COMBINED PATCH-WISE MINIMAL-MAXIMAL PIXELS REGULARIZATION FOR DEBLURRING

Deblurring is a vital image pre-processing procedure to improve the quality of images. It is a classical ill-posed problem. A new blind deblurring method based on image sparsity prior is proposed here. The proposed image sparsity prior combines patch-wise minimal and maximal pixels of latent image, and improves gradually the image sparsity during deblurring. An algorithm that is different with half quadratics splitting algorithm is applied under the maximum a posterior (MAP) framework. Experiment results demonstrate that the proposed method can keep more subtle texture and sharpened edges, reduce the artefacts in visual, and the corresponding evaluated indexes perform favourably against it of the state-of-the-art methods on synthesized, natural and remote sensing images (RSI) quantitatively.


INTRODUCTION
Due to the atmospheric effect (Gerwe et al., 2002), platform high-frequency jitter (Wang and Ma et al., 2018;Ye et al., 2019) and camera defocus (Gajjar et al., 2017), images are somewhat degraded by the blur, in particular for remote sensing images (RSI). Blur leads to degraded images and ruins the considerable useful information of observation. In order to recover the lost information, deblurring is indispensable. In this research, we focus on establishing of image sparsity prior and corresponding model of deblurring for degraded images.
With the assumption that the blur is uniform and spatially invariant, the blurring process can be modelled as a convolution operation (Chan et al., 1998;Perrone and Favaro, 2014) as where Y, I, k and n denote the blurred image, latent (clear) image, blur kernel (which can be seen as point spread function, PSF) and random noise, respectively. The symbol of ⊗ is the convolution operator. The inversion problem of latent image I in Eq. (1) is an ill-posed problem. Based on whether the blur kernel is known or not, deblurring methods to restore the latent image are classified as two types: non-blind deconvolution (NBD) and blind deconvoltuion (BD).
As to BD methods, blur kernel and latent image need to be simultaneously estimated, which leads to more ill-posed problems than NBD methods. Therefore, additional constraints and potential prior knowledge on images and blur kernel are required to make the problems well-posed. Since the pioneering work of Chan et al. (1998) was achieved on employing total variation (TV) as a prior to deblurring, the main successes of deblurring methods have lay on hand-crafted image prior and edge-prediction strategies (Cho and Lee, 2009;Xu and Jia, 2010;Xu et al., 2011). These sophisticated priors contain the gradient sparsity prior (Chan and Wong, 1998;Levin et al., 2009Levin et al., , 2011Perrone and Favaro, 2014;Pan et al., 2014;Zuo et al., 2016), normalized sparsity prior (Krishnan et al., 2011), patch prior (Wen et al., 2019), low-rank prior (Ren et al., 2016), dark or bright channel prior (Pan et al., 2016(Pan et al., , 2017Yan et al., 2017;Cao et al., 2018), a discriminative prior (Li and Pan et al., 2019) and a deep prior (Ren et al., 2019). Furthermore, the following priors of blur kernel are usually adopted, such as gradient sparsity prior (Levin et al., 2009;Pan et al., 2016Pan et al., , 2017Zuo et al., 2016) and spectral prior (Liu et al., 2014). The deblurred images can be estimated by Variational Bayes (VB)-based method (Levin et al., 2009) and MAP-based method. Due to the heavy computational burden of VB-based method, MAP-based method as a popular method has been explored and exploited in recent decade.
Sometimes, single image prior cannot satisfy high requirement of deblurred image quality. Combinations of image priors are employed to achieve better restoration. The image sparsity prior, which is one of the most frequently used priors, is applied to restore degraded images by jointing with priors like low-rank prior (Ren et al., 2016), dark or bright channel priors (Pan et al., 2016(Pan et al., , 2017Yan et al., 2017) and so on. In particular, the dark channel prior, as one of the state-of-the-art methods, has shown promising effectiveness, but still faces with expensive computation and non-rigorous approximation of half quadratic splitting method (Wen et al., 2019). Therefore, Wen et al. (2019) proposed the patch-wise minimal pixels (PMP, which is replaced by PMinP) prior, and the corresponding algorithm avoids the non-rigorous approximation for nonexplicit subproblems and reduce the computation burden.
However, the patch-wise maximal pixels (PMaxP) in clear images are not likely to keep maximal after the blurring process, which is ignored in PMinP prior. Therefore, a new concept of PMaxP prior is defined, and a combined patch-wise minimalmaximal pixels (CPMMP) prior and corresponding regularization method are provided in this research. We briefly illustrate the empirical observation of PMaxP in statistics and exploit corresponding property as a prior to restore the blurred images. For the application of L0-regualrization, a competitive algorithm distinguishing from the classical half quadratic splitting method is recommended.
The rest of this research is organized as follows. In section 2, the definition of PMaxP prior is reviewed and a new image sparsity prior is proposed. In Section 3, a detailed algorithm based on the new image sparsity prior is provided. In Section 4, benchmark dataset, simulated and real blurred GF-2 satellite images, and OVS-A video satellite image are employed to test the performance of the proposed deblurring method. Finally, in Section 5, the major findings are concluded.

Definition of PMaxP prior
The  Similarly, we can define the PMaxP prior of an image I as The symbol of max i Ω denotes the index set of the maximal pixel locations of the i th patch.

Statistic property of PMinPs and PMaxPs
A dataset of over 10,000 natural images from the VGG data (http://www.robors.ox.ac.uk/~vgg/data/) is selected. Every image in this dataset has a blurred version. The blurred images are synthesized by the blur kernel of the dataset (Levin et al., 2009). The histogram of the average number and cumulative probability of PMinPs and PMaxPs for all clear and blurred images are calculated. The histogram of the average number of PMinPs and PMaxPs are shown in Figure 1 (a) and (b). We can find that a larger portion of the PMinPs of clear images are closer to 0, and PMaxPs of clear images are closer to 1 than that of blurred images. Figure 1 (c) and (d) are the corresponding cumulative distribution of the PMinPs and PMaxPs, respectively. It can be found that the PMaxPs is comparable to the PMinPs, and the sparsity of clear images is superior to that of blurred images. As Wen et al., (2019) reported, the PMinps and PMaxPs of clear images (under a threshold such as 0.9) follow a hyper Laplacian distribution and are much sparser than those of blurred images.
Except for experimental statistical results illustrated in Figure 1 Similarly, for the PMaxP prior, we can get the following inequality: Figure 1. Statistics of the minimal and maximal pixels in clear and blurred images over 10,000 natural images, patch size is 35 35 × where "1" presents the maximal value of the PMaxPs. The mentioned inequality property can be easily proven referring to Pan et al. (2016Pan et al. ( , 2017, Yan et al. (2017) and Wen et al. (2019).

Definition of a combined prior
Therefore, we can get a new prior by combining PMinPs and PMaxPs with L0-regularization as This prior can take advantages of both PMinP and PMaxP prior to facilitate the image deblurring problem. A blind deblurring algorithm with this combined prior will be described in next Section.

DEBLURRING ALGORITHM BASED ON THE COMBINED PRIOR
Based on the MAP framework, the objective function is given by where γ and μ are positive weight parameters, and L is a data fidelity term, which confines k I ⊗ to be consistent with the blurred image Y. In order to solve the ill-posed problem, H and R are the priors of the blur kernel and the latent image, respectively.

The Proposed Flowchart
The workflow of CPMMP prior regularization method is illustarated in Figure 2 and Figure 3. The deblurring process contains two main steps: blur kernel estimation and nonblind deconvolution. Figure 2, the process in red rectangle presents the blur kernel estimation. Figure 3 illustrate the i th -level of blur kernel estimation process. 1) Achieve the image prior according to the definition of CPMMP prior in section 2. 2) Compute a corase latent image with the prior by limited iterations. 3) Obtain the accurate blur kernel in i th level of pyramid with the gradient freature of Y(i,0) and latent image I(i,n) (I(i,n)=Y(i,n)). 4) Get the initilized blur kernel of the next level pyramid by upsampling. Figure 2, the blue arrow presents the final step of the deblurring process, i.e., once the final blur kernel is estimated, a classical nonblind decovlution method is implemented to restore the latent image with the estimated blur kernel and input image.

The Proposed Model for Deblurring
On account of the sparsity of the gradient of clear images, R(I) is usually chosen as L0-norm regularizer of I ∇ (the gradient of I), and the loss function L and the regularization term H is select L2-norm. This yield 2 2 The deblurring process is modelled as an optimization problem by alternatively solving the latent image I in the 1 st and 3 rd items of Eq. (7). Introducing the proposed sparsity promotion of CPMMP (see Eq. (5)), the Eq. (7) is rewritten as where α and β are positive weight parameters. The latent image I and blur kernel k can be estimated by the half quadratics splitting algorithm. Motivated by Wen et al. (2019), we employ an efficient algorithm which can avoid non-rigorous approximation.

Optimization Algorithm
Firstly, the blur kernel k is fixed, a soft-threshold method proposed by Wen et al. (2019) is employed to update I by the constraint of CPMMP prior in Eq. (5). Then, fix the latent image I, one step augmented Lagrange method is applied to update blur kernel. The detailed algorithm for estimating I and k is described in the following subsections.

Estimating Latent Images:
Given an interim estimation of the blur kernel k (t-1) as the current estimation at (t-1) th level of pyramid, the problem on latent image I is formulated as ( ) The expression can be diagrammed in 2 nd step shown in where the symbols of (·) and  -1 (·) represent the FFT and inverse FFT operators.

( )
  presents conjugate transpose operator. The objective formulation (22) is in spatial domain. In order to utilize the gradient features of latent images, the equivalent objective formulation in gradient space is,

Estimating Blur
The blur kernel can be efficiently obtained by FFT as follows: The blur kernel estimation needs to satisfy the non-negative constraint and the normalization constraint. Furthermore, a coarse-to-fine multi-scale strategy (Cho, 2009;Pan et al., 2014) illustrated in Figure 2, is employed to make the blur kernel estimation more robust and efficient.
The main steps of latent image I and blur kernel k estimation for blurred images are summarized in Algorithm 1.  (13) and (14) Update , The key step in Algorithm 1 contains two loops. The inner loop enforces smooth result with fewer artefacts in restored image, and the outer loop reduce the artefacts generated by the sparsity prior.

EXPERMENTAL RESULTS AND DISCUSSION
A benchmark dataset of blurred images (Levin et al., 2009) is used to test the performance of the proposed method. Two quantitative metrics, i.e., peak signal-to-noise ratio (PSNR), structural similarity (SSIM) (Wang et al., 2004), are employed here. In order to validate the superiority of the proposed for RSIs, simulated blurred RSIs with deterministic blur kernel in Levin et al. (2009) and real RSIs are used. The real RSIs include two blurred RSIs from GF-2 satellite captured on the second day after launch, and a motion blurred video sequence from commercial satellite OVS-A (Cao et al., 2018). The noreference metric linear regression (LR) index (Liu et al., 2013) is used evaluate the real data experiment results.
Parameters are set as

Benchmark Dataset Experiments
The dataset from Levin et al. (2009) contains 32 blurred samples corresponding to 4 clear images and 8 blur kernels. Figure

Simulated RSI Experiments
In order to validate the performance of the proposed CPMMP prior regularization method on RSIs, a group of blurred RSIs with known blur kernels are simulated. A GF-2 multi-spectral image is selected as shown in Figure 6 (a). We convolute the blur kernels in Levin et al. (2009) with the GF-2 image, and add small random noise to obtain the blurred RSIs, which contains 8 blurred samples corresponding to 8 blur kernels. For example, the 2 nd blurred image is shown in Figure 6 (b). The blurred image suffers the loss of surface features and texture, see the zoomed regions in red rectangles.
After deblurred by the means based on DCP prior, ECP prior, PMinP prior and the proposed CPMMP prior, the details of the restored images reconstructed from different methods are shown in Figure 6 (c)~(e). The zoomed part of the proposed method is smoother than the other three methods compared to an RGB image from Google Earth. The restored images are evaluated with PSNR, as shown in Figure 7. We can find that the proposed method is with the highest PSNR for almost all images and with the highest average PSNR.

Real-Data Experiments
For the real data experiments, two typical blurred data scenarios are selected: 1) focus adjusting of GF-2 satellite. The GF-2 satellite need refocusing after arriving the designed orbit. The GF-2 satellite image shown in Figure 8 (a) was captured on the second day after its launch, which is a typical performance of refocusing. Four different deblurring methods are used to restore the defocusing image. The sharpness of the surface features can be enhanced and a great deal of textures can be shown visually. In particular, as shown in Figure 8 (b)~(e), we can find that the house edges zoomed in red rectangle are sharper. There are least artefacts exist in Figure 8(e), while the other restored images somewhat have repetitive house edges in Figure 8(b)~(d). The LR values of the four methods are listed in Table 1. It is obvious that the proposed method obtains the largest value, which indicates that the restored performance of the proposed method is the best. Furthermore, in Table 2, the runtime of CPMMP can be saved compared to DCP and ECP due to the traversal by patches in CPMMP instead of the traversal by pixels in DCP and ECP, but be inferior to PMinP, because the equivalent regularized prior items (2) and (3) is implemented not parallelly. The runtime of the prior item (2) and (3) are 24s and 23s in theoretically statistics, if (2) and (3) are implemented parallelly to save at least 23s. The efficiency of CPMMP may be improved by the multi-processing of the CPU. 2) Motion Blur from Video Satellite. An OVS-A RGB image located at (14.9610°W, 37.4630°N) is extracted from a video sequence data as illustrated in Figure 9 (a). The satellite uses the area-array sensor, which includes motion in video mode instead of typical TDI (time delay integration) line-array sensor. The blurred video sequence was caused by the mismatch of the velocity before the stabilization of the orbit (Cao et al., 2018).
In Figure. 9, after deblurring using different methods, the clear video can be recovered well. The zoomed region in Figure 9 (e) from CPMMP shows more details than the others. Visual inspection finds that many surface features of the other restored images are homogenized to reduce the representation on the texture. In addition, as displayed in the third row of Table 1, LR value of CPMMP is the highest one among all methods. Meanwhile, CPMMP has advantages over DCP and ECP in computational efficiency. In summary, the proposed CPMMP performs favourably against the three involved state-of-the art methods here for real RSIs.

CONCLUSION
In this research, we find that patch-wise maximal pixels (PMaxP) in clear images will lessen, and the corresponding sparsity will become weaker after the images are blurred. By combining the PMaxP with PMinP, a new prior named CPMMP is proposed to recover the latent images for the blur kernel estimation. Based on the MAP framework and coarse-to-fine optimization strategy, an algorithm discriminating with half quadratics splitting algorithm is carried out. It is more efficient than the existing intensity prior based algorithms, because the proposed CPMMP prior can impose flexibly sparsity promotion during the deblurring procedure. The proposed method can preserve more subtle texture and sharpened edges, and reduce artefacts than other three methods during the experiments for benchmark dataset, panchromatic or multi-spectral GF-2 images and satellite video sequence data.