Abstract
Restoration of blurred images corrupted with impulse noise has attracted increasing attention due to its great potential toward practical applications. To guarantee the noisy image deblurring performance, total variation (TV), one of the most popular regularizers, has been extensively exploited in the literature. This local regularizer is known for its capability of preserving edges and penalizing oscillations but not sharp discontinuities. However, TV regularized variational models commonly suffer from undesirable staircaselike artifacts in homogeneous regions. The reason behind this phenomenon is that TV regularizer favors solutions that are piecewise constant in numerical experiments. To overcome the modeldependent deficiency, we proposed to replace the commonly used local regularizer by its nonlocal extension, i.e., nonlocal total variation (NLTV) regularizer. By taking advantage of the high degree of selfsimilarity within images, the proposed nonlocal variational model is able to remove both blurring and impulse noise while preserving meaningful image details. To guarantee solution stability and efficiency, the resulting nonsmooth image restoration problem was effectively dealt with using an alternating optimization algorithm. In addition, to further enhance the image quality, a local variance estimatorbased automatic method was introduced to calculate the spatially adapted regularization parameters during image restoration. Extensive experiments have demonstrated that the proposed method could achieve superior imaging performance compared with several stateoftheart image restoration methods.
1. Introduction
1.1. Background and Related Work
Realworld images are often degraded by both blurring and additive/multiplicative noise during image acquisition and analogtodigital conversion. Over the past decades, restoration of such images has attracted increasing attention in the fields of image processing, such as astronomical imaging [1], remote sensing [2], biomedical imaging [3], and many others [4].
Image restoration is a typical illconditioned inverse problem, which requires regularization techniques to guarantee a unique and stable solution. In the current literature, the classical regularization for image restoration is a quadratic regularization term or Tikhonov regularization [5]. However, the restoration results often suffer from oversmoothing of important signal features (e.g., edges and textures in images) thus compromising the image visual quality. To overcome this limitation, total variation (TV) regularizer, first proposed by Rudin et al. [6] for Gaussian noise removal, has become one of the most popular regularization tools due to its ability to preserve edges and penalize oscillations but not sharp discontinuities [7, 8]. In addition, TV regularized variational models have also been extensively used for removal of impulse [9], Poisson [10], Rician [11], and Speckle (Gamma) [12] noises.
In the case of images degraded by both blurring and additive Gaussian noise, TV regularized L2 variational models based on the Maximum A Posteriori (MAP) framework have been developed to enhance image quality [13]. In particular, the models were proposed by combining the squared L2norm datafidelity term and TV regularizer. However, these models often tend to cause staircaselike artifacts due to the nature of TV in favoring piecewise constant solution. To overcome the modeldependent deficiency, many extensions of TV have been reported in the current literature. Secondorder total variation (TV^{2}) regularizer [14, 15] was proposed to replace the TV regularizer to suppress the undesirable staircaselike artifacts. Many efforts have been made to combine both TV and TV^{2} regularizers to further improve image quality [16, 17]. Motivated by the observation that gradients of natural images follow a hyperLaplacian distribution [18], the nonconvex hybrid TV regularizer has been developed based on the combination of nonconvex TV and TV^{2} regularizers [19]. From a statistical point of view, the nonconvex regularizer is well capable of edgepreserving image restoration. The secondorder total generalized variation (TGV) regularizer [20–22] has also been proposed to remove the staircaselike artifacts yielded by TV in recovered images. All of these traditional regularizers are developed based on local image operators, which are capable of preserving edges and reducing noise very well but may not perform well in preserving fine texture because texture is nonlocal in nature [23]. Inspired by the concepts of nonlocal means (NLM) [24] and graph Laplacian [25], Gilboa and Osher [26] defined a variational framework based on the nonlocal operators. The corresponding nonlocal total variation (NLTV) regularizer [27, 28] has gained great success in image restoration when images are degraded by both blurring and additive Gaussian noise. In particular, the NLTV regularizer is capable of taking advantage of the high degree of selfsimilarity within images. It is able to preserve the meaningful image details while removing both blurring and impulse noise in practice.
In this paper, we mainly focus on the variational models for restoring blurred images with impulse noise. Thus, the abovementioned variational models with squared L2 datafidelity term cannot be used directly to deal with the image restoration problems addressed in this paper. To restore blurred images with impulse noise, TV regularized L1 variational (TVL1) models have attracted considerable attention during recent years [29–31]. In particular, both L1norm datafidelity term and TV regularizer were combined in these variational models. Besides, in [32, 33], the authors successfully extended the TVL1 model to deblurring multichannel images corrupted by impulse noise. On the other hand, to recover the image with pixel values in the same range, a simple but useful box constraint was incorporated into the TVL1 model to further enhance image quality [34, 35]. Hintermüller and RinconCamacho [36] developed a spatially adapted regularization parameter choice rule in the TVL1 model where small details were desired to be preserved. However, these proposed models still tend to cause staircaselike artifacts since TV favors a piecewise constant solution. More recently, Liu et al. [37] proposed a spatially adapted variational model by replacing TV with a hybrid TV (TV^{1,2}), which was based on the combination of both TV and TV^{2} regularizers.
1.2. Motivation and Contributions
It is well known that nonlocal selfsimilarity property has been widely exploited in the fields of both low and highlevel computer vision [38–40]. This property is based on the observation that image patches in natural images tend to redundantly repeat themselves many times [41]. Therefore, if images are degraded by blurring and impulse noise, the variational models based on local regularizers (e.g., TV and TV^{1,2}) may not perform well in preserving fine textures in restored images. To guarantee high image quality, it is necessary to incorporate the nonlocal regularizer into the variational image restoration model. Inspired by the previous works in [27, 28, 42, 43], we tend to propose a NLTV regularized L1 variational model by replacing squared L2norm datafidelity term with L1norm datafidelity term to handle blurred images with impulse noise.
On the other hand, the regularization parameter plays an important role in TV regularized variational models. Since images are always comprised of image features of different scales, the constant parameter can lead to significant degradation of the image quality [44]. Thus, to achieve high image quality, the regularization parameter should be selected adaptively according to local image features. As discussed in [36, 44–46], the authors proposed to develop a local variance estimatorbased automated method for selection of spatially adapted regularization parameter. Similar to these previous works, we will introduce a spatially adapted regularization parameter selection scheme for our proposed nonlocal variational model. To the best of our knowledge, no research has been conducted on NLTV regularized L1 variational model with spatially adapted regularization parameter selection thus far.
Due to the nondifferentiability of NLTV regularizer and nonsmoothness of L1norm datafidelity term, it is difficult to solve our proposed nonlocal variational model through the commonly used optimization algorithms. In the current literature, the alternating optimization algorithm [29] has been widely used for image restoration problems and achieved satisfactory results on optimization performance. Motivated by these previous works, to get a numerically stable solution, we will develop an alternating optimization algorithm to solve the proposed image restoration model. In particular, the original nonsmooth optimization problem will be decomposed into simpler subproblems by introducing several intermediate variables. Each of these subproblems has a closedform solution or could be efficiently dealt with using a simple numerical method.
In conclusion, the main ideas and contributions of this paper, given the current stateoftheart research work, can be summarized as follows:(1)A spatially adapted nonlocal variational model is proposed to restore blurred images with impulse noise. This model considers the image nonlocal selfsimilarity property, and the spatially adapted regularization parameter is selected automatically to enhance image quality.(2)An alternating optimization algorithm is developed to optimize the proposed nonlocal variational image restoration model, in which the optimization of the image restoration model is decomposed into two subproblems. Each subproblem has a closedform solution or can be efficiently solved using existing optimization algorithms.(3)The experimental results on several grayscale images have shown the superior performance of our proposed method in terms of both quantitative and visual quality evaluations. It may well provide a new idea for restoration of blurred images with impulse noise.
The main benefit of our proposed method is that it takes full advantage of the image nonlocal selfsimilarity property and spatially adapted regularization parameters. Therefore, the proposed method is capable of restoring degraded images while preserving edges and fine details in practice.
1.3. Organization
The remainder of this paper is organized into several sections. In Section 2, the image degradation model is introduced, and following this, the MAPbased image restoration framework is presented. The NLTV regularized L1 variational model for restoration of noisy blurred images is specifically developed in Section 3. In Section 4, an alternating optimization algorithm is developed to effectively solve the resulting nonlocal regularized image restoration problem. To further improve image quality, a local variance estimatorbased automatic method for selection of spatially adapted regularization parameter is introduced in Section 5. Section 6 presents numerous experiments using several grayscale images, which verify the effectiveness of the proposed method. Finally, we conclude this paper by summarizing our contributions and discussing the future work in Section 7.
2. Problem Formulation
2.1. Image Degradation Model
Images are often degraded by blurring and different kinds of noise. In this paper, we will concentrate mainly on the impulse noise, such as saltandpepper noise and randomvalued impulse noise. We denote by the image domain, which is often considered to be a bounded and piecewise smooth open subset. Let be a real function defined on . The corrupted image is then given bywhere denotes the image degradation by impulse noise. For the sake of simplicity, we assume that the linear and continuous blurring operator is known beforehand. Denote the dynamic range of to be , i.e., , the degradation model is defined as follows [47].(i)Saltandpepper noise: the gray level of at pixel location is where determines the level of the saltandpepper noise. The gray values of degraded pixels change to either the minimum value or the maximum value .(ii)Randomvalued impulse noise: the gray level of at pixel location is where is the level of the randomvalued impulse noise. The gray values of are identically and uniformly distributed random numbers in .
Recently, many variational image restoration models have been developed by combining the nonsmooth L1 datafidelity term with different regularizers. In this paper, we will restrict our attention to the local TV regularizer and its nonlocal extension. The corresponding MAPbased image restoration model will be proposed in the next subsection.
2.2. MAPBased Image Restoration Model
The MAP estimation theory, which inherently includes prior constraints in the form of prior probability density functions, has attracted increasing attention in the field of image processing, computer vision, and medical imaging. For a given degraded image , the aposteriori likelihood function via Bayesian modeling is given by . To determine an approximation to the unknown image , the MAP estimator can be used to minimize the negative loglikelihood function, i.e.,where the additive term is neglected due to its independence to . In practice, the probability density should be assumed suitably according to the model of the noise in the given image . In the case of impulse noise, the negative loglikelihood function is commonly defined as a L1 datafidelity term, i.e., . To stabilize the image restoration, it is necessary to incorporate the additional prior information via the apriori probability density function (PDF) into the restoration process. The most frequently used PDFs are Gibbs functions in current literature which satisfywhere is a positive scale parameter and denotes a regularization functional. If the Gibbs apriori PDF equation (5) is incorporated into the MAP framework equation (4), minimizing the negative loglikelihood function is equivalent to solving the following minimization problem:where is the regularization parameter which controls the tradeoff between the datafidelity and regularization terms. In [29, 33], the regularization term is selected as a TV prior, i.e., . However, the results often tend to cause staircaselike artifacts since TV regularizer favors a piecewise constant solution in bounded variation (BV) space. The undesirable solutions could introduce false edges that do not exist within the true image and then fail to guarantee high visual quality. To improve the image restoration capability, Liu et al. [37] proposed to replace TV by TV^{1,2}, i.e., , where denotes the gradient dependent weight. Essentially, both TV and TV^{1,2} are local regularizers, which have constrained the further performance improvement in image restoration. There is a significant potential to incorporate the nonlocal regularizer into the image restoration framework equation (6). In particular, the nonlocal regularizer is well capable of taking full advantage of the pattern redundancy and selfsimilarity within an image [24]. Thus, it can generate superior restoration performance compared with local regularizers. In next section, we will propose a nonlocal regularized L1 variational model for restoration of blurred images corrupted by impulse noise.
3. NLTV Regularized L1 Image Restoration Model
3.1. Nonlocal Total Variation Functional
The neighborhood filter NLM, first proposed by Buades et al. [24] for Gaussian noise removal, has attracted increasing attention in recent years. The underlying idea of NLM is to make full use of the nonlocal similar patches within an image. To further enhance image quality, Gilboa and Osher [26] have extended TV to NLTV by combining the concepts of NLM and graph Laplacian. In the following, we will introduce some definitions and notations regarding the NLTV functional [26, 27, 48] that will be used throughout this paper. Let denote the image domain. Given a reference image , the similarity between two pixels can be measured using a weight function , which satisfieswhere of size denotes the neighborhood window, is the Gaussian kernel with standard deviation , and is a positive smoothing factor to control the degree of filtering. Let be a real function . Given the symmetric weight function from a reference image , the nonlocal gradient at point is defined as follows:where with of size denoting the search region. The magnitude of at point is then obtained:
Let denote the nonlocal divergence operator, which is the adjoint of the nonlocal gradient, i.e., . The divergence for a nonlocal vector can be defined as follows:which satisfies the conjugation with the nonlocal gradient, i.e., . Thus, the nonlocal Laplacian operator and NLTV regularizer can now be, respectively, defined byand
NLTV regularizer has achieved significant success due to its weighted nonlocal gradient. In conventional local regularizers (e.g., TV [6], TV^{2} [14, 15], TGV [20], and TV^{1,2} [17, 37]), the gradient is calculated using only a few neighbors of the target pixel; whereas all pixels are considered as the neighbors in NLTV regularizer. More detailedly, weights in local regularizers are applied equally for the gradients of all pixels within an image, whereas the weights in NLTV regularizer are spatially adaptive according to the similarities between the neighborhoods of the targeting and neighboring pixels [24].
3.2. Nonlocal Regularized L1 Variational Model
Due to the significant advantage of NLTV regularizer, it is necessary to incorporate NLTV into the MAPbased image restoration model equation (6). In this case, we assume that the original image follows a Gibbs prior of the form:
Using the MAP estimation theory introduced in Section 2.2, the minimization problem equation (6) can be rewritten as follows:where is a predefined regularization parameter. This variational image restoration model (termed NLTVL1 for short) consists of two terms: the first term, called the L1 datafidelity term, denotes a measure of the difference between the restored data and the observed version. The second term is the NLTV regularizer which can stabilize the restoration result. Due to nonsmooth natures of these two terms, it is intractable to efficiently solve the unconstrained minimization problem equation (14) via commonly used numerical methods. To achieve a robust and efficient solution, an alternating optimization algorithm will be developed to solve the NLTVL1 model.
4. Numerical Optimization Algorithm
In this section, we propose to develop an alternating optimization algorithm for solving the NLTVL1 model equation (14). The main advantage of this algorithm is that the two nonsmooth terms in equation (14) can be effectively decoupled. In each step of this algorithm, the corresponding subproblem can be efficiently solved through existing optimization algorithms. In particular, we will deal with the L1related minimization problem in the first step via a soft shrinkage operator. To further enhance the image quality, the second step related with NLTV regularized L2 deconvolution will be implemented using the preconditioned Bregmanized operator splitting (PBOS) proposed by Zhang et al. [27].
4.1. Overview of PBOS
We tend to give a brief overview of PBOS used in this paper. Consider the following unconstrained optimization problem:where is a predefined regularization parameter and denotes a general convex regularizer, such as TV [6], TV^{2} [14, 15], TGV [20], TV^{1,2} [17, 37]), and NLTV [27]. By combining the Bregman iterative scheme [49] and forwardbackground operator splitting technique [50], the Bregmanized operator splitting (BOS) method is introduced to effectively solve equation (15) as follows:for a positive parameter . The superscript denotes the transpose (conjugate transpose) operator for real (complex) matrices or vectors. In this paper, we mainly focus on the NLTV regularizer, i.e., . However, the image restoration performance is highly dependent on the updating of weight function in equation (7). In the case of image denoising, can be estimated accurately because the noisy image still contains the most image similarity information. In contrast, it is difficult to achieve the satisfactory for image deblurring due to loss of structural similarities in blurred image. To overcome this limitation, Lou et al. [51] and Zhang et al. [27] proposed to estimate from the preconditioned image generated by Tikhonov’s regularization method. Although the preconditioned image would suffer from noise amplification, the main geometric structure preserved will be beneficial for the accurate estimation of since the updating scheme equation (7) is insensitive to noise. Thus, the preconditioned Bregmanized operator splitting (PBOS) for solving equation (15) can be defined as follows:with , where is a small regularization parameter to stabilize the solution of subproblem in equation (17).
4.2. Alternating Optimization Algorithm for the NLTVL1 Model
We introduce one auxiliary variable in equation (14), which can be written as an equivalent unconstrained optimization problem as follows:where is a penalty parameter. It could be easily proved that, as , the solution of equation (18) is convergent to the solution of equation (14). However, when is not so large, the approximation in equations (18) to (14) is inaccurate in practice. We note that equation (18) can be rewritten as follows:
In this work, the solution of equation (18) can be obtained by applying the alternating optimization algorithm. Staring from an initial guess and , we compute the sequencevia the following formulas:
Now, we alternatively solve equation (18) with respect to and until the solution converges to an optimal solution.
4.2.1. Solving the Subproblem Equation (21)
The first step of the alternating optimization algorithm is to perform the L1 minimization problem. For fixed , the solution of subproblem in equation (21) is equivalent to solving the following minimization problem:where and . The exact minimizer of can be obtained by a soft shrinkage operator [52, 53] as follows
Therefore, the solution of the optimization problem equation (21) can be obtained using , i.e.,
4.2.2. Solving the Subproblem Equation (22)
The last step is to apply a nonlocal variational model to recover the image generated by the previous step. For fixed , the solution of subproblem in equation (22) is equivalent to optimizing the following NLTV regularized L2 minimization model:
As introduced in Section 4.1, PBOS can be directly adopted for NLTV regularized L2 minimization problem. We first replace by , the solution of subproblem equation (26) can be achieved by considering the following alternating iteration scheme:for . The variable satisfies the following updating scheme, i.e., and . Both the first and third steps in equation (27) can be updated easily, but it is difficult to effectively update the second step due to the nonsmooth nature of NLTV regularizer. In the current literature, the split Bregman method [54] has been successfully used in NLTVbased multiplicative noise removal [48]. To apply this method, we first replace by and add a penalty function term. The solution of subproblem in equation (27) can be achieved by considering the following split Bregman iteration equation:with and . Here, is a penalty parameter and the variable is updated using for . The variable can be obtained for updating the third step in equation (27).
The subproblem in equation (28) is in essence a convex L1 minimization problem:which can be explicitly computed using the following soft shrinkage operator [52, 54], i.e.,
This shrinkage operator is extremely fast and requires only a few operations per element of . Given the fixed , , and , solution of the subproblem in equation (28) is equivalent to solving the following minimization probem:
We can easily obtain the optimal solution because the subproblem is both smooth and differentiable. In particular, the solution consists in solving the following system of linear equation:which provideswhere denotes the identity element. Since the nonlocal Laplacian is negative semidefinite, the operator is diagonally dominant with nonlocal weight . To ensure optimal efficiency, the widely used GaussSeidel method can be adopted to achieve an approximate solution of equation (33). This method is capable of guaranteeing a fast convergence. The implementation of PBOS for equation (26) is described by the following pseudocode shown in Algorithm 1.

4.3. Optimization Procedure
As described in Section 4.2, an alternating optimization algorithm was proposed to decompose the original NLTVL1 minimization problem equation (14) into two simpler subproblems, i.e., subproblem equation (21) and subproblem equation (22). Each subproblem has a closedform solution or could be efficiently handled using existing optimization algorithms. In particular, the subproblem was solved by a soft shrinkage operator. Solution of the subproblem was achieved using PBOS. In conclusion, Algorithm 2 shows the pseudocode of our alternating optimization algorithm for the NLTVL1 minimization problem equation (14).

5. Automatic Selection of Spatially Adapted Regularization Parameter
It is well known that the regularization parameter plays an important role in image restoration. In particular, larger easily leads to little noise reduction; whereas smaller will result in oversmoothing of texture regions. Since images are always comprised of image features of different scales, the constant parameter will degrade the image visual quality [44]. Therefore, to achieve satisfactory restoration performance, the regularization parameter should be selected adaptively according to the local image features. Roughly speaking, the smaller parameter enables effective noise reduction in homogeneous regions (with largescale features); whereas larger one performs well in fine detail preservation in texture regions (with smallscale features). In [36, 44–46], many efforts have been made to take full advantage of the local image features and achieved remarkable restoration results. To the best of our knowledge, no research has been conducted on NLTVL1 model with spatially adapted regularization parameter selection thus far. In this section, we will investigate how to adaptively select the regularization parameter for our proposed NLTVL1 model in terms of both saltandpepper and randomvalued impulse noise.
Motivated by the previous works in [36, 44–46], we employ the similar automated method in spatially adapted regularization parameter selection for our NLTVL1 model equation (14). Before the selection step is performed, we define a local window centered at pixel as follows:and assume that is a local window filter, i.e.,with and . In this paper, the local expected absolute value estimator is defined as follows:for . Based on the formula in (36), we can obtain the following NLTV regularized minimization problem with local constraints:where “a.e.” stands for “almost everywhere,” and denotes the expected absolute value that depends on the type of noise [36]. It should be noted that the constrained minimization problem (37) is related to an unconstrained optimization model with spatially adapted regularization parameters as follows:
For fixed regularization parameter , the minimization problem (38) can be solved via the alternating optimization algorithm given in Algorithm 2. Similar to the previous works in [36, 44–46], we propose the updating scheme for the regularization parameter as follows:where is a step size, is a constant parameter adopted in this paper, is a large positive parameter to ensure that stays bounded, and is the current estimate of the latent image . More details about the determination of the parameter and expected absolute value in equation (39) will be discussed, respectively, in Sections 5.1 and 5.2.
5.1. Selection of in the Case of SaltandPepper Noise
In the case of saltandpepper noise, we are able to obtain the expected absolute value for any , where is the noise level defined in equation (2). To keep the updated at the same scale as , the parameter is set as as done in [36, 44]. Therefore, for given recovered image and intermediate variable , we can define the following scheme to update the regularization parameter as follows:where is the current estimate of the latent image which is obtained by implementing the Algorithm 2 with fixed . According to the iterative update of in equations (41)–(43), we can obtain the NLTVL1 model with spatially adapted regularization parameters for restoring blurred images with saltandpepper noise given by Algorithm 3.

5.2. Selection of in the Case of RandomValued Impulse Noise
Analogous to the saltandpepper noise, is set as to keep the updated at the same scale as . For a given associated with a recovered image , the current expected absolute value can be calculated as follows where for any . This spatially adapted expected absolute value is significantly more complex than the constant value in equation (41). Then, we propose the updating scheme of as follows:
For this type of noise, the recovered image can be obtained by implementing the Algorithm 2 with fixed . As a conclusion of this subsection, the NLTVL1 model with spatially adapted regularization parameters for restoring blurred images with randomvalued impulse noise can also be found in Algorithm 3. For more details on automatic selection of spatially adapted regularization parameters in terms of both saltandpepper noise and randomvalued impulse noise, we refer the interested reader to see [36] and references therein. Owing to the proposed NLTVL1 model and automatic selection technique, the new image restoration version developed in this paper is called spatially adapted NLTVL1 algorithm (termed SANLTVL1 for short).
6. Experimental Results and Discussion
In this section, numerical results are presented to illustrate the superior performance of our proposed SANLTVL1 for restoring blurred images with impulse noise. We compare the proposed method with several stateoftheart image restoration techniques on different numerical experiments. All image restoration methods are implemented using Matlab (The MathWorks, Natick, Inc., MA) on a machine with 3.10 GHz Intel Core i52500 CPU and 4 GB RAM. Eight different grayscale images (with size ) used throughout this paper are displayed in Figure 1. Using the Matlab builtin function fspecial, both Disk and Gaussian blur kernels with size , visually illustrated in Figure 2, are generated in our experiments. In particular, these two blur kernels are fspecial (‘disk’, 7) and fspecial (‘gaussian’, [15, 15], 3), respectively. To objectively evaluate the restoration performance, the quality of restored images by different methods is compared quantitatively using three popular assessment methods, i.e., peaktonoise ratio (PSNR), structural similarity (SSIM) index [55], and feature similarity (FSIM) index [56].
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(a)
(b)
6.1. Comparison with Other Image Restoration Techniques
The proposed method will be compared with five image restoration techniques as follows:(1)TVL1: TV Regularized L1 Variational Model [29]. TVL1 was proposed by combining the L1 datafidelity term with TV regularizer. In particular, this model is typically capable of preserving edge features without amplifying the impulse noise. In this paper, TVL1 was solved using an alternating minimization algorithm.(2)NLTVL1: NLTV Regularized L1 Variational Model. This model takes full advantage of the natural redundancy of patterns within an image. Thus it is capable of suppressing the staircaselike artifacts often yielded by TVL1 in recovered image. NLTVL1 was solved using the alternating optimization algorithm given in Algorithm 2.(3)SATVL1: Spatially Adapted TV Regularized L1 Variational Model [36]. This method is an improved version of the TVL1 by introducing an automated method for selection of spatially adapted regularization parameter. For the sake of comparison, the corresponding TVL1 model was also solved using the alternating minimization algorithm proposed in [29].(4)SATVL1: Spatially Adapted TV^{1,2}Regularized L1 Variational Model [37]. Compared with SATVL1, Liu et al. [37] proposed to replace TV by TV^{1,2} to further improve the restoration performance. In this paper, the alternating direction method of multipliers (ADMM) was adopted to efficiently solve the resulting optimization problem.
The first two variational methods (i.e., TVL1 and NLTVL1) were implemented by using only constant regularization parameter. In contrast, SATVL1 and SATV^{1,2}L1 take full advantage of the spatially adapted regularization parameter. However, these three variational methods were proposed based on local regularizers, which could constrain the further performance improvement in image restoration. In contrast, the proposed SANLTVL1 takes into consideration both nonlocal regularizer and spatially adapted regularization parameters. These two properties are capable of enhancing the performance of our image restoration method.
6.2. Optimal Parameter Selection
Nonlocal variational image restoration techniques are dependent on several parameters, i.e., , , and . In the current literature, many efforts have been made to investigate the influence of these parameters on image restoration [27, 57, 58]. To the best of our knowledge, no research has been conducted on the NLTVL1 model for restoring blurred images with impulse noise. It is generally thought that the patch size is less important (common choices are between 5 and 9) [58]. Without loss of generality, we set the patch size to be throughout the rest of this paper. To keep a good balance between computational complexity and restoration performance, we only select the 10 best and 4 nearest neighbors for each pixel in the searching window [27]. The searching window size is empirically set as as conducted in [27]. The smoothing parameter is also an important parameter that we will study more carefully. To estimate the optimal parameter , manual experiments are implemented on the eight different grayscale images, shown in Figure 1. In our experiments, the images are corrupted by two different blur kernels and different levels (i.e., and ) of impulse noise ranged from to with in step.
In Figure 3 and Figure 4, the optimal value of for different images was determined using the image quality measures in terms of PSNR and SSIM. It can be observed that the optimum gradually becomes larger as noise level increases. The relationship between and can be expressed as a linear approximation formula, i.e., . The fitting coefficients and can be estimated empirically from Figures 3 and 4. To illustrate, the approximation results are summarized in Tables 1 and 2. It can be observed that most linear approximation formulas have the similar coefficient which is more important than in practice. For the sake of simplicity, we, respectively, obtained the mean version for saltandpepper and randomvalued impulse noise by averaging the formulas in Tables 1 and 2, i.e.,where and denote the noise levels of saltandpepper and randomvalued impulse noise, respectively. The other parameter values in our numerical experiments were empirically set as , , , . was adaptively calculated using the automatic parameter selection technique introduced in Section 5. The maximal iterations and in Algorithms 1 and 2 were set as 5 for all examples since the algorithms could attain a steady state. We determined the iterative scheme in Algorithm 3 by the following stopping criterion:or the maximal number of iterations was reached. Here, denotes the recovered image at the iteration of the proposed method. All other image restoration methods were implemented with their optimum default parameter setting in numerical experiments. We refer the interested reader to see [29, 36, 37] for more details on the optimal setting of parameters.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
6.3. Quantitative Comparison of Restoration Performance
This section is devoted to compare SANLTVL1 with some current stateoftheart image restoration techniques. To evaluate the stability of our restoration method, four different images were corrupted by different Gaussian blur kernels and saltandpepper/randomvalued impulse noise with different ranged from to with in step. To evaluate the image restoration performance, both PSNR and SSIM metrics were used simultaneously. Figures 5 and 6 depict the quantitative results. Essentially, both NLTVL1 and SANLTVL1 are nonlocal variational methods which take full advantage of the natural redundancy of patterns within an image. The nonlocal property is reliably capable of noise suppression and detail preservation during image restoration. Thus both NLTVL1 and SANLTVL1 can yield the most satisfactory restoration results compared with other local variational methods (i.e., TVL1, SATVL1, and SATV^{1,2}L1). Among these local methods, SATV^{1,2}L1 outperforms other local methods under consideration. In particular, TV^{1,2} can suppress the staircaselike artifacts generated by TV in recovered images. Compared with TVL1, the image quality can be further improved using the spatially adapted regularization parameter for SATVL1. The superior performance of our SANLTVL1 also benefits from the spatially adapted regularization parameter. As shown in Figures 5 and 6, the advantage of SANLTVL1 over other competing methods increases with higher noise level in terms of both PSNR and SSIM.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
6.4. Qualitative Visual Quality Assessment
6.4.1. Influence of Nonlocal Regularizer
Qualitative visual quality comparison plays a crucial role in evaluating the performance of an image restoration method. In particular, the recovered image should contain important geometrical structures, few (or ideally) no visible artifacts. In this subsection, we will investigate the influence of nonlocal regularizer on image restoration performance. Both “Barbara” and “Boat” were selected to evaluate the visual quality of restoration results. As shown in Figures 7 and 8, these two images were corrupted by type II blur and different levels of saltandpepper/randomvalued impulse noise ranged from to with in step. The smoothing parameter for both NLTVL1 and SANLTVL1 was determined by the linear approximation formula introduced in Section 6.2. Some of the visual results in Figures 7 and 8 show that both TVL1 and SATVL1 suffer from some disadvantages such as undesirable staircaselike artifacts and blockylike structures. This is mainly due to the fact that TV favors piecewise constant solutions. SATV^{1,2}L1 could further enhance the image quality owing to the TV^{1,2} regularizer and spatially adapted regularization parameter. However, the recovered results easily produce blurrededge effects which significantly degrade the visual quality. In contrast, NLTVL1 could achieve the most satisfactory visual quality in terms of noise/blur removal and detail preserving. The advantage of NLTVL1 was further confirmed by the local magnification views of regions of interest in recovered images shown in Figures 9 and 10. NLTVL1 performed well on these two images, and produced more similar geometric structures to those of the original version. The superior performance of NLTVL1 benefits from the selfsimilarity properties within an image. The qualitative analysis in this subsection could be confirmed by the quantitative results shown in Figures 5 and 6.
(a)
(b)
(c)
(d)
(e)
(a)
(b)
(c)
(d)
(e)
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
(c)
(d)
(e)
(f)
6.4.2. Influence of Spatially Adapted Regularization Parameter Selection
As shown in Figures 5 and 6, SANLTVL1 generates better quantitative results than NLTVL1 in terms of PSNR and SSIM under different image degradation conditions. In this subsection, we will investigate the influence of spatially adapted regularization parameter selection on qualitative visual appearance. In Figure 11, we first assess the image restoration performance for image “Cameraman” corrupted by type I blur and saltandpepper noise with . In the case of small type I blur, both NLTVL1 and SANLTVL1 were capable of maintaining a good balance between noise/blur removal and detail preserving. To facilitate the qualitative visual evaluation, we also presented the absolute difference images between original and degraded/recovered images shown in the bottom panel of Figure 11. To achieve highquality image restoration, the corresponding absolute difference images should contain as little structural details as possible. As observed in Figure 11, the geometrical structures were more noticeable in absolute difference images yielded by NLTVL1. In contrast, our SANLTVL1 generated smaller absolute differences due to the spatially adapted regularization parameter shown in Figure 12. As can be observed, the values of the adaptive parameter are illustrated in a gray scale. Light gray regions refer to larger , whereas dark gray belongs to regions where is smaller. From the figure, we find that Figure 13 becomes large in the regions of the camera and tripod which contain significative structural information. Note that larger regularization parameters can perform well in fine detail preservation, whereas smaller ones in homogeneous regions demonstrate the results of artifact reduction.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
In the case of large type II blur and randomvalued impulse noise with ranged from to , we will assess the restoration performance for image “Lena” under different degradation conditions. The restoration results and their associated magnified views are displayed in Figures 13 and 14, respectively. Similar effects as those obtained in the preceding experiments can be found in this example. Both NLTVL1 and SANLTVL1 were capable of strong impulse noise removal. However, due to the large type II blur, restoration results generated by NLTVL1 suffered from the ripple artifacts in homogeneous regions (e.g., face and bare shoulder). In contrast, SANLTVL1 could effectively remove the ripple artifacts and preserved the meaningful geometrical structures. The improvements of the recovered images are due to the spatially adapted regularization parameter shown in Figure 15. It can be observed that is large in the regions of hair and is small in the homogeneous regions. Thus, our SANLTVL1 could establish a proper balance between detail preservation and artifact reduction and generate better visual appearance. In conclusion, the superior performance of the proposed method benefits from the nonlocal regularizer and spatially adapted regularization parameter selection.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(a)
(b)
(c)
(d)
7. Conclusions and Future Work
In this paper, we have studied the variational method for restoring blurred images with impulse noise. Based on the good features of the nonlocal regularizer and the spatially adapted parameter selection scheme proposed by Hintermüller and Rincon–Camacho in [36], we proposed a spatially adapted NLTV regularized L1 variational image restoration method. This method is capable of restoring degraded images while preserving edges and fine details. The major contributions of this paper were threefold. First, a NLTV regularizer was incorporated into the MAPbased image restoration framework. The main benefit of this nonlocal regularizer is that it can take full advantage of the natural redundancy of patterns within an image. Therefore, edges and fine details can be preserved during image restoration. Second, to guarantee solution stability, an alternating optimization algorithm was proposed to efficiently solve the proposed NLTVL1 model. Third, to future improve image visual quality, we introduced an automated method for selection of spatially adapted regularization parameter in terms of both saltandpepper and randomvalued impulse noise. Numerous experiments preformed on several images have demonstrated the superior performance of our proposed method in terms of both quantitative and visual quality evaluations. However, the proposed method still suffers from several potential limitations in current version. To make the proposed method available in practical applications, the work presented in this paper can be extended in the following directions:(i)The results for computational CPU time are illustrated in Figure 16. It can be observed that both NLTVL1 and SANLTVL1 suffer from high computational cost under different image degradation conditions. The high computational cost is mainly due to the cost of weights calculation for all pixels during image restoration. To significantly accelerate the image restoration, both NLTVL1 and SANLTVL1 will be implemented using graphics processing unit (GPU). Over the past few decades, GPU has rapidly evolved into a costeffective hardware platform for parallel computing [59] and has been successfully implemented to accelerate the NLTVbased motion estimation [60] and NLTGVbased optical flow estimation [61]. There is thus a strong incentive to accelerate both NLTVL1 and SANLTVL1 for realtime applications on GPUbased parallel computing platform.(ii)In this paper, both blur kernel and noise level are assumed to be known before image restoration. Therefore, the recovered image can be directly induced from the sufficient information provided by the blur kernel and image noise. However, both blur kernel and noise level are commonly unknown in practical applications. In current literature, a significant amount of effort has been dedicated to impulse noise estimation and reduction [9, 62]. By contrast, blind image restoration, which means that the blur kernel needs to be estimated, is a much more challenging illposed problem. Many methods have been proposed for blind image restoration [63], but they often tend to suffer from severe ringing artifacts especially in the presence of image noise. In future research, we will propose to first identify image pixels contaminated by impulse noise, as done in [31] and to simultaneously implement noise reduction and blind image restoration based on the estimated location of noisefree pixels. Due to the spatially adapted regularization parameter selection and nonlocal regularizer insensitive to noise, we believe that our SANLTVL1 can be successfully generalized to blind image restoration in the presence of impulse noise.(iii)Due to the powerful learning capacity, deep learning has significantly improved the image restoration results [64–66]. It is well known that deep learning is an essentially datadriven method, which does not take into consideration the image degradation process. The restored images sometimes suffer from instabilities and unwanted artifacts under complex imaging conditions. In addition, the variational method is essentially related to the image prior which indicates the intrinsic image knowledge. It is able to robustly restore the degraded images. In future study, there is a great potential to incorporate the deep learning strategy into our nonlocal regularized variational model. The learning module is capable of accurately estimating the important variables, which are difficult to handle in traditional variational methods. Therefore, the imaging performance will be further enhanced, learning to more reliable postprocessing steps, such as object detection, recognition, and tracking in practical applications.
(a)
(b)
(c)
(d)
Honestly, our current study has some potential limitations, but it is still worthy of consideration since the proposed SANLTVL1 could guarantee superior restoration performance compared to several stateoftheart methods. We believe there is a great potential to use the proposed method to recover degraded image in practical applications.
Data Availability
The image data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work was supported by the 2022 Research Topics of Huanggang Scientific and Technological Innovation Think Tank and the Scientific Research Project of Education Department of Hubei Province (No: Q20202905).