Abstract
This paper presents an image interpolation model with local and nonlocal regularization. A nonlocal bounded variation (BV) regularizer is formulated by an exponential function including gradient. It acts as the Perona-Malik equation. Thus our nonlocal BV regularizer possesses the properties of the anisotropic diffusion equation and nonlocal functional. The local total variation (TV) regularizer dissipates image energy along the orthogonal direction to the gradient to avoid blurring image edges. The derived model efficiently reconstructs the real image, leading to a natural interpolation which reduces blurring and staircase artifacts. We present experimental results that prove the potential and efficacy of the method.
1. Introduction
Image interpolation has been an active research topic in both academia and industry due to its huge number of applications. It aims at producing a high-resolution (HR) image from its low-resolution (LR) counterpart . The relationship between HR and LR images can be described as where the matrix combines both filtering and downsampling processes. Equation (1) is an ill-posed inverse problem since the null space of is nonzero as stated in [1]. This ill-posed inverse problem is generally approached in a regularization-based framework, which was formulated as an energy functional, where is a regularization parameter that controls the tradeoff between the data fidelity term and the regularization term . The data fidelity function generally is formulated in the classical least-squares sense as .
Many regularizers have been developed to specify (2). Generally speaking, variational regularization approaches to image processing can be divided into two categories: the local regularization and the nonlocal regularization. In the local regularization approaches, the -Laplacian functional and variations of it have been extensively studied in the literature. It possesses the following form: TV-based regularization, , as first proposed by Rudin et al. [2] does an excellent job at preserving edges in image denoising, image interpolation [1], image restoration [3–7], and image inpainting [8]. Choosing results in isotropic diffusion which solves the staircasing problem. Together with other constraints, it formulates the regularization for image segmentation [9]. Different values of result in anisotropic diffusion between TV-based and isotropic smoothing.
These local regularization methods which penalize derivatives involve only the values and derivatives of at the same point [10]. Moreover, the classical gradients which are used to extract the image feature direction are local operators [11]. The information contained in the gradient is limited to a point and its immediate neighbors, while the edge curve is not a local event. As pointed out in the literatures, for example, [12, 13], this sometimes causes “blocky" solutions, staircasing effects, and “false edges," which can mislead a human or computer into identifying erroneous features not present in the true image.
In order to better respect edges in images, the nonlocal regularization schemes have been proposed. Kindermann et al. [10] were the first to interpret nonlocal means and neighborhood filters as regularization based on nonlocal functionals which possesses the following general form: with an appropriate positive weight function , a differentiable filter function , and a parameter . Choosing , the nonlocal -Laplacian functional is obtained as follows: which shares many properties of the classical -Laplacian functional (3). The nonlocal -Laplacian regularization of (5) has been used recently in the study of saliency detection [14] and image denoising and segmentation [15, 16].
Another type of nonlocal regularization is based on the graph gradient operator. The graph gradient operator is defined by [17–19] Thus, this type of nonlocal regularization is defined by Taking in the functional (7), Gilboa and Osher [17] focus on the nonlocal TV regularization and use it to detect and remove irregularities from textures and image inpainting. Elmoataz et al. [18] present nonlocal discrete -Laplacian regularization on image and manifold by the choice in (7) which leads to the corresponding nonlocal regularization The above nonlocal functionals do not take into account the gradient of the image . As pointed out in [10], if the problem is ill-posed a functional involving gradients will do better.
In order to make use of the advantages of local and nonlocal functional, local and nonlocal regularization are presented in this paper. It is a linear combination of local TV and nonlocal BV. The nonlocal BV function is an exponential function including gradient, which shares the properties of the classical Perona-Malik equation. This makes our nonlocal BV functional possess the property of anisotropic diffusion, which preserves image edges better. The rest of the paper is organized as follows. In Section 2, the motivation of our nonlocal BV regularization was introduced. The local and nonlocal regularization appear in Section 3. In Section 4, we demonstrate the experimental results to verify the effectiveness of our method, and the last section is for conclusion.
2. Background
Here we briefly review some related nonlocal functional presented in [10]. Considering the functional (4), the authors chose , which gives the functional The corresponding Euler-Lagrange descent flow of (10) is
As stated in [10], the above functionals might suffice for denoising purposes, but not for the deblurring or interpolation case which is ill-posed in the space , since adding a Prokhorov term will in general not regularize the problem in a reasonable space. Hence, a functional involving gradients is beneficial for ill-posed problem. So for image deblurring or image interpolation it is a good choice to use nonlocal functionals derived from the space of BV. The more general functional of the this type is In [10], the following nonlocal regularization term was selected () to image denoising: where .
For , the corresponding Euler-Lagrange descent flow of (13) is Note that does not include image information. Equation (14) indicates that (13) bears some resemblance to the TV regularization of which the corresponding Euler-Lagrange descent flow is . This kind of dissipation process diffuses along the image contours. It will preserve the location and the intensity transitions of the contours to avoid blurring image edges. In order to enhance image edges it is important to consider the diffusion along the direction orthogonal to the image contours, similar to the Perona-Malik anisotropic diffusion equation [20], The general selection of is or , where is a constant. It is difficult to obtain the Euler-Lagrange equation as above from a functional, but we can derive its nonlocal form similar to (15) from a nonlocal functional motivated by (10) and (11). In the next section we consider this kind of nonlocal functional.
3. Local and Nonlocal Regularization to Image Interpolation
In this section we formulate a nonlocal BV functional by incorporating (10) and (12), which has the form The directional derivative of is given by We can split the integral into a term involving and . In the latter a change of variables gives the same integral as that with ; hence, we end up with Integrated by parts, the Fréchet derivative of as a functional from to is given by The corresponding Euler-Lagrange descent flow is As can be seen, (20) is similar to the Perona-Malik equation (15) with . At the points lying in flat area or gentle slope, is small and (20) acts as a Laplacian evolution equation, which is used in image denoising and so forth, while, for the points on image edges, is large and (20) diffuses image energy backward like the Perona-Malik equation. This process enhances image edges, which represents our desirable requirement. To eliminate oscillatory image contour, the TV regularizer will suffice.
Based on the above, our proposed energy functional for regularized image interpolation is given by The gradient flow associated with the functional is where The kernel is assumed to be nonnegative, bounded continuous radial function, with and .
We develop a fully discrete numerical method to approximate problem (22). We recall first the notations in the finite differences scheme used in our paper. Let and be the space and time steps, respectively, and let be the grid points. Let be an approximation of the function , with . Equation (22) can be discretized as follows: In all numerical experiments, we choose the following kernel function: the constant is selected such that and .
4. Experimental Results
In this section, we tested the proposed interpolation method (24) on a variety of images. The nine natural images of them are shown in Figure 1. The image sizes are (Barbara), (race), (fingerprint), (eye), (ball), and (hat, parrot, flower, and boat). The images were downsampled without low-pass filtering and upsampled two times, such that the peak signal-to-noise ratio (PSNR) and the mean structural similarity (MSSIM) index [21] were measured, as shown in Tables 1 and 2. The algorithms for comparison are the NEDI [22], BSAI [23], adaptive gradient magnitude self-interpolation (AGMS) (using the default parameters) [24], the LMMSE estimation [25], and nonlocal BV [10]. The MATLAB codes of the NEDI, LMMSE, and AGMS and the C codes of the BSAI were obtained from the authors' corresponding websites, respectively. The MATLAB code of AGMS is available at http://www.datatang.com/data/44058. Some of the results are shown in Figures 2–6. The choice of the parameters in (24) is based on subjective quality of the results assessed informally by our personal preference as human viewers in terms of edge sharpness, contour crispness, no ringing in smooth regions, and no ringing near edges. We use the parameters , , , and time step for the proposed interpolation method. There is no visible improvement on subjective or objective quality of the results when the parameters are not badly changed. The iteration is terminated, when , normally within a few decade iterations.
Figure 2 shows a portion of the eye image. It is obvious that the NEDI suffers from zigzagging around the fence in the pupil. It is observable that the fence in the eye is blurring in LMMSE and BSAI interpolation and is deformed and distorted in AGMS interpolation. The artificial effects are clear in NEDI and NLBV. The proposed method produces crisp and smooth image edges, which contributes to 2.6 dB improvement in PSNR in the eye image. In Figure 3, the interpolation artifacts around black plumage strip appear in LMMSE and NLBV result. The deformation on black plumage strip appears in AGMS result. The shape of black plumage strip is natural and clear in our result, since local TV regularizer smoothes image edges and avoids staircase effects while nonlocal BV regularizer acts as anisotropic diffusion to enhance image edges in our method. But the blurring is visible in NEDI and BSAI. The same effects arise in Figures 4 and 5. In Figure 4, the teeth in tire are clear in our result. Although the teeth in tire are the clearest in NLBV, it is a false appearance produced by zigzagging effect. In Figure 5, the NEDI and our method obtain clear pistils, but our result possesses natural petal. Figure 6 shows that our method works well for texture (fingerprint) image.
We use two measures, the classic PSNR and MSSIM, to characterize the difference between the reference images and the outputs of a method. If and are two images, the PSNR is defined as , where . The detailed definition of MSSIM refers to [21]. MSSIM index takes values in and increases as the quality increases. It is calculated by the code available at http://www.cns.nyu.edu/~lcv/ssim/, using the default parameters. From the two tables, the proposed method is about 2 dB PSNR and 0.05 MSSIM higher than the other algorithms on average.
5. Conclusion
In this paper, a local and nonlocal image interpolation model based on local TV and nonlocal BV regularization is proposed. It combines the advantages of TV regularizer and nonlocal BV regularizer, that is, allowing discontinuities and preserving 1D image structures and the diffusion of the grey values of images along image feature direction. The direction of anisotropic diffusion is indicated by the information of image feature in a larger neighborhood. This results in minimal smoothing in the directions across the image features preserving sharp edges and maximal smoothing in the directions along the image features reducing zigzagging artifacts and oscillatory. We have shown improvement over local and nonlocal regularization on a subjective scale and in many cases with an improvement in PSNR and MSSIM.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work was supported by the NSF of China (nos. 61202349, 11101453, and 11226268), the Natural Science Foundation Project of CQ CSTC (no. cstc2013jcyjA40058), Scientific and Technological Research Program of Chongqing Municipal Education Commission (nos. KJ120709, KJ131209), and the Research Foundation of Chongqing University of Arts and Sciences (no. R2012SC20).