Research Article | Open Access

Volume 2014 |Article ID 230348 | https://doi.org/10.1155/2014/230348

Yi Zhan, Sheng Jie Li, Meng Li, "Local and Nonlocal Regularization to Image Interpolation", Mathematical Problems in Engineering, vol. 2014, Article ID 230348, 8 pages, 2014. https://doi.org/10.1155/2014/230348

# Local and Nonlocal Regularization to Image Interpolation

Revised15 Jan 2014
Accepted06 Feb 2014
Published12 Mar 2014

#### Abstract

This paper presents an image interpolation model with local and nonlocal regularization. A nonlocal bounded variation (BV) regularizer is formulated by an exponential function including gradient. It acts as the Perona-Malik equation. Thus our nonlocal BV regularizer possesses the properties of the anisotropic diffusion equation and nonlocal functional. The local total variation (TV) regularizer dissipates image energy along the orthogonal direction to the gradient to avoid blurring image edges. The derived model efficiently reconstructs the real image, leading to a natural interpolation which reduces blurring and staircase artifacts. We present experimental results that prove the potential and efficacy of the method.

#### 1. Introduction

Image interpolation has been an active research topic in both academia and industry due to its huge number of applications. It aims at producing a high-resolution (HR) image from its low-resolution (LR) counterpart . The relationship between HR and LR images can be described as where the matrix combines both filtering and downsampling processes. Equation (1) is an ill-posed inverse problem since the null space of is nonzero as stated in [1]. This ill-posed inverse problem is generally approached in a regularization-based framework, which was formulated as an energy functional, where is a regularization parameter that controls the tradeoff between the data fidelity term and the regularization term . The data fidelity function generally is formulated in the classical least-squares sense as .

Many regularizers have been developed to specify (2). Generally speaking, variational regularization approaches to image processing can be divided into two categories: the local regularization and the nonlocal regularization. In the local regularization approaches, the -Laplacian functional and variations of it have been extensively studied in the literature. It possesses the following form: TV-based regularization, , as first proposed by Rudin et al. [2] does an excellent job at preserving edges in image denoising, image interpolation [1], image restoration [37], and image inpainting [8]. Choosing results in isotropic diffusion which solves the staircasing problem. Together with other constraints, it formulates the regularization for image segmentation [9]. Different values of result in anisotropic diffusion between TV-based and isotropic smoothing.

These local regularization methods which penalize derivatives involve only the values and derivatives of at the same point [10]. Moreover, the classical gradients which are used to extract the image feature direction are local operators [11]. The information contained in the gradient is limited to a point and its immediate neighbors, while the edge curve is not a local event. As pointed out in the literatures, for example, [12, 13], this sometimes causes “blocky" solutions, staircasing effects, and “false edges," which can mislead a human or computer into identifying erroneous features not present in the true image.

In order to better respect edges in images, the nonlocal regularization schemes have been proposed. Kindermann et al.  [10] were the first to interpret nonlocal means and neighborhood filters as regularization based on nonlocal functionals which possesses the following general form: with an appropriate positive weight function , a differentiable filter function , and a parameter . Choosing , the nonlocal -Laplacian functional is obtained as follows: which shares many properties of the classical -Laplacian functional (3). The nonlocal -Laplacian regularization of (5) has been used recently in the study of saliency detection [14] and image denoising and segmentation [15, 16].

Another type of nonlocal regularization is based on the graph gradient operator. The graph gradient operator is defined by [1719] Thus, this type of nonlocal regularization is defined by Taking in the functional (7), Gilboa and Osher [17] focus on the nonlocal TV regularization and use it to detect and remove irregularities from textures and image inpainting. Elmoataz et al. [18] present nonlocal discrete -Laplacian regularization on image and manifold by the choice in (7) which leads to the corresponding nonlocal regularization The above nonlocal functionals do not take into account the gradient of the image . As pointed out in [10], if the problem is ill-posed a functional involving gradients will do better.

In order to make use of the advantages of local and nonlocal functional, local and nonlocal regularization are presented in this paper. It is a linear combination of local TV and nonlocal BV. The nonlocal BV function is an exponential function including gradient, which shares the properties of the classical Perona-Malik equation. This makes our nonlocal BV functional possess the property of anisotropic diffusion, which preserves image edges better. The rest of the paper is organized as follows. In Section 2, the motivation of our nonlocal BV regularization was introduced. The local and nonlocal regularization appear in Section 3. In Section 4, we demonstrate the experimental results to verify the effectiveness of our method, and the last section is for conclusion.

#### 2. Background

Here we briefly review some related nonlocal functional presented in [10]. Considering the functional (4), the authors chose , which gives the functional The corresponding Euler-Lagrange descent flow of (10) is

As stated in [10], the above functionals might suffice for denoising purposes, but not for the deblurring or interpolation case which is ill-posed in the space , since adding a Prokhorov term will in general not regularize the problem in a reasonable space. Hence, a functional involving gradients is beneficial for ill-posed problem. So for image deblurring or image interpolation it is a good choice to use nonlocal functionals derived from the space of BV. The more general functional of the this type is In [10], the following nonlocal regularization term was selected () to image denoising: where .

For , the corresponding Euler-Lagrange descent flow of (13) is Note that does not include image information. Equation (14) indicates that (13) bears some resemblance to the TV regularization of which the corresponding Euler-Lagrange descent flow is . This kind of dissipation process diffuses along the image contours. It will preserve the location and the intensity transitions of the contours to avoid blurring image edges. In order to enhance image edges it is important to consider the diffusion along the direction orthogonal to the image contours, similar to the Perona-Malik anisotropic diffusion equation [20], The general selection of is or , where is a constant. It is difficult to obtain the Euler-Lagrange equation as above from a functional, but we can derive its nonlocal form similar to (15) from a nonlocal functional motivated by (10) and (11). In the next section we consider this kind of nonlocal functional.

#### 3. Local and Nonlocal Regularization to Image Interpolation

In this section we formulate a nonlocal BV functional by incorporating (10) and (12), which has the form The directional derivative of is given by We can split the integral into a term involving and . In the latter a change of variables gives the same integral as that with ; hence, we end up with Integrated by parts, the Fréchet derivative of as a functional from to is given by The corresponding Euler-Lagrange descent flow is As can be seen, (20) is similar to the Perona-Malik equation (15) with . At the points lying in flat area or gentle slope, is small and (20) acts as a Laplacian evolution equation, which is used in image denoising and so forth, while, for the points on image edges, is large and (20) diffuses image energy backward like the Perona-Malik equation. This process enhances image edges, which represents our desirable requirement. To eliminate oscillatory image contour, the TV regularizer will suffice.

Based on the above, our proposed energy functional for regularized image interpolation is given by The gradient flow associated with the functional is where The kernel is assumed to be nonnegative, bounded continuous radial function, with and .

We develop a fully discrete numerical method to approximate problem (22). We recall first the notations in the finite differences scheme used in our paper. Let and be the space and time steps, respectively, and let be the grid points. Let be an approximation of the function , with . Equation (22) can be discretized as follows: In all numerical experiments, we choose the following kernel function: the constant is selected such that and .

#### 4. Experimental Results

In this section, we tested the proposed interpolation method (24) on a variety of images. The nine natural images of them are shown in Figure 1. The image sizes are (Barbara), (race), (fingerprint), (eye), (ball), and (hat, parrot, flower, and boat). The images were downsampled without low-pass filtering and upsampled two times, such that the peak signal-to-noise ratio (PSNR) and the mean structural similarity (MSSIM) index [21] were measured, as shown in Tables 1 and 2. The algorithms for comparison are the NEDI [22], BSAI [23], adaptive gradient magnitude self-interpolation (AGMS) (using the default parameters) [24], the LMMSE estimation [25], and nonlocal BV [10]. The MATLAB codes of the NEDI, LMMSE, and AGMS and the C codes of the BSAI were obtained from the authors' corresponding websites, respectively. The MATLAB code of AGMS is available at http://www.datatang.com/data/44058. Some of the results are shown in Figures 26. The choice of the parameters in (24) is based on subjective quality of the results assessed informally by our personal preference as human viewers in terms of edge sharpness, contour crispness, no ringing in smooth regions, and no ringing near edges. We use the parameters , , , and time step for the proposed interpolation method. There is no visible improvement on subjective or objective quality of the results when the parameters are not badly changed. The iteration is terminated, when , normally within a few decade iterations.

 Image NEDI LMMSE BSAI AGMS NLBV Proposed Hat 28.6606 28.6328 28.8784 30.0459 30.9239 31.3346 Parrot 27.6830 27.6966 28.0056 28.1065 31.3969 31.6225 Boat 25.6305 25.6840 25.8219 27.3422 28.4654 28.5086 Race 23.6013 23.6792 23.8557 24.3376 26.0261 26.1027 Eye 29.7578 29.8821 29.9564 31.0001 32.7364 32.6488 Ball 31.9757 32.1733 32.2449 34.6244 36.3935 36.5681 Flower 26.7856 26.7789 26.9631 28.6568 30.2598 30.5425 Barbara 23.8678 24.3268 24.3028 25.1123 25.8749 25.5891 Fingerprint 23.7430 24.2690 24.3663 25.5444 26.3480 26.8346
 Image NEDI LMMSE BSAI AGMS NLBV Proposed Hat 0.8575 0.8548 0.8590 0.8807 0.8949 0.9037 Parrot 0.9018 0.9037 0.9066 0.9182 0.9378 0.9433 Boat 0.7786 0.7823 0.7862 0.8309 0.8653 0.8634 Race 0.7286 0.7367 0.7445 0.7732 0.8412 0.8344 Eye 0.8834 0.8879 0.8883 0.9226 0.9443 0.9430 Ball 0.9093 0.9147 0.9149 0.9468 0.9595 0.9630 Flower 0.8311 0.8335 0.8401 0.8741 0.9073 0.9131 Barbara 0.9046 0.9011 0.9030 0.9391 0.9811 0.9650 Fingerprint 0.8911 0.8942 0.8973 0.9364 0.9420 0.9480

Figure 2 shows a portion of the eye image. It is obvious that the NEDI suffers from zigzagging around the fence in the pupil. It is observable that the fence in the eye is blurring in LMMSE and BSAI interpolation and is deformed and distorted in AGMS interpolation. The artificial effects are clear in NEDI and NLBV. The proposed method produces crisp and smooth image edges, which contributes to 2.6 dB improvement in PSNR in the eye image. In Figure 3, the interpolation artifacts around black plumage strip appear in LMMSE and NLBV result. The deformation on black plumage strip appears in AGMS result. The shape of black plumage strip is natural and clear in our result, since local TV regularizer smoothes image edges and avoids staircase effects while nonlocal BV regularizer acts as anisotropic diffusion to enhance image edges in our method. But the blurring is visible in NEDI and BSAI. The same effects arise in Figures 4 and 5. In Figure 4, the teeth in tire are clear in our result. Although the teeth in tire are the clearest in NLBV, it is a false appearance produced by zigzagging effect. In Figure 5, the NEDI and our method obtain clear pistils, but our result possesses natural petal. Figure 6 shows that our method works well for texture (fingerprint) image.

We use two measures, the classic PSNR and MSSIM, to characterize the difference between the reference images and the outputs of a method. If and are two images, the PSNR is defined as , where . The detailed definition of MSSIM refers to [21]. MSSIM index takes values in and increases as the quality increases. It is calculated by the code available at http://www.cns.nyu.edu/~lcv/ssim/, using the default parameters. From the two tables, the proposed method is about 2 dB PSNR and 0.05 MSSIM higher than the other algorithms on average.

#### 5. Conclusion

In this paper, a local and nonlocal image interpolation model based on local TV and nonlocal BV regularization is proposed. It combines the advantages of TV regularizer and nonlocal BV regularizer, that is, allowing discontinuities and preserving 1D image structures and the diffusion of the grey values of images along image feature direction. The direction of anisotropic diffusion is indicated by the information of image feature in a larger neighborhood. This results in minimal smoothing in the directions across the image features preserving sharp edges and maximal smoothing in the directions along the image features reducing zigzagging artifacts and oscillatory. We have shown improvement over local and nonlocal regularization on a subjective scale and in many cases with an improvement in PSNR and MSSIM.

#### Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

#### Acknowledgments

This work was supported by the NSF of China (nos. 61202349, 11101453, and 11226268), the Natural Science Foundation Project of CQ CSTC (no. cstc2013jcyjA40058), Scientific and Technological Research Program of Chongqing Municipal Education Commission (nos. KJ120709, KJ131209), and the Research Foundation of Chongqing University of Arts and Sciences (no. R2012SC20).

#### References

1. H. A. Aly and E. Dubois, “Image up-sampling using total-variation regularization with a new observation model,” IEEE Transactions on Image Processing, vol. 14, no. 10, pp. 1647–1659, 2005. View at: Publisher Site | Google Scholar | MathSciNet
2. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, vol. 60, no. 1–4, pp. 259–268, 1992.
3. P. L. Combettes and J.-C. Pesquet, “Image restoration subject to a total variation constraint,” IEEE Transactions on Image Processing, vol. 13, no. 9, pp. 1213–1222, 2004. View at: Publisher Site | Google Scholar
4. Y. Wang, W. Yin, and Y. Zhang, “A fast algorithm for image deblurring with total variation regularization,” Rice University CAAM Technical Report TR07-10, 2007. View at: Google Scholar
5. D. Goldfarb and W. Yin, “Second-order cone programming methods for total variation-based image restoration,” SIAM Journal on Scientific Computing, vol. 27, no. 2, pp. 622–645, 2006. View at: Publisher Site | Google Scholar | MathSciNet
6. A. Chambolle and P.-L. Lions, “Image recovery via total variation minimization and related problems,” Numerische Mathematik, vol. 76, no. 2, pp. 167–188, 1997.
7. Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM Journal on Imaging Sciences, vol. 1, no. 3, pp. 248–272, 2008.
8. T. F. Chan and J. Shen, “Mathematical models for local nontexture inpaintings,” SIAM Journal on Applied Mathematics, vol. 62, no. 3, pp. 1019–1043, 2002.
9. A. Tsai, A. Yezzi Jr., and A. S. Willsky, “Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation, and magnification,” IEEE Transactions on Image Processing, vol. 10, no. 8, pp. 1169–1186, 2001.
10. S. Kindermann, S. Osher, and P. W. Jones, “Deblurring and denoising of images by nonlocal functionals,” Multiscale Modeling and Simulation, vol. 4, no. 4, pp. 1091–1115, 2005.
11. R. A. Carmona and S. Zhong, “Adaptive smoothing respecting feature directions,” IEEE Transactions on Image Processing, vol. 7, no. 3, pp. 353–358, 1998. View at: Publisher Site | Google Scholar
12. W. Ring, “Structural properties of solutions to total variation regularization problems,” Mathematical Modelling and Numerical Analysis, vol. 34, no. 4, pp. 799–810, 2000.
13. D. C. Dobson and F. Santosa, “Recovery of blocky images from noisy and blurred data,” SIAM Journal on Applied Mathematics, vol. 56, no. 4, pp. 1181–1198, 1996.
14. M. Li, Y. Zhan, and L. Zhang, “Nonlocal variational model for saliency detection,” Mathematical Problems in Engineering, vol. 2013, Article ID 518747, 7 pages, 2013. View at: Publisher Site | Google Scholar
15. G. Gilboa and S. Osher, “Nonlocal linear image regularization and supervised segmentation,” Multiscale Modeling and Simulation, vol. 6, no. 2, pp. 595–630, 2007.
16. Q. Xin, X. Zhang, and L. Xu, “Image segmentation model based on the nonlocal p-Laplacian equation,” Journal of Computational Information Systems, vol. 9, no. 1, pp. 345–351, 2013. View at: Google Scholar
17. G. Gilboa and S. Osher, “Nonlocal operators with applications to image processing,” Multiscale Modeling and Simulation, vol. 7, no. 3, pp. 1005–1028, 2008.
18. A. Elmoataz, O. Lezoray, and S. Bougleux, “Nonlocal discrete p-Laplacian driven image and manifold processing,” Comptes Rendus, vol. 336, no. 5, pp. 428–433, 2008.
19. G. Peyré, S. Bougleux, and L. Cohen, “Non-local regularization of inverse problems,” in Proceedings of the European Conference on Computer Vision (ECCV '08), 2008. View at: Google Scholar
20. P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, no. 7, pp. 629–639, 1990. View at: Publisher Site | Google Scholar
21. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. View at: Publisher Site | Google Scholar
22. X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Transactions on Image Processing, vol. 10, no. 10, pp. 1521–1527, 2001. View at: Publisher Site | Google Scholar
23. K.-W. Hung and W.-C. Siu, “Fast image interpolation using the bilateral filter,” IET Image Processing, vol. 6, no. 7, pp. 877–890, 2012. View at: Publisher Site | Google Scholar | MathSciNet
24. L. Wang, S. Xiang, G. Meng, H. Y. Wu, and C. Pan, “Edge-directed single image super-resolution via adaptive gradient magnitude self-interpolation,” IEEE Transaction on Circuits and Systems for Video Technology, vol. 23, no. 8, pp. 1289–1299, 2013. View at: Publisher Site | Google Scholar
25. L. Zhang and X. Wu, “An edge-guided image interpolation algorithm via directional filtering and data fusion,” IEEE Transactions on Image Processing, vol. 15, no. 8, pp. 2226–2238, 2006. View at: Publisher Site | Google Scholar

Copyright © 2014 Yi Zhan et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.