Single Image Super-Resolution via <svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-4.5247pt" id="M1" height="16.402pt" version="1.1" viewBox="-0.0657574 -11.8773 18.3609 16.402" width="18.3609pt"><g transform="matrix(.018,0,0,-0.018,0,0)"><path id="g113-77" d="M541 160L512 170C485 116 462 88 439 67C411 41 376 35 318 35C272 35 238 37 224 48C207 61 205 85 212 121L290 533C305 611 308 615 391 622L397 650H139L133 622C217 615 221 612 206 533L126 118C111 41 103 34 23 26L17 0H474C489 31 528 124 541 160Z"/></g><g transform="matrix(.013,0,0,-0.013,11.41,4.308)"><path id="g50-49" d="M245 635C92 635 37 457 37 312C37 149 91 -12 244 -12C395 -12 449 166 449 312C449 469 395 635 245 635ZM243 598C332 598 358 454 358 312C358 173 334 26 245 26C158 26 128 174 128 313S152 598 243 598Z"/></g></svg> Image Smoothing

Liu, Zhang; Huang, Qi; Li, Jian; Wang, Qi

doi:https://doi.org/10.1155/2014/974509

Mathematical Problems in Engineering

On this page

Abstract Introduction Related Work Results Conclusions References Copyright Related Articles

Research Article | Open Access

Volume 2014 | Article ID 974509 | https://doi.org/10.1155/2014/974509

Single Image Super-Resolution via Image Smoothing

Zhang Liu,^1,2Qi Huang,¹Jian Li,¹and Qi Wang^1,3

Academic Editor: José R. C. Piqueira

Received19 Oct 2014

Accepted04 Dec 2014

Published29 Dec 2014

Abstract

We propose a single image super-resolution method based on a smoothing approach. We consider a low-resolution image as two parts: one is the smooth image generated by the smoothing method and the other is the error image between the low-resolution image and the smoothing image. We get an intermediate high-resolution image via a classical interpolation and then generate a high-resolution smoothing image with sharp edges by the smoothing method. For the error image, a learning-based super-resolution approach, keeping image details well, is employed to obtain a high-resolution error image. The resulting high-resolution image is the sum of the high-resolution smoothing image and the high-resolution error image. Experimental results show the effectiveness of the proposed method.

1. Introduction

Image super-resolution is an active field recently in image processing. It tends to obtain the high-resolution image via one or multiple low-resolution images. If we get the high-resolution image only by one low-resolution input, we call it single image super-resolution. The other case is called classical image super-resolution. Obviously, single image super-resolution is more useful and challenging for the practical applications, since only one low-resolution image is available sometimes due to the limitation of hardware or other reasons. In particular, we address single image super-resolution in the paper. Image super-resolution has many applications, for example, medical imaging magnetic resonance imaging (MRI), synthetic aperture radar (SAR), and high definition television. How to develop an effective super-resolution model and algorithm is very important.

There are many image super-resolution methods up to now. They can be classified into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods. Interpolation-based methods tend to interpolate the unknown points via the known neighbor points. The two most famous interpolation methods are nearest-neighbor interpolation and bicubic interpolation. In addition, more interpolation methods can be found in [1–3]. Interpolation-based methods are very popular and fast, but they have to face some drawbacks, for example, jaggy effect or blur effect on image edges. Many reconstruction methods [4–10] have been proposed. They obtain the resulting super-resolution images via some reconstruction ideas. In [9], Shan et al. propose a fast image/video super-resolution approach by minimizing an energy function. This method can use fast Fourier transformation (FFT) algorithm to accelerate the speed. It upsamples a low-resolution image and then applies a fast reconstruction method to the upsampled image to get the resulting high-resolution output. In [10], Chambolle and Pock propose a first-order primal-dual algorithm for convex models and apply this algorithm to single image super-resolution. In addition, authors also give the convergence analysis of corresponding algorithm. Learning-based methods are a very popular tool to address image super-resolution problems [11–24]. Excellent results can be obtained via learning-based methods. However, before implementing the method, we have to generate two training data sets. One data set is formed by a mass of low-resolution patches; the other is formed by corresponding high-resolution patches. After getting the relation between the two training data sets, we apply it to a given low-resolution image to get high-resolution image. Obviously, this approach depends on the selection of training data sets and leads to expensive computation. In [19, 20], Yang et al. propose a sparse signal representation method for single image super-resolution. They search a sparse representation for each patches of the low-resolution image and compute the coefficients and then apply the computed coefficients to generate the high-resolution image. Based on the work of [19, 20], Zeyde et al. [22] simplify the overall process and reduce the computation complexity of algorithm. In [23], Timofte et al. propose a fast image super-resolution method which makes no compromise on image quality. It proposes an anchored neighborhood regression and utilizes global collaborative coding for speedup. In [24], Dong et al. learn a mapping between low-resolution and high-resolution images and then consider the mapping as a deep convolutional neural network (CNN). The CNN takes the low-resolution image as input and outputs the resulting high-resolution image.

In addition, some different methods also have been proposed, for example, frequency technique [25], pixel classification method [26], and others [10, 27–29].

In this paper, we divide a low-resolution image into two parts: one part is smoothing image with sharp edges which is generated by smoothing method; the other part is error image which is the difference between the low-resolution image and the smoothing image. We get the high-resolution smoothing image by applying smoothing method to an intermediate high-resolution image. The high-resolution smoothing image contains very important features of super-resolution images: sharp edges. For the error image, we upsample it using an approach of keeping image details to get high-resolution error image. The final resulting super-resolution image is the sum of high-resolution smoothing image and high-resolution error image.

The organization of this paper is as follows. In Section 2, we introduce an important image smoothing method which is utilized in our work. In addition, we give the proposed method which uses the smoothing method and one learning-based method. The experimental results are shown in Section 3. Finally, we give the conclusions in Section 4.

2.1. Image Smoothing

In [30], Xu et al. proposed a novel image smoothing method via gradient minimization which is used to control the number of nonzero gradients. By controlling the number of nonzero gradients, they establish a model to smooth the high-frequency details of image, preserving prominent image structures.

1D Signal Smoothing. For 1D case, counts the number of neighbor pixels and , where is the smoothing result and represents the counting operator and outputs the number of satisfying . Note that it can give one value to to control the smoothness of image. Although is not functional, the specific objective function can be given as follows: where is the given value and is the known discrete signal at location . This optimization model is very powerful to abstract the image structure. It flattens image details effectively and preserves sharp image edges obviously. The property of preserving sharp edges is the most important motivation of this paper to deal with image super-resolution.

Actually, can be set from zero to thousands according to practical case. A general regularity form is employed to balance the structure flattening term and the fidelity term; it is shown as follows: where is the regularity parameter which can control the significance of . The parameter also can be viewed as a smoothing factor: a larger will get fewer image edges. In particular, the number of nonzero gradients is monotone with respect to .

2D Image Smoothing. For 2D image case, is denoted as the input image and is denoted as the resulting smoothing image. The gradient of image at pixel is represented by . Similar to (1), it can get the number of nonzero gradients in the following formula:

Similar to (3), we get the image smoothing model according to (4), where is the balance parameter similar to (3). Actually, the term smoothes the image details and keeps the main image structure similarly. The corresponding solver of (5) will be given in the following.

Solving. For the optimization problem (5), it is a nondeterministic polynomial-time (NP) hard problem to find the global optimal solution. Motivated by Wang et al. [32], a splitting scheme is proposed to solve the optimization problem. Two auxiliary variables and are introduced to substitute and . Equation (5) can be rewritten as where is an adaptive parameter to control the distance between and their gradients . For solving one variable, it should fix the other variable obtained from the previous iteration. This process can be depicted as two subproblems.

S-Subproblem. We get by solving the following minimization problem:

The problem has global minimum solution due to quadratic function. It may be speed up by following fast Fourier transform (FFT) (see details in [32]). Consider where is FFT operator, is the Fourier transform of delta function, and is the complex conjugate. Note that is computed very fast due to the FFT operator which has complexity .

-Subproblem. For -subproblem, the corresponding minimization problem is as follows: where , if , and otherwise. Thus (9) can be rewritten as

In [30], it has been proved that, for each pixel , reaches its minimum under the following condition:

The resulting algorithm for image smoothing is as shown in Algorithm 1.

Input: Image , smoothing weight , parameter , rate
(1) Initialization: .
(2) While
(3) Solving via (10) when fixing
(4) Solving via (8) when fixing
(5) Update:
(6)
(7) end
Output: Smoothing image

Step (5) is to update adaptively via a parameter for each iteration. The resulting is the smoothing image which flattens image details but preserves sharp edges.

2.2. The Combined Method for Image Super-Resolution

Sharp edges are very important features to super-resolution images. In our work, we utilize the property of preserving sharp edges of image smoothing to enhance the quality of super-resolution image. We divide one low-resolution image into two parts: (1) smoothing image obtained by smoothing method and (2) error image obtained by the difference of the low-resolution image and the smoothing image. After this process, we take two steps to realize the proposed method. First, we upsample the low-resolution image to get intermediate high-resolution image by a fast and classical interpolation method and then apply smoothing method to it to get high-resolution smoothing image. Note that the high-resolution smoothing image has sharp edges. Second, the error image will be upsampled to high-resolution error image via one learning-based super-resolution method of keeping image details well.

In Figure 1, we show the proposed strategy for image super-resolution. This strategy combines two methods of different image applications for image super-resolution. In particular, “LR” represents the low-resolution image. “” is the low-resolution smoothing image via smoothing method. “” is the error image between “LR” and “.” Actually, “” is just to get the low-resolution error image . “” is the upsampled image by one classical interpolation method (here we use “bicubic” interpolation method as “Method 1”). “” is the high-resolution smoothing image by the same smoothing method. “” is the high-resolution error image by one learning-based method which can preserve image details well (in our work, we use one learning-based method [31] as “Method 2”).

We present Figure 2 to specify the rationality of the proposed work. From Figure 2, the resulting high-resolution image (Figure 2(a)) is the sum of high-resolution smoothing image (Figure 2(b)) and high-resolution error image (Figure 2(c)). Note that the high-resolution smoothing image has sharp image edges and the high-resolution error image contains obvious image details. It demonstrates that our proposed work can combine the advantages of image smoothing (preserving sharp edges) and the detail-preserved method [31] (keeping obvious image details).

(a)

(b)

(c)

3. Results

In this section, we compare the proposed method with some competitive image super-resolution methods, for example, 01’TIP [1], 06’TIP [3], and 07’TIP [8]. The experimental computer is a laptop with 3.25 Gb RAM and Intel Core i3-2370M CPU: @2.40 GHz. All experiments are implemented on MATLAB (R2010a). For the corresponding parameters, we set , , , and . In addition, the images with different types and upscaling factors are employed to test the effectiveness of the compared methods.

In Figure 3, we compare our method with two classical interpolation methods (nearest-neighbor interpolation and bicubic interpolation). Different images, for example, butterfly, and flower, are employed for experiments. From the figure, it is easy to know that the resulting super-resolution images by the proposed method keep sharper image edges than nearest-neighbor and bicubic interpolations. In particular, the resulting super-resolution images by nearest-neighbor interpolation show jaggy effect, and the super-resolution images by bicubic interpolation show blur effect.

In Figure 4, a competitive kernel regression method [8] is employed to compare with the proposed method. We know that the resulting images by our method preserve sharp edges better than other methods. The kernel regression method [8] and bicubic interpolation all show blur effect on the edges.

From Figure 5, more competitive super-resolution methods are compared with our method. We employ two images with different types in the experiments: one type is natural image and the other type is comic image. From the figure, bicubic interpolation, 01’TIP [1], 06’TIP [3], and 07’TIP [8] all show blur effect, especially on image edges. The proposed method performs best, not only on image edges but also on nonedge regions.

4. Conclusions

In this paper, a novel image super-resolution method was proposed based on an image smoothing strategy. We applied image smoothing method to a given low-resolution image to get its low-resolution smoothing image which preserved sharp edges. Besides, a low-resolution error image was obtained by the difference between the low-resolution image and the low-resolution smoothing image. For the low-resolution error image and the given low-resolution image, we took two steps to generate the resulting super-resolution image. First, we upsampled the low-resolution image by bicubic interpolation to get intermediate high-resolution image and then applied the smoothing method to the intermediate high-resolution image to generate the high-resolution smoothing image. Second, a learning-based super-resolution method was utilized to upsample the low-resolution error image to get high-resolution error image. The learning-based method could preserve image details well. The final super-resolution image was the sum of the high-resolution smoothing image and the high-resolution error image. In experimental section, we employed low-resolution images of different types to test our method. Results demonstrated that the proposed method performed better than some competitive image super-resolution methods, especially preserving sharp image edges.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

The authors thank the reviewers for their valuable comments.

References

X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Transactions on Image Processing, vol. 10, no. 10, pp. 1521–1527, 2001.
View at: Publisher Site | Google Scholar
N. Mueller, Y. Lu, and M. Do, “Image interpolation using multiscale geometric representations,” in Computational Imaging, vol. 6498 of Proceedings of SPIE, 2007.
View at: Google Scholar
L. Zhang and X. Wu, “An edge-guided image interpolation algorithm via directional filtering and data fusion,” IEEE Transactions on Image Processing, vol. 15, no. 8, pp. 2226–2238, 2006.
View at: Publisher Site | Google Scholar
M. Irani and S. Peleg, “Super resolution from image sequences,” in Proceedings of the 10th International Conference on Pattern Recognition (ICPR '90), pp. 115–120, June 1990.
View at: Google Scholar
M. Irani and S. Peleg, “Motion analysis for image enhancement: resolution, occlusion andtransparency,” Journal of Visual Communication and Image Representation, vol. 4, no. 4, pp. 324–335, 1993.
View at: Publisher Site | Google Scholar
T. Komatsu, T. Igarashi, K. Aizawa, and T. Saito, “Very high resolution imaging scheme with multiple different-aperture cameras,” Signal Processing: Image Communication, vol. 5, no. 5-6, pp. 511–526, 1993.
View at: Publisher Site | Google Scholar
P. Chatterjee, S. Mukherjee, S. Chaudhuri, and G. Seetharaman, “Application of Papoulis-Gerchberg method in image super-resolution and inpainting,” The Computer Journal, vol. 52, no. 1, pp. 80–89, 2009.
View at: Publisher Site | Google Scholar
H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Transactions on Image Processing, vol. 16, no. 2, pp. 349–366, 2007.
View at: Publisher Site | Google Scholar | MathSciNet
Q. Shan, Z. Li, J. Jia, and C. Tang, “Fast image/video upsampling,” ACM Transactions on Graphics, vol. 27, no. 5, pp. 1–7, 2008.
View at: Google Scholar
A. Chambolle and T. Pock, “A first-order primal-dual algorithm for convex problems with applications to imaging,” Journal of Mathematical Imaging and Vision, vol. 40, no. 1, pp. 120–145, 2011.
View at: Publisher Site | Google Scholar | MathSciNet
W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, “Learning low-level vision,” International Journal of Computer Vision, vol. 40, no. 1, pp. 25–47, 2000.
View at: Publisher Site | Google Scholar
W. T. Freeman and E. C. Pasztor, “Markov networks for super-resolution,” in Proceedings of the 34th Annual Conference on Information Sciences and Systems, 2000.
View at: Google Scholar
E. Gur and Z. Zalevsky, “Single-Image digital super-resolution: a revised Gerchberg-Papoulis algorithm,” IAENG International Journal of Computer Science, vol. 34, no. 2, pp. 251–255, 2007.
View at: Google Scholar
J. Sun, N. N. Zheng, H. Tao, and H. Shum, “Image hallucination with primal sketch priors,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR' 03), vol. 2, pp. 729–736, 2003.
View at: Google Scholar
Y. Zhao, J. Yang, Q. Zhang, L. Song, Y. Cheng, and Q. Pan, “Hyperspectral imagery super-resolution by sparse representation and spectral regularization,” EURASIP Journal on Advances in Signal Processing, vol. 2011, no. 1, article 87, 2011.
View at: Publisher Site | Google Scholar
M. F. Tappen, B. C. Russell, and W. T. Freeman, “Exploiting the sparse derivative prior for super-resolution and image demosaicing,” in Proceedings of the IEEE Workshop on Statistical and Computational Theories of Vision, 2003.
View at: Google Scholar
W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-based super-resolution,” IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56–65, 2002.
View at: Publisher Site | Google Scholar
J. Sun, Z. Xu, and H.-Y. Shum, “Image super-resolution using gradient profile prior,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '08), pp. 1–8, Anchorage, Alaska, USA, June 2008.
View at: Google Scholar
J. Yang, J. Wright, T. Huang, and Y. Ma, “Image super-resolution as sparse representation of raw image patches,” in Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition (CVPR '08), pp. 1–8, Anchorage, Alaska, USA, June 2008.
View at: Publisher Site | Google Scholar
J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010.
View at: Publisher Site | Google Scholar | MathSciNet
J. Yang, Z. Wang, Z. Lin, S. Cohen, and T. Huang, “Coupled dictionary training for image super-resolution,” IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3467–3478, 2012.
View at: Publisher Site | Google Scholar
R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” in Curves and Surfaces, vol. 6920 of Lecture Notes in Computer Science, pp. 711–730, Springer, 2012.
View at: Google Scholar
R. Timofte, V. Smet, and L. Gool, “Anchored neighborhood regression for fast example-based super-resolution,” in Proceedings of the International Conference on Computer Vision (ICCV '13), December 2013.
View at: Google Scholar
C. Dong, C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in Computer Vision-ECCV 2014, vol. 8692 of Lecture Notes in Computer Science, pp. 184–199, Springer, New York, NY, USA, 2014.
View at: Publisher Site | Google Scholar
S. Borman and R. L. Stevenson, “Super-resolution from image sequences-a review,” in Proceedings of the Midwest Symposium on Circuits and Systems, pp. 374–378, Notre Dame, Ind, USA, 1998.
View at: Publisher Site | Google Scholar
C. B. Atkins, C. A. Bouman, and J. P. Allebach, “Optimal image scaling using pixel classification,” in Proceedings of the IEEE International Conference on Image Processing (ICIP '01), pp. 864–867, Thessaloniki, Greece, October 2001.
View at: Google Scholar
E. Candes and C. Fernandez-Granda, “Towards a mathematical theory of super-resolution,” Communications on Pure and Applied Mathematics, vol. 67, no. 6, pp. 906–956, 2014.
View at: Publisher Site | Google Scholar
F. Viola, A. W. Fitzgibbon, and R. Cipolla, “A unifying resolution-independent formulation for early vision,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR '12), pp. 494–501, 2012.
View at: Google Scholar
G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Transactions on Graphics, vol. 30, no. 2, article 12, 2011.
View at: Publisher Site | Google Scholar
L. Xu, C. Lu, Y. Xu, and J. Jia, “Image smoothing via $L_{0}$ gradient minimization,” ACM Transactions on Graphics, vol. 30, no. 6, article 174, 2011.
View at: Publisher Site | Google Scholar
K. I. Kim and Y. Kwon, “Single-image super-resolution using sparse regression and natural image prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 1127–1133, 2010.
View at: Publisher Site | Google Scholar
Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A new alternating minimization algorithm for total variation image reconstruction,” SIAM Journal on Imaging Sciences, vol. 1, no. 3, pp. 248–272, 2008.
View at: Publisher Site | Google Scholar | MathSciNet

Copyright

Copyright © 2014 Zhang Liu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

2197

Downloads

1342

Citations