Abstract
We introduce an efficient superresolution algorithm based on advanced nonlocal means (NLM) filter and iterative back projection for hyperspectral image. The nonlocal means method achieves the to-be-interpolated pixel by the weighted average of all pixels within an image, and the unrelated neighborhoods are automatically eliminated by the trivial weights. However, spatial location distance is also an important issue to reconstruct the missing pixel. Therefore, we proposed an advanced NLM (ANLM) filter considering both neighborhood similarity and patch distance. In the conventional NLM method, the search region was the whole image, while the proposed ANLM utilizes the limited search to reduce the complexity. The iterative back projection (IBP) is a very famous method to deal with the image restoration. In the superresolution issue, IBP is able to recover the high-resolution image iteratively from the given low-resolution image which is blurred due to the noise by minimizing the reconstruction error, while, because the reconstruction error of IBP is back projection and isotropic, the conventional IBP suffers from jaggy and ringing artifacts. Introducing the ANLM method to improve the visual quality is necessary.
1. Introduction
Recently, the space technology developed very fast and it could take the visual observation of earth from the artificial satellites. Hyperspectral image obtains terrestrial information in various contiguous and narrow spectral bands. Hyperspectral image is attracting more and more researchers due to its wide applications in aerial and space imagery fields, agriculture, ecology, geology, medicine, and meteorology. Hyperspectral image has a high resolution in spectral domain, while the resolution in spatial domain is limited. Moreover, when we acquire the hyperspectral image, many issues degrade the quality of hyperspectral image such as atmospheric scattering, secondary illumination, and sensor noise. Therefore, improving spatial resolution is a key issue in the hyperspectral image applications and high resolution can make various applications easier. However, it is very expensive to change the resolution in the hardware way. Thus, it is necessary to develop the technology to improve the spatial resolution in software method and superresolution is becoming a popular technique to solve this problem.
Image superresolution is a technique that reconstructs a high-resolution (HR) image from one or few given low-resolution (LR) images. These LR images are downsampled and blurred due to degradation of imagery system [1]. Huang and Tsai [2] firstly introduced the superresolution problem in the frequency domain but disregarded the blur in the image processing. Some researchers present maximum a posteriori (MAP) estimator-based algorithms to solve the interpolation and deblurring problems [3–5]. Projection onto convex sets (POCS) is also a very popular technique in image restoration issues and therefore is implemented in superresolution algorithms [6–8].
The superresolution methods have been developed for a long time for natural images. However, with the development of space technology, hyperspectral images become more popular and widely used. Many researchers start to research the superresolution method on hyperspectral images to improve the spatial resolution. For example, Akgun et al. [1] present a hyperspectral image acquisition model to simulate the degradation process and provide a POCS-based superresolution method to improve the spatial resolution of hyperspectral images. Mianji et al. [9] reviewed some superresolution methods for hyperspectral imagery and provided the challenges in this area. Ma et al. [10] proposed an operational superresolution approach for multitemporal and multiangle remotely sensed imagery.
In this paper, we propose an advanced nonlocal means (ANLM) and iterative back projection- (IBP-) based superresolution method for hyperspectral imagery. IBP is a famous superresolution method proposed by Irani and Peleg [11]. The main concept of IBP is to minimize the error during the iterative process. It is reported that IBP can give good performance and achieve the superresolution and deblurring simultaneously. However, this iterative process often causes jaggy and ringing artifacts in the detail region. The proposed ANLM filter is an improved version of the traditional nonlocal means (NLM) filter established by Buades et al. in [12], which was inspired from Yaroslavsky neighborhood filter [13]. The NLM has been widely used in image processing such as denoising and deblurring [14, 15]. NLM estimates the missing pixel as the weighted average of the pixels whose neighborhoods seem like the neighborhood of the missing pixel. Known from Yaroslavsky neighborhood filter, the spatial patch distance also affects the performance of the superresolution method because long distance has smaller chance to have similar neighborhood than short distance. Moreover, NLM needs to compute all the neighborhood similarities in the whole image which is computationally expensive. Thus, we propose ANLM where the missing pixel is smoothed as the weighted average of all pixels that have similar Gaussian neighborhoods and close patch distance.
The rest of the paper is organized as follows. The proposed ANLM method is described in Section 2. Simulation results and their corresponding discussion are explained in Section 3. Finally, conclusions are drawn in Section 4.
2. Proposed Method
2.1. Advanced Nonlocal Means (ANLM) Filter
A basic digital signal superresolution system can be realized as weighted linear average. Given a discrete LR image , we want to estimate the HR image ; the conventional methods adopt the uniform weighted sum of average values and can be expressed aswhere denotes the search region of a fixed size and is centered at the missing pixel located at , is the weight, is the pixel value of the given LR image, and is the estimation of the missing pixel of the reconstructed HR image.
The NLM filter restores the missing pixel by taking an average value of the neighboring pixels with a similar neighborhood. Using a Gaussian kernel for evaluating the similarity, the formula can be expressed aswhere is the search region and is the weight of NLM filter defined aswhere and are square neighborhood centered at and , and are the intensity gray level vectors which are composed of pixels in the neighborhoods and , and is a filtering parameter. It is apparent that the similarity between two pixels at and depends on the similarity of the intensity gray level vectors and .
Most NLM-based methods only consider neighborhood similarity and ignore the spatial locality distance which results in poor performance. Therefore, we propose an ANLM method in order to obtain an improved performance while considering the spatial distance [15]. In ANLM, the patch distance is introduced as a new weight; then we redesign the weight function asWhere and are filtering parameters. Then, we apply to the ANLM filter as follows:The search region of conventional NLM was the entire image. In the proposed ANLM, the introduced patch distance is increasing when the given pixel is farther from the missing pixel . When is larger than some constant value, is close to zero. Particularly, the weight is trivial for the larger patch distance. Thus, we can neglect these pixels that are located far from the missing pixel . We search the similar neighborhood in a limited search region instead of the whole image as in NLM, which reduce the computational complexity considerably.
2.2. Iterative Back Projection (IBP)
The goal of superresolution algorithms is to restore the desired HR image from downsampled and blurred LR image. Irani and Peleg proposed an IBP superresolution reconstruction method [11]. In IBP, the difference between the simulated and given LR images is repeatedly iteratively back projected to obtain the desired HR image. The process continues until some conditions are satisfied or the maximum iteration number is reached. IBP can be formulated as where is estimated HR image in the iteration, is estimated HR image in the iteration, and is the estimation error which is computed bywhere is the simulated LR image which is the downsampling and blurring of , is upsampling, and is back projection kernel.
2.3. The Proposed ANLM-IBP Method
To improve the performance of IBP, we propose an ANLM-IBP-based superresolution algorithm for hyperspectral imagery. First, we use ANLM to reconstruct the initial HR image. Then, during the iterative process of IBP, ANLM guides the error propagation of IBP. By combining ANLM and IBP, one can achieve better objective and subjective performance. The flowchart of the proposed ANLM-IBP is drawn in Figure 1.
In summary, the proposed ANLM-IBP method is described in the following steps:(1)apply ANLM to obtain initially estimated HR image;(2)use degradation model for downsampling and blurring the estimated HR image to obtain the simulated LR image;(3)calculate estimation error between the given original LR image and the simulated LR image;(4)utilize ANLM to interpolate the estimation error;(5)judge if the terminal condition is satisfied. If it is Yes, computed image is the reconstructed HR image. If it is No, we add the error to the estimated HR image and reobtain an updated HR image for the next iteration.
3. Experimental Results
In the experimental results, we adopt a well-known dissimilarity criterion called peak signal-to-noise ratio (PSNR) in decibels (dB), which is calculated as where and are the original and reconstructed images, respectively. We measured the objective performance in terms of PSNR, which is a widely adopted criterion in the literature.
The proposed method was tested with 191-band airborne multispectral scanner data set [16] and 220-band spectral image acquired with the AVIRIS data set (shown in Figures 2 and 3). For the subjective performance evaluation in terms of visual effect, we show part of perceived image quality in Figures 4 and 5. As can be seen in Figure 3, result images show that the proposed method yields a better visual quality with details which has sharper edge and better object boundary.
(a)
(b)
(c)
(d)
(e)
(a)
(b)
(c)
(d)
(e)
Table 1 shows objective performance comparison in PSNR metric. Our proposed method yields better performance than the other exiting methods.
4. Conclusion
We propose an advanced nonlocal means and iterative back projection-based superresolution algorithm for hyperspectral imagery. We introduce an improved version of nonlocal means filter that automatically selects the edge orientation using the weighted average of a similar neighborhood and reduces the complexity of nonlocal means filter by patch distance. In addition, we combine the advanced nonlocal means with iterative back projection to reduce the artifacts caused by the iterative back projection. The experimental results show that we could obtain better objective and subjective performance compared to conventional methods.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was supported by the Incheon National University Research Grant in 2012.