Abstract
A new method for superresolution image reconstruction based on surveying adjustment method is described in this paper. The main idea of such new method is that a sequence of low-resolution images are taken firstly as observations, and then observation equations are established for the superresolution image reconstruction. The gray function of the object surface can be found by using surveying adjustment method from the observation equations. High-resolution pixel value of the corresponding area can be calculated by using the gray function. The results show that the proposed algorithm converges much faster than that of conventional superresolution image reconstruction method. By using the new method, the visual feeling of reconstructed image can be greatly improved compared to that of iterative back projection algorithm, and its peak signal-to-noise ratio can also be improved by nearly 1 dB higher than the projection onto convex sets algorithm. Furthermore, this method can successfully avoid the ill-posed problems in reconstruction process.
1. Introduction
Obtaining images is a process from ideal high-resolution (HR) image to observation image. Superresolution image reconstruction (SRIR) is a reverse process of obtaining images, which uses many low-resolution (LR) images obtained by multiple observations on the same target to calculate the real image of the target. SRIR was firstly proposed by Harris [1] and Goodman [2]. It is similar to the process of surveying adjustment, which gets optimal value by multiobservations on the same object in field of surveying. SRIR can obtain one or more HR images from multiple low-resolution images with complementary information. It can overcome the limitation of hardware and correspondingly increase the image resolution. Nowadays, SRIR techniques play an important role in remote sensing, video security monitoring, military aerial reconnaissance, medical digital image processing, and video standard conversion. One of the most typical applications of SRIR technology is in SPOT5 satellite of France. The satellite takes two images with resolution of 5 m, but with mutual dislocation of 0.5 pixel. Images with resolution of 2.5 m are obtained by SRIR and provided to the whole world.
Up to now, a lot of algorithms have been proposed [3–16]. These algorithms can be roughly divided into two categories: one is frequency-domain algorithm [17–19] and the other is spatial-domain algorithm [4, 13]. In frequency-domain algorithms, images are transformed onto frequency domain and SRIR is carried in frequency domain. In spatial-domain algorithm, image gray value are computed directly in the process of SRIR. Frequency-domain algorithms are usually based on idealistic assumptions, so their application is very limited and cannot be effectively applied in most cases. While spatial-domain algorithm [20–25], such as nonuniform spatial interpolation, iterative back projection (IBP), has difficulty to make use of a priori information, hence the quality of SRIR will be severely limited [26]. Even the projection onto convex sets (POCS) algorithm, a real popular method, sometimes is unstable in computation and heavily depends on the initial estimation of SRIR, and its solution is often not unique and so on. Actually, most of the above two kinds of algorithms are proposed according to some specific situations; the performance of SRIR from multiple images usually depends on the imaging system and the accurate degradation model. The algorithm universality of SRIR is rarely dealt with. In surveying and mapping field, if the same object is observed multiple times, the optimum value of the object can be obtained by using surveying adjustment. SRIR is similar to this process. If a real image can be represented by a gray function, any pixel value will be the integration of its gray function in the pixel, and hence the relationship between observation images (namely, LR images in SRIR) and real image (namely, HR image in SRIR) can be described by an integration relationship. Based on this principle, a general surveying adjustment model containing integration is established in this paper. And, on this model, a new method for SRIR is proposed. The main idea of this new method for SRIR is that a sequence of low-resolution images are taken firstly as observations, and then observation equations are established by using survey adjustment method. The gray function of the real image is estimated on the observation equations. After that, the estimated gray function is used to calculate HR pixel value of the corresponding area. Finally, HR image can be obtained.
In order to evaluate quantitatively reconstruction image, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are usually used as two evaluation indicators. PSNR describe gray similarity between two images, and SSIM describe structural similarity of two images. PSNR and SSIM are simultaneously used to get a comprehensive evaluation for reconstruction images in this paper [6].
2. The Principle of SRIR Based on the Surveying Adjustment
Pixel value of any image can be regarded as integration of its gray function within the scope of the pixel; namely, where is pixel value, and are pixel coordinates, is pixel scope, and is the gray function of object surface. If of the real image is known, superresolution image with any resolution can be calculated by (1). Conversely, if gray values of pixels are known, we can estimate on these gray values of pixels. Basic idea is to take pixel gray values of LR images as observations and real value of gray function as unknowns and then use surveying adjustment method to estimate , which will be the closest to the real value of gray function, and then pixel values of SR can be calculated by gray function .
For one or more LR images coexist within the same area, it can be regarded as multiple observation for the real gray function in such area. Considering the errors which possibly existing in the different pixel gray values, the observation equations can be approximately expressed as follows: where The above observation equations can also be expressed as a matrix: where is the vector of observation, is correction vector of observations, gray function is unknown, and is the number of pixels of LR images related to the same small region. It is an integral relationship between observations and unknowns in the above observation equations; thus we call (4) as “adjustment model with integration.” Different from traditional adjustment model, the functions here must be determined rather than parameters. However, how to find the solution of (4) needs to be further studied. For images, gray function can be denoted by Fourier series or by wavelet function. Here, we use polynomial to substitute the gray function and transform (4) into a model of parameters estimation. If gray function is replaced by quadratic polynomial within a sufficiently small region, such as Substituting (5) into (2), the following is obtained: where Equation (4) can be written again as follows: According to the least square adjustment method, the solution can be found as follows: where is the weight of observations. Estimation of real gray function can be calculated by putting into (5), and then (10) is obtained by putting into (1): According to (10), we can calculate pixel values of HR image in SRIR.
3. The Application of SRIR Based on Surveying Adjustment
When using (4) or (8) for SRIR, one key is to determine the region that the gray function will deal with, namely, how to select pixel values as observations. Generally, real gray function of object surface is very complicated; if we use a quadratic function to substitute the real gray function, it must be in a small region. In this paper, we determine the region according to the following method: for any pixel of the reference image, one can get its surrounding 8 pixels, and then total of 9 pixels can be taken as observations. For the second image after registration, one should select the pixel closest to pixel in reference image and its surrounding 8 pixels as observations, and so on. For every LR image, 9 pixels can be selected as observations. Based on these observations, an estimate of real gray function can be obtained according to (9). And then pixel value of coordinate and the surrounding 8 pixel values of SR image can be calculated by (10). Obviously, the computation of this method will converge and can successfully avoid the ill-posed problems.
When (9) is used, it is necessary to determine the weight of observations. In fact, pixel value above contains two kinds of noise: one is the noise of image itself and the other is the model error brought in by the substitution of quadratic polynomial. The model errors will depend on the distance to the center of the region using quadratic polynomial. Accordingly, the scheme of weighting observations is taken as follows: for any pixel of the reference image, one can calculate the distances between and the surrounding 8 pixels and the distances between and the corresponding 9 pixels of the second image. Then, the weight values of 17 observations are defined by inverse distances square; that is, where denotes the distance between the concerned pixel and pixel and denotes the weight of the observation corresponding to the concerned pixel. The weight value of pixel itself will use where is half pixel width.
Pixel values of HR image exist overlapping area according to (10); overlapping area can be shown in Figure 1. For the overlapping region, we use the average value as the last HR pixel value.
4. Testing and Results Analysis
In order to check the feasibility of the suggested method, we design tests according to the following idea: we add some noise to a HR image and then take samples from the HR image to obtain a series of LR images. By using these LR images, we try to reconstruct the HR image. Comparing with the original image, we can know the feasibility of the suggested method. The whole process can be divided into the following four steps: obtaining LR images, image registration, fuzzy estimation, and image reconstruction.
4.1. Obtaining LR Images
We can firstly select a HR image, and then we transform the HR image into a sequence of LR images. Image transformation includes image translation and rotation, fuzzy processing, down sampling, and image noise. An example with a size and gray-scale level 256 image is as shown in Figure 2; we simulate four LR images with subpixel displacement. The sequence of LR images is created through the following steps. Firstly, original HR image is shifted with different motion vectors in vertical and horizontal directions to produce four images. Secondly, these images are converted to frequency domain for setting frequency range 0-0.1 to zero; we can get four blurred images. Thirdly, these images are downsampled by the factor of 2 in each direction to produce four LR images of a size of . Finally, we add Gaussian noise to downsampled images, where the variance is 0.05 and the mean value is 0.
4.2. Image Registration
Image registration is to choose one LR image as reference frame, and the other LR images are aligned with reference frame on space. LR images registration is irregular sampling of HR grids. Registration models usually include translation model, affine transformation model, bilinear model, projection model, and rigorous model [26]. The scale invariant feature transform (SIFT) algorithm is adopted here to get subpixel precision in image registration. Image registration based on characteristics can be generally divided into the following three steps: feature extraction, feature description, and feature matching [27].
4.3. The Fuzzy Function and Noise Estimation
Fuzzy function describes image formation process affected by fuzzy degradation. The precision of the estimation of fuzzy function can directly affect the quality of SRIR. In these experiments, Gaussian fuzzy function is adopted. And Gaussian noise with mean 0 and variance 0.05 is added into LR image. Wiener filtering is used to deblur and denoise for LR images in the image preprocessing stage.
4.4. Image Reconstruction
A sequence of LR images are used to reconstruct the HR image with the same size of original image by IBP, POCS, and this algorithm. Reconstruction results are shown in Figure 3. We can see that the visual feeling of Figures 3(b) and 3(d) is better than that of Figure 3(c). For the detailed comparison, we can take a small part of the images and enlarge it in Figure 4. From Figure 4, we can clearly see that the reconstruction images have richer details. Finally, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are both used to evaluate the objective reconstruction. PSNR is a measure of gray similarity between two images. SSIM is a measure of structural similarity of two images. PSNR and SSIM can be simultaneously used to evaluate comprehensively reconstruction image from gray similarity and structural similarity [24]. The results are shown in Table 1. One can see that PSNR of the suggested method is nearly 1 dB higher than that of the other two algorithms in Table 1. This means that reconstruction image by the suggested algorithm has higher gray level similarity with original image than the other two methods. From the value of SSIM, one can see that reconstruction image from the suggested algorithm in this paper has the same structural similarity with the other two algorithms.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
5. Conclusions
The method proposed in this paper is based on surveying adjustment theory. Examples show that the method is feasible. The quadratic polynomial can be used as gray function of object surface and the convergence speed of the methods is faster than that of POCS and IBP algorithm. Results also showed that the proposed method can obtain a good result of reconstruction image through the subjective and objective comprehensive evaluation.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The authors would like to thank the National Basic Research Program (no. 2013CC733303), the National Natural Science Foundation of China (nos. 41274010 and 40974007), the National High Technology Research and Development Program of China (no. 2012AA121301), and the Fundamental Research Funds of Central South University for financial support.