Abstract
The spatial resolution of digital images is the critical factor that affects photogrammetry precision. Single-frame, superresolution, image reconstruction is a typical underdetermined, inverse problem. To solve this type of problem, a compressive, sensing, pseudodictionary-based, superresolution reconstruction method is proposed in this study. The proposed method achieves pseudodictionary learning with an available low-resolution image and uses the -SVD algorithm, which is based on the sparse characteristics of the digital image. Then, the sparse representation coefficient of the low-resolution image is obtained by solving the norm of minimization problem, and the sparse coefficient and high-resolution pseudodictionary are used to reconstruct image tiles with high resolution. Finally, single-frame-image superresolution reconstruction is achieved. The proposed method is applied to photogrammetric images, and the experimental results indicate that the proposed method effectively increase image resolution, increase image information content, and achieve superresolution reconstruction. The reconstructed results are better than those obtained from traditional interpolation methods in aspect of visual effects and quantitative indicators.
1. Introduction
High-quality images are required for high-precision measurements, and their spatial resolution is the key indicator in the assessment of digital image quality; an image with a higher spatial resolution contains more information content. The level of image detail has a decisive role in absolute photogrammetry precision and the range of applications for which a given image can be used. However, limited by flight condition and cost as well as optical diffraction, certain difficulties exist in directly acquiring high-resolution images. To satisfy the demand of image resolution using single-frame-photogrammetry of local geospatial data, superresolution image reconstruction was investigated in this study.
There are two types of single-frame-image, superresolution reconstruction methods: interpolation and learning. Traditional interpolation methods achieve high resolutions using an interpolation kernel function and include methods such as the nearest neighbor, bilinear interpolation, bicubic interpolation, and spline function methods. Scholars later proposed edge-oriented interpolation methods. For example, Jensen and Anastassiou [1] improved visual effects by template-fitting after image edge detection; Carey et al. [2] proposed a wavelet-based interpolation method; and Chang et al. [3] used a wavelet transform to describe edge points, infer extreme point coefficients at finer scales, and reconstruct high-resolution images. Li and Orchard [4] also proposed an edge-guide interpolation algorithm based on the least squares method, where the edge feature of an interpolated image was maintained based on the edge-guide feature of the adaptive covariance. Zhou et al. [5] proposed a multisurface fitting-based interpolation algorithm that could increase the edge clarity of an interpolated image; that algorithm had good robustness against noise. The interpolation method is simple and applicable to many situations; however, it lacks prior information, can only enhance image visual effects, and typically has difficulty in recovering high-frequency information that is lost in low-resolution images, producing blurred reconstructed images.
Learning-based superresolution reconstruction is achieved by establishing corresponding relationships between high- and low-resolution images using prior information. The example-based superresolution method proposed by Freeman et al. [6] obtained prior information between high- and low-resolution images via Markov random field learning; this type of method achieved excellent reconstruction results in human-facial [7] and word-processing applications. The sparse-representation-based SR algorithm proposed by Yang et al. [8] used the sparse characteristics of natural images and linear programing to solve for the sparse representation of the low-resolution images and then combined the obtained representation coefficient with a high-resolution dictionary to create high-resolution image tiles. Dong et al. [9] proposed an image interpolation algorithm based on a nonlocal autoregression model under the framework of sparse representation. This method used similar image blocks that were common in natural images to construct a nonlocal autoregression model; it then used self-similarity in the image structure as additional information to reconstruct high-resolution images.
The premise of the learning method is to obtain prior knowledge of the high-resolution images. Studies typically start with existing images, produce corresponding low-resolution images via blurring and downsampling, and then use a reconstruction algorithm to approximate the original images. The reconstructed images typically do not exhibit substantial improvements in image quality or resolution. Although the amount of information in the interpolated, superresolution, image reconstruction did not increase, the resolution can be increased markedly. Due to the limitations of traditional reconstruction methods, this study uses the local, geometrically consistent, spatial relationship between the interpolated and original images; organically combines the interpolation and learning methods; and proposes a compressed sensing, pseudodictionary-based, superresolution reconstruction method.
2. Method and Model
The proposed method treats an existing image as a low-resolution image, uses bicubic interpolation images as guides, and creates a joint dictionary for image training based on the sparse characteristics of digital images by requiring that the high- and low-resolution images have the same sparse representation in the corresponding high- and low-resolution dictionary. Then, the related rule between the high- and low-resolution images is applied to the bicubic interpolation image, and the optimization problem is solved using compressed sensing theory. The proposed method can effectively retrieve more detailed high-frequency information while producing a superresolution image.
2.1. Training Sample Extraction and Pseudodictionary Learning
The goal of the superresolution reconstruction of a single-frame image is to restore a given low-resolution image to a high-resolution image of the same scenery. To resolve this ill-posed problem, the requirement of a consistent observation model for high- and low-resolution images must be satisfied [10]. Based on the digital imaging mechanism, the low-resolution image can be obtained by sampling the high-resolution image, and the observation model can be expressed as follows:where and describe the blurring and downsampling operations on the image, respectively; is the Gaussian noise; and are the high- and low-resolution images, respectively; and the approximation image of the original high-resolution image can be reconstructed by processing . To avoid complexity induced by the differences in the high- and low-resolutions of the images, the low-resolution image is upsampled to the same dimension as the high-resolution image via interpolation, thus obtaining high- and low-resolution image sample sets and , respectively.
The preprocessing of the samples is first performed after the construction of the training samples. Because the images’ structural characteristics are primarily represented in the high-frequency range, the image interpolation value is used as the high-frequency information of the high-resolution image. The preprocessing of the low-resolution image is performed by a convolution integral of degree on the geometric structure components; the characteristic vector components are combined to form the characteristic vector of the low-resolution image. Finally, the characteristic vectors of the high- and low-resolution images at the same position are formed into a training sample data pair .
The compressed sensing, pseudodictionary-based, superresolution reconstruction problem requires the simultaneous use of an over-defined dictionary pair and of the high- and low-resolution image tiles. Because the pair belongs to a heterogeneous data type, the following equation must be solved for the training data to have the same sparse representation in both dictionaries [11]:where and are the high- and low-resolution image tile dictionaries, respectively; is the sparse coefficient that simultaneously satisfies in dictionary and in dictionary ; and and are the regularization parameters of the second and third terms, respectively. The joint dictionary receives the correlation between the high- and low-resolution training samples in full consideration, and learning is required to use prior knowledge of the high-resolution image to effectively guide the superresolution reconstruction of other low-resolution images. This study uses the -SVD method [12] to solve an over-defined dictionary.
2.2. Mathematical Model of Compressed Sensing
The other key element in superresolution reconstruction is to reconstruct the low-resolution dictionary’s sparse representation with respect to the low-resolution image tiles. Compressed sensing is a new theory in the field of signal processing. dimensional signal can be expressed by the superposition of a set of orthonormal basis as follows [13, 14]:If has only nonzero elements or elements that are far greater than other points, then the signal is called sparse signal, where , and is the sparse domain of . The sampling process of compressed sensing is achieved using a sensing matrix that is not correlated to the basis [13]:where is an -dimensional sample matrix, , and is the sensing matrix, which is also called the sparse dictionary.
Signal reconstruction is the core of compressed sensing. For a sparse signal , the sparse coefficient can be reversely derived with a high probability from the -dimensional observation value and the -sparsity constraint condition when the sensing matrix satisfies the isometric condition. The original signal can then be restored with a high probability based on the reversibility of the orthogonal transformation. Thus the signal reconstruction problem can be considered as a search for the sparsest solution to the underdetermined equation set .
2.3. Flow of the Proposed Superresolution Reconstruction Algorithm
This study uses the bicubic interpolation image of an existing image as the high-resolution image . Then, is processed by the same blurring and downsampling as in the training stage, to obtain the corresponding low-resolution . Based on the detailed characteristics of the bicubic interpolation that remain unchanged in spatial location, the pseudodictionary obtained in the training stage is directly applied to the higher level bicubic interpolation image. The superresolution reconstruction problem can be described as the problem of solving for the optimum approximation of the superresolution image with the given . After completing the high- and low-resolution pseudodictionaries, the superresolution reconstruction process can be summarized as follows:
(1) Use the interpolation operator to perform a bicubic interpolation of , producing , which has the same resolution as .
(2) Use the same method as that used in image training to decompose the image into tiles, producing .
(3) Use the image tile sparsity as prior information. Because the natural information is compressible, the superresolution construction of the compressed sensing can be solved using norm optimization. The mathematical model of this process is shown as follows:However, this is NP-Hard problem, and the norm and norm have equivalency under certain conditions [15, 16]. Thus, the above equation can be converted into the convex optimization problem under a minimum norm:
This study estimates the sparse representation coefficient of the low-resolution characteristic block with respect of the pseudodictionary via the optimal orthogonal matching pursuit (OMP) algorithm for the norm.
(4) Based on the same sparse representation in the high- and low-resolution images in their respective sparse pseudodictionaries, their superresolution image tiles are reconstructed by , and the superresolution image is then created.
2.4. Quality Evaluation Method of Reconstruction Results
This study evaluates the superresolution reconstruction results from both subjective and objective perspectives. Image information entropy (InEn) and average contrast (AC) are used as the objective quality evaluation index (NR-IQA) [17] for the superresolution reconstruction results of a no-reference image:where represents the distribution probability of the th gray/color scale; and are the image average contrasts in the - and -directions, respectively; and can be calculated by the following equation:where is the gray-scale of pixel () and and are the number of pixels in the - and -directions, respectively. The calculation method for is the same as that for .
When , let , where . The information entropy and the average contrast of the color image can be obtained from the three-channel information entropy normalization. The information entropy is a measure of the information richness contained in the image; a higher information entropy indicates richer information contained in the image, and a higher average contrast indicates a clearer image.
3. Experiment and Analysis
3.1. Data Source
The foremost task in achieving superresolution reconstruction is to construct a high-resolution training sample. In this study, 96 randomly selected natural images are used as training samples. To verify the effectiveness of the superresolution reconstruction model and algorithm and analyze the results of the superresolution reconstruction of photogrammetric images using the proposed method, the 1 : 5000 photogrammetric image of Xuzhou city (resolution = 5 cm), Hammer Aerial Photographic data (resolution = 375 cm), and a close-up image (resolution = 6 mm) collected by a Canon EOS 5D nonmetric digital camera are used in the superresolution reconstruction experiment in this study. The results are compared to those produced by the bicubic interpolation method. Because all experimental data are colored, the conversion from RGB to YUV space is performed first in a MATLAB environment; the superresolution reconstruction of component uses proposed method, while the other two channels use bicubic interpolation method and finally reconvert the colored image to RGB space. To increase the speed of the experiment, a test image with a size of 512 × 512 pixels is used for this superresolution experiment.
3.2. Parameter Setting
The training image is used as a high-resolution image, and the corresponding low-resolution image is obtained by downsampling, where the downsampling factor is 3, and a one-dimension filter of 1, 3.4, 3, 1/12 is used to blur the training image in the horizontal and vertical directions, respectively. The image is then magnified using the bicubic interpolation method, and the training sample set is composed of 50,000 image tile pairs that were extracted during the experiment. The extraction of low-resolution image characteristics is achieved using first- and second-order gradient operators:
The size of a low-resolution image tile is 3 × 3 pixels, and the overlap is 1 pixel. The number of iterations of the pseudodictionary-pair learning process using the -SVD algorithm is 30, and the dictionary size is 1024. In the testing stage, the bicubic interpolation method is used to magnify the image three times, creating the high-resolution test image . Then, the proposed method is used to achieve a reconstruction that has a superresolution that is three times larger. The iteration termination value of the OMP reconstruction algorithm is = 0.05.
3.3. Results and Analysis
Experiment 1 (impact of ground-feature type on reconstruction accuracy). To analyze the impact of different ground-feature types on the image reconstruction results, the Xuzhou 1 : 5000 aerial photogrammetric image is used as source data for capturing images of four typical ground features (a building, vegetation, bare soil, and a body of water) used as test data in this study. The reconstruction results are shown in Figure 1. For comparison, the original image is simply magnified using the nearest neighbor interpolation method.
To evaluate the quality of a color image, the human objective effect is strong. As shown in the resulting images, the image produced by the traditional bicubic interpolation method is blurred, and the reconstructed image lacks clear details at edges and exhibits an irregular structural region with serious losses of texture. The reconstructed image produced by the compressed sensing method is clearer; the locally magnified image that is 10 times larger shows more detailed textures and an improved edge effect.
The quantitative index values of the reconstructed images produced by the bicubic interpolation method and the proposed method are shown in Table 1. Compared to the bicubic interpolation method, the compressed sensing method exhibits certain improvements in both the information entropy and average contrast. The information entropy of the four typical features (i.e., building, vegetation, bare soil, and water body) increases by 0.0284, 0.0342, 0.0425, and 0.0505, respectively; the image contrast also increases by 0.5014, 1.2047, 0.8966, and 0.4926, respectively. The information entropy of the image reconstructed by the proposed method increases by 0.0347, and the average image contrast increases by 0.7128.
To describe the degree of enhancement of various ground-feature image reconstruction indices, the increased values are drawn into related curves based on ground-feature types, producing Figure 2.
With regard to image information entropy, the entropy gradually increases from building to vegetation, bare soil, and water body. The increase in entropy is most marked for the water body. In terms of image contrast, the increase in vegetation is the most prevalent, while that of the water body is least prevalent.
(a) Original LR image
(b) Bicubic interpolation method
(c) Proposed method
Experiment 2 (impact of image resolution on reconstruction accuracy). Image spatial resolution is an important index when evaluating image quality; it directly reflects the level of detail of the information of the imaged object. A higher spatial resolution indicates that the image has more detailed information, including edges. The purpose of superresolution reconstruction is to increase the spatial resolution of an image and restore more image information. To examine the impact of the original image resolution on the superresolution reconstruction results, the reconstruction of a 375 mm resolution Hammer photogrammetric image, a 50 mm resolution photogrammetric image of Xuzhou city, and a 6 mm resolution close-up image captured by a nonmetric digital camera are compared; the reconstruction results are shown in Figure 3.
Considering the visual effect, the images reconstructed by the proposed method show more detailed high-frequency information, and their geometric textures are more prevalent. The corresponding quantitative indices are listed in Table 2. To mitigate the influence of different ground-feature types on the reconstruction results, the averages of building, vegetation, bare soil, and water body in the reconstructed image of Xuzhou city were determined. As shown in Table 2, for images with resolutions of 375, 50, and 6 mm, the information entropy of the reconstructed images produced by the proposed method increased by 0.0417, 0.0389, and 0.0232, respectively, compared to those produced by the bicubic interpolation method; additionally, the image contrast increased by 1.131, 0.7739, and 0.5659, respectively.
To describe the impact of image resolution on the reconstruction accuracy in more detail, the reconstruction results in Table 2 are used to create quantitative indices of improvement curves with regard to image resolution, as shown in Figure 4. Because the original image’s resolution continues to decrease, the improvement in the accuracy of the reconstructed image shows a continuously increasing trend.
4. Discussion
Currently, many studies of the superresolution reconstruction of single-frame images using the sparse representation and dictionary-learning methods have been published in the literature [8, 18, 19]. These studies generally use methods such as the adaptive sparse domain and adaptive regularization parameters [20] to improve reconstruction accuracy using the improved dictionary-learning method [21, 22], and they typically produce good reconstruction results. However, these studies used existing images as high-resolution images, and the superresolution reconstruction process is merely a signal restoration of the downsampled images; the image resolution did not receive essential improvement. For this reason, this study used compressed sensing as a theoretical construct along with the similarity of the high- and low-resolution image characteristics to train high- and low-resolution image dictionaries using the -SVD method. Using a bicubic interpolation image as prior information, the proposed method performed a superresolution reconstruction of the existing images, which resulted in an image with a resolution triple that of the original image. The experimental results of this study indicated that the image obtained by the proposed reconstruction method increased both in visual effect and quantitative index compared to the image reconstructed using the bicubic interpolation method.
The superresolution reconstruction of single-frame images has been successfully applied in the reconstruction of remote-sensing images [23], medical imaging, “Resource Satellite III” images [24], Moon Rover images [25], and SAR images [26]. High-resolution imagery is necessary for high-precision photogrammetry and can increase the absolute measurement precision of photogrammetry. Thus, this study of the superresolution reconstruction of a single-frame photogrammetric image is important. This study introduces the superresolution reconstruction concept into the photogrammetry field, producing superresolution reconstructions of single-frame photogrammetric image experiments from the MATLAB platform and superresolution images that exhibit improved visual effect and accuracy compared to those produced by the bicubic interpolation method.
The current studies in the field all concentrate on improving the sparse domain and dictionary-learning methods to improve algorithm accuracy. However, the impact of different ground-feature types and the image resolution of the photogrammetric images on the reconstruction accuracy is marked; this study conducted detailed studies of these phenomena. The texture structure and gray-scale variation of different types of ground features are reflected differently on photogrammetric images, which inevitably induce differences in reconstruction accuracy. Thus, to achieve superresolution reconstruction, this study used a 5 cm resolution image of Xuzhou city as an experimental object to create a superresolution reconstruction of four typical ground features in the photogrammetric image. This study then conducted an accuracy comparison with the traditional bicubic interpolation method based on the quantitative index analysis. The experimental results of this study indicate that the building image has the most improved reconstruction result due to the clarity and richness of the image. Additionally, the four resulting images all exhibit certain degrees of improvement, and the improvement in the image information entropy exhibits an increasing trend from building, vegetation, and bare soil to water body. In general, lower original image entropy produces more marked improvements. Conversely, the increase in image contrast is marked because vegetation exhibits the richest color information and the most frequent variation in gray-scale. The texture of the water-body image is the most monotone, and thus, its increase in contrast is the lowest. Additionally, the increment in image contrast of different ground features maintains a consistent relationship with the self-contrast of the images of these features.
Image spatial resolution is an important index for judging the amount of detailed information in an image. The direct purpose of superresolution reconstruction is to increase image resolution; however, the reconstruction process is built on the basis of existing images, and thus, the resolution of the existing image will inevitably influence the reconstruction accuracy. Thus, this study performed a superresolution reconstruction experiment on three images of different resolutions. The results of this study indicate that as the original image resolution decreases, the reconstructed images exhibit increases in the quantitative indices. Thus, the proposed superresolution reconstruction method is particularly suitable for low-resolution images.
5. Conclusions
Blurry images produced during image reconstruction are common when using the traditional interpolation method. Additionally, the learning method requires prior information of the resulting high-resolution image to perform superresolution reconstruction; reconstructed images with superresolutions are thus not produced. This study uses compressed sensing as a theoretical framework; organically combines the interpolation method with the learning method, guided by bicubic interpolation images; and produces a superresolution reconstruction of an existing image at three times the scale. This study uses the classical OMP reconstruction algorithm to solve the sparse optimization problem and effectively increases image resolution. Compared to traditional bicubic interpolation method, the reconstruction based on the proposed method increases the image information content and clarity. The proposed method provides an effective way to improve the application range and accuracy of aviation and aerospace photogrammetric images. In the future, the sparse optimization method will be improved further to produce even better reconstruction results.
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
The authors wish to acknowledge funding from the National Fund Committee of China, which supports a Project no. 41171343 through National Natural Science Foundation for Research on precise mark location and precision estimation of digital photogrammetry image.