Regularized Multiframe Super-Resolution Image Reconstruction Using Linear and Nonlinear Filters

Khattab, Mahmoud M.; Zeki, Akram M.; Alwan, Ali A.; Bouallegue, Belgacem; Matter, Safaa S.; Ahmed, Abdelmoty M.

doi:https://doi.org/10.1155/2021/8309910

Journal of Electrical and Computer Engineering

On this page

Abstract Introduction Experimental Results and Discussion Conclusion Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2021 | Article ID 8309910 | https://doi.org/10.1155/2021/8309910

Regularized Multiframe Super-Resolution Image Reconstruction Using Linear and Nonlinear Filters

Mahmoud M. Khattab,^1,2Akram M. Zeki,¹Ali A. Alwan,³Belgacem Bouallegue,²Safaa S. Matter,⁴and Abdelmoty M. Ahmed²

Academic Editor: Cesare F. Valenti

Received17 Oct 2021

Accepted04 Dec 2021

Published18 Dec 2021

Abstract

The primary goal of the multiframe super-resolution image reconstruction is to produce an image with a higher resolution by integrating information extracted from a set of corresponding images with low resolution, which is used in various fields. However, super-resolution image reconstruction approaches are typically affected by annoying restorative artifacts, including blurring, noise, and staircasing effect. Accordingly, it is always difficult to balance between smoothness and edge preservation. In this paper, we intend to enhance the efficiency of multiframe super-resolution image reconstruction in order to optimize both analysis and human interpretation processes by improving the pictorial information and enhancing the automatic machine perception. As a result, we propose new approaches that firstly rely on estimating the initial high-resolution image through preprocessing of the reference low-resolution image based on median, mean, Lucy-Richardson, and Wiener filters. This preprocessing stage is used to overcome the degradation present in the reference low-resolution image, which is a suitable kernel for producing the initial high-resolution image to be used in the reconstruction phase of the final image. Then, L₂ norm is employed for the data-fidelity term to minimize the residual among the predicted high-resolution image and the observed low-resolution images. Finally, bilateral total variation prior model is utilized to restrict the minimization function to a stable state of the generated HR image. The experimental results of the synthetic data indicate that the proposed approaches have enhanced efficiency visually and quantitatively compared to other existing approaches.

1. Introduction

Over the last two decades, the world has experienced an enormous advancement in software and hardware technologies. Industrial sectors have made the best use of modern technology to generate electronic devices such as computer systems, cellular mobile phones, personal digital assistant (PDA), and innumerable devices at inexpensive costs [1]. Moreover, the manufacturing methods of camera sensors have been highly developed to generate high-quality digital cameras. Additional information is provided from high-resolution (HR) images to have a greater visual perspective. In various imaging applications, HR images are used widely, including but not limited to video surveillance [2], medical imaging [3], forensic imaging [4], and remote sensing [5]. They are still in urgent need for HR image which frequently exceeds the abilities of the HR digital cameras [6, 7]. However, the existing imagery system produces low-resolution (LR) images which must be improved in order to obtain HR images.

In many real-life imaging systems, there are a variety of possible factors for reducing the image resolution due to various physical constraints, inadequate photo detectors, a lower rate of spatial sampling, and an inefficient method for capturing images [8–11]. In the past years, super-resolution (SR) image reconstruction approaches have become a powerful and cost-effective solution to increase the quality of the recorded LR images, meeting the growing market demand for HR images [12–14].

SR image reconstruction increases the image quality retrieved from a series of corresponding LR images. Sensitive information is fused to rebuild the HR image by passing LR images through three processes, namely, image registration, image interpolation, and image restoration. However, the SR image is limited to three major problems: the inaccurate subpixel registration among a sequence of LR images, the ill-conditional nature of the degrading matrix, and the inadequate amount of LR images [15]. The SR issue for multiframe SR image reconstruction was first presented by Tsai and Huang based on the concept of frequency domain introduced by [16]. Since then, various approaches for overcoming the SR problem have been proposed in the literature [8–11, 17, 18]. The main aim of the regularization method is to resolve the SR issue due to its ill-conditional nature. Recent works have been conducted aiming at resolving the SR issue by exploiting the regularization framework [8–11, 19–26]. The idea of reconstruction of the SR image relies on the regularization process that generates the HR image through attempting to minimize the main objective function, where it incorporates both the fidelity of data and the regularization terms. According to the fidelity of the data term that calculates the convergence of both the measured HR image and the obtained LR images, while the main problem is resolved and the stable solution is achieved using the regularization term, the regularized SR image reconstructing employs a maximum a posteriori (MAP) method to match a posteriori distribution across an HR image as the data-fidelity term, whereas the regularization term may be used as an image prior [10, 27].

The process of SR image reconstruction is usually an inverse problem that is difficult to overcome without using any prior knowledge [8, 9, 13, 28]. As a result, a variety of regularization-based SR methods were proposed in the previous works, which integrated the prior information to estimate the unknown HR image. Tikhonov prior-based SR approach proposed using the L₂ norm, where it can easily eliminate noises from the image and blur the image boundaries [29]. Farsiu et al. proposed the bilateral total variation (BTV) prior to penalize gradient magnitudes, calculated according to the L₁ norm [10, 27, 30]. BTV is used to preserve edges well but artifacts are produced in the smoothed areas [17].

In this paper, we intend to enhance the efficiency of multiframe super-resolution image reconstruction in order to optimize both analysis and human interpretation processes by improving the pictorial information and enhancing the automatic machine perception. The regularized framework is used to provide additional information as well as an attempt to overcome the inverse problem that is not appropriate for the image reconstruction process. Therefore, the main contributions are as follows.

An efficient initialization approach is developed based on estimating the initial HR image through the preprocessing stage on the reference LR image. This approach uses linear and nonlinear filters to propose two initialization methods. The proposed initialization approaches are as follows:(i)Proposing a new method inspired by applying the median, mean, and Wiener filters at the reference LR image to remove the noise from the image, smooth the image by reducing the amount of intensity variation between neighboring pixels, and eliminate the blurring from the image, respectively.(ii)Developing another initialization approach based on the use of median, Lucy-Richardson, and Wiener filters at the reference LR image. It is significantly cleaner and more vivid due to the use of a median filter that is able to remove the noise from the image, Lucy-Richardson filter can smooth the image by reducing the amount of intensity variation between neighboring pixels, and Wiener filter is able to eliminate the blurring from the image.

The rest of this paper is organized as follows: Section 2 explains the SR problem formulation for image observation or degradation model and the Bayesian framework is shown in Section 3. The image SR reconstruction prior model is illustrated in detail in Section 4. The proposed multiframe SR approaches are presented and discussed in Section 5, while Section 6 demonstrates the experimental results. Lastly, the conclusion is outlined in Section 7.

2. Image Observation Model

A suitable image model, which can simulate the physical process of the image degradation for SR reconstruction, has to be established. There are some degrading influences in the practical sampling process including atmospheric instability, the motion of objects, blur effect, and downsampling equipment [12]. Consequently, the choice of a suitable degradation model is the primary key to reconstruct the SR image as shown in Figure 1 which is the core kernel of the image reconstruction process [31].

In this paper, let us assume that the procedure of image acquisition consists of warping, blurring, downsampling, and noise degradations. Initially, let us assume that X presents the HR image and Y_k is a set of LR images and M is its number. Then, F_k represents the movement matrix which is used to construct the movement degrading process of the k^th LR image, H_k is the blurring matrix which represents the point spread function (PSF) for the sensor of the camera, and D_k describes the matrix of downsampling that passes through both the movement and the blur matrices of the k^th image. Lastly, the additive system noise is indicated by n_k. Accordingly, the image degradation model which is used to simulate the whole process can be rewritten as in the following equation [8, 9, 11, 13]:

In general, we presume that the generated LR images are produced in similar situations, and the model in the following equation can be derived:where D and H describe the same motion and blur matrices, respectively. The image degradation model is demonstrated in Figure 1 [11, 32].

3. Bayesian Framework of Super Resolution

The main purpose of the Bayesian framework is to measure the probability of the uncertainty problem based on the available knowledge or information by integrating various priors into a mathematical model which is more powerful than statistical inference. Therefore, the SR technology for estimating the unknown HR image from a sequence of LR observed images is the uncertainty inference problem. In other words, the observed LR images are evidence of HR image deduction, and the regularization that represents the image prior is limited by the reconstructed solution of SR [17].

In the Bayesian framework, SR reconstruction is equivalent to the estimation of the HR image with given LR images. The maximum a posteriori (MAP) method is a popular one for the Bayesian framework because it integrates prior restrictions of the image and achieves results by optimizing the posterior probability cost function. Besides, it is known for its versatility in the preservation of edges and the estimation of joint parameters. In general, it is possible to use Bayesian estimation when there is a need to estimate the posteriori probability distribution for unknown parameters rather than using the specified parameters [33]. MAP estimator is extensively utilized as described in the following equation:where explains the conditional probability of the LR image (Y_k) by looking for the HR image (X) and P(X) indicates the prior probability of the HR image.

In equation (2), the image degrading model describes the process of producing LR images and also inversely estimates the corresponding unknown HR image. The fundamental structure of multiframe SR includes a data-fidelity term and also a regularization term. M estimator is implemented for the data-fidelity term to minimize the residual among the predicted HR image and the observed LR images. The regularization term is designed to restrict the minimization function to a stable state of the generated HR image [33]. Therefore, the classical multiframe SR framework can be described as follows:where the data-fidelity term is represented by ρ(·), R(·) describes the regularization term, and the regularization parameter is defined by λ.

4. Image Reconstruction Model

As discussed earlier, the regularization-based SR framework in spatial domain primarily includes data-fidelity and regularization terms [8]. This section explains the multiframe SR image reconstruction models that need to reproduce the HR image. A brief review of L_p norm is discussed as the data-fidelity term and also the BTV model is described as the regularization term. Farsiu et al. [30] proposed that the L_p error norm is used as the data-fidelity term, and the estimated HR image is solved when by

Farsiu et al. [30] proposed the suitable model called BTV, which integrates a bilateral filter and the total variation (TV) regularization. BTV regularization takes into account the use of a large number of neighbors to measure the gradient at a given pixel. In addition, BTV tends to preserve sharp edges in images with little artifacts as well as being computationally inexpensive and easily implemented. The main formula of BTV prior is defined as follows:where shifts X in the horizontal direction by l pixels, shifts X in the vertical direction by m pixels, and . Furthermore, α represents a scaling weight in the range of 0 < α <1 and the control parameter is P.

Due to the fact that SR is an ill-posed problem, an unlimited number of solutions are possible and the problem can be solved in unknown situations. In addition, the little amounts of noise in the measurements lead to large perturbations in the final solution which is an unstable solution. Consequently, the methods of using regularization in SR image reconstruction are a very useful way to reach a stable solution to eliminate artifacts from the final solution and enhance the convergence rate. There are a large number of regularization methods and we would like to use one of them to obtain the HR images with sharp edges and ease of implementation. The major benefit of using the regularization term in image reconstruction stage is to compensate for the lost data with some prior knowledge that is used as a penalty function for the minimization objective function [27]. The cost function of the BTV regularization term [30] in equation (6) is presented as follows:

In this paper, the steepest descent approach is applied to equation (7) to find a closed solution, where the steepest descent approach is used for fast convergence to the closely true HR image and estimating the HR image with low number of iterations. The measurement iteration using the steepest descent approach is shown below:where β is a scalar that regulates the step size in the gradient direction, λ is the regularization parameter, and and describe the transposing of matrices and , respectively.

5. Proposed Multiframe SR Approaches

As mentioned earlier, the HR image is measured by a series of LR images. Nevertheless, the inverse SR problem is really ill-posed in the case of noisy and blurring effects. In order to resolve this issue by the use of a Bayesian framework, we consider inserting prior information into the HR image X. Due to the fact that the HR image X is known to have a white Gaussian noise, the estimated vector Y_k is considered to be Gaussian. The process of generating the HR image through the Bayes rule means that the minimization problem is solved using the MAP framework in equation (5).

The traditional SR approaches use interpolation methods on the reference LR image such as bicubic or bilinear methods to estimate the initial HR image. However, the reference LR image has a poor quality leading to weakness in estimating the initial HR image. In the event that the initial HR image is properly estimated, the minimization function may go to a steady status very quickly. As a result, the final HR image can be recovered with a high quality by attempting to treat the reference LR image. The initial HR image has a significant impact on the reconstruction process which helps to reduce the impact of noise and artifacts, improve the quality of the output solution, and speed up the rate of convergence to converge towards the true estimate.

Numerous strategies of the multiframe SR image reconstruction have been proposed to improve the quality of the estimated HR image with various success degrees. The traditional regularization terms, including Tikhonov and TV, cannot differentiate edges from noises. Thus, despite the elimination of noise, the texture information is also suppressed, which reduces the effectiveness of these techniques. Therefore, BTV is the widely used prior function. The BTV approach is effective in retrieving sharp-edged images well because it includes the larger neighborhood in computing the gradient at a specific pixel, but it fails on images with smooth surfaces which suffer from the staircase effect. Also, BTV still suffers from noise and blur effect.

In this paper, we propose the new multiframe image SR approaches that firstly rely on estimating the initial HR image through preprocessing of the reference LR image with different filters. This preprocessing stage is used to overcome the degradation present in the reference LR image, which is a suitable kernel for producing the initial HR image to be used in the reconstruction phase of the final image. Then, the L₂ norm is employed for the data-fidelity term to minimize the residual among the predicted HR image and the observed LR images. Finally, the BTV prior model is utilized to restrict the minimization function to a stable state of the generated HR image.

The two proposed initialization methods are shown in Figures 2 and 3. The first proposed initialization method is inspired by applying the median, mean, and Wiener filters at the reference LR image to produce the composed LR image as shown in Figure 2. Meanwhile, the second proposed initialization method is based on the use of median, Lucy-Richardson, and Wiener filters at the reference LR image to generate the composed LR image as shown in Figure 3. Compared with these four filters, the median filter is used to remove the noise from the image. The mean filter is used to smooth the image by reducing the amount of intensity variation between neighbouring pixels. Wiener and Lucy-Richardson filters are used to eliminate the blurring from the image. In the end, regardless of whether the first or the second proposed method is selected to generate the composed LR image, the initial HR image is estimated through the bicubic interpolation method form the composed LR image. Then, the L₂ norm is employed for the data-fidelity term to minimize the residual among the predicted HR image and the observed LR images. Finally, BTV prior model is utilized to restrict the minimization function to a stable state of the generated HR image.

The main aim of the proposed approaches is to preserve the important image characteristics as much as possible including borders and corners, preserve the sharp image edges, and overcome artifacts. In addition, these proposed approaches increase the high-frequency components, eliminate degradations in the image capturing systems, and achieve a good balance for edge protection and noise reduction. The process flow of the proposed approaches comprises the following steps: Step (1): Generate a series of LR images from the original HR image by using the above observation model in equation (3), so that it is used as an input to the proposed approach. Step (2): Assign one of its LR images as the LR reference image to be used for initializing the HR image. Step (3): Estimate the motion parameters by calculating both of the rotated angle and the vertical and horizontal shift among the LR reference image and the next LR image to enhance the warp matrix F_k. Step (4): Apply the first or second proposed initialization method on the reference LR image in Figure 2 or Figure 3, respectively, to produce the composed image which is used as an input to the image interpolation stage. Step (5): Apply the bicubic interpolation on the composed image by using the magnification factor to measure the initial HR image (X₀) which is used as an input to the image reconstruction stage. Step (6): Input both of the initial estimation image (X₀) and the LR images into an SR image reconstruction model that depends on the L₂ norm and BTV prior model according to equation (8) beside using the motion parameters to reconstruct the final HR image (X_f).

The schematic diagram of the whole procedure flow for SR image reconstruction approach based on the BTV prior model is shown in Figure 4.

6. Experimental Results and Discussion

This section presents and discusses the actual performance evaluation of the approach proposed in comparison with several well-known state-of-the-art approaches in the literature. The performance analysis of the proposed approaches is evaluated by many experimental results. Section 6.1 defines the experimental environment and also describes the parameter settings. The performance analysis and criteria are illustrated in Section 6.2. In Section 6.3, the simulated experimental results are performed from four common SR approaches compared to our proposed SR approaches.

6.1. Experimental Setup

In this subsection, different simulated results are presented to estimate the efficiency of the proposed SR approaches. The proposed SR approaches are evaluated on a large benchmark database on the Internet [15] consisting of photographs of different scenes. We select eight images from a benchmark database of various styles to generate our simulated LR sequences as elaborated in Figure 5. Figures 5(a)–5(f) illustrate the original grayscale images for Acen, Cartap, Foreman, Text, and Brain with the size of 120 × 120 and License Plate image with the size of 250 × 100. Moreover, the original RGB images for Natural Scene and Parrot with the size of 120 × 120 are shown in Figures 5(g)–5(h). All experiments are performed directly on grayscale images and RGB channels for color images. These eight original HR images are very different from each other. Accordingly, the main reasons for choosing these images are their content, their availability for comparisons, and them being realistic, and these images have been frequently used by most recent works related to [3, 34–36]. Additionally, they are very different from each other in the following characteristics: complex structure, amount of irregular lines, texture features, gray level histogram, contrast, and smoothness.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

In practical terms, the performance of SR approaches cannot be measured arithmetically, because the HR images of the ground truth are still unavailable. In all experiments, the proposed approaches are initially examined on synthetic data, because the ground-truth images are available for synthetic LR images. Throughout all experiments, the same data and parameter settings are used to compare the proposed SR approaches to the four following state-of-the-art SR approaches:(1)An SR method with the Lorentzian error norm in both the data-fidelity and regularization terms is proposed by Patanavijit and Jitapunkul [37] (denoted as “LOR”).(2)The robust L₁ and L₂ norms with BTV prior are proposed by Farsiu et al. [30] (denoted as “L₁-BTV” and “L₂-BTV”).(3)The adaptive norm with BEP regularization is proposed by Zeng and Yang [27] (denoted as “BEP”).

Table 1 summarizes the parameter settings to guarantee a fair comparison of all experiments. The parameter settings are determined as follows:(1)20 LR images are produced from a single HR image and simulate a rigid motion of the displacement of each image. Therefore, uniform distribution of random values for translations and rotations displaces the HR image.(2)The random translation values vary between −2 and 2 pixels, while the values of the rotation angles shift randomly from −0.5° to 0.5°.(3)The displaced HR images are blurred with the width of isotropic Gaussian PSF σ = 0.4 and subsampled with magnification factor r = 2.(4)In the simulated LR sequence, an additive white Gaussian noise is provided. The standard deviation of an additive white Gaussian noise is assigned to 0.025.(5)The maximum number of iterations is 50 for Lorentzian, BTV, and BEP priors.(6)The regularization parameter λ for Lorentzian, BTV, and BEP priors is chosen as 0.0025.(7)Setup image prior for the Lorentzian prior: the window size is set to h = 1.8 and the scale factor to α = 1.(8)Setup image prior for the BTV prior: the window size is set to and the scale factor to α = 0.6.(9)Setup image prior for the BEP prior: the window size is set to q = 1, the scale factor to α = 1, and the threshold parameter to c = 0.2.

All applied experiments are coded in MATLAB R2020a and run on a computer system with Intel(R) Core (TM) i7-3612QM CPU @ 2.10 GHz, 8 GB memory, 512 MB Radeon graphics card, 1 TB hard disk drive, and 64-bit Windows 7 Home Premium operating system. Table 2 summarizes the system setup requirements to implement the comprehensive experiment.

6.2. Performance Analysis

To perform the processes of quantitative analysis, evaluation, and comparison for the reconstruction performance, we generate synthetic data and calculate the peak-signal-to-noise ratio (PSNR) indicator and the structural similarity (SSIM) indicator of our estimated HR image with regard to a ground-truth image [8, 26, 31, 38, 39]. Therefore, the PSNR measurement is calculated from the mean square error (MSE), where the MSE reflects the average error among the reconstructed SR image and the original HR image. By giving a reconstructed SR m × n image and its original HR image , MSE and PSNR are described as

The SSIM indicator is used to assess the symmetry among the recreated SR image and the original HR image. Luminance, contrast, and structural differences between two images are also considered in the SSIM indicator. The SSIM indicator is represented aswhere and are the means. and are the standard deviations of the original HR images and the reconstructed SR images, respectively. is the covariance of and . C₁ and C₂ present the fixed values to guarantee a stable SSIM computation. The constants C₁ = (k₁L)² and C₂ = (k₂L)², where k₁ and k₂ have tiny-fixed values, while L is the pixel dynamic set (255 for 8-bit grayscale images). In particular, a greater value of PSNR shows a good image quality for the restored image. The SSIM value falls within the range (0, 1) so that the similarity is large when its value is closer to 1. Then, the effect of image reconstruction is better.

6.3. Experimental Results on Synthetic Data

This research paper uses two types of evaluation criteria, namely, subjective (qualitative or visual analysis) and objective (quantitative analysis) evaluations. On one hand, subjective evaluation is considered the most intuitive and clear approach. Therefore, this approach needs from people to know a few essential image features in advance before starting to analyze the reconstructed HR images. Consequently, the reconstruction results should be evaluated by the human eye to identify the best reconstructed HR images. On the other hand, objective evaluation is considered an approach intended to evaluate the results by calculating the correlation coefficient between the images. Normally, PSNR and SSIM criteria are used to evaluate the effect of SR image reconstruction. In general, if these two values are large, the reconstruction effect becomes better.

This subsection presents and discusses the experimental results of the proposed SR approaches on the simulated data to show the SR reconstruction improvement result effectively. The first proposed method is denoted as MMW-BTV, while the second proposed method is denoted as MLRW-BTV. To guarantee a fair comparison, the image formation model is typically executed in equation (2) for all algorithms. After that, we employ the four selected multiframe SR approaches and the proposed SR approaches to rebuild the SR image from the degraded LR images as shown in Figures 6–13.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Comparisons are provided to assess the performance of the proposed SR approaches through both visual and quantitative comparisons. Visual quality evaluations of the restored SR images for “Acen,” “Cartap,” “Foreman,” “Text,” “Brain,” “License Plate,” “Natural Scene,” and “Parrot” from Figure 5 using the proposed SR approaches are illustrated in Figures 6–13, respectively, in comparison with several current state-of-the-art approaches. Figures 6–13 display the reconstruction results generated for each experiment. The explanations of these figures are described as follows: (a) shows the original HR image, (b) displays a single measured LR image, and ((c)–(f)) demonstrate the restored HR images with LOR, L₁-BTV, BEP, and L₂-BTV approaches, respectively. Here, (g) and (h) indicate the SR image of the two proposed approaches. Also, parts of these reconstructed images are enlarged for a perfect visualization by displaying them in the downmost right corner of every image.

6.4. Qualitative and Visual Analysis

The visual experiments in Figures 6–13 represent the reconstruction result for HR images. A set of conclusions can be drawn based on Figures 6–13. Firstly, LOR and L₁-BTV are inadequate to eliminate the noise, remove the blurring, and have a zigzagging effect in the reconstructed images as shown in Figures 6–13(c) and 13(d). For example, in the Foreman image as shown in Figures 8(c) and 8(d), a lot of noise remains in the background area as well as a lot of zigzagging at the edges. Secondly, L₂-BTV is robust to suppress the white Gaussian noise and preserve the image edges as shown in Figures 6–13(f). This method achieves a result that is the most visible and close to the image quality of the reconstructed HR image in the proposed MMW-BTV approach. For example, in the Foreman image, as shown in Figure 8(f), the image is clear and has better reconstruction; however, there is a little staircase effect at the edges. Finally, the BEP method can adaptively choose error norm to suppress the Gaussian noise but some noise still exists due to its simple initial estimation as shown in Figures 6–13(e). For example, in the Foreman image as shown in Figure 8(e), some noises remain in the background area as well as some zigzagging at the edges.

As a result, the proposed MMW-BTV and MLRW-BTV approaches obviously produce the restored HR image, which offers the best visual effects, as opposed to other SR methods (LOR, L₁-BTV, BEP, and L₂-BTV) and they better prevent the staircase effect. In comparison, image information such as sharp edges is retained, while the noise is suppressed. This improvement is due to the proposed MMW-BTV and MLRW-BTV approaches relying on estimating the initial HR image through preprocessing of the reference LR image based on the median, mean, Lucy-Richardson, and Wiener filters, because the reference LR image has a poor quality and contains some distortions such as blur and noise which lead to weakness in estimating the initial HR image. This preprocessing stage is used to overcome the degradation present in the reference LR image, which is a suitable kernel for producing the initial HR image to be used in the reconstruction phase of the final image. Moreover, median, mean, Wiener, and Lucy-Richardson filters are able to remove the noise from the image, smooth the image by reducing the amount of intensity variation between neighboring pixels, eliminate the blurring from the image, and smooth the image by reducing the amount of intensity variation between neighboring pixels, respectively. Therefore, the proposed MMW-BTV and MLRW-BTV are, finally, considerably clearer, more realistic, and closer to the image of the ground reality.

6.5. Quantitative Analysis

Objective quality measurement is an alternative of a tedious and time-consuming subjective assessment. In literature, there are a plenty of metrics that model or not the human visual system (HVS). Most of them are not very popular due to their complexity, difficult calibration، or lack of freely available implementation. This is why metrics like PSNR and SSIM are widely used to compare algorithms [3, 10, 11, 13, 14, 28, 32, 39–42].

A computational analysis is required to support the visual appearance. For this reason, the quantitative quality estimation of the generated HR images is performed by the PSNR indicator and the SSIM criterion. The PSNR index is used to assess the generated HR image quality, while the SSIM index is a supplementary measurement that indicates that the image quality is dependent on the recognized features of the HVS. Tables 3 and 4 show the computed values of the PSNR and SSIM metrics, respectively, for the eight examined images in Figure 5. PSNR and SSIM have the best value in bold numbers in each row.

From quantitative quality estimation, Table 3 presents the proposed approaches with the best PSNR values that confirm their effectiveness against blurring and noise reduction. Furthermore, the values of SSIM in Table 4 demonstrate the high image quality in terms of image preservation information. Based on the data in the above tables (Tables 3 and 4), the following conclusions can be drawn:(1)In terms of PSNR and SSIM, the proposed MMW-BTV and MLRW-BTV approaches are better than other SR methods for image reconstruction. They allow the reconstructed images to be more similar to the original images with good deblurring and denoising performance.(2)In comparison to the BEP method, SSIM measures for the proposed MMW-BTV approach are higher by 0.03814–0.15671. For example, the SSIM value of the proposed MMW-BTV approach (0.96593) is 0.15671 higher than that of the BEP method (0.80922) for the image of Text.(3)The PSNR and SSIM values of the proposed MLRW-BTV approach are higher than those of the BEP method by 1.4073–4.8426 dB and 0.03809–0.15608, respectively. For example, PSNR of the proposed MLRW-BTV approach (30.6008 dB) is 1.4073 dB higher than that of the BEP method (29.1935 dB) for the image of Brain, and the SSIM difference becomes 0.08097 for the image of Foreman (0.9365 versus 0.85553).(4)The proposed MMW-BTV approach has SSIM values that are 0.00002–0.00276 higher than the L₂-BTV method. For the Foreman image, SSIM of the proposed MMW-BTV approach (0.93735) is 0.0016 higher than that of the L₂-BTV method (0.93580).(5)The proposed MLRW-BTV approach has PSNR and SSIM values that are 0.0159–0.2503 dB and 0.0001–0.0023, respectively, higher than those of the L₂-BTV method. For the Text image, PSNR of the proposed MLRW-BTV approach (28.384 dB) is 0.0159 dB higher than that of the L₂-BTV method (28.3681 dB). Meanwhile, the SSIM difference becomes 0.0021 (0.9653 versus 0.96317).

As a result, the proposed MMW-BTV and MLRW-BTV approaches always have better results as compared to other methods (LOR, L₁-BTV, BEP, and L₂-BTV) which guarantee their efficiency, because they estimate the initial HR image well through preprocessing of the reference LR image in order to remove the noise, smooth the image, and eliminate the blurring.

6.6. Computational Complexity Analysis

In this subsection, we conduct a detailed evaluation of the computational complexity of our proposed SR approaches (MMW-BTV and MLRW-BTV) and the compared SR approaches (LOR, L₁-BTV, BEP, and L₂-BTV) in terms of run-time. The run-time evaluation is performed using the parameter settings and system setup requirements previously explained in Tables 2 and 3, respectively. The run-time complexity of our proposed SR approaches and the compared SR approaches is shown in Table 5. The proposed SR approaches are less computationally complex in comparison with the state-of-the-art SR approaches such as L₁-BTV and BEP. The BEP approach has the highest computational complexity, while the LOR and L₂-BTV approaches have lower computational complexity than our proposed SR approaches. However, the quality of our super-resolved images is much better than theirs in terms of both visual evaluation and PSNR/SSIM measurements as shown in Tables 3 and 4.

7. Conclusion

In this paper, we propose new multiframe image SR approaches that firstly rely on estimating the initial HR image through preprocessing of the reference LR image based on median, mean, Lucy-Richardson, and Wiener filters. Then, the L₂ norm is employed for the data-fidelity term and BTV prior model is utilized to the regularization term. The proposed multiframe SR approaches are used for increasing the components of high frequency and preventing image degradation produced by the imaging systems. Also, they ensure a good balance among the preservation of edge and the suppression of noise and adaptively enhance the SR reconstruction process. The efficiency of the proposed SR approaches is measured for synthetic images and they are comparable to several related state-of-the-art approaches. Extensive experiments have been conducted and the experimental results illustrate that our proposed approaches increase the visual efficiency of state-of-the-art algorithms. Furthermore, we illustrate that the proposed approaches always generate the best PSNR and SSIM values quantitatively. However, the high computational cost of the proposed approaches remains a challenging point.

Data Availability

The data are available any time if needed from the author Mahmoud M. Khattab (e-mail: [email protected]).

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

The authors extend their appreciation to the Deanship of Scientific Research at King Khalid University for funding this work through General Research Project (Grant no. GRP/332/42).

References

L. Yue, H. Shen, J. Li, Q. Yuan, H. Zhang, and L. Zhang, “Image super-resolution: the techniques, applications, and future,” Signal Processing, vol. 128, pp. 389–408, 2016.
View at: Publisher Site | Google Scholar
H. Seibel, S. Goldenstein, and A. Rocha, “Eyes on the target: super-resolution and license-plate recognition in low-quality surveillance videos,” IEEE Access, vol. 5, pp. 20020–20035, 2017.
View at: Publisher Site | Google Scholar
T. Köhler, “Multi-frame super-resolution reconstruction with applications to medical imaging,” 2018, https://arxiv.org/abs/1812.09375.
View at: Google Scholar
J. Satiro, K. Nasrollahi, P. L. Correia, and T. B. Moeslund, “Super-resolution of facial images in forensics scenarios,” in Proceedings of the 2015 International Conference on Image Processing Theory, Tools and Applications (IPTA), Orleans, France, November 2015.
View at: Google Scholar
W. Ma, Z. Pan, J. Guo, and B. Lei, “Super-resolution of remote sensing images based on transferred generative adversarial network,” in Proceedings of the IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, July 2018.
View at: Publisher Site | Google Scholar
S. C. Sung Cheol Park, M. K. Min Kyu Park, and M. G. Moon Gi Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21–36, 2003.
View at: Publisher Site | Google Scholar
H. Hou, M. Wang, and X. Wang, “RL-MS-L filter function for CT image reconstruction,” TELKOMNIKA (Telecommunication Computing Electronics and Control), vol. 14, no. 1, pp. 195–202, 2016.
View at: Publisher Site | Google Scholar
S. C. Mohan, “Adaptive super-resolution image reconstruction with lorentzian error norm,” Indian Journal of Science and Technology, vol. 10, no. 16, 2017.
View at: Publisher Site | Google Scholar
S. Huang, J. Sun, Y. Yang, Y. Fang, P. Lin, and Y. Que, “Robust single-image super-resolution based on adaptive edge-preserving smoothing regularization,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2650–2663, 2018.
View at: Publisher Site | Google Scholar
X. Liu, L. Chen, W. Wang, and J. Zhao, “Robust multi-frame super-resolution based on spatially weighted half-quadratic estimation and adaptive BTV regularization,” IEEE Transactions on Image Processing, vol. 27, no. 10, pp. 4971–4986, 2018.
View at: Publisher Site | Google Scholar
A. Laghrib, A. Hadri, A. Hakim, and S. Raghay, “A new multiframe super-resolution based on nonlinear registration and a spatially weighted regularization,” Information Sciences, vol. 493, pp. 34–56, 2019.
View at: Publisher Site | Google Scholar
L. Wang, Z. Lin, X. Deng, and W. An, “Multi-frame image super-resolution with fast upscaling technique,” 2017b, https://arxiv.org/abs/1706.06266.
View at: Google Scholar
A. Laghrib, A. Ben-Loghfyry, A. Hadri, and A. Hakim, “A nonconvex fractional order variational model for multi-frame image super-resolution,” Signal Processing: Image Communication, vol. 67, pp. 1–11, 2018.
View at: Publisher Site | Google Scholar
M. Hakim, A. Ghazdali, and A. Laghrib, “A multi-frame super-resolution based on new variational data fidelity term,” Applied Mathematical Modelling, vol. 87, pp. 446–467, 2020.
View at: Publisher Site | Google Scholar
K. Nasrollahi and T. B. Moeslund, “Super-resolution: a comprehensive survey,” Machine Vision and Applications, vol. 25, no. 6, pp. 1423–1468, 2014.
View at: Publisher Site | Google Scholar
R. Tsai and T. S. Huang, “Multiframe image restoration and registration,” Advances in computer vision and Image Processing, vol. 1, no. 2, pp. 317–339, 1984.
View at: Google Scholar
S. Zhao, R. Jin, X. Xu, E. Song, and C.-C. Hung, “A variational Bayesian superresolution approach using adaptive image prior model,” Mathematical Problems in Engineering, vol. 2015, Article ID 469859, 2015.
View at: Publisher Site | Google Scholar
S. Zhao, H. Liang, and M. Sarem, “A generalized detail-preserving super-resolution method,” Signal Processing, vol. 120, pp. 156–173, 2016.
View at: Publisher Site | Google Scholar
Q. Yuan, L. Zhang, and H. Shen, “Multiframe super-resolution employing a spatially weighted total variation model,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 3, pp. 379–392, 2012.
View at: Publisher Site | Google Scholar
H. Zhang, L. Zhang, and H. Shen, “A super-resolution reconstruction algorithm for hyperspectral images,” Signal Processing, vol. 92, no. 9, pp. 2082–2096, 2012.
View at: Publisher Site | Google Scholar
D. Kim and H. Byun, “Regularization based super-resolution image processing algorithm using edge-adaptive non-local means filter,” in Proceedings of the 7th International Conference on Ubiquitous Information Management and Communication, Kota Kinabalu, Malaysia, January 2013.
View at: Publisher Site | Google Scholar
Z. Ren, C. He, and Q. Zhang, “Fractional order total variation regularization for image super-resolution,” Signal Processing, vol. 93, no. 9, pp. 2408–2421, 2013.
View at: Publisher Site | Google Scholar
W.-Z. Shao, H.-S. Deng, and Z.-H. Wei, “A posterior mean approach for MRF-based spatially adaptive multi-frame image super-resolution,” Signal, Image and Video Processing, vol. 9, no. 2, pp. 437–449, 2013.
View at: Publisher Site | Google Scholar
R. M. Bahy, G. I. Salama, and T. A. Mahmoud, “Adaptive regularization-based super resolution reconstruction technique for multi-focus low-resolution images,” Signal Processing, vol. 103, pp. 155–167, 2014.
View at: Publisher Site | Google Scholar
B. Jacob Maiseli, N. Ally, and H. Gao, “A multi-frame super-resolution method based on the variable-exponent nonlinear diffusion regularizer,” Signal Processing: Image Communication, vol. 34, no. 1, pp. 1–13, 2015.
View at: Publisher Site | Google Scholar
T. Köhler, X. Huang, F. Schebesch, A. Aichert, A. Maier, and J. Hornegger, “Robust multiframe super-resolution employing iteratively re-weighted minimization,” IEEE Transactions on Computational Imaging, vol. 2, no. 1, pp. 42–58, 2016.
View at: Publisher Site | Google Scholar
X. Zeng and L. Yang, “A robust multiframe super-resolution algorithm based on half-quadratic estimation with modified BTV regularization,” Digital Signal Processing, vol. 23, no. 1, pp. 98–109, 2013.
View at: Publisher Site | Google Scholar
V. K. Ghassab and N. Bouguila, “Light field super-resolution using edge-preserved graph-based regularization,” IEEE Transactions on Multimedia, vol. 22, no. 6, pp. 1447–1457, 2019.
View at: Google Scholar
X. Zhang, E. Y. Lam, E. X. Wu, and K. K. Wong, “Application of Tikhonov regularization to super-resolution reconstruction of brain MRI images,” in Proceedings of the International Conference on Medical Imaging and Informatics, Beijing, China, August 2007.
View at: Google Scholar
S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Transactions on Image Processing, vol. 13, no. 10, pp. 1327–1344, 2004.
View at: Publisher Site | Google Scholar
M. M. Khattab, A. M. Zeki, A. A. Alwan, and A. S. Badawy, “Regularization-based multi-frame super-resolution: a systematic review,” Journal of King Saud University-Computer and Information Sciences, vol. 32, 2018.
View at: Google Scholar
L. Wang, Z. Lin, X. Deng, and W. An, “Fast multi-frame image super-resolution based on MRF,” 2017a, https://arxiv.org/abs/1706.06266.
View at: Google Scholar
Z. Wang, H. Yang, W. Li, and Z. Yin, “Super-resolving IC images with an edge-preserving Bayesian framework,” IEEE Transactions on Semiconductor Manufacturing, vol. 27, no. 1, pp. 118–130, 2014.
View at: Publisher Site | Google Scholar
M. Kumar and M. Diwakar, “A new exponentially directional weighted function based CT image denoising using total variation,” Journal of King Saud University-Computer and Information Sciences, vol. 31, 2016.
View at: Google Scholar
R. Nayak and D. Patra, “Super resolution image reconstruction using penalized-spline and phase congruency,” Computers & Electrical Engineering, vol. 62, 2016.
View at: Google Scholar
I. E. Mourabit, M. E. Rhabi, A. Hakim, A. Laghrib, and E. Moreau, “A new denoising model for multi-frame super-resolution image reconstruction,” Signal Processing, vol. 132, pp. 51–65, 2017.
View at: Publisher Site | Google Scholar
V. Patanavijit and S. Jitapunkul, “A Lorentzian stochastic estimation for a robust iterative multiframe super-resolution reconstruction with Lorentzian-Tikhonov regularization,” EURASIP Journal on Applied Signal Processing, vol. 2007, Article ID 034821, 21 pages, 2007.
View at: Publisher Site | Google Scholar
A. Laghrib, A. Ghazdali, A. Hakim, and S. Raghay, “A multi-frame super-resolution using diffusion registration and a nonlocal variational image restoration,” Computers & Mathematics with Applications, vol. 72, no. 9, pp. 2535–2548, 2016.
View at: Publisher Site | Google Scholar
W. Long, Y. Lu, L. Shen, and Y. Xu, “High-resolution image reconstruction: an envℓ1/tv model and a fixed-point proximity algorithm,” International Journal of Numerical Analysis & Modeling, vol. 14, no. 2, 2017.
View at: Google Scholar
S. Huang, J. Sun, Y. Yang, Y. Fang, and P. Lin, “Multi-frame super-resolution reconstruction based on gradient vector flow hybrid field,” IEEE Access, vol. 5, pp. 21669–21683, 2017.
View at: Publisher Site | Google Scholar
M. Rossi and P. Frossard, “Geometry-consistent light field super-resolution via graph-based regularization,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4207–4218, 2018.
View at: Publisher Site | Google Scholar
J. Kim, J. Han, and M. G. Kang, “Multi-frame depth super-resolution for ToF sensor with total variation regularized L1 function,” IEEE Access, vol. 8, pp. 165810–165826, 2020.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2021 Mahmoud M. Khattab et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

740

Downloads

670

Citations