Abstract
With the proliferation of portable digital products, image quality degradation has received a lot of attention. As the most common phenomenon in image degradation, the issue of image deblurring is the focus of much attention. Blind motion blur removal is the main target of this paper. The heavy-tailed distribution is the most dominant statistical feature of natural images. However, most image deblurring methods use a gradient prior with fixed parameters to recover a clear image, which leads to loss of details in the recovered clear image and does not consider the higher order prior of the natural image. Therefore, this paper proposes a new regularized image recovery model based on the Gaussian-scale mixture expert field(GSM-FOE) model. First, the GSM-FoE model learns filters and corresponding parameters with higher order prior information of images by training images in a natural image library; second, these learning results are used to guide the image recovery process. The GSM-FoE model and gradient-fidelity based image recovery model is proposed, which can be used with an iterative re-weighted least squares (IRLS) method. Experiments demonstrate that the suggested recovery approach is simple to use and successful at reducing blur and noise, as well as suppressing ringing effects while preserving image information. Moreover, the image restoration method performs well for large blurring kernels. The results fully reflect the effectiveness and robustness of the proposed method for complex noise scenarios. The quality of the generated images is significantly better than that of several classical methods.
1. Introduction
Images are the primary form in which humans acquire, express, and communicate visual information. Motion blur is commonly used to portray the relative motion between the target object and the camera using a blur kernel. The goal of deblurring is to recover a clear-edged image from the observed blurred image for subsequent use in intelligent applications. Therefore, the problem of motion blur image restoration is of great theoretical and practical importance.
Algorithms on image deblurring have also evolved in the image field. Qin et al. proposed to remove motion blur based on the feature information (transparency information) of the image content itself [1]. Liu et al. proposed a deep learning approach to estimate the probability distribution of motion blur blocks using the CNN and Markov random field model and then use the information based on the image blocks to solve the global inconsistent motion blur problem, but the blind deblurring problem of a single image increases the difficulty [2]. Abdelrahim estimated the blur kernel by extracting the salient edges of the blurred image, but there is a large amount of noise and ringing in the recovered image [3] Sun et al. proposed a constraint based on the Laplace prior that can better preserve the edge and detail information of the recovered image [4]. Sun et al. introduces a new model to guide the image restoration process, namely, the use of continuous segmented function stitching to approximate the gradient distribution [5]. To better characterize the sparse nature of image gradients, Kja et al. proposed a constraint based on a super-Laplacian prior that makes the recovered image more consistent with natural scene properties, but the method cannot adaptively adjust the strength of the penalty for different regions in the image. [6]. For the image noncoherent motion blur problem, Wang proposed a deep learning method of convolutional neural networks that can estimate and remove noncoherent motion blur more effectively [7]. Yang et al. used the TV blind convolution method, applying a sparse gradient prior as a constraint to solve the blind convolution problem [8]. Based on the edge information of the image, Xu et al. proposed a blind deconvolution method for MAP estimation model of the image [9]. In order to solve the problem of noncoherent motion blur, Jin et al. proposed the method of estimating blur kernel using a learning convolutional neural network (CNN) [10]. In recent years, some scholars have broken away from the original research idea and proposed the concept of learning-based image restoration to replace the smoothing constraint term based on regularization methods [11–15]. The basic idea of this class of methods is to obtain a priori knowledge of natural images through learning algorithms. For image prior terms, Huang et al. proposes a Gaussian scale mixture learning method combined with the Bayesian minimum mean squared error estimation to train the model [16]. In low-level computer vision, Nazarinezhad et al. proposed to learn prior information of natural images with the higher-order Markov random fields (MRF) [17].
Digital image processing techniques are increasingly used in high-end fields, and the study of image deblurring is the key factor to promote its development. Considering the fact that common algorithms are still prone to multipeaks, this paper proposed the GSM-FOE model, which represents the spatial structure information of images, to mine the higher-order prior knowledge of natural images, and learns eight 3 × 3 filters that contain the higher-order prior knowledge of natural images. In the image restoration process, the gradient information of the image is also introduced into the image prior term in this paper. Experimental results and comprehensive comparison analysis demonstrate its superiority.
Concretely, our contributions are four-fold as follows:(i)This paper argues that although the gradient distributions of natural images all obey a heavy-tailed distribution, it is not appropriate to take a function to approximate this distribution directly, which would increase the error in the image recovery step. This will increase the error in the image recovery step.(ii)This paper argues that it is not sufficient to consider only the first-order a priori information of natural images. Based on a profound learning of the GSM-FoE model, this paper uses the GSM-FoE model to learn higher-order prior knowledge of natural images, and the results of these learned filters acting on the images reflect their intrinsic feature information.(iii)The deblurring algorithm, which combines the learning results with the gradient fidelity term, is used to maintain the image details and edge information well and suppress the ringing effect.(iv)For the image restoration model in this paper, we give an effective solution that works well for large images or large blur kernels.
2. GSM-FoE Model Offline Natural Image a Priori Learning Method
2.1. FoE Method
In a regularization-based approach, the Markov field (MRF) model can be used to model the spatial structure information of an image using potential functions to form a priori constraints. However, it is limited in that it can only use a simple neighborhood structure (each pixel is only related to its four nearest neighboring pixels), whereas the FoE (Field of Expert) model, which represents the spatial structure information of an image, can better address this limitation and can learn higher-order prior knowledge from the training image library. Figure 1 shows the flow process of the FoE model.
An a priori model based on image spatial information is introduced into the objective function of image deblurring. In other words, the FoE model is incorporated into the image restoration model. A neighborhood system is defined for a (m is generally odd) square region such that it connects all nodes within the region. There are systems of neighborhoods that may overlap with each other throughout the image . Each neighborhood center pixel () then has an extremely large group . The potential function of the group is denoted by , . Under the FoE model, the probability density function of the image is as follows:
Among them, denotes the number of expert functions, is the expert function to be defined, is the filter to be learned, denotes the set of parameters to be learned, and the filter is the normalized parameter. The number of parameters in the model depends only on the size of the group and the number of filters, and there is no requirement for the size of the learned image . In practice, the model is often transformed into the form of a Gibbs distribution for convenience.
2.2. Selection of Canonical Terms and Expert Functions under the GSM-FoE Model
The literature [18] mentions, respectively, the use of TV regularization and parametric regularization methods to fit the heavy-tailed distribution of image gradients. In addition to TV regularization and parametric regularization methods, the concept of learning-based image restoration has been proposed by some scholars in recent years. In addition to first-order derivative information, the intrinsic features of an image include higher-order prior information, and learning methods that train the prior can learn these higher-order priors from a given image database and are therefore more accurate than methods that approximate the prior.
If the image in the training image library is transformed into the image in the image restoration process, then the regular term describing the higher order prior of the image under the FoE model can be expressed as follows:
Among them, denotes the number of expert functions, is the expert function to be defined, is the filter to be learned, denotes the set of parameters to be learned, and the filter is the normalized parameter.
Under the MAP model, the selected expert function needs to satisfy the condition that it is guaranteed to be logarithmically continuous and differentiable. There are three classical expert functions: one is based on the student-t distribution, the second is Charbonnier’s light and heavy-tailed expert function, and the third is the Gaussian scale mixture expert function. In spite of losing detail, the Student-t expert function’s logarithmic distribution is more consistent with the heavy-tailed distribution than the Charbonnier expert function. The Gaussian scale mixture expert function [19] not only retains more detail information in the image but can also eliminate noise better, so the GSM expert function is used in this paper.
Although the gradient distributions of natural images all obey a neutral distribution, it is not reasonable to take a function to approximate this distribution directly, which would increase the error in the image recovery step. Therefore, in this paper, the expert function represented by the Gaussian scale mixtures with zero mean is chosen.
Here, and are the parameters of . Filter with higher order information about the image and the parameters can be learned from the Berkeley image library using the EM (Maximum Expectation) algorithm [20] with the following training procedure:
Algorithm 1 describes the basis rotation algorithm.
|
Figure 2 shows the logarithmic distribution of the GSM expert function. The logarithmic distribution shows that the GSM expert function has thicker tails on both sides and a small spike in the middle, so the log-weighted tail distribution is more approximate. The advantage of the Gaussian scale hybrid expert function is that it includes a variety of represented expert functions, including Student-t experts and Charbonnier experts, and the scales and parameters of the GSM experts are adjustable, so the GSM experts are more flexible and diverse. It is not sufficient to consider only the first-order priori information of natural images. Based on a deep learning of the GSM-FoE model, we learn the higher-order prior knowledge of the natural images. 15 scales are selected in the training process, and eight 3 × 3 filters are learned. The result of these filters acting on the image reflects its intrinsic feature information. From the distribution of the weights in Figure 2, it can be seen that the yellow, red, and orange curves are more consistent with the heavy-tailed distribution at the −3, −4 scale, which means that the weight distribution at the −3, −4 scale fits the higher-order prior of the image better and can better reflect the intrinsic characteristics of the image itself. Based on the above analysis, the GSM expert function is selected in this paper for the image restoration process.
3. Image Deblurring Algorithm Based on the GSM-FoE Model
The quality of image recovery can be improved by using a priori constraints on image spatial structure information in regularized image recovery methods. The advantages of the FOE model in representing spatial structure information have attracted increasing attention [21–23].
By taking into account the higher-order prior of natural images, this paper introduces an a priori model based on image spatial information in the objective function of declaring and applies the learning results under the model of GSM FoE to guide image restoration. In the image restoration process, considering that the image gradient term can better suppress noise, this paper incorporates the gradient fidelity term into the image deblurring model as well, improves the traditional regularization term, proposes a regularization method based on GSM FoE and gradient fidelity, applies it to the single image blind deblurring problem, and gives an effective algorithm based on IRLS.
3.1. Regularization Methods Based on the GSM-FoE Model
The energy function describing the higher order prior of the image under the GSM FoE model can be expressed from the reasoning in the literature [24] as follows:
Of these, , represent scales, 15 scales were selected during the course of the training. denotes the filter learned by training, and is the parameter learned.
From the distribution about the weights of the GSM experts at different scales, it is known that the weights at the −3, −4 scale are more in line with the heavy-tailed distribution, so the −3, −4 scale is chosen in this section, which can better fit the higher-order prior of the image. This soft fit can reduce the error in fitting the energy function and improve the accuracy of the image prior when the restriction is changed from to . Based on the above analysis, the energy function used in this paper is as follows:
A conventional image restoration model based on regularization takes the following concrete form:
The second of these is noise suppression through image gradient information. More clear images with intrinsic priori information are obtained using an image prior term approach based on an expert field model. Within the framework of the MAP model, this paper incorporates the higher-order prior knowledge of natural images learned from the GSM-FOE model into the image prior term, while suppressing noise by introducing image gradients information. The optimization model is as follows:
To solve model (9), we use an algorithm based on the iterative reweighted least squares (IRLS) method and the conjugate gradient method [20] for the iterative solution. The weights updated for each iteration are as follows:where denotes the position of the pixel in the image. The advantage of the new improved model is that this soft fit reduces the error of the energy function and thus improves the accuracy of the image prior.
Before giving the algorithms in this paper, we illustrate some notation as follows:(1)The symbol is used to denote a Toeplitz matrix, i.e., the matrix indistinct is first pulled into a row vector form, denotes the matrix formed by the elements of this row cycling backwards each time and then arranging them in rows.(2)The convolution of the image is expressed in a matrix form: .To ensure that the matrix is multipliable, the rest of the values of the elements of each row in , except for the elements of the fuzzy kernel k, are complemented by 0.(3) denotes the learned filter.
Algorithm 2 describes the estimation process for clear images:
|
3.2. Process of Blind Image Deblurring
The recovery of blurred images in this paper is divided into three parts: offline GMS higher-order prior learning training, blur kernel estimation, and clear image recovery, as shown in flowchart Figure 3.
Assuming that motion blur is spatially globally consistent, the recovery model in this paper estimates the blur kernel and the clear image by iterating the following two equations alternatively.
Among them, is the unknown fuzzy kernel and is the 2D convolution operator. Under the assumption of Gaussian noise, the fidelity term is generally denoted as . is an image prior term based on the GSM-FoE model, which guides the recovery of clear images by mining the higher-order prior information of natural images. is based on image gradient information to suppress noise. is the fuzzy kernel prior term. The weights and are the parameters of the kernel prior and the image prior, respectively, are the parameters of the image gradient and are the parameters of the discrete metric .
The fuzzy image is first converted into a gray-scale image; in the estimation stage of the fuzzy kernel, given the input iterative image , this paper uses a constraint-based approach to extract the significant structure; in order to retain more structural information, the strong edges are restored with impact filtering; finally, the fuzzy kernel is estimated.
In the image recovery phase, the offline GMS-FoE model is used to train the Berkeley image database so as to learn higher-order prior knowledge of natural images, which is used to guide the recovery of clear images; the GSM-FoE prior model and the gradient-fidelity regularization term are used as model constraints, and an optimization algorithm is proposed.
The detailed parts of blurred images tend to cause inaccuracy in the estimation of the blur kernel. The double-sided filter allows more low frequency information to be retained. The specific algorithm is as follows:
Among them, x and y are the coordinates of the pixel points in the image, W is the image pixel space, is the normalized term, is the image to be filtered, is the image after bilateral filtering, and both f and denote domain filters.
Considering the problem of noise in the image, this paper performs a bilateral filtering algorithm on of Algorithm 2 to obtain the image . Calculate the image detail layer as . Finally, a clear image with more detailed information is obtained.
In recent years, many papers have considered sparsity constraints in the kernel estimation process, but they tend to lead to non-convexity problems. To avoid this problem and to take into account the continuity of the kernel, the kernel estimation model [24] used in this paper is as follows:
Among them, . Due to the use of discrete metrics , the solution is to alternate iterations of the following two processes. and denote adjustable parameters to balance the strength of the fidelity and kernel similarity terms. The first process is a convex function on the fuzzy kernel k. The second process is the gradient minimization problem.
Once the fuzzy kernel has been estimated, the image can be recovered using the fuzzy image and the estimated fuzzy kernel. In order to make full use of the learned higher order prior knowledge, while avoiding noise in the image recovery process, a GSM FoE prior model and gradient fidelity regular terms are used and the recovery model is as follows: indicates the filter that has been learned and the parameters.
Value . The relative energies ratio of the regularization methods based on the GSM-FOE model is . Experiments show that the property that a regular term based on GSM-FOE model and gradient fidelity has an image energy greater than that of a clear image, making the model converge more towards the clear image.
4. Numerical Experiments and Analysis
In this paper, a coarse to fine multiscale approach is used to recover a clear image. The total number of layers is determined by the size of the fuzzy kernel . The size of determines the number of iteration steps per layer to be 25. First, the color image is converted into a gray-scale image, on which the saliency structure is extracted; second, the estimation of the blur kernel and the recovery of the image are performed; then, the recovered image is upsampled and used as the initial input for the next iteration. In the final image restoration stage, the three channels of the color image are processed separately.
The parameters of the experiments are set as follows: the parameters in the model . The parameters under the GSM-FoE model and are adjustable parameters, and the experiments generally take , the number of expert functions , and the window size . The EM learning algorithm is applied to learn eight filters and their corresponding parameters from the training database set.
The simulation platform is MATLAB R2018a, the computer configuration is: 64 bit windows 10 system, Pentium dual-core 2.8 GHz, running memory is 2 GB. In our experiments, our learning data came from the Berkeley Segmentation Benchmark image library. There are 500 benchmark images in this database, including many common natural scenes in everyday life, such as animals, people, landscapes, buildings, etc. It is the most commonly used image database for image segmentation, edge detection, and other related fields. The eight 3 × 3 filters used in the following experiments are the result of learning this database as a whole, which is undoubtedly very time consuming. Given the efficiency of the experiments, the user can also manually select images in the image library that have some correlation with the blurred image B for training.
4.1. Experimental Comparison of Image Prior Terms with and without the Gradient Fidelity Term
Tikhonov first proposed to use this constraint on the squared norm of the gradient as a regular term to solve the discomfort problem of images to better remove noise. In this paper, we combine the gradient fidelity term with the GSM-FoE model, proposing a novel image regularity term. The experimental results show that the addition of a gradient fidelity term significantly suppresses noise in the recovered image and reduces the generation of the ladder effect. Since the gradient fidelity term is a convex problem that can be solved by the fast Fourier transform, it does not destroy the existence of the optimal solution of the deblurred model.
Figure 4 shows an experimental comparison of the image prior term with and without the gradient fidelity term. Figure (a) shows a clear image, figure (b) shows a blurred image, figure (c) shows the image restoration results with only the GSM FoE model introduced in the image prior term, and figure (d) represents the introduction of the GSM-FoE model and the gradient fidelity term in the image prior term. In terms of the recovery results, some information such as edges and textures are partially missing in figure (c). Figure (d) not only eliminates the noise but also suppresses the ringing effect by adding the image gradient fidelity term, and the recovered image has a clearer and more natural visual effect at the edges and textures.
(a)
(b)
(c)
(d)
4.2. Comparison of Quantitative Experiments
Using accuracy and time-consuming experiments as indicators, literature [5], literature [18], and literature [20] were used as control groups for the experiments and their experimental results were compared with the experimental results of this research method. The Bregman reweighted alternating minimization (BRAM) was applied to image deblurring [5]. Nonblind image deblurring was proposed via deep learning (DL) in a complex field [18]. The image restoration method used in this paper is the GSM-FOE model. The gradient-based conditional generative adversarial network (CGAN) was used to image deblurring [20]. The experimental data samples were obtained from the Berkeley image database, and the effect of different methods on the recovery of degraded images was verified through synthetic datasets.
4.2.1. Recovery Time Comparison
500 blurred images under synthetic data were selected for blind recovery, and the algorithm in this paper was compared with other three algorithms for recovery time, and the experimental results are shown in Figure 5.
According to the analysis of the data in Figure 5, the recovery times for the methods of literature [18], literature [20] and this paper are relatively short when the number of images is small. Between 190 and 230 images, there is a relatively large qualitative change in the recovery time of the methods of literature [5], literature [18], and literature [20] as the number of images increases, followed by a relatively stable oscillation. Overall, as the number of images increases, the recovery time of the method in this paper is relatively stable and the time required for recovery is short and efficient.
4.2.2. Comparison of the Degree of Recovery
To further validate the performance of this method, PSNR (peak signal-to-noise ratio), MSE (mean square error), and the degree of recovery were used as test metrics, and the methods of literature [5], literature [18], and literature [20] were used as control groups to blindly recover blurred images, respectively, to compare the recovery effects of the five algorithms, and the metric data are shown in Table 1.
The experiments show that the peak signal-to-noise ratio value of the method in this paper is 93.25 db, which is at least 7 db higher than that of other methods in the literature. The mean square error value is 15.18%, which is much lower than that of other methods. Figure 6 shows the comparison of the recovery degree of different methods. As shown in Figure 6, as the number of images increases, the recovery degree of the literature [18] method shows a sudden low change between the number of 10 and 50 images, followed by a smooth decrease. Overall, there is a steady oscillation in the degree of recovery in literature [5] and literature [20] as the number of images increases, while the degree of recovery of the method in this paper then increases and is significantly higher than that of the other three literature methods, indicating that the recovery results in this paper are clearer and better maintain the details of the images.
4.3. Recovery Effect
The blind recovery experiments of the method are shown in Figure 7, where the fuzzy kernel is unknown. On the left is the fuzzy image, and on the right is the recovery result. In addition to the first-order derivative information, the intrinsic features of the image also include higher-order prior information, and the training prior learning method can learn these higher-order priors from a given image database, so the regular term method in this paper is more accurate than the approximation prior method. In terms of the overall visual effect, this paper effectively suppresses the ringing effect while removing blur, and maintains the edges of the image well.
5. Conclusion
Image deblurring techniques have a wide range of applications in daily life, industrial production, and other fields and have received widespread attention in research areas such as image processing and computer vision. The method of learning with the GSM-FoE model can better fit the higher-order prior of natural images and accurately portray the global prior knowledge of natural images. The graph of the logarithmic distribution of the expert function shows that the Gaussian mixed-scale expert function fits the heavy-tailed distribution better than other expert functions. In this paper, using the GSM-FoE model, 500 images from the Berkeley database were trained to learn eight filters and the corresponding parameters.
In this paper, these learning results under the line of GSM-FoE model are used to guide the image restoration process in the objective function of image deblurring. In the image restoration process, the gradient fidelity term is also incorporated into the image deblurring model in this paper, considering that the image gradient term can better suppress noise, so the traditional regular term is improved. An effective algorithm based on IRLS is proposed for the GSM-FoE model and the image gradient fidelity based regular term, which adaptively changes the parameter values during the iterative process and the recovered image can better maintain the details. In this paper, an efficient algorithm with alternate iterations of fuzzy kernel and image recovery is used, and experiments show that the algorithm can effectively handle the case of large fuzzy kernels.
In future research work, more consideration needs to be given to the following issues. First, training against the Berkeley image library is time-consuming, so we need to find a relatively fast learning method; second, given the diversity of learning models, further research studies will follow on other learning models and their application in the direction of image deblurring. Therefore, the next step is to investigate a more efficient method of extracting saliency structure, which can incorporate it into the global inconsistent image deblurring problem.
Data Availability
Some or all data, models, or codes generated or used during the study are available in a repository or online in accordance with funder data retention policies.
Conflicts of Interest
The authors declare that there are no conflicts of interest with any financial organizations regarding the material reported in this manuscript.
Acknowledgments
This work was supported by Jilin Province Higher Education Association Project (JGJX2020D516).