Abstract
Existing image inpainting algorithm based on low-rank matrix approximation cannot be suitable for complex, large-scale, damaged texture image. An inpainting algorithm based on low-rank approximation and texture direction is proposed in the paper. At first, we decompose the image using low-rank approximation method. Then the area to be repaired is interpolated by level set algorithm, and we can reconstruct a new image by the boundary values of level set. In order to obtain a better restoration effect, we make iteration for low-rank decomposition and level set interpolation. Taking into account the impact of texture direction, we segment the texture and make low-rank decomposition at texture direction. Experimental results show that the new algorithm is suitable for texture recovery and maintaining the overall consistency of the structure, which can be used to repair large-scale damaged image.
1. Introduction
Image inpainting is a hot research for computer graphics and computer vision. The main purpose is to, in a way of not being noticed, repair the damaged image or remove the unwanted objects in the image. Image inpainting’s mission is to, adopting the most appropriate method, restore the original state and to ensure that the image achieves the ideal artistic effect at the same time.
Image inpainting can be roughly divided into three broad categories: structure-based repairing, texture-based repairing, and learning-based repairing. For the nontexture structure methods, the most commonly used is the high order partial differential equations (PDE) method. PDE method was put forward by Bertalmío et al. [1] firstly, and the idea was to extend the information of the area around the repair area to the damaged regions using the thermal diffusion equation of physics. The contour is estimated based on the edge information of repair area and the external information of the contour along the contour normal spread to the pixel in the middle. Chan and Shen [2], based on the research of Bertalmío, converted the repair problem to a total variational (TV) problem which is a curvature-driven inpainting model. Considering the curvature of the contour in the diffusion process, Chan and Shen [3] extended the TV method and proposed curvature-driven diffusion (CDD) method. It suppresses large curvature contours by curvature. Texture structure-based restoration technology also contains two methods: one is the image decomposing method, which can be decomposed into structural parts and texture part; another is texture synthesis technique based on the sample, and the most typical method is Criminisi algorithm [4]. The repair technology based on study is to find the statistical information or the prior probability through original image or other image databases. It uses some optimization algorithm to reconstruct the image, such as belief propagation (BP) and gradient descent method.
The storage structure of image in the computer is a matrix and an image’s pixels corresponding to a matrix. Each location of the matrix stores the space and color information of a pixel in the image, so the image processing can be done in treatment of matrix. For large sample data, the approximate algorithm of low-rank matrix is very is very suitable. The low-rank approximate (LRA) is a technology which hunts for large-scale matrix low-rank representation, which is a powerful method to seek potential useful information from large-scale complex data. Low-rank decomposition decomposes complex matrix into lower rank matrix and sparse matrix, to make it better for processing, and the same or similar results are obtained. In recent years, low-rank matrix method in signal processing, image processing, computer vision, high-dimensional data analysis, and other fields has important and successful application; this paper mainly studies its application in image restoration.
2. Related Works
Many scholars have proposed a variety of effective methods for low-rank approximation decomposition, and Nyström [5] is one of the most popular methods. Some simple but effective stochastic approximation algorithms are used to construct low-rank matrix factorization, and it can be used in complex field matrix. Also there are many other low-rank approximation methods, such as QR decomposition and sampling low-rank approximation method. In general it can be divided into two kinds: one is sparse rules or column switching operation; another is using sampling principle to set the kernel matrix column random selection probability. Smola and Schőlkopf [6] use sparse approximation method to obtain a matrix compressed form, and the time complexity is for greedy method. Girosi [7] proposed sparse approximation of support vector machine (SVM). Liu et al. [8] put forward the low-rank approximation and decomposition for kernel matrix based on column correlation.
Dimension reduction is a key step to solve the approximate low-rank matrix, which is widely used in feature point extraction, denoising, and other areas of the image processing. Nonnegative matrix factorization (NMF) and singular value decomposition (SVD) are two kinds of common data dimension reduction method. Lee and Seung early used nonnegative matrix decomposition approach to reduce the dimension of large-scale data [9–11]. Using multiplicative iterative rule, a nonnegative matrix approximation is decomposed to a product of two nonnegative matrices, and the decomposition algorithm can be regarded as a nonlinear programming problem with constraints. The nonnegative matrix factorization is related to initial value deeply, and we often get only local minimum value. Using a different objective function, a variety of NMF algorithm was put forward. Li and Zhang [12] proposed a bilinear NMF. Huang et al. [13] proposed a constrained NMF. Shi et al. [14] put forward an acceleration algorithm for incomplete NMF. Cai et al. [15] proposed graph regularized NMF, which retained the neighbor structure of the original sample well. NMF feature vector is nonnegative. The vectors after factorization are not completely orthogonal, and have some redundant information. SVD has obvious physical significance. It can express a complex matrix by several smaller and simpler submatrixs’ multiplication. These submatrixs are the important feature of matrix. The eigenvectors after singular value decomposition are orthogonal to each other. SVD feature vectors are orthogonal to each other, and it can do the best approximation for the sample vector under different dimensions. The error between the sample and approximate result is minimal under the Euclidean distance. As a kind of algebraic features, SVD has the invariance of the algebra and geometry and is not sensitive to noise.
Hong [16] demonstrated that the singular value feature is effective feature for image recognition and proved the stability, scale invariance, and rotation invariance of the singular values characteristics. It is easy to get a matrix optimal approximation for a matrix . It can effectively reflect the characteristics of the matrix, which was widely used for image feature extraction, image retrieval and matching, digital watermarking, and other fields. Zhang and Zha [17] analyzed the SVD structure of column block matrix and sparse low-rank approximation method. Gan and Zhang [18] presented new face recognition ways by singular value feature based neural networks. Wang et al. [19] proposed a face identification method based on SVD and data fusion. He and Sun [20] obtained singular value feature vector using SVD, singular value reduction, the singular value vector standardization, and singular value vector sort.
Considering how to recover low rank from larger and sparse error, it decomposed a matrix into a low-rank matrix and a sparse matrix, sometimes also called sparse and low-rank matrix decomposition or robust principal component analysis (RPCA). Chen and Cheng [21] put forward a kind of image restoration algorithm based on low-rank matrix and joint sparse matrix. Tan et al. [22] proposed a face recovery algorithm based on RPCA, which can recover the face from video stream losing some frames. Liang et al. [23], using low-rank and sparse structure, restored the image with regular or approximate regular texture. The method automatically and accurately fixes the global structure and the texture of damage by convex optimization low-rank and sparse signal recovery. Wang and Zhang [24] proposed an image restoration algorithm based on weighted sparse nonnegative matrix decomposition. Damaged area can be repaired by image block, and good candidate block can be described by low-dimensional linear model. Wen et al. [25] solved the low-rank matrix decomposition model by nonlinear successive overrelaxation algorithm, and each iteration only needs to solve a linear least squares problem. Zhang et al. [26] put forward a kind of transform invariant low-rank textures (TILT) model, which can be used for the extraction of low-rank texture. The algorithm is effective for the image with heavy deformation or damage.
3. Matrix Low-Rank Decomposition
For a very large-scale matrix, sometimes we want to use a matrix with lower rank to approximate, and that is the low-rank approximation. This problem can be converted into matrix decomposition, and the decomposition of a matrix is defined as follows: where is a low-rank matrix and is a sparse matrix.
Figure 1 illustrates the principle of image restoration algorithm based on low-rank approximation. is the repaired image we want to get, and is the damage parts or spare parts. The improving of low-rank decomposition method has a large extent and affects the quality of image restoration. So the above matrix low-rank decomposition problem can be converted to the following optimization problem: where is spectral norm or Frobenius norm and is the rank approximation of matrix .
In NMF algorithm, its feature vector is not completely orthogonal, and there exists information redundancy. SVD can represent a complex matrix with the multiplication of several smaller and simple matrices, which are the important feature of matrix. SVD feature vectors are orthogonal to each other, and they are able to do the best approximation under different dimension.
The following describes the best approximation properties of SVD theorem [27].
Theorem 1. , , and singular value decomposition for matrix is
where , , and .
If the energy of image is expressed by the square of the -norm, one can get
where is nonzero singular value for . is the rank of matrix.
After the singular value decomposition, we can see from formula (4) that the texture and geometry information are concentrated in the and . The singular value of represents the energy information of the image.
If is small enough, is the best rank approximation of , namely,
can be regarded as its theme matrix, and singular value determines the importance of these themes. The smaller singular values reflect unimportant or subject to “noise” affected theme. Therefore the best low-rank approximation of is to get rid of these secondary themes. When is small enough, is the number of important topics of . is the best low-rank decomposition for .
According to the above formula, the low-rank decomposition algorithm steps of SVD can be described as follows.
Step 1. Let .
Step 2. For (1)calculate largest singular triples of ,(2)let
Step 3. Let , , and .
is a diagonal matrix; . The best rank approximation for is
4. Image Inpainting Algorithm
A typical image restoration problem can be described as is an image on area ; is the region to be repaired. Region can be nonconnected; region is areas other than and may contain noise. To get an acceptable effect for the whole image is the goal that we want to achieve.
Texture is widespread in our lives, and people can easily recognize it but find it difficult to define. Texture is a feature for reflecting the spatial distribution of pixel gray. Usually it is irregular in local, but is regular in global. Much attention has been paid to texture synthesis, and image restoration is similar to texture synthesis processing. They all need sampling and analysis and generate similar image information based on the source image to synthetize or fill it into the original image. The images are generally multitexture images, which contain a variety of texture and structure information. So firstly we need segment texture in order to better maintain the texture and border structure information.
In this paper, the basic steps of the image restoration algorithm based on the matrix low-rank approximation are to be as follows.
Step 1 (low-rank decomposition). Firstly, using the matrix decomposition algorithm proposed by Section 3, the image matrix is decomposed into the corresponding low-rank matrix and sparse matrix (as shown in Figure 2).
Step 2 (level set interpolation). After image low-rank decomposition, usually we cannot get the desired repairing effect, requiring further level set interpolation.
Step 3 (direction repairing). Usually according to the direction feature of the texture, we can find the general direction of the texture. We find the matching block along the direction, and it can reduce the repairing time and improve the repairing effect. Therefore, for different repaired area after segmentation, we, respectively, obtained low-rank texture direction and rotate corresponding angle. Also we make low-rank decomposition and level set interpolation processing at each texture direction. At last we take an inverse rotation to back its original position.
4.1. Texture Segmentation
Texture segmentation is a means of regional segmentation using texture characteristics. The image is divided into a set of limited area sets with uniform texture feature, including texture feature extraction, boundary processing, and texture feature classification. There are a variety of texture segmentation methods, such as deformation model, MRF model [28], Markov model [29], level set model [30], and multiscale texture image segmentation based on EHMM-HMT and MSWHMT [31].
In this paper texture feature extraction is based on wavelet transform pyramid structure. We use local statistical method to estimate texture boundaries and adopt d-tree as the data structure for -means clustering. Firstly using Mallat algorithm to decompose the images into levels where and represent the low pass and high pass filter, respectively. and are the row and column, . Consider
Formula (8) is the mean value, and formula (9) is the variance. We use variance as “energy measure” to obtain each dimension’s characteristics of texture segmentation. Then we use quarter method to smoothen the texture characteristics. According to formula (9), in the th decomposition, it is easy to obtain the mean and variance of four neighborhood subwindows, and the minimum variance is
We create a binary d-tree to store multidimensional clustering data. Making -level wavelet decomposition for original image (take, e.g., ), 4D vectors are formed at each position for all decomposition levels. In order to get the same size with the first level wavelet decomposition, the image after second decomposition is extended two times in the horizontal and vertical directions, and an eight-dimensional feature vector is created. Fixing the number of clusters and cluster centers , for each node , the nearest center from the center point of clustering center set is
The formula assumes is the smallest box that contains all the data. is the minimum value along the segmentation dimension, and is the maximum value. Consider
If formula (12) is established, will be ignored and ultimately forming a new cluster center:
Because the texture is regular at the macro, the damaged region is divided into several different regions according to the texture feature. The region with the same texture belongs to the same structure region. Boundary curve can be connected smoothly by tensor voting algorithm. Figure 3 shows the image segmentation by texture feature. Different color represents different texture region.
4.2. Level Set Interpolation
After image low-rank decomposition, the repairing effect may not still be very ideal (see Figure 4(a)), and it can be improved by level set interpolation. Literature [32] proposes an image restoration algorithm based on horizontal line interpolation. The damaged area is interpolated from outside to inside, ring by ring, and the horizontal line gradually extends into damaged region.
(a)
(b)
(c)
(d)
(e)
(f)
AMLE model [33] is put forward by Almansa et al. They proved that the second-order partial differential equations have a unique viscous solution: where is the boundary of the domain . is a curve in . is a point in .
The steady-state solution of elliptic partial differential equations is consistent with the corresponding steady-state solution of parabolic partial differential equations, so the level set interpolation of image can be solved by the corresponding parabolic partial differential equation. The two expressions of formula (14) are the generalized boundary conditions.
By formula (14), we obtain
The equation shows that AMLE can be seen as a directivity diffusion problem. It is similar to coding method based on level set of image, only encoding the data of the boundaries. When decoding a new image can be reconstructed using AMLE, and it is the values on the boundary of level set. If there is enough level set, the transition is smooth from a level set to the adjacent level set, so we can consider that is an approximation of the original image.
If only keeping level sets, then the boundaries values of level sets can reconstruct a new image (see Figure 4(b)). Then we perform low-rank decomposition operation, and this process can be repeated many times. But next time, the number of level sets must be more than the previous interpolation (usually is 1.5 to 2 times). With the level sets (line) number increasing, it is smoother for the transition between level sets, so the accuracy of approximation is higher.
4.3. Texture Direction
Texture direction is an essential feature of the texture image, which has a strong practical significance. It is a concept on a region and is a statistical characteristic in large neighborhood region. Texture direction estimation occupies an important position in segmentation, fusion, classification, reconstruction, retrieval, and so on. If not considering the local difference of the image, image reconstruction can produce degradation and distortion. Classification-based and wavelet-based reconstruction methods do not consider the texture direction characteristics of image edge, so the reconstruction exists blur and aliasing phenomenon. The boundary information can be introduced to improve the reconstruction effect. Considering only the relationship between horizontal and vertical direction and ignoring other directions, the effect is not very ideal. Image reconstruction method based on adaptive texture direction can estimate the most abundant texture direction. Not only is the reconstruction quality of smooth region higher than traditional method, the effect is more obvious on edge and texture rich regional.
In Figure 5, we can see that the image has obvious texture direction. Along the texture direction, low-rank decomposition can get good inpainting effect, and it can be seen in Figure 5(d). The accuracy of texture direction detection will directly affect the image restoration effect. The commonly used method is based on image texture spectrum features, mainly using the multichannel filtering, including the discrete cosine transform (DCT), Gabor filter, wavelet transform, Bandelet transform, and directional filter bank. For image restoration, this method is not appropriate, due to the sampled values of the texture function being subject to many types of corruption such as noise, damage, and occlusions. In addition, the change of viewpoint usually induces a transformation on the domain of the texture function. Using low-rank method to compute texture direction, it can well express the deformed texture and corrupted texture. It can recover the exact texture from an image that contains a deformed and corrupted version of it. The results shown in Figures 6 and 7 demonstrate the robustness of the low-rank texture direction to damage, noise, rotation, and corruptions, respectively.
(a)
(b)
(c)
(d)
(a)
(b)
(a)
(b)
We consider a two-dimensional texture as a function defined on . One-dimensional function spans a finite low-dimensional linear subspace [26], such as
If is finite, then is rank texture.
According to formula (16), for low-rank texture, its rank is invariant for rotating translation, namely,
To some extent, image is a transformation of a low-rank texture function: where belongs to a texture group , .
Considering the influence of the noise and the surrounding background, the image can be expressed as where is error matrix.
We assume that only small regions of the image pixels are damaged, so is a sparse matrix. For a distorted or damaged image, we calculate the low-rank textures and domain transform ; then we can get the texture direction:
This problem can be converted into the following optimization problem: where is the number of nonzero data in sparse matrix .
After texture segmentation, the area to be repaired is divided into four texture regions (see Figure 7(a)). Low-rank texture direction of the hair is shown in Figure 7(b). According to the low-rank texture direction, we rotate certain angle and then make low-rank decomposition. At last, making an inverse rotate, the damage part is repaired by the corresponding region.
5. Experiment
In order to check the validity of our restoration algorithm, take standard Lena, Barbara, and boat image as an example. We compared our algorithm with low-rank decomposition LMaFit algorithm [34]. LMaFit method repair effect is not ideal for a large damaged area, and there are still obvious repair traces. Our approach has achieved good results, and repairing effect is better than LMaFit, which is closer to the original image and more consistent with human visual. After texture segmentation, the image has strong direction texture. We can better repair any directional texture images using low-rank texture direction.
From the three groups’ image restoration results (Figure 8), we can find that our algorithm gets better repair effect than traditional low-rank method, especially for the image with strong structure or larger damaged area.
There are mainly two kinds of methods (subjective and objective methods) to evaluate the restoration quality. Subjective method mainly is the subjective feeling of person’s visual sense, and objective method is mainly the peak signal to noise ratio (PSNR). PSNR is used to measure difference of two images. The greater the value, the better the repair effects and the smaller the difference of original image and repaired image. Table 1 is a comparison for our algorithm and the LMaFit algorithm using PSNR.
In Figure 9, we make a comparison for our method, Oliveira algorithm, TV algorithm, CDD algorithm, PDE algorithm, and RBF algorithm. It can be seen that our algorithm is superior to other methods. Influenced by texture, repairing effect of traditional methods is not very ideal. The essence of image restoration problem is an ill-condition problem, and we usually need to introduce prior knowledge constraints. By limiting the scope of the problem domain, ill-condition problem transforms into benign problem. Most traditional algorithms get the optimal solution via variational methods. For smaller regional repair, variational method can restore image edge structure, but it is not suitable for processing texture detail.
(a)
(b)
(c)
(d)
(e)
(f)
6. Conclusion
Low-rank matrix approximation is an important method of data acquisition and representation and is a very effective way to obtain low-rank structure of high-dimensional data. In this paper, we propose a new image restoration algorithm based on matrix low-rank approximation. First the image is decomposed by low-rank approximation and then interpolated by level set interpolation method and, at last, making texture segmentation according to the image features and getting the texture direction by low-rank approximation; we consider the impact of texture direction to get good repair effect. Seen from the experimental results, our method is better than traditional image restoration algorithms, but there is still a gap compared with the best result. Because of iterations, there is slightly more computation; the efficiency of the algorithm needs to be improved. In practical application, the singular value decomposition is the most time-consuming, so in the next step we will study the fast algorithm.
In the future, we plan to utilize homotopy perturbation method for measurable improvement. Homotopy perturbation method can solve various kinds of nonlinear functional equations. The image inpainting problem is a typed nonlinear ill-posed inverse problem. There are many image restoration methods based on Perona-Malik equation (PM). Using homotopy perturbation method, we can obtain the correspondent approximated solutions for the PM equation with regard to relevant error analysis, which will lead us to more speedy computations. In addition, we use level set interpolation to get more ideal effect. It needs to solve second-order partial differential equations. Homotopy perturbation method has faster convergence than the classical method, which has been proved and analyzed in literature [35]. So adopting homotopy perturbation method, the efficiency of our algorithm will be improved.
Conflict of Interests
The authors declare no conflict of interests.
Acknowledgments
This work is supported by the National Natural Science Foundation of China under Grant nos. 61173173, 61272430, 61373079, and 61472227 and the Provincial Natural Science Foundation of Shandong under Grant no. ZR2013FM015.