Abstract
Image restoration is a research hotspot in computer vision and computer graphics. It uses the effective information in the image to fill in the information of the designated damaged area. This has high application value in environmental design, film and television special effects production, old photo restoration, and removal of text or obstacles in images. In traditional sparse representation image restoration algorithms, the size of dictionary atoms is often fixed. When repairing the texture area, the dictionary atom will be too large to cause blurring. When repairing a smooth area, the dictionary atom is too small to cause the extension of the area, which affects the image repair effect. In this paper, the structural sparsity of the block to be repaired is used to adjust the repair priority. By analyzing the structure information of the repair block located in different regions such as texture, edge, and smoothing, the size of the dictionary atom is adaptively determined. This paper proposes a color image restoration method that adaptively determines the size of dictionary atoms and discusses a model based on the partial differential equation restoration method. Through simulation experiments combined with subjective and objective standards, the repair results are evaluated and analyzed. The simulation results show that the algorithm can effectively overcome the shortcomings of blurred details and region extension in fixed dictionary restoration, and the restoration effect has been significantly improved. Compared with the results of several other classic algorithms, it shows the effectiveness of the algorithm in this paper.
1. Introduction
With the rapid development of science and technology, especially the development of computer technology, people began to associate images with computers [1, 2]. Many precious calligraphy and paintings, files, classic videos, especially ancient paintings, and murals with historical significance can be scanned. However, they will inevitably have scratches or stains when they are kept for a long time. For example, long-term storage of photos will produce scratches or mildew stains, and movie films will inevitably produce scratches when played repeatedly [3]. These actions will affect the visual effect of the image to varying degrees. If its professionals perform manual repairs, it will definitely consume a lot of manpower, material, and financial resources, and the risk is very high [4]. The emergence of digital image restoration is of great significance to image processing. People can use computer technology for automatic restoration, so that it will not cause damage to the original work but also can restore the image repeatedly until a satisfactory visual effect is obtained [5, 6]. Nowadays, digital image restoration computing has become an active research direction in the image field [7].
The construction and decoration industry is developing very fast, not only the number of employees has increased sharply, but also the design level is improving day by day [8, 9]. The key engineering projects of environmental art design are also presided over by early foreign designers, and some projects are completed by Chinese environmental art designers in cooperation with foreign designers [10]. Whether it is residential or commercial office buildings, a lot of design is required, which provides a large market for environmental art design [11]. Because of the human pursuit of new science and new technology and the emphasis on practical functions, modern environmental art design presents new characteristics, diverse design styles, and the comfort of space is constantly improving [12]. Environmental art design makes full use of industrial materials and mass production of industrial products, as well as high-tech technology and methods. When material and technological conditions permit, it pays attention to human touch and pursues individuality and originality as much as possible; environmental art design pays more attention to multiple artistic styles [13, 14]. The development of environmental art design is both modern and traditional. The current environmental art design presents a multilevel and multistyle development trend, and some are simple and fresh, and rich in pastoral colors. Modern environmental art design is a holistic art, whether it is level or style. It must deal with space, shape, and color reasonably and create a style that is in harmony with the surrounding environment [15]. Contemporary people pay more attention to the communication between people in the internal space, as well as the spiritual factors of space environment and culture, and pay more attention to the originality of design [16].
The focus of current research is to convert complex image information into less information for storage and use [17]. Therefore, image restoration algorithms based on sparse representation have received extensive attention. Image sparse representation is based on compressed sensing (CS) theory, which represents an image as a linear combination of a few atoms in a dictionary [9, 18]. Then, if the dictionary is known, how to represent the image information in the sparsest way, that is, the sparse coding process, is a problem to be processed by image restoration [19]. At present, there are many optimization methods for sparse coding: relaxation methods, which smooth the l0 norm and use continuous optimization techniques, such as basis pursuit (BP) [20]. Greedy algorithm constructs solutions with nonzero elements in each iteration, such as matching pursuit (MP) and orthogonal matching pursuit (OMP); the successive approximation method uses the approximation equation to solve the sparse solution of the norm [21, 22]. Training the learning dictionary is also a crucial step in image restoration [23]. Currently, two methods can be used to construct a dictionary. The first is to use the method of harmonic analysis to construct a fixed dictionary, which can use complex wavelet, contourlet, and other methods. The dictionary structure constructed by this method cannot be updated and changed, so it cannot adapt to various image restoration situations. Another method is to learn from training samples to obtain a sparse dictionary [24, 25]. This kind of dictionary is adaptive to different types of image restoration [26–28]. Compared with a fixed dictionary, learning a dictionary can often achieve better results. Related scholars use color image libraries as training samples, use independent component analysis methods to train the dictionary, and then use sparse coding methods to repair image blocks one by one [29–31]. However, the algorithm does not fully consider the global sparsity of the image [32, 33].
In daily life and certain application scenarios, images that do not meet the requirements are often encountered, such as old photos with scratches or stains, damaged ancient cultural relics, calligraphy and painting, and obscured objects. This affects the completeness and validity of image information and even affects the later work of image processing, such as target detection and image registration. The digital image restoration technology automatically detects and repairs damaged areas by inputting the corresponding algorithm into the computer, so this technology is currently the most direct and effective method for restoring works of art and old photos. The research on the image restoration algorithm is to accurately restore the structure and texture information inside the damaged area, obtain the restored image with minimal error from the original image to the greatest extent, and improve the restoration efficiency of the corresponding method to meet the real-time requirements. In traditional sparse representation algorithms, the size of dictionary atoms is often fixed. When repairing texture regions, the dictionary atoms will be too large to cause blurring. When repairing smooth regions, the dictionary atoms will be too small to cause region extension, thereby affecting the effect of image restoration. In order to overcome this defect, this paper adopts the structure sparsity of the block to be repaired to adjust the repair priority. Specifically, the technical contributions of this article can be summarized as follows.
First, by analyzing the structural information of the repair block in different regions such as texture, edge, and smoothing, we adaptively determine the size of the dictionary atom. A color image restoration method that adaptively determines the size of dictionary atoms is proposed. Experimental results show that the algorithm can effectively overcome the shortcomings of blurred details and region extension in fixed dictionary restoration, and the restoration effect has been significantly improved.
Second, in this paper, a sufficient simulation experiment and performance analysis of the proposed algorithm are carried out. We carried out the removal of portraits, the removal of subtitle scratches, and the completion of the simulation of the image structure. By comparing with related algorithms, the effectiveness of this algorithm in repair quality is analyzed. The time cost of algorithm repair, peak signal-to-noise ratio and structural similarity index are used as objective evaluation criteria. A comprehensive comparative analysis of the experimental results proves the advantages of the algorithm in this paper.
The rest of this article is organized as follows: Section 2 discusses the related theoretical basis of image restoration. Section 3 designs an image restoration algorithm with block structure sparsity constraints and dictionary atom size adaptation. In Section 4, a simulation experiment was carried out. Section 5 summarizes the full text.
2. Related Theoretical Basis of Image Restoration
2.1. Classification of Damaged Images and Description of Image Repair Problems
2.1.1. Classification of Damaged Images
Image restoration is to reconstruct the original high-quality image x from a low-quality image or a damaged image y. It is a typical ill-conditioned linear inverse problem, which can generally be expressed aswhere x and y are the dictionary stack representations of the original image and the degraded image, respectively, H is the matrix representing the linear irreversible degenerate operator, and n is usually additive white Gaussian noise.
The repair of damaged images can be regarded as an ill-conditioned linear inverse problem. At this time, H is divided into several situations, corresponding to several types of image damage.
When H is a mask, that is, H is a diagonal matrix whose diagonal item is 1 or 0, the cause of image damage is mostly improper preservation of old photos or artificial damage, causing some blocks in the image to be missing.
When H is an identity, for the environmental conditions in image acquisition and the quality of the sensing components themselves, the image distortion and noise are caused by the interference of the transmission channel during the compression and transmission process.
When H is the blur operator, when the object shot by the camera is blocked, or the lens moves during image capture, or the mosaic is artificially added to cover up the information in the image, the image is partially blurred.
2.1.2. Description of Image Repair Problem
Image restoration is a process of visual cognition. People use prior knowledge and inherited experience to reasonably reconstruct the information of the image defect area under the control of the perceptual psychological organization characteristics. In digital image restoration, the Gestalt organization law is of important guiding significance. Only by simulating the characteristics of human perceptual organization in the algorithm design can we obtain restoration results that conform to human visual psychological habits. In the current image restoration, the similarity, proximity, and continuity of perceptual tissues are the most widely used [34, 35]. The same-directional motion law of perceptual organization is also widely used in video motion segmentation.
From a mathematical point of view, digital image repair is to use the known information in the image to fill in the unknown area to be repaired in the image through certain rules and algorithms, so that the image to be repaired can maintain certain integrity.
Image restoration belongs to the field of image restoration. However, image restoration is usually a morbid problem, because there is still not enough information to ensure that the damaged part can be restored uniquely and correctly. Therefore, people have put forward various assumptions to solve this problem from the perspective of visual cognition.
Usually, in the process of image acquisition, it will be affected by some unknown factors, which will degrade the image quality. In the field of image restoration, the commonly used degradation models are
Among them, I0 is the obtained target image, I is the original image, and N is the additive white Gaussian noise. For most digital image restoration problems, the data model has the following form:
Among them, D represents the area to be repaired in the image, and I is the target image. Assuming that N is Gaussian, the data model of the energy function E can be defined as the minimum mean square error:
Since there are no available data in the repair area D, the image prior model is more important to the image repair algorithm than other traditional restoration problems (such as denoising).
The image prior model can be obtained from the image data through filtering, estimation and image entropy. Although these statistical methods are very important for the restoration of images with rich textures, for most image restoration problems, important geometric information (such as edges) in the image is often lost in the area to be repaired. In order to reconstruct this geometric information, the image prior model must resolve these geometric features in advance, but most traditional probability models lack this feature.
2.2. Analysis of Traditional Image Restoration Methods
Traditional image restoration methods mainly include partial differential equation image restoration models, sample image restoration models, and image restoration methods based on hybrid models. Among them, image restoration models based on partial differential equations mainly include Bertalmio–Sapiro–Caselles–Bellester (BSCB) model, total variation (TV) model, and curvature-driven diffusion-based restoration model. This type of algorithm has a mechanism for simulating microrepair, which can be derived from each other through differential and variational principles. The repair process is based on the edge information of the area to be repaired, and the filling information of the area to be repaired is calculated according to the diffusion factor of the partial differential equation. The advantage of this type of method is that it has a better repair effect on images with small damaged areas and denoising. However, this type of method also has its shortcomings; that is, the effect of repairing images with large damaged areas is relatively poor, blurring will be introduced, and the repairing will be incomplete. Based on the method of sample image restoration, the idea is to first sample in the undamaged area of the image and select appropriate image blocks to fill continuously along the damaged edge.
2.2.1. BSCB Repair Model
The BSCB algorithm iteratively diffuses the information around the area to be repaired in the damaged image into the area to be repaired. Due to the irregularity of the damaged area, the isoilluminance lines will cross, so the iterative diffusion process of the BSCB algorithm is not simple direct diffusion, but anisotropic diffusion. You repeat iteratively until the entire damage is completed. The specific process is shown in Figure 1.
As shown in Figure 1, the BSCB model regards the repair process as a series of diffusion processes and extends the edge area information along the normal direction of the damaged edge to the area to be repaired. Given a two-dimensional grayscale image, the size of the image is . First, a sequence of images is constructed. The transmission process uses an iterative equation as
Among them, n is the number of iterations, ∆t is the iteration step size, and I0 (i, j) is the pixel value of point (i, j) obtained in the nth iteration.
In the process of image restoration, you should not only pay attention to the overall restoration effect, but also pay attention to specific details. Ordinary diffusion methods cannot satisfy texture details or structural details when repairing a damaged image. Therefore, using anisotropic diffusion can not only remove noise, but also effectively keep the edges of the image smooth.
The process of anisotropic diffusion first performs anisotropic diffusion in each direction of a damaged image to filter the interference of noise on the image, then performs n iterations of repair, and then performs anisotropic diffusion and n times of repair.
2.2.2. CDD Repair Model
When the CDD model transmits information, in addition to considering the gradient information, its diffusion direction is also restricted by the curvature of the isoilluminance line, which makes the information diffusion more accurate. That is, a penalty function is added to the original second-order nonlinear diffusion equation, and the CDD repair model is obtained as shown in the following formula:
Among them, the function is a monotonically increasing function of the variable s.
In the CDD model, the magnitude of the equal illumination curvature determines the magnitude of the conductivity coefficient, and the two are in a proportional relationship. According to the image gradient information, where the curvature is relatively smooth, the diffusion coefficient is close to zero, and where the image gradient information changes greatly, the curvature is large and the diffusion coefficient is also large.
The BSCB model is the earliest third-order partial differential equation proposed; the total variational repair model is a second-order partial differential equation, so when the same damaged image is repaired under the same conditions, the repair speed of the total variation repair model is faster than BSCB. The model is repaired, but it also has its own shortcomings, and the CDD model is an improvement to the TV model. Although it overcomes the shortcomings of the TV model, the solution process is also to calculate the third-order partial differential equation, and the time complexity is still relatively higher.
2.2.3. Sample-Based Repair Method
Image restoration models based on partial differential equations are not suitable for restoring images with larger damaged areas. In actual problems, we often encounter many large-area damaged images. In response to this situation, a sample-based repair method uses image blocks as the operating unit and can easily target images with large damaged areas. You can also repair the image with damaged texture structure. Figure 2 shows a schematic diagram of Criminisi algorithm repair.
(a)
(b)
(c)
(d)
In the repair process, there are two problems: one is how to search for the sample block that matches the area to be repaired most, and the other is how to determine which area to be repaired is the first to be repaired, that is, to determine the order of repair.
When dealing with the first problem, most methods use image texture synthesis to search and match the texture that best matches the area to be repaired. In response to the second problem, the researchers adopted an image restoration method based on partial differential equations, which used the tangent direction of the isoilluminance line to spread the data, and successfully solved the sequence of the areas to be restored.
The sample-based repair method uses more information about the completed area of the image than the repair method based on partial differential equations and can well repair the large-area damaged image. For the image restoration of large damaged areas, the sample-based restoration method also has application scenarios. For example, when the target in the image needs to be cut out, the sample restoration method can be used for processing. However, this type of method also has shortcomings. For example, when the global search matching algorithm is used for matching, the local structure information of the image may be ignored, resulting in differences in the overall image after repair. In addition, the use of global search when matching image blocks will increase time complexity.
2.3. Image Restoration Method Based on Sparse Representation Theory
Sparse representation uses less basic information to linearly combine to express most or all of the original information. Among them, this basic information is selected from the overcomplete dictionary and is called atoms, while the overcomplete dictionary is made up of atoms with more than the signal dimension. Therefore, any signal has different sparse representations under different atomic combinations.
In order to reduce the complexity of the modified model, the only way to change the sample into a sparse representation is the process of dictionary learning. The expression of dictionary learning is as follows:
Among them, D is the dictionary matrix, xi is the i-th sample, λ is the regularization parameter, and is the sparse representation of xi.
Other greedy algorithms for sparse representation mostly improve matching pursuit and orthogonal matching pursuit, and they all have some common shortcomings; that is, the sparse optimization is not thorough, the effect is not good, and the signal cannot be sparsely compressed efficiently. However, their advantages are fast processing speed and high efficiency.
The relaxation algorithm of sparse representation is a solution proposed for NP-hard problems. Solving NP-hard problems is similar to solving linear programming, and it is also the focus of many researchers. Among the many relaxation algorithms, the better algorithm for optimizing the l1 norm is the Bregman iterative algorithm.
The L1 regularization problem is a classic topic in convex optimization. It is very computationally intensive and difficult to solve with traditional methods. The Bregman iterative algorithm has shown good results in solving the L1 norm optimization problem and reducing the difficulty of model solving.
Dictionary learning is a highly complex process. Through learning and training a large amount of data, a dictionary matrix containing similar target data is finally obtained. One of the hypotheses of dictionary learning is that the dictionary has a sparse representation for the specified signal. The idea is to learn the complex information of the natural world and possess the characteristics of this information, which can represent simple atoms of this complex information. Therefore, the principle of dictionary learning is to use sparse information as much as possible to express the original information. There are roughly two stages, namely, the learning stage and the presentation stage, namely, the dictionary generation stage and the sample sparse representation stage, as shown in Figure 3.
(a)
(b)
To construct an overcomplete dictionary, not only its adaptive effect must be considered, but also its redundancy and scale transformability. The traditional orthogonal basis is very sensitive to the information number, so the more dictionary primitives, the more accurately the characteristics of the signal can be represented. However, traditional dictionary learning methods are mostly selected from transform bases, and such methods cannot adapt to signals.
3. Image Restoration Algorithm with Block Structure Sparsity Constraint and Dictionary Atom Size Adaptive
3.1. The Basic Principle of the Algorithm
Image restoration based on the sparse representation restoration algorithm essentially treats the image blocks in the known information area as a set of bases and can perform sparse representation of the target block.
For the determination and update method of the repair priority function, people use isoilluminance lines to define the priority of the block to be repaired and cannot effectively distinguish the edge texture. During the repair process, the consistency of local texture information and structural information is maintained. There are shortcomings. For this reason, based on the sparse representation of samples, people use structural sparsity to determine the repair order of the blocks to be repaired, and keep the local texture information and structural information consistent. The structural sparsity is used to determine the priority of the block to be repaired, and the similar blocks of the block to be repaired are used for sparse representation, and the effect of repairing gray image and color image is better. However, only relying on sparsity to determine the repair priority of the block to be repaired cannot guarantee the priority of the edge, resulting in poor repair effect.
This type of method uses sparsity to determine the repair priority and does not guarantee the highest priority of the edge, and it will cause the repaired image to have texture extension or edge fracture. At the same time, when this kind of method is reconstructing, the atom size of the dictionary used is fixed. When repairing the smooth area, the dictionary atom will be too small to cause the extension of the area. When repairing the texture area, the dictionary atom will be too large, resulting in poor repair effect.
Therefore, for the abovementioned defects, this paper starts from the sparsity constraint priority and adaptive selection of dictionaries with different atom sizes and discusses a color image restoration algorithm with sparsity constraint and dictionary atom size adaptation. In this paper, the structural sparsity of the block to be repaired is used to adjust the repair priority to ensure the priority repair of the edge area. And by analyzing the structural information of the repair block located in different regions such as texture, edge, and smoothing, the size of the dictionary atom is adaptively determined to improve the repair effect of different structural regions.
3.2. Analysis and Improvement of Structure Sparse Propagation Mode
In the image restoration process based on sample blocks, because the damaged area contains not only structural information but also rich texture information, the phenomenon of texture extension is prone to occur in the restoration process. Therefore, the edge restoration of the image should take precedence over the restoration of the texture area. Therefore, in the image restoration process, priority should be introduced to determine the order of restoration of the blocks to be repaired. Since the point and line structure of the image is sparse and has low similarity to the neighborhood, the structure sparsity can be used to define the priority and determine the repair order of the blocks to be repaired.
In the definition of priority function, the degree of confidence indicates the amount of known information in the block to be repaired, which has a limited impact on its priority, and the structural sparsity determines the priority of the block to be repaired. The sparsity of the block structure can be used to better judge the repair priority of the block to be repaired.
In the experiment, this article found that although the block structure sparsity can be used to distinguish the smooth part well, in the texture and edge areas, using the block structure sparsity to determine the priority of the repair block will cause errors. Figure 4 shows 7 different areas to be repaired in two images. In each area to be repaired, the texture, edge, and smooth blocks to be repaired are taken, and then the structure sparsity of each block to be repaired is calculated. And their sizes are shown in Tables 1 and 2, respectively. Combining the analysis of Figure 4 and Tables 1 and 2, we can find the following.
(a)
(b)
First, the structure sparseness of the block to be repaired located in the smooth area is smaller than the structure sparseness of the block to be repaired located in the edge area or the texture area. Therefore, the structure sparsity can be used to determine whether the block to be repaired is located in a smooth area or in a texture area or edge area. Secondly, although the structure sparsity of most of the to-be-repaired blocks located in the edge area is greater than that of the to-be-repaired blocks located in the texture area, there is also a similar or even opposite conclusion; that is, the structure of the to-be-repaired blocks located in the texture area is similar. The structure sparsity is greater than the structure sparsity of the block to be repaired located in the edge area. Therefore, using the structure sparsity cannot accurately determine whether the block to be repaired is located in the edge area or in the texture area. Third, from the repair priority of the block to be repaired given in Table 2, it can be found that no matter in the area 5 and 6 in Figure 4(a), or in the area 2 and 5 in Figure 4(b), the repair priority of the to-be-repaired block in the texture area is greater than the repair priority of the to-be-repaired block located in the edge area, which breaks the image repair rule that the edge repair priority is higher than the texture repair priority, which will result in poor repair effect. Therefore, in order to overcome these shortcomings, this paper introduces an edge discrimination factor and redefines the priority function.
In order to overcome the shortcomings of only using the structure sparsity to judge whether it is an edge area or a texture area, this paper introduces an edge discrimination factor to judge it. Figure 5 divides the block to be repaired along the normal direction. The block to be repaired is divided into a block 1 and a block 2 of uniform size. You calculate the average values and of the known pixels of and , respectively. The discriminant factor is defined as
Among them, is the normalization parameter, generally 255.
The value of the discriminant factor of the edge area is much larger than the value of the texture area, which can easily distinguish the edge area from the texture area, which better overcomes the shortcomings of only using the structure sparsity to determine the existence.
In order to ensure that the restoration of the edge part takes precedence over the restoration of the texture, this paper combines the characteristics of the structure sparsity and the edge discrimination factor to redefine the priority function , and its size is
In the formula, is the confidence level, the value of depends on the area structure of the block to be repaired, and its size is determined by the following principles:(1)The value of can better distinguish the smooth area from the texture area. Therefore, a threshold can be set (empirical constant, generally 0.05). When the structure sparsity is less than the threshold, the repair block is located in the smooth area; otherwise, it is located in the edge texture area. In this way, when the repair block is located in a smooth area,(2)When distinguishing texture and edge area, you use the discriminant factor to make the discrimination; that is, set a threshold (empirical constant, generally 0.1). When the discrimination factor is less than the threshold, it indicates that the block to be repaired is located in the texture area. Otherwise, the block to be repaired is located in the edge area.
3.3. Adaptively Determine the Dictionary Atom Size
Because the edge area and texture area have more detailed information, and the small-scale dictionary atoms can capture the fine features of the image, it is conducive to the sparse reconstruction of the damaged edge and texture area. The detailed information of the smooth area is less integral, and the large-scale dictionary atoms can describe the overall characteristics of the image, which is more suitable for sparse reconstruction of the smooth area. However, the atomic size of the dictionary used by the traditional sparse algorithm is fixed. For this reason, the reconstruction of image blocks in different structure regions lacks adaptability, which will greatly affect the image restoration effect. Therefore, for the information reconstruction of different structural regions of the image, a better dictionary atom size should be selected adaptively according to the structural characteristics of the region to be repaired to meet the needs of repairing different structural regions.
Small-scale dictionary atoms are conducive to the restoration of edges and textures, while the dictionary of smooth parts of large-scale atoms has more advantages. The adaptive repair effect is better than the dictionary repair effect of fixed atom size. The effect of a dictionary with an atom size of 7 × 7 is better than that of 5 × 5 and 9 × 9, so too large or too small atoms will make the repair effect worse. Using larger dictionary atoms in simple texture regions can overcome the extension phenomenon caused by too small dictionary atoms. However, in complex texture areas and edge areas, smaller dictionary atoms are needed because of richer textures, which can avoid the blur of repair due to excessively large dictionary atoms. Therefore, only using a fixed-size dictionary cannot meet the needs of different structures in each region.
To this end, this article uses the difference in the structure of the area, where the block to be repaired is located, and adaptively selects the dictionary with the appropriate atom size, that is, combining the structure sparsity and the edge discrimination factor to adaptively select the dictionary atom size. The selection principle is as follows:(1)When the structural sparsity is less than its threshold, the block to be repaired is located in the smooth area. When reconstructing the block to be repaired, a dictionary with a size of 9 × 9 atoms is used.(2)When the structural sparsity is greater than the threshold and the discrimination factor is less than the threshold, the block to be repaired is located in a complex texture area. When reconstructing the block to be repaired, a dictionary with a size of 7 × 7 atoms is used.(3)When the structural sparsity is greater than the threshold and the discrimination factor is greater than the threshold, the block to be repaired is in the edge area. When reconstructing the block to be repaired, a dictionary with a size of 5 × 5 atoms is used.
Therefore, in the dictionary training, in order to improve the completeness of the dictionary, the dictionary training block is not only extracted from the image to be repaired but also extracted from a part of the natural image selected from the public image library, so as to use the K-SVD algorithm to train the dictionary. The atom sizes are 9 × 9, 7 × 7, and 5 × 5 dictionary.
In summary, the algorithm flow of this paper is shown in Figure 6. Specific steps are as follows: Step 1: you map the RGB image to the YUV color space. Step 2: you extract the edge of the part to be repaired. Step 3: you calculate the priority of all the blocks to be repaired at the edge, sort according to the priority size, and repair the block with the highest priority among the blocks to be repaired first. Step 4: according to the conditions of the area where the block to be repaired is located, you adaptively select a dictionary with a reasonable dictionary atom size, and reconstruct the block to be repaired. Step 5: you update edge information. Step 6: you repeat Steps 3–5 until the restoration of the image is completed.
4. Simulation Experiment and Analysis
4.1. Portrait or Target Removal Experiment
The first introduction is the experiment of portrait removal, as shown in Figure 7. In Figure 7, Figure 7(a) is the original image, Figure 7(b) is the repair result of the Criminisi algorithm, Figure 7(c) is the result of the adaptive repair algorithm, Figure 7(d) is the result of the sparsity repair algorithm, and Figure 7(e) is the result of this article. From the comparison of the repair results of the various algorithms in the figure, it can be seen that in the Criminisi algorithm, the repaired area is not necessarily consistent with the surrounding image blocks, such as the area marked with a green wire frame. The reason for this result has a lot to do with the imprecise matching criterion in the Criminisi algorithm. The result of Figure 7(c) is much better, because it uses the feature vector of the image block to perform matching repair, making the repair more perfect and accurate, reducing the mismatch between image blocks, but the area marked in green is vague. There are also some mismatches in the map repair results. Because the improved algorithm in this paper uses the value of the block structure sparsity to calculate the priority value, the block structure sparsity can reflect the regional characteristics of the image, so the texture structure and other information can be repaired in a targeted manner, and the color histogram is introduced in the matching criterion, further improving the accuracy of matching. Therefore, in the result of the algorithm restoration in this paper, the green area and the surrounding image look more harmonious and natural.
(a)
(b)
(c)
(d)
(e)
Figure 8 shows an experimental analysis of the removal of obstructed objects in the image. It can be seen from Figure 8 that the target object to be removed is the diamond-shaped area in the image, and the environment around the diamond-shaped area is relatively simple, the image level is relatively simple, and the color is relatively light. From the repaired results, the effect of Figure 8(e) is the worst, and there is no mismatch information, as shown in the area marked by the blue box in Figure 8(e). This is because the priority calculation method in the Criminisi algorithm is unreasonable. This calculation method will lead to an uncertain filling order, which matches the wrong information. The sparsity repair algorithm and the algorithm in this paper are based on the sparsity of the block structure, and this paper chooses a more reasonable priority calculation method, so the repair effect is more ideal.
(a)
(b)
(c)
(d)
(e)
4.2. Subtitle Removal Experiment
The character removal experiment shows that the algorithm in this paper is more effective in removing the repair of larger targets. In addition, this method can still be used to repair some images with relatively small damaged areas, such as subtitles on movie screens or text and letters appearing in images. The experiments in this section illustrate these applications.
Figure 9 shows the removal of an English letter. The application in this area is still very extensive, including the removal of various news and newspaper image fonts and the removal of movie screen subtitles. It can be seen from the images in Figures 9(c)–9(e) that the repair results of these algorithms are almost the same. First of all, the background of this image is relatively simple, and the color is relatively light. As a result of the repair, the repaired area blends well with the surrounding environment, looks natural and smooth, and there is no seam effect or traces of letters.
(a)
(b)
(c)
(d)
(e)
Figure 10 shows another experiment of letter removal. The selected image is a brightly colored parrot. The texture details of the image are richer, and the structure is relatively simple. Judging from the results of the algorithm’s restoration, the effect is relatively satisfactory. The four algorithms have been successful in restoring texture details. The adaptive restoration takes the feature vector of the image block as the measure of similarity, so the texture is more considered when repairing. The algorithm in this paper uses the color histogram to optimize the matching items and also maintains the texture and color information well.
(a)
(b)
(c)
(d)
(e)
4.3. Scratch or Spot Repair Experiment
The image restoration technology based on sample blocks can not only be used for the removal of target objects in portraits and subtitles, but also can be used for the restoration of image scratches or spots like methods based on partial differential equations.
Figure 11 shows the repair of the scratches on the picture, and the scratches are relatively few. In fact, this type of image restoration can be based on partial differential equations. From the results of the repair, the scratches were repaired very cleanly, and the overall picture looks very natural and coordinated. If you observe some details, you will find that some blur effects appear in the adaptive repair algorithm. However, there is no blur effect in the repaired result of the Criminisi algorithm. The repair result of the sparsity repair algorithm is the most ideal, because this algorithm uses a smaller sample block window size to keep the texture detail information better. The algorithm in this paper also has some slight dents, but it is lighter than the Criminisi algorithm, so the effect looks more ideal. On the whole, the algorithm repair effect in this paper is the most significant.
(a)
(b)
(c)
(d)
(e)
Figure 12 shows to repair an image that has lost part of its information to maintain its integrity in structure and content. It can be seen from Figure 12(b)–12(e) that the edges and structure are kept relatively intact, and the edge curves are relatively natural. It does not seem to be much repaired. The results of the algorithm repair in this article are quite satisfactory.
(a)
(b)
(c)
(d)
(e)
4.4. Distance Function Parameter Test
We add damage information to the test image, and then set to take different values in the range of 0 to 1 to repair the damaged image, and record the PSNR value of the corresponding repair result. Figure 13 shows a simulation diagram of the PSNR value changing with .
(a)
(b)
It can be seen from Figure 13(a) that when is increased appropriately, both color information and feature information play a role, causing the matching result to fluctuate up and down. It can be seen from Figure 13(a) that the corresponding value is different when the highest PSNR value is obtained for each image, but when the value of is between 0.01 and 0.03, a higher PSNR can be obtained. It can be seen from Figure 13(b) that when the value of is 0.02, the average PSNR value obtained is the highest, so we set to 0.02.
4.5. Priority Parameter Test
The following tests the specific value of the weighting factor in the priority function α. Since there is a relationship between between the two parameters, only one parameter value needs to be changed. Here, let take different values between 0 and 1 to repair the damaged image, and record the PSNR value of the repair result. Figure 14 shows a simulation diagram of the PSNR of the repair result changing with .
(a)
(b)
It can be seen from Figure 14 that the PSNR value of the test image restoration result gradually increases with the value of α. This is because the two parts of the priority function slowly reach a balance at this time. The included known information can play a better role. However, with the excessive increase of , the PSNR of the test image restoration result will show a downward trend. At this time, the proportion of the two-dimensional information entropy in the priority function is small, and the structural information contained in the image block itself cannot be fully utilized. It can be seen from Figure 14 that when the value of is 0.05, the average PSNR of the test image can reach the highest, so this article sets the value of to 0.05.
4.6. Analysis of Objective Indicators of Experimental Results
This section mainly analyzes the above groups of experiments from some objective indicators, such as repair time, SSIM, and PSNR. Figure 15 shows a comparison of the repair time of each algorithm.
It can be seen from Figure 15 that the Criminisi algorithm takes longer to repair. This is because in its algorithm, the priority calculation method is unreasonable. The search for the optimal matching block uses a global search strategy. This step increases time to search. In the algorithm in this paper, the local search strategy is used, and it can be calculated that the image repair time is reduced by up to 55 seconds compared with the Criminisi algorithm. Therefore, for image restoration, the time cost of this algorithm is reduced a lot.
Figure 16 shows a comparison of the PSNR of each algorithm. It can be seen from Figure 16 that the PSNR of the other two algorithms is generally higher than that of the Criminisi algorithm, and the peak signal-to-noise ratio of the image restoration result of the adaptive image restoration algorithm with block structure sparsity in this paper is the highest.
Figure 17 shows the SSIM comparison of each repair algorithm. The larger the SSIM value, the higher the similarity between the two images, and the maximum value is 1. It can be seen that the adaptive repair algorithm and sparsity repair algorithm have improved structural similarity compared with the Criminisi algorithm. The structure similarity of the adaptive image inpainting algorithm for block structure sparsity proposed in this paper is the highest.
5. Conclusion
In the context of the continuous development of computers and digital technology, digital image processing has slowly entered our lives. Images are an important way for us to obtain information. There is a lot of information in life that cannot be clearly expressed by language alone, but through an image, it can be clear at a glance, showing the importance of the image. However, in the fields of life and scientific research, many images will be damaged to varying degrees due to various reasons, which seriously affects the appreciation and use value. The above problems have seriously affected the readability and usability of images. In order to reproduce the complete information of the image or remove some unwanted objects, digital image restoration technology has emerged at the right time and has been continuously developed and matured, and it has also been reflected in this period.
Since the use of sparsity to determine the repair priority will lead to the extension of the repaired image texture, the use of dictionary reconstruction with a fixed atomic size will lead to the image repair with blurred edges and smooth area extension. This paper proposes a color map with sparsity constraint and dictionary atom size adaptation. The algorithm uses the structure sparsity of the block to be repaired to adjust the repair priority to ensure priority repair of the edge; by analyzing the structure information of the repair block in different regions such as texture, edge, and smoothing, it adaptively determines the size of the dictionary atom, which is effective. To overcome the shortcomings of blurring of details and region extension in fixed dictionary repair, this paper proposes an improved algorithm based on block structure sparsity. We use the block structure sparsity to calculate the priority of the sample blocks to be repaired, so that the repair order is more reasonable; we select the size of the sample block according to the size of the block structure sparsity, which improves the flexibility of repair; introduces color histograms when matching the sample blocks. The graph makes the matching more accurate; finally, the local search strategy is used to replace the global search strategy, which reduces the time cost. Simulation experiments show that the repair effect of the algorithm discussed in this article has been significantly improved. Since the reconstructed image block is an approximation of the original image block, and the algorithm is also greedy, in the restoration of large-area object removal, the priority restoration image block will affect the subsequent restoration image block. Experimental results show that the algorithm in this paper overcomes the shortcomings of Criminisi algorithm. It has a good repair effect for different degrees of damaged images. In addition, compared with the Criminisi algorithm, the algorithm in this paper has reduced time and improved peak signal-to-noise ratio and SSIM value.
The algorithm in this paper will produce serious blocking and seam effects when repairing random texture areas. At the same time, if there are curved edges in the damaged area, the arc-shaped structures will be connected by straight lines, making the repair traces very obvious. How to construct a more reasonable repair model is also a problem worthy of in-depth study. The current repair algorithm based on sample blocks always selects the relatively best matching block. If there is no block in the current search area that has a similar texture structure, the algorithm has to select a block from it for filling, which is obviously unreasonable. For such problems, consider whether the current block to be repaired can be put on hold through certain restrictions, and the second priority block is selected for repair first. In short, there is no definite solution to the problem of image restoration, so it has serious morbidity. This article only discusses and studies some of the problems. There are still many problems to be solved, and more and more in-depth research and analysis are needed in the future.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest regarding the publication of this paper.
Acknowledgments
The study was supported by Shaanxi Provincial Social Science Fund, beautiful countryside construction in study on the protection and upgrading of local landscape along Qinling in Shaanxi Province, project no. 2018K22, and Shaanxi Provincial Education Department Project, s study on the precise poverty alleviation of regional culture based on public art education in colleges and universities scientific research, project no. 19JK0483.