Abstract
Mask of damage region is the pretreatment step of the image inpainting, which plays a key role in the ultimate effect. However, state-of-the-art methods have attached significance to the inpainting model, and the mask of damage region is usually selected manually or by the conventional threshold-based method. Since manual method is time-consuming and the threshold-based method does not have the same precision for different images, we herein report a new method for automatically constructing the precise mask by the joint filtering of guided filtering and smoothing. It can accurately locate the boundary of damaged region in order to effectively segment the damage region and then greatly improves the ultimate effect of image inpainting. The experimental results show that the proposed method is superior to state-of-the-art methods in the step of constructing inpainting mask, especially for the damaged region with inconspicuous boundary.
1. Introduction
Image inpainting as a technique of restoring missing or damaged areas of an image in an undetectable form for observers has drawn considerable attention over recent years due to the demand for revisiting damaged works of paintings or photography to make them more legible [1]. Remote sensing images are generally degraded by motion blur, the random noise, and the cloud that becomes a popular topic in remote sensing field [2]. Image inpainting can obtain the high quality image from degraded image and it can inpaint the random noise region. The noise-like white or off-white region can be located and further inpainted using the CDD or other inpainting models in this paper. The fundamental process of image inpainting is to construct a mask to locate the boundary of damaged region followed by subsequent inpainting process. State-of-the-art methods have attached significance to the inpainting model, and the mask of damage region is usually selected manually or by the conventional threshold-based method. Since manual method is time-consuming and the threshold-based method does not have the same precision for various images, therefore, it is of vital significance to automatically construct a precise mask for the damaged region of an image in order to fully restore the missing message, which is the aim of this paper.
To date, the numerous approaches [3–8] have been proposed for image inpainting since the seminal work of BSCB [3, 4]: for example, the classical Total Variation (TV) [7] and Curvature-Driven Diffusion (CDD) [8] models, which are based on solving Partial Differential Equation (PDE) to preserve the directions of the isophotes in the missing region. In practice, it is assumed that the basement layer is white and off-white photo letter or wall, so it is with the damaged region. Then a certain range of white area is taken as the mask based on the threshold which applies to the threshold of the pixels in the image to be inpainted.
However, the boundary between damaged and undamaged regions is usually inconspicuous, which may result in two main drawbacks using the threshold method: some pixels belonging to damaged region may be lost when the threshold is high, which leads to the occurrence of bleached inpainting region; some bright pixels in undamaged region are probably treated as the region which needs to be inpainted when the threshold is low. Therefore, the validity of conventional threshold method is strongly restricted; that is, it works on condition that the boundary of damaged region is conspicuous. In this work, we introduce a new method to construct a more precise mask based on guided filtering and smoothing in order to preciously locate the boundary of damaged region. Experimental results demonstrate that our method shows superiority over the conventional method in image inpainting especially for the images with inconspicuous boundary of damaged region.
2. Related Work
Two main steps comprise the conventional method for image inpainting.
Step 1. The construction of mask image: In detail, setting two values—255 and 0—to represent absolutely white and black, respectively. Supposing that the inpainting region approaches white, when each RGB value of pixels is higher than the limit value, the pixels are labeled as damaged pixels, and then the values of corresponding pixels in mask are assigned to 255. Otherwise, the values are assigned to 0. The principle of constructing mask with threshold method is shown in Figure 1, where and denote the damaged image and the mask, respectively.
Step 2. After the construction of mask image, the damaged region is inpainted iteratively by CDD model with the mask above. CDD inpainting is like the process of heat diffusion with the diffusion intensity depending on both the gradient and curvature. The diffusion intensity is high when the curvature is large and decreases along with the isophote furtherly smoothing. Hence, the CDD model can eliminate the large curvature and stabilize the small curvature.
3. The Proposed Method
As mentioned above, it is difficult to build mask when the boundary of damaged region is inconspicuous. Inspired by dark channel prior [9], a new method is proposed to deal with this issue. The definition of dark channel is described aswhere is a color channel of and is a local patch centered at . is the corresponding dark channel image. The size of is in this paper. As the natural outdoor images are usually colorful and full of shadows, the dark channel of these images is really dark. The bright parts in these images have very high intensity in each RGB channel, so that the brightest point in the dark channel image is usually close to white.
Since the damaged region (like noise or cloud region) in an image is generally white, it is assumed that the damaged region approaches white and its boundary is inconspicuous. Resultantly, the brightest pixel is considered as the most obviously damaged pixel. In addition, the colors of the damage region and its undamaged adjacent region are obviously different, which means that the damage region and undamaged region can be identified by our eyes. The most obvious damage pixel is labeled as reference point . The pixels whose RGB channel is approximately balanced may be damaged pixels. Therefore, the pixels are selected as damaged pixels according to their balanced degree of RGB and approaching degree to reference point .
The method mentioned above is sufficient for building mask when the boundary of damaged region is clear. However, it fails to work well when the boundary is inconspicuous. Motivated by this challenge, we here propose a new method to build precise mask for the boundary of damaged region based on guided filtering [10] and smoothing [11, 12]. In brief, guided filtering is employed to enhance image details, and then smoothing is used to sharpen major edges and eliminate low-amplitude structures. These two processes act as the first two steps of our proposed method as illustrated in Figure 2 and the theoretical details are provided as follows.
In [10], it is assumed that the output image is linearly related to the guide image in a local place in guide filter; then the output image in window with radius centered at pixel can be expressed as the linear coefficient are constant in . This linear model ensures that and have the same edges due to .
To determine the linear coefficient , the filtering input is constricted, and the difference between output and input is considered to be noise, expressed as and then the linear coefficient can be solved by minimizing the cost function:where is a regularization parameter. The guided filter behaves as an edge-preserving smoothing operator when and are identical. Under this circumstance, the filtered information is the difference between and , which is expressed as . This information represents the detailed information of and is used for the following enhancement process.
The enhancement via guided filtering is written aswhere is the result of the enhancement through guided filtering and is the enhancement factor. As seen from (4), increases with the increase of , meaning that the detailed information and therefore the boundary of damaged region will be enhanced as is high; otherwise, the enhancement will be weak.
smoothing is used to suppress low-amplitude details and sharpen the salient boundary. smoothing is realized by where is the input image and the gradient of pixel in the output image is expressed as . Here and are the gradients at the directions of and , respectively. is defined as a counter to count the number of the pixels whose gradient is nonzero; that is, . The aim of minimization is to sharpen the major boundary, so that the boundary of damaged region can be preserved in and then found in the gradient of .
Figure 2 shows the sketch of our method and the steps are listed below.
Step 1. The damaged image is enhanced by guided filtering, and the result is represented by .
Step 2. is smoothed by smoothing, and the result is denoted by .
Step 3. The interim mask is built by RGB balance and approaching degree of from .
Step 4. The mask is built by and the gradient map of .
Step 5. is represented by .
Firstly, the original damaged image is enhanced by (4), and the enhancement of is represented as . The smoothed result of by smoothing is denoted as .
The brightest pixel in the dark channel of is regarded as the reference point mentioned above. Regarding the construction of mask the pixels are treated as damaged pixels when the following two conditions are satisfied: the pixels’ RGB approaching degree does not exceed the given coefficient ; the differences between RGB average of and the pixels’ RGB average do not exceed coefficient . Otherwise, the pixels will be treated as undamaged pixels. As stated in Section 2, in our method the pixels of damaged region are all set to 255 and the rest are set to 0, and we thus construct a new grayscale image denoted as mask . Generally, the white region in is mainly the inner region of damaged region; the other pixels especially the ones at the boundary in the damaged region are probably not included in . Therefore, acts as an interim mask and will be utilized to construct more precise masks in the following steps.
The next step is to locate the boundary of damaged region more precisely using guided filtering enhancement and smoothing. Then a more precise mask instead of is needed, which is constructed in the following way. We employ a small patch through which contains inner damaged region to match the gradient map of and then denote the included boundary of damaged region as mask . Likewise, the boundary is set to 255 and the rest is assigned to 0 in . Based on this, we can obtain a more precise mask defined as . Note that the size of the patch here should not be large, or the boundary will be too thick.
4. Experimental Results
In this section, we further test the performance of the proposed method using Matlab2011b in a personal computer with an Intel Core i7 processor. We collect the tested images from http://imagedatabase.cs.washington.edu/. In this paper, the damaged regions like white text and curves in original images are manually made and superimposed on undamaged images with Photoshop.
We compared the proposed method with the traditional methods, and the results are shown in Figure 3. Figures 3(a) and 3(b) are the undamaged and damaged image, respectively. Figures 3(c) and 3(d) illustrate the inpainting regions (the orange regions) found by our method and threshold method, respectively. As can be seen from Figure 3(d), some undamaged regions (e.g., mountains indicated by the red arrow) are wrongly selected to be inpainted by using the threshold method. In comparison, fewer undamaged regions are found with the proposed method as indicated by Figure 3(d).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figures 3(e) and 3(f) show the inpainting results using the proposed and threshold methods with both basing on CDD model, respectively. The zoom-in views of the red rectangle patches in Figures 3(e) and 3(f) are shown in Figures 3(g) and 3(h), respectively. Obviously, some damaged regions are not inpainted using the threshold method as indicated by the red arrows in Figure 3(g). In contrast, our method is superior to the threshold method especially for the images with inconspicuous boundary of damaged region.
5. Conclusion
In this paper, we report a new method for building a precise mask of damaged region for image inpainting. With the precise mask and the combinations of the guided filtering enhancement and smoothing, our method exhibits satisfactory performance in image inpainting and shows superiority over conventional methods for images with inconspicuous boundary of damaged region. It is also interesting to apply our method to the inpainting of images colorful damaged region; meanwhile, a precise cloud region can be precisely detected and acquired as a byproduct of image inpainting. We leave this problem for future research.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant 61503300, Natural Science Foundation of Shaanxi Province of China under Grant 2014JQ8327, China Postdoctoral Science Foundation under Grant 2014M560801, Scientific Research Program Funded by Shaanxi Provincial Education Department under Grant 14JK1750, Science Foundation of Northwest University under Grant 13NW40, and Foundation of Key Laboratory of Space Active Opto-Electronics Technology of Chinese Academy of Sciences under Grant AOE-2016-A02.