Abstract
Defogging algorithms based on dark channel prior have color shift in light color areas because of inaccurate estimation of transmittance. To resolve this problem, a novel improved image clearness method is proposed. Based on the dark channel prior, the essential causes of color shift are analyzed, with two important factors summarized. Then, transmission map is calculated by using 33 fixed region, and the restoration module based on self-adaptive threshold mechanism for transmission map is provided. Some experiments are carried out to determine parameters in restoration module to correct the transmission map. According to the corrected transmission map, a transmission restoration algorithm is constructed based on the self-adaptive threshold mechanism to improve the performance of the fog-free image. The experiment results show that this method can resolve the color shift in light color areas effectively and guarantee the overall framework of defogging method based on dark color theory unchanged.
1. Introduction
In fog weather, due to the influence of atmospheric scattering, image taken by outdoor surveillance system would get serious degradation problems in the color and contrast fidelity. It not only directly affects the safety of sea, land, and air transportation by making the outdoor surveillance system abnormal [1, 2], but also hinders images feature extraction, which causes some monitoring systems based on feature extraction ineffective, such as production monitoring system [3, 4].
Currently, most of image defogging algorithms are based on image restoration, the core idea of which is as follows: firstly, an imaging mode should be established; secondly, the degraded part of the imaging model is compensated and the interferential part of it is filtered; thirdly, the clear image is restored [5–17]. Theoretically, the defogging effect of these algorithms can be ideal; however, most of the existing imaging models are dependent on the image depth information; unfortunately, they cannot be accurately calculated using a single image. Therefore, how to obtain the accurate image depth information is a bottleneck in image restoration field.
In 2009, Professor Kaiming He proposed a dark channel prior (DCP) theory on CVPR conference [8]. According to his theory, image depth information can be accurately acquired, which makes breakthrough progress in image restoration field. The defogging algorithm based on DCP is simple and effective; however, the inaccurate estimates of transmission would lead to distortion of the image, such as halo artifacts and overly enhanced restoration in light color areas, and also the optimization algorithm of transmission has high spatial complexity. Therefore, some improved methods have been proposed [9–17] based on DCP. Li et al. [11] proposed an edge-preserving decomposition method to estimate transmission map. This method removes haze effectively and overcomes halo artifacts. However, the result of dehazing partly depends on the accuracy of haze level estimation and the algorithm cannot do that at present. Song et al. [12] estimate the transmission by using small patch size dark channel (DC), and several gray images as guided filter (GF) are used to optimize the transmission. This method has excellent performance in edge maintenance and halo reduction; nevertheless, the dehazing effect of it is not so outstanding because of the limitation of DC size. Yang et al. [13] proposed a edge of alternative method to refine transmission map, which could overcome halo artifacts effectively. Chen et al. [14] proposed a dehazing-parameter adaptive method to adjust the degree of haze removal, and an iteration method based on GF is used to optimize transmission. Experimental results show that this algorithm performs well in different levels of haze conditions. However, the optimization method of transmission is replaced by an approximate scheme to reduce high computation.
Other methods are proposed and also have an outstanding effect on dehazing. Zhu et al. [18] exploited the characteristics of brightness and saturation of the pixels in hazy image, proposing a new linear model and learning the parameters of the model by using a supervised learning method to estimate the depth information. Their experimental results indicated that dehazing effects are good and efficient, but the insufficient estimation of transmission map is still an unsolved problem.
Based on DCP, this paper proposes an improved algorithm combined with self-adaptive threshold mechanism (STM), which enhances the transmittance in light color area. The remaining parts of this paper are arranged as follows: in Section 2, atmospheric scattering model will be introduced briefly; in Section 3, the original defogging algorithm based on dark channel prior theory will be described in detail; Section 4 performs a detailed analysis of the color shift problem and construct an improved algorithm combined with self-adaptive threshold mechanism (STM). In Section 5, the comparison of the experiment results between our defogging algorithm and other algorithms will be used to illustrate the validity of the improved algorithm. And then, Section 6 summarizes our work and discusses the future direction.
2. Introduction of Atmospheric Scattering Model
In image restoration filed, the following mathematical model is widely used to describe the imaging process of fogging image:
where is the fogging image light intensity acquired by visual system (i.e., input image); presents the light intensity reflected at the surface of the scene (i.e., needed fog-free image); is used to mark the two-dimensional location of the pixels in image; and describes the atmospheric light coming from infinity (skylight), which is usually assumed as a global constant. In other words, there is no relationship between location parameter and . denotes the transmission map of the image, which reflects the depth information of the scene, and it can be expressed as follows:where is scattering coefficient of the atmosphere and describes the distance between the target scene and the observer. In general, the atmosphere is uniform.
Formula (1) is widely used in image defogging field [8–15] at present, which describes the imaging process of fogging image and gives us the tip about the core idea of image defogging algorithm, i.e., eliminating the atmospheric light participated in imaging, and compensating for the lost reflected light of scene caused by atmospheric scattering.
3. Imaging Defogging Algorithm Based on Dark Channel Prior Theory
3.1. Basic Concept of Dark Channel Prior Theory
After observing large numbers of outdoor fog-free images, empirical statistical regularity called dark channel prior theory is proposed by Professor Kaiming He in [8]. It points out that any local regions of vast majority outdoor clear images contain some pixels, which has very low intensity values in at least one color channel. This intensity value that belongs to local area is called dark channel value and we can use the following formula to calculate the dark channel values:
where represents one color channel of image and is local area of image, which is centered on meaning two-dimensional position. Generally, these dark channel values always exist in object shadow, dark object, and object with bright colors.
3.2. Image Defogging Algorithm Based on Dark Channel Prior Theory
Due to the fact that dark channel values of the fogging image are always close to zero in fog weather, the dark channel values obtain certain brightness at the process of imaging. Notice that the dark channel values may be calculated at both sides of (1), and then the first term at the right of (1) is set to zero. Thus, formula (1) could be rewritten as follows:
According to (4), the image depth information can be expressed as follows formula.
There are two unknown parameters in formula (5): dark channel values which can be calculated by using formula (3) and atmospheric light A. Unfortunately, we do not know how to acquire atmospheric light A. To solve the problem, Professor Kaiming He sorted these dark channel values and extracted the 10% brightest dark channel values and then took the mean of these 10% brightest dark channel values as the value of atmospheric light A in [8].
In fact, atmospheric scattering exists even in a cloudless day. Particles are suspended in air, so the vague area of image will appear when we observe distant objects. On the other hand, images have depth because of atmospheric scattering. If we remove atmospheric light thoroughly, the restoring image may tend to look fake and unnatural. For solving the problem, Professor He introduces parameter to keep a small amount of atmospheric light for distant objects in [8]. And formula (5) is adapted as
Meanwhile, the defogging model of images may get combined with imaging model (1).
Therefore, once the transmission map is calculated using formula (6), the clear fog-free image can be restored. Notice that the clear fog-free image restored by defogging algorithm based on dark channel prior theory is always darker than atmospheric light, so the parameter is introduced to enhance and increase the intensity of the image in [8]. Therefore, formula (7) may be adapted as follows:
In practice sense, the value of is set according to the requirements of real application. For example, the value of has been set as 0.1 in [8].
3.3. Insufficiency of Defogging Algorithm Based on Dark Channel Prior Theory
Generally speaking, if the fogging images acquired by outdoor surveillance system contain no obvious light color areas, they can obtain a satisfactory defogging effect by defogging algorithm based on dark channel prior theory. This is because the vast majority of the pixels in these images meet dark channel prior theory; i.e., there has to be at least one color channel whose value is close to 0 among these pixels. For example, the original fogging images are shown as Figure 1 and the defogging images processed by dark channel prior theory are shown as Figure 2.
In a real world situation, however, some fogging images may always contain obvious light color areas shown as Figure 3.
(a)
(b)
(c)
In this case, if we use the defogging method based on dark channel prior theory to restore these fogging images, the color shift problem should appear in light color areas. This is due to the fact that the pixel values in three color channels are very high for all pixels in the light color area, and then the defogging algorithm based on dark channel prior theory is invalid in these light color areas. It is illustrated as Figure 4.
Notice that, in Figure 4, the restored images look too saturated when we calculate the dark channel values by using 33 fixed region. In this paper, our main purpose is how to restore the invalid image caused by the dark channel prior theory in these light areas.
To solve the color shift problem, we will first analyze the characteristics of original fogging image and restored image and then figure out the essential cause of image color shift problem. Based on this, a repaired model of inaccurate transmission map in light color is constructed directionally.
4. Improved Image Defogging Algorithm Based on Dark Channel Prior Theory and Self-Adaptive Threshold Mechanism
4.1. Cause Analysis of Color Shift in Light Color Areas
In order to analyze the characteristics of original fogging image and restored image, find out the essential cause of image color shift problem; the Histograms of dark channel values between images containing obvious light color areas and images containing no obvious light areas are shown in Figures 2 and 4. The corresponding statistical comparison results are illustrated in Figures 5 and 6.
From Figure 5, it can be found that the histogram of dark channel values of fogging images which contain no obvious light color areas is darker and its statistical values are close to zero as a whole. In contrast, we can find from Figure 6 that some inconsistent larger dark channel values exist in the images that contain obvious light color areas. This indicated that the dark channel value may be inexistent in these light color areas; i.e., the dark channel prior theory has failed in these areas of image.
However, how do these larger incorrect dark channel values affect the final defogging effect? For answering this question, we rewrite formula (7) according to formula (6) and (8) as follows:
where and denote restored fog-free image and original fogging image, respectively. As every image has three color channels as , , or , formula (9) may also be remarked as follows:where c presents color channels of image. Obviously, three color channels of image can be marked as , , and, analogously, and those of original image can be described as , , and .
Notice that the dark channel value is usually larger in light color area, which will result in transmittance tending to be smaller directly. In this fact, even if the pixel values of three color channels are quite close, their difference will become larger while they are divided by a small transmittance value, which can lead to significant change of color values of these pixels essentially. At this point, the color shift happens.
To better analyze the problem, the maximum difference of three channels between original fogging images and restored free-fog images by dark channel prior theory is experimented. Figure 7 describes three color channels’ maximum difference of each pixel with obvious light color areas.
It can be clearly found that vast majority of three color channels’ maximum difference values of pixels have to be increased several times, which brings a great change for the original color of these pixels, after using defogging algorithm based on dark channel prior theory. To precisely locate these pixels in primary images, the three color channels’ maximum difference values are presented in image form shown as Figure 8.
Obviously, the distribution of these maximum difference values and that of light color areas are coincidental. This indicates that the conclusion is reasonable and fit in with facts. Next, we need to find a method to solve these types of problems.
4.2. Transmittance Restored Algorithm Based on Self-Adaptive Threshold Mechanism
According to the analysis in previous section, the essential reason of color shift problem in light color areas is the incorrect smaller transmittances generated by the larger dark channel values that come from incorrect calculations. Therefore, the original defogging algorithm can be improved by optimizing these incorrect smaller transmittances. Firstly, we assume that the optimized transmittance and primary transmittance satisfy the following relationship:
In other words, the transmittance can be optimized in light color areas by multiplying a weight coefficient and can be kept in other areas. The parameter is used to enhance the intensity of the image in defogging algorithm based on dark channel prior theory. Therefore, the key of improving defogging algorithm is how to locate light color area and estimate parameter .
To improve the effect, Professor Jiang proposed an improvement model to calculate the parameter and improve original transmittance in [10]. They assumed that all pixel values in three color channels are close to atmospheric light in light color area. Based on the assumption, the improved formula of transmittance is set as follows:For simplicity, let in [10].
Unfortunately, according to the analysis of the pixels in light color area in Figures 7 and 8, the pixels in light color area have another characteristic yet; i.e., the maximum difference values of three color channels are smaller which can be found by the red curve in Figure 7. These indicate that the following two characteristics are true in practice:
(1) The maximum difference values of three color channels are smaller, which can be found by the red curve in Figure 7.
(2) All pixel values in three color channels are close to atmospheric light.
Obviously, Jiang only considered second feature of the light color area, so formula (12) cannot satisfy the dynamical need for restoring image and the algorithm has poor robustness.
To better improve the effect of restored images and meet the dynamical need, the expression of could be assumed combining the above two characteristics as follows: where
indicates the two-dimension coordinate of the pixels in image;
presents the difference values of three color channel in original fogging image;
denotes the value of pixel of the original fogging image in color channel;
can be selected as , , .
In reality, is calculated as the value located at the 99% position in ascending order of because of the existence of noise.
According to the analysis above, the following formula can be used to optimize the primary transmittance:
In addition, there have been cases where the primary fogging image is dim on the whole. Meanwhile, the pixel values in different color channels are close; i.e., the vast majority of three color channels’ maximum difference values of pixels are close to zero. However, the light color areas still exist relatively. In this case, the weight coefficient computed by formula (13) is too small that the improved effect of transmittance cannot be achieved. Therefore, the following method is proposed to solve the problem brought by the situation.
The concrete processing steps are as follows.
(1) Locating this situation. Observing the statistical result, it is found that which presents the mean of is small and most of values in are smaller than . Thus, the characteristic could be described using the following model:
where is a self-adaptive fault-tolerant threshold. In particular, experiments have shown that when threshold value is set to be 0.1, almost this situation can be successfully located. is the number of pixels whose three color channels’ maximum difference values are less than . And presents the number of pixels whose three color channels’ maximum difference values are greater than .
(2) Recalculation for weight coefficient . Considering the fact that even though the vast majority of color channels’ maximum difference values are close to zero, the differences among them exist still. Therefore, the original can be processed using linear stretch algorithm to increase these differences:
where , represent the maximum value and minimum value of , respectively
As a result, it not only retains the relative difference of color channels’ maximum difference values, but also increases their absolute difference values. Then we could get a new maximum value by using the linear stretch formula (16).
To guarantee the improvement effect of transmittance, the self-adaptive maximum value model of is constructed combining the primary maximum value and linear stretch maximum .
Based on this information, a final could be calculated again by using the stretch difference value and the original difference value . Further, the weight coefficient could be gotten by formula (13) and formula (17).
5. Experimental Simulation and Analysis
For verifying the availability of our algorithm, in this section, we will compare it with experimental results of DCP algorithm and Improved DCP algorithm in two ways: subjective visual evaluation and objective quality of defogging images. Here the original images containing large light color areas, which are shown as Figure 3, are used as the experimental images.
5.1. Subjective Visual Evaluation
The fogging images are restored by using DCP algorithm shown as Figure 4; serious color shift problem exists in light color areas. Improved DCP algorithm and STM algorithm that we proposed could improve the insufficiency well. Two different visual results are shown as Figure 9.
(a) Clearness images obtained by Improved DCP algorithm
(b) Clearness images obtained by STM algorithm
According to formula (12), the improvement effect of transmittance of Improved DCP algorithm relies mainly on the selection of . In actual engineering, if these images have different characteristics, should be selected as different value. However, [10] overlooked the problem and had been selected as a constant value 50. We can find out from Figure 9(a) that although the algorithm proposed by Professor Jiang solved the color shift problem, it has poor robustness. This is mainly because the different constant value may result in different optimized effects for different images.
In comparison, the algorithm proposed in this paper prevents artificial arbitrariness of selection for and changes adaptively weight coefficient according to color channels’ maximum difference values. The defogging images shown as Figure 9(b), which are restored by our algorithm, would have more robustness.
5.2. Objective Evaluation
Considering the limitation of subjective evaluation coming from human visual system, here, we introduce image’s standard deviation that can reflect the contrast of an image to evaluate the quality of restored images defogging by different algorithms. The computing model is as follows:where indicates gray value of different pixel and represents the probability of in images.
Meanwhile, the information entropy, which represents the amount of information in an image, is used to evaluate the quality of restored images defogging by different algorithms. And the computing model is as follows:where and are the same as described in formula (18). The symbol denotes the statistical mean of gray value of the whole image. The formula is shown as follows:
Normally, if the size of image block is too large, it will result in overly enhanced restoration and then the following problems of restored images arise: image is so saturated that some image details are lost. In addition, our goal is to improve the insufficiency of defogging method based on dark channel prior theory used in light color areas, so a 33 image block is used to calculate the dark channel values for better comparison.
It is generally known that a fogging image always has a low contrast and less depth information. Therefore, the better the algorithm is, the bigger the standard deviation and entropy of the defogging image are. Thus, the performance comparison of quality of defogging images, restored by three algorithms, is listed in Table 1 from (18)-(20).
Observing the above table, we can find that all the algorithms can increase the standard deviation and entropy of the images, i.e., improving the objective quality of images. STM algorithm makes more contribution for standard deviation and entropy of the images, when compared with DCP and Improved DCP algorithm. These results in Table 1 show that STM algorithm/model is effective.
6. Conclusion
In this paper, a new improved defogging algorithm based on dark color theory is presented combined with self-adaptive threshold mechanism. First we make a detailed analysis and experimental verification for characteristics of the light color area in image, and then the invalidity of dark channel prior theory in light color is discussed in detail. Based on the dynamical need for defogging image in light color areas, a self-adaptive threshold mechanism is proposed to optimize transmittance. The optimized transmission helps to avoid color shift problem in light color areas. The experiments indicate that our algorithm is in line with practical situation.
On the other hand, we do not consider the situation in which dark prior channel theory is out of work when image contains large range of white objects. It would be worth studying in the future.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
The research was sponsored by Natural Science Foundation of China (NSFC) under Grants 61573076, 61663008, and 61703063; the Scientific Research Foundation for the Returned Overseas Chinese Scholars and Program for Excellent Talents of Chongqing Higher School under Grant 2015-49; the Program for Excellent Talents of Chongqing Higher School under Grant 2014-18; Science and Technology Project under Grants KJ1705121 and KJ1705139; and the Chongqing Natural Science Foundation of China under Grant CSTC2017jcyjA1665.