Abstract
There exist two main drawbacks for traffic images in classic image enhancement methods. First is the performance degradation that occurs under frontlight, backlight, and extremely dark conditions. The second drawback is complicated manual settings, such as transform functions and multiple parameter selection mechanisms. Thus, this paper proposes an effective and adaptive parameter optimization enhancement algorithm based on adaptive brightness baseline drift (ABBD) for color traffic images under different luminance conditions. This method consists of two parts: brightness baseline model acquisition and adaptive color image compensation. The brightness baseline model can be attained by analyzing changes in light along a timeline. The adaptive color image compensation involves color space remapping and adaptive compensation specific color components. Our experiments were tested on various traffic images under frontlight, backlight, and during nighttime. The experimental results show that the proposed method achieved better effects compared with other available methods under different luminance conditions, which also effectively reduced the influence of the weather.
1. Introduction
Video-based traffic surveillance has become essential in city management due to the rapid development of transportation. However, image sequences taken from surveillance systems display various properties in various environments, including frontlight, backlight, and extremely dark environments. These fluttering properties may reduce the surveillance quality and affect consequential detection processes, such as vehicle detection and vehicle plate recognition. As a result, the enhancement of traffic image sequences under complex environments plays a critical role in video-based surveillance.
There are two tracks for this topic. One is image enhancement under certain conditions. Xu et al. [1] proposed methods that involved a denoiser and a tone mapper, which enhanced very dark image sequences in particular. Klaus and Warrant [2] proposed a theoretical biological model of visual systems in nocturnal animals to determine the optimum spatiotemporal receptive fields for vision in dim light. He et al. [3] presented an effective method using dark channel prior to remove haze from a single image. Tan [4] proposed a method that maximized the local contrast of the image in inclement weather, such as fog and haze. Im et al. [5] described a contrast enhancement method for backlit images that first extracted underexposed regions using the dark channel prior map and then performed spatially adaptive contrast enhancements. Tsai and Yeh [6] gave a contrast compensation method using fuzzy classification and image illumination analysis especially for backlit and front-lit color faces. All these proposals perform well under certain conditions.
The other track involves the enhancement of images for certain objects. Yang [7] introduced a method that maximized the image contrast and then employed morphology processing for road sign recognition in natural scenes. The authors in [8] used normalization for local luminance to eliminate the impact of light changes so as to achieve the automatic detection and recognition of signs in natural scenes. In [9, 10], studies on license plate images in natural scenes were performed by means of histogram equalization and tensile contrast. In [11], Tan et al. employed the combination of high frequency enhancement and neural networks to strengthen the contrast in van images under different light conditions.
Additionally, histogram equalization and its variations are one of the most commonly used algorithms to perform contrast enhancement due to its simplicity and effectiveness [12–16]. Recently, Wu and Juang [17] proposed a histogram extension method specifically for vehicle detection systems. Wu [18] proposed an image enhancement method via optimal contrast-tone mapping. Xu et al. [19] analyzed the relationships between image histograms and contrast enhancement/white balancing and then proposed a generalized equalization model for image enhancement.
These research topics have contributed a great deal to improving image quality. However, as regards traffic images, these methods are not robust enough to deal with luminance changes during the course of a day or given the impact of extreme weather. In this paper, we have presented a new method to improve the quality of traffic surveillance images based on adaptive brightness baseline drift. In this method, the impacts of both brightness variation and time change have been considered simultaneously, and a brightness variation model has automatically obtained the parameters for enhancement. This method could remove the effects resulting from the impacts of both weather and light and could consequently produce satisfactory enhancement results under complex environments.
The remainder of this paper is organized as follows. Section 2 describes the proposed ABBD method. Section 3 presents the experimental results. Conclusions are finally drawn in Section 4, along with recommendations for future research.
2. Adaptive Brightness Baseline Drift
The traffic image enhancement approach discussed in this paper consists of two steps: brightness baseline model acquisition and adaptive image compensation based on such a model. Figure 1 illustrates the proposed approach flow diagram. In the first step, the brightness baseline model was obtained by an offline brightness baseline curve and online brightness feedback. In the second step, the color image to be processed was initially transformed into an HSV color space in which the brightness component was adjusted according to the baseline value, and then the backward color transformation is applied to produce the enhanced image.
2.1. Brightness Baseline Model Acquisition
Because brightness changes with time, the baseline model has been used to illustrate the overall trend of illumination; the model is comprised of two parts: the first part is brightness baseline analysis and compensation and the other is real-time brightness feedback.
2.1.1. Brightness Baseline Analysis and Compensation
In order to precisely measure the illumination variations with time, two groups of images were obtained from a north-south intersection and an east-west intersection, respectively, in different light directions. The mean values of the brightness were calculated after certain intervals for both groups. The statistical result is illustrated in Figure 2. It can be concluded from this figure that the mean brightness values do vary significantly over time. Therefore, a baseline of this value is proposed in this paper, and these baseline values form a brightness baseline curve.
Define the brightness baseline value in time as , which is relatively invariable at night, since it is free from the impact of sunlight, while this value changes during daytime. The camera that faces north or south has the brightest surveillance image at noon but is much darker in the morning and in the evening. The change in brightness goes from low to high and then drops back to the low level. This process can be described as a Gaussian function during daytime and as a constant during nighttime inwhere , , , and are the brightness value during the brightest moment, the current moment, the mean value of this curve, and the standard deviation of the Gaussian function, respectively. Here, is set to 150, according to experiments, and . was chosen for this procedure: assume that daytime lasts from to , and the interval covers more than 99.7% of the area in the Gaussian distribution. Thus . The constant denotes the brightness value at night.
Figure 3 illustrates the condition for the camera facing east and west. When the luminance angle changes, frontlight and backlight occur. The image is bright under frontlight conditions, as is the area exposed to the direct light present in backlight images, but the area in shadow is dark. Figure 3(a) shows the relationship between camera orientation, the object, and the sun. The traffic image shows frontlight in the morning and backlight in the afternoon when the camera is facing west; the opposite occurs when it is facing east. Figures 3(b) and 3(c) give examples of the frontlight image and backlight image, respectively. The brightness histograms shown in Figures 3(d) and 3(e) can quantitatively depict brightness distributions.
(a)
(b)
(c)
(d)
(e)
To move the range of the traffic image to a reasonable range of brightness, the entire brightness of the frontlight image should be reduced. For the backlight image, the brightness in the area exposed to direct sunlight should be reduced, while the brightness should be increased for those in the shadow. The vehicle plate and other useful information can be found in shadow, whereas highlighted areas are on the ground; under circumstance of backlight the brightness baseline value should be increased. Therefore, the brightness baseline value should be reduced in the morning and increased in the afternoon when the camera is facing west; the opposite should occur when facing east. Because the overall trend of illumination is slowly changing, the trigonometric function in the following equation is utilized to compensate the image:where is the compensation factor, which varies with the angle of the camera and seasons. Thus it is larger in summer because the backlight and frontlight are more intense, whereas the value is smaller during winter, when lighting is mild. This value is greater than zero when the camera is facing west, less than zero when facing east, and equal to zero when facing north and south. indicates when the time compensation starts; is when it ends. Then we have
Therefore, the value of brightness baseline after compensation at time is equal to where represents the brightness baseline in normal situations and represents the compensated brightness. The curves of and are illustrated as shown in Figure 4. The red curve is the compensation brightness curve in frontlight while the blue curve is the compensation brightness curve in backlight. In contrast, the red curve stands for the baseline in frontlight and the blue curve stands for the baseline in backlight.
2.1.2. Real-Time Brightness Feedback
Considering that images taken at the same times differ under different weather and luminance conditions, it is not sufficient to confirm the brightness reference value solely by applying the brightness baseline curve. Therefore, a real-time feedback system was introduced to resolve this problem. First, the reference value was obtained from the brightness baseline curve, . Next, the mean brightness of the real-time images was calculated as , along with the difference between the baselines, . Defining the correction value for brightness, , we have where is the correction step; it takes the value between 0 and 1.
However, in a real experiment, two frames in consequence may differ greatly due to violent changes in lightning and object movement. Therefore, the previous correction value, defined as the “inertia term,” was taken into consideration in the current experiment. Figure 5 shows this process, where stands for the correction value of the current frame and the previous frame, and is the “inertia coefficients” that ranges from 0 to 1. Thus
The final brightness reference value, shown in the following equation, is composed of the value from the curve and the feedback value:
2.2. Adaptive Color Image Compensation
After the reference value for brightness was attained, it was then used in the brightness histogram to enhance the image. In order to maintain the color, the color image must first be transformed into the HSV color space; the enhancing process is performed in the V component. The steps for image compensation are listed as follows.
Firstly, the RGB image was transformed into the HSV color space, and the mean of brightness was calculated using where is the brightness value and is the pixel count in brightness .
Secondly, the adaptive reference value for brightness was obtained by (7).
Thirdly the histogram of brightness was shifted by . Consequently, the drift coefficients and were calculated, using (9), where the two regions were separated by the brightness baseline.
Then brightness transformation was processed in both regions using (10). Figure 6 is an example of adaptive image compensation, where the left area has a value from 0 to , while the right one has a range from to 255.
Finally, the image was transformed back into the RGB color space:where and are the minimum and maximum for the brightness, respectively. is the brightness value that is less than the reference brightness and is the value greater than the reference brightness. and are the corresponding values after the shift.
3. Experimental Results
3.1. Effectiveness of Traffic Image Enhancement
In order to testify the effectiveness of this enhancement method, the experiments were conducted at one crossroad in Hubei province, China. As a result, frontlight images at 11 h, backlight images at 16 h, and images at 21 h were randomly selected among the samples. The methods in [1–6] only focused on certain environments, and [13–16] aimed at detecting certain objects; the two latest and typical enhancement methods that are effective under different luminance conditions, histogram extension [17] and ESIHE [16], were employed as comparisons.
Figure 7(a) is the original frontlight image, which is considerably bright because of the intense sunlight and the installation angle of the camera. Figure 7(b) is the image enhanced by our method, while Figures 7(c) and 7(d) are the results of histogram extension and ESIHE, respectively. From these figures, it can be concluded that the proposed method could reduce the entire brightness while increasing the contrast at the same time. The color of the vehicles and the ground processed by our method is superior to other comparison methods in visual perception. For the purpose of strengthening details, Figures 7(e) and 7(f) show the amplified images of the red rectangle areas. In comparison with other methods, our proposed method enjoys significant reduction of exposure and a wider range of brightness.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 8(a) is the original backlight image; most of the areas are extremely bright except for those in shadow. Figure 8(b) is the image enhanced by our method, which has a smoother transition in the area where brightness and shadow meet; moreover, there is less road texture and better contrast in shadow area. For example, the vehicle plate and other features become clearer. Figures 8(c) and 8(d) show the results of the histogram extension and ESIHE. As can be observed, the histogram extension method barely changed the brightness value, and ESIHE method had a limited effect on this occasion. Observed from Figures 8(e) and 8(f), the vehicle detail is enhanced in shadow, and thus a better visual effect has been fulfilled.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Figure 9(a) is the original image during nighttime. The vehicle is too dark to be observed. Figure 9(b) is the result processed by our method, which illustrates clear details without causing distortion. Figures 9(c) and 9(d) are the histogram extension and ESIHE results. Even though the former increased the brightness, it added a severe block-effect, while the ESIHE enhanced the contract but meanwhile brought in more noise. Figures 9(e) and 9(f) indicate again that our method augments the contract without causing overexposure in the vehicle lamp and vehicle plate area.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
3.2. Experimental Results for Eliminating Effects of Weather
The quality of traffic surveillance videos can be greatly affected by environment change, which results in reduced precision in vehicle detection and plate recognition. Experiments were conducted to demonstrate that ABBD is a method that eliminates the effects of weather impact, with samples collected every 10 h, 12 h, 16 h, and 21 h on sunny days (S), rainy days (R), and cloudy days (C), respectively. The root mean square error (RMSE) [17] and absolute mean brightness error (AMBE) [14] in the following equations, respectively, were employed to measure the differences between the two images with different sceneries:where and are defined as the pixel value at in images and . and stand for the height and width of such images. and are the mean values of the image. Table 1 lists the RMSE result for each sample by applying ABBD, HE, and ESIHE. The corresponding AMBE is listed in Table 2.
It can be concluded from Table 1 that the RMSE varies at the same times under different weather conditions; however, the RMSE was reduced after our method was implemented and produced the lowest RMSE for all images. The HE method, which could reduce original RMSE values, had RMSE values higher than the proposed method. The ESIHE method seldom cut down the RMSE values.
Table 2 shows the absolute difference between images under different weather conditions. The less the value was, the greater the similarities between the images were. The AMBE values of the HE and ESIHE methods for all the images were higher than the AMBE value of the proposed method; therefore, our method proved effective in diminishing the absolute average brightness difference.
Based on the experimental results, the weather effects were significantly reduced from all the source images. These results benefit processes such as moving object segments, vehicle detection, and plate recognition. Moreover, the parameters set for the surveillance system could be simplified, and better robustness could be achieved in complex environments.
4. Conclusions
This paper proposed a new method based on adaptive brightness baseline drift for traffic image enhancement. First, changes in lighting were analyzed and then an adaptive model was applied. After the modeling, a nonlinear enhancement method was processed onto the image. The performance analysis indicated that our method achieved better effects compared with other available methods under different luminance conditions and could effectively reduce the influence of the weather. In our further work, we will continue to optimize this method, especially as regards evening performance; moreover, we will apply this method to vehicle detection and license plate recognition.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
This work was supported by the Major State Basic Research Development Program of China (973 Program) (no. 2012CB719900).