Abstract
For displaying high-dynamic-range images acquired by thermal camera systems, 14-bit raw infrared data should map into 8-bit gray values. This paper presents a new method for detail enhancement of infrared images to display the image with a relatively satisfied contrast and brightness, rich detail information, and no artifacts caused by the image processing. We first adopt a propagated image filter to smooth the input image and separate the image into the base layer and the detail layer. Then, we refine the base layer by using modified histogram projection for compressing. Meanwhile, the adaptive weights derived from the layer decomposition processing are used as the strict gain control for the detail layer. The final display result is obtained by recombining the two modified layers. Experimental results on both cooled and uncooled infrared data verify that the proposed method outperforms the method based on log-power histogram modification and bilateral filter-based detail enhancement in both detail enhancement and visual effect.
1. Introduction
For producing raw infrared data with a wide dynamic range, the high-quality infrared cameras accommodate the temperature range about 50 K, and the thermal detector used can catch the temperature difference within 0.01 K. Normally, the uncooled infrared detector acquires 14-bit data, while a human observer can distinguish only about 128 levels of gray in an image [1]. In addition, the display device can present an image with only 256 levels of gray (8 bits). Consequently, data compression and enhancement lead to infrared image display. Considering the fact that a fine infrared image should trade off between high contrast for human observers and rich detail without artifacts, diverse methods on contrast enhancement while keeping some constraints are presented in some related literature [2–5].
From the point of range mapping, infrared image contrast enhancement is a process of high-dynamic-range compressing, and the mapping function determines the final image display performance via contrast/brightness adjustment. Recently, plenty of work is devoted to designing the mapping function directly or indirectly. However, most of the work is devoted to refining the contrast of the gray image which is converted from 14-bit data to 8-bit data [6–10]. In their work, the performance is presented by using only two infrared data examples which is not persuasive due to the complex distribution of the raw infrared data. The first step in their work is the process of min-max mapping (linear mapping the minimum to 0 and the maximum to 255) that leads to great trouble which is analyzed in Section 3.
As for infrared image enhancement, automatic gain control- (AGC-) and histogram equalization- (HE-) based methods [11–14] are the representatives of linear mapping and nonlinear mapping. AGC method removes extreme values (e.g., 2% minimum and maximum of the total pixel number) and linearly maps the middle range of values onto an 8-bit domain for display. Histogram equalization normalizes the intensity distribution by using its cumulative distribution function to make the output image tend to have a uniform distribution of intensity. AGC method is more likely to be data-compressing and keep the differences of the pixel values while HE-based methods increase the contrast obviously by stretching the background and compressing the detail. As for visible optical images, He-based methods increase the contrast out of control that may make the image loss detail or other artifacts [11]. The same thing happens to infrared image. To overcome this disadvantage, plateau histogram equalization (pHE) [12] has been proposed to display infrared images by modifying the histogram with a plateau threshold value to suppress the enhancement of background. Both AGC and (p)HE mentioned treat the pixels of the image globally. For detail enhancement or contrast enhancement locally, adaptive histogram equalization (AHE) [13] adopts the neighbors’ information of a given pixel to determine the mapping for that pixel. The satisfied performance for AHE is ensured by using linear interpolation. Then, a generalization of AHE, contrast limited AHE (CLAHE) [14], refines the performance of modified histogram by computing the clipping level of the histogram. In addition, linear transformed histogram modification outperforms the HE-based methods for infrared image enhancement [15].
From the point of effeteness for operating on the hardware and display performance, HE-based methods stretch the raw infrared data more strongly than AGC, and pHE refines the performance more than HE in some scenes; however the plateau value is difficult to determine and the adaptive plateau value in some work cannot solve this problem effectively. In addition, these HE-based methods only adopt the histogram to determine the mapping function, and histogram itself is an information-loss feature description. As for the local HE-based methods, there are two drawbacks. On the one hand, modified histogram is hand-designed and equalization also introduces noise; on the other hand, refinement relies on linear interpolation between one window and another, and this is awkward for hardware implementation.
Recently, infrared image enhancement turns the attention to the detail information enrichment for image display. Bilateral filter and dynamic range partitioning (BF&DRP) proposed by Branchitta et al. [16] outperforms the traditional contrast enhancement for infrared image display. In BF&DRP method, bilateral filter is adopted to separate the detail layer from the base layer, and then the two layers are handled, respectively. At last, the two layers are recombined for the final display. However, it sometimes produces severe gradient reversal artifacts and highlights the noise in flat regions. To refine the performance, Zuo et al. [17] present a new method on detail enhancement for high-dynamic-range compressing. In Zuo’s work, bilateral filter is also adopted to decompose the raw data into two layers. With the two layers, it compresses the base layer and adds the adaptive weighted detail layer and rearranges to the 8-bit domain for display. Compared with Branchitta’s work [16], the great contribution of Zuo’s work [17] is removal of gradient reversal artifacts by using the Gaussian filter.
Compared with the BF&DRP method, Zuo’s work decreases the gradient reversal artifacts by blurring the base layer obtained via bilateral filtering. Totally, Zuo’s work decomposes the image into two layers; though it refines the base layer for removing artifacts, the base layer derived by BF may lead to cross-region mixing (discussed in the next section) and degrade the detail layer. In this work, a new image filter is adopted to generate the base layer and detail layer. Different from the BF used layer decomposition, no cross-region mixing happens to the detail layer. In addition, the gain control for the detail layer depends on the weight computed in the filtering process, and the propagated image filter [18] used in this work outperforms BF used in RBF-based method on detail enhancement. Experiment on raw infrared data captured in different scenes verifies the proposed method. The proposed method is visualized in Figure 1.
This work is organized as follows: the implementation of the proposed method is presented in Section 2. Section 3 presents the experiment on the infrared data and discusses the experiment results to show the performance of the proposed method. Section 4 is the conclusion.
2. Implementation of the Proposed Method
As for the related work on detail enhancement for infrared image, the normal procedure is decomposing the raw infrared data into two layers, the base layer and detail layer, by using an edge-preserving filter [19]:
For obtaining the two layers, we can filter the raw data and compute the difference between the raw data and the base layer . Hence, the detail-enhanced image can be derived from the mixed data :where and are the refined layers and is the gain factor to control the degree of the detail enhancement; the bigger the value of is, the stronger the detail presence in the final image is. The following subsections present the details of the proposed method.
2.1. Propagated Image Filtering for Base Layer
As discussed above, the base layer can be acquired by filtering the raw data. In the related work, bilateral filter (BF) [20] is used as the default filtering to smooth the raw data, while preserving the edge well. BF computes the value for pixel as follows:
In the formulation above, is the output of the pixel , is the normalization term, is the neighborhood of the pixel , is the Gaussian probability with variance , and is the distance metric, where is the spatial distance and is the photometric distance. Most of the filters like BF finish the filtering task by computing the weights for the pixels. As for BF, the term is the weight of the pixel to reconstruct the value of .
Combining the spatial information and intensity relationship, BF is always used as an effective filtering tool for edge preservation. However, cross-region mixing is a typical problem for existing filters when performing image processing tasks like denoising or smoothing. For instance, although bilateral filter involves the photometric distances between pixels into generating the filter weights, their use of explicit spatial filtering kernels would inevitably assign weights to pixels across image regions.
Based on the observation above, the propagated image filtering is presented in [18] to solve the cross-region mixing problem. In this paper, for obtaining a smooth base layer without texture mixing, we adopt the propagated image filter which does not use the explicit spatial kernel information. We first briefly introduce the propagated image filter and then refine it for computation in this work. Then we turn back to the filtering formulation:
In (4), is the weight of the pixel for reconstructing the pixel . As presented in [18], the weight is computed bywhere is denoted as the adjacent photometric relationship between pixels and ; suppose that the probability value of the two adjacent pixels is photometric related and proportional to the value of a Gaussian function of their pixel value difference. In the same way, is measured as the adjacent photometric relationship between and :
Intuitively, the weight is determined by each pixel in the connected path; if one pixel is little related to the central pixel, the pixel in the end of the path is also little related to the central pixel. This constraint ensures that two different texture regions do not mix with each other. In a window, the real pixels which are taken into account for constructing the central pixel form a path tree.
Hence, this work adopts the propagated image filter for generating the base layer. But the original propagated image filter calculates the weights across each pixel in the neighbor region. This work generates the weight from the central pixel to the surroundings alternatively for time saving. The detail of the weight computation will be discussed in the later subsection. So far, we can obtain the detail layer by subtracting the base layer from the raw data bywhere is the filtered image by using the propagated image filter.
2.2. Compressing the Base Layer by Using Histogram Projection
Normally, the base layer determines the contrast of the final image performance. In Zuo’s work [17], parameter-based histogram projection is adopted to compress the base layer for increasing the contrast of the base layer since it consists of high-dynamic-range pixels. Hence, a plateau-like threshold changes to the original by setting the plateau as 1 for the traditional histogram projection. In this section, we first present the traditional histogram projection and refine it as to satisfy the application, which is also different from the modification in [17].
Histogram projection [12] is a typical case of the generalized histogram equalization. Given the histogram of an image, if the value of the bin is bigger than zero, modify it as 1, otherwise 0. Hence, the modified histogram is a binary vector, and the sum of the elements for the vector denotes the image gray information:
In (8), the histogram is modified by clipping the value of the bins with a threshold 1; hence, the traditional histogram projection is a pHE-based method with . In the family of HE-based methods, histogram projection leads to the lowest contrast while keeping the most information. In the contrast, traditional HE loses much detail information. Reference [17] handles the high-dynamic-range data, for obtaining a relatively great contrast, and it replaces 1 with , which is bigger than 1 and sets it as 0.1% of the total number of pixels in [17]. Figure 2 shows the mapping change by using pHE-based methods with respect to different thresholds on the famous testing image named lena. We can see that the real gray bins of the processed image decrease with turning the threshold up. Meanwhile, the mapping curve turns a greater slope. The blue lines denote the setting , and if the image data has a compact distribution, the blue setting is just linear mapping. Intuitively, pHE-based methods modify the original histogram via changing the value of the parameter (plateau value) , where is the maximum value of and corresponding to the peak of the histogram. Typically, when , pHE-based method is the classical histogram equalization method. Many works are devoted to selecting a satisfactory that trades off between histogram projection performance and histogram equalization performance.
(a)
(b)
In addition, Zuo’s work computes the range of the mapping. If the number of the bins whose statistic value is bigger than is less than the device displaying range (e.g., 255), it rearranges the base layer between 0 and for decreasing the overenhancement phenomenon. However, this is limited to the data used in Zuo’s work. For example, the narrow range of the raw data leads to a low-intensity image according to the computation of . In addition, the nonadaptive threshold will also make the video display great difference between the two frames on both brightness and contrast. The rule used in Zuo’s work is too sensitive to the range of the raw data to handle the complex scene.
Considering the disadvantage discussed above, this work presents a reasonable solution to the problem of compressing the base layer. Given an image and the corresponding histogram, we first order it by the value of the bins and then select the th value as the threshold. Meanwhile, if the number of the bins whose statistic value is bigger than the threshold is less than the device displaying range, we not only limit the mapping range but also drop the data in the middle range. This rule is formulated as
In (9), and are the input data and output data, respectively. We define the active range of the input data as the number of the bins whose statistic value is in the list of Top , . In addition, is the output range, typically set as 255, is the adjusted parameter to control the brightness of the output if is smaller than the value of , and is the modified histogram equalization and output the data in the range . Since the human vision system only can distinguish the gray level of 128, for the high dynamic data, the information loss due to strong compressing by using Top policy does not affect the display performance while it can keep a stable contrast for diverse scenes.
2.3. Strict Gain Control of Detail Layer
In the subsections above, the base layer and its compressed output for the display device are obtained. In this subsection, we refine the gain control of the detail layer. As for the high-dynamic-range data, we compress the base layer and then add the detail layer to make the output have a presentation of detail enhancement since the base layer is compressed while the detail layer is not [21]. However, the detail layer consists of noise which should be suppressed, especially in the flat region. Hence, if the pixel in the filtering process owns a relatively big weight, it is more likely to appear in the flat region. Otherwise, it is close to the edge. Based on this observation, the pixel in the flat region is suppressed in the process of detail enhancement. Simply define the gain of the pixel of the detail layer:where and are the parameters to control the range of the gain factor and is the weight generated in the filtering process. In this work, we set as 2.5 and as 1.0.
The gain computed in (10) is the regular design and is adopted in many related works. However, the weight generated in this work is stricter than that generated by using other common filters. Propagated image filter designs the weight without spatial information for preventing from the cross-region mixing. This takes an advantage of generating a relative disconnected detail and makes the detail layer have a sense of hierarchy.
As discussed above, the weights of the pixels’ contribution to the central pixel in the window determine whether the central pixel stays in a flat region or not. The bilateral filtering, designing the weight by involving spatial relationship, computes the weight between each pixel in the neighborhoods and the central pixel. However, the propagated image filtering measures the weight by considering all the pixels in the connected path from the current pixel to the central pixel. As shown in Figure 3, is the central pixel of the window with size of .
In the process of filtering, the weight of each pixel in the window is measured. Bilateral filtering computes the weight between and directly as marked in blue curve. Propagated image filtering refines the weight not only by using the metric between and but also by consulting the pixels on the path between them. Hence, the propagated image filter has a strict weight computation. If () in the path is far from the central pixel with a metric, will have a small value of weight even if it is close to the central pixel with the same metric. By using the strict weight, only limited pixels in the window participate in the filtering process for ensuring that no cross-region mixing happens.
3. Experiment Results
3.1. Database Used in This Work
For verifying the performance of the proposed method, two kinds of raw data produced by the cooled infrared camera and uncooled infrared camera are used. Normally, uncooled raw data has a relatively narrow range compared to cooled one. For one thing, uncooled data is generated by mixing a degree of noise. For another, the discriminative ability of the uncooled infrared detector, left alone, is weaker than that of cooled infrared detector. Uncooled data makes a challenge of high-dynamic-range data compressing with noise problem. Figure 4 presents the used infrared data visualized by min-max AGC and the corresponding raw range. Figures 4(a) and 4(b) are the cooled infrared data, and Figures 4(c) and 4(d) are the uncooled infrared data. In addition, the resolution of the cooled infrared image is , and the uncooled one is .
(a) Visualized image of the raw cooled infrared data
(b) The distribution of the raw cooled infrared data
(c) Visualized image of the raw uncooled infrared data
(d) The distribution of the raw uncooled infrared data
In order to display the histogram information directly, we subtract the minimum value of the image while computing the histogram of the image. From Figure 4, we can see that the cooled infrared data has a relatively wide range, especially the first cooled infrared data with hot objects in the scene. We name the first scene as building and the second one as park for ease of exposition. In the meanwhile, we capture two big scenes of the uncooled infrared data. The uncooled infrared scene includes relatively more objects. Particularly the second scene consists of tree, sunshade, bike, road, people, and machines in operation, such as sedan and bus. In addition, we name the first scene as house, and the second one as bus stop. Since the visualized image is obtained by using the min-max AGC method which is a totally linear mapping to the raw data, the gray image may does not present the noise obviously.
3.2. Compared Methods and Performance Measurement
In this section, we compare the proposed method with the four classical methods on infrared image contrast enhancement: AGC, HE method, CLAHE, and the recently proposed DPL [15]. Meanwhile, we also compare the proposed method with the work [17] for detail enhancement. As for AGC method, we disregard the extreme 2% of the number of the pixels and then linearly map the rest into the 8-bit image. CLAHE is performed using the standard MATLAB adapthisteq function, dividing the images into 8 × 8 tiles and limiting contrast stretching to 0.05. DPL is carried out by setting . For a fair of comparison, we set the same parameters with the work [17], such as the same window size and standard deviation.
As for the performance measurement, no reference image quality assessment metrics, including contour-volume (CV [22]), uniform intensity distribution (UID [22]), and root-mean-square contrast (RMSC [23]), are adopted to measure the enhancement effect.
The three metrics are introduced as follows:(I)CV can reflect edge power and is obtained by summing the absolute value of the edge map which is generated by filtering the image with a Laplace filtering:where denotes convolution operation on with the sliding window and is the image of size pixels. Bigger value of CV indicates more edge information.(II)UID is derived from the information theory that the image includes more information if the distribution of pixels is more uniform. Hence, we havewhere is the gray level (e.g., 256) and is the number of pixels belonging to the th gray level. The same is with CV; bigger value of UID is preferred too.(III)RMSC is the basic index to determine the contrast effect. It is derived by calculating the average of the square of difference between the image and its average pixel value: where represents the average of all pixel values in the image of size pixels. RMSC measures the degree of the difference of the pixel.
In addition, all of the metrics may be influenced by different cases. We present them to compare the methods in some point. Among the methods, AGC, HE, CLAHE, and DPL concern infrared contrast enhancement, while Zuo’s work and ours concern infrared detail enhancement. Here, the metrics also reflect the different aspects of emphasis.
The results of the three metrics are presented in Table 1. The last column presents the mean index values of 16 scenes. From Table 1, we can see that the upper part of the table (corresponding to the four contrast enhancement methods) has relatively bigger value of RMSC than the two detail enhancement methods except CLAHE. This is just because detail enhancement prefers to boost the detail information while compressing the flat parts, leading to a low RMSC. Moreover, AGC has a relatively bigger value of RMSC, and this is because the raw data has a wide range with a large inner variance. As for the detail enhancement methods, ours has a bigger value of RMSC than that of Zuo’s work. As for CV metric, the two detail enhancement methods have slightly big value compared with the contrast enhancement methods. Typically, ours is significantly larger than Zuo’s. The same thing happens to UID metric. Totally, the four contrast enhancement methods refine the contrast obviously, while overlooking the detail information which can improve the visual performance. During the contrast enhancement processing, more or less detail information is missing by the gray level merger. In contrast, the detail enhancement compresses the base layer to some degree and enhances the detail layer for stunning visual effect.
3.3. Experimental Results and Analysis
Figures 5 and 6 show the cooled infrared data processing result. AGC method presents the gray image with a reasonable linear compressing and keeps the original temperature relationship. However, as a disadvantage of the infrared data, low spatial discrimination turns up in the gray image. Hence, the edge is blurred and the texture information is poor. HE method stretches the data nonlinearly and increases the contrast. Due to the contrast enhancement to some degree, the edge seems stronger than that of AGC. But for (b), the lines of the image are still coarse and blurred and some regions are overflat without detail. The blue rectangle shows the detail missing caused by the HE method. As the method of histogram modification, (c) and (d) outperform AGC on contrast and (b) on reducing washout effect. In addition, both (e) and (f) have a good performance on detail enhancement. Even so, (f) presents a better contrast and detail than (e). As shown in red tangles, (f) turns up a sense of hierarchy. This is because bilateral filter computes the weight considering the whole pixels in the window and makes a high spatial relationship. So (e) seems more flat than (f), and (f) presents a realistic scene, such as the branch and leaves of the trees. In addition, (f) seems better than (e) on the brightness and contrast, and (e) seems dim.
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
(c)
(d)
(e)
(f)
Figures 7 and 8 present the performance of different methods on the uncooled infrared data. With the same parameter setting, the blue rectangle in (b) points out the detail missing caused by HE method. Obviously, (c) and (d) outperform AGC on contrast and (b) on reducing washout effect. As for (e) and (f), we only select one region which is marked as the red rectangle. Both (e) and (f) perform a detail enhancement for the house infrared data; strictly speaking, (f) also outperforms (e) on contrast. The bus stop infrared data is a big scene including many objects with diverse temperatures. Compared with (e) in the scene of bus stop, (f) seems relatively soft. Some of the regions amplify the noise and make the image degrade.
(a)
(b)
(c)
(d)
(e)
(f)
(a)
(b)
(c)
(d)
(e)
(f)
4. Conclusion
This work presents a new detail enhancement for the infrared image. The raw infrared data is decomposed into the base layer and detail layer by using the novel propagated image filter which prevents the cross-region mixing. Meanwhile, the strict gain control generated in the filtering process guides a reasonable detail enhancement. Experiment on both cooled and uncooled infrared data compared with other classical methods proves the performance of the proposed method on detail enhancement and contrast/brightness refinement.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (51374063) and the Fundamental Research Funds for the Central Universities (N140303008, N141008001, and N150308001).