Abstract
We propose a novel fusion method of IR (infrared) and visual images to combine distinct information from two sources. To decompose an image into its low and high frequency components, we use Gaussian and Laplacian decomposition. The strong high frequency information in the two sources can be easily fused by selecting the large magnitude of Laplacian images. The distinct low frequency information, however, is not as easily determined. As such, we use histogram distributions of the two sources. Therefore, experimental results show that the fused images can contain the dominant characteristics of both sources.
1. Introduction
Multisensor fusion is widely used for signal, image, feature, and symbol combination [1]. To fuse images, variable images, such as visual image, infrared (IR) image, millimeter wave (MMW) image, X-ray image, and depth image are used for concealed weapon detection [2, 3], remote sensing [4, 5], multifocus imaging [6], and so forth [1, 7, 8].
Forward looking infrared (FLIR) cameras can sense IR radiation (i.e., thermal radiation). Therefore, IR images can contain useful information that is not apparent in visual images. Alternatively, detailed information within the visual band is not included in IR images. Therefore, the fusion of IR and visual images can provide the advantages of both types of images. Also, the fusion of IR and visual images can apply various research fields, such as night vision [9, 10], face recognition [11], human detection [12], and detecting concealed object [13].
Visual and IR images can be easily fused by averaging; however, in this scheme, the advantages of the two sources are eliminated and the details are annihilated in the worst case.
To preserve the dominant advantages in the fused images, two methods can be utilized. Various methods using multiscale decomposition, such as Laplacian pyramid [14, 15], discrete wavelet transform (DWT) [16, 17], and contrast pyramid [18] can be used. One stage of Laplacian pyramid decomposition is shown in Figure 1. Additionally, some methods using region-segmentation [19–21] have also been proposed.
The multiscale decomposition-based methods are performed with low computation; however, selecting the correct and distinct values of the low frequency information is not easily determined. So, many stages of pyramid and DWT are used to select significant values, and distinct values are only detected from Laplacian images or high frequency bands [22]. Although distinct values can be selected from both low and high frequency images, most of all methods used only strong intensity and predetermined weights.
On the other hand, the latter methods can simply select the distinct values of the low frequency information; accurate segmentation is not guaranteed, and segmentation methods have a higher computation complexity. In addition, seam boundary regions should be blended by two image sources to prevent discontinuities.
In this paper we are aimed at developing a fusion method that has the advantages of the methods mentioned above. To do this, we use simple Gaussian and Laplacian decomposition and utilize histogram distributions and edge selection to determine the distinct values of the low and high frequency information, respectively. Because we use histogram distributions, significant low frequency information can be selected, such as locally hot or cold regions of IR images and locally bright or dark regions of visual images. In addition, only one decomposition is used and fast averaging filters are used, so processing speed is fast enough for real-time applications.
2. Fusion of IR and Visual Images
The proposed fusion method is similar to methods that use the Laplacian pyramid or DWT. We use Gaussian smoothing, as opposed to Gaussian or wavelet scaling, so our method is identical to methods that perform single scaling. A Gaussian image is the filtered image of an original image by the Gaussian convolution, and a Laplacian image is calculated by () as shown in Figure 2.
To obtain a fused image , the distinct information of is related to the magnitude; a larger indicates a strong edge. To determine the distinct value of , we use the histogram distributions of the visual and IR Gaussian images.
To fuse a visual image and an IR image , we first compute the Gaussian images and and decompose the Laplacian images and , and then the Gaussian and Laplacian image of fused image are generated by selecting distinct values using histogram distributions and edge selection as shown in Figure 3.
The Laplacian component of the fused image is easily determined by comparing edge strength such as the absolute Laplacian values at each pixel. If the large absolute Laplacian pixels are directly used, boundary discontinuities have an evil effect in terms of visuality. This can cause the fused images to contain discontinuous artefacts. Accordingly, we use weight map of the absolute Laplacian values. To compute this weight map , we compute the binary weight map calculated by
Finally, we determine using or as follows:where (2a) can be used for the fusion image having smooth boundaries of objects, and (2b) can be used for the fusion image having distinct boundaries of objects.
The distinct intensity values of most images have a low population. In particular, intensities with a low population in the IR images show the highest or the lowest temperature as shown in Figure 4. Therefore, we use the histogram distribution to select .
(a)
(b)
(c)
To select the Gaussian component of the fusion image, we use the low population map given bywhere and denote the histogram distribution functions for and , respectively. However, has extreme discontinuities, so we substitute for .
In all processes for Gaussian filter, we use a fast mean filter using spatial buffers to reduce computation complexities.
3. Experimental Result
The proposed fusion method was tested with three image sequences of TNO image fusion dataset [23]: “UN Camp,” “Dune,” and “Trees” (360 × 270, grayscale). These test images have small intensity rages, so we tested modified images by linear histogram normalization instead of original images. In this experiment, we used two parameter sets as shown in Table 1, where the first three parameters are the mask sizes. The optimal parameters are experimentally determined to make visually nature fusion images not having artificial discontinuities.
Examples of our results using the optimal parameters are shown in Figures 5, 6, and 7. Edge discontinuities are observed in and , while they are blurred in and . It seems that the distinct information of and is well fused in . In images, strong high frequency components are well selected. In images, the blurred distinct values are well revealed regardless of images. Particularly, isolated dark regions having low frequency are distinctly shown in images.
More results for the three image sequences using optimal parameters are shown in Figure 8; the red fused regions are more influenced by the visual images, while orange fused regions are more influenced by the IR images.
To objectively evaluate the proposed method, we consider two evaluation metrics: entropy () and the Xydeas and Petrovic index () [24]. The performance comparison with the averaging method is shown in Table 2. To compare our results with those reported in [12], we compared the improvement in the ratio of the averaging method as shown in Table 3, because the existing methods used a different image enhancement methods which were not appeared in literature.
Although our method using the optimal parameters does not yield the best objective performance, these metrics do not completely agree with human subjective evaluation as shown in Figure 9. The performance of Figure 9(b) is higher than Figure 9(a); however, Figure 9(b) has many discontinuities as indicated by the circles. In addition, in Figure 9(c), the stitching of the two source images (after clipping and mean filtering) yields the highest compared to the other results. Therefore, and for segmentation-based fusion methods may be overestimated to the human subjective evaluation.
(a)
(b)
(c)
Visual comparison of two results using two parameter sets is shown in Figure 10. As shown in Table 3, and of the results using the large parameters are higher than the results using the optimal parameters; however the results using the optimal parameters show better visuality.
(a)
(b)
(c)
Using single thread processing with a 3.60 GHz CPU, the average computation time of all of the sequences is 5.237 msec. This processing time is inconceivable for segmentation-based methods. Therefore, the proposed method can be used for real-time fusion applications.
4. Conclusion
In this paper, we have proposed a novel fusion method for IR and visual images based on Gaussian and Laplacian decomposition using histogram distributions and edge selection. This method can easily determine the distinct values of Gaussian and Laplacian images. The distinct values of Laplacian images are selected by edge strength, and the distinct values of Gaussian images are selected by using histogram distributions. So, the fused images can contain the dominant characteristics of two source images and can be obtained via relatively simple computation. In addition, we showed that the object evaluation, entropy and Xydeas and Petrovic index, does not completely agree with human visual evaluation by showing the results using two different parameter sets. Therefore, the proposed method can be used for image fusion and blending instead of other existing methods.
Conflict of Interests
The authors declare that they have no competing interests.