Research Article | Open Access
An Improved Infrared/Visible Fusion for Astronomical Images
An undecimated dual tree complex wavelet transform (UDTCWT) based fusion scheme for astronomical visible/IR images is developed. The UDTCWT reduces noise effects and improves object classification due to its inherited shift invariance property. Local standard deviation and distance transforms are used to extract useful information (especially small objects). Simulation results compared with the state-of-the-art fusion techniques illustrate the superiority of proposed scheme in terms of accuracy for most of the cases.
The visible light astronomy due to reflection, refraction, interference, and diffraction enables scientists to unearth many of nature’s secrets; however, brightness of stars creates a haze in the sky. On the other hand, the infrared (IR) astronomy enables us to peer through the veil of interstellar dust and see objects at extreme cosmological distances. IR images have good radiometric resolution whereas visible images provide detailed information. In this regard, various image fusion techniques have been developed to combine the complementary information present in both images. These techniques can be grouped into wavelet, statistical decomposition, and compressive sensing.
The wavelet transform based fusion schemes generally decompose the visible and IR images into different base and detail layers, to combine the useful information. In , contourlet transform fusion is used to separate foreground and background information; however, the separation is not always accurate, which causes loss in target information. In , nonsubsampled contourlet transform, local energy, and fuzzy logic based fusion claims better subjective visual effects; however, merger and description of necessary components of IR and visible images in fusion model require improvements especially in case of noisy images. In , wavelet transform and fuzzy logic based scheme utilizes dissimilarity measure to assign weights; however, some artifacts are also introduced in the fused image. Contrast enhancement (using ratio of local and global divergence of IR image) based fusion lacks color consistency . In adaptive intensity hue saturation method , the amount of spatial details injected into each band of multispectral image is appropriately determined by the weighting matrix, which is defined on the basis of the edges present in panchromatic and multispectral bands. The scheme preserves the spatial details; however, it is unable to control the spectral distortion sufficiently . In , gradient-domain approach based on mapping contrast defines the structure tensor matrix onto a low-dimensional gradient field. However, the scheme effects the natural output colours. In , wavelet transform and segmentation based fusion scheme is developed to enhance targets in low contrast. However, the fusion performance is dependent on segmentation quality and large segmentation errors can occur for cosmological images (especially when one feature is split into multiple regions).
Statistical fusion schemes split the images into multiple subspaces using different matrix decomposition techniques. -means and singular value decomposition based scheme suffers from computational complexity . In , spatial and spectral fusion model uses sparse matrix factorization to fuse images with different spatial and spectral properties. The scheme combines the spectral information from sensors having low spatial but high spectral resolution with the spatial information from sensors having high spatial but low spectral resolution. Although the scheme produces better fused results with well preserved spectral and spatial properties, its issues include spectral dictionary learning process and computational complexity. In , an internal generative mechanism based fusion algorithm first decomposes source image into a coarse layer and a detail layer by simulating the mechanism of human visual system for perceiving images. Then the detail layer is fused using pulse coupled neural network, and the coarse layer is fused by using the spectral residual based saliency method. The scheme is time inefficient and yields weak fusion performance. In , independent components analysis based IR and visible image fusion scheme uses kurtosis information of the independent components analysis based coefficients. However, further work is required for determining fusion rules of primary features.
Compressive sensing based fusion schemes exploit the sparsity of data using different dictionaries. Adjustable compressive measurement based fusion scheme suffers from empirical adjustment of different parameters . In , a compressive sensing approach preserves data (such as edges, lines, and contours); however, design of appropriate sparse transform and optimal deterministic measurement matrix is an issue. In , a compressive sensing based image fusion scheme (for infrared and visible images) first compresses the sensing data by random projection and then obtains sparse coefficients on compressed samples by sparse representation. The fusion coefficients are finally combined with the fusion impact factor and the fused image is reconstructed from the combined sparse coefficients. However, the scheme is inefficient and prone to noise effects. In , a nonnegative sparse representation based scheme is used to extract the features of source images. Some methods are developed to detect the salient features (which include the target and contours) in the IR image and texture features in visible image. Although the scheme performs better for noisy images, the sparseness of the image is controlled implicitly.
In a nutshell, the above-mentioned state-of-the-art fusion techniques suffer from limited accuracy, high computational complexity, or nonrobustness. To overcome these issues, a UDTCWT based visible/IR image fusion scheme for astronomical images is developed. The UDTCWT reduces noise effects and improves object classification due to its inherited shift invariance property. Local standard deviation along with distance transforms is used to extract useful information (especially small objects). Simulation results illustrate the superiority of proposed scheme in terms of accuracy, for most of the cases.
2. Proposed Method
Let be the input source IR () and visible () registered images (with dimensions ). The local standard deviation for estimating local variations of is where is local mean image computed as The local standard deviation measures the randomness of pixels in a local area where the high values indicate presence of astrobodies and the low value values correspond to smooth/blank space (without any object or astrobody).
The image is obtained by thresholding to remove the pixel containing large variations: that is, where is a controlling parameter and and are mean and variance of , respectively. The gray distance image (to classify different points which are present inside/outside any shape/object) is computed using and mask as The distance transform (used to eliminate oversegmentation and short sightedness) measures the overall distance of the pixel from other bright pixels. For instance, a pixel closer to a cluster of stars (objects) tends to be part of the segmented mask and vice versa.
Let be the binary image obtained from the distance image : that is, where denotes mean image and is a positive constant. The image segments the foreground from background regions. The connected components image (to segment different binary patterns) with structure element (a matrix of all ones) is Let and represent area and perimeter of the th connected component placed at th location, respectively; a binary segmented image is constructed as where and are thresholding parameters. UDTCWT is applied on the source images to obtain coefficient matrix of dimensions (where represents wavelet coefficients). The decomposition obtained using UDTCWT not only eliminates noise/unwanted artifacts, but also is effective in preserving the useful information present in the input images (due to its undecimated property). The binary coefficient matrix is obtained by assigning nonzero values at pixel locations where visible image provides more information than the IR image. This binary thresholding ensures that the fused image contains the significant/important information of both source images (as the higher value of UDTCWT corresponds to presence of significant/important information): A binary fuse map is computed as where represents operation. LetThe final fused image is obtained by computing the inverse UDTCWT of fused coefficients . Figure 1 shows the flow diagram of proposed technique.
3. Results and Discussion
To verify the significance of the proposed technique, simulations are performed on various visible/IR datasets. Quantitative analysis is performed using (luminance/contrast distortion), (mutual information), (weighted quality index), (edge dependent quality index), (structural similarity index measure), (human perception inspired metric), (edge transform metric), and (image feature metric) [17–22].
The metric [17, 18] is designed through modality image distortion as combination of loss of correlation, luminance distortion, and contrast distortion. The metric  represents the orientation preservation and edge strength values. It models the perceptual loss of information in fused results in terms of how well the strength and orientation values of pixels in source images are represented in fused image. It deals with the problem of objective evaluation of dynamic, multisensor image fusion, based on gradient information preservation between the inputs and the fused images. It also takes into account additional scene and object motion information present in multisensor sequences. The metric  is defined by assigning more weight to those windows, where saliency of the input image is high. It corresponds to the areas that are likely to be perceptually important parts of the underlying scene. The index  takes into account aspects of the human visual system, where it expresses the contribution of the edge information of the source images to the fused images. The measure  is the similarity between two images and is designed to improve traditional measures of mean square error and peak signal to noise ratio, which are inconsistent with human eye perception. The metric  evaluates the performance of image fusion for night vision applications, using a perceptual quality evaluation method based on human visual system models. Image quality of fused image is assessed by contrast sensitivity function and contrast preservation map. The metric  assesses the pixel-level fusion performance and reflects the quality of visual information obtained from the fusion of input images. The metric  evaluates the performance of the combinative pixel-level image fusion, based on an image feature measurement (i.e., phase congruency and its moments), and provides an absolute measurement of image features. By comparing the local cross-correlation of corresponding feature maps of input images and fused output, the quality of the fused result is assessed without a reference image.
These quality metrics [17–22] work well for noisy, blurred, and distorted images, using multiscale transformation, arithmetic, statistical, and compressive sensing based schemes for multiexposure, multiresolution, and multimodal environments. These are also useful for remote and airborne sensing, military, and industrial engineering related applications. The normalized range of these measures is between 0 and 1 where high values imply better fusion metric for each quality measure.
Figure 2(a) shows Andromeda galaxy (M31) JPEG IR image taken by Spitzer space telescope  while Figure 2(b) shows the corresponding visible image taken using 12.5′′ Ritchey Chretien Cassegrain (at F6) and ST10XME . Figures 2(c)–2(f) are the outputs of local variance, distance transform, and segmentation steps.
Figure 3 shows the fusion results obtained by ratio pyramid (RP) , dual tree complex wavelet transform (DTCWT) , nonsubsampled contourlet transform (NSCT) , multiresolution singular value decomposition (MSVD) , Ellmauthaler et al. , and proposed schemes. By visual comparison, it can be noticed that the proposed scheme provides better fusion results, especially the preservation of background intensity value as compared to existing state-of-the-art schemes.
Figures 4(a) and 4(b) show visible and IR Jupiter’s moon JPEG images taken by new Horizons spacecraft using multispectral visible imaging camera and linear Etalon imaging spectral array . The fusion results obtained by RP , DTCWT , NSCT , MSVD , Ellmauthaler et al. , and proposed schemes are shown in Figures 4(c)–4(h), respectively. Note that only the proposed scheme is able to accurately preserve both the moon texture (from IR image) and other stars (from visible image) in the fused image.
Figures 5(a) and 5(b) show visible and IR Nabula (M16) JPEG images taken by Hubble space telescope . The fusion results obtained by RP , DTCWT , NSCT , MSVD , Ellmauthaler et al. , and proposed schemes are shown in Figures 5(c)–5(h), respectively. The fused image using proposed scheme highlights the IR information more accurately as compared to existing state-of-the-art schemes.
Table 1 shows the quantitative comparison of existing and proposed schemes (where the bold values indicate best results). It can be observed that the results obtained using proposed schemes are significantly better in most of the cases/measures as compared to existing state-of-the-art schemes.
A fusion scheme for astronomical visible/IR images based on UDTCWT, local standard deviation, and distance transform is proposed. The use of UDTCWT is helpful in retaining useful details of the image. The local standard deviation variation measures presence or absence of small objects. The distance transform activates the effects of proximity in the segmentation process and eliminates effects of oversegmentation in addition to short sightedness. The scheme reduces noise artifacts and efficiently extracts the useful information (especially small objects). Simulation results on different visible/IR images verify the effectiveness of proposed scheme.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
- L. Kun, G. Lei, L. Huihui, and C. Jingsong, “Fusion of infrared and visible light images based on region segmentation,” Chinese Journal of Aeronautics, vol. 22, no. 1, pp. 75–80, 2009.
- Z. Fu, X. Dai, Y. Li, H. Wu, and X. Wang, “An improved visible and Infrared image fusion based on local energy and fuzzy logic,” in Proceedings of the 12th IEEE International Conference on Signal Processing, pp. 861–865, October 2014.
- J. Saeedi and K. Faez, “Infrared and visible image fusion using fuzzy logic and population-based optimization,” Applied Soft Computing, vol. 12, no. 3, pp. 1041–1054, 2012.
- S. Yin, L. Cao, Y. Ling, and G. Jin, “One color contrast enhanced infrared and visible image fusion method,” Infrared Physics and Technology, vol. 53, no. 2, pp. 146–150, 2010.
- Y. Leung, J. Liu, and J. Zhang, “An improved adaptive intensity-hue-saturation method for the fusion of remote sensing images,” IEEE Geoscience and Remote Sensing Letters, vol. 11, no. 5, pp. 985–989, 2014.
- B. Chen and B. Xu, “A unified spatial-spectral-temporal fusion model using Landsat and MODIS imagery,” in Proceedings of the 3rd International Workshop on Earth Observation and Remote Sensing Applications (EORSA '14), pp. 256–260, IEEE, Changsha, China, June 2014.
- D. Connah, M. S. Drew, and G. D. Finlayson, Spectral Edge Image Fusion: Theory and Applications, Springer, 2014.
- A. Ellmauthaler, E. A. B. da Silva, C. L. Pagliari, and S. R. Neves, “Infrared-visible image fusion using the undecimated wavelet transform with spectral factorization and target extraction,” in Proceedings of the 19th IEEE International Conference on Image Processing (ICIP '12), pp. 2661–2664, October 2012.
- M. Ding, L. Wei, and B. Wang, “Research on fusion method for infrared and visible images via compressive sensing,” Infrared Physics and Technology, vol. 57, pp. 56–67, 2013.
- B. Huang, H. Song, H. Cui, J. Peng, and Z. Xu, “Spatial and spectral image fusion using sparse matrix factorization,” IEEE Transactions on Geoscience and Remote Sensing, vol. 52, no. 3, pp. 1693–1704, 2014.
- X. Zhang, X. Li, Y. Feng, H. Zhao, and Z. Liu, “Image fusion with internal generative mechanism,” Expert Systems with Applications, vol. 42, no. 5, pp. 2382–2391, 2015.
- Y. Lu, F. Wang, X. Luo, and F. Liu, “Novel infrared and visible image fusion method based on independent component analysis,” Frontiers in Computer Science, vol. 8, no. 2, pp. 243–254, 2014.
- A. Jameel, A. Ghafoor, and M. M. Riaz, “Adaptive compressive fusion for visible/IR sensors,” IEEE Sensors Journal, vol. 14, no. 7, pp. 2230–2231, 2014.
- Z. Liu, H. Yin, B. Fang, and Y. Chai, “A novel fusion scheme for visible and infrared images based on compressive sensing,” Optics Communications, vol. 335, pp. 168–177, 2015.
- R. Wang and L. Du, “Infrared and visible image fusion based on random projection and sparse representation,” International Journal of Remote Sensing, vol. 35, no. 5, pp. 1640–1652, 2014.
- J. Wang, J. Peng, X. Feng, G. He, and J. Fan, “Fusion method for infrared and visible images by using non-negative sparse representation,” Infrared Physics and Technology, vol. 67, pp. 477–489, 2014.
- G. Piella and H. Heijmans, “A new quality metric for image fusion,” in Proceedings of the IEEE Conference on Image Processing Conference, pp. 171–173, September 2003.
- Z. Wang and A. C. Bovik, “A universal image quality index,” IEEE Signal Processing Letters, vol. 9, no. 3, pp. 81–84, 2002.
- Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004.
- Y. Chen and R. S. Blum, “A new automated quality assessment algorithm for image fusion,” Image and Vision Computing, vol. 27, no. 10, pp. 1421–1432, 2009.
- C. S. Xydeas and V. Petrović, “Objective image fusion performance measure,” IET Electronics Letters, vol. 36, no. 4, pp. 308–309, 2000.
- J. Zhao, R. Laganière, and Z. Liu, “Performance assessment of combinative pixel-level image fusion based on an absolute feature measurement,” International Journal of Innovative Computing, Information and Control, vol. 3, no. 6, pp. 1433–1447, 2007.
- Andromeda galaxy (M31) IR, http://sci.esa.int/herschel/48182-multiwavelength-images-of-the-andromeda-galaxy-m31/.
- Andromeda galaxy (M31) visible, http://www.robgendlerastropics.com/M31Page.html.
- A. Toet, “Image fusion by a ratio of low-pass pyramid,” Pattern Recognition Letters, vol. 9, no. 4, pp. 245–253, 1989.
- J. J. Lewis, R. J. O'Callaghan, S. G. Nikolov, D. R. Bull, and N. Canagarajah, “Pixel- and region-based image fusion with complex wavelets,” Information Fusion, vol. 8, no. 2, pp. 119–130, 2007.
- Q. Zhang and B.-L. Guo, “Multifocus image fusion using the nonsubsampled contourlet transform,” Signal Processing, vol. 89, no. 7, pp. 1334–1346, 2009.
- V. P. S. Naidu, “Image fusion technique using multi-resolution singular value decomposition,” Defence Science Journal, vol. 61, no. 5, pp. 479–484, 2011.
- Jupiter's moon, http://www.technology.org/2014/12/03/plutos-closeup-will-awesome-based-jupiter-pics-new-horizons-spacecraft.
- Nabula (M16), http://webbtelescope.org/webb_telescope/technology_at_the_extremes/keep_it_cold.php.
Copyright © 2015 Attiq Ahmad et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.