Abstract
Improved guided image fusion for magnetic resonance and computed tomography imaging is proposed. Existing guided filtering scheme uses Gaussian filter and two-level weight maps due to which the scheme has limited performance for images having noise. Different modifications in filter (based on linear minimum mean square error estimator) and weight maps (with different levels) are proposed to overcome these limitations. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.
1. Introduction
Medical images from different modalities reflect different levels of information (tissues, bones, etc.). A single modality cannot provide comprehensive and accurate information [1, 2]. For instance, structural images obtained from magnetic resonance (MR) imaging, computed tomography (CT), and ultrasonography, and so forth, provide high-resolution and anatomical information [1, 3]. On the other hand, functional images obtained from position emission tomography (PET), single-photon emission computed tomography (SPECT), and functional MR imaging, and so forth, provide low-spatial resolution and functional information [3, 4]. More precisely, CT imaging provides better information on denser tissue with less distortion. MR images have more distortion but can provide information on soft tissue [5, 6]. For blood flow and flood activity analysis, PET is used which provide low space resolution. Therefore, combining anatomical and functional medical images through image fusion to extract much more useful information is desirable [5, 6]. Fusion of CT/MR images combines anatomical and physiological characteristics of human body. Similarly fusion of PET/CT is helpful for tumor activity analysis [7].
Image fusion is performed on pixels, features, and decision levels [8–10]. Pixel-level methods fuse at each pixel and hence reserve most of the information [11]. Feature-level methods extract features from source images (such as edges or regions) and combine them into a single concatenated feature vector [12, 13]. Decision-level fusion [11, 14] comprises sensor information fusion, after the image has been processed by each sensor and some useful information has been extracted out of it.
Pixel-level methods include addition, subtraction, division, multiplication, minimum, maximum, median, and rank as well as more complicated operators like Markov random field and expectation-maximization algorithm [15]. Besides these, pixel level also includes statistical methods (principal component analysis (PCA), linear discriminant analysis, independent component analysis, canonical correlation analysis, and nonnegative matrix factorization). Multiscale transforms like pyramids and wavelets are also types of pixel-level fusion [11, 14]. Feature-level methods include feature based PCA [12, 13], segment fusion [13], edge fusion [13], and contour fusion [16]. They are usually robust to noise and misregistration. Weighted decision methods (voting techniques) [17], classical inference [17], Bayesian inference [17], and Dempster-Shafer method [17] are examples of decision-level fusion methods. These methods are application dependent; hence, they cannot be used generally [18].
Multiscale decomposition based medical image fusion decompose the input images into different levels. These include pyramid decomposition (Laplacian [19], morphological [20], and gradient [21]); discrete wavelet transform [22]; stationary wavelet transform [23]; redundant wavelet transform [24]; and dual-tree complex wavelet transform [25]. These schemes produce blocking effects because the decomposition process is not accompanied by any spatial orientation selectivity.
To overcome the limitations, multiscale geometric analysis methods were introduced for medical image fusion. Curvelet transform based fusion of CT and MR images [26] does not provide a proper multiresolution representation of the geometry (as curvelet transform is not built directly in the discrete domain) [27]. Contourlet transform based fusion improves the contrast, but shift-invariance is lost due to subsampling [27, 28]. Nonsubsampled contourlet transform with a variable weight for fusion of MR and SPECT images has large computational time and complexity [27, 29].
Recently, guided filter fusion (GFF) [30] is used to preserve edges and avoid blurring effects in the fused image. Guided filter is an edge-preserving filter and its computational time is also independent of filter size. However, the method provides limited performance for noisy images due to the use of Gaussian filter and two-level weight maps. An improved guided image fusion for MR and CT imaging is proposed to overcome these limitations. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.
2. Preliminaries
In this section, we briefly discuss the methodology of GFF [30]. The main steps of the GFF method are filtering (to obtain the two-scale representation), weight maps construction, and fusion of base and detail layers (using guided filtering and weighted average method).
Let be the fused image obtained by combining input images and of same sizes (). The base ( and ) and detail ( and ) layers of source images are where is the average filter. The base and detail layers contain large- and small-scale variations, respectively. The saliency images are obtained by convolving and and with a Laplacian filter followed by a Gaussian filter ; that is,
The weight maps and are where is a function with value 1 for and value 0 for (similarly for ). and are the saliency values for pixel in and , respectively.
Guided image filtering is performed to obtain the refined weights , , , and as where , , , and are the parameters of the guided filter and and and are the base layer and the detail layer weight maps.
The fused image is obtained by weighted averaging of the corresponding layers; that is,
The major limitations of GFF [30] scheme are summarized as follows.(1)The Gaussian filter from (2) is not suitable for Rician noise removal. Thus, the algorithm has limited performance for noisy images. Hence filter of (2) needs to be modified to incorporate noise effects.(2)The weight maps and from (3) can be improved by defining more levels. The main issue with binary assignment (0 and 1) is that when the saliency values are approximately equal, the effect of one value is totally discarded, which results in degraded fused image.
3. Proposed Methodology
The proposed scheme follows the methodology of GFF [30] with necessary modifications to incorporate the above listed limitations. This section first discusses the modification proposed due to noise artifacts and then the improved weight maps are presented.
3.1. Improved Saliency Maps
The acquired medical images are usually of low quality (due to artifacts), which degrade the performance (both in terms of human visualization and quantitative analysis).
Beside other artifacts, MR images often contain Rician Noise (RN) which causes random fluctuations in the data and reduces image contrast [31]. RN is generated when real and imaginary parts of MR data are corrupted with zero-mean, equal variance uncorrelated Gaussian noise [32]. RN is a nonzero mean noise. Note that the noise distribution tends to Rayleigh distribution in low intensity regions and to a Gaussian distribution in regions of high intensity of the magnitude image [31, 32].
Let be image obtained using MR imaging containing Rician noise . The CT image has higher spatial resolution and negligible noise level [33, 34].
The source images are first decomposed into base and and detail and layers following (1): and have an added noise term compared to (1). Linear minimum mean square error estimator (LMMSE) is used instead of Gaussian filter for minimizing RN, consequently improving fused image quality.
The saliency maps and are thus computed by applying the LMMSE based filter and following (2): The main purpose of is to make the extra term as in small as possible while enhancing the image details.
3.2. Improved Weight Maps
The saliency maps are linked with detail information in the image. The main issue with 0 and 1 weight assignments arises in GFF [30] when different images have approximately equal saliency values. In such cases, one value is totally discarded. For noisy MR images, the saliency value may be higher at a pixel due to noise; in that case it will assign value 1 (which is not desirable). An appropriate solution is to define a range of values for weight maps construction.
Let , and ,
Let , and ,
These values are selected empirically and may be further adjusted to improve results. Figures 1(a) and 1(b) show CT and noisy MR images, respectively. Figures 1(c)–1(f) show the results of applying different weights. The information in the upper portion of the fused image increases as more levels are added to the weight maps.
(a)
(b)
(c)
(d)
(e)
(f)
The weight maps are passed through guided filter to obtain , , , and . Finally the fused image is
LMMSE based filter reduces the Rician noise and the more levels of weight maps ensure that more information is transferred to the fused image. The incorporation of the LMMSE based filter and a range of weight map values makes the proposed method suitable for noisy images.
4. Results and Analysis
The proposed method is tested on several pairs of source (MR and CT) images. For quantitative evaluation, different measures including mutual information (MI) [35] measure , structural similarity (SSIM) [36] measure , Xydeas and Petrović’s [37] measure , Zhao et al.’s [38] measure , Piella and Heijmans’s [39] measures and , and visual information fidelity fusion (VIFF) [40] metric are considered.
4.1. MI Measure
MI is a statistical measure which provides the degree of dependencies in different images. Large value of MI implies better quality and vice versa [11, 33, 35]: where , , and are the entropies of , , and images, respectively. is the jointly normalized histogram of and , is the jointly normalized histogram of and , and , , and are the normalized histograms of , , and , respectively.
4.2. SSIM [36] Measure
SSIM [36] measure is defined as where is a sliding window and is where and are the variance of images and , respectively.
4.3. Xydeas and Petrović’s [37] Measure
Xydeas and Petrović [37] proposed a metric to evaluate the amount of edge information, transferred from input images to fused image. It is calculated as where and are the product of edge strength and orientation preservation values at location , respectively. The weights and reflect the importance of and , respectively.
4.4. Zhao et al.’s [38] Metric
Zhao et al. [38] used the phase congruency (provides an absolute measure of image feature) to define an evaluation metric. The larger value of the metric describes a better fusion result. The metric is defined as the geometric product of phase congruency, maximum and minimum moments, respectively.
4.5. Piella and Heijmans’s [39] Metric
Piella and Heijmans’s [39] metrics and are defined as where and are the local quality indexes calculated in a sliding window and is defined as in (13). Consider where is the mean of and and are the variance of and the covariance of , , respectively. Consider where and are the variance of images and within the window , respectively.
4.6. VIFF [40] Metric
VIFF [40] is a multiresolution image fusion metric used to assess fusion performance objectively. It has four stages. (1) Source and fused images are filtered and divided into blocks. (2) Visual information is evaluated with and without distortion information in each block. (3) The VIFF of each subband is calculated and the overall quality measure is determined by weighing (of VIFF at different subbands).
Figure 2 shows a pair of CT and MR images. It can be seen that the CT image (Figure 2(a)) provides clear bones information but no soft tissues information, while in contrast to CT image the MR image in Figure 2(b) provides soft tissues information. The fused image must contain both the information of bones and soft tissues. The fused image obtained using proposed scheme in Figure 2(d) shows better results as compared to fused image obtained by GFF [30] in Figure 2(c).
(a)
(b)
(c)
(d)
Figure 3 shows the images of a patient suffering from cerebral toxoplasmosis [41]. A more comprehensive information consisting of both the CT and MR images is the requirement in clinical diagnosis. The improvement in fused image using proposed scheme can be observed in Figure 3(d) compared to image obtained by GFF [30] in Figure 3(c).
(a)
(b)
(c)
(d)
Figure 4 shows a pair of CT and MR images of a woman suffering from hypertensive encephalopathy [41]. The improvement in fused image using proposed scheme can be observed in Figure 4(d) compared to image obtained by GFF [30] in Figure 4(c).
(a)
(b)
(c)
(d)
Figure 5 shows a pair of images containing acute stroke disease [41]. The improvement in quality of fused image obtained using proposed scheme can be observed in Figure 5(d) compared to Figure 5(c) (image obtained by GFF [30]).
(a)
(b)
(c)
(d)
Table 1 shows that proposed scheme provides better quantitative results in terms of , , , , , , and as compared to GFF [30] scheme.
5. Conclusions
An improved guided image fusion for MR and CT imaging is proposed. Different modifications in filter (LMMSE based) and weights maps (with different levels) are proposed to overcome the limitations of GFF scheme. Simulation results based on visual and quantitative analysis show the significance of proposed scheme.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.