Abstract
An image fusion technique for magnetic resonance imaging (MRI) and positron emission tomography (PET) using local features and fuzzy logic is presented. The aim of proposed technique is to maximally combine useful information present in MRI and PET images. Image local features are extracted and combined with fuzzy logic to compute weights for each pixel. Simulation results show that the proposed scheme produces significantly better results compared to state-of-art schemes.
1. Introduction
Fusion of images obtained from different imaging systems like computed tomography (CT), MRI, and PET plays an important role in medical diagnosis and other clinical applications. Each imaging technique provides a different level of information. For instance, CT (based on X-ray principle) is commonly used for visualizing dense structures and is not suitable for soft tissues and physiological analysis. MRI on the other hand provides better visualization of soft tissues and is commonly used for detection of tumors and other tissue abnormalities. Likewise, information of blood flow in the body is provided by PET (a nuclear imaging technique) but it suffers from low resolution as compared to CT and MRI. Hence, fusion of images obtained from different modalities is desirable to extract sufficient information for clinical diagnosis and treatment.
Image fusion integrates (complementary as well as redundant) information from multimodality images to create a fused image [1–6]. It not only provides accurate description of the same object but also helps in required memory reduction by storing fused images instead of multiple source images. Different techniques are developed for medical image fusion which can be generally grouped into pixel, feature, and decision level fusion [7]. Compared to feature and decision, pixel level methods [1, 2] are more suited for medical imaging as they can preserve spatial details in fused images [1, 8].
Conventional pixel level methods (including addition, subtraction, multiplication, and weighted average) are simpler but are less accurate. Intensity Hue saturation (IHS)-based methods fuse the images by replacing the intensity component [1, 5, 9]. These methods generally produce high-resolution fused images but cause spectral distortion (due to inaccurate estimation of spectral information) [10]. Similarly, principal components analysis based methods fuse images by replacing certain principle components [11].
Multiresolution techniques including pyramids, discrete wavelet transform (DWT), contourlet, curvelet, shearlet, and framelet transform image into different bands for fusion (a comprehensive comparison is presented in [12]). DWT-based schemes decompose the input images into horizontal, vertical, and diagonal subbands which are then fused using additive or substitutive methods. Earlier DWT-based fusion schemes cannot preserve the salient features of the source images efficiently, hence producing block artifacts and inconsistency in the fused results [2, 3]. Human visual system is combined with DWT to fuse the low frequency bands using visibility and variance features, respectively. Local window approach is used (to adjust coefficients adaptively) for noise reduction and maintaining homogeneity in fused image [4]. However, the method often produces block artifacts and reduced contrast [3, 5]. Consistency verification and activity measures combined with DWT can only capture limited directional information and hence are not suitable for sharp image transitions [13].
Texture features and visibility measure are used with framelet transform [5] to fuse high and low frequency components, respectively. Contourlet transform based methods use different and flexible directions to detect the intrinsic geometrical structures [13]. The common methods are variable weight using nonsubsampled contourlet transform [14]; and bio-inspired activity measurer using pulse-coded neural networks [15]. However, the down- and up-sampling in contourlet transform lack shift invariance and cause ringing artifacts [14]. Curvelet transform uses various directions and positions at length scales [16]; however, it does not provide a multiresolution representation of geometry [17]. Shearlet transform carries different features (like directionality, localization, and multiscale framework) and can decompose the image into any scale and direction to fuse the required information [17].
Prespecified transform matrix and learning techniques are used with kernel singular value decomposition to fuse images in sparse domain [18]. In [19], image fusion has been performed using redundancy DWT and contourlet transform. A pixel level neuro-fuzzy logic based fusion adjusts the membership functions (MFs) using backpropagation and least mean square algorithms [20]. A spiking cortical model is proposed to fuse different types of medical images [21]. However, these schemes are complex or work under certain assumptions/constraints.
A fusion technique for MRI and PET images using local features and fuzzy logic is presented. The proposed technique maximally combines the useful information present in MRI and PET images. Image local features are extracted and combined with fuzzy logic to compute weights for each pixel. Simulation results based on visual and quantitative analysis show the significance of the proposed scheme.
2. -Trous-Based Image Fusion: An Overview
In contrast to conventional multiresolution schemes (where the output is downsampled after each level), à-trous or undecimated wavelet provides shift invariance, hence better suited for image fusion.
Let different approximations of MRI image (having dimensions ) be obtained by successive convolutions with a filter , that is, where and is a bicubic B-spline filter. The th wavelet plane of is, The image is decomposed into low and high frequency components as where is the total number of decomposition levels. Similarly PET image in terms of low and high frequency components is where , as PET images are assumed to be in pseudocolor [9].
Different methods are present in literature to fuse low and high frequency components which are generally grouped into substitute wavelet (SW) and additive wavelet (AW). The fused image using SW is Note that SW method fuses image by completely replacing the high frequency components of PET by high frequency components of MRI image, which can cause geometric and spectral distortion. SW and IHS (SWI) are combined to overcome the limitation in fused image ; that is, where the intensity image is The substitution process in SWI method sometimes results in loss of information as the intensity component is obtained by simple averaging/weighting.
In AW method, the fused image is obtained by injecting high frequency components of into : AW method adds the same amount of high frequencies into low-resolution bands which causes redundancy of high frequency components (hence resulting in spectral distortion).
To cater the limitation, AW luminance proportional (AWLP) method injects the high frequencies in proportion to the intensity values [22]. Consider where are total number of bands. The fused image of AWLP preserves the relative spectral information amongst different bands. The fused image using improved additive wavelet proportional (IAWP) [23] method is where are wavelet planes of a low-resolution (a spatially degraded version of ) MRI image . The is obtained by filtering the high frequencies (by applying a smoothing filter). The major limitations of the above schemes includes induction of redundant high/low frequencies; and consequently spatial degradations.
3. Proposed Technique
The proposed scheme first decomposes the MRI and PET images into low and high frequencies using à-trous wavelet. High and low frequencies are then fused separately according to defined criterion. The overall fused image in terms of high and low frequencies is
3.1. Fusion of Low Frequencies
Fusion of low frequencies and is critical and challenging task. Various schemes utilize different criterions for fusion of low frequencies. For instance, one choice is to totally discard the low frequencies of one image, another choice is to take average or weighted average of both and so forth. However, the schemes provide limited performance as they do not cater the spatial properties of image. We have proposed fusion of low frequency using different weighting average for each pixel location. The weights are computed based on the amount of information contained in vicinity of each pixel.
3.1.1. Local Features
Local variance (LV) and local blur (LB) features are used with fuzzy inference engine to compute the desired weights for fusing low frequencies.
LV [24] is used to evaluate the regional characteristics of image and is defined as : where is the mean value of window centered at pixel. Note that image containing sharp edges results in higher value (and vice versa).
LB is computed using local Rényi entropy [25] of image. Let be the probability (or normalized histogram) having intensity values within a local window (of size ) centered at pixel. is defined as [25]
High values of and show that contain more information and need to be assigned more weight as compared to image.
3.1.2. Fuzzy Inference Engine
Let high and low Gaussian Membership functions (MFs) having means , and variances , for LV be [26] Similarly let high and low Gaussian MFs having means , and variances , for LB be
The inputs and are mapped into fuzzy set using Gaussian fuzzifier [27] as where and are noise suppression parameters. The inputs are then processed by fuzzy inference engine using pre defined IF-THEN rules [26, 27] as follows. : IF is high and is high THEN is high. : IF is low and is high THEN is medium. : IF is high and is low,THEN is medium. : IF is low and is low THEN is low.
The output MFs for high (having mean and variance ), medium (having mean and variance ), and low (having mean and variance ) are defined as The output of fuzzy inference engine is where and . The weights are obtained by processing fuzzy outputs using center average defuzzifier [27].
The image is obtained by weighted sum of and as
3.2. Fusion of High Frequencies
Let represent a wavelet plane of the resultant image . This ensures that only those high frequency components are used for image fusion, which are not already present in . By the virtue of this, the proposed scheme not only avoids redundancy of information but also results in improved fusion results as compared to early techniques. The fused high frequency image is Note that is not dependent on the bands because is gray-scale image.
4. Results and Discussion
The simulations of proposed and existing schemes are performed on PET and MRI images obtained from Harvard database [28]. The fusion database for brain images is classified into normal, grade II astrocytoma, and grade IV astrocytoma images. The MRI and PET images are coregistered with spatial resolution. The proposed fusion scheme is compared visually and quantitatively (using entropy [29], mutual information (MI) [29], structural similarity (SSIM) [30], Xydeas and Petrovic [31] metric, and Piella [32] metric) with DWT [12], GIHS [6], IAWP [23], and GFF [33] schemes.
The original MRI images belonging to normal brain, grade II astrocytoma, and grade IV astrocytoma are shown in Figures 1(a)–1(c), respectively. Fluorodeoxyglucose (FDG) is a radiopharmaceutical commonly used for PET scans. The PET-FDG images of normal, grade II, and grade IV astrocytoma are shown in Figures 1(d)–1(f), respectively. It can be seen that different imaging modalities provide complementary information for the same region.
(a)
(b)
(c)
(d)
(e)
(f)
Figure 2 shows fused images (of normal brain) obtained by using different techniques. It can be seen from Figure 2(e) that the proposed technique has preserved the complementary information of both modalities and the fuzzy based weight assessment has enabled offering less spectral information loss as compared to other state-of-art techniques.
(a)
(b)
(c)
(d)
(e)
Figure 3 shows fused images (of grade II astrocytoma class) obtained by using different techniques. From Figure 3(e), it can be observed that the proposed technique provides complementary information contained in both modalities and the fuzzy based weight assessment has enabled offering less spectral information loss as compared to other state of art techniques. The improvement in fused images is more visible in the tumorous region (bottom right corner).
(a)
(b)
(c)
(d)
(e)
Figure 4 shows fused images (of Grade IV astrocytoma) obtained by using different techniques. Similar improvement (as that of Figures 2(e) and 3(e)) can be observed in Figure 4(e). It is easy to conclude that the proposed scheme provides better visual quality compared to the existing schemes.
(a)
(b)
(c)
(d)
(e)
Table 1 shows the quantitative comparison of different fusion techniques. Note that a higher value of the metric represents better quality. The fused images obtained using proposed technique provide better quantitative results in terms of entropy [29], MI [29], SSIM [30], Xydeas and Petrovic [31], and Piella [32] metrics.
5. Conclusion
An image fusion technique for MRI and PET using local features and fuzzy logic is presented. The proposed scheme maximally combines the useful information present in MRI and PET images using image local features and fuzzy logic. Weights are assigned to different pixels for fusing low frequencies. Simulation results based on visual and quantitative analysis show that the proposed scheme produces significantly better results compared to state of art schemes.
6. Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.