Abstract
In many remote sensing applications, users usually prefer a multispectral image with both high spectral and high spatial information. This high quality image could be obtained by pan-sharpening techniques which fuse a high resolution panchromatic (PAN) image and a low resolution multispectral (MS) image. In this paper, we propose a new technique to do so based on the adaptive intensity-hue-saturation (IHS) transformation model and evolutionary optimization. The basic idea is to reconstruct the target image through a parameterized adaptive IHS transformation. An optimization objective is thus introduced by considering the relations between the fused image and the original PAN and MS images. The control parameters are optimized by an evolutionary algorithm. Experimental results show that our new approach is practical and performs much better than some state-of-the-art techniques according to the performance metrics.
1. Introduction
With the development of satellite technologies, many satellites, such as IKONOS and QuickBird, provide two types of images: the multispectral (MS) image and the panchromatic (PAN) image. The MS images are with high spectral information but low spatial information; on the contrary, the PAN images are with high spatial information but low spectral information. However, in practice, users usually prefer a single image with both high spectral and high spatial information. This high quality synthetic image could be reconstructed by the so-called pan-sharpening techniques which fuse a high resolution PAN image and a corresponding low resolution MS image.
In the last two decades, a variety of pan-sharpening methods as well as the performance metrics has been proposed. Comprehensive surveys are referred to [1, 2]. In early stages, the component-substitution (CS) methods, such as intensity-hue-saturation (IHS) [3] and principal component analysis (PCA) [4], inject spatial information from PAN image to the MS image. In later stages, the multiresolution analysis (MRA) based methods, such as wavelet and contourlet [5], inject high-frequency details inferred from the PAN image to the MS bands. Recently, the CS based and/or MRA based methods are also hybridized [6, 7] or combined with other techniques, such as evolutionary algorithms (EAs) [8] and compressed sensing [9, 10], to get high spectral and spatial performance.
Among different pan-sharpening methods, the wavelet transform fusion method has good spectral information, but the texture of the fused image is coarse and the detail is not clear enough; the IHS transformation based method [3] is popular due to its efficiency. However, the major limitations of this method are obvious: () it can only handle MS image with three bands and () the ISH transform fusion method can improve the texture characteristics of the fused image and enhance the detail information of the fused image, but the spectral distortion is larger. A generalized IHS transform has been proposed to overcome the former limitation [11]. By using the intensity band to approximate the PAN image, some adaptive IHS methods have been proposed to overcome the latter limitation [12, 13].
Till now, due to the complexity of pan-sharpening, all of the existing methods are more or less deficient. Especially in the variational pan-sharpening methods, the assumptions might not be very perfect. This naturally leads us to consider some new assumptions from new perspectives and to build a new model from them to obtain pan-sharpened images more accurately and effectively. In this letter, we propose a new pan-sharpening method based on both EA and adaptive IHS [13] as shown in Figure 1. This method is based on swarm intelligence algorithm and global optimization, the algorithm uses ISH transform and fusion method to obtain the fused image as the initial population and then the image based on the quantitative evaluation index to construct the objective function, finally using evolution algorithm to optimize the objective function, so as to obtain the appropriate image fusion results. Both experimental results and objective evaluations show that the proposed algorithm is better than conventional fusion methods such as wavelet transform and ISH transform in expressing detailed information and fusing quality parameters.
Our method targets finding an optimal fused multispectral () image by assuming that () the PAN image could be a linear combination of the optimal synthetic image bands and () the MS image could be degraded from the optimal synthetic image. An optimization objective is thus introduced based on the two concepts. An EA is used to optimize the objective to get the optimal control parameters, and the optimal synthetic image is reconstructed by using adaptive IHS model with the optimal control parameters.
2. An Evolutionary Pan-Sharpening Method
Before fusing the two images, we upsample the original MS image to make it with the same size as the PAN image. Let be the preprocessed -band MS image, is the th band, and a pixel at position of band is denoted as . Let denote the given PAN image and be the unknown fused synthetic multispectral image.
2.1. Adaptive IHS [13]
In adaptive IHS, the intensity () band is estimated as where , , are the unknown coefficients. It is further assumed that the PAN image could be approximated by the MS images; that is, Therefore, the coefficients can be obtained by minimizing where is the Lagrange multiplier.
With the optimized coefficients , the synthetic image is reconstructed byfor . is an edge detector defined as where and are two control parameters and is the gradient of the PAN image at .
It is clear that, on the edges when , the spatial information from PAN image will be injected into the synthetic image; otherwise, on the off edge when , , which means that only the spectral information from the MS image will be injected into the synthetic image. By this way, the spectral distortion in the original IHS method is overcome in a sense.
2.2. Evolutionary Optimization
Evolutionary algorithms are global heuristic optimization methods which simulate the natural evolution process [14]. The model tackled by an EA can be generally formulated aswhere is a decision variable, is the search space, and is the objective function. A widely used EA framework is shown as follows.
Step 1. Uniformly randomly sample a set of solutions which means population size from the search space , and calculate their objectives.
Step 2. Generate a set of offspring solutions from by using genetic operators, and calculate their objectives.
Step 3. Calculate the fitness of each solution, and select solutions to from .
Step 4. If the stop condition is satisfied, return the best solution found so far; otherwise, go to Step 2.
A variety of EAs have been proposed and new ones are still coming. These EAs differ from each other by using different genetic operators in Step 2 and the selection operators in Step 3. An EA usually stops when a given maximum number of iterations are reached or the best objective value does not change in some iterations. The algorithm flow is shown in Figure 2.
From the above framework, we can see that EAs have two major advantages over classical programming methods: () they work on a set of solutions simultaneously; thus they are insensible to the initial solutions and () they do not need the gradient of the objective function, which makes them suitable to nonlinear and black-box optimization problems.
2.3. Evolutionary Pan-Sharpening Model
We aim to design a model, which is suitable for EAs, by extending the adaptive IHS method. Like adaptive IHS, our basic idea is to find the optimal coefficients to reconstruct the synthetic image. Instead of optimizing the function to obtain , we consider more to design such an objective function. More precisely, we assume that(i)the PAN image could be approximated by a linear combination of the optimal synthetic image; that is, where , , are unknown coefficients;(ii)the upsampled MS image could be degraded from the optimal synthetic image; that is, where is a unknown convolution template.
By considering the above two assumptions and the model in (7), we design the following model: where is norm and is calculated as in (5). It is clear in (10), the first part reflects the first assumption and the second part reflects the requirement of the second assumption.
We would like to make the following comments on the above model.(i)We use norm in (10) to measure the difference between between two points, but other measures can also be applied here. The reason is that the optimization technique we use do not consider the form of the model.(ii)In EA algorithm, before calculating the objective value, is normalized to satisfy the equation constraints in (10).
2.4. Adaptive Evolutionary Clustering [15]
In this letter, we use composite differential evolution (CoDE) as the optimizer. It should be noted that other EAs can also be applied here. In evolutionary clustering, coding scheme, reproduction procedure, and fitness function are the three main components.
(1) String Representation. In CoDE, each decision variable is a sequence of real numbers and it falls into a feasible solution space, and the initial population is randomly generated.
(2) Fitness Evaluation. To guarantee that the evolution has a meaningful direction, a fitness function must be set. The fitness function of this experiment is as above (10).
(3) Reproduction Procedure. In an EA, the roles of reproduction procedure are () selecting some parent solutions from the current population and () generating offspring solutions by the selected parent solutions. In other words, it is the combination of crossover and mutation. The basic idea behind the reproduction procedure is to extract information from the current population to guide the search towards the global optimum. In this latter, we use the reproduction operator of CoDE to generate the new trial solutions. In CoDE, three offspring points are generated by applying three reproduction operators for each point in the paternal population. The three operators generate component one by one as follows: rand/1/bin: rand/2/bin: current-to-rand/1: where , returns a uniform random number from , is a random integer from , and – are randomly selected from the population and they are different from each other and different from the parent . and are two algorithm parameters which are randomly selected from the combinations of , , and .
(4) Selection Procedure. The selection procedure selects some solutions from the current population and the newly generated offspring solutions to survive to the next generation. A naive way is to replace the current population directly by the offspring population. A more widely used strategy is to always keep the best solution found so far. This strategy is also called elitist selection strategy.
CoDE uses a simple elitist selection strategy by comparing each parent with its best offspring and keeping the better one. Let be the parent and , , and be the three children. Let and the surviving one is
3. Experiments
The effect evaluation of remote sensing image fusion is very complex. Subjective assessment method has one sidedness and is greatly influenced by the observer’s own factors. Therefore, the subjective evaluation of images should be carried out at the same time. Assuming that is the size of the fused image, the objective quantitative evaluation used in this letter is mainly as follows.
(1) CC (Correlation Coefficient). Correlation factor (CC) is used to analyze and compare the quality of spectra and is widely used in image quality evaluation. In this letter, the fusion of multispectral images is compared with the original multispectral images; its formula is
Among them, and are the original multispectral image and the mean value of the fused multispectral image . The correlation factor of the image is calculated by the whole image. The value of CC is between 0 and 1; when CC is 0, there is no correlation, and when CC is 1, the correlation is the highest.
(2) ERGAS (Relative Global Dimensional Synthesis Error). ERGAS is mainly used to calculate spectral distortion, and its formula is
where is the resolution of the image with high spatial resolution, and this is the resolution of panchromatic image. is the resolution of multispectral images with low spatial resolution. is the band number of multispectral images. is the mean of each band after fusion of multispectral images.
(3) QAVE (A Universal Image Quality Index). QAVE treats any distortion as a combination distortion of the three distortion factors (loss of correlation, luminance distortion, and contrast distortion). and represent the original multispectral images and the fused multispectral images. The formula for QAVE is
(4) RASE (The Relative Average Spectral Error). RASE is used to evaluate the average performance of image fusion methods for each spectral band, and the formula is as follows:
Among them, is the average radiation value of band of original multispectral image, and RMSE is minimum square error.
(5) RMSE (The Root Mean Squared Error). The Root Mean Squared Error (RMSE) is a common method for image evaluation, and its formula is as follows:
Among them, and are original multispectral images and fused multispectral images, respectively.
(6) SAM (Spectral Angle Mapper). SAM is also widely used in multispectral image analysis to calculate the average change of all angles in the spectral component. In order to calculate the SAM between two multispectral images, assume that each image has bands, the pop vectors are and , and each has components, and the formula is as follows:
(7) SID (Spectral Information Divergence). SID treats the spectral components of each pixel as random variables and then measures the difference probabilities between these variables; is a multispectral image with a probability density . Similarly, is also a multispectral image and the probability density of . Its formula is
The CoDE algorithm [16], a variation of EA, is used to tackle the proposed optimization model. The parameters of CoDE are as follows: the size of the population is ; the algorithm stops after iterations. For simplicity, we name the proposed evolutionary IHS method with parameter as EIHS(). Since CoDE is a heuristic method which may get different results on different runs, the following results are based on independent runs of EIHS(). Adaptive IHS (AIHS) [13], the stationary wavelet, and PCA based methods are compared with EIHS on two -band multispectral Quickbird images with size of pixels [17]. All the algorithms are implemented in Matlab 7.11 and executed on a server computer with two Xeon 3.2 G CPUs and 96 G RAM. Seven spectral performance metrics, which are used in [13], are applied to assess the performance of the compared methods (the details of CoDE algorithm and more experimental results are referred to in the supplemental materials. The source codes are available from [email protected]).
Firstly, we compare the results visually. The MS image, PAN image, and the fused images obtained by the algorithms are shown in Figures 3 and 4, respectively. It is clear that AIHS and EIHS methods obtained better spatial information than the other algorithms. Of course, these methods can effectively fuse PAN and MS images, but each fusion image is not the same. In Figures 3 and 4, the spectral information in (e) is poor; the spectral information is rich and the profile is clear in (d), but the spectral resolution is low and there is artifact; there is no obvious disadvantage, but there is no prominent advantage in (c); in (h), spatial information and spectral information are well preserved and visually smooth.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Secondly, we consider the performance metric values, which are listed in Table 1. The evaluation indexes of wavelet method are the worst, which also confirms the poor spectral information of Figures 3(e) and 4(e). Obviously, the EIHS algorithm is the best of all. Even the worst runs of EIHS(2.0) perform better than AIHS, PCA, and wavelet based methods on the metrics except SID. We can also see that EIHS(2.0) is slightly better than EIHS(1.0) and EIHS(1.0) is slightly better than EIHS(0.5). EIHS methods are heuristic methods and they get different results in different runs. However, the performance metric values show that the std. values are relatively small, which means that they are stable for the two problems.
4. Conclusion
In this paper, we proposed an evolutionary approach for pan-sharpening. In the approach, a multispectral image with both high quality of spectral and spatial information is reconstructed from the PAN and MS images based on adaptive IHS method; the optimal control parameters for image fusion are obtained by optimizing an objective function, which considers the relations between the fused image and the given PAN and MS images, through an evolutionary algorithm. The new method was compared with three state-of-the-art pan-sharpening methods on two test images. The test results indicated that our method performed much better than the other methods on most of the performance metrics. The major reason might be that () the assumptions and the proposed objective function are reliable for pan-sharpening tasks and () the evolutionary algorithm succeeded in finding good optimal control parameters to reconstruct the synthesis image. It should be noted that the IHS transformation model is not able to avoid the local dissimilarities between the MS and PAN images [2]; thus we will consider combining other models with evolutionary optimization in the future.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work is supported by National Natural Science Foundation of China (no. 61372147).