Advanced F-Transform-Based Image Fusion

Abstract

We propose to use the modern technique of the F-transform in order to show that it can be successfully applied to the image fusion. We remind two working algorithms (SA—the simple algorithm, and CA—the complete algorithm) which are based on the F-transform and discuss, how they can be improved. We propose a new algorithm (ESA—the enhanced simple algorithm) which is effective in time and free of frequently encountered shortcomings.

1. Introduction

Image processing is nowadays one of the most interesting research areas where traditional and new approaches are applied together and bring significant advantages. In this contribution, we consider the image fusion, which is one of many subjects of image processing. The image fusion aims at integration of complementary distorted multisensor, multitemporal, and/or multiview scenes into one new image which contains the “best” parts of each scene. Thus, the main problem in the area of image fusion is to find the less undistorted scene for every given pixel.

A local focus measure is traditionally used for selection of an undistorted scene. The scene which maximizes the focus measure is selected. Usually, the focus measure is a measure of high frequency occurrences in the image spectrum. This measure is used when a source of distortion is connected with blurring which suppresses high frequencies in an image. In this case, it is desirable that a focus measure decreases with an increase of blurring.

There are various fusion methodologies currently in use. The methodologies differ according to different mathematical fields: statistical methods (e.g., using aggregation operators, such as the MinMax method [1]), estimation theory [2], fuzzy methods (see [3, 4]), optimization methods (e.g., neural networks, genetic algorithms [5]), and multiscale decomposition methods, which incorporate various transforms, for example, discrete wavelet transforms (for a classification of these methods see [6], a classification of wavelet-based image fusion methods can be found in [7], and for applications for blurred and unregistered images, refer to [8]).

In our approach, we propose to use the modern technique of the F-transform and to show that it can be successfully applied to the image fusion. Our previous attempts have been reported in [9–12]. The original motivation for the F-transform (a short name for the fuzzy transform) came from fuzzy modeling [13, 14]. Similarly to traditional transforms (Fourier and wavelet), the F-transform performs a transformation of an original universe of functions into a universe of their “skeleton models” (vectors of F-transform components) in which further computation is easier. Moreover, sometimes, the F-transform can be more efficient than its counterparts. The F-transform proves to be a successful methodology with various applications: image compression and reconstruction [15, 16], edge detection [17, 18], numeric solution of differential equations [19], and time-series procession [20].

The F-transform-based approach to the image fusion has been proposed in [11, 12]. The main idea is a combination of (at least) two fusion operators, both are based on the F-transform. The first fusion operator is applied to F-transform components of scenes and is based on a robust partition of the scene domain. The second fusion operator is applied to the residuals of scenes with respect to inverse F-transforms with fused components and is based on a finer partition of the same domain. Although this approach is not explicitly based on focus measures, it uses the fusion operator which is able to choose an undistorted scene among available blurred. In this contribution, we analyze two methods of fusion that have been discussed in [11, 12] and propose a new method which can be characterized as a weighted combination of those two. We show that •the new method is computationally more effective than the complete algorithm of fusion and has better quality than the simple algorithm of fusion, both have been proposed in [11, 12].

2. F-Transform

Before going into the details of image fusion, we give a brief characterization of the F-transform technique applied herein (we refer to [13] for a complete description).

Generally speaking, the F-transform is a linear mapping from a set of ordinary continuous/discrete functions over domain 𝑃 onto a set of discrete functions (vectors) defined on a fuzzy partition of 𝑃. We assume that the reader is familiar with the notion of fuzzy set and the way(s) of its representation. In this paper, we identify fuzzy sets with their membership functions. In the below given explanation, we will speak about the F-transform of an image function 𝑢 which is a discrete function 𝑢∶𝑃→ℝ of two variables, defined over the set of pixels 𝑃={(𝑖,𝑗)∣𝑖=1,…,𝑁,𝑗=1,…,𝑀} and taking value from the set of reals ℝ. Throughout this text, we will always assume that 𝑀,𝑁, and 𝑢 have the same meaning as above.

Let [1,𝑁]={𝑥∣1≤𝑥≤𝑁} be an interval on the real line ℝ, 𝑛≥2, a number of fuzzy sets in a fuzzy partition of [1,𝑁], and ℎ=(𝑁−1)/(𝑛−1) the distance between nodes 𝑥₁,…,𝑥_𝑛∈[1,𝑁], where 𝑥₁=1, 𝑥_𝑘=𝑥₁+(𝑘−1)ℎ, 𝑘=1,…,𝑛. Fuzzy sets 𝐴₁,…,𝐴_𝑛∶[1,𝑁]→[0,1] establish an ℎ-uniform fuzzy partition of [1,𝑁] if the following requirements are fulfilled. (1)For every 𝑘=1,…,𝑛, 𝐴_𝑘(𝑥)=0 if 𝑥∈[1,𝑁]⧵[𝑥_𝑘−1,𝑥_𝑘+1], where 𝑥₀=𝑥₁, 𝑥_𝑁+1=𝑥_𝑁; (2)for every 𝑘=1,…,𝑛, 𝐴_𝑘 is continuous on [𝑥_𝑘−1,𝑥_𝑘+1], where 𝑥₀=𝑥₁, 𝑥_𝑁+1=𝑥_𝑁; (3)for every 𝑖=1,…,𝑁, ∑^𝑛_𝑘=1𝐴_𝑘(𝑖)=1; (4)for every 𝑘=1,…,𝑛, ∑^𝑁_𝑖=1𝐴_𝑘(𝑖)>0; (5)for every 𝑘=2,…,𝑛−1, 𝐴_𝑘 is symmetrical with respect to the line 𝑥=𝑥_𝑘.

The membership functions of the respective fuzzy sets in a fuzzy partition are called basic functions. The example of triangular basic functions 𝐴₁,…,𝐴_𝑛, 𝑛≥2 on the interval [1,𝑁] is given below: 𝐴₁(𝑥)=1−𝑥−𝑥₁ℎ,𝑥∈𝑥₁,𝑥₂,𝐴0,otherwise,_𝑘(𝑥)=𝑥−𝑥_𝑘ℎ,𝑥∈𝑥_𝑘−1,𝑥_𝑘+1,𝐴0,otherwise,_𝑛(𝑥)=𝑥−𝑥_𝑛−1ℎ,𝑥∈𝑥_𝑛−1,𝑥_𝑛,0,otherwise. (1) Let us remark that(1)the shape (e.g., triangular or sinusoidal) of a basic function in a fuzzy partition is not predetermined and can be chosen according to additional requirements, for example, smoothness, and so forth, see [13]; (2)if the shape of a basic function of a uniform fuzzy partition of [1,𝑁] is chosen, then the basic function can be uniquely determined by the number 𝑛_𝑝 of points, which are “covered” by every “full” basic function 𝐴_𝑘 where 𝑛_𝑝=𝑖∈[1<𝑛]∣𝐴_𝑘(𝑖)>0,𝑘=2,…,𝑛−1, (2) (in this case, we assume that 𝑛≥3).

Similarly, a uniform fuzzy partition of the interval [1,𝑀] with 𝑚≥2 basic functions 𝐵₁,…,𝐵_𝑚 can be defined. Then the fuzzy partition of 𝑃=[1,𝑁]×[1,𝑀] is obtained by 𝑛×𝑚 fuzzy sets 𝐴₁⋅𝐵₁,…,𝐴_𝑛⋅𝐵_𝑚. Below, we will always assume that 𝑛,𝑚 denote quantities of fuzzy sets in fuzzy partitions of [1,𝑁] and [1,𝑀], respectively.

Let 𝑢∶𝑃→ℝ and fuzzy sets 𝐴_𝑘⋅𝐵_𝑙, 𝑘=1,…,𝑛, 𝑙=1,…,𝑚, establish a fuzzy partition of [1,𝑁]×[1,𝑀]. The (direct) F-transform of 𝑢 (with respect to the chosen partition) is an image of the map 𝐹[𝑢]∶{𝐴₁,…,𝐴_𝑛}×{𝐵₁,…,𝐵_𝑚}→ℝ defined by 𝐹[𝑢]𝐴_𝑘⋅𝐵_𝑙=∑^𝑁_𝑖=1∑^𝑀_𝑗=1𝑢𝑖,𝑗𝐴_𝑘(𝑖)𝐵_𝑙𝑗∑^𝑁_𝑖=1∑^𝑀_𝑗=1𝐴_𝑘(𝑖)𝐵_𝑙𝑗, (3) where 𝑘=1,…,𝑛,𝑙=1,…,𝑚. The value 𝐹[𝑢](𝐴_𝑘⋅𝐵_𝑙) is called an F-transform component of 𝑢 and is denoted by 𝐹[𝑢]_𝑘𝑙. The components 𝐹[𝑢]_𝑘𝑙 can be arranged into the matrix representation as follows: 𝐅_𝑛𝑚[𝑢]=𝐹[𝑢]₁₁…𝐹[𝑢]_1𝑚𝐹[𝑢]⋮⋮⋮_𝑛1…𝐹[𝑢]_𝑛𝑚. (4)

The inverse F-transform of 𝑢 is a function on 𝑃, which is represented by the following inversion formula, where 𝑖=1,…,𝑁,𝑗=1,…,𝑀: 𝑢_𝑛𝑚𝑖,𝑗=^𝑛𝑚_𝑘=1𝑙=1𝐹[𝑢]_𝑘𝑙𝐴_𝑘(𝑖)𝐵_𝑙𝑗. (5) It can be shown that the inverse F-transform 𝑢_𝑛𝑚 approximates the original function 𝑢 on the domain 𝑃. The proof can be found in [13, 14].

3. The Problem of Image Fusion

Image fusion aims at the integration of various complementary image data into a single, new image with the best possible quality. The term “quality” depends on the demands of the specific application, which is usually related to its usefulness for human visual perception, computer vision, or further processing. More formally, if 𝑢 is an ideal image (considered as a function of two variables) and 𝑐₁,…,𝑐_𝐾 are acquired (input) images, then the relation between each 𝑐_𝑖 and 𝑢 can be expressed by 𝑐_𝑖𝑥,𝑦=𝑑_𝑖𝑢𝑥,𝑦+𝑒_𝑖𝑥,𝑦,𝑖=1,…,𝐾, (6) where 𝑑_𝑖 is an unknown operator describing the image degradation, and 𝑒_𝑖 is an additive random noise. The problem of fusion consists in finding an image 𝑢 such that it is close to 𝑢 and it is better (in terms of a chosen quality) than any of 𝑐₁,…,𝑐_𝐾. This problem occurs, for example, if multiple photos with focuses on different objects of the same scene are taken.

4. Image Decomposition for Image Fusion

Let us explain the mechanism of fusion with the help of the F-transform. It is based on a chosen decomposition of an image. We distinguish a one-level and a higher-level decomposition. We assume that the image 𝑢 is a discrete real function 𝑢=𝑢(𝑥,𝑦) defined on the 𝑁×𝑀 array of pixels 𝑃={(𝑖,𝑗)∣𝑖=1,…,𝑁,𝑗=1,…,𝑀} so that 𝑢∶𝑃→ℝ. Moreover, let fuzzy sets 𝐴_𝑘⋅𝐵_𝑙, 𝑘=1,…,𝑛, 𝑙=1,…,𝑚, where 2≤𝑛≤𝑁,2≤𝑚≤𝑀 establish a fuzzy partition of [1,𝑁]×[1,𝑀].

We begin with the following representation of 𝑢 on 𝑃: 𝑢𝑥,𝑦=𝑢_𝑛𝑚𝑒𝑥,𝑦+𝑒𝑥,𝑦,0<𝑛≤𝑁,0<𝑚≤𝑀,𝑥,𝑦=𝑢𝑥,𝑦−𝑢_𝑛𝑚𝑥,𝑦, (7)(8) where 𝑢_𝑛𝑚 is the inverse F-transform of 𝑢 and 𝑒 is the respective first difference. If we replace 𝑒 in (7) by its inverse F-transform 𝑒_𝑁𝑀 with respect to the finest partition of [1,𝑁]×[1,𝑀], the above representation can then be rewritten as follows: 𝑢𝑥,𝑦=𝑢_𝑛𝑚𝑥,𝑦+𝑒_𝑁𝑀𝑥,𝑦,∀𝑥,𝑦∈𝑃, (9) We call (9) a one-level decomposition of 𝑢 on 𝑃.

If function 𝑢 is smooth, then the function 𝑒_𝑁𝑀 is small, and the one-level decomposition (9) is sufficient for our fusion algorithm. However, images generally contain various types of degradation that disrupt their smoothness. As a result, the function 𝑒_𝑁𝑀 in (9) is not negligible, and the one-level decomposition is insufficient for our purpose. In this case, we continue with the decomposition of the first difference 𝑒 in (7). We decompose 𝑒 into its inverse F-transform 𝑒_{𝑛′𝑚′} (with respect to a finer fuzzy partition of [1,𝑁]×[1,𝑀] with 𝑛′∶𝑛<𝑛′≤𝑁 and 𝑚′∶𝑚<𝑚′≤𝑀 basic functions, resp.) and the second difference 𝑒′. Thus, we obtain the second-level decomposition of 𝑢 on 𝑃: 𝑢𝑥,𝑦=𝑢_𝑛𝑚𝑥,𝑦+𝑒_{𝑛′𝑚′}𝑥,𝑦+𝑒^′,𝑒𝑥,𝑦_′𝑥,𝑦=𝑒𝑥,𝑦−𝑒_{𝑛′𝑚′}𝑥,𝑦. (10) In the same manner, we can obtain a higher-level decomposition of 𝑢 on 𝑃: 𝑢𝑥,𝑦=𝑢_𝑛1𝑚1𝑥,𝑦+𝑒_𝑛⁽¹⁾_2𝑚2𝑥,𝑦+⋯+𝑒_𝑛^(𝑘−2)_{𝑘−1𝑚𝑘−1}𝑥,𝑦+𝑒^(𝑘−1)𝑥,𝑦, (11) where 0<𝑛₁≤𝑛₂≤⋯≤𝑛_𝑘−1≤𝑁,0<𝑚₁≤𝑚₂≤⋯≤𝑚_𝑘−1𝑒≤𝑀,₍₁₎𝑥,𝑦=𝑢𝑥,𝑦−𝑢_𝑛1𝑚1,𝑒𝑥,𝑦_(𝑖)𝑥,𝑦=𝑒^(𝑖−1)𝑥,𝑦−𝑒_𝑛^(𝑖−1)_𝑖𝑚𝑖𝑥,𝑦,𝑖=2,…,𝑘−1. (12) Below, we will be working with the two decompositions of 𝑢 that are given by (9) and (11).

5. Two Algorithms for Image Fusion

In [12], we proposed two algorithms: (i)the simple F-transform-based fusion algorithm (SA) and (ii)the complete F-transform-based fusion algorithm (CA). These algorithms are based on the decompositions (9) and (11), respectively.

The principal role in fusion algorithms CA and SA is played by the fusion operator 𝜅∶ℝ^𝐾→ℝ, defined as follows: 𝜅𝑥₁,…,𝑥_𝐾=𝑥_𝑝,if𝑥_𝑝=max𝑥₁,…,𝑥_𝐾. (13)

5.1. Simple F-Transform-Based Fusion Algorithm

In this section, we give a “block” description of the SA without technical details which can be found in [12] and not be repeated here. We assume that 𝐾≥2 input images 𝑐₁,…,𝑐_𝐾 with various types of degradation are given. Our aim is to recognize undistorted parts in the given images and to fuse them into one image.(i)Choose values 𝑛, 𝑚 such that 2≤𝑛≤𝑁,2≤𝑚≤𝑀 and create a fuzzy partition of [1,𝑁]×[1,𝑀] by fuzzy sets 𝐴_𝑘⋅𝐵_𝑙,𝑘=1,…,𝑛,𝑙=1,…,𝑚. (ii)Decompose input images 𝑐₁,…,𝑐_𝐾 into inverse F-transforms and error functions according to the one-level decomposition (9). (iii)Apply the fusion operator (13) to the respective F-transform components of 𝑐₁,…,𝑐_𝐾 and obtain the fused F-transform components of a new image. (iv)Apply the fusion operator to the to the respective F-transform components of the error functions 𝑒_𝑖, 𝑖=1,…,𝐾, and obtain the fused F-transform components of a new error function. (v)Reconstruct the fused image from the inverse F-transforms with the fused components of the new image and the fused components of the new error function.

The SA-based fusion is very efficient if we can guess values 𝑛,𝑚, that characterize a proper fuzzy partition. Usually, this is done manually according to user's skills. The dependence on fuzzy partition parameters can be considered as a main shortcoming of this otherwise effective algorithm. Two recommendations follow from our experience. (i)For complex images (with many small details), higher values of 𝑛, 𝑚 give better results. (ii)If a triangular shape of a basic function is chosen, then the generic choice of 𝑛, 𝑚 is such that the corresponding values of 𝑛_𝑝,𝑚_𝑝 are equal to 3 (recall that 𝑛_𝑝 is a number of points, which are covered by every full basic function 𝐴_𝑘).

In this section, the algorithm SA is illustrated on examples “Table” and “Castle,” see Figures 2 and 4 below. There are two inputs of the image “Table” (Figure 1) and four ones of the image “Castle” (Figure 3).

(a)

(b)

Figure 1 

Two inputs of the image “Table.” The toy is blurred in the left image, and vice versa, it is the only sharp part in the right one.

Figure 2 

The SA fusion of the image “Table,” approximate run time: 1,6 sec.

(a)

(b)

(c)

(d)

Figure 3 

Four inputs (sharp zones vary from the northwest to the southeast quatre) of the image “Castle.”

Figure 4 

The SA fusion of the image “Castle,” approximate run time: 1,9 sec.

5.2. Complete F-Transform-Based Fusion Algorithm

The CA-based fusion does not depend on one choice of fuzzy partition parameters (as in the case of the SA), because it runs through a sequence of increasing values 𝑛,𝑚. The description of the CA is similar to that of the SA except for the step 4 which is repeated in a cycle. Therefore, the quality of fusion is high, but the implementation of the CA is rather slow and memory consuming, especially for large images. For illustration, see Figures 5 and 6.

Figure 5 

Table—the CA fusion of the image “Table,” 2 input images, approximate run time: 111 sec.

Figure 6 

Table—the CA fusion of the image “Castle,” 4 input images, approximate run time: 359 sec.

5.3. Fusion Artefacts

In this section, we characterize input images, for which it is reasonable to apply the SA or CA. By doing this, we put restrictions on inputs which are acceptable by the algorithms SA and CA. First of all, input images should be taken without shifting or rotation. Secondly, blurred parts of input images should not contain many small details like leaves on trees, and so forth. If it is so, then the fusion made by SA or CA can leave “artefacts,” like “ghosts” or “lakes,” see the explanation below where we assume that there are two input images for the fusion. (i)Ghosts: this happens when a sharp edge of a nondamaged input image is significantly blurred in the other one. As a result of the SA or CA, the edge is perfectly reconstructed, but its neighboring area is affected by the edge presence (see Figure 7).(ii)Lakes: this may happen in both cases when the fusion is performed by the SA or CA. In the case of SA, a “lake” is a result of choosing neighboring areas with significantly different colors from different input images. In the case of SA, a “lake” is a result of rounding off numbers (see Figure 8).

(a)

(b)

Figure 7 

Example of “ghost”—the white area around the left button (a) and the doubled edge of the roof (b).

Figure 8 

Example of “lake”—the color of this area is significantly different from the respective colors of input images.

6. Improved F-Transform Fusion

The main purpose of this contribution is to create a method which will be as fast as the SA and as efficient as the CA. The following particular goals should be achieved. (i)Avoid running through a long sequence of possible partitions (as in the case of CA). (ii)Automatically adjust parameters of a fusion algorithm according to a level of blurring and a location of a blurred area in input images.(iii)Eliminate situation which can lead to “ghosts” and “lakes” in a fused image.

6.1. Proposed Solution

The main idea of the improved F-transform fusion is to enhance the SA by adding another run of the F-transform over the first difference (7). Our explanation is as follows: the first run of the F-transform is aimed at edge detection in each input image, while the second run propagates only sharp edges (and their local areas) to the fused image. The informal description of the enhanced simple algorithm (ESA) is given in Algorithm 1.

for all input images do  Compute the inverse F-transform  Compute the first absolute difference between the original image and the inverse F-transform of it  Compute the second absolute difference between the first one and its inverse F-transform and set them as weights of pixels end for for all pixels in an image do  Compute the value of sow—the sum of weights over all input images  for all input images do   Compute the value of wr—the ratio between the weight of a current pixel and sow  end for  Compute the fused value of a pixel in the resulting image as a weighted (by wr) sum of input image values end for

Algorithm 1 

Although the algorithm ESA is written for gray scale input images, there is an easy way how to extend it to color images which are represented in RGB or YUV models. Our tests were performed for both of them. In the case of RGB, the respective R, G, or B channels were processed independently and then combined. In the second case of YUV, the Y-part of the model was used to achieve weights (this part contains the most relevant information about the image intensity), while the U-part and the V-part were processed with the obtained weights.

Let us remark that the ESA-fused images are (in general) better than each of the SA or CA. It can be visually seen on the chosen Figures 9 and 10. The main advantages of the ESA are as follows.(i)Time: the executing time is smaller than in the case of the CA (in the examples above it is as follows: 11 versus 111 (“Table”), 18 versus 359 (“Castle”). The quality of the ESA fusion is better than that of the SA. Examples of run times and memory consumption are presented in Table 1 (notice that the memory consumption significantly depends on memory management of implementation environment.) (ii)Ghosts: ghosts effect is reduced. The “ghost” effects (they are seen around the tower roof in the image “Castle” and around the buttons and the clock in the image “Table”) are removed as it can be seen in Figure 11. (iii)Lakes: lakes effect is eliminated. The “lakes” are almost eliminated as it can be seen from Figures 8, 9, and 12.

Table 1 

Basic characteristics of the three algorithms applied to the tested images.

Figure 9 

Table—the ESA fusion of the image “Table,” 2 input images, approximate run time: 10 sec.

Figure 10 

Table—the ESA fusion of the image “Castle,” 4 input images, approximate run time: 18 sec.

(a)

(b)

Figure 11 

Example of  “ghost” artefact reduction by ESA.

Figure 12 

Example of  “lakes” artefact reduction by ESA.

6.2. Comparison between Three Algorithms

In this section, we show that in general, the ESA fusion has better execution parameters than the SA or CA fusion. We experimented with numerous images which due to the space limitation cannot be presented in this paper. An exception is made for one image “Balls” with geometric figures to show how the fusion methods reconstruct edges. In Figure 13, two inputs of the image “Balls” are given, and in Figure 14, three fusions of the same image are demonstrated.

(a)

(b)

Figure 13 

Two inputs of the image “Balls.” The central ball is blurred in (a) and vice versa, it is the only one sharp ball (b).

(a)

(b)

(c)

Figure 14 

The SA (a), CA (b), and ESA (c) fusions of the image “Balls.” The best quality has the ESA fusion (cf. Table 2).

In Table 1, we demonstrate that the complexity (measured by the execution time or by the used memory) of the newly proposed ESA fusion is greater than the complexity of the SA and less than the complexity of the CA.

In Table 2, we demonstrate that the quality of fusion (measured by the values of MSE and PSNR) of the newly proposed ESA fusion is better (the MSE value is smaller) than the quality of the SA and in some cases (the image “Balls”) is better than the quality of the CA. Table 2 does not contain the values of MSE and PSNR for the image “Table,” because (as it happens in reality) there was no original (nondistorted) image at disposal.

Table 2 

MSE and PSNR characteristics of the three fusion methods applied to the tested images.

7. Conclusion

In this paper, we continued our research started in [9–12] on effective fusion algorithms. We proposed the improved method of the F-transform-based fusion which is free from following imperfections: long running time, dependence on initial parameters which characterize a proper fuzzy partition, and presence of fusion artefacts, like “ghosts” or “lakes”.

Acknowledgments

This work relates to Department of the Navy Grant N62909-12-1-7039 issued by Office of Naval Research Global. The US Government has a royalty-free license throughout the world in all copyrightable material contained herein. Additional support was given also by the SGS12/PRF/2012 (Image Processing and Artefacts Detection Using Soft Computing).