A Fast Image Stitching Algorithm via Multiple-Constraint Corner Matching

Abstract

Video panoramic image stitching is in general challenging because there is small overlapping between original images, and stitching processes are therefore extremely time consuming. We present a new algorithm in this paper. Our contribution can be summarized as a multiple-constraint corner matching process and the resultant faster image stitching. The traditional Random Sample Consensus (RANSAC) algorithm is inefficient, especially when stitching a large number of images and when these images have quite similar features. We first filter out many inappropriate corners according to their position information. An initial set of candidate matching-corner pairs is then generated based on grayscales of adjacent regions around each corner. Finally we apply multiple constraints, e.g., their midpoints, distances, and slopes, on every two candidate pairs to remove incorrectly matched pairs. Consequently, we are able to significantly reduce the number of iterations needed in RANSAC algorithm so that the panorama stitching can be performed in a much more efficient manner. Experimental results demonstrate that (i) our corner matching is three times faster than normalized cross-correlation function (NCC) rough match in traditional RANSAC algorithm and (ii) panoramas generated from our algorithm feature a smooth transition in overlapping image areas and satisfy human visual requirements.

1. Introduction

To stitch images and form a video panoramic image, the similarity of overlapping regions among adjacent images needs to be calculated in the first place. State-of-the-art algorithms for image registration (sometimes also referred to as “image alignment”) can be classified into intensity-based, frequency domain-based, and feature-based ones [1–7]. Intensity-based algorithms usually involve a large amount of computation and therefore are not appropriate for image alignment when there is image rotation and scaling. On the other hand, algorithms based on frequency-domain are in general faster and can handle well small translation, rotation, and scaling. Unfortunately, the performance of frequency domain-based algorithms will be degraded when dealing with scenarios where smaller overlapping regions exist. Feature-based algorithms utilize a small number of invariant points, lines, or edges to align images. One significant advantage of these algorithms is that the computational complexity will be reduced due to less information that needs to be processed. Additionally, feature-based algorithms are robust to changes in image intensity. However, there is one serious issue identified for many existing algorithms. Most of these algorithms make use of an exhaustive search that is based on template matching. As a result, the computation, although already decreased to some extent, is still intensive, which does not meet the real-time requirement usually found in video panorama stitching.

We present in this paper a new algorithm to handle the aforementioned challenge. Our algorithm is motivated by the observation that adjacent images usually have small overlapping and small difference of translation, rotation, and scaling between each other. The proposed algorithm is based upon our innovative multiple-constraint corner matching. First, we filter out large numbers of candidate corners according to their position information. We then generate an initial set of matching-corner pairs based on grayscales of each corner’s adjacent regions. Finally, multiple constraints, for example, their midpoints, distances, and slopes, will be applied on every two candidate pairs to remove incorrectly matched pairs. Consequently, we are able to significantly reduce the number of iterations that are needed in conventional Random Sample Consensus (RANSAC) algorithm [8]. As a result, the video panoramic image stitching can be performed a lot more efficiently.

The rest of this paper is organized as follows. Section 2 introduces in detail our methodology; Section 3 describes experimental results; and Section 4 concludes with future research directions.

2. Methodology

2.1. Corner Selection

Harris algorithm [2] detects corners through the differential of the corner score and the autocorrelation matrix. Suppose that an image has an intensity of 𝐼(𝑥,𝑦) and an image patch over the area 𝑤(𝑥,𝑦) is shifted by (𝑢,𝑣) the intensity change, 𝐸(𝑢,𝑣), of the pixel (𝑥,𝑦) can then be calculated by (1) where 𝑀=^𝐴𝐶_𝐶𝐵=𝑤(𝑥,𝑦)⊗^{𝐼2𝑥𝐼𝑥𝐼𝑦}_{𝐼𝑥𝐼𝑦𝐼2𝑦}  (𝐼(𝑥) and 𝐼(𝑦) are partial derivatives of the pixel, respectively, and 𝑤 is the Gaussian function to filter noises). One has 𝐸()=[]𝑀𝑢𝑣𝑢,𝑣𝑢𝑣. (1) The corner response function is defined in (2) with 𝑘 in the range of [0.04, 0.06]. Any pixels whose 𝑅 value is greater than a threshold, 𝜆, can be selected as candidate corners: 𝑅=det(𝑀)−𝑘(trace(𝑀))². (2) Note that 𝜆 depends on characteristics of actual images, size and texture for example; Usually 𝜆 is determined indirectly: pixels are sorted in a descending order of their 𝑅 values; then the first Sum pixels are selected as corners.

Harris detector only involves the first order difference and filtering operations of pixel grayscale, with low computational complexity. A large number of corners can be detected in regions with rich texture, whereas fewer corners will be selected in regions with less texture information. Therefore, selected corners are not evenly distributed; that is, corners tend to cluster around regions with richer texture. Zhao et al. proposed an algorithm in [9] where they fragmented the original image into several regions. A fixed number of corners with top 𝑅 values were selected in each region as candidate corners; all such candidate corners (a total of Sum) were then sorted in their 𝑅 values. Finally a scaling parameter, 𝑘, whose range is (0, 1), was applied to finalize the corner selection, that is, generating a total of 𝑘×Sum corners. To assure that each region contains some finalized corners, this algorithm iteratively applied different 𝑘 values in an ascending order and the iteration was terminated as soon as there was at least one finalized corner for each region. Because of its ability to select corners with relatively high quality, we adopt this algorithm when selecting Harris corners from adjacent images to be stitched.

2.2. Multiple-Constraint Corner Matching

The traditional RANSAC algorithm is inefficient, especially when stitching a large number of images and when these images have quite similar features. Thus, it does not meet the real-time requirement commonly found in video panorama stitching. Note that in the field of video panorama stitching, more often than not, adjacent images have highly similar features with each other, that is, small difference of translation, rotation, and scaling between each other. Based on this insight, we propose to apply multiple constraints on candidate matching-corner pairs to remove incorrectly matched pairs. As such, we can significantly reduce the number of iterations needed in RANSAC algorithm.

2.2.1. Create a Corner Similarity Matrix between Adjacent Images

Suppose that the image 𝐼 has a resolution of 𝑊×𝐻, and the 𝑘th corner in 𝐼 is denoted by 𝐼_𝑘 (with coordinates 𝐼_𝑘⋅𝑥 and 𝐼_𝑘⋅𝑦, and intensity 𝐼(𝑥_𝑘,𝑦_𝑘)). One corner from the left image (𝐼^𝑙_𝑖) and another corner from the right image (𝐼^𝑟_𝑗) can be matched with each other if the following conditions are satisfied: (i) the difference between 𝑦 coordinates of these two corners is no greater than 𝐻/3; (ii) the 𝑥 coordinate of the left corner is greater than or equal to that of the right corner; and (iii) there is a high intensity correlation between two corners. Accordingly, we utilize (3) to calculate pairwise corner similarity and create a similarity matrix between adjacent images 𝐼^𝑙 and 𝐼^𝑟: =𝐼sim𝑖,𝑗NCC^𝑙_𝑖,𝐼^𝑟_𝑗if𝐼^𝑙_𝑖⋅𝑦−𝐼^𝑟_𝑗⋅𝑦<𝜆_ℎ,𝐼^𝑙_𝑖⋅𝑥≥𝐼^𝑟_𝑗⋅𝑥,0else. (3) In (3), 𝜆_ℎ is the threshold of the difference between 𝑦 coordinates of two corners, and normalized cross-correlation (NCC) function is the one described in [10]. Suppose that the similarity window size is (2𝑤+1)×(2𝑤+1); NCC is then calculated as NCC𝐼^𝑙_𝑖,𝐼^𝑟_𝑗=∑^𝑤_𝑢=−𝑤∑^𝑤_𝑣=−𝑤𝐷_𝑙(𝑖,𝑢,𝑣)⋅𝐷_𝑟𝑗,𝑢,𝑣∑^𝑤_𝑢=−𝑤∑^𝑤_𝑣=−𝑤𝐷_𝑙(𝑖,𝑢,𝑣)²⋅∑^𝑤_𝑢=−𝑤∑^𝑤_𝑣=−𝑤𝐷_𝑟𝑗,𝑢,𝑣², (4) where 𝐷_𝑙(𝑖,𝑢,𝑣)=𝐼^𝑙𝑥_𝑖+𝑢,𝑦_𝑖+𝑣−𝐼^𝑙_𝑖,𝐷_𝑟𝑗,𝑢,𝑣=𝐼^𝑟𝑥_𝑗+𝑢,𝑦_𝑗+𝑣−𝐼^𝑟_𝑗, (5) and 𝐼^𝑙_𝑖 and 𝐼^𝑟_𝑗 are the mean intensity of windows around corners 𝐼^𝑙_𝑖 and 𝐼^𝑟_𝑗, respectively. In addition, we further filter out corner pairs with low similarity using (6), where 𝜆_𝑛 is the similarity threshold (a real number that is greater than 0.5): sim𝑖,𝑗=sim𝑖,𝑗ifsim𝑖,𝑗>𝜆_𝑛,0else. (6) In brief, we use (3) and (6) to calculate pairwise corner similarity, sim(𝑖,𝑗), resulting in a similarity matrix between two adjacent images.

2.2.2. Generate an Initial Set of Matching-Corner Pairs

A set of indexes of matching-corner pairs is generated by the following procedure: in each row of the similarity matrix obtained previously, we find the column index so that the corresponding cell in the matrix has the maximum value for that row, and the pair of (row index, column index) is added into the set. After we process all rows in the matrix, we will obtain a set of index pairs, 𝐿^𝑙. Such a procedure is formally described in (7), where Sum_𝑙 is the predefined total number of corners in the left image 𝐼^𝑙: 𝐿^𝑙=𝑖,𝑗∣∀𝑖∈1,Sum_𝑙,𝑠𝑖,𝑗=max(𝑠(𝑖,∶)),𝑠𝑖,𝑗≠0. (7) Similarly, we can obtain another set of index pairs, 𝐿^𝑟, by searching the maximum row index for each column. Equation (8) is a formal description of this procedure, where Sum_𝑟 is the predefined total number of corners in the right image 𝐼^𝑟: 𝐿^𝑟=𝑖,𝑗∣∀𝑗∈1,Sum_𝑟,𝑠𝑖,𝑗=max𝑠∶,𝑗,𝑠𝑖,𝑗≠0. (8) In general, Sum_𝑙 in (7) and Sum_𝑟 in (8) can take different values. In our algorithm we use the same value for these two parameters. Now we compare two sets, 𝐿^𝑙 and 𝐿^𝑟. If a row index and a column index happen to have each other as the other component in a pair, their similarity will be adjusted to 1. That is, if two corners mutually find their “best” match as each other, such a pair will have an updated similarity value of 1. Equation (9) formalizes this procedure of similarity adjustment: sim_updated𝑖,𝑗=1if𝑖,𝑗∈𝐿^𝑙∩𝐿^𝑟,sim𝑖,𝑗else. (9) Finally we generate an initial set of matching-corner pairs, 𝐿, by a union of 𝐿^𝑙 and 𝐿^𝑟, shown in (10). Note that this initial set of pairs is already reduced in size compared with NCC rough match in traditional RANSAC algorithm because as shown in (3) we have already filtered out some inappropriate corners according to their positions in respective regions (i.e., their coordinate values). One has 𝐿=𝐿^𝑙∪𝐿^𝑟=(𝑙,𝑟)∣(𝑙,𝑟)∈𝐿^𝑙or(𝑙,𝑟)∈𝐿^𝑟. (10)

2.2.3. Apply Multiple Constraints on Matching-Corner Pairs

Consider two initial matching-corner pairs in Figure 1, (𝑙_𝑚,𝑟_𝑚) and (𝑙_𝑛,𝑟_𝑛), along with their respective midpoints, that is, 𝑙_𝑚𝑛 between 𝑙_𝑚 and 𝑙_𝑛 and 𝑟_𝑚𝑛 between 𝑟_𝑚 and 𝑟_𝑛. Let 𝛿_𝑚 and 𝑚 be the slope and length of the segment formed by 𝑙_𝑚 and 𝑟_𝑚, respectively, and let 𝛿_𝑛 and 𝑛 be the slope and length of the segment formed by 𝑙_𝑛 and 𝑟_𝑛, respectively. We design three constraints to be applied to these two matching-corner pairs, as follows: constraint1:𝛿_𝑚−𝛿_𝑛<𝜆_𝛿,constraint2:|𝑚−𝑛|<𝜆_𝑑,constraint3:NCC𝐼^𝑙_𝑙𝑚𝑛,𝐼^𝑟_𝑟𝑚𝑛>𝜆_𝑛. (11) The intuition of (11) is that, between two matching pairs, not only the intensity of their respective midpoints (constraint 3) should be correlated, but also the slope (constraint 1) and length (constraint 2) of the segments formed between these two pairs should be similar with each other as well. According to multiple constraints specified in (11), we calculate pairwise similarity between every two initial matching pairs using (12) and generate a matrix 𝐷 of size 𝐾×𝐾, with 𝐾 being the cardinality of 𝐿 generated in (10). One has 𝐷()=𝑚,𝑛sim_updated𝑙_𝑚,𝑟_𝑚+sim_updated𝑙_𝑛,𝑟_𝑛+NCC𝐼^𝑙_𝑚𝑛,𝐼^𝑟_𝑚𝑛3,ifsatifyingall3constraints0else. (12)

Figure 1 

Two initially matching-corner pairs.

2.2.4. Generate the Final, Reduced Set of Matching-Corner Pairs

Among a total of 𝐾 initial matching-corner pairs, according to (13), we search for a special pair, 𝑡, which has the strongest correlation with all other pairs: 𝑡=argmax_{[]𝑖∈1,𝐾}^𝐾_𝑗=1𝐷𝑖,𝑗. (13) Then we refer back to the matrix 𝐷 generated previously and find out all initial matching pairs that have some correlation with the aforementioned special pair, 𝑡; that is, an initial matching-corner pair will be output to the final, further reduced set as long as the cell in 𝐷 corresponding to this pair and the special pair 𝑡 has a nonzero value. Equation (14) formally specifies this final selection step, and the resultant set 𝑇 is the finalized, reduced set of matching-corner pairs. Note that the size of 𝑇 is further reduced from that of 𝐿, and we explained earlier that 𝐿 is already reduced in size compared with NCC rough match in traditional RANSAC algorithm: 𝑇={𝑙∣∀𝑙∈𝐿,𝐷(𝑡,𝑙)≠0}. (14)

2.3. Image Stitching

After we obtain a reduced set of matching-corner pairs between two original images to be stitched, we select one as the reference image and calculate the affine transformation parameters using RANSAC algorithm. Based on these parameters we map pixel coordinates in the other image into the coordinate system of the reference image. The light conditions may vary among different cameras; therefore, the panorama to be generated may be inconsistent in terms of its intensity. To obtain a smooth transition in overlapping areas among images to be stitched, we utilize the weighted-sum method introduced in [10] to perform a gradual fading-in and fading-out image stitching process to generate the final video panoramic image.

3. Experimental Results and Analysis

3.1. Experimental Environment and Parameter Setup

Experimental Environment and Parameter Setup are as follows: PC: CPU E2200 + 2.2 GHz, 4 GB memory, Matlab 7.0; image resolution: 1280 × 720.

Various parameters described earlier in Section 2 were set as follows. Note that the setting of these experimental parameters was based upon our previous experience from numerous experiments.(i)The difference of 𝑦 coordinates of adjacent cameras was not greater than 𝐻/3; that is, 𝜆_ℎ in (3) was set to 𝐻/3;(ii)The horizontal overlapping was not great than 𝑊/3;(iii)The original image was segmented into regions of size 80 × 80, and the number of corners for each region was set to six;(iv)The similarity threshold, 𝜆_𝑛, in (6) was set to 0.75, and the similarity window size in (4) was set to 7×7; that is, 𝑤 was set to three.

3.2. Evaluation on Corner Matching

The experimental results are demonstrated in Figure 2. Two original images with corners selected using algorithm in [9] are exhibited in Figure 2(a). We chose the right one-third region of the left image and the left one-third region of the right image as two regions to perform corner matching. So we had a total of 45(=(𝐻/80)×((𝑊/3)/80)) segmented regions, and the total number of corners is 270(=6×45). The total number of matching-corner pairs from NCC rough match in traditional RANSAC algorithm was 388 (Figure 2(b)), whereas the total numbers of initial and finalized matching-corner pairs from our algorithm were 332 (Figure 2(c)) and 35 (Figure 2(d)), respectively. This result verified our earlier discussion in Section 2.2; that is, the initial set of pairs is already reduced in size compared with NCC rough match in traditional RANSAC algorithm because as shown in (3) we have already filtered out some corners according to their positions in respective regions (i.e., their coordinate values). Note that most of the 35 matching pairs in Figure 2(d) were correct ones. In addition, as demonstrated in Table 1, our multiple-constraint corner matching was three times faster than NCC rough match in traditional RANSAC algorithm. The reason for us to obtain a much shorter matching process is that traditional RANSAC algorithm needs to calculate the NCC function, which is very time consuming, for all pairwise combinations of corners, whereas in our algorithm only a small number of combinations need to be considered. To be more specific, (3) ignores all corners that do not meet the position requirement, and we further avoid NCC calculation if two initial pairs do not satisfy the first two constraints specified in (11). More experimental results can be found at the following project Web link: http://www.soc.southalabama.edu/~huang/ImageStitching/ExperimentResults.rar.

Table 1 

Efficiency comparison between traditional RANSAC and our algorithm.

(a)

(b)

(c)

(d)

Figure 2 

Experiments on corner matching.

3.3. Evaluation on Image Stitching

The experimental results are demonstrated in Figure 3. We performed both the regional Harris corner selection and multiple-constraint corner matching between Figures 3(a) and 3(b) and between Figures 3(b) and 3(c), respectively. After we obtained two finalized sets of matching-corner pairs, we selected Figure 3(b) as the reference image and calculated the affine transformation parameters as discussed earlier in Section 2.3. We then mapped pixel coordinates in Figures 3(a) and 3(c) into the coordinate system of Figure 3(b), respectively. Finally we performed a gradual fading-in and fading-out image stitching process. The final result in Figure 3(d) clearly demonstrated that (i) our corner matching was accurate; (ii) we obtained a smooth transition in overlapping areas among images to be stitched; and (iii) the panorama generated satisfied human visual requirements. Similarly, more experimental results can be found at the following link: http://www.soc.southalabama.edu/~huang/ImageStitching/ExperimentResults.rar.

(a)

(b)

(c)

(d)

Figure 3 

Experiments on image stitching.

4. Conclusions

We presented an innovative algorithm to handle challenges in video panoramic image stitching, for example, small overlapping regions and extremely time-consuming stitching processes. Our contribution can be summarized as (i) a multiple-constraint corner matching and (ii) a more efficient image stitching process. To overcome the inefficient corner matching in traditional RANSAC algorithm, we first filtered out a large number of corners according to their position information. We then generated an initial set of matching-corner pairs based on grayscales of adjacent regions around each corner. Finally we applied multiple constraints on every two candidate pairs to remove incorrectly matched pairs. We were able to significantly reduce the number of iterations needed in RANSAC algorithm, resulting in a much more efficient panorama stitching process. Experimental results (both those that were detailed in this paper itself and those additional ones in the Web link provided) demonstrated that (i) our corner matching is three times faster than traditional RANSAC matching and (ii) panoramas generated from our algorithm feature a smooth transition in overlapping image areas and satisfy human visual requirements.

One possible future research direction is to investigate on automatically determining the total number of corners according to the image texture information. Another interesting future work is to handle the motion ghost challenge during image stitching.

Acknowledgment

This research was supported by the Project of Fujian Province under Grant nos. 2011Y0040, 2012J01263, and 2013J01186.