Frame Interpolation Based on Visual Correspondence and Coherency Sensitive Hashing

Abstract

The technology of frame interpolation can be applied in intelligent monitoring systems to improve the quality of surveillance video. In this paper, a region-guided frame interpolation algorithm is proposed by introducing two innovative improvements. On the one hand, a detection approach is presented based on visual correspondence for detecting the motion regions that correspond to attracted objects in video sequences, which can narrow the prediction range of interpolated frames. On the other hand, spatial and temporal mapping rules are proposed using coherency sensitive hashing, which can obtain more accurate predicted values of interpolated pixels. Experiments show that the proposed method can achieve encouraging performance in terms of visual quality and quantitative measures.

1. Introduction

Frame interpolation plays a very important role in intelligent monitoring systems. This technology can not only increase the frame rate of surveillance video to meet the requirements of monitoring display devices [1] but also forecast the missing frames to obtain smooth surveillance video. The essence of frame interpolation is to predict intermediate frames by using two given frames in video sequences [2], and it can be roughly divided into two categories: the optical flow method and the block matching method. The former uses time-domain variation and correlation of pixel intensity data to determine the location and value of each pixel [3]. However, in practice, because of the existence of a variety of factors such as multiple light levels, transparency, and noise, assumptions of brightness constancy and spatial smoothness in the optical flow basic equations cannot be always satisfied [4, 5]. The latter finds the best match for each block by minimizing the matching difference [6, 7]. Due to its simplicity, this method is easy to implement, but blocking artifacts will occur in the interpolation frames [8]. In brief, these existing methods still suffer from poor quality of video frames. Therefore, it would be beneficial to develop an algorithm to obtain high-quality interpolation frames.

In this paper, a novel region-guided frame interpolation algorithm (RGFI) is proposed, the goal of which is to improve the definitions of intermediate frames of the surveillance video. To achieve this, two techniques are introduced for frame interpolation. The first technique is visual correspondence using local descriptor matching [9, 10]. For this purpose, compact real-time descriptors (CARD) [11] are used, an approach which was proposed recently to establish visual correspondence quickly between two images. Moreover, the computation time of each descriptor is approximately 16 times faster than that of the scale-invariant feature transform (SIFT) [12]. The second technique is the latest approximate nearest neighbor (ANN) method [13], called coherency sensitive hashing (CSH) [14], which was proposed to find matching patches quickly. CSH relies on hashing to propagate information through similarity in appearance space and neighborhood in the image plane. Its advantage is the use of special observations to establish candidate block sets, which can avoid many artifacts along edges when reconstructing an original image. Through the advanced techniques described above, the interpolation task for video sequences is achieved satisfactorily from a different perspective.

The main contributions of this paper can be summarized as follows: (1) a novel and comprehensive frame interpolation framework based on spatial and temporal correlations in video sequences and (2) a detection approach based on visual correspondence to capture motion regions so as to narrow the prediction range of interpolation frames. The foundation of the detection approach is the frame difference technique and CARD technique, (3) the spatial and temporal block definition and spatial and temporal mapping relationship definition for better implementation of the RGFI algorithm. On this basis, this approach is combined with CSH to construct spatial and temporal mapping rules so as to predict values of interpolated pixels accurately, (4) application of the proposed algorithm to video resizing. Using accurate motion regions obtained in the proposed algorithm, the important content in the surveillance video can be preserved on the premise of ensuring global visual effect and can be displayed on the monitoring display devices with different resolutions.

The rest of the paper is structured as follows. Section 2 provides an overview of the proposed RGFI algorithm. Section 3 demonstrates the implementation details of RGFI. Section 4 presents experimental work carried out to demonstrate the effectiveness of the algorithm. Section 5 shows another contribution of our algorithm. Section 6 concludes the paper.

2. RGFI Overview

In this paper, a novel RGFI algorithm is proposed according to characteristics of video sequences. The RGFI framework is shown in Figure 1 and is divided into motion region detection and interpolated pixel computation. First, the detection approach using visual correspondence to capture motion regions is presented. Then the spatial and temporal mapping scheme based on CSH is demonstrated to compute the unknown pixel values in the motion regions obtained. At the same time, for the pixels of other regions of input video frames, only the original values are kept. Through these two steps, high-quality interpolation frames are produced.

Figure 1 

Architecture of the proposed algorithm.

3. Implementation of RGFI

3.1. Motion Region Detection Using Visual Correspondence

The motion regions of the surveillance video are very important in the intelligent monitoring system. Therefore, by finding these regions, it is possible to speed up the accomplishment of the interpolation task. Unlike previous methods, the method proposed here takes advantage of visual correspondences based on CARD to obtain motion regions between video frames. The detailed implementation of the detection approach includes three parts: motion region initial estimation, key point correspondence establishment, and motion region determination.

Because the frame difference method [15] can quickly find the outline of the moving target in video sequences, it is first used to estimate rough motion regions, as shown in the following: 𝐼_𝑖𝑥,𝑦∈𝑀^′_𝑛,if𝑓_𝑛+1𝑥,𝑦−𝑓_𝑛𝑥,𝑦≥𝐷^𝑛×𝑐, (1) where 𝑓_𝑛(𝑥,𝑦) and 𝑓_𝑛+1(𝑥,𝑦) denote the two consecutive frames of the input video sequences, 𝑀^′_𝑛 is the initial motion region corresponding to 𝑓_𝑛(𝑥,𝑦), 𝑐 is a predefined value, and 𝐷^𝑛 is the difference between the maximum value and the minimum value in two consecutive frames as follows: 𝐷^𝑛=max𝑓_𝑛+1𝑥,𝑦−𝑓_𝑛𝑓𝑥,𝑦−min_𝑛+1𝑥,𝑦−𝑓_𝑛𝑥,𝑦. (2)

Second, inspired by [11], comparatively accurate key point correspondences are established in the initial motion regions; the establishment process includes the following four steps.

Step  1.  Construct an image pyramid [16] for the initial regions as follows: 𝐿𝑥_𝑚,𝑦_𝑚,𝑛,𝜎−1=𝐿_2↓𝑥_𝑚,𝑦_𝑚,𝑛,𝜎, (3) where 𝐿(𝑥_𝑚,𝑦_𝑚,𝑛,𝜎) is a scale function, 𝑥_𝑚 and 𝑦_𝑚 are the values of the abscissa and the ordinate, 𝜎 is the level of the image pyramid, and 2↓ is the downsampling factor.

Step  2.  A corner detector technique [17] is used to find the key points in the image pyramid, and the corresponding point set is denoted as 𝐾_𝑛={𝑘_𝑛(𝑥_𝑘1,𝑦_𝑘1)∣(𝑥_𝑘1,𝑦_𝑘1)∈𝑀^′_𝑛}. Similarly, the key point set 𝐾_𝑛+1 corresponding to the initial region 𝑀^′_𝑛+1 of 𝑓_𝑛+1(𝑥,𝑦) is obtained.

Step  3.   For ∀𝑘_𝑛′(𝑥_𝑖,𝑦_𝑖)∈(𝐾_𝑛∪𝐾_𝑛+1), the orientation histogram HG(ℎ^𝑛′_𝑘) is determined; the corresponding magnitudes and orientations are given in (4). At the same time, log-polar binning is used to achieve good discrimination ability [18], as follows 𝑚_HG𝑥_𝑖,𝑦_𝑖=𝑀^′_𝑛𝑥_𝑖+1,𝑦_𝑖−𝑀^′_𝑛𝑥_𝑖−1,𝑦_𝑖²+𝑀^′_𝑛𝑥_𝑖,𝑦_𝑖+1−𝑀^′_𝑛𝑥_𝑖,𝑦_𝑖−1^21/2,𝜔_HG𝑥_𝑖,𝑦_𝑖𝑀=arctg^′𝑛𝑥_𝑖+1,𝑦_𝑖−𝑀^′_𝑛𝑥_𝑖−1,𝑦_𝑖𝑀^′_𝑛𝑥_𝑖,𝑦_𝑖+1−𝑀^′_𝑛𝑥_𝑖,𝑦_𝑖,𝑟−1_𝑘,𝑛′=0,0≤𝑢_𝑖²+𝑣_𝑖²<𝜑₁1+𝑄₈𝑢arctg^𝑖𝑣_𝑖−2𝜋𝑙_𝑘mod𝐿𝐿,𝜑₁≤𝑢²_𝑖+𝑣²_𝑖<𝜑₂9+𝑄₈𝑢arctg^𝑖𝑣_𝑖−2𝜋𝑙_𝑘mod𝐿𝐿,𝜑₂≤𝑢²_𝑖+𝑣²_𝑖<𝜑₃, (4)(5) where 𝑟_𝑘,𝑛, denotes a rotated binning pattern, 𝐿 is a predefined value, 𝜑₁, 𝜑₂, and 𝜑₃ are constants, 𝑢_𝑖 and 𝑣_𝑖 denote the coordinate values of 𝑘_𝑛′(𝑥_𝑖,𝑦_𝑖) in relative coordinates, 𝑄_𝐹(𝜃) represents the quantization function, given in (6), and 𝑙_𝑘 denotes the quantization result of 𝜔_HG(𝑥_𝑖,𝑦_𝑖). On this basis, a spatial binning table [19] is used to extract HG(ℎ^𝑛′_𝑘) and to obtain the descriptor 𝑑^𝑛_𝑘 of 𝑘_𝑛′(𝑥_𝑖,𝑦_𝑖), setting the descriptor set 𝐷^𝑛_𝑘𝑄_𝐹(𝜃)=𝐹𝜃+𝜋2𝜋mod𝐹. (6)

Step  4. For ∀𝑑^𝑛_𝑘(𝑑^𝑛_𝑘∈𝐷^𝑛_𝑘), compute its corresponding descriptor 𝑑_𝑚^𝑛+1 in the next frame according to the following: 𝑑_𝑚^𝑛+1=argmin_{𝑑𝑖𝑛+1∈𝐷𝑘𝑛+1}disc𝑑_𝑖^𝑛+1−𝑑^𝑛_𝑘=argmin_{𝑑𝑖𝑛+1∈𝐷𝑘𝑛+1}𝑑_𝑖^𝑛+1−𝑑^𝑛_𝑘1=argmin_{𝑖∈𝐾𝑛}^𝐶_𝑗=1𝛿_𝑖^𝑛+1𝑗−𝛿^𝑛_𝑘𝑗𝛿^𝑛_𝑘=1+sgn𝑊^𝑇𝑑^𝑛_𝑘2, (7) where 𝛿^𝑛_𝑘 is the changed short binary code of any descriptor 𝑑^𝑛_𝑘, 𝐶 is the length of changed binary code, and 𝑊 is a weight matrix. In this way, the corresponding set 𝐷_𝑚^𝑛+1 of descriptor set 𝐷^𝑛_𝑘 and the matching key point 𝐾_𝑚 of 𝐾_𝑛 can be obtained.

Finally, more accurate boundary values for the motion regions can be determined according to the locations of key points and their correspondences between consecutive frames. Let 𝐾=𝐾_𝑛∪𝐾_𝑚; the boundary values of motion region 𝑀^𝑛 can then be computed using the following equation: 𝑀^𝑛_𝐿=min_{𝑘𝑛𝑥𝑛𝑖,𝑦𝑛𝑖∈𝐾}𝑥^𝑛_𝑖−𝜀_𝐿,𝑀^𝑛_𝑅=max_{𝑘𝑛𝑥𝑛𝑖,𝑦𝑛𝑖∈𝐾}𝑥^𝑛_𝑖+𝜀_𝑅,𝑀^𝑛_𝑇=min_{𝑘𝑛𝑥𝑛𝑖,𝑦𝑛𝑖∈𝐾}𝑦^𝑛_𝑖−𝜀_𝑇,𝑀^𝑛_𝐵=max_{𝑘𝑛𝑥𝑛𝑖,𝑦𝑛𝑖∈𝐾}𝑦^𝑛_𝑖+𝜀_𝐵, (8) where 𝑀^𝑛_𝐿, 𝑀^𝑛_𝑅, 𝑀^𝑛_𝑇, and 𝑀^𝑛_𝐵 are the left border, right border, top border, and bottom border and 𝜀_𝐿, 𝜀_𝑅, 𝜀_𝑇,  and 𝜀_𝐵 are the predefined minimal deviation values.

Figure 2 shows one example of motion region detection. Figures 2(a) and 2(b) are the two consecutive frames of the walk video, and the key point correspondences between them are presented. From the results for the motion region shown in Figure 2(c), it is evident that the proposed detection approach can perform well in expressing the motion content of video sequences, which lays the foundation for interpolated pixel computation.

(a)

(b)

(c)

Figure 2 

One example of motion region detection. The first two figures from the left show the two consecutive images of the walk video; red lines are used to denote key point correspondences. Marked with yellow lines, the obtained motion region in the last figure contains key points denoted by blue dots.

3.2. Interpolated Pixel Computation Using Coherency Sensitive Hashing

To facilitate the computation of interpolated pixels, it is necessary to define some concepts in advance.

Definition 1. For obtained motion regions in two consecutive frames, the original motion region 𝑀^𝑛 and the mapping motion region 𝑀^𝑛+1 are defined. 𝑀^𝑛 or 𝑀^𝑛+1 is divided into 𝑠×𝑠 overlapping image blocks, and each block is defined as a spatial and temporal block, ST for short. Denote the total numbers and sets of ST as 𝑇_𝑀^𝑛,  𝑇_𝑀^𝑛+1, and 𝐵^𝑛={𝑏^𝑛_𝑘},  𝐵^𝑛+1={𝑏_𝑘^𝑛+1}, respectively.

Definition 2. Given ∀𝑏^𝑛_𝑘(𝑏^𝑛_𝑘∈𝐵^𝑛),  assume that 𝑏_𝑙^𝑛+1(𝑏_𝑙^𝑛+1∈𝐵^𝑛+1) is the corresponding mapping block of 𝑏^𝑛_𝑘 and that it is computed through spatial and temporal mapping rules. Then 𝑏^𝑛_𝑘↦𝑏_𝑙^𝑛+1 is defined as the spatial and temporal mapping relationship between 𝑏^𝑛_𝑘 and 𝑏_𝑙^𝑛+1.

As far as is known, the block matching method generally finds the only match for each original divided block. However, the computation method proposed here can obtain two or more matching blocks for each ST so as to approach more closely the true values of interpolated pixels. This computation method includes four parts: ST projection and conversion, mapping block computation, nearest block determination, and unknown pixel computation.

First, the projection of ST is computed on Walsh-Hadamard kernels [20]. Specifically, for each ST 𝑏_𝑗, 𝑏_𝑗∈(𝐵^𝑛∪𝐵^𝑛+1), gray-code filter kernels [21] are computed as the transform kernel, as shown in (9) and Figure 3. The results are stored in the temporary set 𝐻={𝑊𝐻𝑇_𝑗∣1≤𝑗≤𝑇^𝑁}, in which 𝑇^𝑁=𝑇_𝑀^𝑛+𝑇_𝑀^𝑛+1: 𝑉⁽⁰⁾_𝑠1=𝑉⁽⁰⁾_𝑠2𝑉=1_(𝑘)𝑠1=𝑣^(𝑘−1)_𝑠1𝛼_𝑘𝑣^(𝑘−1)_𝑠1𝑉^(𝑘)_𝑠2=𝑣^(𝑘−1)_𝑠2𝛼_𝑘𝑣^(𝑘−1)_𝑠2𝑉_𝑠^(𝑘)=𝑉^(𝑘)_𝑠1×𝑉^(𝑘)_𝑠2s.t.𝑣^(𝑘−1)_𝑠1∈𝑉^(𝑘)_𝑠1,𝑣^(𝑘−1)_𝑠2∈𝑉^(𝑘)_𝑠2,𝛼_𝑘∈{+1,−1}. (9)

Figure 3 

Tree scheme for computing projections of a 2D signal onto WH kernels of order 8. In the Walsh-Hadamard kernel array, yellow denotes the value 1 and blue denotes the value −1.

To accelerate ST mapping, a hash value is assigned to each transform result, and each hash function ℎ_𝑎,𝑏(𝑊𝐻_𝑗)∶𝑅^dim→𝑁 maps a dim-dimensional vector 𝑊𝐻_𝑗 onto the set of integers [22]. In this way, the corresponding hash table 𝑡_𝑚(1≤𝑚≤𝑇^𝑁) is constructed, and each ST is saved in the entry 𝑡_𝑚[𝑔_𝑚(𝑏_𝑗)], in which 𝑔_𝑚(𝑏_𝑗) is given by the following: 𝑔_𝑚𝑏_𝑗=𝑊𝐻𝑇₁⋅𝑏_𝑗+𝜌₁𝛾∘𝑊𝐻𝑇₂⋅𝑏_𝑗+𝜌₂𝛾∘⋯∘𝑊𝐻𝑇_𝑗⋅𝑏_𝑗+𝜌_𝑗𝛾∘⋯∘𝑊𝐻𝑇_𝑇^𝑁⋅𝑏_𝑗+𝜌_𝑇𝑁𝛾. (10) In (10), 𝛾 is a fixed integer value which is set in advance and 𝜌_𝑗 is a random number according to a uniform distribution on the interval [0,𝛾). At the same time, for the two vectors 𝑊𝐻_𝑙, 𝑊𝐻_𝑙′, let 𝑐=‖𝑊𝐻_𝑙−𝑊𝐻_𝑙′‖_𝑝; they are hashed to the same value with probability 𝑃(𝑐): 𝑃(𝑐)=Pr_𝑎,𝑏ℎ_𝑎,𝑏𝑊𝐻_𝑙=ℎ_𝑎,𝑏𝑊𝐻_𝑙′=^𝑟₀1𝑐𝑓_𝑝𝑡𝑐𝑡1−𝑟𝑑𝑡. (11)

Then spatial and temporal mapping rules are established for ST to expand the number of mapping blocks of ST. Here four mapping rules are established, shown as Rules 1–4. The first three mapping rules take advantage of a coherency sensitive hashing technique [14] to mine spatial correlations between frames, and Rule 4 makes direct use of temporal correlations of video sequences.

Rule 1. If 𝑔_𝑚(𝑏^𝑛_𝑘)=𝑔_𝑚(𝑏_𝑙^𝑛+1), then 𝑏^𝑛_𝑘↦𝑏_𝑙^𝑛+1, where 𝑏^𝑛_𝑘∈𝐵^𝑛, 𝑏_𝑙^𝑛+1∈𝐵^𝑛+1.

Rule 2. If left(𝑏^𝑛_𝑘)↦𝑏_𝑙^𝑛+1, then 𝑏^𝑛_𝑘↦𝑏_𝑙^𝑛+1, where 𝑏^𝑛_𝑘∈𝐵^𝑛, 𝑏_𝑙^𝑛+1∈𝐵^𝑛+1, and left(𝑏^𝑛_𝑘) is the left neighboring block of 𝑏^𝑛_𝑘.

Rule 3. If 𝑔_𝑚(𝑏^𝑛_𝑘)=𝑔_𝑚(𝑏^𝑛_𝑘′) and  𝑏^𝑛_𝑘′↦𝑏_𝑙^𝑛+1_′, then 𝑏^𝑛_𝑘↦𝑏_𝑙^𝑛+1_′, where 𝑏^𝑛_𝑘∈𝐵^𝑛, 𝑏^𝑛_𝑘′∈𝐵^𝑛, 𝑏_𝑙^𝑛+1_′∈𝐵^𝑛+1.

Rule 4. If the coordinate values of the upper left corner of 𝑏^𝑛_𝑘 are equal to those of 𝑏_𝑙^𝑛+1, then 𝑏^𝑛_𝑘↦𝑏_𝑙^𝑛+1, where 𝑏^𝑛_𝑘∈𝐵^𝑛, 𝑏_𝑙^𝑛+1∈𝐵^𝑛+1.

The third step is to determine the nearest blocks from the obtained mapping blocks of ST. Here a freedom search technique [8] using the patchmatch method is used to initialize the nearest blocks for each ST as follows: 𝑏^𝑛+1_near=𝑏^𝑛_𝑘+𝑤𝛼^𝑖𝑅^𝑖. (12) In (12), 𝑅^𝑖∈[−1,1]×[−1,1], α is a constant value, 𝑤 is a search radius, and 𝑏^𝑛+1_near is the best block of any ST 𝑏^𝑛_𝑘. Then for each hash table, (13) is used to compare and update the nearest blocks: 𝑏^𝑛+1_near=𝑏^𝑛+1_map,ifdisc𝑏^𝑛_𝑘,𝑏^𝑛+1_map<disc𝑏^𝑛_𝑘,𝑏^𝑛+1_map. (13)

Finally, the predicted value of the pixels in the ST can be computed through a smoothing operation for the nearest blocks obtained as follows: 𝐼^′𝑥_𝑖,𝑦_𝑖=∑^{|𝐴𝑛(𝑏𝑛𝑘)|}_𝑗=1𝐼𝑥_𝑗^𝑛+1,𝑦_𝑗^𝑛+1𝐴^𝑛𝑏^𝑛_𝑘, (14) where 𝐼^′(𝑥_𝑖,𝑦_𝑖) denotes the predicted value of an interpolated pixel, 𝐴^𝑛(𝑏^𝑛_𝑘) is the nearest blocks set of 𝑏^𝑛_𝑘, 𝐼(𝑥_𝑗^𝑛+1,𝑦_𝑗^𝑛+1)∈𝑏_𝑗^𝑛+1, and 𝑏_𝑗^𝑛+1∈𝐴^𝑛(𝑏^𝑛_𝑘). In addition, to obtain color frame interpolation, the 𝑅, 𝐺, and 𝐵 channel values of each pixel can be computed in the same way.

4. Experimental Results and Discussion

In this section, the proposed RGFI algorithm is compared with other frame interpolation algorithms, including the block matching methods of three-step search (TSS), the adaptive rood pattern search (ARPS) [23], the optical flow method of Horn and Schunck (H&S) [24], Classic+NL-Full [5], and CSH [14]. Each algorithm was run in MATLAB using a PC with Intel(R) Pentium(R) 4 CPU processor, 3.00 GHz, and 3 GB of main memory. To evaluate algorithm performance, one in every two frames of the original video sequences was removed. Then this removed frame was reconstructed using different frame interpolation algorithms, and the reconstructed frame was compared with the removed frame. In all experiments, the detailed parameters were as follows. (1) Since 𝑐 affects the accuracy of motion region detection, we try several different values and finally determine its value to be 0.2. (2) According to [11], we found that the best choice of parameters {𝐿,𝜙₁,𝜙₂,𝜙₃} is {17,3,10,20}, respectively, and this provides good discrimination ability. (3) Adjustment parameters {𝜀_𝐿,𝜀_𝑅,𝜀_𝑇,𝜀_𝐵} are used to complete the conversion from key points to motion regions and by experiments we found that the best choice of each parameter is 8. (4) The value of 𝛼 is 0.5 and this is the same with [8]. (5) Based on [14], we choose three different values for 𝑠, namely, 8, 16, and 32. At the same time, we find that the appropriate value is 16 according to the obtained experiment results. The test video sequences used for these experiments were walk (640 × 480), jump in place (320 × 240), silent (176 × 144), and space (320 × 240). Walk and jump in place were provided by the image sequence evaluation research laboratory in Barcelona [25]. Silent was obtained from the video trace research group at Arizona State University [26]. Space was obtained from Youku Web. The selection of test video sequences covered a variety of different background and object motions, which are frequently found in real video.

Figure 4 shows the frame interpolation results using the six algorithms. Every row in the figure shows the interpolated frames for the same video using different algorithms, and every column shows the interpolation results for different videos using the same algorithm. From the red-bordered region of each figure, the visual differences among algorithms can be determined. TSS exhibited a poor interpolation effect, ARPS could not reveal the whole motion content, H&S and Classic+NL-Full introduced a suspension effect, and CSH produced disappointing effects, such as vague legs on the walking man. From these figures, it is evident that the proposed algorithm shows comparatively better performance in terms of visual quality.

Figure 4 

Comparison of interpolation frames obtained using six methods. From top to bottom, every row shows different interpolation results: interpolation frame 44 of walk, interpolation frame 31 of jump in place, interpolation frame 205 of silent, and interpolation frame 3 of space. From left to right, different algorithms are used: TSS, ARPS, H&S, Classic+NL-Full, CSH, and RGFI.

To validate the algorithm further, objective measurements are also provided. Peak signal-to-noise ratio (PSNR) and root mean square error (RMSE) are traditional quantitative measures of accuracy. Figures 5 and 6 show their values for the interpolation frames of different video sequences using different algorithms. In these figures, the red curves represent the proposed algorithm, and the five black curves using a different format represent comparison algorithms. It can be observed that the PSNR values for the interpolation frames obtained using the proposed algorithm are generally the highest, and the RMSE values are the lowest. However, the proposed method might produce unsatisfactory results, for example, the PSNR value of interpolation frame 58 of walk. Overall, it can be clearly demonstrated that the proposed new method outperforms the other five algorithms.

(a)

(b)

(c)

(d)

Figure 5 

Comparison of PSNR using six algorithms for walk, jump in place, silent, and space, respectively.

(a)

(b)

(c)

(d)

Figure 6 

Comparison of RMSE using six algorithms for walk, jump in place, silent, and space, respectively.

The MSSIM measure was also used to evaluate the visibility quality of interpolation frames. MSSIM assesses the image visibility quality from an image formation point of view under the assumption of a correlation between human visual perception and image structural information [27, 28]. Figure 7 shows MSSIM comparison results using the six frame interpolation algorithms for the different video sequences. Note that the proposed RGFI algorithm generally achieved a greater MSSIM than the other five algorithms, and the results show that the interpolation frames obtained using the proposed algorithm are closer to the original frames in terms of image structure similarity.

(a)

(b)

(c)

(d)

Figure 7 

Comparison of MSSIM using six algorithms for walk, jump in place, silent, and space, respectively.

Table 1 summarizes the average PSNR, RMSE, and MSSIM values for each video sequence using different algorithms. From this table, it can be significantly determined that RGFI always obtains the highest PSNR and MSSIM and the lowest RMSE. In a word, because of its encouraging performance in terms of video visualization and quantitative quality assessment, the proposed algorithm is very competitive in frame interpolation.

Table 1 

Comparison of average PSNR, RMSE, and MSSIM values.

5. Another Contribution

We combine RGFI with a seam carving approach [29] to achieve video resizing, so as to obtain high-quality resizing results displayed on the monitoring display devices with different resolutions. Figure 8 shows comparative results for frame 10 of the video of “walk with dog” using four methods: scaling method with uniform resizing ratio, the best cropping method directly using cutting, the seam carving method by removing or duplicating seams, and the proposed algorithm using accurate motion regions obtained from RGFI. It can be seen that using the scaling method (see Figure 8(b)), the walking man and his dog all become vaguer than before. Using the best cropping (see Figure 8(c)), the walking man is only partly displayed, resulting in missing original information. Only using seam carving (see Figure 8(d)) does the prominent part of this video sequence change much less than before. From Figure 8(e), it is apparent that the proposed method can protect the prominent object of the original frame when the video sequence resolutions are changed. Table 2 summarizes five evaluation indicators, including average gradient (AG), information entropy (IE), edge intensity (EI), spatial frequency (SF), and image definition (ID) values. These indicators are used to measure the resizing quality of the video frames. From this table, it is clear that the proposed method achieved the highest AG, IE, EI, SF, and ID values, which indicates that the proposed method can effectively improve resizing quality when image sequence resolutions are changed.

Table 2 

Comparative results for the four resizing methods.

(a) The original frame

(b) Using scaling

(c) Using best cropping

(d) Using seam carving

(e) Using our algorithm

Figure 8 

Comparative results for a walking man with a dog for the four resizing methods when the resolutions are resized from 180 × 144 to 200 × 288.

6. Conclusions

In this paper, promising RGFI method for intelligent monitoring systems has been presented. The main feature of this method is its ability to obtain relatively high-quality of interpolated frames according to spatial and temporal correlations in video sequences. The implementation process involves two steps: motion region detection and interpolated pixel computation. The former determines the prediction range of interpolation frames through a detection approach based on visual correspondence, and the latter computes interpolated pixels using spatial and temporal mapping rules based on coherency sensitive hashing. Experimental results show that the proposed algorithm outperforms the other five representative frame interpolation algorithms examined on subjective quality and in quantitative measures. At the same time, RGFI combined with a seam-carving approach can achieve video resizing. In a word, a promising frame interpolation method has been proposed for the intelligent monitoring systems. However, as a new frame interpolation algorithm, RGFI also has its disadvantage and the disadvantage is to take comparatively longer time caused by higher complexity. In the future, we will exploit a multicore architecture to do parallel computing, so as to reduce running time of the algorithm.

Acknowledgments

This work was supported by the National Basic Research Program of China (973 Program) 2012CB821200 (2012CB821206), the National Natural Science Foundation of China (nos. 91024001, and 61070142), and the Beijing Natural Science Foundation (no. 4111002).