An Integrated New Scheme for Digital Video Stabilization

Abstract

In many digital video applications, video sequences suffer from jerky movements between successive frames. In this paper, an integrated general-purpose stabilization method is proposed, which extracts the information from successive frames and removes the translation and rotation motions that result in undesirable effects. The scheme proposed starts with computation of the optical flow between consecutive video frames and an affine motion model is adopted in conjunction with the optical flow field obtained to estimate objects or camera motions using the Horn-Schunck algorithm. The estimated motion vectors are then used by a model-fitting filter to stabilize and smooth video sequences. Experimental results demonstrate that the proposed scheme is efficient due to its simplicity and provides good visual quality in terms of the global transformation fidelity measured by the peak-signal-noise-ratio.

1. Introduction

Video captured by cameras often suffers from unwanted jittering motions. In general, this problem is usually dealt with by means of compensation for image motions. Most video stabilization algorithms presented in the recent literature try to remove the image motions by either totally or partially compensating for all motions caused by camera rotations or vibrations [1–9]; therefore the resultant background remains motionless. The motion models described in [1, 2] proposed a pyramid structure to compute the motion vectors with an affine motion model representing rotational and translational camera motions. Hansen et al. [3] described an image stabilization scheme which uses a multiresolution, iterative process to calculate the affine motion parameters between levels of Laplacian pyramid images. The parameters obtained through the refinement process achieve the desired accuracy. The method presented in [4] used a probabilistic model with a Kalman filter to reduce the motion noises and to obtain stabilized camera motions. Chang et al. [5] used the optical flow between consecutive frames based on the modification of the method in [6] to estimate the camera motions by fitting a simplified affine motion model. Tsubaki et al. [7] developed a method that uses two threshold parameters to describe the velocity and the frequency of oscillations of unstable video sequences. More recently, Zhang et al. [8] proposed a 3D perspective camera model based method, which works well in situations where significant depth variations exist in the scenes and the camera undergoes large translational movement. The technique developed in [9] adopted a spatially and temporally optimized approach to achieve high-quality camera motion on videos where 3D reconstruction is difficult or long feature trajectories are not available. The technique formulates stabilization as a spatial-temporal optimization problem that finds smooth feature trajectories and avoids visual distortion.

In this paper, an integrated video stabilization scheme is proposed, which primarily has two objectives. First of all, rather than developing novel and complicated individual algorithms, it aims to simplify the stabilization process by integrating the well-researched techniques, such as motion estimation, motion modeling, and motion compensation, into a new single framework that is of modular nature and can reduce the complexity for implementation, particularly in hardware. Secondly, the scheme aims to provide better performance in terms of the global transformation fidelity (a typical measure of stabilization performance), compared to other existing methods. This is achieved by combining optical flow estimation with motion models to increase accuracy of estimation. The scheme is based on estimating the motion field between consecutive frames using the Horn-Schunck algorithm [10]. An iterative process based on a coarse-to-fine technique is adopted here. The motion vectors are firstly estimated using the block matching method between two consecutive fields and then the dense motion field is estimated using the motion vectors and the Horn-Schunck algorithm. By fitting an affine motion model, the motion parameters of the model are computed and smoothed. Thus, by analyzing the directions of motion vectors and their standard deviations as well as using previously stabilized frame or a reference frame, the image motions caused by three-dimensional rotation and translation can be determined and the current video frame can be stabilized.

The rest of this paper is organized as follows. In the next section, we present an overview of the proposed video stabilization scheme. Sections 3, 4, 5, and 6 describe in detail key components of the scheme, namely, optical flow field estimation, motion model fitting, motion parameter smoothening, and motion compensation. The experimental and simulation results of the proposed method are presented in Section 7. Finally, the conclusions are drawn in Section 8.

2. Overview of the Video Stabilization Scheme

The flowchart of the proposed stabilization scheme is shown in Figure 1, which integrates four key components: optical flow field estimation, motion model fitting, motion parameters smoothening, and motion compensation. Each of these components will be analytically presented in detail in the following sections.

Figure 1 

Flowchart of the video stabilization scheme.

3. Optical Flow Estimation Technique

The accuracy of the stabilization scheme mainly depends on the motion vectors produced during the interframe motion estimation. Here, a coarse-to-fine technique is used to perform block correlation, initially at a coarse scale, and then to interpolate the resulting estimates before they pass through iterations of Horn and Schunck’s optical flow algorithm. Optical flow is an approximation of the local image motion based upon local derivatives in a given sequence of images. That is, in two dimensions, it specifies how much each image pixel moves between adjacent images, while, in three dimensions, it specifies how much each volume voxel moves between adjacent volumes.

To estimate optical flow of any pixel(𝑥,𝑦)in an image, we use the “intensity constancy” assumption which states that the intensity of any pixel on any object in an image remains constant with time; that is, 𝐼𝑥,𝑦,𝑡=𝐼𝑥+𝛿𝑥,𝑦+𝛿𝑦,𝑡+𝛿𝑡. (1) Assuming small motions between consecutive frames (small 𝛿𝑥 and 𝛿𝑦), we can perform a first-order Taylor series expansion on 𝐼(𝑥,𝑦,𝑡) in (1) to obtain 𝐼+𝑥+𝛿𝑥,𝑦+𝛿𝑦,𝑡+𝛿𝑡=𝐼𝑥,𝑦,𝑡𝜕𝐼𝜕𝑥𝛿𝑥+𝜕𝐼𝜕𝑦𝛿𝑦+𝜕𝐼𝜕𝑡𝛿𝑡+H.O.T. (2) H.O.T. are the higher order terms of the Taylor series, which we assume to be small and can safely be ignored. Using (1) and (2), we can obtain 𝜕𝐼𝑣𝜕𝑥^𝑥+𝜕𝐼𝑣𝜕𝑦^𝑦+𝜕𝐼𝜕𝑡=0 (3) or 𝐼_𝑥𝑢+𝐼_𝑦𝑣+𝐼_𝑡=0, (4) where𝐼_𝑥=𝜕𝐼/𝜕𝑥, 𝐼_𝑦=𝜕𝐼/𝜕𝑦, 𝐼_𝑡=𝜕𝐼/𝜕𝑡, 𝑢=𝛿𝑥/𝛿𝑡, and 𝑣=𝛿𝑦/𝛿𝑡.

Equation (4) has two variables, 𝑢 and 𝑣, which means that, for an image with 𝑁 pixels, there will be 𝑁 equations with 2𝑁 variables. Hence additional constraints are required to solve these equations. Horn and Schunck proposed to use the smoothness constraint; that is, to find 𝑢, 𝑣, and their derivatives, we minimize the following energy function: 𝐼()=𝑢,𝑣_Ω𝐼_𝑥𝑢+𝐼_𝑦𝑣+𝐼_𝑡²+𝜆²𝑢²_𝑥+𝑢²_𝑦+𝑣²_𝑥+𝑣²_𝑦𝑑𝑥𝑑𝑦, (5) where 𝜆 controls the weight given to the smoothness constraint and Ω denotes the image domain. Also we assume that 𝑢 and 𝑣 are zero at the boundaries of the image domain. The minimization of (5) is achieved using calculus of variations and the approximation of the Laplacian: ∇²𝑢≈𝑘𝑢_{𝑖,𝑗,𝑘}−𝑢_{𝑖,𝑗,𝑘},∇²𝑣≈𝑘𝑣_{𝑖,𝑗,𝑘}−𝑣_{𝑖,𝑗,𝑘}. (6) The derivatives of brightness are estimated from the discrete set of image brightness measurements as follows: 𝐼_𝑥=14𝐼_{𝑖,𝑗+1,𝑘}−𝐼_{𝑖,𝑗,𝑘}+𝐼_{𝑖+1,𝑗+1,𝑘}−𝐼_{𝑖+1,𝑗,𝑘}+𝐼_{𝑖,𝑗+1,𝑘+1}−𝐼_{𝑖,𝑗,𝑘+1}+𝐼_{𝑖+1,𝑗+1,𝑘+1}−𝐼_{𝑖+1,𝑗,𝑘+1},𝐼_𝑦=14𝐼_{𝑖+1,𝑗,𝑘}−𝐼_{𝑖,𝑗,𝑘}+𝐼_{𝑖+1,𝑗+1,𝑘}−𝐼_{𝑖,𝑗+1,𝑘}+𝐼_{𝑖+1,𝑗,𝑘+1}−𝐼_{𝑖,𝑗,𝑘+1}+𝐼_{𝑖+1,𝑗+1,𝑘+1}−𝐼_{𝑖,𝑗+1,𝑘+1},𝐼_𝑡=14𝐼_{𝑖,𝑗,𝑘+1}−𝐼_{𝑖,𝑗,𝑘}+𝐼_{𝑖+1,𝑗,𝑘+1}−𝐼_{𝑖+1,𝑗,𝑘}+𝐼_{𝑖,𝑗+1,𝑘+1}−𝐼_{𝑖,𝑗+1,𝑘}+𝐼_{𝑖+1,𝑗+1,𝑘+1}−𝐼_{𝑖+1,𝑗+1,𝑘}, (7) where 𝑖 and 𝑗 are the grid space intervals and 𝑘 is the image frame sampling period. The local averages 𝑢 and 𝑣 are defined as follows: 𝑢_{𝑖,𝑗,𝑘}=16𝑢_{𝑖−1,𝑗,𝑘}+𝑢_{𝑖,𝑗+1,𝑘}+𝑢_{𝑖+1,𝑗,𝑘}+𝑢_{𝑖,𝑗−1,𝑘}+1𝑢12^{𝑖−1,𝑗−1,𝑘}+𝑢_{𝑖−1,𝑗+1,𝑘}+𝑢_{𝑖+1,𝑗+1,𝑘}+𝑢_{𝑖+1,𝑗−1,𝑘},𝑣_{𝑖,𝑗,𝑘}=16𝑣_{𝑖−1,𝑗,𝑘}+𝑣_{𝑖,𝑗+1,𝑘}+𝑣_{𝑖+1,𝑗,𝑘}+𝑣_{𝑖,𝑗−1,𝑘}+1𝑣12^{𝑖−1,𝑗−1,𝑘}+𝑣_{𝑖−1,𝑗+1,𝑘}+𝑣_{𝑖+1,𝑗+1,𝑘}+𝑣_{𝑖+1,𝑗−1,𝑘}. (8) We can compute a new set of velocity estimates (𝑢^𝑛+1,𝑣^𝑛+1) per frame from the estimated derivatives and the average of the previous velocity estimates (𝑢^𝑛,𝑣^𝑛) by 𝑢^𝑛+1_𝑖,𝑗=𝑢^𝑛_𝑖,𝑗−𝛼𝐼_𝑥,𝑣^𝑛+1_𝑖,𝑗=𝑣^𝑛_𝑖,𝑗−𝛼𝐼_𝑦,𝐼𝛼=^𝑥𝑢^𝑛_𝑖,𝑗+𝐼_𝑦𝑣^𝑛_𝑖,𝑗+𝐼_𝑡𝜆²+𝐼²_𝑥+𝐼²_𝑦. (9) At the iteration𝑛=0, the initial values of 𝑢 and 𝑣 are set equal to zero.

In motion estimation, there are occasions that the motion vectors produced fall outside normal values. When a motion vector is above a certain value, it is characterized as an outlier. The above method is very sensitive to outliers; that is, it is prone to produce outliers or unexpected data. Therefore, an alternative value has to be considered to substitute these outliers. Here, the median value of motion vectors is adopted. This is because, among geometric mean, harmonic mean, standard deviation, median and trim-mean, all of which have been applied and tested, the median and trim-mean are found to be the most robust, that is, resistant to outliers.

4. Motion Model Fitting

A camera projects a three-dimensional world point onto a two-dimensional image point. The motion of the camera may be regarded as a single motion such as rotation, translation, or zoom or a combination of any two or three of these motions. Such camera motion can be well categorized by a set of parameters. In our case, the first frame of a video sequence is used to define the reference coordinate system, and a two-dimensional affine model is used to estimate a parametric form describing the displacement of the video content between consecutive frames by identifying the correspondence between local invariant features. The affine model was employed since it is more resilient to noisy data and it can represent all the basic camera motions which often occur in video applications. If we denote a pixel position in the first frame by (𝑥,𝑦) and the corresponding position in the second frame by (𝑥^′,𝑦^′), the two-dimensional affine motion model can be formulated as 𝑥^′𝑦^′=𝑎₀𝑎₃+𝑎₁−𝑎₂𝑎₂𝑎₁𝑥𝑦, (10) where the motion parameters𝑎₁,𝑎₂control the scaling and rotation (𝑎₁=cos𝜃 and 𝑎₂=sin𝜃 if there is only rotation; 𝜃 is rotation angle) and the parameters 𝑎₀ and 𝑎₃ correspond to the horizontal and vertical translations.

Assuming that we have 𝑁 motion vectors, which correspond to the image pixels (𝑥_𝑖,𝑦_𝑖) and (𝑥^′_𝑖,𝑦^′_𝑖) (𝑖=1,2,3,…,𝑁) between two consecutive video frames, we can estimate the simplified affine motion between the two frames from the 𝑁 motion vectors by solving the following overconstrained linear equation: 𝑥₁−𝑦₁𝑦10₁𝑥₁𝑥01₂−𝑦₂𝑦10₂𝑥₂..𝑥01_𝑁−𝑦_𝑁𝑦10_𝑁𝑥_𝑁𝑎01¹𝑎₂𝑎₀𝑎₃^𝑇=𝑥^′₁𝑦^′₁𝑥^′₂𝑦^′₂..𝑥^′_𝑁𝑦^′_𝑁=𝑥₁+𝑢₁𝑦₁+𝑣₁𝑥₂+𝑢₂𝑦₂+𝑣₂..𝑥_𝑁+𝑢_𝑁𝑦_𝑁+𝑣_𝑁.𝐀𝐗𝐘 (11) The affine motion parameters are obtained through solving this linear equation by the least-square solution. Equation 𝐀𝐗=𝐘 has least square solution if 𝐀^𝑡𝐀 is invertible and rank(𝐀)=4. In this case, the least square solution 𝐗 is given by 𝐗=(𝐀^𝑡𝐀)⁻¹𝐀^𝑡𝐘.

5. Motion Parameters Smoothing

In order to produce high quality stabilized video sequences, the motion parameters obtained need to be smoothed. This can be achieved by space-domain filtering. Different types of filters have been applied and tested. These include the recursive Kalman filtering which removes camera vibrations, the moving average filter that smoothes data by replacing each data with the average of the neighboring data defined within a span, and the locally weighted scatterplot smoothing which uses weighted linear regression to smooth data. In our scheme, the Savitzky-Golay filter [11, 12] is used to process the originally estimated affine global motion parameters as it is a generalized moving average filter with the properties of simplicity and efficiency for implementation.

6. Global Motion Compensation

Motion compensation is performed frame by frame using previously stabilized frames (apart from the first frame) and their corresponding global smoothed parameters; that is, the first stabilized frame is obtained by compensating the first original frame with its corresponding smoothed affine motion parameters; the second stabilized frame is achieved by compensating the first stabilized frame with its corresponding smoothed affine motion parameters, and so forth. The block diagram of this compensation process is shown in Figure 2. Due to utilization of the first original frame (rather than a previously stabilized one) at the very beginning of the process, an error will be produced and propagated to the subsequent frames. In order to mitigate this effect, synchronization is performed to control the error as follows. For the frame𝑖, the unsmoothed motion parameters are compared with the smoothed motion parameters. If the result of the comparison is less than a threshold (synchronization distance threshold), the original frame𝑖−1 together with the corresponding smoothed motion parameters are used to obtain the stabilized frame 𝑖, that is, synchronizing the stabilized frame 𝑖 with the original frame𝑖−1; otherwise the stabilized frame 𝑖−1with the corresponding smoothed motion parameters is used to obtain the stabilized frame𝑖. The higher the threshold is, the more synchronized the stabilized video sequence would be with the original video sequence, therefore reducing errors. In order to guarantee quality of the stabilized output video, regardless of the error control described above, a synchronization frame is enforced every 30 frames.

Figure 2 

Compensation process to stabilize video sequence.

7. Simulation Results

In order to evaluate the effectiveness and performance of the stabilization scheme proposed, the simulations are carried out using a range of the QCIF format (176 pixels by 144 lines) video sequences captured. Figure 3 shows the stabilization results from the captured video sequence “My Office.” Figure 3(a) shows an original frame (no. 14) from the video sequence. The frame clearly contains the “tremor effect,” which is deliberately introduced into the video sequence. Figure 3(b) shows the optical flow field, which is estimated from frames 14 and 15. The random vectors detected, as shown in the figure, are due to zooming in/out effects produced during the video recording. Since it is very difficult to visually distinguish two consecutive frames, here we compare the difference between two original frames (numbers 14 and 15) to the difference between two corresponding stabilized frames, as shown in Figures 3(c) and 3(d). The results of another experiment from the video sequence “Jerky” are shown in Figure 4. Similarly, the frames 14 and 15 of the sequence are used here. From these experiments, it is obvious that the stabilized frames in Figures 3(d) and 4(d) have much less motion (white pixels/regions) than the original frames in Figures 3(c) and 4(c) do. This demonstrates that, after the stabilization process, a significant amount of the undesirable movements have been compensated.

(a)

(b)

(c)

(d)

Figure 3 

Example of stabilization results from the stabilization scheme proposed: (a) original frame No. 14 of the video sequence “My Office”; (b) dense optical flow field-estimated motion vectors between frames 14 and 15; (c) difference between original frames 14 and 15; (d) difference between stabilized frames 14 and 15.

(a)

(b)

(c)

(d)

Figure 4 

Experimental results from the video sequence “Jerky”: (a) original frame no. 14; (b) dense optical flow field-estimated motion vectors between frames 14 and 15; (c) difference between original frames 14 and 15; (d) difference between stabilized frames 14 and 15.

Since dynamic processes, such as stabilization, cannot be illustrated with still images, we present and compare in Figure 5 the three motion parameters (rotation, horizontal, and vertical displacements) of the first 80 frames of the video sequence “My Office.” The comparisons between original (blue curve) and smoothed (red curve) motion parameters show that all three motion parameters have been smoothed for the length of the video sequence, therefore reducing the unwanted movements captured during the generation of the video. It is also shown that the parameter smoothing process has been improved with synchronization. Without this synchronization (green curve), accumulated estimation errors will increase significantly after a certain number of frames. The difference between the red and green curves indicates clearly that the parameter smoothing process is synchronized at frames 7, 13, 25, 41, 57, and 73 in order to correct the accumulated estimation errors, which occur at these frame locations on the green curves.

(a)

(b)

(c)

Figure 5 

Original and smoothed motion parameters of the video “My Office.”

In order to objectively and quantitatively evaluate the performance of the scheme proposed, we use global transformation fidelity (GTF) [13] as a measure of how well stabilization compensates the motion of a camera; that is, how precisely the motion model fits the actual camera motion. Here, the Peak-Signal-Noise-Ratio (PSNR) between two frames is used to measure the GTF, which is defined as PSNR𝑓2,𝑓1=10log255²,MSE𝑓2,𝑓1 (12) where MSE (mean squared error) measures the average difference between the two frames 𝑓1 and 𝑓2. Figure 6 shows the performance comparison in terms of PSNR between the proposed stabilization scheme and the well-known Gray-Coded Bit-Plane Matching based stabilization method [14, 15], which uses simple Gray-coding and has low computational load for hardware implementation, therefore being of equivalent complexity to that of the proposed scheme. The calculations of PSNR in the figure make use of the first 50 frames of “My Office” sequence, which contains the anticlockwise camera rotation of 90 degrees. The GTF (PSNR) is calculated between the reference frame (the first frame in this case) and the currently stabilized frame. It can be observed from the GTF curves that the proposed stabilization procedure performs significantly better than the Gray-Coded Bit-Plane Matching during the first 10 frames which correspond to the rotation part of the sequence. This result is anticipated because the Gray-Coded Bit-Plane Matching method does not compensate for rotation very well. The GTF of the proposed scheme drops from frame to frame since each following frame has less overlap with the reference frame. After about 40 frames, the sequence almost does not overlap with the reference frame.

Figure 6 

GTF of the video sequence “My Office,” measured by PSNR.

8. Conclusions

This paper presents a general-purpose video stabilization scheme, aiming at a simple and effective solution for a wide range of video-based applications. The scheme features integration of optical flow and motion model based motion estimation, space-domain filtering, and motion compensation, thus offering an efficient computation method for video stabilization. It is successfully implemented in MATLAB. The simulation result shows that the scheme is effective for a broad range of real-time applications. Compared to other video stabilization methods, it has the advantages of simplicity and robustness while maintaining better or comparable performance in terms of the global transformation fidelity measured by PSNR.