Abstract
The Horn and Schunck (HS) optical flow model cannot preserve discontinuity of motion estimation and has low accuracy especially for the image sequence, which includes complex texture. To address this problem, an improved fractional-order optical flow model is proposed. In particular, the fractional-order Taylor series expansion is applied in the brightness constraint equation of the HS model. The fractional-order flow field derivative is also used in the smoothing constraint equation. The Euler-Lagrange equation is utilized for the minimization of the energy function of the fractional-order optical flow model. Two-dimensional fractional differential masks are proposed and applied to the calculation of the model simplification. Considering the spatiotemporal memory property of fractional-order, the algorithm preserves the edge discontinuity of the optical flow field while improving the accuracy of the estimation of the dense optical flow field. Experiments on Middlebury datasets demonstrate the predominance of our proposed algorithm.
1. Introduction
Since the optical flow was proposed by Gibson in 1950, many scholars have done research on it. In these studies, the HS optical flow model proposed by Horn and Schunck [1] in 1981 achieved high reliability and accuracy and accordingly began to attract more and more scholarly attention. But the accuracy of the motion estimation with the HS optical flow model would be greatly reduced in the image sequence, which would include complex texture and nonrigid motion; meanwhile the discontinuity of motion estimation is difficult to retain.
Many scholars have devoted themselves to the modification or improvement of HS model to increase the accuracy [2], deal with large displacement motion [3], solve the occlusion problem [4], and track nonrigid motion targets [5, 6]. All these models are based on first-order differentiation techniques.
Lv et al. [7] added a nonlinear fourth-order diffusive term to the Euler-Lagrange equations of the variational TV model, and the improved model can preserve sharp jump discontinuities, but it would deteriorate when too much noise exists. A high-order total variation minimization method for image deblurring and denoising was proposed by Chan et al. [8]; the model can filter out noise while the edge discontinuity was retained. But details of the edge were neglected.
Because of its ability to preserve the texture details of the smooth region while highlighting the image edge feature and its spatiotemporal memory of the target point neighborhood, a fractional differential is applied to many image processing fields [9], such as image denoising [10], image enhancement [11], and motion estimation [12–16]. In [12, 13], the fractional-order smoothing constraint equation was used in the HS optical flow model to preserve the discontinuity of motion estimation, but it does not consider the correlation of the pixel intensity. Although the edge discontinuity of the image can be preserved to a certain extent, the accuracy of the optical flow field estimation is not high. In order to further enhance the accuracy of motion estimation, an improved fractional-order optical flow model is proposed in this paper.
The main characteristics of the present paper can be summarized as follows:(1)Fractional-order was first used in a brightness constraint equation of the HS optical flow model.(2)A two-dimensional fractional-order differential mask, which was used to compute dual fractional-order differentials, was first proposed and applied to the simplification of the improved fractional-order optical flow model.(3)Combined with the Euler-Lagrange equation and two-dimensional fractional-order differential masks, the iteration formula of the optical flow field estimation of the improved fractional-order optical flow model was constructed, which is different from any of the former counterparts.
2. The Construction of Two-Dimensional Fractional-Order Differential Masks
The mathematical preliminaries of the fractional-order theory used in this paper can be found in [17, 18].
In this chapter, six two-dimensional fractional differential masks were presented. There are , , , , , , where is the order of fractional differentials.
We define as the two-dimensional fractional differential mask of dual derivatives on the -axis. The two-dimensional fractional derivatives of order of a discrete image brightness function on the -axis can be defined as
is the mask size, when :
Similarly, where , when :Also,where , , . When ,
What is more, we can get , .
Normalization should be done in all the mask items. More details about the expansion of dual fractional-order derivatives can be found in [12].
3. Improved Fractional-Order Optical Flow Model
3.1. The Fractional-Order Optical Flow Model
We substitute the brightness constraint equation of the HS model with the fractional-order Taylor series expansion and get the fractional-order brightness constraint equation aswhere is the abbreviation of .
Combined with (2.19) and (2.20) in [12], we can get the fractional-order optical flow model in this paper:
3.2. Numerical Algorithm
The Euler-Lagrange equation [19] was applied in this paper to minimize energy function .
After simplification, we get the equation as follows: where
After a series of computations, the iteration formula for the proposed fractional-order optical flow model can finally be written as where , represented the optical flow vector in the neighborhood of point , and
After repeated iteration utilizing the Gauss-Seidel iteration method, the accurate optical flow field can be achieved, and the convergence of the algorithm can be demonstrated by the method mentioned in [13].
4. Experimental Results and Analysis
We have conducted our experiments from the following three aspects: the comparison of the performance of previously tested optical flow algorithms; the influence of the order of fractional-order differentials for the algorithm; and the influence of the size of the dual fractional differential mask window for the algorithm. The Middlebury datasets [20] are chosen for evaluating our proposed algorithm for its characteristics of having different structures and attributes, like textures, blurred images, discontinuities in motion, shadows, nonrigid motion, and so on.
The angular error (AE) and flow endpoint error (EE) are selected for evaluation. As we are concerned with the performance of the algorithm in the whole image, the AE/EE in the entire image (AVAE/AVEE) are used to visually evaluate our proposed optical model.
For convenient description, we call the HS optical flow model the HS model, the model in [12] the FOVOF model, and our proposed model the DFOVOF model. The mask window size is .
Figure 1 shows the optical flow field of the HS model, the FOVOF model, and the DFOVOF model in different image sequences. In the DFOVOF model, , , . Among them, different colors represent different movement directions and different color depth represents different speed of movement. The deeper the color, the higher the speed.
(a) Yosemite
(b) Grove
(c) Urban
(d) Army
The first row in Figure 1 shows the acquired optical flow field produced by the traditional HS model. It can be seen that the edge of the image discontinuity is not well reserved and the details of the image texture are neglected.
The second row shows the optical flow field obtained by the FOVOF model. The edge discontinuity is reserved to some extent, but the contour of the object is not very clear.
In the third row, the optical flow field obtained by the DFOVOF optical flow model is presented. The contour edge is much more obvious; meanwhile the details of image texture can be seen clearly. The DFOVOF model expands the brightness constraint equation in the form of a Taylor fractional-order, and the local brightness correlation of the image is also added for the calculation of the optical flow field, which further reduces the error and improves the accuracy of the model.
In order to evaluate the influence of fractional-order parameters on the performance of the model and find out the correlation between the fractional-order and the image characteristics, the AVAE/AVEE of the optical flow field of each image sequence at different fractional orders is estimated, as shown in Figure 2. At this time we choose . The smaller the value of AVAE/AVEE, the higher the accuracy of the optical flow field estimation. The first row is the AVAE and AVEE of image sequence “Yosemite”; the second row is the AVAE and AVEE of image sequence “Army.” The third row is the AVAE and AVEE of image sequence “Urban.”
We can find in Figure 2 that the minimum AVAE/AVEE is distributed in a range of and the minimum AVAE/AVEE is in a range of ; the simpler the image texture, the smoother the surface representing the variation of the value of AVEE/AVAE to variable and ; the more complex the image texture, the steeper the surface; the surface would also increase in incline when there exists a nonrigid motion in image sequences. Thus, it is clear that the algorithm presented here can improve the accuracy of motion estimation, especially for image sequences with complex texture and nonrigid motion. In conclusion, we choose a small for a complex texture image and choose a relatively large for an simple texture image sequence. What is more, nonrigid motion would also make the optimal small. The optimal changes little in different image sequence.
To test the effect of the mask window size on the algorithm’s performance and find out the correlation between mask window length and image features, we calculate the AVEE of optical flow estimation of six image sequences at different mask window size which was shown in Table 1, where , , where (a) is Venus; (b) is dimetrodon; (c) is hydrangea; (d) is Rubberwhale; (e) is grove; and (f) is urban. For comparison, the AVEE of H-S model and FOVOF model were also added to the table. For comparison, the AVEE values for the optical flow field estimation of the H-S model and the FOVOF model are also added to the table.
We can see in Table 1 that the value of AVEE reduces further when applied to the DFOVOF model; in the DFOVOF model the error rate becomes smaller when the mask window size increases—in general, we get the smallest AVEE value when is within (2, 7). For all the images, AVEE will decrease first when rises; after AVEE arrives at its optimal value, it will then increase when rises. The more complex the image texture is or the more nonrigid the motion is, the smaller the value of will be.
In conclusion, we should choose a large value for relatively simple image sequences and choose a small value for complex texture images and nonrigid motion images.
5. Conclusion
We have presented an improved fractional-order optical flow model, which applies fractional-order both in brightness constraint equation and in smoothness constraint equation of the HS model. What is more, we have proposed two-dimensional fractional differential masks and used them to simplify the computation of dual fractional-order derivatives. Experiments show that the model created here improves accuracy further. We have also found the relationships between the values of , , and and the image content. So different image content should use different , , and , but the algorithm we employ utilizes the same , , and in the whole image. Further studies will focus on the adaptive adjustments of the value of , , and in relation to the image content.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (61503141), Guangdong Province Frontier and Key Technological Innovation Special Funds (2016B090912001), the Production Research Project in Guangzhou (201604010114), the International Cooperation Projects of Science and Technology Information Bureau in Guangzhou City (2012J5100001), Sea Defense Public Welfare Project (201505002), and the Fundamental Research Funds for the Central Universities.