Robust Online Object Tracking Based on Feature Grouping and 2DPCA

Abstract

We present an online object tracking algorithm based on feature grouping and two-dimensional principal component analysis (2DPCA). Firstly, we introduce regularization into the 2DPCA reconstruction and develop an iterative algorithm to represent an object by 2DPCA bases. Secondly, the object templates are grouped into a more discriminative image and a less discriminative image by computing the variance of the pixels in multiple frames. Then, the projection matrix is learned according to the more discriminative image and the less discriminative image, and the samples are projected. The object tracking results are obtained using Bayesian maximum a posteriori probability estimation. Finally, we employ a template update strategy which combines incremental subspace learning and the error matrix to reduce tracking drift. Compared with other popular methods, our method reduces the computational complexity and is very robust to abnormal changes. Both qualitative and quantitative evaluations on challenging image sequences demonstrate that the proposed tracking algorithm achieves more favorable performance than several state-of-the-art methods.

1. Introduction

Online object tracking is a fundamental problem in many computer vision applications such as surveillance, driver assistance systems, and human-computer interactions [1–4]. Although researchers have made great progresses in this area, object tracking remains a challenging problem due to the difficulty arising from the appearance variability of an object. Intrinsic and extrinsic changes inevitably cause large appearance variation. Due to the nature of the tracking problem, an effective appearance model is of prime importance for the success of a tracking algorithm [5–8].

In recent years, as a popular dimensionality reduction and feature extraction technique, linear subspace learning has been successfully used in robust visual tracking. Supervised discriminative methods for classification and regression have also been exploited to solve visual tracking problems. For example, Avidan developed a tracking algorithm that employs the support vector machine (SVM) classifier within an optic flow framework [9]. Along similar lines, Williams et al. developed a method in which an SVM-based regressor was used for tracking [10]. As a result of training the regressor on in-plane image motion, this method is not effective in tracking objects with out-of-plane movements. In [11], the author combines the sparse coding and Kalman filtering together and chooses the color histogram and gradient histogram as features to track the object. The template updating strategy of the algorithm is to replace a random template of the original template library with the last tracking result. This updating strategy can easily introduce tracking errors when abnormal changes happen to the object, which would result in failure of tracking. A method by casting object tracking as a sparse approximation problem in a particle filter framework is proposed in [12]. The author solves the problem of object occlusion through the introduction of trivial template which, however, extends the number of templates greatly and increases the computational complexity. In that case, the practical value of algorithm is greatly reduced.

Motivated by the abovementioned discussions, we propose an online object tracking algorithm based on feature grouping and 2DPCA. Firstly, we introduce regularization into the 2DPCA reconstruction and develop an iterative algorithm to represent an object by 2DPCA bases. Secondly, the object templates are grouped into a more discriminative image and a less discriminative image by computing the variance of the pixels in multiple frames. Then, the projection matrix is learned according to the more discriminative image and the less discriminative image, and the samples are projected using the projection matrix. The object tracking results are obtained using Bayesian maximum a posteriori probability (MAP) estimation (Figures 1, 2, and 3). Finally, to further reduce tracking drift, we employ a template update strategy which combines incremental subspace learning and the error matrix. This strategy adapts the template to the appearance change of the target and reduces the influence of the occluded target template as well. The experimental results show that our proposed method has strong robustness to abnormal changes.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 1 

Tracking results of test video “Lemming.”

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2 

Tracking results of test video “Car4.”

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3 

Tracking results of test video “Singer1.”

2. Robust Online Object Tracking Based on Feature Grouping and 2DPCA

2.1. The Theory of 2DPCA

Principal component analysis (PCA) is a well-established linear dimension-reduction technique, which has been widely used in many areas (such as face recognition [13]). It finds the projection directions along which the reconstruction error to the original data is minimum and projects the original data into a lower dimensional space spanned by those directions corresponding to the top eigenvalues. Recent studies demonstrate that two-dimensional principal component analysis (2DPCA) could achieve performance comparable to PCA with less computational cost.

Given a series of image matrices 𝐘=[𝐘_𝟏;𝐘_𝟐;…;𝐘_𝐝], 2DPCA aims to obtain an orthogonal left-projection matrix 𝐔, an orthogonal right-projection matrix 𝐕, and the projection coefficients 𝐀=[𝐀_𝟏;𝐀_𝟐;…;𝐀_𝐝] by solving the following objective function: min_{𝐔,𝐕,𝐀𝑖}1𝑑^𝑑_𝑖=1𝐘_𝑖−𝐔𝐀_𝑖𝐕^′2_𝐹. (1)

Then the coefficient 𝐀_𝑖 can be approximated by 𝐀_𝑖≈𝐔^′𝐘_𝑖𝐕. We note that the underlying assumption of (6) is that the error term is Gaussian distributed with small variances. This assumption is not able to deal with partial occlusion as the error term cannot be modeled with small variances when occlusion occurs. In this paper, we propose an object tracking algorithm by using 2DPCA basis matrices and an additional MLE error matrix 𝐘≈𝐔𝐀𝐕^′+𝐞.

Let the objective function be 𝐿()=1𝐀,𝐄2𝐘−𝐔𝐀𝐕^′−𝐄²_𝐹+𝜆‖𝐞‖₁, (2) the problem is min_𝐀,𝐄𝐿(𝐀,𝐄)s.t.𝐔^′𝐔=𝐈;𝐕^′𝐕=𝐈, (3) where 𝐘 denotes an observation matrix, 𝐀 indicates its corresponding projection coefficient, 𝜆 is a regularization parameter, and 𝐞 describes the error matrix.

2.2. Feature Grouping

In object tracking problem, 𝐓=[𝐭_𝟏,𝐭_𝟐,…,𝐭_𝐧]∈ℝ^𝑑×𝑛 consists of 𝑛 templates and forms the object template library, where 𝐭_𝑖∈ℝ^𝑑,𝑑≫𝑛. In our experiments, we make 𝑛=10 and the object template size 32×32; that is, 𝐝=1024. The result image block in current frame is denoted as 𝐲,𝐲∈ℝ^𝑑, which has the same size as object template 𝐭_𝑖.

We would like to decompose each template 𝐭_𝑖,𝑖=1,2,…,𝑛, into a more discriminative image 𝐭^𝑎_𝑖 and a less discriminative image 𝐭^𝑏_𝑖. First, we rewrite 𝐭_𝑖 as 𝑡_𝑖,𝑗,𝑗=1,2,…𝑑, and compute the variance of each pixel in each template. if the pixel 𝑡_𝑖,𝑗 has a smaller variance, we can think that this pixel is more stable. Based on this heuristic, we can decompose those templates into a more discriminative image and a less discriminative image.

Denote by 𝑡_𝑗 the mean of the templates 𝑡_𝑖,𝑗. The variance of the 𝑗th pixel in template 𝐭_𝑖,𝑗 is 𝜎_𝑗=1𝑑^𝑑_𝑗=1𝑡_𝑖,𝑗−𝑡_𝑗. (4)

We want to put those pixels having smaller 𝜎_𝑗 into a more discriminative image and the remaining into a less discriminative image 𝐭^𝑏_𝑖.

After decomposing, each template can be written as 𝐭_𝑖=𝐭^𝑎_𝑖+𝐭^𝑏_𝑖,𝑖=1,2,…,𝑛, and we have 𝐓=𝐓_𝑎+𝐓_𝑏. However, if we learn the projection matrix 𝐏 only based on 𝐓_𝑎, the result cannot be very satisfying because 𝐓_𝑏 is also useful for determining the projection direction. To effectively exploit the information in both 𝐓_𝑎 and 𝐓_𝑏, we propose to learn a subspace where the energy in 𝐓_𝑎 is well preserved and the energy in 𝐓_𝑏 is suppressed. Denote by 𝐭, 𝐭^𝑎, and 𝐭^𝑏 the mean vectors of 𝐓, 𝐓_𝑎, and 𝐓_𝑏, respectively, and let 𝐭_𝑖=𝐭_𝑖−𝐭, 𝐭^𝑎_𝑖=𝐭^𝑎_𝑖−𝐭^𝑎, and 𝐭^𝑏_𝑖=𝐭^𝑏_𝑖−𝐭^𝑏 be the centralized image vectors. Accordingly we have the centralized datasets 𝐓, 𝐓_𝑎, and 𝐓_𝑏. Clearly, we have 𝐭=𝐭^𝑎+𝐭^𝑏, 𝐭_𝑖=𝐭^𝑎_𝑖+𝐭^𝑏_𝑖, and 𝐏𝐭_𝑖=𝐏𝐭^𝑎_𝑖+𝐏𝐭^𝑏_𝑖. After projection, the average energy of 𝐏𝐭^𝑎_𝑖 is 𝐸_𝑎=1𝑛^𝑛_𝑖=1𝐏𝐭^𝑎_𝑖²₂=1𝑛^𝑛_𝑖=1𝐏𝐭^𝑎_𝑖^𝑇𝐏𝐭^𝑎_𝑖𝐏1=tr𝑛𝐓_𝑎𝐓^𝑇_𝑎𝐏^𝑇=tr𝐏𝐒_𝐚𝐏^𝑇, (5) where 𝐒_𝐚=(1/𝑛)𝐓_𝑎𝐓^𝑇_𝑎 is the total scatter matrix of 𝐓_𝑎 and “tr” is the matrix trace operator.

Similarly, the average energy of 𝐏𝐭^𝑏_𝑖 is 𝐸_𝑏=1𝑛^𝑛_𝑖=1𝐏𝐭^𝑏_𝑖²₂=tr𝐏𝐒_𝑏𝐏^𝑇, (6) where 𝐒_𝑏=(1/𝑛)𝐓_𝑏𝐓^𝑇_𝑏 is the total scatter matrix of 𝐓_𝑏.

To preserve the 𝐓_𝑎 while suppressing the 𝐓_𝑏, we seek for a projection matrix 𝐏 to maximize the energy 𝐸_𝑎 while minimizing the energy 𝐸_𝑏 by solving the following optimization problem: 𝐽_𝑃=argmax_𝑃𝐸_𝑎𝐸_𝑏=argmax_𝑃tr𝐏𝐒_𝐚𝐏^𝑇tr𝐏𝐒_𝑏𝐏^𝑇. (7)

An equivalent form of (4) is 𝐽_𝑃=argmax_𝑃𝐸_𝑎𝐸_𝑏=argmax_𝑃tr𝐏𝐒_𝐚𝐏^𝑇s.t.𝐏𝐒_𝐚𝐏^𝑇=𝐈. (8)

Apparently, the desired 𝐏 can be computed by using generalized eigenvalue decomposition; that is, matrix 𝐏 is composed of the generalized eigenvectors of 𝐒_𝐚𝑤=𝜆𝐒_𝑏𝑤 corresponding to the 𝑝 largest eigenvalues.

3. Bayesian Map Estimation

We can regard object tracking as a hidden state variables’ Bayesian MAP estimation problem in the hidden Markov model; that is, with a set of observed samples 𝑌_𝑡={𝑦₁,𝑦₂,…,𝑦_𝑡}, we can estimate the hidden state variable 𝑥_𝑡 using Bayesian MAP theory.

According to the Bayesian theory, 𝑝𝑥_𝑡∣𝑌_𝑡∝𝑝𝑦_𝑡∣𝑥_𝑡𝑝𝑥_𝑡∣𝑥_𝑡−1𝑝𝑥_𝑡−1∣𝑌_𝑡−1𝑑𝑥_𝑡−1, (9) where 𝑝(𝑥_𝑡∣𝑥_𝑡−1) stands for a state transition model for two consecutive frames and 𝑝(𝑦_𝑡∣𝑥_𝑡) stands for an observation likelihood model. We can obtain the object’s best state in 𝑡th frame through maximum a posteriori probability estimation; that is, 𝑥_𝑡=argmax_𝑥𝑙𝑡𝑝𝑥^𝑙_𝑡∣𝑌_𝑡,𝑙=1,2,…,𝑁, (10) where 𝑥^𝑙_𝑡 stands for the 𝑙th sample of state variable 𝑥_𝑡 in 𝑡th frame. In this paper, we choose 𝑁=400.

3.1. State Transition Model

We choose object’s motion affine transformation parameters as state variable 𝑥_𝑡={𝑥_𝑡,𝑦_𝑡,𝜃_𝑡,𝑆_𝑡,𝛼_𝑡,𝜙_𝑡}, where 𝑥_𝑡 and 𝑦_𝑡, respectively, represent the 𝑥-direction and 𝑦-direction translation of the object in 𝑡th frame, 𝜃_𝑡 stands for the rotation angle, 𝑆_𝑡 represents the scale change, 𝛼_𝑡 stands for the aspect ratio, and 𝜙_𝑡 stands for the direction of tilt.

We assume that the state transition model follows the Gaussian distribution; that is, 𝑝𝑥_𝑡∣𝑥_𝑡−1=𝑁𝑥_𝑡;𝑥_𝑡−1,Ψ, (11) where Ψ is a diagonal matrix whose diagonal elements are motion affine parameter’s variation 𝜎²_𝑥,𝜎²_𝑦,𝜎²_𝜃,𝜎²_𝑆,𝜎²_𝛼, and 𝜎²_𝜙.

3.2. Observation Likelihood Model

We use object’s reconstruction error to build observation likelihood model; that is, 𝑝𝑦_𝑡∣𝑥_𝑡=_{𝑗=1,2,…,𝑑}ℕ𝑒^𝑡_𝑗,𝜇,𝜎², (12) where ℕ(⋅) means Gaussian distribution, 𝜇 and 𝜎², respectively, represent the mean and variation of Gaussian distribution, 𝑑 stands for the number of pixels of an object template, and 𝑒^𝑡_𝑗 stands for the reconstruction error of 𝑗th pixel of object templates in 𝑡th frame.

4. Experimental Results and Analysis

In order to show the robustness of the object tracking algorithm based on projection discussed in this paper, we choose several sets of public test videos taken under different environments to test the performance of our algorithm (Figures 4, 5, and 6). Given the limited space, in this section we only list three of them to show the tracking results and error curves. Different abnormal changes have happened to the moving objects in chosen test videos such as occlusion, rotation, illumination variation, or scale variation (Table 1).

Table 1 

The description of test videos.

Figure 4 

Quantitative evaluation in terms of center location error for test video “Lemming.”

Figure 5 

Quantitative evaluation in terms of center location error for test video “Car4.”

Figure 6 

Quantitative evaluation in terms of center location error for test video “Singer1.”

The implementation of the algorithm is based on Windows operating system. The configuration of the computer is AMD Athlon (TM) X2 Dual Core QL-62 2.00 GHz, 1.74 GB memory. In order to evaluate the performance of the algorithm, we choose six currently most representative and classic tracking algorithms to do the comparison. The six classic algorithms are L1 Tracker [8], IVT Tracker [14], PN Tracker [15], VTD Tracker [16], MIL Tracker [17], and Frag Tracker [18].

The average processing speeds of our method for different test videos are listed in Table 2. The statistical data is obtained using Opencv2.2. Table 2 shows that the proposed algorithm can meet the real-time performance.

Table 2 

The average processing speed of our method.

5. Conclusions

This paper presents a robust tracking algorithm via feature grouping and 2DPCA. In this work, we represent the tracked object by using 2DPCA bases and a feature grouping. With the proposed model, we can reduce the effect of abnormal pixels on tracking algorithms. We obtained the tracking result using Bayesian maximum a posteriori probability estimation framework and designed a stable and robust tracker. Then, we explicitly take partial occlusion and misalignment into account for appearance model update and object tracking. Experiments on challenging video clips show that our tracking algorithm performs better than several state-of-the-art algorithms. Our future work will be the generalization of our representation model into other related fields.

Acknowledgments

This research described in this paper was supported by the Fundamental Research Funds for the Central Universities (DC110321, DC120101132, and DC120101131). This work was supported by project of Liaoning Provincial Department of Education (L2012476 and L2010094) and by National Natural Science Foundation of China (61172058).