Abstract
The behavior identification of the target trajectory is one of the important issues in space behavior analysis. Since the target trajectory model obtained from a fixed view cannot be adapted to the change of the observation perspective, it needs to be retrained when being faced with a new view, which leads to a great amount of increment in application cost. This study proposes a hidden Markov model (HMM) based on the cross-view transfer learning and the recognition method that firstly constructs a linear mapping relationship between the observation matrices of the source and target view utilizing the domain trajectory of the HMMs and obtains the observation matrix parameters of the target domain through the mapping system. Secondly, the transfer probability of the source domain is further optimized to obtain the target domain of the HMM and to identify the behavior of the target domain trajectory utilizing a small number of samples from the view of the target domain. The experimental results denote that the proposed method could effectively realize the identification of the trajectory behavior utilizing a small sample size in the target domain and would greatly reduce the application cost of the identification of the cross-view target trajectory.
1. Introduction
Target trajectories reveal important information on target behavior, and the identification of the trajectory is an important part pertinent to understanding an event and behavior analysis. In real applications, a video surveillance system often composes of multiple cameras located at different angles. The motion trajectory of the same target is displayed differently from diverse perspectives. The training of the trajectory model relies on samples from a fixed perspective. When the perspective alters, new samples need to be collected and retrained, leading to the excessively high training cost for the trajectory model, which would greatly raise the application cost of the identification of the trajectory behavior. This is not conducive to its promotion. Pointing out that it has a high application value and practical importance to research on the behavior identification method of the low-cost cross-view target trajectory.
Hidden Markov model (HMM) [1–8] could effectively model time-series signals and is a powerful tool to model the target tracking. Qiming and Cheng propose that trajectory modeling is executed according to the trajectory coordinate sequence [1]. In this method, the trajectory sequence is firstly clustered, and then, the corresponding HMM trajectory model is attained utilizing the training of the diverse categories. Dapeng et al. suggest that the HMM is utilized to model the trajectory of the enemy submarine and to predict the trajectory of the target to provide operational information support for warship operations [2]. Hervieu et al. propose that the curvature and velocity of the coordinate points are utilized for the HMM trajectory modeling, and the trajectory characteristics are invariant to translation, rotation, and scaling [3]. Qian and Lau suggest that a layered HMM is proposed to model the continuous target trajectory in a multilayer scene that leads to both a low-level HMM to model the single-layer trajectory and a high-level HMM connecting the multilayer trajectory to form the target trajectory model [4].
On the other hand, transfer learning has gradually become an important research area in machine learning in recent years since it mainly has resolved the problem related to the learning of the cross-domain knowledge and could realize the model migration when the data of the target domain is unlabeled and the data of the source domain is called a multisource. Moreover, it has a high theoretical value in many practical problems. Transfer learning, which relaxes two basic assumptions in traditional machine learning, is a new method in the field of machine learning. Its main characteristics could be summarized as follows: (1) there must be enough samples to learn for a good classification model. (2) Both the training samples were utilized for learning, and the new test samples meet the conditions of independence and identical distribution properties. Some scholars have proposed various HMM transfer methods for some cross-domain applications. For instance, Van Kasteren et al. propose that a parameter transfer learning method utilizes the HMM that constructs the sensor mapping relationship according to the location information of the spatial structure and then finds out the parameters of the HMM utilizing the EM algorithm [9]. Zheng et al. propose that, it takes the received real-time signal intensity of mobile devices as the observed value to construct the HMM positioning model and realizes the transfer of it at different times in the same position [10]. Bingtao et al. suggest that giving weight to the sample data of the source domain by calculating the similarity between the source domain and the target domain sample data, then finds out the HMM parameters in the weighted data set, and improves the learning algorithm to realize the instance-based HMM transfer learning [11]. Similar research studies conducted by Kim et al. propose that an HMM transfer method utilizing the maximum a posterior (MAP) and maximum likelihood linear regression (MLLR) is applied to speech and text recognition [12–15]. Some empirical research studies related to HMM are Fei et al. in [16, 17].
As a contribution to this research area, to solve the low-cost application problem of the HMM trying to determine the trajectory behavior in different perspectives, this manuscript proposes an HMM-based transfer learning and recognition method for the behavior of the target trajectory where the cases consist of the perspective samples of the sufficient source domain, and the perspective samples of few marked target domain exist. This method achieves the purpose of transfer learning and recognition by transferring the trajectory behavior model in the source perspective domain and optimizing a small number of samples in the target domain, which provides an effective way to resolve the mentioned problem.
2. The Proposed Model: Trajectory Modeling and Simulation
2.1. Target Trajectory Model Based on the HMM
Hidden Markov model (HMM) is a probabilistic model describing the process of time series, which can be described by five components denoted by a quintuple . Moreover, its simplified form is represented by :(1)N indicates the number of hidden states in the model. All states in the model are interconnected, and any state can be reached from other states.(2)M represents the observation symbol of each hidden state in the model, i.e., the number of observation states.(3)A is called the probability distribution of state transition. , where, , , and . Qt is called the hidden state at time t.(4)B is called the probability distribution of the observable state in the hidden state at moment j, , where, . Qt is called the observable state at time t.(5)π is called the distribution of the initial state , where .
According to the basic definition of the HMM, this manuscript firstly conducts the HMM modeling for the target trajectory. The direction angle of the trajectory of the target in a unit of time is utilized as the observation characteristic. The sequence of the angle representing the trajectory information is called the observable state vector. The hidden state is called the transfer characteristic of the angle change of the target trajectory. The angular direction of the target trajectory is computed bywhere and are called the target positions at time t and t − 1, respectively. According to , the observed value Qt is attained by discretizing the 24-direction freeman code depicted in Figure 1. The parameters of the trajectory of the HMM are obtained by training.
The trajectory presented in Figure 2 is utilized for modeling. The coordinate sequence of the target trajectory is presented in Table 1. The angle characteristic is computed by (1), and its discretization is presented in Table 2. While the number of the hidden states called N is set to 4, that of observation called M is set to 24. The probability vector of the initial state denoted by π is defined as uniform distribution. The transfer probability and the probability distributions of the observation are initialized randomly. The trajectory samples above are trained by the Baum–Welch algorithm to attain the parameters of the initial distribution π, transfer probability matrix A, and observation probability matrix B, respectively, that are presented in Table 3.
2.2. Simulation of the Target Trajectory
There are two important assumptions in the HMM, which are expressed as follows: (1) homogeneous Markov Chain hypothesis, that is, the hidden state at any time only depends on its previous hidden state. (2) Observation independence hypothesis, that is, the observed state at any time only depends on the hidden state at the current time.
As presented in Figure 3, the hidden Markov Chain based on the above hypothesis is determined by the initial state probability vector and the state transfer probability matrix A. Hence, the result of which is to generate an unobservable hidden state sequence. The observation probability matrix B and the hidden state sequence are combined to determine the way of generating an observable sequence.
Given the HMM model in the form of , the observation sequence could be generated by the algorithm steps presented in Table 4. According to the initial coordinates of the trajectory and the sequence of the observed values, the coordinates of the trajectory sequence could be uniquely determined.
The trajectory of the HMM is trained and simulated under four different circumstances, and the results are presented in Figure 4.
(a)
(b)
(c)
(d)
2.3. Statistical Analysis of the Trajectory Characteristics from Different Perspectives
To further analyze the relationship between the characteristics of the target trajectory from diverse perspectives, this manuscript utilizes both a straight line and curve trajectories as illustrative examples to examine their statistical characteristics of the observation characteristics from diverse perspectives. When the target linear trajectory is a concern, Figure 5 denotes the observation of the linear motion of the same target from two different perspectives. The statistical features of the characteristics of the target trajectory observation from two different perspectives are presented in Figure 6 suggesting that the statistical envelope of the trajectory characteristics between the two perspectives is very similar, while the center of gravity is different.
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
The trajectory of the curve motion is presented in Figure 7. Statistical analysis is conducted on the observation characteristics from different perspectives, and the characteristic of the statistical curve is presented in Figure 8 suggesting that the statistical characteristic of the curved trajectory from different perspectives have a certain similarity.
(a)
(b)
To sum up, although there exist differences in the perspective, a certain correlation exists in the performance of the feature sequence of the target trajectory. If there were more tracks along a certain direction, they would accumulate near the characteristic coding of that direction. Hence, it can be considered that the characteristic frequency curves of the target trajectory from different perspectives denote a certain linear translation.
3. Transfer Learning Based on the Trajectory of the HMM
For the recognition of the cross-view target trajectory, small sample data are employed in this research utilizing the transfer learning strategy to obtain the trajectory of the HMM model for the target view and to realize the low-cost modeling of the behavior model of the cross-view trajectory. The basic idea of the HMM model based on the transfer learning in this manuscript is presented as follows: firstly, according to the statistical translation of the trajectory characteristics in the different perspectives in Section 2.3, the linear regression model is constructed to attain the mapping between the source domain and the target domain concerning the HMM observation probability matrices of the characteristics, which leads to obtaining the observation probability. Secondly, the transfer probability matrix is randomly initialized, and the simulation data are generated. The objective optimization function is constructed according to the similarity between the simulation and the domain data of the target. Finally, the parameters of the transfer probability matrix for the trajectory HMM utilizing the target domain are obtained through the iterative optimization, and the transfer learning process of the cross-view HMM is realized.
3.1. Transfer of the Observation Probability Matrix B
The key to the transfer of the observation probability matrix is to determine the mapping relationship between the feature spaces of the target domain and the source domain. By utilizing this mapping relationship to transfer the HMM parameters of the source domain, a parameter model is formed that can reflect the target trajectory characteristics of the target domain.
In this manuscript, the least square method is adopted to construct the mapping model, and the trajectory characteristic of the mapping model under two different perspectives is assumed to be represented bywhere and b are called the coefficients of a curve equation fitting by feature mapping, Os is called a coding sample of the source domain, and is called the coding data of the target domain after the mapping. The objective function is defined bywhere Ot is called the encoding data of the real target domain.
According to the statistical analysis of the trajectory characteristics in Section 2.3, the transfer of the parameter of the observation probability mainly includes three steps. First, both perspective trajectory samples of the source domain and target domain are collected. Each group of samples contains the source domain that is labeled as the samples of the same behavior and as a small number of target domains that are labeled as the samples. The source domain of the HMM trajectory model denoted by is obtained according to the method of the training model in Section 2.1. Then, the linear regression mapping model about source domain and target domain features is constructed. Finally, the observation probability denoted by BT of the target domain is attained utilizing the relation transfer matrix BS of the linear regression.
3.2. Transfer Learning Algorithm of the Transfer Probability Matrix A
Ignoring the effect of the initial distribution, the initial distribution πS of the source domain is directly migrated to the target domain to form πT, and the transition probability of the source domain acts as the transition probability of the initial value of the target domain. Then, the transferred initial HMM is denoted by . In this study, the optimization algorithm is utilized to adaptively optimize the transfer of the model parameters to attain better performance of the target domain model. The parameters of the optimized target source model are more suitable for the target domain data, that is, the more similar the simulation data of the optimized model to the target domain data would be, the better it would converge. The objective function is defined bywhere is the mean value of the data generated by the simulation and is called a variable in the optimization problem. The trajectory data is simulated according to the algorithm steps in Section 2.2. The inherent characteristics of the HMM require that the transition matrix is non-negative and the sum of row elements is equal to 1, so the constraint conditions of this optimization problem are that the elements of the transition probability matrix and are non-negative and the sum of row elements is equal to 1. The measurement of similarity is determined by the Euclidean distance, which is defined bywhere and are the mean values of the feature sequence set of the simulated trajectory model represented by and target domain, respectively. Hence, both of which belong to the same trajectory category. The similarity computation is defined by
Utilizing the constraints and objective function presented above, the interior point method is undoubtedly one of the most suitable methods to resolve the optimal value . Table 5 presents the steps of this solution procedure leading to the optimality of employing the interior point method. The similarity between the simulated data of the target domain model and is calculated. If the similarity is greater than or equal to the similarity threshold denoted by , the obtained from the previous optimization step is re-entered into the iteration of the interior point method as the initial value until the computed similarity is less than the threshold , and then, the HMM of the target domain is attained.
3.3. HMM Transfer Learning and Recognition Framework
To resolve the problem of the recognition of the cross-view target trajectory, this manuscript proposes a cross-view HMM transfer method. The algorithm framework is presented in Figure 9. Utilizing the objective function analysis, the statistics of the characteristics of the target trajectory under the two perspectives suggest a linear relationship. Therefore, the mapping function between the characteristic spaces of the source domain and the target domain is first obtained by resolving an optimization problem based on the linear regression model. Utilizing the mapping function, the parameters of the observation probability of the source domain are transferred. Then, the similarity between the generated data of the simulated HMM and the labeled data obtained from the target domain perspective is computed to further optimize the transition probability of the source domain. Hence, the target domain HMM is obtained. Finally, the forward algorithm is utilized to assess the match, classification, and identification of the cross-view target trajectory.
4. Experimental Results and Analysis
4.1. Experimental Evaluation
The data sets employed in this experiment are all self-collected data, which come from the simultaneous shooting of two cameras installed in the 10 × 12 m2 room. The 100 samples of three kinds of trajectories (300 in total) from view A of the first camera can better reflect the real movement behavior of the moving target. The 300 samples are taken as the source domain data set, while the corresponding 300 samples of the view B of the second camera are taken as the target domain sample. All collected samples are manually marked. In this manuscript, the performance of the model is assessed by the accuracy measurement defined bywhere naccu is called the number of the correct trajectory classification and Num is called the total number of the test data.
4.2. Analysis of Experimental Results
The first experiment aims to qualitatively compare the performance of the HMM transfer method described in this manuscript with the one that existed in the literature [9]. In this experiment, k is set to some sample values such as (k = 10, 15, 20, 25, 30, 40, and 50) representing each type of the trajectory in the sample set of the target domain that is randomly selected and combined with the sample set of the source domain as the training data. In addition to k number of the selected samples, all other data in the sample set of the target domain are employed as the test data. Figures 10–12 are the comparisons of the accuracies of the test data between the source and the target domains when the number k of each type of trajectory sample changes with the angle of the view θ = 30°, θ = 45°, and θ = 60°, respectively. The experimental results suggest that the accuracy of the method in the experiment is improved with the increment of the number of the target domain samples, k. As the number of the target domain samples increases, the learning algorithm could extract more knowledge from the target domain. When only θ = 60°, the accuracy of trajectory 2 does not improve, or even decreases, indicating that overfitting occurs in this case. It is worth noting that the accuracy of this algorithm increases faster than other algorithms with the increment of samples. When the included angle varies from 30° to 45° and 60°, and k is kept constant, the corresponding accuracy of most trajectories decreases, indicating that the increment of the angle of the view leads to more obvious deformation of the trajectory at different included angles (Figure 12). The experimental results also suggest that when compared with the method researched in [9], this method could achieve a higher recognition rate of the behavior trajectory of the cross-view target utilizing only a small amount of the marked data from the target domain perspective, which would greatly reduce the annotation work of the target domain perspective.
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
(a)
(b)
(c)
(d)
In the second experiment, according to the results of the previous experiment, the number of training data of the target domain perspective is fixed to 50. To verify the effectiveness of this method, the proposed method is compared with both Method 1 and Method 2, and the results are recorded. In Method 1, the marked training data from view B are utilized to find out the parameters of the model. The marked data in the number of view B are small, so the leave-one-out cross-validation technique is implemented. In this technique, a group of 20 samples is taken from the target domain data set as the test set, the rest are left as the training data until all samples have been tested. A total of M times (the size of the data set) are conducted and calculated, and finally, the accuracy is averaged. In Method 2, only the marked training data in view A are utilized to conduct the Baum–Welch algorithm to attain the HMM, and the determined parameters of the model are utilized to attain the accuracy of the test data set of view B.
Tables 6–8 present the accuracy rates of the different methods at the different θ values. The purpose of the comparison with Method 1 is to decide whether a trained model of the source domain could be utilized directly across visual angles. The results imply that the effect is poor, especially from different views since the knowledge extracted from the source domain cannot be directly applied to the target domain, especially when the two domains differ much. Compared with Method 1, the method in this manuscript greatly improves the outcomes of the recognition. This indicates that although the HMM trajectory modeling is very successful when implemented from a single perspective, its performance would significantly decrease when both cross views and crossing view have a great impact on trajectory recognition. Compared with Method 2, the recognition effect is similar, indicating that when a small data set of the target domain are provided, the performance of the HMM transfer model is almost the same as that of the training model with a large data set of the target domain. In conclusion, the transfer learning method in this manuscript makes use of the prior knowledge that has been already extracted in the source domain model and hence, could effectively identify the cross-view target trajectory. Besides, it does not require a high number of target domain samples. Since there are enough samples in the source domain, which contains a large amount of characteristic information, the transfer learning could utilize only a small target data set to combine with these prior pieces of knowledge to attain better outcomes.
In the third experiment, according to the outcomes of the first experiment, the training data of the target domain perspective are fixed to 50, and the performance of the proposed method is compared with the HMM transfer method described in the literature [10, 11].
Seen from Tables 9–11 that the HMM transfer method in this manuscript is more effective than other HMM transfer methods under the same experimental settings. In the transfer recognition of the cross-view HMM trajectory model, the accuracy rate of the proposed method reaches over 85% in most cases, and the half could be over 90%.
The experiments present that the linear regression model based on the least square method can be successfully applied to the transfer learning method of target behavior recognition. The characteristic representation of both the source domain and target domain differs due to the variation of a visual point. To overcome this issue, we need to find statistical patterns between the characteristics of the source domain and the target domain. Then, the curve needs to be fitted. By doing so, transferring the probability distribution of the emission of the source domain adaptively is transferred to the target domain. Hence, it leads to transferring the correlation knowledge between different perspectives and behaviors from one domain to another. The existence of a small number of the labeled samples in the target domain provides a reference for the optimization of the state transition probability of the source domain. The improved and optimized HMM performs well on the data set with higher accuracy and better robustness.
5. Conclusion
This manuscript mainly deals with identifying the behavior categories of the trajectory data of the target behavior when the cross perspective is a concern. In the case of having insufficient labeling data of the target domain, having difficulty in labeling data and dealing with a higher cost, the problem is transformed into the identification problem of the feature sequence of the trajectory utilizing the HMM transfer learning. Through knowledge transfer of relevant data from different perspectives, the classification model is constructed, and then, the behavior trajectory category of the data to be classified in the target domain is identified. In this manuscript, first, according to the data that is labeled from the perspective of the source domain, an HMM is constructed to train the parameters of the triples; second, the frequency of the coding sequence features utilizing the same type of data that is labeled from the perspectives of the source domain and target domain is computed. Then, the mapping curves of the two perspectives are fitted by the least square. Hence, the observed probability matrix is adaptively transferred by mapping relationships. Afterward, the transition probability based on a small labeled data of the target domain is transferred to the target domain through an optimization algorithm. Therefore, the target domain classification model is constructed. The experimental results on the data set denote that the transfer learning method based on the HMM could construct the classification model, and it has a better performance in the recognition of the cross-view target trajectory when there exist a small number of labeled data in the target domain perspective.
Data Availability
All the data are included in the manuscript, and further data can be requested from the corresponding author upon reasonable request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Acknowledgments
This study was supported by the National Natural Science Foundation of China (no. 61673318, target trace analysis and abnormal behavior warning in structural space), the Key Industrial Chain Project of Shaanxi Province (no. 2020ZDLGY04-04), and the Science and Technology Planning Project of Xi’an City (no. GXYD14.1, intelligent identification system of residue state for Subway construction).