Expressway Traffic Flow Missing Data Repair Method Based on Coupled Matrix-Tensor Factorizations

Jiang, Hui; Deng, Hongxing

doi:https://doi.org/10.1155/2021/2919073

Mathematical Problems in Engineering

On this page

Abstract Introduction Results and Discussion Conclusions Data Availability Conflicts of Interest Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2021 | Article ID 2919073 | https://doi.org/10.1155/2021/2919073

Expressway Traffic Flow Missing Data Repair Method Based on Coupled Matrix-Tensor Factorizations

Hui Jiang^1,2and Hongxing Deng¹

Academic Editor: Frederic Kratz

Received05 Aug 2020

Revised15 Jan 2021

Accepted28 Jan 2021

Published11 Feb 2021

Abstract

Traffic flow data is the basis of traffic management, planning, control, and other forms of implementation. Once missing, it will directly affect the monitoring and prediction of expressway traffic status. Regarding this, this paper proposes a repair method for the traffic flow missing data of expressway, combined with the idea of coupled matrix-tensor factorizations (CMTF), to couple the auxiliary traffic flow data into the main traffic flow data and to construct the coupling matrix-tensor expression of traffic flow data, and the alternating direction multiplier algorithm is used to realize the repair of missing traffic flow data. Combined with the measured data of expressway traffic flow, the experimental results show that, under different missing data types and missing rates, the proposed method outperforms the methods lacking auxiliary traffic flow data and achieves a good repair effect, especially for high miss data rates.

1. Introduction

The traffic management department requires traffic flow data to be complete to realize accurate assessment and prediction of the traffic status. Thus, the repair of the missing data is crucial for effective management of the traffic flow.

In the past few decades, researchers have shown much interest in the problem of repairing the missing traffic flow data. The existing research on this area can be categorized into three types of data composition: vector, matrix, and tensor. Below, we elaborate on the existing studies to provide an overview of the state of the art.

First of all, researchers used the traffic flow data in the form of a one-dimensional vector to repair the missing parts based on spatial or temporal correlation. Gold et al. classified the reasons for the missing data based on the characteristics of expressway traffic flow data and repaired the missing data using nuclear regression, polynomial regression, and linear interpolation [1]. Based on the hotspot analysis of the spatial correlation of traffic data, Conklin et al. used the similarity of the adjacent lanes to repair the missing data [2]. Smith et al. utilized the temporal correlation of traffic data and repaired the missing parts by using the average value of the historical data at the corresponding time and the exponential smoothing value of data at adjacent periods from several days ago [3]. Based on the spatial and temporal correlation, Jiang and Gang predicted the value of data in previous periods and adjacent sections to repair the missing data [4]. Jin and Wang proposed a traffic flow parameter correction method based on correlation analysis from the perspective of statistical analysis of traffic flow data [5]. Gheyas and Smith mainly focused on the measured traffic flow data and utilized the methods of historical trend and exponential smoothing to repair the missing parts. By comparison, these two methods were proved to be more effective in dealing with continuous abnormal data [6]. In consideration of the spatial-temporal characteristics of the traffic flow data, Wang et al. proposed a repair method based on the 3D shape function and the spatial-temporal interpolation [7]. Chen et al. proposed a long short-term memory (LSTM) method, which was built on the normal time-series using the calculation error of the Gaussian Bayesian model. Experimental results on three real datasets showed the advantages of this method [8].

Then, the data is organized into a matrix to repair the missing data. Kurucz et al. proposed a low-rank matrix completion algorithm using the path norm minimization of the matrix to replace the minimization of the rank function [9]. Qu et al. performed (Bayesian) principal component analysis to reconstruct the traffic flow data into a matrix. The results showed the effectiveness of this method in data repair for low missing rates [10]. Tang et al. made use of fuzzy c-means clustering and genetic algorithm to repair the missing data based on a matrix. The repair effect of data on different time scales was quantified. The results revealed that the fusion model had a significant repair effect [11]. Lu et al. proposed a data repair model structured on an improved multiscale principal component analysis. Based on the spatial-temporal correlation characteristics of the traffic flow data, they calculated the correlation coefficient of the missing data to estimate its true value [12]. Similarly, Jiang proposed a data repair algorithm using the fusion matrix low-rank decomposition [13].

The data composition in the form of multidimensional tensor expands the vector and matrix to higher dimensions. In recent years, it has proliferated in machine learning (clustering and dimensionality reduction) and other fields [14]. This method can repair the missing data by organizing the traffic flow data into tensors. Acar et al. proposed an algorithm based on CP decomposition and gradient optimization and then proved that the algorithm had high flexibility [15]. Silva and Herrmann proposed a tensor repair method based on the hierarchical Tucker decomposition to repair the missing seismic data [16]. Tan et al. proposed a traffic data repair method based on tensor for the first time, which achieved good experimental results, especially in the case of high data missing rates. Subsequently, Tan et al. proposed several other tensor-based repair methods, which proved that the tensor-based method was still useful, especially for the cases with high missed data [17]. Liu et al. proposed a low-rank tensor repair algorithm to solve the problem of image repair and proved that their method was better than the tensor model using CP decomposition [18]. Based on the tensor theory, Xiang used the spatial-temporal characteristics of the traffic data for modelling, which significantly improved the accuracy of data repair [19]. Zhang focused on the microwave data of the urban expressway and repaired the data using a CP-ALS algorithm based on tensor decomposition. The results proved that their method could effectively improve data quality [20]. Zhang et al. proposed an iterative tensor decomposition (ITD) method to repair the lost data in the ANPR system. They used the multidimensional intrinsic correlation of the traffic flow data for the detection of missed parts and their repair. The results showed that this method could accurately detect and repair the missing data under various missing rates [21].

To summarize, the one-dimensional vector was highly dependent on the historical information, requiring stable traffic flow data within a period or space. Thus, it could not use the information collected after the data missed. Besides, if the data missed within a long period, the error in the repair results turned to be high. On the other hand, although the two-dimensional matrix made full use of the spatial-temporal correlation characteristics of the traffic flow data, it was limited to the 2D data. Furthermore, it could not simultaneously utilize the multimode correlation of the traffic flow data, which resulted in low repair accuracy. In recent years, the high-dimensional tensor model has attracted significant attention in data repair, and some progress has been achieved, especially for the repair of traffic flow missing data. However, most of the existing efforts have focused on the model construction of spatial and temporal correlation of the traffic flow data.

In summary, it is necessary to introduce a data repair method, which makes full, reasonable, and effective use of multidimensional information between the traffic flow data. To do so, this work for the first time proposes a traffic flow missing data repair model based on the coupled matrix-tensor factorizations. Compared with the existing work, we make two contributions: (1) we aim to obtain better multidimensional information of traffic flow data, which is divided into main traffic flow data and auxiliary traffic flow data, and coupled the auxiliary traffic flow data into the main traffic flow data to construct the coupling matrix-tensor expression of traffic flow data. (2) In order to get better data repair effect, we used alternating direction multiplier (ADMM) algorithm to decompose large global problems into many smaller and easier local subproblems to solve the optimal solution. By comparison with the prior method, we validate the effectiveness and feasibility of the proposed methodology in data repair.

The rest of this paper is organized as follows. Firstly, the tensor theory is explained in Section 2. Then, the traffic flow data analysis is provided in Section 3. Section 4 presents the missing data repair method based on CMTF, where the results and discussion on the findings are elaborated in Section 5. Finally, Section 6 concludes the paper.

2. Tensor Theory

As an extension of vector and matrix to a high-dimensional space, tensor is a mathematical representation of a multidimensional data. The dimension of a tensor is also called the modulus or the order [22, 23]. Hence, if we consider a vector as a tensor of the first order and a matrix as a tensor of the second order, it refers to an array of three or more dimensions as a tensor of a high order, tensor for short. The third-order tensor with dimensions , , and shown in Figure 1 is an example of this.

Tensors can be expressed in either a fibrous or a slice form [24, 25]. Any two dimensions in the third-order tensor are kept unchanged. In this case, the tensor can be expressed by a column fiber line , a fiber , and a pipe , where the colon represents the dimension all of elements. Alternatively, the tensor can be expressed in the form of a horizontal slice , a side slice , and a front slice when the other two dimensions are changed.

2.1. Tensor Matrixing

Tensor matrixing is a process, in which the elements of a tensor are rearranged into matrices. To be specific, tensor operations are simplified by expanding tensor into the matrix in the modulus.

The CP decomposition is one of the earliest and most in-depth tensor decomposition methods. It can be interpreted as decomposing a tensor into the form of a sum of rank-1 tensors. For , the CP decomposition can be expressed in the following matrix form:where is the vector corresponding to rank-1 tensor. Similarly, B and C are the vectors and corresponding to rank-1 tensor. Thus, A, B, and C are the factor matrices decomposed by CP decomposition, where , , and . The slice of a tensor can be represented as follows:where the symbol “⊙” can be specifically described as the output value of 1 when two input variables are the same, otherwise the output value is.

In practice, however, the column vectors in the factor matrix are usually normalized: . Thus, the CP decomposition can be expressed aswhere is the regularization parameter.

2.2. The Coupled Matrix-Tensor Factorizations

The joint analysis of data is quite useful in understanding the underlying structure of complex datasets. Here, the coupled matrix-tensor factorizations (CMTF) can capture the potential structural characteristics of heterogeneous data by coupling the matrix data into the high-order tensor model.

To explain this coupling model in detail, we assume that a third-order tensor and a matrix are derived from different datasets, and U is coupled to a dimension of . The schematic diagram of the CMTF is shown in Figure 2.

The underlying structure of the dataset can be extracted by the CMTF, where the R-component of a tensor and a matrix U is defined aswhere are the factor matrices of the tensor extracted by CP decomposition, where , , and .

The factor matrices and are extracted from matrix U, which can be represented aswhere A is the common mode factor matrix of the tensor and the matrix U on a certain dimension (modulus).

To ensure that the datasets are codecomposed and share the same potential space, the matrix should have the same underlying low-rank structure as the tensor on at least one dimension.

3. Analysis of the Traffic Flow Data

3.1. Traffic Flow Missing Data: Causes and Types

The complete traffic flow data is key for the traffic management department to assess the traffic status. However, due to the influence/restriction of traffic conditions, meteorological environment, and many other factors, the traffic system has formed a complex operating environment. There are four main reasons of the missing data: detection equipment failure, power failure, communication network failure, and environmental factors.

The missing data types can be summarized as follows:(1)Missing Completely at Random (MCAR). It refers to the type where the missing traffic flow data are completely random and independent of each other; that is, the probability of the missing data is not affected by other factors. The missing data location is represented as a random discrete distribution.(2)Missing at Random (MAR). It refers to the type where the missing traffic flow data is not completely random. The missing data is not correlated with itself but with other adjacent data points. The missing data location is a random continuous distribution.(3)Missing at Nonrandom (MANR). It refers to the type where there is a certain correlation between the missing data and its characteristics, which also depends on other missing data. This loss is generally caused by the system itself.

One of the earlier studies in the field [26, 27] assumed that the MANR data was found and then deleted, where subsequent studies mainly used the MCAR and MAR. To comprehensively reflect the real traffic flow under the condition that the complexity of the data is missing, the follow-up work will be according to the three following case studies:(1)The missing data type 100% are missing completely at random (MCAR)(2)Lack of missing at random (MAR) data type 100%(3)Lack of missing completely at random data type and random loss, 50% each, namely, lack of hybrid (MIX-M)

3.2. Analysis of the Traffic Flow Data

In analysis of traffic flow data variation, the traffic flow data is divided into main traffic flow data and auxiliary traffic flow data according to whether it can directly reflect the actual traffic status of the road.

3.2.1. Analysis of the Main Traffic Flow Data

This paper defines the main traffic flow data as the data that can directly reflect the state change of the system. Specifically, it is described as the index value that can represent the state of the traffic flow through the data, such as flow, speed, and occupancy, which plays a leading role in the repair of the missing data.

The phenomena of the main traffic flow data changes with time and space are called the spatial and temporal distribution of traffic flow, which has the characteristics of temporal and spatial correlation. The time correlation analysis of the traffic flow data describes the change characteristics in the time domain. The travellers determine the time regularity of the traffic flow. According to the traffic data of different times of a month, week, and day, the similarity of the different degrees is reflected. Considering the different time organization modes of traffic flow data, the time correlation analysis will be performed according to the correlation analysis of different weeks of a month, days of a week, and periods of a day. Some studies have revealed that the traffic flow data has strong daily and weekly correlations [3–5, 7, 16, 18, 28–31].

The spatial correlation analysis of the traffic flow data includes two main parts: (i) the data correlation analysis of a lane and its transverse adjacent lanes and (ii) the data correlation analysis of a lane and its longitudinal upstream and downstream sections of the same lane. Some studies have proved that the flow data of an urban expressway has a strong correlation with its horizontal and vertical spaces. In theory, the spatial and temporal correlation of the traffic flow data can be used to effectively repair the missing data.

3.2.2. Analysis of the Auxiliary Traffic Flow Data

The core of data repair is to establish a relationship between the known and missing data. The more information related to missing data is contained in the data repair model, the better the effect of data repair will be. The auxiliary traffic flow data refers to the fact that the traffic flow data itself does not have a direct response state of the quantitative indicators. However, in the process of data repair, it can provide certain information about road conditions and environmental characteristics. Thus, the acquisition and use of auxiliary data will greatly help to restore the missing data.

The importance of auxiliary traffic flow data in data repair is explained as follows: Firstly, different weather conditions may affect the operational behaviours of vehicles, for example, in rainy weather, due to the slippery road surface, which will directly affect the speed of vehicles, thus leading to similar running state of vehicles in similar weather conditions. Secondly, the analysis of the running speed of vehicles in different lanes of the same section shows that the vehicles using the middle lane will travel at different speeds compared to the ones in the edge lanes [32, 33]. The difference in the information added for the traffic flow data repair helps to improve the restoration precision of the traffic flow data.

4. The Missing Data Repair Method Based on CMTF

The joint analysis of multisource data improves the understanding of the underlying structure of complex datasets. In the existing research, the repair of traffic flow missing data is mostly based on spatiotemporal information, while the importance of auxiliary information in data repair is neglected. Furthermore, due to the heterogeneity of the latter, there are different quantitative standards in practical application. Therefore, a method is required to integrate the auxiliary information about traffic flow with the spatial-temporal information to repair the missing data.

The CMTF model can fit the matrix data into the tensor model through the coupling and capture the underlying structural characteristics of the heterogeneous data, realizing the joint analysis. Thus, combining with the CMTF model, this paper proposes a missing data repair method based on multisource information.

4.1. The CMTF Representation of the Traffic Flow Data

The schematic diagram of the CMTF model is shown in Figure 3. There are three datasets in the figure: the tensor with missing traffic flow data, the feature matrix of weather feature data, and the lane feature data matrix.

The third-order tensor is constructed from the perspective of the spatiotemporal correlation of the main traffic flow data, where represents the longitudinal time-series data of the same day in consecutive weeks, represents the lanes of the same detection section, and represents the transverse time-series data of a day.

According to the auxiliary traffic flow data, also considering that the third-order tensor contains the same underlying low-rank structure with the construction of the third-order tensor, the two following feature matrices are constructed:(1)Weather Feature Matrix. Different weather conditions, for example, rain, will affect the operational behaviours of vehicles. The wet roads and low visibility will directly affect the speed of vehicles, resulting in a similar running state for all vehicles. The weather data matrix is in the composition of feature dimension, where represents the longitudinal time-series data of the same day in consecutive weeks, and represents the transverse time-series data of a day. Here, the sunny, cloudy (foggy), and rainy/snowy days are assigned the values of 1, 2, and 3, respectively. Thus, the element in the matrix is an integer varying from 1 to 3.(2)Lane Feature Matrix. The running state of the vehicles traveling towards the same direction in the different lanes of the same section will have typical differences. For example, near the separator, the operating characteristics of the vehicle lane edge will be different, which will be considered in the traffic flow data repair. For the lane feature matrix , where represents the J lanes of the same detection section and N represents the lane attribute values. Starting from the median, lane of attribute assignment is, respectively, 1, 2, 2, and 3. Thus, the elements in matrix are integers varying between 1 and 3.

The tensor containing missing data is constructed for the traffic flow spatiotemporal data, where the lane and weather feature matrices are coupled in lane and date dimensions, respectively. In this way, the data fusion problem is transformed into the CMTF decomposition problem. Thus, the missing data in the original tensor can be repaired by decomposing the tensor and matrix into multiple datasets and then reconstructing the tensor.

4.2. Objective Function Construction

The factor matrices , , and are obtained by the CP decomposition of tensor , where , , and . Decomposition of matrices and , respectively, gives the factor matrices , , , and , where , , , and . Thus, the objective function of CMTF decomposition can be expressed as

The missing data in tensor can be repaired by coupling it with a known matrix.

Next, denote by the set of the squared errors between the original tensor and the factor matrices , , and , where the reconstruction tensor is obtained by CP decomposition. The smaller is, the better the repair effect of missing data in the original tensor will be. is the rank of the tensor, and is the regularization parameter. , , and are the norms corresponding to each factor matrix. Thus, the tensor is decomposed and expanded according to one module CP aswhere is the matrix representation of the tensor on one modulus. The sets and represent the squared error between the original matrix and the matrix reconstructed from the factor matrix, which are obtained by matrix decomposition as

According to the analysis of the CMTF model, the matrix should have the same underlying low-rank structure as the tensor in at least one dimension. Therefore, the matrix and the tensor should have a common modulus factor matrix. The assumption is that and are equal; and and are equal. Therefore, two global optimal variables and are introduced to satisfy the application premise of the CMTF model:

Thus, the optimization problem of the objective function is described aswhere represents the factor matrix set: . Therefore, the problem of missing data repair is transformed into the problem of solving the optimal value of the objective function .

4.3. Optimization Solution

For optimization, the alternating least square (ALS) and gradient descent method are commonly used in the literature [34–40]. Since both the matrix and the tensor decomposition are a nonconvex problem, the commonly used methods cannot achieve the global optimum. To resolve this issue, the alternating direction multiplier algorithm (ADMM) is suggested [41]. In this method, large global problems are decomposed into many smaller and easier local subproblems, obtaining large global optimal solutions.

Thus, when is finding the optimal solution, it can be interpreted by solving the local , , and optimal solution problems. The objective function of this problem can be given aswhere are the Lagrangian multipliers [31], is the penalty coefficient , and is the trace of the matrix, that is, the sum of the diagonal elements in the matrix. The iterative calculation formula of (9) is

Concerning , the partial derivative of each factor matrix can be obtained as

By setting ,

Similarly,

Similarly, it can be inferred further that

If set , then , and it can be inferred further that

Based on the above-given analysis, the main steps of the CMTF-based traffic flow missing data repair method are shown in Table 1.

5. Results and Discussion

5.1. Data Source and Analysis

5.1.1. Data Sources

The main section of an urban expressway is selected as the area to be analysed. The section includes five detection sections (in the east and west) with four lanes in each section and a total of 40 coil detectors. The data that will be collected by the detectors is shown in Table 2, where “TY” indicates the code of the detection section, “D” refers to the east, and “10 (2)” means Lane 2 of the detection section #10. The data is sampled from August 27, 2018, to September 29, 2018, with a sampling interval of 5 min. There are 161280 groups of data.

5.1.2. Judgment Criteria for Data Repair

The existence and reparability of the missing flow data have to be judged before starting a repair process. To determine whether anything is missing, the traffic flow data can be used. For example, if the flow data is displayed as “0,” it might be due to no vehicle passing through the detector or might be due to some missed data.

Under normal traffic conditions, the missing traffic flow data can be repaired according to the changing trend in the traffic flow. However, under abnormal conditions, such as traffic accidents conditions, once the data is “0,” it is impossible to judge the existence and (if exists) the reparability of the data.

Following the continuous progress in detection technology, the detectors have become more capable of realizing the preliminary discrimination of data anomalies. For example, the location detector can classify the data as valid, invalid, and/or lost. For repair judgment, this paper only considers the invalid data with the value of “0,” which is denoted as the missed data that will be repaired.

5.2. Experimental Scheme Design

(1)Data Representation. Let us construct the tensor for the target data, where “5” represents “Day-mode,” which means the longitudinal time-series data of five consecutive Mondays, “4” represents “Lane-mode,” which means the transverse spatial sequence data of four lanes in the same detection section, and “288” represents “Time-mode,” which means the 288 transverse time-series data obtained in one day. The weather feature matrix is , where “5” refers to five longitudinal time-series data of Monday, and “288” represents the 288 transverse time-series data obtained in one day. The lane feature matrix is , where “4” refers to the data of four lanes of the same detection section, and “3” represents the lane attribute values.(2)The evaluation index of the missing data repair effect. The root mean square error (RMSE) and mean absolute percentage error (MAPE) are used as evaluation indexes [14]. It should be noted that the smaller the difference between the repaired and actual data is, the smaller the RMSE and MAPE will be. These errors are calculated as follows: where is the actual data, is the repaired data, and is the number of repaired pieces of data.(3)Missing Data Type Design. To illustrate the data repair performance of the CMTF, three types of missing data were manually set up: MCAR, MAR, and MIX-M. Then, the data missing rates of each type are set to 10%, 20%, …, 90%. In order to avoid the influence of the randomly set data missing position on the repair result, ten settings were made for different data missing types, and the average value of the 10 repair effects (RMSE and MAPE) corresponding to each method was selected as the final repair result.

5.3. Analysis of the Experimental Results

The experiment environment is designed using Matlab-R2014a software, where the data repair based on the CMTF algorithm is realized by the CMTF_Toolbox_v1_1toolbox [15]. The relevant parameters of the model are obtained by cross-validation.

The TYD12(2) detector with relatively complete flow data on August 27, 2018, was selected for tests, which were loaded with three missing data types (MCAR, MAR, and MIX-M) and nine missing rates (10% to 90%).

Three methods, namely, matrix repair method [13] (Method 1), tensor repair algorithm (Method 2) [20], and the optimization algorithm of CMTF based on alternating least square (Method 3), are compared with the algorithm proposed in this paper (Method 4). The specific analysis of the data repair effect is as follows.

5.3.1. Repair Effect Evaluation of the MACR

For the MACR, the repair results are shown in Tables 3 and 4.

The results show that the RMSE and MAPE of all methods are relatively small and close to each other for low data missing rates. However, with the gradual increase of the missing rate, the RMSE and MAPE for all methods also increase gradually. For high missing rates, the RMSE of Method 3 and Method 4 remains around 15, which is significantly lower than that of the other two methods. This indicates that the multidimensional flow data has a significant effect on the repair of the missing data. The RMSE and MAPE of Method 4 are generally lower than the corresponding values of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS.

5.3.2. Repair Effect Evaluation of the MAR

For the MAR, the repair results are shown in Tables 5 and 6.

The results show that the RMSE and MAPE of all methods are relatively small for low missing rates similar to MACR. However, with the gradual increase of the miss rate, the RMSE and MAPE values also increase gradually, where Method 4 has the slowest growth rate. For high missing rates, the RMSE of Method 3 and Method 4 is still lower than 16.5%, which is significantly lower than the other two methods. This indicates that the multisource flow data has a more significant effect on the repair.

The repair effect of Method 4 is lower than that of Method 3, indicating that the local optimization solution based on ADMM is better than the one based on ALS. Compared to the MACR, the RMSE and MAPE of all methods under the MAR are relatively larger. This is because the missing data is continuous, which will affect the repair effect to some extent.

5.3.3. Repair Effect Evaluation of the MIX-M

For the MIX-M, the repair results are shown in Tables 7 and 8.

The overall trend is similar to that of MACR and MAR, but the RMSE and MAPE of all methods are in the middle. This is because the missing data in MIX-M includes both the discrete missing data of MACR and the continuous missing data of MAR. Thus, the existence of continuous missing data will affect the repair process.

The MIX-M can reflect the lack of traffic flow data. From Figure 4, for the missing rate of 50%, the MAPE of all methods is lower than 21%. When the missing rate exceeds 60%, the MAPE difference between the methods becomes larger. Here, Method 4 still has the best repair effect for high data missing rates.

Under the three types of data missing, all methods can repair the missing data at certain levels, where the repair effect of the MACR is the best, and the MAR is the worst.

In comparison, the repair effect of Method 2 is better than that of Method 1. This indicates that the tensor algorithm based on multidimensional data can improve the repair effect.

The repair effect of Method 3 and Method 4 is better than the other two, indicating that the addition of auxiliary data is useful in data repair.

The repair effect of Method 4 is better than that of Method 3; that is, the optimal solution of ADMM is better than the traditional solution that ALS offers.

Our proposed method has a best repair effect, which has the lowest RMSE and MAPE among all methods. Even for the MAR with a miss rate of more than 80%, the RMSE and MAPE of our method are still around 16.5 and 20%, respectively. This shows that the accuracy of traffic flow missing data repair can be effectively improved by adding auxiliary data based on sufficient and reasonable use of the spatial and temporal correlation between the traffic flow data.

6. Conclusions

To improve the repair effect of traffic flow missing data, this paper proposes a CMTF-based repair method. Different from the existing methods, the proposed method can make full and reasonable use of the spatial and temporal data of traffic flow to further increase the auxiliary traffic flow data and use the ADMM algorithm for optimization solution. The results showed that the proposed algorithm outperformed the other three comparison methods; that is, it has a better repair effect under three data deletion modes and nine data deletion rates. It can also achieve the best repair accuracy under high data deletion rates. The acquisition of complete traffic flow data realized by the proposed method will be highly beneficial in traffic management and control.

Since this paper only considered the coupling of a 3D tensor and a 2D matrix, future research will focus on the coupling of multiple matrices based on the multidimensional tensor. While doing that, we will also consider the total time cost of the running model.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (no. 51638004).

References

D. L. Gold, S. M. Turner, and B. J. Gajewski, Imputing Missing Values in its Data Archives for Intervals under 5 Minutes, Transportation Research Board 80th Annual Meeting, Washington, DC, USA, 2001.
J. H. Conklin and B. L. Smith, “Use of local lane distribution patterns to estimate missing data values from traffic monitoring systems,” Transportation Research Board, vol. 1811, no. 1, pp. 50–56, 2002.
View at: Google Scholar
B. L. Smith, W. T. Scherer, and J. H. Conklin, “Exploring imputation techniques for missing data in transportation management systems,” Transportation Research Record: Journal of the Transportation Research Board, vol. 1836, no. 1, pp. 132–142, 2003.
View at: Publisher Site | Google Scholar
G. Y. Jiang and L. H. Gang, “Dynamic traffic data fault diagnosis and repair method,” Journal of Transportation Engineering, vol. 4, no. 1, pp. 121–125, 2004.
View at: Google Scholar
Y. W. Jin and J. C. Wang, “Research on traffic flow parameter correction method based on correlation analysis,” Control Engineering of China, vol. 15, no. 0, pp. 142–144+147, 2008.
View at: Google Scholar
I. A. Gheyas and L. S. Smith, “A neural network-based framework for the reconstruction of incomplete data sets,” Neurocomputing, vol. 73, no. 6, pp. 3539–3560, 2010.
View at: Publisher Site | Google Scholar
W. Wang, Z. Y. Cheng, M. Y. Liu et al., “Traffic flow fault data repair method based on temporal and spatial correlation,” Journal of Zhejiang University (Engineering Science), vol. 51, no. 9, pp. 1727–1737, 2017.
View at: Google Scholar
Z. Chen, L. Zhang, G. Tian, and E. A. Nasr, “Economic maintenance planning of complex systems based on discrete artificial bee colony algorithm,” IEEE Access, vol. 8, no. 8, pp. 108062–108071, 2020.
View at: Publisher Site | Google Scholar
M. Kurucz, A. A. Benczur, and K. Csalogany, “Methods for large scale SVD with missing values,” Proceedings of KDD Cup and Workshop, pp. 31–38, 2007.
View at: Google Scholar
L. Q. Qu, L. Y. Li, J. Zhang, and J. Hu, “PPCA-Based missing data imputation for traffic flow volume: a systematical approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 10, no. 3, pp. 512–522, 2009.
View at: Google Scholar
J. Tang, G. Zhang, Y. Wang, H. Wang, and F. Liu, “A hybrid approach to integrate fuzzy C-means based imputation method with genetic algorithm for missing traffic volume data estimation,” Transportation Research Part C: Emerging Technologies, vol. 51, pp. 29–40, 2015.
View at: Publisher Site | Google Scholar
B. C. Lu, G. L. Guo, W. Q. Xiao et al., “Fault diagnosis and repair of dynamic traffic data based on multi-scale principal component analysis,” Natural Science, vol. 1, pp. 134–174, 2016.
View at: Google Scholar
Y. Q. Jiang, “An algorithm of imputing missing network data based on matrix low-rank decomposition,” Chang’an University, Xi’an, China, 2018, Dissertation.
View at: Google Scholar
J. Wang, W. Deng, and Y. Guo, “New Bayesian combination method for short-term traffic flow forecasting,” Transportation Research Part C: Emerging Technologies, vol. 43, no. 1, pp. 79–94, 2014.
View at: Publisher Site | Google Scholar
E. Acar, T. G. Kolda, and D. M. Dunlavy, “All-at-once optimization for coupled matrix and tensor factorizations,” KDD Workshop on Mining and Learning with Graphs, Sandia National Laboratories, Livermore, CA, USA, 2011.
View at: Google Scholar
C. D. Silva and F. J. Herrmann, “Hierarchical tucker tensor optimization-application to tensor completion,” in Proceedings of the 10th International Conference on Sampling Theory and Application, pp. 384–387, Bremen, Germany, July 2013.
View at: Google Scholar
H. Tan, J. Feng, Z. Chen et al., “Low multilinear rank approximation of tensors and application in missing traffic data,” Advances in Mechanical Engineering, vol. 6, pp. 1–12, 2015.
View at: Publisher Site | Google Scholar
J. Liu, P. Musialski, P. Wonka, and J. Ye, “Tensor completion for estimating missing values in visual data,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 208–220, 2013.
View at: Publisher Site | Google Scholar
Y. Xiang, “Traffic data recovery based on tensor filling and tensor matrix correlation analysis,” Yunnan University, Kunming, China, 2019, M. S. thesis.
View at: Google Scholar
C. H. Zhang, “Research on data quality evaluation and repair method of urban expressway traffic flow,” China Beijing Jiaotong University, Beijing, China, 2019, M. S. thesis.
View at: Google Scholar
H. Zhang, P. Chen, J. Zheng et al., “Missing data detection and imputation for urban ANPR system using an iterative tensor decomposition approach,” Transportation Research Part C: Emerging Technologies, vol. 107, no. 3, pp. 337–355, 2019.
View at: Publisher Site | Google Scholar
B. Yang, “Research and application of machine learning methods based on tensor data,” China Agricultural University, Beijing, China, 2014, M. S. thesis.
View at: Google Scholar
X. Luo, H. Wu, H. Yuan, and M. Zhou, “Temporal pattern-aware QoS prediction via biased non-negative latent factorization of tensors,” IEEE Transactions on Cybernetics, vol. 50, no. 5, pp. 1798–1809, 2020.
View at: Publisher Site | Google Scholar
C. Leng, H. Zhang, G. Cai, I. Cheng, and A. Basu, “Graph regularized Lp smooth non-negative matrix factorization for data representation,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 2, pp. 584–595, 2019.
View at: Publisher Site | Google Scholar
Y. Zhang, P. Zhou, G. M. Cui et al., “Multi-model based PSO method for burden distribution matrix optimization with expected burden distribution output behaviors,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 6, pp. 1506–1512, 2019.
View at: Google Scholar
X. Chen, Z. Wei, Z. Li, J. Liang, Y. Cai, and B. Zhang, “Ensemble correlation-based low-rank matrix completion with applications to traffic data imputation,” Knowledge-Based Systems, vol. 132, no. 11, pp. 249–262, 2017.
View at: Publisher Site | Google Scholar
L. Li, J. Zhang, Y. Wang, and B. Ran, “Missing value imputation for traffic-related time series data based on a multi-view learning method,” IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 8, pp. 2933–2943, 2019.
View at: Publisher Site | Google Scholar
Y. R. Huang, Z. Xiao, D. Wang, H. B. Jiang, and D. Wu, “Exploring individual travel patterns across private car trajectory data,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 12, pp. 168394–168404, 2020.
View at: Publisher Site | Google Scholar
M. Tang, Z. Li, G. Tian et al., “A Data-Driven-Based wavelet support vector approach for passenger flow forecasting of the metropolitan hub,” IEEE Access, vol. 7, no. 2, pp. 7176–7183, 2019.
View at: Publisher Site | Google Scholar
L. Chao, B. W. Du, Y. G. Wang et al., “Estimation of missing values in heterogeneous traffic data: application of multimodal deep learning model,” Knowledge-Based Systems, vol. 194, no. 22, 2020.
View at: Google Scholar
H. Jiang and H. Deng, “Traffic incident detection method based on factor Analysis and weighted random forest,” IEEE Access, vol. 8, pp. 168394–168404, 2020.
View at: Publisher Site | Google Scholar
J. J. Tang, X. S. Zhang, W. Q. Yin et al., “Missing data imputation for traffic flow based on combination of fuzzy neural network and rough set theory,” Journal of Intelligent Transportation Systems, 2020.
View at: Publisher Site | Google Scholar
X. Y. Chen, J. M. Yang, L. J. Sun et al., “A nonconvex low-rank tensor completion model for spatiotemporal traffic data imputation,” Transportation Research Part C: Emerging Technologies, vol. 117, 2020.
View at: Publisher Site | Google Scholar
M. Safaeian, A. M. Fathollahi-Fard, G. Tian, Z. Li, and H. Ke, “A multi-objective supplier selection and order allocation through incremental discount in a fuzzy environment,” Journal of Intelligent & Fuzzy Systems, vol. 37, no. 1, pp. 1435–1455, 2019.
View at: Publisher Site | Google Scholar
S. Gao, M. Zhou, Y. Wang, J. Cheng, H. Yachi, and J. Wang, “Dendritic neuron model with effective learning algorithms for classification, approximation, and prediction,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 2, pp. 601–614, 2019.
View at: Publisher Site | Google Scholar
Z. Shi and Z. Wang, “Optimal control for a class of complex singular system based on adaptive dynamic programming,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 1, pp. 188–197, 2019.
View at: Publisher Site | Google Scholar
W. J. Wang, G. D. Tian, M. N. Chen et al., “Dual-objective program and improved artificial bee colony for the optimization of energy-conscious milling parameters subject to multiple constraints,” Journal of Cleaner Production, vol. 245, no. 1, pp. 124–135, 2020.
View at: Publisher Site | Google Scholar
X. Luo, M. C. Zhou, Y. Xia et al., “Generating highly accurate predictions for missing QoS-data via aggregating non-negative latent factor models,” IEEE Transactions on Neural Networks and Learning Systems, vol. 27, no. 3, pp. 579–592, 2016.
View at: Publisher Site | Google Scholar
Z. M. Lu, L. Q. Wang, Z. Y. Han et al., “Surrogate-assisted particle swarm optimization algorithm with Pareto active learning for expensive multi-objective optimization,” IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 3, pp. 838–849, 2019.
View at: Google Scholar
X. Luo, M. Zhou, Z. Wang, Y. Xia, and Q. Zhu, “An effective scheme for QoS estimation via alternating direction method-based matrix factorization,” IEEE Transactions on Services Computing, vol. 12, no. 4, pp. 503–518, 2019.
View at: Publisher Site | Google Scholar
B. Stephen, P. Neal, C. Chu et al., “Distributed optimization and statistical learning via the alternating irection method of multipliers,” Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1–122, 2010.
View at: Google Scholar

Copyright

Copyright © 2021 Hui Jiang and Hongxing Deng. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

280

Downloads

694

Citations