Robust Missing Traffic Flow Imputation Considering Nonnegativity and Road Capacity

Abstract

There are increasing concerns about missing traffic data in recent years. In this paper, a robust missing traffic flow data imputation approach based on matrix completion is proposed. In the proposed method, the similarity of traffic flow from day to day is exploited to impute missing data by the low-rank hypothesis of constructed traffic flow matrix. And the physical limitation of road capacity and nonnegativity is also considered through the optimization process, which avoids the possibility of producing negative and overcapacity values. Moreover, the proposed algorithm can impute missing data and recover outlier in a unify framework. The experiment results show that the proposed method is more accurate, stable, and reasonable.

1. Introduction

Traffic information collected by various kinds of sensors are a vital component of intelligent transportation system (ITS) which aims to influence travel behavior, reduce traffic congestion, improve mobility, and enhance air quality [1]. For example, the real-time traffic information can be provided to drivers before and during their travels for supporting their decision of route choice [2] and it is also an important guideline for modern traffic control system to adjust the signal timing [3]. Moreover, after proper preprocessing, the real-time traffic data can be used as the real-time traffic state estimation of transportation networks [4]. On the other hand, several data mining techniques have been applied to mine time related association rules from historical traffic databases and its results have been used for traffic prediction such as the works of Qiao et al. [5] and Zargari et al. [6].

However, the missing traffic data problems remain inevitable due to detector faults or transmission distortion in many places. About 10% of daily traffic flow is usually missing in Beijing, China [7]. Turner et al. [8] reported that almost a quarter of data from San Antonio, Texas, is missing; and more than 5% of data are lost within the PeMS traffic flow database [9]. The missing data adversely affect the applications of intelligent transportation system; for example, the traffic control system requires sufficient traffic flow data (i.e., traffic volumes, occupancy rates, and flow speeds) to generate appropriate traffic management strategies [10, 11]. In traffic forecast area, if there exists missing data, the predicting performance will reduce sharply [12, 13]. Clearly, missing data problem is a large obstacle for any of the functions for which ITS data is to be used.

In the past decades, numerous imputation methods have been proposed to handle missing traffic data problem. These imputation methods can be roughly divided into two parts: interpolation based and inductive learning based methods.

Interpolation based methods always fill the missing data with a weighted average value calculated from part of known data. Yin et al. [14] use historical averages from the same detector at the same time period but in neighboring days to replace missing data. Zhong et al. [15] interpolate error traffic data by traffic data from similar daily flow variation patterns considering the error type and traffic condition. Such approaches only use part of the traffic information and always fail to accurately estimate missing value at high missing ratio.

On the other hand, inductive learning methods try to build imputation modeling from the a priori characteristics of traffic data. Most of these methods are based on some assumptions of traffic flow data. Autoregressive Integrated Moving Average (ARIMA) is based on the assumption that the historical value and future value of traffic flow provide an indication of the missing value [16]. The probabilistic principle component analysis (PPCA) based methods assume that the basic characteristic of traffic flow variations can be captured by the probability distribution of PPCA [7, 17]. The recent tensor-based methods assume that traffic data are highly correlated in multimode (day, week, link) and construct traffic flow data into multiway array (tensor) to capture these correlations. By utilizing the essential characteristics of traffic flow information by the assumptions, these kinds of imputation methods often outperform traditional interpolation methods [7]. It can be concluded that most of the inductive missing traffic data imputation methods make the following assumptions.

Assumption 1. Traffic flow has a high similarity from day to day and week to week but also link to the neighboring link which can be utilized to impute missing traffic flow.

Assumption 2. The traffic flow data have not been spoiled by the outliers, which frequently occur in real-world traffic information system.

While the inductive based imputation methods achieve somewhat success by the assumptions for traffic flow data, there are still some shortcomings in these methods. Firstly, according to the traffic flow theory, the volume of traffic flow is a certain value from zero to the road-capacity (the maximum traffic flow obtainable on a given roadway using all available lanes). But most data-driven imputation methods ignore this limitation of traffic volume. Secondly, the traditional methods such as the PCA-based methods cannot work well with big outlier or errors without the preprocessing of corrupted traffic data [7]. In fact, we have proposed several tensor completion methods [17–20], which make full use of multimode correlations [21] of traffic flow data, to impute missing traffic data. However, in our former works, the nonnegativity (lower bound) and capacity (upper bound) of traffic flow are still ignored. As a result, it is possible that these methods will produce some unreliable results.

To tackle these shortcomings, this paper proposes a traffic flow data imputation methods based on matrix completion. In the proposed method, the traffic data are constructed into a day×interval matrix. The similarity of day mode is captured by the assumption that the constructed matrix is low-rank. By adding a limitation in the objective function, the proposed method can restrict the reconstructed traffic flow data between zero and road capacity. Moreover, for the traffic flow data corrupted by outliers, the proposed method can simultaneously impute the missing data and recover the outliers by the sparse assumption. It should be noted that the method we proposed here can be considered as an idealized version of robust PCA (RPCA) but different from the natural approaches to robust PCA [22]. The proposed approach needs not to preprocess the data and isolate the outlier before the imputation. This advantage allows the proposed matrix completion (MC) methods outperform traditional imputation methods especially for the traffic flow corrupted by outliers.

To give a detailed explanation of the proposed method, the rest of this paper is organized as follows. The methodology and algorithm of the proposed method are proposed in Section 2. Section 3 presents the imputing testing results including comparison with other methods. The conclusion and future works are conducted in Section 4.

2. Methodology

In this section, a brief description of the matrix completion is presented. Then, we described the proposed missing traffic flow data imputation method in detail.

2.1. Review of Matrix Completion Methods

Let 𝑀 be an 𝑚×𝑛 matrix of rank 𝑟 (<𝑚 or 𝑛); the low matrix 𝑀 has some available sampled entries {𝑀_𝑖𝑗∶(𝑖,𝑗)∈Ω} where Ω is a subset of sampled cardinality. Then [17] proves that most matrices 𝑀 of rank 𝑟 can be perfectly reconstructed by solving the optimization problem: min‖𝑋‖_∗s.t𝑋_𝑖𝑗=𝑀_𝑖𝑗𝑖,𝑗∈Ω. (1)

In (1), the functional ‖𝑋‖_∗ is the nuclear norm of the matrix 𝑀, which is the sum of its singular values.

In [23], the singular value thresholding (SVT) algorithm is used to solve an approximate optimization problem of (1): min𝜏‖𝑋‖_∗+12‖𝑋‖²_𝐹s.t𝑃_Ω(𝑋)=𝑃_Ω(𝑀), (2) where 𝑃_Ω is the orthogonal projector onto the span of matrices vanishing outside of Ω so that the (𝑖,𝑗)th component of 𝑃_Ω(𝑋) is equal to 𝑋_𝑖𝑗 if (𝑖,𝑗)∈Ω and zero otherwise.

The Lagrange multiplier of (2) is 𝐿(𝑋,𝑌)=𝜏‖𝑋‖_∗+12‖𝑋‖²_𝐹+𝑌,𝑃_Ω𝑀−𝑋^𝑘, (3) where {𝐴,𝐵}=trace{𝐴𝐵^𝑇}, with optimization variable 𝑋∈𝑅^𝑚×𝑛. Fix 𝜏>0 and a sequence {𝛿_𝑘}_𝑘≥1 of scalar step sizes. Then starting with 𝑌⁰=0∈𝑅^𝑚×𝑛, the algorithm inductively defines 𝑋^𝑘=𝐷_𝜏𝑌^𝑘−1,𝑌^𝑘=𝑌^𝑘−1+𝛿_𝑘𝑃_Ω𝑀−𝑋^𝑘, (4) where 𝐷_𝜏(𝑋)=𝑈𝑆_𝜏(Σ)𝑉^𝑇if𝑋=𝑈Σ𝑉^𝑇. (5)

𝑆_𝜏(𝑥) is the constriction factor of 𝑥, where S_𝜏(𝑥)=𝑥−𝜏if𝑥>𝜏𝑥+𝜏if𝑥<−𝜏0else. (6) More details for SVT can be found in [24].

Literature [25] develops an augmented Lagrange multiplier method to solve MC. In their methods, the MC problem is formulated as min‖𝐴‖_∗s.t𝐴+𝐸=𝐷𝑃_Ω(𝐸)=0 (7) as 𝐸 will compensate for the unknown entries of 𝐷 and the unknown entries of 𝐷 are simply set as zeros. Then the partial augmented Lagrangian function is 𝐿𝐴,𝐸,𝑌,𝜇=‖𝐴‖_∗+{}+𝜇𝑌,𝐷−𝐴−𝐸2‖𝐷−𝐴−𝐸‖²_𝐹. (8)

Then, 𝐴 and 𝐸 are updated according to the subproblems of (8),where 𝐴 is updated by argmin_𝐴‖𝐴‖_∗+𝜇2𝐷−𝐴−𝐸+𝜇⁻¹𝑌²_𝐹=𝐷_𝜇−1𝐷−𝐸+𝜇⁻¹𝑌. (9)𝐸 is updated by argmin_{𝑃Ω(𝐸)=0}𝐷−𝐴−𝐸+𝜇⁻¹𝑌²_𝐹=𝑃_Ω𝐷−𝐴+𝜇⁻¹𝑌. (10)

More detailed information of the augmented Lagrange multiplier (ALM) method can be found in literature [25].

2.2. The Proposed Algorithm

The goal of the proposed method is to impute the missed traffic data considering both the physical limitation of traffic flow data and the possible corruption by outliers. Firstly, the traffic flow data in a local place are formed into the matrix mode as follows: 𝑎𝐴=¹¹…𝑎_1𝑛𝑎⋮⋱⋮_𝑚1…𝑎_𝑚𝑛. (11) In this matrix 𝑎_𝑖𝑗 represents the discretized volume on day 𝑗 at time interval 𝑖 within the given day. For the physical limitation, 𝑎_𝑖𝑗 changes in a particular range from zero to road capacity. The total number of days is 𝑛 and each day is divided into 𝑚 time intervals. Supposing the set of observed traffic volume data is Ω, and the traffic volume is corrupted by sparse outliers. Hence, the missing traffic data imputation problem is translated into a corrupted matrix completion problem [26–28]; the optimization problems can be described as minrank(𝐀)+𝜆𝑃_Ω(𝐄)₀s.t𝐀+𝐄=𝐃,0≤𝐀≤𝐂, (12) where ‖‖₀ represents the number of the nonzero entries and 𝑃_Ω is the orthogonal projector onto the span of matrices vanishing outside of Ω.

The minimums of ‖‖₀ (ℓ⁰-norm) and the rank of matrix are NP-hard problem [29]. To convert the objective function (12) into a convex optimization problem, the rank of 𝐴 is approximated by the nuclear norm (the sum of the singular values) of the matrix, and the ‖‖₀ of 𝑃_Ω(𝐸) is approximated by the ℓ¹-norm of the matrix (the sum of the absolute value of its entries) [29] as follows: min‖𝐀‖_∗+𝜆𝑃_Ω(𝐄)₁s.t𝐀+𝐄=𝐃,0≤𝐀≤𝐂. (13)

Because the solution of 𝐸 is easier than 𝐴, the function is converted into the following form: min‖𝐀‖_∗+𝜆𝑃_Ω(𝐄)₁s.t𝐀+𝐄=𝐃,𝐃−𝐂≤𝐄≤𝐃. (14) Considering the faster computation speed and higher accuracy, the augmented Lagrangian method (ALM) [25] is employed to optimize the problem.

By introducing a Lagrange multiplier 𝑌 to remove the equality constraint, one has the Lagrangian function of (14): 𝐿𝐴,𝐸,𝑌,𝜇=‖𝐴‖_∗+𝜆𝑃_Ω(𝐸)₁+⟨⟩+𝜇𝑌,𝐷−𝐴−𝐸2‖𝐷−𝐴−𝐸‖²_𝐹. (15)

Lin et al. [25] proved that updating 𝐴 and 𝐸 once when solving this subproblem is sufficient for 𝐴 and 𝐸 to converge to the optimal solution of (15). 𝐴 is updated by argmin_𝐴‖𝐴‖_∗+𝜇2𝐷−𝐴−𝐸+𝜇⁻¹𝑌²_𝐹=𝐷_𝜇−1𝐷−𝐸+𝜇⁻¹𝑌, (16) where 𝐷_𝜇−1𝐷−𝐸+𝜇⁻¹𝑌=𝑈𝑆_𝜇−1(Σ)𝑉^𝑇if𝐷−𝐸+𝜇⁻¹𝑌=𝑈Σ𝑉^𝑇. (17)𝑆_𝜇−1() is the constriction factor, where 𝑆_𝜏(𝑥)=𝑥−𝜏if𝑥>𝜏𝑥+𝜏if𝑥<−𝜏0else. (18)𝐸 is updated by argmin_{𝐷−𝐶≤𝐸≤𝐷}𝜆𝑃_Ω(𝐸)₁+𝜇2𝐷−𝐴−𝐸+𝜇⁻¹𝑌²_𝐹=𝑃_{𝐷−𝑐≤𝐸≤𝐷}𝑃_Ω𝑆𝜇𝜆𝐷−𝐴+𝜇⁻¹𝑌+𝑃_Ω𝐷−𝐴+𝜇⁻¹𝑌. (19)𝑃_{𝐷−𝑐≤𝐸≤𝐷} is the projector onto the span of matrices ranging from 𝐷−𝐶 to 𝐷. This leads to a matrix completion based traffic data imputation method (MCI) described in Algorithm 1.

Input: Observation samples 𝐷𝑖𝑗, 𝑖,𝑗∈Ω, of matrix 𝐷∈𝑅𝑚×𝑛 (1) 𝑌0,𝐸0=0,𝜇0,𝑘=0,𝐶(roadcapacity) (2) while not converged do (3) (𝑈,𝑆,𝑉)=𝑠𝑣𝑑𝐷−𝐸𝑘+𝜇𝑘−1𝑌𝑘, (4) 𝐴𝑘+1=𝑈𝑆𝜇𝑘−1𝑉𝑇, (5) 𝐸𝑘+1=𝑃𝐷−𝑐≤𝐸≤𝐷𝑃Ω𝑆𝜇/𝜆𝐷−𝐴𝑘+1+𝜇𝑘−1𝑌𝑘+𝑃Ω𝐷−𝐴𝑘+1+𝜇𝑘−1𝑌𝑘 (6) 𝑌𝑘+1=𝑌𝑘+𝜇𝑘𝐷−𝐴𝑘+1−𝐸𝑘+1 (7) updating 𝜇𝑘 (8) 𝑘=𝑘+1 (9) end while (10) output 𝐴

Algorithm 1 

Matrix completion based traffic data imputation method.

3. Experiments

The evaluation of an imputation method’s performance is a multiobjective problem. In this section, four key performance indicators of the proposed method are discussed, which are the accuracy, stability, robustness, and computation complexity.

3.1. The Test Data

To evaluate the proposed method, traffic flow datasets from PeMS [9] open database are used. The dataset is collected from Detector 400141. The detector is located at north bound freeway I880. The freeway has four lanes under surveillance. The sampling period is between July 11, 2013 and July 30, 2013. The data are almost all observed with a 99.9% observed ratio.

The assumption is an important premise for the missing traffic data imputation methods based on inductive learning. For the proposed MCI, the traffic volume matrix is assumed to be low-rank. The correctness of this assumption is validated by the low-rank approximation of the original data. The low-rank approximation data is computed by singular value decomposition according to Eckart-Young theorem [30]. If we consider singular value decomposition (SVD) of the constructed traffic volume matrix 𝑀_𝐺, we get 𝐌_𝐆=𝐔_𝐆∇_𝐆𝐕^𝐓_𝐆, (20) where columns of 𝑈_𝐺 and 𝑉^𝑇_𝐺 are left-singular vectors and right-singular vectors of 𝑀_𝐺, respectively. The diagonal entries of ∇_𝐺 are equal to the singular values of 𝑀_𝐺.

The full-rank matrix 𝑀_𝐺 can be approximated as a low-rank matrix 𝑀_𝐺 by the SVD of 𝑀_𝐺, namely, 𝐌_𝐆=𝐔_𝐆∇_𝐆𝐕^𝐓_𝐆, (21) where ∇_𝐺 is the same matrix as ∇_𝐺 except that it contains only the 𝑟 largest singular values (the other singular values are replaced by zero). The low-rank approximation of the selected traffic data is given in Figure 1.

Figure 1 

The original data (blue) and approximated low-rank data (red) from Detector 400141. The low-rank data basically keeps the characteristic of origin data.

As we can see from Figure 1, the approximated low-rank data basically keeps the characteristic of origin data. From the results, it can be concluded that the low-rank hypothesis of traffic volume matrix is reasonable.

3.2. Quantitative Measures

The set of measures including MAE, MAPE, and SDE allows one to directly evaluate the performance of multiple imputation techniques

3.2.1. Accuracy

In this paper, the mean absolute percentage error (MAPE) is used to evaluate the performance of missing traffic data imputation. However, the MAPE will be lower if the traffic volumes are higher [31]. In observance of this phenomenon, this paper also applies the mean absolute error (MAE) as a complementary measure for MAPE.

The mean absolute error (MAE) is defined to be 1MAE=𝑛^𝑛_𝑡=1𝑥_𝑡−𝑀_𝑡. (22) The mean absolute percentage error (MAPE) is defined to be 1MAPE=𝑛^𝑛_𝑡=1𝑥_𝑡−𝑀_𝑡𝑥_𝑡×100%, (23) where 𝑛 is the total number of missing data, 𝑥_𝑡 is the observed value, and 𝑀_𝑡 is the reconstructed value.

3.2.2. Stability

The standard deviation of errors (SDE) of the test methods is evaluated. The smaller SDE means that the errors are tightly clustered around the mean value [29]: SDE=var𝑀_𝑡−𝑥_𝑡. (24)

3.2.3. Robustness

The robustness is evaluated by the accuracy on the dataset added outlier under different missing ratio.

3.3. The Results without Outlier

In this part, we evaluate the performance of MCI algorithm and compare it with other state-of-the-art algorithms including PCA-based PPCA [7], SVT [22], and IALM [23] on random missing case without outlier. For MCI, the tolerance on the ‖‖_𝑓of 𝐷−𝐴−𝐸 divided by ‖‖_𝑓 of 𝐷 in the gradient is set to 0.01, and the maximum number of iterations is set to 10³𝜆 which is set to 150. For SVT and IALM, the tolerance on the ‖‖_𝑓 is set to 0.01, and the maximum number of iterations is also set to 10³. For PPCA, similar to [7], the tolerance is set to 0.01, and the latent space is set to 15.

In order to better verify the change in imputation performance, the total missing ratio (the number of missing data points divided by the total number of data points) is set from 5% to 70%. The MAE and MAPE curves are shown in Figure 2.

(a)

(b)

Figure 2 

MAE and MAPE curves for three matrix completion methods and PPCA method.

In Figure 2, all the methods achieved equal results under missing ratio lower than 30%. However, the performances of other methods except for MCI degrade sharply when the missing ratio is higher than 50% except MCI. The reason may be that the MCI can utilize the physical limitation of traffic flow in the imputation process while the other methods ignored the physical limitation of traffic flow data.

Traffic volume must be nonnegative and less than the value of road capacity. The PPCA imputation strategy is a kind of statistical method which imputes data through the a priori statistical characteristics of data. As shown in Figure 3, the PPCA method may give a negative value of volume during the low flow rate interval. The phenomenon also can be found in the other two matrix completion strategies without the constraints of nonnegativity and road capacity in their objective function. Our proposed MCI can tackle this shortcoming by adding the limits to the algorithm. The negative and overcapacity value is not observed in the experiments of MCI. The possibilities of four methods that produce unreliable results in our experiments are given in Table 1 (the frequency of experiment results with unreliable results: negative or overcapacity).

Table 1 

The possibilities of producing unreliable results.

Figure 3 

The negative value reconstructed by PPCA.

In the above experiments, the accuracy of MCI has been tested. Then, we will test the stability of imputation methods by SDE under different missing ratio. As shown in Figure 4, the SDEs of MCI and IALM using augmented Lagrangian function are lower than PPCA and SVT. It suggests that the MCI not only can compute missing data more accurate but also more stable by employing the augmented Lagrangian function.

Figure 4 

The standard deviation of errors (SDE) of the test methods.

3.4. Missing Data Imputation with Outlier

The above experiments assume that the data have not been spoiled by outliers. However, the traffic flow series are often corrupted by the outliers which are caused by numerous reasons [32]. Unfortunately, these outliers are usually not easy to be isolated by the traditional missing traffic data imputation approaches. Thus, the recovery of outlier and imputation of missing data are often completed in different frameworks separately [7, 29].

For the problem, the MCI algorithm makes it possible to impute missing data and recover outlier in a unify framework by adding the sparse matrix 𝐸.

There are various kinds of outlier in traffic data. Here, we only consider two common scenarios of outliers:(a)volume out of range (VOR): percentage of the detector records with volumes larger than 1000 v/5 min;(b)volume repeating zero (VRZ): percentage of the detector records with repeating zero volumes for 30 min.

It is hard to enumerate all the situations with different mixing ratios of the two outliers’ scenarios. In the experiments, the methods are tested on a typical situation by assuming that the mixed VOR and VRZ data have a ratio of 1 : 1. Ratios of outlier data are set from 5% to 15% and the outlier data are produced randomly. The missing data ratio is set to 30%. All the results are averaged by 10 instances. The MAE and MAPE for missing data and outlier recovery are both given in Table 2. Figure 5 presents the part of traffic volume data and reconstructed volume data. The results show that MCI could impute the missing data and recover the traffic volume outlier data with a reliable performance.

Table 2 

The recovery and imputation accuracy with different ratio of outliers.

Figure 5 

Comparisons with raw traffic volume data, data with 30% missing data and corrupted by outliers with 15% ratio, and data reconstructed by MCI.

3.5. Computation Complexity of MCI Approach

As the same as IALM [25], it is not necessary to compute the full SVD in MCI. By using Lansvd [21], a fast SVD method that only computes singular values larger than a particular threshold and their corresponding singular vectors, the complexity of the singular value decomposition is not a problem for MCI. And the computation speed of MCI is faster than traditional matrix completion based methods such as SVT by utilizing the augmented Lagrangian function [25].

It is not easy to choose the parameter 𝜆 which is the weight parameter between the rank of matrix and the number of sparse outlier. For the traffic data without outliers, setting 𝜆 larger than 100 can obtain a good performance. But for data corrupted by outliers, a proper lower value of 𝜆 will achieve better results. In this paper, we suggest 𝜆=150 for real application for data without corruption of outlier and 𝜆=0.05 for corrupted data.

4. Conclusion and Future Works

In this paper, a matrix completion method which fully utilizes the physical limitation of traffic volume and the day mode similarity has been proposed dealing with missing traffic flow problem. The experiment shows that the proposed method is more reasonable, accurate, and stable than the state-of-art methods for traffic flow data. Moreover, the proposed MCI can impute missing data and recover the outlier in a unify framework with a reliable performance.

Future research should look into missing traffic data imputation method that incorporates spatial and temporal correlations among adjacent detectors to improve imputation accuracy. In addition, future studies may evaluate the performance of MCI on other parameters such as speed and occupancy. It still needs more researches on the appropriate choice of parameter for the MCI.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

The research was supported by NSFC (Grant nos. 61271376, 51308115, and 91120010), National Basic Research Program of China (973 Program no. 2012CB725405), and Beijing Natural Science Foundation (4122067).