Abstract
Freeway crashes occupied 30% of total crashes. The advanced driver-assistance system (ADAS) often underestimates or omits the necessary collision warning. To investigate the forward collision in complex traffic conditions, the TTC (time-to-collision) is regarded as a surrogate for collision risk assessment. The study aims to design a forward-collision warning method for ADAS in the urban freeway scenario. The testing vehicle is equipped with sensors and a satellite navigation system. The TTC was collected from the Xi’an Rao Cheng expressway for the car-following scenario for three days. A comprehensive Gaussian model, which consists of three sub-GMM models is applied to describe the TTC distribution. The vehicle trajectories were extracted in the car-following state. To improve the efficiency of the forwarding collision system, four seconds are chosen as an analysis window in the car-following state. Two time-series machine models were used to predict TTC in advance. The TTC prediction models were constructed on the basis of the long short-term memory (LSTM). The prediction result was compared with the deep belief network (DBN) model. The comparison results show that the LSTM model is faster than the DBN model. The number of iterations is 100. The loss function of the LSTM is 0.06. The LSTM outperforms the DBN model and is suitable for car-following warnings.
1. Introduction
The ADAS (advanced driver-assistance system) has improved rapidly in the recent several years. The overarching role of ADAS is to assist drivers while driving and warn drivers in the car-following and lane-changing states. Driver error is responsible for nearly 90% of all traffic collisions (NHTSA, 2015); thus, the ADAS is employed to warn drivers of any potential conflict [1]. It is essential to avoid forward-collision accidents to assist the drivers in safe driving status. Until now, there are two types of ADAS: sensor-based systems and connected-vehicle-based systems. A large number of field tests, simulation experiments, and data analysis indicate that ADAS systems can effectively prevent vehicle collision, especially in forwarding collision warnings [2, 3]. In the sensor-based systems, the floating car equipped with a satellite navigation system and various sensors can obtain the surrounding information about traffic flow at any point on the road [4]. The floating vehicle can collect real-time vehicle information such as speed, longitude, and latitude. The relationship between vehicles can be analyzed. The vehicle can share real-time information with roadside infrastructures in the connected-vehicle-based systems.
The ADAS consists of perception, recognition, decision, and reaction process [5]. At present, many subsystems of ADAS are used in vehicles, including lane keeping assistance, driver drowsiness detection, and collision avoidance system [6]. The forward-collision warning (FCW) proved an efficient ADAS in the car-following state. There are two types of existing collision warning algorithms: the Safety Time Algorithm and the Safety Distance Algorithm [7].
The Safety Time Algorithm determines whether the state is safe by comparing collision time with the limit value of the current state. And, the Safety Distance Algorithm avoids collision by calculating the minimum distance between vehicle and barrier under current driving conditions [8]. Time-to-collision (TTC) is widely used as a surrogate indicator to quantify traffic conflicts because it can be directly and easily obtained from the traffic trajectories video. A TTC threshold is usually chosen to distinguish safe situations from dangerous scenarios exposed to traffic conflicts. The probability of a traffic collision is calculated based on TTC. The Monte Carlo simulation and driver experiment show that 3.6 s and 5.6 s are the thresholds. The 15th percentile is always used as the minimum TTC threshold to be the collision threshold.
DENG designed a monocular vision-based collision avoidance warning system and used the time-to-collision (TTC) algorithm to judge the vehicle's state [9]. Han proposed an AEB strategy that considers the influence of different road friction on the TTC braking threshold, which can adapt to different road surfaces and make the function of the AEB system more reliable [10]. Kilicarslan proposes a method to calculate time-to-collision (TTC) on-the-fly based on motion information captured by in-vehicle cameras to detect hazardous events and degrees from motion divergence in driving videos [11]. Davis proposes NH-TTC, a general method for autonomous robots with arbitrary equations to avoid collision predictably [12]. For the overtaking behavior of intervehicle communication technology, Chen et al. divided the overtaking process into the approach and succeeding stages. They propose a linear estimation method of TTC and collision probability in the approach phase. This estimation method is used as a quantitative index to evaluate the safety of overtaking behavior in the succeeding stage [13]. Li et al. proposed a special transition process based on the TTC index to change the vehicle from a safe situation to a dangerous situation to evaluate the risk of rear-end collision [14]. Cosic et al. proposed an ADAS algorithm that enables high-accuracy collision warning predictions. The ADAS algorithm only uses video frames captured by cameras in the exterior rear-view mirrors to estimate time-to-potential collision (TTC) with vehicles from behind [15]. Pyo et al. propose a vehicle detection-based FCW system for highway environments with a CNN as the classifier and time-to-collision (TTC) as the collision warning index [16]. Salari puts forward a novel camera-based forward-collision warning (FCW) system via TCC as a threshold switch for the collision warning system [17]. Combined with the sliding tire model, the road friction peak is estimated. The braking force threshold corresponding to the TTC is calculated according to the friction peak so that the collision avoidance system has better adaptability to different road surfaces [10]. The TTC upper and lower critical points of its membership function are set to 2.5 s and 0.5 s [18]. Lian et al., Wu et al., and Yang et al. use the safety distance as the warning threshold. The influence of road and driver factors is taken into account by altering the distance threshold while the algorithm is operating [19–21]. Guo et al. proposed an AEB control strategy based on a BP neural network to predict the TTC of collision time was proposed. The prediction model was obtained by using the BP neural network algorithm, and then the threshold of collision departure warning time and emergency braking time were calculated [22].
Zong et al. designed an integrated model that couples a Hidden Markov chain model and an artificial neural network to identify the driving purpose and forecast the driver’s maneuvering behavior [23, 24]. Nakaoka et al. proposed a unique FCW algorithm considering the distance, the vehicle speed, and other parameters. Especially the algorithm takes into account both safe braking distance and TTC [25]. However, previous studies indicated that the TTC is difficult to build mathematical problems. Markov decision process is an effective method to solve this problem Ting applied HMM to decision-making.
Yuan proposed a forward vehicle collision warning framework by aggregating monocular distance measurement and precise vehicle detection. A multiscale detection algorithm that regards the result of calibration as distance prior was proposed to improve the precision. Abnormal driver behaviors are introduced to make FCW adaptive. The model is too simple to predict vehicle crashes effectively [26]. Lee proposed a collision warning system based on an individual driver’s driving behavior using an artificial neural network learning algorithm so that the collision risk could be determined according to the driving characteristics of the driver. It was experimentally verified that the proposed driver behavior model for the driver’s driving behaviors, created using the neural network, accurately reflected the driver’s driving behaviors. However, the performance of the driving behavior-based CWS was only evaluated at the laboratory scale using experimental results from actual vehicles. Therefore, its practicality and effectiveness need to be verified [27].
Deep learning methods are used to predict potential traffic conflicts using data from vehicles and satellite navigation systems combined with floating sensors. Lee and Yeo proposed a real-time rear-end collision warning algorithm employing an MLPNN [28, 29]. Until now, it is exceptional to use other machine learning methods such as Nave Bayes, Random Forest, and so on in collision warning algorithms. Consequently, the performance of these models needs to be compared [30, 31].
The previous research illustrates that the TTC distribution is essential for warning threshold determination. As we know, traffic condition has a dramatic impact on TTC distribution.
The forward-collision warning in advance is a TTC prediction problem. An efficient prediction model has not been addressed yet. The previous studies have ignored the time-series attributes of the vehicle. Therefore, the objectives of this study are to design a rational model using the probabilistic approaches for TTC distribution. A model of TTC prediction is also derived. And, this paper innovatively proposes to use the deep learning method to predict the collision occurrence indicator TTC.
This paper designed a forward-collision risk based on the Gaussian distribution for the Xi’an Rao Cheng expressway. The future motion uncertainty is modelled concerning lanes, therefore, the model can accurately forecast potential road traffic crashes in real-life driving scenarios [32]. Machine learning is used to predict the TTC for forwarding collision warnings at the longitudinal direction. The contributions of this paper are as follows.
The testing vehicle equipped with a satellite navigation system and sensors was collected from the Rao Cheng urban expressway in Xi’an.
TTC can be extracted and calculated from vehicle trajectories, and the distribution is estimated by three sub-Gaussian models.
The deep learning methods, including LSTM and DBN, were used to predict TTC in the next second, which can be used to measure the forwarding warning collision risks.
The paper consists of 6 sections, which are as follows. Section 1 is the Introduction. Section 2 is the process of data acquisition. Section 3 is the raw data processing and vehicle trajectory extraction. Section 4 is TTC calculation and presents the GMM modelling. Section 5 is about the time-series machine learning application in the TTC prediction. Section 6 is the Conclusion.
2. Data Source
2.1. Data Collection
To evaluate traffic safety on the Xi’an Rao Cheng freeway to explore the longitudinal direction safety, which is composed of a six-lane expressway and the lane width is 3.75 m. The Xi’an Rao Cheng expressway is located in Xi’an, with a total length of 82.3 kilometers. It is a fully controlled and interchange expressway. The speed limit is 100 km/h for passenger cars and 80 km/h for trucks. The vehicle is equipped with a satellite navigation system, inertial measurement unit, onboard-diagnosis (OBD) unit, a camera, and microwave radar, as shown in Figure 1(a). The 5-hour natural driving information includes speed, location, and distance between ego vehicles and surrounding vehicles. Each driving test was scheduled during the busiest period of the day. Each driving test was scheduled during the most active period from Oct. 2nd to Oct. 3th, 2020 when the weather was clear. The vehicle locations are obtained from the satellite navigation system at 1s intervals. Figure 1(b) obtained the microtraffic flow from the roadside unit. The lanes from the inner to the outer are marked 1 to 3.
(a)
(b)
The testing route is the red line in Figure 2.
In order to obtain the traffic flow of the testing route, traffic flow features at 10 sites were collected. Table 1 is the data summary of traffic flow. The microtraffic flow parameters were collected at different locations of the testing route during the investigation time.
2.2. Vehicle Trajectory Extraction
OBD in-vehicle units were used to obtain the individual speed, acceleration speed, and the distance between the ego vehicle and the front vehicle [33]. Generally, the vehicles in lanes 1 and 2 were in the car-following state. The speed, the distance, the longitude, and the latitude information were read from CAN cable at 1 Hz. And, the acceleration can be calculated from OBD data.
The lane-changing and car-following behavior units are two primary components. In the video, the lane-changing lasts for 4 seconds. Therefore, 4 seconds was chosen as the analysis window for the behavior trajectory extraction.
Observing the driving video and OBD dataset, to predict the vehicle speed in 4 seconds, each sample from Table 2 was extracted at a frequency of 1 Hz. One thousand and two hundred car-following trajectory samples were obtained and the satellite navigation system collected trajectories.
The vehicle position at time t − 1 is and the location displacement at time t is as follows:where and are the acceleration rate at the longitude and lateral direction. is the time step. The step is 1 s in this study.
2.3. Data Processing
Missing data or hardware problems may appear during the investigation time. Some errors or outliers can be observed. The outliers should be removed from datasets.
Speed values which are smaller than 0 km/h and higher than 130 km/h were considered as outliers.
In this study, the OBD unit outputs natural data including time, speed, mileage, and real-time position of vehicles. The data output frequency is 1 Hz, that is, the data collection interval is 1 s. Acceleration can be calculated from speed and time.
All of these sensors’ observations were gathered in their respective software, which was then pulled from the sensing system and saved in a database. The vehicle position was captured by the satellite navigation system. Two thousand and three hundred car-following was collected. The traffic features during investigation time are shown in Figure 3.
(a)
(b)
Figure 3 indicates that the highest volume is about 450 pcu/5 min, that is, 5400 pcu/h. The speed of the vehicle is around 80 km/h∼90 km/h.
Table 2 is the vehicle features extracted from the individual vehicle.
3. TTC Analysis
3.1. Time-to-Collision Calculation
Traffic conflict is one in which road participants are infinitely close in space and time. Collision behavior refers to a behavior process in which the motion states of two or more objects change significantly in a very short time when they move relative to each other. In the process of driving at high speed, due to various reasons such as the driver’s reaction is not timely, the change of the surrounding environment, and the abrupt change of road conditions, the vehicle’s motion state changes, which leads to the occurrence of collision behavior. The main function of the vehicle forward-collision warning system is to send an alarm to the driver when there is a potential conflict danger between the vehicle and the vehicle ahead.
The forward-collision warning for the individual vehicle is studied the TTC is considered a surrogate safety indicator for vehicle safety assessment for ADAS design.
The TTC in the car-following state can be calculated by the following equation: where and are the TTC and the position of the following vehicle at time t, respectively, denotes the following vehicle’s position at time t, and are the position and the length of the leading vehicle, denotes the following vehicle’s speed, m/s, and denotes the leading vehicle’s speed. The distance between the leading vehicle and following vehicle can be described by . The smaller TTC indicates an increased probability of crash occurrences.
3.2. TTC Distribution Based on the Gaussian Model
To design an appropriate ADAS, the Gaussian mixture model (GMM) was employed in the TTC estimation’s probability density function (PDF). It is imperative to understand the distribution of the TTCs in the safety assessment and danger determination. The complete estimated model is the sum of several sub-Gaussian models with different weights. The values are associated with the safety condition of traffic flow.
The subcomponent density function is a Gaussian function as shown in the following equation:
In this study, the number of sub-Gaussian models is consistent with the safety level. Therefore, ADAS safety conditions are divided into high, medium, and low levels.
A completed GMM for TTC is as follows:where the parameter is TTC time t, which can be obtained by choosing a mixture weight, which is between 0 and 1, and the PDF is the mean vector. is the variance vector.
The complete GMM can be estimated through the variances and the mixture weights from sub-Gaussian model densities. Each submodel can be collectively represented as .
The maximum likelihood (ML) estimation is used to estimate the parameters of a GMM. The ML function of the GMM can be written as follows:where the function of is the likelihood of parameters given the training data. The likelihood is a function of with a certain TTC value.
The estimation of parameters estimation is solved by two steps E-M. Table 3 shows the regression results.
The three Gaussian distributions and the sum of the TTC are in Figure 4.
According to the standard ISO 15623 Intelligent Transport Systems-Forward Vehicle Collision Warning Systems-Performance Requirements And Test Procedures and China national standard GB/T 33577–2017 Intelligent Transportation Systems Vehicle Forward-Collision Warning System Performance Requirements And Test Procedures, the warning threshold is 2.97 s in this study, which indicates that the vehicle is likely to occur when TTC is smaller than 2.97 s.
4. Collision Risk Classification Based on Deep Learning
In a car-following scenario, the study proposed a forwarding collision method based on a satellite navigation system. Then, the driving trajectories were extracted. The framework is implemented as deep learning, and I trained the network.
4.1. LSTM Learning Method
Long short-term memory (LSTM) is a particular form of recursive neural network (RNN). LSTM can learn, remember, and forget information during training. LSTM can alleviate the problem of RNN gradient disappearance and gradient explosion to some extent. It is usually used in event processing prediction with long discontinuities and delays in time. The structure of LSTM is shown in Figure 5.
LSTM networks looping mechanism and three logistic forgotten got gate , input gate , and output gate . The two basic units are cell state and hiding unit .
Forget gate adjusts the retention of information in the previous unit by controlling the self-circulating weight, that is, to “forget” previously useless information. According to the output result of the hidden layer at the last moment and the input at the current moment, the forget gate governs how much of the cell state is reserved to the cell state at the last instant [33].
The vehicle position at time t is denoted as and the time gap between the ego vehicle and the leading vehicle in the ego lane is denoted as TTC. The surrounding vehicle location information by the satellite navigation system.
The LSTM-based time-series data prediction model’s purpose is to generate a sequential output Y = from a set of observation input characteristics [34].
It states that the advanced warning time should be less than 4 seconds according to the standard in China. The behavior analysis window is 4 seconds. In this study, the last four seconds of vehicle information were used to predict the TTC in the next second.
4.2. Deep Belief Network (DBN)
In this study, the deep belief network (DBN) is used to estimate the TTC [35]. A combination of restricted Boltzmann machines (RBMs) was stacked with classification or regression layers, as shown in Figure 6. Therefore, the model converges to the best advantage of the adjacent optimal value to achieve the best model training. The data from RBM are used as DBN inputs, and then, the input data are converted to the hidden layer through RBM.
The DBN training process can be divided into unsupervised training and supervised training. RBM is trained by the contrastive divergence (CD) algorithm in layer-by-layer and RBM takes the output of the previous unit as the input of the next unit. Therefore, the initial parameters of the network are produced. Labeled data are not required for this unsupervised training approach. The BP algorithm fine-tunes the BP layer with labeled data to obtain the final parameters in supervised training.
RBM is a deformation of Boltzmann Machine (BM). In the RBM model, is the visible layer, . is the hidden layer, . a and b are the bias of the visible layer and hidden layer, respectively. is the bias from the ith unit of the visible layer to the hidden layer [36]. is the deviation from the jth unit of the hidden layer to the visible layer. is the weight matrix between the visible layer and the hidden layer. And, its element is the connection weight coefficient of the ith element in the visible layer and the jth element in the hidden layer.
The goal of the RBM model is to obtain the parameters , which can be used to fit the given training data. Given the training sample , after training RBM, the parameter is adjusted and the training samples are fitted. The probability distribution represented by the corresponding RBM is consistent with the empirical distribution of training data as much as possible. CRBM can be regarded as RBM with fixed additional input of the first n moment data of the observable layer. Thus, it increases the time association between the first n moment and the current moment.
5. Experiments
5.1. Data Validation
The framework of ADAS is shown in Figure 7.
The TTC distribution can be expressed as the sum of the three sub-Gaussian models. Table 4 shows the input features for the car-following state.
The first 1000 samples were selected at random as training samples, while the remaining 200 were used as test samples. The length of the trajectory analysis was 4 seconds.
The output variable is the vehicle TTC in the LSTM vehicle longitudinal time distance prediction model under the car-following condition. The LSTM parameters setting is shown in Table 5.
One thousand samples were used for training, and 200 samples were used for validation. The train samples are shown in Table 6.
The DNB was developed for prediction comparison. The parameters’ setting is shown in Tables 7 and 8.
The two types network of TTC prediction results and loss are shown in Figure 8. Two hundred samples were used to validate the time-series model.
(a)
(b)
(c)
(d)
5.2. Discussion
Two time-series machine learning methods for forwarding collision surrogate prediction were developed. The vehicle position data, leading vehicle speed, and following vehicle speed were considered as input variables. Four seconds is used as an analysis window in the car-following state. 2400 samples were used for training, and 600 samples were validated. The previous three seconds were predicted for the next second. Figure 8(a) is the DBN model prediction results and Figure 8(b) is the LSTM model result. The epoch of both methods is 100 iterations. When the TTC is smaller than 7 s, both models have a good performance. The prediction results are more significant than the real TTC. When TTC fluctuates dramatically between 50 and 100 seconds, both the prediction models are fair and good. The iteration of LSTM is 100 which is quicker than the iteration of DBN. The loss function of the LSTM is approximately 0.4 which is lower than the DBN. The trend of LSTM and DBN is almost the same. The performance of the two machine learning methods is similar. The LSTM is little better than the DBN. The safety level s determined according to the previous GMM model. When the TTC is lower than 3.7 seconds, drivers would be a warning.
The LSTM model can be integrated into the FCW or AEB (Autonomous Emergency Braking) system designed to enhance vehicle safety.
6. Conclusion
In this study, the individual vehicle equipped with sensors and a satellite navigation system was tested for the forward-collision warning. The testing route is the Xi’an Rao Cheng expressway. The speed, distance between the vehicle and the leading vehicle, and position information were collected at 1 HZ. 1200 Trajectories were extracted from the vehicle in the car-following state. A TTC model is designed for the ADAS forward-collision warning system. The complete TTC distribution of urban expressways consists of three sub-Gaussian models. Three seconds of TTC and speed were input variables for machine learning, and the output is the TTC in the next second, according to the standard ISO 15623 and GB/T 33577–2017. When the TTC is smaller than 2.97 s, there is a forward-collision warning for drivers.
The machine learning method for time-series time analysis. LSTM and DBN were chosen to model the TTC. Data in the part 3 seconds were used to predict the TTC in the next second. The iteration of both models is almost 100. The results can be compared with the forwarding warning threshold. The loss function of the LSTM is smaller than and DBN methods.
In this paper, we proposed an ADAS forward-collision warning system with machine learning to predict TTC values. This intelligent driving system informs the driver of the possible danger warning in advance. This study can provide a theoretical basis and threshold determination for collision warning systems. The outcomes of the paper can apply to the highway and urban roads as well.
Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this article.
Acknowledgments
This work was supported by the scientific research of the System Engineering Institute, Academy of Military Sciences, and the National Science Foundation (Grant no. 51878088).