Abstract
Traffic accident is an important problem in modern society. Vehicle collision prediction is one of the key technical points that must be broken through in the future driving system. However, due to the complexity of traffic environment and the difference of emergency ability of drivers, it is very difficult to predict vehicle collision. Although experts and scholars have tried to monitor and predict accidents in real time according to environmental conditions, overly agile warning or inaccurate prediction may cause serious consequences. Therefore, in order to more accurately predict the occurrence of vehicle collision, this paper analyses and models the driving mode of the vehicle based on transfer learning and using the previous performance data of the vehicle, so as to predict the future collision situation and even the collision time of the vehicle. Finally, using a real-world Internet of Vehicles data set, this paper implements a large number of experiments to verify the effectiveness of the proposed model.
1. Introduction
With the development of society, the number of vehicles is gradually increasing, and the problem of traffic safety has attracted more and more people’s attention. The frequent occurrence of traffic accidents is worrying. More than 10 million people worldwide are injured in road accidents every year. Among these accidents, vehicle collision is a serious safety problem, accounting for almost 30% of all accidents [1].
However, many accidents are closely related to the improper operation of drivers and the lack of timely and effective response to emergencies. In fact, with the development of technology, especially the development of Internet of Vehicles, automatic driving, and other technologies, the state of various parts of the vehicle can be tracked completely. It is very possible to predict whether the vehicle will collide or even predict the specific collision time in the future based on these data. However, since the implementation and deployment of intelligent transportation system and Internet of Vehicles are still in the initial stage, these data are still very difficult to obtain. At present, aiming at the problem of vehicle collision prediction, some researchers have proposed to monitor the vehicle environment in real time through the radar set on the road, the vehicle’s own infrared, camera, and other sensing equipment, so as to use these data to predict the vehicle collision and give timely warnings to the drivers on the vehicle [2–4]. However, these data based on the external environment, such as radar signals and images that are vulnerable to weather, have strong uncertainty. Therefore, some researchers turn their attention to the relatively stable interior of the vehicle, such as assessing the possibility of collision by paying attention to the driver’s behaviour [5, 6]. However, vehicle interior data is usually difficult to be effectively dynamically modelled because the driving habits of drivers are very personalized.
Recently, the rapid development of deep learning has brought great technological changes to various fields. Among them, the transfer learning technology, which can make full use of the previously collected data, makes the current model perform better on less data and has been favoured by more and more people [7–9]. Inspired by these works, this paper intends to use transfer learning to realize the complex task of vehicle collision prediction. In fact, compared with complex external data, real-time vehicle operation data from the interior of the vehicle, such as vehicle speed, accelerator pedal position, and brake pedal state, are less vulnerable to environmental interference and are directly related to the driving state of the vehicle. Transfer learning can discover more features related to vehicle collisions from limited vehicle operation data by means of knowledge transfer. In order to explore a more safe and effective vehicle collision prediction method, this paper uses the above data to carry out the research of vehicle collision prediction task. However, these data are still very difficult to obtain. In order to fully mine the correlation law between vehicle running state and vehicle collision from these limited data, this paper uses the efficient modelling method of transfer learning to build a model that can not only accurately predict whether the vehicle has a collision but also clearly point out the possible collision time. Specifically, the contributions of this paper are as follows: (1)This paper proposes a vehicle collision prediction model based on transfer learning, which is called TLVC. This method explores the new use of Internet of Vehicles data and provides a strong technical support for safe driving in the future(2)In this paper, a special feature analysis method is developed for the operation data from the inside of the vehicle, which provides a reference for the dynamic behaviour modelling of drivers. Moreover, this method of using vehicle internal data is more reliable than the previous methods based on image and radar signal(3)Using a small amount of Internet of Vehicles data and EfficientNet, this paper constructs a transfer learning model which is more accurate and clearer than the previous vehicle collision prediction model. Finally, through a large number of experiments, we confirm the effectiveness and accuracy of the proposed model
The remaining chapters of this paper are arranged as follows: the second section introduces the research related to vehicle collision and transfer learning. The third section introduces the vehicle collision prediction model proposed in this paper. The fourth section will introduce the real data set of this paper. The last section will summarize the full text and discuss our future work.
2. Related Work
In this part, this paper will introduce the previous work of vehicle collision prediction in detail and review the previous research on transfer learning.
2.1. Vehicle Collision Prediction
In recent years, many researchers have done research on vehicle avoidance and vehicle collision prediction. For example, Wang et al. [3] based on convolutional neural network and using the collected real trajectory data proposed a set of methods from data collection to preprocessing and then to prediction but did not deeply explore how to use the collected data to make more accurate prediction. Lyu et al. [4] established the lane change intention recognition model by tracking the driving direction of the vehicle and then established the collision early warning model by comprehensively predicting the vehicle trajectory. Candela et al. [10] combined with road layout information, statistical agent dynamics, and discrete Gaussian process for future vehicle position estimation, so as to realize vehicle collision prediction. Peng et al. [5] constructed a comprehensive “driver-vehicle-road” data set for actual driver behaviour evaluation, mainly analysing driver behaviour and relevant factors that significantly affect driving safety in emergency situations. Lee et al. [11] constructed a dynamic riding simulator that can control rolling motion, quantified driving behaviour by using lateral control ability, driver’s head movement, and emotional state, and predicted the overall collision avoidance ability by using multiple regression analysis of driving behaviour. Zhang et al. [12] proposed a multipedestrian collision risk assessment framework according to the motion characteristics of vehicles, including motion prediction module, collision inspection module, and collision risk assessment module. According to Katrakazas et al. [13], under the joint framework of interactive perception motion model and dynamic Bayesian network (DBN), network level collision estimation and vehicle-based risk estimation are integrated in real time, and machine learning classifier is used for real-time network level collision prediction. Wang et al. [14] proposed a collision prediction method based on the bivariate extreme value theory framework, taking into account the driver’s perceived response failure to take appropriate avoidance actions.
To sum up, the current vehicle collision prediction is mainly based on road information, vehicle external motion characteristics, and vehicle trajectory, combined with various roadside sensing units, etc., but there is a lack of attention to the characteristics of the vehicle itself and the driver’s operation state, and the vehicle collision problem is mainly modelled as a classification problem, and there is a lack of prediction of the collision time.
2.2. Transfer Learning
Migration learning improves the performance of the model in the target domain by migrating the knowledge contained in other source domains, which can greatly reduce the dependence of the model on the data of the target domain. Due to its wide application prospects, migration learning has attracted extensive attention recently [15]. For example, Ruder et al. outlined modern transfer learning methods in natural language processing (NLP), how models are pretrained, and what information is captured in their learning representation and reviewed examples and case studies on how these models are integrated and adjusted in downstream NLP tasks [16]. Raghu et al. discussed the characteristics of transfer learning for medical imaging [17]. Through a series of analysis of migrating to block shuffled images, Neyshabur et al. separated the effect of feature reuse from the high-level statistical information of learning data and showed that some benefits of migrating learning came from the latter [18]. Pathak et al. used deep transfer learning technology to classify patients infected with COVID-19 [19]. Wang et al. proposed dynamic distributed adaptation (DDA), which can quantitatively evaluate the relative importance of each distribution to solve the problem of transfer learning [20]. Tammina et al. used one of the pretraining models VGG-16 and deep convolution neural network to classify images [21]. Chen et al. proposed a joint transfer learning framework for wearable healthcare [22]. Rao et al. learned a model that can evaluate protein embedding tasks by migrating five biologically related semisupervised learning tasks [23]. Lotfollahi et al. used transfer learning and parameter optimization to achieve efficient, decentralized, and iterative reference construction and the upper and lower culture of new data sets and existing references [7]. Zhang et al. proposed a transfer learning (TL) technology through domain adaptation to bridge the gap between biased numerical model and real structure, guide Bayesian model update (BMU), and supervise structural damage identification [9].
To sum up, at present, due to some outstanding advantages, transfer learning has been widely concerned in various fields such as medicine and biology, but few people are involved in the emerging field of vehicle network. The work of this paper is to fill the vacancy of this data modelling method in the field of vehicle networking, explore the use of migration learning to model the real-time driving data of vehicles, analyse the vehicle state, learn the vehicle collision prediction model, and finally improve the safety of the driving system.
3. Method
This part will focus on the vehicle collision prediction model based on transfer learning proposed in this paper. The overall structure of the model is shown in Figure 1.
3.1. Problem Definition
Before introducing the model, we first give the definition of the vehicle collision prediction problem solved in this paper.
3.1.1. Definition Vehicle Collision Prediction
Given vehicle sets , according to the operation status , ( represents the number of historical running state data of the acquired vehicle , and represents the dimension of monitoring data) of each vehicle obtained from the monitoring system inside the vehicle. The problem of vehicle collision prediction in this paper finally comes down to the training prediction model , so that it can predict the collision of a given vehicle in the future according to the operation state data of the given vehicle , as follow:
where represents the final prediction result, . means that the modified vehicle will not collide in the future, and represents the specific time when the collision occurred. Figure 2 shows an example of vehicle collision prediction. In this example, the vehicle running state data in the Internet of Vehicles is first used to analyze the vehicle operation law, and the proposed TLVC model is used to simulate the law, and then, it is used to predict whether there will be a collision in the future and the time of the collision.
For the real label (physical time) of training data, we process it as follows:
is the training label finally sent into the model; formula (2) indicates the time point when the current vehicle starts monitoring. The above formula makes . The specific details of the prediction model will be described in detail below.
3.2. Preprocessing of Vehicle Operation Status Data
This part corresponds to the left half of Figure 1. The vehicle running data processed in this part include “accelerator pedal position,” “collect time,” “battery pack main negative relay status,” “battery pack main positive relay status,” “brake pedal state,” “driver leaving prompt,” “main driver seat occupancy status,” “driver seat belt status,” “driver demand torque value,” “handbrake status,” “vehicle key status,” “low voltage battery voltage,” “the current gear status of the vehicle,” “the current total current of the vehicle,” “the current total voltage of the vehicle,” “vehicle mileage,” “speed,” and “steering wheel angle.” Among them, features such as “battery pack main positive relay status” and “brake pedal state” are categorical features, while features such as “speed” and “steering wheel angle” are numerical features. For each category of features, this paper adopts three different coding methods to fully mine the relationship between these original features and vehicle collision. The first coding method is simple one hot coding. The final coding result is , where is the total number of samples and represents the dimension of the final one hot code. The other two types of coding are realized by probability distribution. Specifically, for an original feature , its possible value is , where is the total number of categories.
The second coding method is realized by using the collinear probability of collision between the feature and the vehicle:
where indicates that there is a collision and the value of the last time point of feature is the current number of vehicles to be coded, where represents the total number of vehicles in the sample set and represents a probability value, which is the coding of features . The final coding result of all category features obtained by this coding method is , where represents the feature coding dimension using cooccurrence probability for category features.
In the third coding method, we consider the frequency distribution of the vehicle for feature in the whole time detection window:
where represents the number of times that the value of feature is in the detection window of the vehicle in collision. represents the total number of samples for all vehicles involved in the collision. The third coding method reflects the occurrence probability of various features in the detection window. The final coding result obtained by this coding method is , where represents the coding dimension of the frequency distribution used in the window period for category features.
For numerical features, through simple normalization, the final feature code of this part is , where represents the final dimension of this part of features. Finally, by splicing the above feature code , , , and as the input of the collision prediction model, follows .
3.3. Prediction Model
This part corresponds to the right part of Figure 1. In order to obtain better results on the limited vehicle operation data set, this paper uses some parameters of the pretrained EfficientNet [24] to realize vehicle collision prediction through migration learning. In fact, the transfer learning method is widely used in the field of image processing. In this paper, the running state data of two-dimensional vehicles in the time window is compared with the pictures in image processing, and then, the correlation between vehicle running data and vehicle collision is learned through EfficientNet. Then, make the model learn the dynamics of vehicle operation, so as to realize the modelling of dynamic behaviour habits of drivers.
Figure 3 shows the structure of B0 version of EfficientNet. EfficientNet achieves good results without consuming more computing resources by coordinating and controlling the depth, width, and input data size of the network at the same time. This relatively small and refined model is quite cost-effective for applications in the Internet of Vehicles. EfficientNet has been widely used in transfer learning in recent years and has performed well in various tasks, so this model is used in this paper. In addition, this paper also verified through experiments that compared with other transfer learning models, EfficientNet is a better choice for this task.
In our task, in order to make effective use of the parameters of the pretraining model, we frozen half the parameters in EfficientNet in the training process, and let the other half of the network parameters participate in the training of vehicle collision prediction model, which helps to localize the model parameters as much as possible. That is, on the basis of making full use of the existing network parameters, let the model learn the task characteristics of the current vehicle collision data set. In the EfficientNet model using migration, its input is . As shown in Figure 1, its output will be input to the final full connection layer FC layer and then output the prediction .
Finally, the parameters of the whole model are updated by Adam W. Since the two tasks of whether the final vehicle collides and the time of collision are combined into a unified regression problem through formula (2), MSE is used as the final optimization goal in the process of model training.
4. Experiment
In this section, we will use the real vehicle signal data of the Internet of Vehicles to verify the effectiveness and accuracy of the proposed vehicle collision prediction model TLVC based on transfer learning.
4.1. Experimental Settings
This study conducted all experiments on a computer with Intel(R) Core(TM) i7-11700F @ 2.50 GHz and 16 GBDRAM. We implemented these algorithms in Python 3.6. The data used in this paper comes from 2021 Digital China Innovation Contest (https://www.datafountain.cn/competitions/500) and includes a series of vehicle operation data, vehicle collision labels, and collision time. The data includes the operation data of 120 vehicles in 2-5 days in total. The number of detection data of each vehicle is at least 4324 and at most 114460. For building a more accurate vehicle collision prediction model, this paper takes the data of 10 vehicles as the verification set and the remaining 110 vehicle data as the training set.
To enrich the data set and deal with the long vehicle state data, we truncate all vehicle detection data; that is, take consecutive records as a sample data. After such processing, we finally obtained 355,509 training samples and 42,058 test samples. The relevant parameter design of this paper is shown in Table 1.
To verify the effectiveness of the proposed TLVC method, this paper compares the proposed model with the following methods:
MLP: multilayer perceptron, a simple neural network model, is used as a comparison method in this paper
RNN: recurrent neural network (RNN) compared with the general neural network; this method can deal with the data with sequence changes
LSTM: long short-term memory; LSTM is a special RNN, which is mainly used to solve the problems of gradient disappearance and gradient explosion in the process of long sequence training. Compared with ordinary RNN, LSTM can perform better in longer sequences
BiLSTM: bidirectional LSTM is an extension of LSTM. Because two LSTM can be trained in two directions, the performance of sequence prediction model can be improved
Self-attention: this method is widely used in the field of natural language processing because of its excellent performance [25]. Vehicle running state data is a time series data, which is very similar to the text in natural language processing. Therefore, this method is used as a comparison method in this paper
CNN: convolutional neural network [26], a method widely used in the field of image processing, is used to process two-dimensional vehicle running state data in this paper
ResNet: deep residual network; ResNet is an excellent model that improves CNN in the field of image processing [27]
To measure the accuracy of the proposed vehicle collision prediction model, MSE is used to measure the accuracy of the prediction model. In addition, in order to evaluate the robustness of the model, another evaluation index in the regression problem is used to assist the evaluation of the model.
4.2. Comparison Result
In this part, we show the results of the proposed TLVC model compared with various benchmark methods. For the proposed TLVC, we use EfficientNet B4 version. The pretraining model has 1,918,200 parameters in total. The comparison results between the proposed TLVC model and each benchmark model are shown in Figure 4.
As shown in Figure 4, we compared the proposed model with other model benchmark models on the Internet of Vehicles data set from the real world. Among them, MSE reflects the prediction accuracy of the model. It can be seen from the results in Figure 4 that the prediction error of the proposed TLVC model is smaller than that of other benchmark models. As can be seen from Figure 4, using the methods of processing serialized data, LSTM and BiLSTM will obtain large model errors, and the results of these two models are even worse than RNN. This may be because these two models can only learn the characteristics of time series, and this length may lead to the disappearance of gradients in the learning process of these sequence models due to the sampling window . Unexpectedly, the MLP with simpler structure and composed of two-layer fully connected neural network obtained lower prediction error than LSTM and BiLSTM in the experiment. This may be because the fully connected neural network learns more effective features in the whole monitoring window, rather than paying too much attention to the vehicle state transition process in the window as the time series model. The prediction error of CNN model is slightly better than MLP, but its stability is much lower than other models according to the results of . This may be because only the local information of vehicle state change in the monitoring window is extracted through convolution, so it is difficult to infer the final collision of vehicles. The MSE performance of self-attention model is second only to the proposed TLVC model, because it can fully learn the dynamic changes and even correlation of various vehicle characteristic signals in the monitoring window. However, its value is not high, which may be due to the lack of training data, so the value of the model is low. The of ResNet and TLVC with migration model is slightly higher than that of self-attention, mainly because the migration model is less dependent on the amount of data. However, ResNet is similar to CNN model. Because it only focuses on the local features in the monitoring window, the final MSE is large.
On the one hand, the proposed TLVC has better model performance when there is only small training data because the way of transfer learning depends less on data. On the other hand, when using the transfer learning method EfficientNet, we only retain half of the parameters of the original model, and the other half of the model parameters can be learned with the model training, which makes the TLVC model not only do not rely too much on large quantities of data but also carry out effective localization learning.
4.3. Ablation Experiment
In order to verify the effectiveness of the vehicle state preprocessing part proposed in this paper, this paper compares the proposed model TLVC with the preprocessing module of removing early features, and the comparison results are shown in Table 2.
As can be seen from the data in the table, the model accuracy of without feature preprocessing is slightly poor. This shows the effectiveness of vehicle running state data processing in this paper. In particular, this paper encodes a series of category data. The results of ablation experiments show the effectiveness of these coding features in the proposed model, which provides ideas for the effective mining and application of Internet of Vehicles data.
4.4. Parameter Learning
In order to make the model as effective as possible, in this part, we analyse the influence of some parameters in the proposed model, such as model learning rate and number of reserved copies of migration model parameters , on the final effect of the model, as shown in Figures 5 and 6.
As shown in Figure 5, is the number of pretrained transfer learning model parameters reserved for the model, means that half of the parameters in efficientnet B4 are retained, and the other half of the parameters are trained and learned through the local vehicle collision data set. It can be seen from the figure that when , the model can achieve low prediction error.
Figure 6 shows our discussion on the learning rate of the whole TLVC model. Through experiments, we found that the model can achieve better results when .
5. Conclusion
In this paper, we use the vehicle running state data and propose a model TLVC which can predict whether and when the vehicle will collide in the future based on the migration learning model. Compared with the previous methods, this method does not need to rely on external unstable environmental data. It only needs to effectively process the vehicle operation signal by using the proposed preprocessing method and then use the semiparametric migration model for local data training to achieve high accuracy. In particular, because some parameters of TLVC migration model do not need training, learning on less vehicle operation data can achieve the purpose of task localization. Furthermore, by comparing with a series of state-of-the-art benchmark models, we verify the outstanding effect of the proposed method through a large number of experiments.
However, there are still some limitations. For example, the preprocessing of the vehicle operating state also relies on human understanding of the data, which is one of the issues we will further explore later. In addition, traditional transfer learning is usually the transfer of data in the same field, and this paper is limited by the limited sources of Internet of Vehicles data and uses data from different fields. Therefore, if there is a chance to obtain the same type of data in the future, we will discuss more effective transfer model.
Data Availability
The data set used to support the findings of this study are included within the article.
Conflicts of Interest
The authors declare that they have no conflicts of interest.