Abstract
The prediction of lanechange behavior is a challenging issue in intelligent and connected vehicles (ICVs), which can help vehicles predict in advance and change lanes safely. In this paper, a novel intelligent approach, which considering both the driving stylebased lanechange environment and the driving trajectoryrelated parameters of the ICV and surrounding vehicles, is proposed to predict the lanechange behaviors for ICVs. By analyzing the characteristics of the lanechange behavior of the vehicle, a modified dataset for the prediction of lanechange behavior was established based on the NextGeneration Simulation (NGSIM) dataset, which is made up of real vehicle trajectories collected by camera. In the proposed approach, the hidden Markov model (HMM)based model is designed to judge whether the current environment is suitable for lane change according to the driving environment parameters around the vehicle; then according to the driving state of the vehicle, a learningbased predictionthenjudgment model is proposed and designed to realize the prediction of the ICV’s lanechange behavior. Experiments are implemented by using the modified dataset. From the experimental results, the lanechange probability value on the target lane in the truth of the lanechange behavior calculated by the designed HMMbased model is basically above 0.5, indicating that the model can make a more accurate judgment on the surrounding lanechange environment. The proposed learningbased predictionthenjudgment model has an accuracy of 99.32% for the prediction of lanechange behavior, and the accuracy of the lanechange detection algorithm in the model is 99.56%. The experimental results show that the proposed approach has a good performance in the prediction of lanechange behavior, which could effectively assist ICVs to change lanes safely.
1. Introduction
The intelligent and connected vehicle (ICV) [1] integrates modern communication and network technology and has environment perception, intelligent decisionmaking, and collaborative control functions. It can achieve safe, efficient, comfortable, and energysaving driving and realize a new generation of vehicle that replaces humans [2].
Lanechange behavior detection and prediction plays an important role in the ICV technology. During the driving of the vehicle, the current driving environment may be misjudged due to the occlusion of surrounding vehicles or the driver’s inattention, resulting in greater safety hazards. Therefore, the sensor and communication technology can assist the ICV to perceive and judge the surrounding environment and the state of the vehicle and combined with artificial intelligence technology can predict the lanechanging behavior, thereby improving the driving safety.
Many methods have been proposed for lanechange detection and prediction, in which the main technical means and data sources used can be summarized as trajectories, steering wheel, surrounding environment, driving style, computer vision and roadside LiDAR, etc., as shown in Table 1. Lanechange behavior detection methods based on trajectory data are proposed in Refs. [3–8], such as fuzzy logic [3], support vector machine (SVM) [4], long shortterm memory network and convolutional neural network (LSTMCNN) [5], maneuver classification [6], and hidden Markov model [7, 8]. Panichpapiboon and Leakkaw explore an approach to detect lanechange behavior using steering wheel angles extracted from the smart phone [9]. Zheng and Hansen propose an approach to detect lanechange behavior using the steering angle signal from CANbus [10]. Ali et al. propose a wavelet transform (WT)based method to detect failed lanechanging attempts and used the random parameter binary logic model to study how the connected environment affects related parameters [11]. Woo et al. present a method to determine the driving styles and use the result to detect the lanechange behavior [12]. Nguyen et al. introduce a visionbased lane and vehicle detection approach for the lanechange assistant system [13]. Wang et al. present a method to detect lanechange behavior based on candidate lane markings [14]. Wei et al. develop a computer vision system to detect the lanechange behavior [15]. Cui et al. develop the methods to detect and predict lanechange behavior using vehicle trajectories from roadside LiDAR data [17]. Xu et al. present a V2Xbased lanechange prediction model using vehicle trajectories [18]. Zhang and Fu present a lanechange intention detection method using motion parameters of the vehicle and surrounding vehicles [19]. Gao et al. introduce a lanechange behavior detection approach using multiple differing modality data [20]. Jin et al. present an optimal lanechange timing prediction model based on the driver’s habits [21]. Huang et al. present a trajectory planning and control approach based on user preferences [22]. Xing et al. propose a driving pattern analysis and motion prediction system that determines the trajectory according to user’s preference [23]. Xing et al. develop a driver intention inference system for highway lanechange maneuvers [16]. Xing et al. present a leading vehicle trajectory prediction approach that considers different driving styles [24].
In the above studies, different methods and models using different technical means and considering the influence of different characteristic parameters have been designed and proposed, fully verified, and achieved great results. However, few studies have simultaneously considered the effects of the vehicle, environment and driver, and the relationship between them when the ICV changes lanes. In this paper, a novel intelligent approach combines the driving state of the vehicle, the surrounding driving environment, and the driving style is proposed to predict the lanechange behaviors for ICVs. First, based on the learning of the driving habits of manual drivers, the current lanechange environment is judged according to the driving state of surrounding vehicles. If the current environment is suitable for lane change, then the vehicle driving state parameters are predicted, and the lanechange behavior detection method is proposed to judge the predicted value, so as to predict the lanechange behavior. The main contributions can be summarized as follows.(i)According to the relevant characteristic parameters of the vehicle lane change, the NGSIM dataset is processed and analyzed, so that a modified dataset for the lanechange behavior prediction is established(ii)Based on the driving habits of manual drivers, a HMMbased model is designed to judge whether the current surrounding environment of the vehicle is suitable for the lane change(iii)Based on the analysis of lanechange behavior characteristics, a prediction model based on LSTM and lanechange feature judgment method is proposed to predict the state parameters of the vehicle and determine whether it will change lanes(iv)A novel approach based on intelligent and connected technology, which in combination with the driving stylebased lanechange environment and the driving trajectoryrelated parameters of the vehicle and surrounding vehicles, is proposed and performed on the established dataset to predict the lanechange behavior of vehicles
The rest of the paper is organized as follows. In Section 2, the establishment process of the dataset is described. In Section 3, on the basis of fully analyzing the characteristics of lanechange behavior, the proposed approach to lanechange behavior prediction is introduced in detail. Section 4 gives the experimental results and analysis of the proposed approach performed on the modified dataset. Section 5 concludes the research work and presents the future work.
1.1. Dataset Establishment
In this paper, the NGSIM dataset is processed to obtain the vehicle’s trajectory and surrounding driving environment data, so as to combine the driver’s driving style to build and verify the vehicle’s lanechange prediction model.
1.2. Data Description
The NGSIM is a dataset of different sections initiated by the United States Department of Transportation (US DOT) Federal Highway Administration (FHWA) [25]. In the NGSIM, I80 and US101 are the datasets collected in highway, which are studied in this paper. As shown in Figure 1, both I80 and US101 consist of five main lanes, one distribution lane, one onramp, and one offramp (the offramp of I80 is not located within the study area). In I80, the 1650footlong study area is divided into seven subareas by seven cameras to record the relevant data, while in US101, the 2100footlong study area is divided into eight subareas by eight cameras. The dataset contains the trajectory data of all vehicles in the study area during the recorded time period.
(a)
(b)
1.3. Data Preprocessing
In order to analyze the characteristics of the lanechange behavior, characteristic parameters such as the coordinates and velocity of the vehicles are extracted from the NGSIM dataset. The coordinates of the ramp and the most adjacent lane have a large overlap, which will cause great interference to the study. Therefore, the data related to the ramp and the most adjacent lane are eliminated in the study. To further analyze the influence of the surrounding lanechange environment on lanechange behavior and the relationship between them, the distance between the vehicle and the front and rear vehicles on the current lane and adjacent lanes is calculated. The lateral speed of the vehicle is also calculated in order to predict the lanechange behavior. The complete data of 92 lanechanging vehicles, a total of 92932 frames, are finally screened out and processed; then a modified dataset is established.
In the processed data, the surrounding lanechange environment at the time of a certain vehicle lanechange frame is shown in Figure 2. The range of lane coordinates calculated according to the data in the processed dataset is shown in Table 2.
2. Methodology
2.1. Analysis of LaneChange Characteristics
According to the study of Balal et al. [26], the main characteristics that affect drivers’ lane change are D_{ft}, D_{pft}, D_{pt}, D_{pc}, and V_{c}. Lanechange behaviors can be divided into leftlane change and rightlane change, so D_{ft}, D_{pft}, and D_{pt} can be divided into D_{fl}, D_{fr}, D_{pfl}, D_{pfr}, D_{pl}, and D_{pr}, which were defined in Table 3. The typical lanechange scenario taking the rightlane change as an example can be described in Figure 3.
2.2. Intelligent Prediction Approach
Based on the analysis of lanechange characteristics in real scene dataset, an intelligent prediction approach is proposed and established, in which the HMMbased model is used to judge the lanechange conditions, LSTMbased model is used to predict the current vehicle motion data that are suitable for change lanes, and then the designed lanechange detection algorithm is performed to complete the lanechange behavior prediction.
2.2.1. HMMBased LaneChange Environment Judgment
HMM can be used to predict the probability of whether a vehicle changes lanes [27–29]. The vehicle and surrounding vehicles’ driving state determines to a large extent whether the vehicle has the conditions for changing lanes. In this paper, based on the analysis of vehicle lanechange characteristics, eight parameters, D_{fl}, D_{fr}, D_{pfl}, D_{pfr}, D_{pl}, D_{pr}, D_{pc}, and V_{c}, are selected as observations to judge the surrounding lanechange environment.
As shown in Figure 3, in the HMM model, the eight continuous observation values, D_{fl}, D_{fr}, D_{pfl}, D_{pfr}, D_{pl}, D_{pr}, D_{pc}, and V_{c}, are used as observation vectors. In order to simplify the model and facilitate implementation in practical applications, the continuous values of the observation vector are divided into eight segments according to the importance of each observation vector’s influence on lanechange behavior. The observation vector can be defined as follows:where V = [V_{1}, V_{2}, …, V_{T}] is the observation sequence, T is the sequence length, O = [O_{1}, O_{2}, …, O_{M}] is the observation state, M is the distribution of the observable state, which is divided into 8 states according to the value of the observation vector and its importance [26], and M∈{1, 2, 3, 4, 5, 6, 7, 8}. The hidden states are the lanechange behaviors, including nonlane change, leftlane change, and rightlane change, which are represented as H_{1}, H_{2}, and H_{3}, respectively.
As shown in Figure 4, the parameters of designed model can be defined as follows:where A means the state transition probability matrix, in which a_{ij} is the probability of transition to state h_{j} at time T + 1 under the condition that time T is in state h_{i},
B represents the observation probability matrix, in which b_{j(M)} is the probability of generating the observation O_{M} under the condition that time T is in state h_{j}:
indicates the initial state probability distribution, in which is the probability of being in state h_{i} at time t = 1:
In this paper, the dataset contains the observation sequence and the corresponding state sequence. Therefore, the supervised learning method can be used to estimate parameters of HMM. The maximum likelihood estimation method is used, and the specific method is as follows:(1)Estimate the transition probability. Assume that the frequency of the sample at time t in state i and transition to state j at time t + 1 is A_{ij}, then the estimation of state transition probability a_{ij} is as follows:(2)Estimate the probability of observation. Assume that the frequency of the sample state is j and the observation is M is B_{jM}, then the estimation of the probability b_{j(M)} that the state is j and the observation is M is as follows:(3)Estimate the initial state probability. The estimate of the initial state probability is the frequency at which the initial state is h_{i} in the sample.
After the model parameters are determined, using the forward probability and the backward probability, given the model and the observation O, the probability of being in the state h_{i} at time t can be obtained:
From the definition of forward probability and backward probability ,
Then
2.2.2. LSTMBased Vehicle Trajectory Prediction
After judging the surrounding lanechange environment, the data suitable for lane change would be screened out to predict the lanechange behavior. An LSTM [30] model is designed to predict the current vehicle motion data. The structure of an LSTM block [31] is shown as Figure 5, in which f_{i} is the input activation function, f_{o} is the output activation function, and is the gate activation function. At time t, x_{t} is the input, h_{t} is the hidden layer state, it is the output state of input gate, f_{t} is the output state of forget gate, and o_{t} is the output state of output gate, which can be expressed as follows:where , , and are the input weight matrices; , , and who are the feedback weight matrices; and b_{i}, b_{f}, and b_{o} are the bias vectors.
The intermediate states at time t are as follows: the output state corresponding to the input function, the output state C_{t} corresponding to the output function, and the output state h_{t} corresponding to the hidden layer.where , , and are the input weight matrix and the corresponding bias vector, respectively. , as the input function, the output state at time t will participate in the overall update of the input state at time t together with the output state i_{t} of the input gate at time t. As the output state of the input function at time t, participates in the overall update of the input state at time t together with the output state i_{t} of the input gate at time t.
At time t, through the new input and state feedback at previous time, the entire LSTM unit is updated, including the update of C_{t} and h_{t}:
In the update process of each gate function and the output state of the entire unit, the key information in the input feature is retained and transferred through the forget gate function and the transfer of the state.
2.2.3. The LaneChange Behavior Prediction Approach
The predicted data with conditions for lane change are used to determine whether the current vehicle will change lanes through the lanechange detection algorithm, which can be described as shown in Algorithm 1. An optimal sampling interval length (δ sampling period) is obtained according to the data training, and then the velocity data on xaxis are used as input according to the obtained sampling interval length. In the process of lanechange detection, first, calculate all zerocrossing points in the input data and sort them by time; then, calculate the lateral velocity integral between all adjacent zerocrossing points; finally, train the calculation results based on the KNN [32, 33] method, and output lanechange detection result (the lanechange behavior category) and the start and end time of lanechange process.

The structure of proposed multimodel fusion lanechange behavior prediction approach can be described as shown in Figure 6. First, the surrounding environment parameters related to the lanechange condition are used to judge the current lanechange condition through the HMMbased model. Then, the vehicle trajectory data which meet lanechange conditions are predicted by the LSTMbased model. Finally, the lanechange behavior is predicted by the proposed lanechange detection algorithm; the predicted lanechange behavior and the start and end time of lanechange process are output.
3. Results and Analysis
3.1. Evaluation Metrics
When evaluating the prediction results, the confusion matrix definition of the prediction results is shown in Table 4. Accuracy, precision, recall, and F1 value are usually used as evaluation indicators [34] for learningbased classification and prediction models. Among them, the accuracy represents the proportion of the sample size correctly classified in the total sample size, which can be defined as follows:
Precision (P), which indicates the proportion of samples with the correct class label among the samples of a particular class found by the classifier, can be defined as follows:
The recall (R) represents the classifier’s ability to find samples of a certain category, which can be defined as follows:
The F1 value is a comprehensive index that considers the balance between precision and recall, which can be defined as follows:
The closer the F1 value is to 1, the better the effect.
3.2. Experimental Results
The proposed prediction approach, including HMMbased lanechange condition judgment, LSTMbased vehicle lane changerelated parameter prediction, and lanechange detection algorithm, is trained and tested on the established modified dataset to evaluate the performance.
3.2.1. Performance of HMMBased Model
In order to verify the judgment performance of the HMMbased model on the lanechange environment, parameters related to lane change, including D_{fl}, D_{fr}, D_{pfl}, D_{pfr}, D_{pl}, D_{pr}, D_{pc}, and V_{c}, are processed and then trained and tested. The results show that at all lanechange times, the lanechange probability of target lane is basically above 0.5. At the moment when the vehicle does not change lanes, some lanechange environments meet the lanechange conditions, and some do not. Therefore, the designed model can screen out the moments that do not meet the lanechange conditions and improve the prediction accuracy.
The schematic fragment of the designed HMMbased judgment result of lanechange condition is shown in Figure 7. In the figure, the data of the green line represent the truth of the lanechange behavior (1 means rightlane change, −1 means leftlane change, 0 means nonlane change), the data of the red line indicate the probability of the rightlane change calculated by the designed model, while the data of the blue line denote the probability of the leftlane change calculated by the designed model. It can be seen that the calculated rightlanechange probability at the time of rightlane change and the leftlanechange probability at the time of leftlane change in the figure are all greater than 0.5, which meets the lanechange conditions.
3.2.2. Performance of LSTMBased Model
In order to predict the lanechange behavior of the ICV, the LSTMbased model is designed to predict the lanechangerelated motion data (lateral velocity) at the next moment. The dataset is divided into training set and test set at a ratio of 2 to 1 to verify the performance of the designed model. The loss curve of the training process is shown in Figure 8, in which the loss value is stable at around 2.32E05.
The prediction result of designed LSTM is shown in Figure 9, in which the blue line represents original data and yellow line and green line indicate the prediction result of the training set and the test set, respectively. The root mean square error (RMSE) of the prediction is 0.37 m/s for the training set and 0.68 m/s for the test set. From the prediction results in the figure, it can be found that the overall prediction error of the designed model is small, and the prediction error is greater when the data have large and sudden changes than when the data are flat. The maximum prediction error of the dataset is 5.5668 m/s (the original data is 50.0055 m/s).
3.2.3. Performance of the Detection Algorithm
The designed lanechange behavior detection algorithm was performed on the established dataset to verify its detection effect on lanechange behavior. The dataset is divided into training set and test set at a ratio of 2 to 1, the experimental result of detection is shown in Table 5, and the confusion matrix of it is shown in Figure 10.
It can be seen from the experimental result of detection that the P of leftlanechange detection has reached 100% and R of it is 83.05%, the P of rightlanechange detection is 90.91% and R of it is 100%, while P and R of nonlane change are 99.55% and 100%, respectively. The F1 values of leftlane change, nonlane change, and rightlane change are 90.74%, 99.77%, and 95.24%, respectively.
From the confusion matrix of detection result, it can be found that the accuracy of the detection algorithm is 99.56%. Among them, 10 samples in nonlane change are detected as leftlane change, and 1 sample is detected as rightlane change, while no lanechange behavior is detected as nonlane change and there is no error detection between leftlane change and rightlane change, which shows that the proposed detection algorithm could accurately detect lanechange information for the safe lane change of ICV.
Taking entire driving process of vehicle 2458 as an example, the detection result is shown in Figure 11. It can be seen from the figure that vehicle 2458 made a lane change at t = 600 during the whole process, and its lateral velocity has an obvious acceleration process. The designed detection algorithm accurately detects the lanechange behavior and calculates the lanechange process that is between t_{1} = 591 and t_{2} = 641.
3.2.4. Performance of the LaneChange Behavior Prediction Approach
Finally, the proposed prediction approach is performed on the established dataset to verify the effect of the approach. The proposed lanechange behavior detection algorithm is performed on the filtered prediction data that meets the lanechange conditions. The dataset is also divided into training set and test set at a ratio of 2 to 1, the experimental result of prediction is shown in Table 6, and the confusion matrix of prediction result is shown in Figure 12.
From the experimental result of prediction, it can be seen that P of leftlanechange detection is 89.36% and R of it is 95.45%, the P of rightlanechange detection is 90.00% and R of it is 81.82%, while P and R of nonlane change are 99.72% and 99.65%, respectively. The F1 values of leftlane change, nonlane change and rightlane change are 92.31%, 99.65%, and 85.71%, respectively.
It can be found from the confusion matrix of prediction result that the accuracy of the prediction approach is 99.32%. Among them, 2 samples in nonlane change are predicted as leftlane change, and 2 samples are predicted as rightlane change. 5 samples in leftlane change are predicted as nonlane change, with a precision of 89.36%, and 1 sample in rightlane change is predicted as nonlane change, with a precision of 90.00%, while there is no error detection between leftlane change and rightlane change. The experimental result shows that the proposed prediction approach could effectively provide vehicle lanechange information to assist the ICV in safe lane change.
Taking the entire driving process of vehicle 2458 as an example, the prediction result is shown in Figure 13. It can be seen that the predicted lateral velocity and the truth of lateral velocity are basically consistent in value and trend. The designed prediction approach accurately predicts the lanechange behavior and calculates the lanechange process that is between t_{1} = 590 and t_{2} = 776. The predicted time interval of the lanechange process is longer than that calculated by the detection algorithm, which is because that when the lateral velocity value fluctuates around 0, the predicted lateral velocity value fluctuates slightly and is less than 0. The prediction approach can still accurately predict the lanechange behavior and the time interval of lane change.
4. Conclusions
The paper proposed a novel intelligent approach to lanechange behavior prediction for ICVs, which combines the surrounding lanechange environment and the vehicle’s own motion parameters. A modified dataset is established based on the NGSIM dataset, and then the proposed approach is trained and tested. From the experimental results, the HMMbased model can make relatively accurate judgments on the lanechange environment, and its calculated lanechange probability at the time of lane change is above 0.5. The prediction RMSE of the vehicle lateral speed by the LSTMbased model is 0.37 m/s for the training set and 0.68 m/s for the test set. The proposed lanechange detection algorithm has an accuracy of 99.56% on the established dataset and can accurately calculate the time interval of the vehicle lane change. On the basis of the fusion of the above models and algorithm, the proposed intelligent prediction approach is completed, result shows that the accuracy of the prediction approach on the established dataset is 99.32%, and the time interval of the vehicle lane change can be calculated accurately. The experimental result indicates that the proposed prediction approach could effectively provide vehicle lanechange information to assist the ICV in safe lanechange and has the potentials for application in actual intelligent and connected environment for ICVs.
Since the proposed approach is postprocessing with measured data, its realtime performance in practical applications needs to be further verified. In future work, the proposed approach can be deployed on mobile terminals for realtime testing, and its accuracy and realtime performance can be further improved.
Data Availability
The modified NGSIM data used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Acknowledgments
The research work was supported in part by the National Key R&D Program of China under Grant 2018YFB0105205, in part by Technological Innovation Project (Major Program) of Hubei Province under Grant 2019AAA025, and in part by China Scholarship Council under Grant 202106950043.