Abstract
A driving performance assessment (DPA) system has been proposed in this paper to evaluate drivers’ skills in training. The system is based on centimeter-level localization of the vehicles, thanks to differential BeiDou Navigation Satellite System (BDS). Given a vehicle’s dimensions, its envelopment has been discretized both temporally and spatially as binary images, while the training area is modeled as a grayscale image where the intensity denotes the penalty of the certain area in unit time. The performance index can be obtained from the summation of images along with time. Experiments have been conducted to demonstrate the accuracy of vehicle tracking and the effectiveness of the proposed assessment system.
1. Introduction
Autonomous vehicles (AVs) have been witnessed vigorous and fruitful in the last decade, with many applications in public transportation, logistics services, environmental health, disaster response, and space exploration [1–5]. Driving performance assessment (DPA) plays an essential role in building a robust AV system by providing feedback on the performance of an autonomous driver. Explicitly, learning-based approaches require a criterion such that different driving trajectories could be appropriately rewarded or penalized [6–8]. Besides potential applications on AVs, DPA is also helpful during intelligent driver monitoring and training [9–11].
In related work, driving performance has been assessed through mainly two perspectives, namely, (1) driver-centered and (2) vehicle-centered approaches. As for driver-centered methods, driver’s physical and mental status [12] and other subjective indicators [11] have been used for assessment. These evaluation indexes may reflect driving performance from the source but are sometimes hard to quantify, thus are trying for a fair evaluation in a variety of driving scenarios. On the other hand, vehicle-centered approaches focus on the vehicle’s velocity, acceleration, trajectories, and many other designed metrics. Authors in [11] have used completion time and the number of collisions to compare driving skills in different groups. In [13], an approach measuring maneuvering time, lateral velocity, and acceleration has been proposed to evaluate a lane change algorithm. In [14, 15], traveling distance and clearance to obstacle have been used as the cost function to optimize in AV trajectory planning problem.
This work proposes a DPA approach by evaluating vehicles’ trajectories. Firstly, a BeiDou Navigation Satellite System- (BDS-) based position tracking module which accurately records the vehicle’s trajectory has been built. BeiDou Navigation Satellite System (BDS) [16, 17] has been in rapid expansion since its first launch in 2000. As an alternative of GPS in the Asia-Pacific region, BDS has been widely used in transportation [18], agriculture [19], disaster prevention and mitigation [20], and many more. The tracking module includes a fixed base station and a mobile base station, where the fixed base station receives BeiDou satellite ephemeris data, and the mobile base station, which is mounted on the vehicle, receives differential data from the fixed base station. Then, an assessment approach has been proposed to evaluate trajectories such that the performance index could be obtained for each recorded trajectory based on the prebuilt map. The map has to be manually defined as a grayscale image, where the intensities indicate local penalties in unit time. Thanks to the high-accuracy trajectory recorded from the tracking module, driving performance can be scored by integration concerning space and time. Software to collect and process data has been developed for rapid and easy implementation. Experiments have been done on passenger vehicles to validate the effectiveness of the proposed DPA system.
The rest of this paper is organized as follows: Section 2 presents an accurate position tracking system. The proposed DPA algorithm is detailed in Section 3. Experimental studies and results are shown in Section 4. Finally, Section 5 concludes the paper and discusses possible future extensions.
2. BDS-Based Accurate Position Tracking
This section presents our proposed position tracking system structure, which includes both hardware configurations and front-end software.
2.1. Systematic Architecture
The architectural diagram of the proposed position tracking system is shown in Figure 1 [21]. Two BeiDou base stations need to be installed at different places. One is fixed as the reference point, for which the position is chosen depending on the practical situation. The other station, a mobile station, is placed properly on the vehicle to be tracked. The fixed station consistently receives the ephemeris data from BeiDou satellites, and the data are used for differential data calculation. The resulting differential data are then broadcasted out to the mobile station through the wireless or UHF radio data link. When data are captured and processed with necessary calculations, the tracked vehicle can be located in real time [22, 23]. The position and other related information calculated on the terminal computer will be sent to the monitoring center where data are displayed and logged.
2.2. Hardware Design
2.2.1. Fixed Base Station
As shown in Figure 2(a), the fixed base station includes a Global Navigation Satellite System (GNSS) receiver, GNSS antennae, a wireless or UHF digital radio station, the power supply equipment, and the lightning protection equipment.
(a)
(b)
BeiDou fixed base station, which is the origin in the system reference frame, receives and processes satellite signals continuously. It is powered by automotive batteries or any stable voltage of 12-220 V. The difference correcting value (DCV) of the base station is broadcasted through the onboard wireless/UHF digital radio station in real time, such that high-precision carrier phase differential data and initial coordinates for the mobile station can be obtained. DCV is the basis to achieve centimeter-level vehicle positioning precision, since the continuity and reliability of the differential information broadcast are directly related to the GNSS accuracy. As for base station location selection, the following items should be considered:(1)The site is stable and solid without any visible sinking(2)The height cutoff angles of visual obstacles’ heights should not exceed 15 degrees(3)The distance to any high-power radio source is not less than 200 m(4)The distance to any high voltage transmission line and microwave radio transmission channel is not less than 50 m(5)Lightning rods should be installed 1–3 m away from BeiDou fixed reference station and above its position
2.2.2. Mobile Station
The mobile station consists of a mobile terminal GNSS receiver host, GNSS antennae, and the power supply, as shown in Figure 3(b). In this work, the mobile terminal station GNSS receiver host is placed inside the mobile terminal. The GNSS antennae are at the top of the mobile terminal, installed, and fixed by a sucker or welding at a distance of 70 cm.
(a)
(b)
Similar to the base station, the mobile station is also powered by a car battery or any stabilized voltage source of 12 V to 220 V. The data stream between the BeiDou fixed base station and the mobile station is shown in Figures 3(a) and 3(b), respectively.
2.2.3. Error Analysis
The principle of the proposed position tracking system is shown in Figure 4. Since the BDS-based system outputs ENU coordinates only, transformations are necessary if coordinates in global frame are required. In particular, vectors and in ENU frame can be obtained from BDS. With a measured in the vehicle’s body frame, is then transformed to the global frame as the tracking result. Without considering the roll angle, it is noted that baseline position error δ would lead to angle error , where L denotes the baseline length. By writing the position error of the BDS-based system as δ, we have errors of baseline, pitch, and yaw in the ENU frame as
The errors of baseline, pitch, and yaw caused by the transformation from the ENU to the global frame are , , and , respectively, since the global frame is simply rotated from the ENU frame with the baseline length . Thus, the resultant errors of baseline, pitch, and yaw are derived as squared summations:
From (2)–(4), it is observed that , , but after transformation. The error of yaw would contribute to the total position error:
In this case, the position tracking error becomes large when . In practice, it is desired to set a large . Theoretically, the length would not affect tracking error from coordinate transformation, but larger is helpful in reducing pitch and yaw errors in the vehicle’s body frame.
2.3. Database and Application Design
The mobile application has been developed in Android 4.4.4, a Linux-based open-source operating system which consists of an application layer, an application framework, a system runtime, and a Linux kernel. The model-view-controller (MVC) simplified model-view-presenter (MVP) framework is used in development. The framework is divided into a view layer for graphical user interface (GUI) and interaction design, a presentation layer which handles the responsibility logic in interaction, and a model layer, which operates the database. In order to complete the training and evaluation task, four functional modules, namely, “sign in,” “training,” “inquiry,” and “configuration,” have been designed in the application, as shown in Figure 5.
3. Driving Performance Assessment
Since the exact position of the vehicle has been obtained, this section focuses on designing a straightforward approach to evaluate the driver’s performance. Suppose vehicle’s position has been converted to East-North-Up (ENU) frame. Then, the two-dimensional approximation is applied by removing z-axis from the coordinates by assuming that the vehicle’s trajectory is relatively short compared to the curvature of the earth’s surface. By writing the state vector as that contains the vehicle’s two-dimensional coordinates in translation plus one-dimensional coordinate in rotation, each trajectory can be represented as a function in state space . The DPA problem is then formulated as follows: Design a function such that for each given trajectory , a performance score can be computed.
3.1. Trajectory Envelopment and Map Representation
As vehicles have distinct dimensions which should be taken into consideration during DPA, the envelopment of the vehicle is defined as a static binary scalar field which satisfieswhere denotes the set containing all points within two-dimensional ( to ) projection of the vehicle in its body frame. Note that θ is not included here as is in the vehicle’s body frame instead of the global frame.
Based on the concept above, we further define the trajectory envelopment , which satisfieswhere x and y denote coordinates; , , and are obtained from trajectory . It is noted that is varying with time due to vehicle motion, but we omit the variable t for a concise notation. From (7), is actually represented in the global frame.
To evaluate if the vehicle is at the desired position quantitatively, a map in the global frame is introduced as an integrable static scalar field in two dimensions satisfying . The map reflects the area where the vehicle should stay on: The larger is, the more penalty will be received for the trajectory in unit time. The explicit form of is case-by-case depending on the application. In this paper, we apply the following map:where denotes the goal set, which is defined manually; represents the distance between point and the nearest drivable route that could be extracted from the map; the positive scale s represents how we measure the influence of distance; and a is a constant offset.
3.2. Graphical Trajectory Evaluation
With a map , the following operation is proposed to evaluate the trajectory envelopment derived from trajectory :
Firstly, we multiply and to create a new scalar field which indicates the driving performance of the vehicle’s current position. Both the dimension and position of the vehicle are considered during evaluation. Then, the new scalar field is integrated with respect to space and time such that the vehicle needs to reach the goal area as fast as possible while keeping close to drivable route in order to achieve higher evaluation score.
3.3. Approximated Version of DPA Algorithm
In practical implementation, the approximated DPA is required to ensure efficiency on a variety of platforms. In this work, we approximate and as binary and 8-bit grayscale images and , respectively, for convenient visualization. Each pixel in the image represents a grid with particular dimensions such that the coordinate transformation is linear. As the trajectory envelopment is varying with time, we finally obtain a sequence of images where k denotes the discrete time index. It is required that the time sampling interval remains the same in order for a fair evaluation at different times.
As matrices are used to represent images, the approximated evaluation function is written asin which denotes the Hadamard product (or the Schur product).
4. Experiments
In this section, the effectiveness of the proposed DPA approach has been demonstrated by actual tests. Since GPS has been widely applied to various applications, a criterion system with a GPS-based tracking module has been used in experiments for performance comparison.
4.1. Evaluation on Tracking Accuracy
We first evaluate position tracking accuracy before implementing DPA on the vehicle such that its trajectory can be precisely obtained. During all experiments, data recorded by the tracking system have been converted to 2D by assuming the vehicle moves around a relatively small region where the ground could be deemed as flat.
As shown in Figure 6, the testing vehicle runs three loops along the same route over 900 m. As the output frequency for the BDS and GPS systems are different, the Hausdorff distance [24] is used to measure the error of trajectories of the proposed system. The Hausdorff distance measures the maximum distance from points in one trajectory to the nearest point in the other trajectory. It is shown that the two trajectories are almost the same for most of the time, and it is computed that the average Hausdorff distance for the three loops is 5.44 m.
(a)
(b)
4.2. Evaluation on DPA System
Experiments have been conducted to verify the effectiveness of the proposed DPA system. During map generation, different resolutions may be chosen. Figure 7 illustrates maps from coarse to fine. It is noted that a finer map leads to smoother and more accurate approximation but would also increase computational complexity. We approximate the map as 8-bit grayscale image , where each pixel represents 0.02 meters. The map intensities at the starting area and the target area are set to offset a and 0, respectively. As shown in Figure 8, two trajectories from the starting area to the target area have been collected in this work, and the physical dimension of the vehicle has been deemed as zero. In other words, is a single-pixel image that moves along with the trajectory.
The results are shown in Figure 9, which illustrate the corresponding evaluation scores with respect to discrete time. It is noted that the evaluation score is monotonically increasing due to time integration. As for the groundtruth trajectories from GPS, the 1st trial obtains lower score owing to its slight deviation to drivable roads and faster completion. However, as shown in the 2nd trial, the BDS-based tracking system sometimes outputs intermittent tracking results, which explain why DPA gives different results. Figure 10 shows the influence on evaluation score with different map resolutions ranging from (0.005 m/pixel) to (1 m/pixel). The scores are quite steady while the resolution is smaller than 0.1, and the slight disparity comes from approximation errors.
We further show the real-time performance of DPA as in Figure 11. It takes 0.08 s to compute scores for the two pairs of trajectories on average, with 0.02 m/pixel resolution. It is evident that the time consumption for score computation decreases with a coarser map. Based on the above discussion, it is a case-by-case trade-off between algorithm efficiency and approximation accuracy in practical implementation.
5. Conclusion
In this paper, a driving performance assessment system, which includes a differential BeiDou Navigation Satellite System-based accurate position tracking and a back-end trajectory evaluation method, has been designed. Experimental validation has been made during field tests to demonstrate the effectiveness of assessment, given a predefined map. The proposed system can be applied to automated driver training, or as a quantifiable criterion to be fed into learning-based autonomous driving approaches.
Data Availability
The data used to support the findings of this study were supplied by the Army Engineering University of PLA under license and so cannot be made freely available. Requests for access to these data should be made to the corresponding author.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.