Abstract
To improve the technology of unmanned ground vehicles, it is necessary to conduct a proper evaluation on various technologies. Previous evaluation methods are mainly based on completion of the task; this may mislead most of teams of unmanned ground vehicles using a conservative strategy during the evaluation. In this paper, a new evaluation method is proposed. Based on typical working conditions including intersection, car-following, and obstacle-avoiding, the new evaluation indicator system is established, and the entropy-cost function method is applied to the comprehensive evaluation of unmanned ground vehicles. As reported in a numerical example, the proposed evaluation method can get a quantitative result that authentically reflects the intelligent behavior level of unmanned ground vehicles.
1. Introduction
Based on the preestablished evaluation system, the technology of intelligent vehicles abroad has developed rapidly in recent years. Obviously an excellent evaluation system can guide participants to improve the performance of intelligent vehicles in evaluation/test. For example, none unmanned vehicles finished the entire race in the US intelligent vehicle competitions DARPA 2004 [1]. Guided by the rules/evaluation rules of Grand Challenge, the teams improved their unmanned vehicles and five teams accomplished the entire distance in 2005 competition [2]. In the history of technology development, evaluation system has played an important role.
The 2nd DARPA Grand Challenge simply used the “the number of finishing the races” and “the total number of gates through” to rank the teams [3]. The 3rd DARPA Grand Challenge, Urban Challenge, required that all participants must obey traffic rules [4]. “Future Challenge 2009” was held in June 2009 [5]; the race used the third-party testing to evaluate the implementation efficiency of intelligent vehicles in competitions. This attempt has promoted and facilitated the innovation and development of vehicle verification platform [6]. “Future Challenge 2012” competition was held in November 2012, Chifeng City, Inner Mongolia [7]. The race mainly tested the research level of cognitive computing about visual and auditory information [8]. U-Turn evaluation environment is accepted for more competitions in recent years, but the evaluation environment cannot take more uncertain factors into consideration [9, 10].
Most of existing foreign evaluation methods of domestic and abroad intelligent vehicle competitions used the mission-driven evaluation approach, which has an obvious shortcoming, leading many teams to adopt a conservative strategy [11]. For example, in order to avoid the violation of traffic regulations, many teams chose to park and wait [12] rather than interact with the environment as manned vehicles do. Even though they can get high score, it is a departure from the purpose of technological development [13]. As for domestic methods, the evaluation system is obviously not comprehensive; many factors are not considered in evaluation/test. For example, the papers [7–10] just focus on dynamic driving abilities and ignore static conditions, interactive abilities with environment. In the future, intelligent vehicles will interact with more people and cars in complex scenes [14–16]; thus the evaluation method needs improvement. The method must take more comprehensive evaluation factors into account and reflect the performance of unmanned vehicles objectively.
This paper concentrates on proposing a novel evaluation method for intelligent vehicles that is based on information entropy and cost function. The information entropy checks all evaluation indicators’ weight from information amount aspect to handle uncertainty problems in evaluation/test; the cost function checks the intelligent level of each of abilities to get specific evaluation score. The evaluation method also concludes a new rigorous evaluation indicator system based on typical working conditions; each typical working condition of intelligent vehicles is subdivided into different physical indicators to reflect real situation. Not only the completion time of indicators but also all the details are taken into account, even down to the completion quality and veracity of the various secondary indicators. The new evaluation method refines three aspects of evaluation process: evaluation indicator system establishment, indicators’ weight arrangement, and evaluation score classification to guarantee the objectivity, comprehensiveness, and scientificalness of evaluation results. The logic and construction of the paper are shown in Figure 1.
The final evaluation results can show the intelligent level of unmanned vehicles and its weakness and then guide the participating vehicles to move in the right direction and goals of high-tech development.
2. Foundation of the Evaluation Indicator System
As described above, most of the practical evaluation activities were mission-driven, which resulted in the incomprehensive evaluation indicator system. Thus the evaluation results were partial. Most participants could get higher score just because of their own research superiorities. It is not fair and objective. That means a proper selection of evaluation indicators is an important part in the evaluation.
Evaluation of unmanned vehicle intelligent behavior is a multilevel comprehensive evaluation problem. Considering the characteristics of unmanned vehicles’ data which are scattered and the advantages and disadvantages of traditional evaluation indicators now, expert opinions and analysis of typical working conditions are selected; this can not only make full use of the experts’ cognitive knowledge of unmanned vehicles but also avoid missing important indicators. The result is relatively accurate with the rigorous indicators selection process (see Figure 2).
The paper mainly takes two factors into consideration: objective one and subjective one, which are given in Figure 2. Objective working conditions reflect the external environment; subjective opinions from experts reflect the internal attitude in the evaluation/test.
The typical working conditions are summarized from many intelligent vehicle competitions. In each of intelligent vehicle competitions, the participants will encounter three main working conditions; those working conditions almost conclude all of intelligent driving behaviors in evaluation/test. According to the typical working conditions including intersection, car-following [17], and obstacle-avoiding, the paper analyzes the performance of different situations that can show the real level of unmanned vehicles’ intelligence and then summarizes essential features of each typical working condition. Each typical working condition of intelligent vehicles reflects main intelligent behavior; each behavior has its tasks. Evaluation indicator structure is given in Table 1. All indicators are established through considering real situations in the driving process; the physical characteristics can be reflected in the indicator system.
The indicator called “Parking precision” tests the ability to park the vehicle at the right place and the right time when facing intersection. “Restart ability” tests the ability to identify complex intersection conditions and participate in the traffic intelligently. “Speed capability” reflects the performance on the speed controlling, acceleration, and deceleration. “Braking deceleration” tests braking quality. “Early warning” tests video detection system and the ability to identify different obstacles. “Avoidance in right angle” tests whether the vehicle can use less space to avoid the obstacles. “Path replanning” can guarantee the vehicle will not abort the original destination. “Stimulation” tests the ability to calculate the fore car’s real-time speed, thus taking actions timely. “Safe distance” evaluates the stationary and the following model of the intelligent vehicle [18, 19]. “Speed optimization” tests the ability to control distance through keeping optimal speed.
3. Information Theory and Entropy for Indicators’ Weight
This paper introduces knowledge of information theory to determine the weight distribution parameters. In 1948, in the paper “A Mathematical Theory of Communication,” Shannon used Probability and Statistics approach to the study of communication systems, revealing that the object of communication system is information and then proposed the concept of entropy through describing the information quantitatively.
Based on Shannon’s theory, the recipient cannot predict the message that will be received; therefore, uncertainty exists objectively because of the message’s randomness. Delivery of messages is a process to eliminate the uncertainty of information, a process from uncertainty to certainty. Uncertainty in information theory is a kind of inherent property existing in working process, so the concept of entropy can be applied to the evaluation of intelligent behaviors of unmanned ground vehicles. Information entropy can determine the key indicators in terms of the amount of information to calculate the indicator weight parameters, and the entropy of information can be calculated as follows:
3.1. Judgment Matrix
To judge all factors with 1~9 scaling method to determine each indicator’s importance, experts’ suggestion is summarized to construct the judgment matrix of fractional values, and evaluation indicator system has aspects behaviors, objects, auxiliary indicators, , and .
3.2. Matrix Data
According to the linear proportional relationship, the decision matrix is reformed as standardization to get standardized matrix and then normalize :
3.3. Information Entropy
Calculation of indicator’s information entropy is as follows:
3.4. Information Utility Value
The real parameters that reflect indicator weights are the values of information utility; the values can be calculated by the following formula:
3.5. Total Indicator Weight
The importance weight of all factors of this layer corresponding to the above layer of a certain factor that can be calculated using the result of all single hierarchical sorts at the same hierarchy is as follows: is objective weight of each indicator.
4. Cost Function for Score Classification
According to the 2007 DARPA Urban Challenge’s low-speed, low-density traffic environment issues, scholars at Carnegie Mellon University proposed a kind of robust highway autonomous driving technology that is combined with the cost evaluation method [20]. The cost function method can also be used in Chinese intelligent vehicle competitions to quantify the indicators. The cost function method is bound to promote competition vehicles toward the direction of the minimum cost function in the process of technology development, thus leading unmanned ground vehicles to finish the task with high-quality. The technological level of Chinese unmanned vehicles on its natural environment perception and decision-making capacity is gradually improving.
Taking the indicator “Path replanning” as a research object, unmanned vehicles need to consider the process cost control that means more planning time and more useless operating range deserve more penalty and higher process cost. The cost can be calculated by the following formula:
To make sure of the comprehensive consideration for all indicators and the full use of evaluation information, the paper adopts a weighted average algorithm to get final evaluation results by decomposing calculation. As for typical intelligent behaviors, the paper presents each cost function as , , and . can be calculated as follows:
presents the cost of indicator of intersection behavior; presents the weight of indicator of intersection behavior; presents the maximum value of the cost values; is the penalty factor. The final cost of unmanned vehicles can be calculated as follows:
5. The Entropy-Cost Function Evaluation Method
5.1. Weight Distribution of Each Indicator
According to the evaluation indicators for unmanned vehicle intelligent behavior comprehensive evaluation model, the judgment matrix of each level is constructed following the 1~9 scaling methods by expert group composed by research members in related fields. , calculated by formulae (2)–(5), is the weight in accordance with the overall goal layer of 10 evaluation indicators. According to the additivity of information entropy, the weight of typical working conditions’ behavior can be calculated through accumulating affiliated indicators directly as follows.
Typical working conditions’ behavior weight is as follows:
Affiliated indicators weight is as follows:
Then the level of indicators’ weight coefficients is determined, and the evaluation results will be more objective because of different treatments of indicators.
5.2. Comprehensive Score Based on Cost Function
The weight of each indicator in the evaluation system has been calculated in Section 5.1, and the cost combined with weight will be the final value that represents the comprehensive score of unmanned vehicles. In the testing, the paper uses ranking method to define the basic cost value; the real performance will be quantified in the corresponding rank. Taking “Parking precision” as a research object, the cost can be represented as follows:
Since the data are rough, the score can just estimate a general intelligent level of unmanned vehicles; two teams with almost same score must make further comparison about more details. The rest of the indicators’ cost value can be calculated identically, and the first class indicators’ cost is
6. Unmanned Vehicle Intelligent Behavior Evaluation
6.1. Each Indicator’s Score
The score of each indicator is given based on the comparison between optimal performances and minimum acceptable performances from manned driving conditions roughly; the gap will be divided into several score ranks (see Figure 3).
The method for calculating score is a transition from manned driving to driverless. As unmanned vehicles’ development direction is human-like capacity in driving, the method is accordingly beneficial for technical updating.
6.2. Each Indicator’s Cost Value
Also taking the indicator “Path replanning” as the research object, some data in “Future Challenge 2012” competition are shown in Table 2.
Table 2 shows that car F’s data are invalid because of the lane departure in this test. The score is corresponding to the planning time and operating range, optimal condition in manned driving cannot be quantified with 7–9 (s) and 5–7 (m), and minimum acceptable condition is 18–20 (s) and 15–17 (m). The threshold value of score classification is based on human experience, so the data can only be processed roughly to get the score-rank of performance.
Analyzing the data of 10 indicators of team A in the competition, “Parking precision” is corresponding to parking time and the distance between the front of the vehicle and the stop line; “Restart ability,” restart time and acceleration; “Speed capability,” horsepower; “Braking deceleration,” deceleration and braking distance; “Early warning,” lead time and accuracy rate; “Avoidance in right angle,” offset distance and maximum offset angle; “Stimulation,” time gap in gear shift; “Safe distance,” reasonable distance; “Speed optimization,” optimal speed and adjustment time. The indicators’ data and the corresponding score are listed in Table 3.
6.3. Total Cost Value
Calculating the data from Table 3 to get A’s total cost value as described in Section 5.2, the cost values of three typical intelligent behaviors are
The total cost is .
The result shows that team A is at 2nd (20–40) level in the competition.
7. Conclusion
In this paper, the evaluation of unmanned ground vehicles is studied. Based on the typical working conditions of unmanned ground vehicles, a multilevel indicators evaluation system is established. Because the uncertainty is intrinsic property of each evaluation process, information entropy is applied to quantify the weight of each indicator, and each factor matches different weight coefficients to highlight the importance of the evaluation factor. Then entropy-cost function evaluation method is proposed to evaluate team A’s unmanned vehicle in “Future Challenge 2012” competition. From the quantitative results, the teams can learn the vehicle’s intelligent level generally and find their technical shortcomings in some specific indicators; thus the team will get the right development direction.
The first issue to be developed is more detailed quantification of indicators’ score. The technology development is from manned driving to driverless, but manned driving behaviors are also difficult to quantify. Therefore, the cost function should be reformed for more precise cost value through both qualitative and quantitative manned driving empirical data.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This project was supported by the National Natural Science Foundation of China (no. 90920304 and no. 91120010) and the Fundamental Research Foundation of Beijing Institute of Technology (no. 20120342026).