About this Journal Submit a Manuscript Table of Contents
Advances in Mechanical Engineering
Volume 2014 (2014), Article ID 681326, 13 pages
http://dx.doi.org/10.1155/2014/681326
Research Article

Test and Evaluation of Autonomous Ground Vehicles

1School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China
2School of Mechanical Engineering, Hebei University of Engineering, Handan, Hebei 056038, China

Received 16 September 2013; Revised 10 December 2013; Accepted 25 December 2013; Published 23 January 2014

Academic Editor: Chamaillard Yann

Copyright © 2014 Yang Sun et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

A preestablished test and evaluation system will benefit the development of autonomous ground vehicles. This paper proposes a design method for a scientific and comprehensive test and evaluation system for autonomous ground vehicles competitions. It can better guide and regulate the development of China’s autonomous ground vehicles. The test and evaluation system includes the test contents, the test environment, the test methods, and the evaluation methods. Using a hierarchical design approach, the test content is designed to be stage by stage, moving from simplicity to complexity and from individual modules to the entire vehicle. The hierarchical test environment is established according to the levels of test content. The test method based on multilevel platforms and sensors is put forward to ensure the accuracy of test results. A fuzzy comprehensive evaluation method combined with analytic hierarchy process (AHP) is used for the comprehensive evaluation which can quantitatively evaluate the individual module and the overall technical performance of autonomous ground vehicles. The proposed test and evaluation system has been successfully applied to real autonomous ground vehicle competitions.

1. Introduction

Since the 1980s, autonomous driving has become a fast-developing and promising area. The relevant evaluation methods and test items involved the actual application environment, such as VaMP [1], ARGO [2], NavLab [3], and Demo [4]. With the development of individual technology and integrated systems, the test and assessment methods of autonomous ground vehicles were developed from a single test to a complex capability test. Urmson et al. described the tests for evaluating and comparing navigational skills of autonomous ground vehicles [5]. The tests include blind path tracking test, perception assisted path tracking test, and perception planning test.

Krotkov et al. proposed third-party test experiments for autonomous ground vehicles [6]. They proposed to conduct the experiments by a group independent of the developers. The unrehearsed testing experiments provide little prior knowledge of the test courses. This makes the evaluation experimental tests not demonstration. They described the detailed information of the test environment and the test content. However, they did not discuss the evaluation approach in detail.

Third-party test method is used in many autonomous ground vehicle competitions, such as the Grand Challenge in 2004 and 2005 and the Urban Challenge in 2007 [711]. An extremely talented and dedicated team of DARPA staff and contractors put an enormous amount of work into preparing comprehensive rules and test procedures. Teams have to undergo a series of tests to determine the ability of the systems to autonomously navigate and avoid obstacles, as well as thorough inspections to ensure that they meet safety and performance requirements [12].

None of the autonomous ground vehicles finished the course in the first DARPA Grand Challenge competition. However, five vehicles successfully completed the race in the second competition. The third-party test played an important role and formulated the related technical indicators to guide the development of autonomous ground vehicles. However, the evaluation method used in the DARPA Challenge mainly cares about whether a vehicle completes the task and the time required to complete the task. It does not focus on the quality of the work. For example, Wei and Dolan indicated that some teams in 2007 DARPA Urban Challenge preferred to avoid difficult maneuvers in high-density traffic by stopping and waiting for a clear opening instead of interacting with it [13].

In China, the first third-party test for the autonomous ground vehicle competition named Future Challenge (FC) was held in 2009 [14]. The competition pushed autonomous ground vehicles to go out from laboratories into real-world environments [15]. Up to now, five Future Challenge competitions have been held in China. Of these five competitions, we took part in the design of the test and evaluation system four times. We analyzed and compared evaluation methods over the past competitions; AHP was used in [16, 17]. The AHP is a widely used multicriteria method. It has firm theoretical underpinnings and has been used successfully to help people make better decisions in a wide variety of complex circumstances [18, 19].

In summary of the previous design experiences, in this paper, we propose to build a complete test system that includes the test contents, the test environment, and the test methods to meet the demands of testing for autonomous ground vehicles. The fuzzy comprehensive evaluation method combined with AHP which was used in our previous work [20] is further introduced to solve fuzzy and hard-to-quantify problems. The proposed test and evaluation system is concerned more about how the autonomous ground vehicles complete the task. It can quantitatively evaluate the overall technical performance and the individual technical performance of autonomous ground vehicles. Specifically, we describe the design of the third competition (FC’2011) to demonstrate the proposed test and evaluation architecture for autonomous ground vehicles. The proposed test and evaluation system can also be used by other researchers and autonomous ground vehicle performance testers with a few adjustments.

2. Design of the Test Environment

Environment perception, perception of the state of the vehicle, behavior decision making, and vehicle control are the key technologies and abilities of autonomous ground vehicles. These key technologies and abilities of autonomous ground vehicles are hard to test directly. The behaviors of autonomous ground vehicles reflect the key technologies and abilities; therefore, the behaviors of autonomous ground vehicles should be tested to evaluate their key technologies and abilities. It is an important means for realizing the scientific and rational assessment of key technologies of autonomous ground vehicles.

To ensure that the test meets requirements such as repeatability and safety, the test environment has to be designed scientifically. By analyzing the relationship between the test environment and intelligent behaviors of autonomous ground vehicles, the test environment model can be established in accordance with the definition and classification of the test environment elements [16]. The design method of the test environment is shown in Figure 1. The intelligent behaviors of autonomous ground vehicles are closely related to the environment. Based on the analysis of the key technologies and abilities of autonomous ground vehicles, we have defined the environment elements and their compositions which include road, traffic rules, obstacles, and auditory elements. We combine the difficulty of technical analysis and combination of elements to establish the model of test environment. And then, we can design the test environment according to the test requirements.

681326.fig.001
Figure 1: Design of the test environment.
2.1. Establishment of Test Environment Model

Whether the test environment can simulate the complex and real environment is crucial to check the key research and technology results of autonomous ground vehicles. Through the analysis of the relationship between environmental factors, key technologies of autonomous ground vehicles, and intelligent behavior ability, the influential environmental factors can be classified and the basic elements and composition can be defined. The basic elements are divided into the following factors: road environment, traffic rule, acoustic environment, lighting conditions, and working conditions. By combining various environmental factors and building all kinds of real roads, the environment can be studied and the test environment model can be formed to simulate the complex, real environment. The scheme for establishing a theoretical model of the test environment is shown in Figure 2.

681326.fig.002
Figure 2: Test environment model.
2.2. Hierarchical Test Environment Design

The multilevel test environment needs to be designed according to the hierarchical test contents. On the analysis of the design requirements of the test environment, combining different targets, and focusing on the intelligent behavior level, the matching environmental factors can be chosen. Based on the combination of different complexity elements, the various road environments can be set up.

The design of the test environment for autonomous ground vehicles can be divided into environment design and environment implementation. The environment design includes the compilation of test plan/instruction/report, software, hardware, and support. In the design phase, according to the test content the related environmental elements can be selected to design the combination of environment elements. The implementation environment includes field, software, hardware, and support that carry out all kinds of intelligent behavior tests of autonomous ground vehicles. Through a combination of real environmental factors, the implementation environment can simulate the actual traffic environment of urban roads, rural roads, and highways [21].

3. Design of the Test Content

The test content needs to meet different requirements from basic performance to advanced performance and from individual performance to overall performance, because the level of intelligence of autonomous ground vehicles is developed in the similar order. During the validation of key technologies like visual and auditory perception, the relevant autonomous ground vehicles technologies should be tested. These contents can be used to assess the research and development level of the key technologies. The overall test and evaluation system can then be used to guide the development direction of visual and auditory information cognitive calculation.

Autonomous ground vehicles can be divided into the following modules: intelligent environment perception module, intelligent behavior decision-making module, state perception module, and control module. Based on the analysis of development trends of relevant autonomous ground vehicle technology, the test contents can be designed in five layers according to the natural environment perception and intelligent behavior decision making. Figure 3 shows the test contents of these five layers.

681326.fig.003
Figure 3: Hierarchical test content.

4. The Test Method

4.1. Test Technology Based on Multiplatform and Multisensor

The test platform should be designed without limitation on the development means and interference with the driving and task execution of the autonomous ground vehicle being tested. Figure 4 shows the test platform.

681326.fig.004
Figure 4: Autonomous ground vehicle test platform.

The multisensor information fusion method is used to assess the status of the tested vehicles and to test and evaluate the intelligent behaviors of autonomous ground vehicles.

4.2. Real-Time Image Monitoring and Display System

To provide data support for result evaluation, the confirmation of results, and on-site viewing for all participating teams and experts, the remote real-time image monitoring and relay system were established to monitor the whole competition in real time the autonomous ground vehicles in the third “Future Challenge.” Figure 5 shows the real-time image monitoring and display system [22].

681326.fig.005
Figure 5: Real-time image monitoring and display system.

The core technology of the real-time wireless monitoring system is wireless image transmission. COFDM (coded orthogonal frequency division multiplexing) is the most advanced modulation technology. It can break through sight limit, and it makes full use of radio spectrum resources. Therefore, it has good immunity to noise and interference.

The main technical advantages of COFDM are as follows. (1) Suitable for the non-line-of-sight and blocked city environment, the application equipment has the excellent abilities of out-of-sight, diffraction, and penetration. It can realize the stable transmission of video with the high probabilities in the non-line-of-sight and blocked environments of city, mountain, and building. The impact on environment is small. (2) Suitable for wireless transmission of real-time images in high-speed mobile. (3) Suitable for high-speed transmission, high bandwidth, high code stream, and high-quality audio and video. (4) Excellent antijamming performance. (5) High utilization rate of frequency and channel.

5. The Evaluation Method

Wei and Dolan indicated that some teams used a conservative approach in 2007 DARPA Urban Challenge [13]. They mainly cared about whether their vehicles completed the task and the time of completing the task not the quality of driving. This evaluation method is not helpful for long-term development of autonomous ground vehicles because it may lead to the team using some primitive approaches to complete the task. Therefore, not only the time to complete the task needs to be considered, but also the quality of completion should be considered in the evaluation.

We proposed a cost function-oriented quantitative evaluation method in [17]. This cost function-oriented quantitative evaluation can quantitatively evaluate the overall technical performance and individual technical performance of autonomous ground vehicles; however, the value of the cost function is determined subjectively. In this paper, we further adopt a fuzzy comprehensive evaluation method combined with AHP (fuzzy-AHP) [20]. The fuzzy comprehensive evaluation method is an application of membership degree instead of the cost function. This method can solve fuzzy and hard-to-quantify problems [23, 24].

5.1. Establishing the Evaluation Index System

Evaluating autonomous ground vehicles is a problem of multilevel comprehensive evaluation. It should be divided into different levels of evaluation based on the complexity of environment perception and intelligent behavioral decision making of autonomous ground vehicles. The evaluation index system includes evaluation aspects, evaluation elements, and evaluation factors in this work. The evaluation aspect involves basic intelligent behavior and advanced intelligent behavior. The evaluation element contains five parts: VCB (vehicle control behavior), BDB (basic driving behavior), BTB (basic traffic behavior), ADB (advanced driving behavior), and ATB (advanced traffic behavior). The evaluation factor consists of many subprime indexes, shown in Figure 6.

681326.fig.006
Figure 6: Evaluation index system of autonomous ground vehicles.
5.2. The Fuzzy-AHP Evaluation for a Single Factor

(1) Establishing the Evaluation Factor Set : where is the number of the evaluation factors.

(2) Establishing the Evaluation Grade Set The same number of evaluation grade set elements can be used by various factors, where is the number of the evaluation grade. Evaluation grade is determined by the fuzzy membership grades. We apply the fuzzy statistical method to get the fuzzy membership grades.

(3) Establishing the Fuzzy Matrix . The fuzzy matrix of the th evaluation element in the th evaluation aspect is represented by :

(4) Establishing the Weight Coefficient Matrix. The factor of each level has a different importance level in the evaluation index system. The AHP is therefore used to get a reasonable weight distribution of the evaluation system [20].

The judgment matrix of the upper factor for the lower relevant factor can be given through the relative importance comparison of evaluation factors in accordance with the definition table of importance [25]. The decision makers compare the degree of importance between the two factors shown in Table 1.

tab1
Table 1: Comparison between factors.

A consistency check must be carried out after obtaining the biggest characteristic root of the matrix to ensure the consistency of different factors.

Consistency index C.I. is expressed as If it meets the following condition, then the results from the comparison matrix can be accepted: where C.R. is the consistency index.

The weight coefficient matrix of the evaluation factor is where is the weight value of the th evaluation factor in the th evaluation element which belongs to the th evaluation aspect and meets :

(5) Single Element Comprehensive Evaluation Matrix. The comprehensive evaluation of the th element in the th aspect can be expressed as where .

The quantitative results can be used to better reflect the real situation because of the impact assessment in computing the results of the various factors and their importance in the evaluation are considered.

5.3. The Fuzzy-AHP Evaluation of All Aspects

The fuzzy evaluation results   of each single factor are taken together to form a higher-level evaluation matrix . Using the same method to get the comprehensive evaluation result of the th element combined with and the weight coefficient matrix , form a higher level of matrix and finally get the comprehensive evaluation matrix ; this matrix is a comprehensive evaluation result.

The weight coefficient matrix of the evaluation element is where is the weight value of the th evaluation element in the th evaluation aspect and meets : The weight coefficient matrix of the evaluation aspect is where is the weight value of the th evaluation aspect and meets :

Evaluation of the results of is not only taking into account all factors, but also considering all the information at all levels of evaluation.

The comprehensive evaluation result can be expressed with a total score. The evaluation criteria membership grade set is , and then the specific score of the comprehensive evaluation result can be calculated. Finally, the object can be evaluated according to the score. Higher score indicates a higher intelligence level of autonomous ground vehicles:

6. Test and Evaluation System for FC’2011

6.1. Design of the Test Environment

The test environment of the second competition of autonomous ground vehicles (FC’2010) is shown in Figure 7. It was held on the campus of Chang’ an University. Some of the road lane markings are not clear. In addition, the use of GPS was not allowed. Just relying on radars and cameras, no vehicle completed the task, although the whole route is only 3.6 km.

681326.fig.007
Figure 7: Test course in FC’2010.

In view of this situation, a new design scheme was proposed for the third competition (FC’2011). The competition should focus on testing visual and auditory cognition not the dynamics of the vehicle. The competition should be designed in realistic environments, both structured and unstructured. GPS should not be forbidden. But the test task should be designed such that vehicles can not finish the race if they only depend on GPS.

According to the design scheme, the test environment was selected and designed in a real urban environment in Ordos city, 800 km from Beijing. The whole route is 10 km and is shown in Figure 8.

681326.fig.008
Figure 8: Layout of testing points.
6.2. Design of the Test Content of FC’2011

The test contents were designed in terms of the above analysis and factual requirements including going straight, avoiding static obstacles, identifying traffic lights and turning right, turning left, U-Turn, intersection activity, interaction with other vehicles, and identifying stop signs and stopping. The layout of eight testing points is shown in Figure 8.

In the third competition, two test points were designed to test the ability of avoiding obstacles for autonomous ground vehicles. Figure 9(a) shows the first test point where two vehicles are stopped in the two lanes in the same way. Figure 9(b) shows the second test point where many cones were placed to block all the four lanes in the same way. The autonomous ground vehicle should be able to identify the obstacles ahead, carry out the lane changing behavior, and drive according to the traffic rules.

fig9
Figure 9: Sketch of static obstacles.

Figure 10 shows the test point for intersection activity. When the autonomous ground vehicle is going straight to pass through the intersection with a green traffic light, the human-driven vehicle (yellow car in Figure 10) turns right into the driving lane of the autonomous ground vehicle. Autonomous ground vehicles should be able to identify the traffic condition and make a correct decision.

681326.fig.0010
Figure 10: Intersection activity.
6.3. Implementation of Real-Time Image Monitoring and Display System

The third competition environment is shown in Figure 11 (the blue closed loop).

681326.fig.0011
Figure 11: Competition environment.

In order to realize the transmission of wireless images over such a large area, a relay was added. It was installed on the hillside shown in Figure 12(a). The relay is composed of a receiver and transmitter, as shown in Figure 12(b). The receiver is used to receive the car video of wireless transmission. The transmitter is used to wirelessly transfer the video to the specified terminal. Because the direction of the terminal is known, a directional antenna is used, as shown in Figure 12(c).

fig12
Figure 12: Relay part.

At the terminal, only a receiver and directional antenna need to be installed, as shown in Figure 13. The vehicle with the transmitter drives along the blue line shown in Figure 11. At the same time, the real-time video is observed at the terminal. The test result shows that the signal is good except in the red areas shown in Figure 11.

fig13
Figure 13: Receiver in the known terminal.

To make the signal smoother in the red areas shown in Figure 11, two directional antennas are installed at the relay station on the top of a mountain, as shown in Figure 14(a). Two upright antennas are used to receive the wireless images from two moving vehicles. In order to make the signals better and images smoother, two directional antennas are used and an augmenter is added for the receiver. The video from the two cars is integrated on the one screen, shown in Figure 14(b).

fig14
Figure 14: Improvement and result.

The proposed system continuously worked for eight hours to transfer the video collected by two cars moving in the large environment to the terminal. The experiment shows that the system has the features of small time delay, convenient operation, and high image quality.

6.4. Evaluation of FC’2011

The competition contents include traffic sign recognition, obstacle avoidance, merging into traffic, U-turn, intersection, and lane changing (see Figures 15, 16, 17, and 18). The environment perception and intelligent decision making of autonomous ground vehicles were comprehensively tested.

681326.fig.0015
Figure 15: Static obstacle avoidance.
681326.fig.0016
Figure 16: Dynamic obstacle avoidance.
681326.fig.0017
Figure 17: Through the intersection.
681326.fig.0018
Figure 18: U-turn.

With a team statistic data of Team B (see Table 2), the proposed fuzzy-AHP evaluation method is used.

tab2
Table 2: Performance of Team B in FC’2011.

The evaluation aspects, elements, and factors are assigned different weight coefficients. The evaluation grade can be set to the same number levels: The evaluation process of autonomous ground vehicles is evaluated from the low stage to the higher level.

6.4.1. The Evaluation Process of the “Vehicle Control Behavior” Element in the “Basic Intelligent Behavior” Evaluation Aspect for Team B Is Described as Follows

(1) Establishing the “vehicle control behavior” evaluation factor set :

(2) Establishing the fuzzy evaluation matrix of the “vehicle control behavior” evaluation element.

Each team was evaluated by 10 experts in the field of autonomous ground vehicles based on the quality of completing the task. And then the fuzzy matrix is expressed by the membership grade using the following method combined with the expert investigation: = The number of very good/10; = The number of good/10; = The number of so-so/10; = The number of bad/10; = The number of very bad/10.

From the data in Table 2, the fuzzy evaluation matrix of Team B was

(3) Establishing the weight coefficient matrix of the “vehicle control behavior” evaluation element. The AHP pairwise comparison process is shown in Figure 19.

681326.fig.0019
Figure 19: The AHP pairwise comparison process.

The evaluators estimate the parking to be two times more important than the starting in this decision. If the evaluators feel that they are unfairly judged, they have a veto right and further discussion is then needed. In our case, there was a high consensus (consistency ratio = 0 in Figure 19) and the veto right was not used.

The weight coefficient matrix is For security reasons, the parking has higher importance compared to starting. The weights indicate that the parking has the most influence in VCB (vehicle control behavior).

(4) Calculating the comprehensive evaluation matrix of the “vehicle control behavior” evaluation element: This result was the comprehensive evaluation result of VCB (vehicle control behavior) evaluation element. The comprehensive evaluation results of BDB (basic driving behavior) and BTB (basic traffic behavior) evaluation element can be gotten using the same method:

6.4.2. The Scores of Five Evaluation Elements

(1) If the comprehensive evaluation result is expressed by scores, the membership grade set of the evaluation criterion was

(2) The scores (, , , , and ) of five evaluation elements (vehicle control behavior, basic driving behavior, basic traffic behavior, advanced driving behavior, and advanced traffic behavior) of Team B were

6.4.3. The Evaluation Process of the “Basic Intelligent Behavior” Evaluation Aspect for Team B Is Described As Follows

(1) Synthesizing the fuzzy evaluation of each evaluation element.

The evaluation results ( and ) of three elements form the fuzzy matrix of  “basic intelligent behavior” aspect which includes VCB, BDB, and BTB:

(2) The weight coefficient matrix of the three evaluation elements in the “basic intelligent behavior” aspect was

(3) So the comprehensive evaluation result of the “basic intelligent behavior” aspect can be gotten: Similarly, the comprehensive evaluation result   of the “advanced intelligent behavior” aspect was

6.4.4. The Comprehensive Evaluation Score of Team B

(1) From the data in Table 2, the weight coefficient matrix is

(2) The fuzzy evaluation matrix of Team B was

(3) Calculating the comprehensive evaluation matrix of Team B,

(4) The comprehensive evaluation score of Team B was The competition results of FC’2011 are shown in Table 3.

tab3
Table 3: Competition results in FC’2011.

From the scores of the evaluation elements of Team B, we can see the intelligence levels of vehicle control behavior, basic driving behavior, and advanced driving behavior are high enough to meet the competition requirement, but the intelligence levels of basic traffic behavior and advanced traffic behavior are low.

Table 3 shows the total scores and rank of Team A, Team B, and Team C. The total score is the quantitatively evaluated result. The higher the total score, the better the overall technical performance of autonomous ground vehicles. The intelligence levels of Team A and Team B are high enough to meet the competition requirement because of the relatively high total score. However, the intelligence levels of basic traffic behavior and advanced traffic behavior are low. Therefore, they have to find the problems and deficiencies in these aspects and improve corresponding technologies. The experiment results show that the proposed fuzzy-AHP method can quantitatively evaluate the overall technical performance and individual technical performance of autonomous ground vehicles.

It is noted that the evaluation results are dependent on the weight coefficients. The weight coefficients are calculated by the AHP method based on the importance of the evaluation index in this competition. Different weight coefficients may lead to different evaluation results. In practice, it is suitable for the real application. In each competition, five evaluation elements perhaps have different importance. For example, in the first competition, we much more focused on VCB (vehicle control behavior) and BDB (basic driving behavior). With the development of technologies, ADB (advanced driving behavior) and ATB (advanced traffic behavior) will be emphasized.

7. Discussion and Conclusion

In this paper, a design method for establishing the test environment for autonomous ground vehicles is proposed according to the definition and classification of the test environment elements. The hierarchical test content of autonomous ground vehicles ranged from simplicity to complexity and from local performance to overall performance. This multiplatform, multisensory test method is put forward to ensure the accuracy of test results. A remote real-time image monitoring and relay system for autonomous ground vehicles is established to provide the data support for the result evaluation, the confirmation of results, and on-site viewing for all participating teams and experts. The evaluation system is built using a hierarchical evaluation concept. The fuzzy comprehensive evaluation method combined with AHP is used for the comprehensive evaluation, which can quantitatively evaluate the individual module and the overall technical performance of autonomous ground vehicles. The fuzzy-AHP evaluation method guides the autonomous ground vehicles not only to complete the task, but also to complete the task in high standard and high quality.

Compared with the other test systems for autonomous ground vehicles, the proposed system has some strength. First, this system is more complete which includes the test contents, the test environment, the test methods, and the evaluation method. Second, it can quantitatively evaluate the individual module and the overall technical performance of autonomous ground vehicles. Third, it is much more concerned about the quality of autonomous ground vehicles to complete the task. This system can help to evaluate scientifically the key technologies and abilities of autonomous ground vehicles.

The described methodology is most suitable for testing autonomous ground vehicles. However, it can also be used by other researchers and autonomous ground vehicles performance testers with a few adjustments. For example, on the analysis of the test requirements, combining the different target, the matching environmental factors can be chosen. Based on the combination of different complexity elements and compositions, the various environments can be set up. Moreover, the corresponding test content, test method, and evaluation method can be adopted according to the described process in this paper.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgment

This research was supported by the National Natural Science Foundation of China (Grant nos. 90920304 and 91120010).

References

  1. M. Bertozzi, A. Broggi, and A. Fascioli, “Vision-based intelligent vehicles: state of the art and perspectives,” Robotics and Autonomous Systems, vol. 32, no. 1, pp. 1–16, 2000. View at Publisher · View at Google Scholar · View at Scopus
  2. M. Bertozzi and A. Broggi, “GOLD: a parallel real-time stereo vision system for generic obstacle and lane detection,” IEEE Transactions on Image Processing, vol. 7, no. 1, pp. 62–81, 1998. View at Publisher · View at Google Scholar · View at Scopus
  3. C. Thorpe, R. Craig Coulter, M. Hebert, et al., “Smart cars: the CMU navlab,” in Proceedings of World MED93, 1993.
  4. C. M. Shoemaker and J. A. Bomstein, “Overview of the Demo III UGV program,” pp. 202–211, Proceedings of the Part of the SPIE Conference on Robotic and Semi-Robotic Ground vehicle Technology, 1998.
  5. C. Urmson, C. Baker, J. Dolan et al., “Autonomous driving in traffic: boss and the urban challenge,” AI Magazine, vol. 30, no. 1, pp. 17–29, 2009. View at Scopus
  6. E. Krotkov, S. Fish, L. Jackel, B. McBride, M. Perschbacher, and J. Pippine, “The DARPA PerceptOR evaluation experiments,” Autonomous Robots, vol. 22, no. 1, pp. 19–35, 2007. View at Publisher · View at Google Scholar · View at Scopus
  7. http://www.darpa.mil/grandchallenge/index.asp.
  8. C. Mark, E. Magnus, J. P. How, and M. R. Murray, “Autonomous driving in urban environments: approaches, lessons and challenges,” Philosophical Transactions of the Royal Society A, vol. 368, no. 1928, pp. 1471–2962, 2010.
  9. C. Thorpe, R. Aufrere, J. Carlson et al., “Safe robot driving,” in Proceedings of the International Conference on Machine Automation (ICMA '02), September 2002.
  10. J. Leonard, J. How, S. Teller et al., “A perception-driven autonomous urban vehicle,” Journal of Field Robotics, vol. 25, no. 10, pp. 727–774, 2008. View at Publisher · View at Google Scholar · View at Scopus
  11. C. Urmson, J. Anhalt, D. Bagnell et al., “Autonomous driving in urban environments: boss and the urban challenge,” Journal of Field Robotics, vol. 25, no. 8, pp. 425–466, 2008. View at Publisher · View at Google Scholar · View at Scopus
  12. A. Tether, The DARPA Grand Challenge Commemorative Program, 2004.
  13. J. Wei and J. M. Dolan, “A robust autonomous freeway driving algorithm,” in Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 1015–1020, Shaanxi, China, June 2009. View at Publisher · View at Google Scholar · View at Scopus
  14. http://ccvai.xjtu.edu.cn/news.do?method=listmessage&type=1.
  15. G. Xiong, P. Zhou, S. Zhou et al., “Autonomous driving of intelligent vehicle BIT in 2009 future challenge of China,” in Proceedings of the IEEE Intelligent Vehicles Symposium (IV '10), pp. 1049–1053, June 2010. View at Publisher · View at Google Scholar · View at Scopus
  16. G. Xiong, X. Zhao, H. Liu et al., “Research on the quantitative evaluation system for unmanned ground vehicles,” in Proceedings of the IEEE Intelligent Vehicles Symposium (IV '10), pp. 523–527, San Diego, Calif, USA, June 2010. View at Publisher · View at Google Scholar · View at Scopus
  17. Y. Sun, G. Xiong, H. Chen, S. Wu, J. Gong, and Y. Jiang, “A cost function-oriented quantitative evaluation method for unmanned ground vehicles,” Advanced Materials Research, vol. 301–303, pp. 701–706, 2011. View at Publisher · View at Google Scholar · View at Scopus
  18. Y. X. Feng, Y.-C. Gao, X. Song, and J.-R. Tan, “Equilibrium design based on design thinking solving: an integrated multicriteria decision-making methodology,” Advances in Mechanical Engineering, vol. 2013, Article ID 125291, 8 pages, 2013. View at Publisher · View at Google Scholar
  19. O. S. Vaidya and S. Kumar, “Analytic hierarchy process: an overview of applications,” European Journal of Operational Research, vol. 169, no. 1, pp. 1–29, 2006. View at Publisher · View at Google Scholar · View at Scopus
  20. Y. Sun, G. Tao, G. Xiong, et al., “The Fuzzy-AHP evaluation method for unmanned ground vehicles,” Applied Mathematics & Information Sciences, vol. 7, no. 2, pp. 653–658, 2013.
  21. X. Li, G. Xiong, Y. Sun et al., “Design on hierarchical testing system for unmanned ground vehicles,” Advanced Materials Research, vol. 346, pp. 817–822, 2012. View at Publisher · View at Google Scholar · View at Scopus
  22. X. Wang, G. Xiong, Y. Sun, et al., “Development and implementation of real-time wireless monitor system using COFDM for the competition of intelligent vehicles in China,” in Proceedings of the International Conference on Communications and Information Processing (ICCP '12), vol. 288 of Communications in Computer and Information Science, pp. 591–598, 2012.
  23. A. Piros and Z. Farkas, “Fuzzy-based evaluation of a specific drive train,” Advances in Mechanical Engineering, vol. 2012, Article ID 763171, 7 pages, 2012. View at Publisher · View at Google Scholar
  24. K. Q. He, Y. Jia, B. Wang, R. Wang, and H. Luo, “Comprehensive fuzzy evaluation model and evaluation of the karst collapse susceptibility in Zaozhuang Region, China,” Natural Hazards, vol. 68, no. 2, pp. 613–629, 2013.
  25. T. I. Saaty, The Analytic Hierarchy Process, McGraw Hill, 1980.