Abstract
In highly automated driving, the driver can engage in a nondriving task but sometimes has to take over control. We argue that current takeover quality measures, such as the maximum longitudinal acceleration, are insufficient because they ignore the criticality of the scenario. This paper proposes a novel method of quantifying how well the driver executed an automation-to-manual takeover by comparing human behaviour to optimised behaviour as computed using a trajectory planner. A human-in-the-loop study was carried out in a high-fidelity 6-DOF driving simulator with 25 participants. The takeover required a lane change to avoid roadworks on the ego-lane while taking other traffic into consideration. Each participant encountered six different takeover scenarios, with a different time budget (5 s, 7 s, or 20 s) and traffic density level (low or medium). Results showed that drivers exhibited a considerably higher longitudinal and lateral acceleration than the optimised behaviour, especially in the short time budget scenarios. In scenarios of medium traffic density, the trajectory planner showed a moderate deceleration to let a vehicle in the left lane pass; many participants, on the other hand, did not decelerate before making a lane change, resulting in a dangerous emergency brake of the left-lane vehicle. In conclusion, our results illustrate the value of assessing human takeover behaviour relative to optimised behaviour. Using the trajectory planner, we showed that human drivers are unable to behave optimally in urgent scenarios and that, in some conditions, a medium deceleration, as opposed to a maximal or minimal deceleration, is optimal.
1. Introduction
Over the past decades, improvements in sensor technology, artificial intelligence, and control systems have led to an increase in vehicle automation. Adaptive cruise control (ACC), a technology that was introduced in the 1990s, has the potential to increase driver comfort by automating the longitudinal control task [1–4]. The recent arrival of assistance systems that can perform lateral control has led to a situation where the driver is provided with the option to detach him- or herself from the control loop of the vehicle [5, 6]. However, in current and near-future levels of automation (SAE levels 2/3), the human still has the role of a fallback agent because automated driving systems have limitations regarding their operational design domain and the handling of unforeseen road situations [7, 8].
It has been shown that being out of the loop can lead to a degradation of situation awareness, mental underload, complacency, and mental overload if the automation reaches a system boundary and the operator has to take back control (e.g., [9]). The fact that the drivers of automated vehicles are susceptible to such detrimental effects, but have to take over control when needed, has sparked a wealth of research on automation-to-driver takeovers. In particular, an important topic concerns the quantification of how well the human driver takes over from the automated driving system.
A variety of takeover quality measures have been proposed in the literature (see [10]; for a review). Radlmayr et al. [11] used the maximum longitudinal acceleration as a takeover quality measure, whereas Zeeb et al. [12] used the maximum lateral acceleration and centerline deviation. In many other studies, takeover quality is quantified using the minimum time to collision (MTTC) [13–16].
The interpretation of these measures has been hampered by the fact that different scenarios impose different demands on the driver. Zhang et al. [17] surveyed 129 takeover studies and found that the time budget and the traffic density are among the most frequently used independent variables. Scenarios can differ from nonurgent, without an immediate need to take over on an empty road [18–20], to highly urgent, where the time budget to collision is 5 s or less or where multiple vehicles drive in the vicinity [21–23]. Urgent scenarios will generally result in “poor” scores on takeover quality measures, such as a low MTTC [24, 25] and high accelerations [26]. Gold et al. [27] stated that, with shorter time budgets, driver reactions are “worse in quality” (p. 1938). The problem with this interpretation of Gold et al. is that, in urgent scenarios, high vehicle acceleration may be desirable or even necessary to avoid an accident and thus reflect “good” rather than “poor” driving behaviour. In summary, the type of scenario is expected to have a large influence on takeover quality measures, but the literature tends to ignore this fact in the interpretation of takeover quality. Takeover quality measures may be uninterpretable if not compared to an optimal reference behaviour.
This study aimed at assessing the merits of using a novel scenario-specific approach for defining takeover quality. We assessed the effects of time budget and traffic density on not only “traditional” takeover scores (e.g., maximal accelerations) but also in relation to the scores obtained from a trajectory planner. More specifically, we took scenario criticality into account by comparing the scores on the measures to a scenario-specific optimised behaviour, obtained via a trajectory planner developed at Volkswagen Group Research, Wolfsburg.
A human-in-the-loop driving simulator study was carried out using different driving scenarios. The independent variables of this study were the time budget and traffic density, as these two variables have been shown to strongly affect takeover quality in previous studies using traditional quality measures [11, 28, 29]. We used 5, 7, and 20 s time budgets. A time budget of 5 s is critical, and a 7 s time budget is the minimum for a safe takeover after being engaged in a secondary task [27, 30, 31]. The 20 s scenarios were included to investigate human takeover behaviour in a case where there is little urgency and the human is free to decide when and how to act.
2. Method
2.1. Participants
Thirty-one Porsche AG employees were recruited to participate in this study. They were required to have a valid driver’s license, as well as normal or corrected-to-normal vision. A total of 6 participants had to be excluded due to incomplete data recordings. Therefore, the analysis was performed on data from 25 participants, of which 13 were female and 12 were male. There were 11, 9, 3, and 1 participants in the 21–30, 31–40, 41–50, and 51–60 age categories, respectively. Two participants indicated having participated in a simulator study regarding automated driving before. The research was approved by the Human Subjects Research Ethics Committee of the TU Delft, and all participants provided written informed consent.
2.2. Apparatus
A hexapod driving simulator at the Porsche Research and Development Facility in Weissach, Germany, was used [32–34]. The hexapod was fitted with a fully functional mock-up of a Porsche Macan. The vehicle dynamics software was based on an all-electric Porsche Taycan. The 6-DOF moving base platform (eMove eM6-640-1800) has an actuator stroke of 640 mm. The motion cueing was according to a classical washout algorithm. The visualised field of view was 180 deg, achieved by projectors displaying 3840 × 2160 pixels on all three sides, as well as the ceiling. The side mirrors were not physically present but integrated into the simulation. The visualization was refreshed at a frequency of 60 Hz. The simulator during one of the takeover scenarios is shown in Figure 1. During the experiment, participants wore Dikablis eye-tracking glasses [35].
2.3. Road Environment
The automated vehicle drove at 130 km/hr on a simulated two-lane motorway. The width of all vehicles, including the ego-vehicle, was 1.78 m. The lane widths were 3.88 m. During the approximately 30 min drive, six takeover scenarios occurred. These takeovers were triggered by roadworks on the right lane. The roadworks were 361 m long. All takeovers took place on a straight stretch of road. There was no hard shoulder on the right.
2.4. Human-Machine Interface
The automation could be activated using a button on the steering wheel. Upon activation, a message was shown on the instrument cluster, indicating that the driver was allowed to take the hands off the steering wheel. Activation was confirmed by the grey steering wheel icon (automation available) turning green (automation active) (see Figure 2).
(a)
(b)
(c)
The participants performed an audiovisual nondriving task in the form of watching a comedy television series (see also [36]). The video automatically started playing on the 10.9-inch centre-display when the automated driving function was enabled.
In each takeover scenario, the participants received a one-stage auditory signal with a fundamental tone of 300 Hz and peaks at 342.5 Hz, 432.5 Hz, 666.7 Hz, and 866.7 Hz as well as a visual warning on the instrument cluster. The steering wheel icon switched to red along with two hands holding it, as shown in Figure 2. Additionally, the centre-display turned black and stated: “Limited functionality while driving manually.”
2.5. Independent Variables
A 3 × 2 within-subject design was used, with time budget (3 levels) and traffic density (2 levels) as independent variables. Six takeover scenarios occurred per participant, comprised of each combination of time budget and traffic density, as shown in Table 1.
The traffic density levels were “low” and “medium,” as defined in the literature as 5 and 10 vehicles per km, respectively [37]. For the medium traffic density scenarios, a left-lane overtaking vehicle, which was driving at a constant speed of 140 km/hr, drove directly behind the ego-vehicle at the time of the takeover request, making an immediate lane change unsafe. In the low traffic density condition, an immediate lane change was possible and safe due to the absence of nearby vehicles at the time of the takeover request.
Figure 3 shows the scenario layout for Scenario 1. Similar figures for Scenarios 2–6 are provided in the Supplementary Materials (Figures S1–S5).
The order in which the 6 scenarios appeared was counterbalanced using a Latin-square method. The automated driving time in between takeovers varied between 3 and 5.5 min, to counteract expectations. The total driving distance was 60.4 km, with a portion of automated driving availability of 50.1 km (23.1 min) per participant.
2.6. Dependent Variables
We computed a variety of takeover measures that have been used in the literature before (e.g., [19, 38, 39]), as shown in Table 2. Four subjective dependent variables were included as well, based on Radlmayr et al. [10]. After each takeover, participants were asked to rate the scenario, on a 7-point scale rating for criticality (very noncritical-very critical), complexity (very uncomplicated-very complicated), discomfort (very comfortable-very uncomfortable), and subjective time budget (more than enough-way too little). Note that these are translations from German; the entire study was conducted in the German language. The effects of time budget and traffic density were assessed using a two-way repeated-measures analysis of variance (ANOVA).
2.7. Procedure
The participants were scheduled into one-hour time slots. After welcoming the participant and thanking him or her for participating, the participant was asked to read an information form. This form explained the capabilities of the highly automated driving system, icons on the dashboard, how to activate the automated driving system, the availability of the nondriving task (television series), the takeover requests, and the four questions that had to be completed on the tablet. The participant was informed that the automation cannot cope with every situation and that their priority should be safety and adherence to the traffic rules. Next, the participant signed a consent form, completed a demographics questionnaire, and was asked to take a seat inside the driving simulator.
While the simulator was still stationary, the experimenter in the passenger seat explained how to activate the automated driving function and how the subjective ratings should be completed on a tablet. Next, the eye-tracking glasses were handed over to the participant and calibrated using DLab behavioural research software (version 3.5). A safety briefing followed. The participant fastened the seatbelt, was informed that opening the car door would stop the dynamic simulator immediately, and was informed that radio communication would always be possible if necessary. A 5 min test drive followed during which the participant got used to the manual controls, automation, display icons, performing the nondriving task, and completing the questionnaires on the tablet. After the test drive, the assistant exited the simulator. Once the moving base platform was reactivated, the main study began.
At the start of the drive, participants engaged the automated driving function while driving in the right lane. There was no speed requirement for activation, although it was advised to activate the automated driving function when driving approximately 130 km/hr, as this was the automated driving system’s target speed. The automated driving function would accelerate or decelerate to the target speed of 130 km/hr. If the automated driving function was engaged, a video automatically started playing on the centre-display. The participant’s gaze was recorded using the eye-tracking cameras. In case the gaze was frequently directed at the road instead of the centre-display, the intercom was used to remind the participant to trust the automated driving system and attend to the video.
Once the automated driving function prompted a takeover request, the driver was required to regain manual control while taking into account any potential left-lane overtaking vehicles. After passing the roadworks, the participant returned to the right lane and reengaged the automated driving system. While the car was driving automatically, the participant completed the subjective rating of the previous takeover scenario. From the simulator control room, it was checked whether the form was received properly. If this was not the case, the participant was asked through the intercom to resubmit the rating form.
Once finished with the rating, the participant continued watching the video displayed on the centre-screen. This order of events was repeated for a total of six times (Table 1). After the sixth takeover scenario, the intercom was used to ask the participant to bring the car to a standstill. The moving base platform was lowered, and the participant exited the simulator. A postdrive questionnaire with regard to simulator fidelity was filled out. In this questionnaire, the participant also had the opportunity to give open feedback regarding the study. The participant was offered something to drink as well as a sweet. Once all potential questions from the participant regarding the study had been answered, the participant was once again thanked.
2.8. Trajectory Planner
A trajectory planner was used to generate reference trajectories for the six scenarios. The look-ahead time of the trajectory planner was changed to correspond with the time budget of each scenario. This created an identical urgency level for the participants in the driving simulator study.
The trajectory planner relied on local motion planning and used a linear reward function that maximised route progress, comfort, and safety. More specifically, the trajectory planner maximised progress to the destination (i.e., the end of the roadworks) and distance to objects, while minimizing acceleration, jerk, wheel angle, and wheel angle changes.
The trajectory planner performs an exhaustive forward search of actions. Accordingly, for every cycle of a model predictive control (MPC), the algorithm yielded a large set of driving policies, which implicitly included multiple behaviours, e.g., lane following, lane changes, swerving, and emergency stops. The final selected driving strategy had the highest reward value while satisfying model-based constraints.
The reward function has an impact on the driving style. The weights used in the reward function were determined by Volkswagen Group Research using a process similar to backtracking, where vehicles were driven manually, and for each time step and corresponding state within the world, the “human” weights were computed. More information about the trajectory planner is provided by Rosbach et al. [41].
The trajectory planner and the human-in-the-loop simulator used the same files describing the simulation environment (openDRIVE extensions .XODR and .XML, built using VIRES Virtual Test Drive (VTD) 2.1 software [42]), including the same simulated vehicles.
The trajectory planner assumed a simple kinematic bicycle model. The vehicle parameters of the trajectory planner were those of a 2012-model Volkswagen Golf. Because the optimised reference trajectory was well within the limits of vehicle dynamics, this deviation from the vehicle model used in the driving simulator (Porsche Taycan) was thought to be unimportant for the goals of the present study.
3. Results
For Scenario 2 (time budget = 5 s and traffic density = low), driving data were missing for two participants due to a data-logging error. Figure 4 shows all 25 participants’ trajectories for all six scenarios. Additionally, it shows the optimised trajectory as driven by the trajectory planner.
In Scenario 1 (time budget = 5 s and traffic density = medium), two participants crashed with the roadworks, and three participants crossed the road boundaries. In Scenario 2 (time budget = 5 s and traffic density = low), zero participants crashed with the roadworks, and three participants crossed the road boundaries. No crashes or road departures occurred in the 7 s and 20 s time budget scenarios.
Figure 5 provides information about the speed pattern during the takeover manoeuvre. In Scenarios 1 and 3 (time budget = 5 or 7 s and traffic density = medium), the planner decelerated to let the fast vehicle on the left lane pass before initiating the lane change. Most participants did the same, although somewhat later. A small number of participants brought their vehicle to a full stop in Scenarios 1–3. Figure 5 further shows that a small number of participants in the medium-density scenarios sped up, presumably in an attempt to change lanes before the arrival of the fast-driving vehicle in the left lane.
3.1. Minimum Time to Collision (MTTC)
The participant’s MTTC values for each scenario are shown in Figure 6, along with the values corresponding to the trajectory planner. In Scenarios 1 and 2, the MTTC was often smaller than 1 s, indicating a highly critical interaction (cf. [43–45]). In the noncritical scenarios (5 and 6), the MTTC showed large individual differences. The trajectory planner changed lane relatively late, resulting in an MTTC value that was lower than the MTTC value of most of the participants. According to a two-way repeated-measures ANOVA, the effects of traffic density and time budget on MTTC were statistically significant (Table 3). The means, standard deviations, and pairwise comparisons per dependent variable are provided in the Supplementary Materials (Tables S1–S53).
3.2. Acceleration Measures
The results for the four acceleration measures are shown in Figure 7. Participants showed similar levels of maximum leftward and rightward lateral accelerations, a symmetric pattern that is characteristic of a normal lane change. The trajectory planner drove considerably more smoothly in lateral and longitudinal directions as compared to the participants. However, for the 5 s and 7 s time budget scenarios with medium traffic density (Scenarios 1 and 3), the trajectory planner’s and participants’ decelerations were relatively similar. This similarity can be explained by the fact that the trajectory planner slowed down to let the approaching vehicle in the left lane pass, as did most of the participants. For Scenario 5 (time budget = 20 s and traffic density = medium), the trajectory planner did not decelerate to let the left-lane vehicle pass, as there was enough time to wait until the left-lane vehicle had passed. Some human drivers, on the other hand, did decelerate for the left-lane vehicle, or they accelerated (see Figure 5) and subsequently had to brake for a leading vehicle.
The two-way repeated-measures ANOVA (Table 3) showed that the effect of traffic density was statistically significant for decelerations and accelerations. The effect of time budget was most pronounced for lateral accelerations; that is, participants steered more abruptly when the time budget was shorter.
3.3. Lane Change Duration
Figure 8 shows the lane change duration for the participants and the trajectory planner. In the 5 s and 7 s scenarios, participants generally changed lanes faster than the trajectory planner. For the 20 s scenarios, the average participant lane change durations were similar to the trajectory planner. The effect of traffic density on lane change duration was not significant, but the effect of time budget was significant (Table 3).
3.4. Minimum or Maximum Scores Are Not Optimal: Illustrating the Value of the Trajectory Planner
Figure 9 shows the participants’ MTTC versus their maximum deceleration in Scenario 3 (time budget = 7 s and traffic density = medium). It can be seen that participants who showed a low maximum deceleration (which may normally be regarded as high takeover quality) and made an early lane change (indicated by high MTTC values) ended up in a highly dangerous situation, as the overtaking vehicle in the left lane had to initiate an emergency stop. The trajectory planner showed an optimised deceleration of around 4 m/s2, just sufficient to let the vehicle in the left lane pass. These findings indicate that a low maximum deceleration and high MTTC are undesirable in this scenario.
3.5. Subjective Ratings
The questionnaire results are shown in Figure 10. Generally, the experimental manipulations had the desired effect, with the urgent scenarios (time budget = 5 s) with medium traffic density being perceived as most critical, uncomfortable, and complex. The nonurgent scenario (time budget = 20 s) with low traffic density, on the other hand, was regarded as least critical, most comfortable, and least complex. The effects of traffic density and time budget were significant for all four questions (Table 3).
3.6. Driver Response Times
Results showed that humans needed about 0.9 s for attending to the road, about 2.5 s to attend to the rear-view mirror, and about 2 s to grab/touch the steering wheel (Figure 11). The trajectory planner, on the other hand, was not susceptible to these delays in human information processing. Table 3 shows that the effects of time budget and traffic density were not significant for the eyes-on-road time. This finding is sensible because only after the driver attends to the road, he or she can assess the time budget and traffic density. There was a significant effect of time budget on eyes-in-side-mirror times and hands-on-wheel times; participants took somewhat more time when the time budget was higher.
4. Discussion
In this study, we assessed human takeover quality for different time budgets and traffic density levels and compared the results to an optimised behaviour obtained via a trajectory planner. The acceleration measures showed that the trajectory planner drove extremely smoothly as compared to the human participants. The participants showed especially large lateral accelerations in the more time-critical scenarios, which may have been because participants were startled by the upcoming roadworks. The trajectory planner “knew” everything from the moment of the takeover request onward, whereas humans needed time to assess the situation, as indicated by the fact that the average eyes-on-road time was about 0.9 s. Another reason for the high accelerations is that participants may not have noticed these high accelerations due to the inherent physical limitations of the motion-base simulator [46, 47]. Several participants stated after the experiment that the simulator motion did not feel realistic. In the 20 s time budget scenarios, on the other hand, participants’ median lane change durations were close to the lane change durations of the optimised trajectory. In summary, the results indicate that the differences between a human takeover manoeuvre and that of the trajectory planner are large in time-critical scenarios. That is, humans seem unable to behave optimally when temporal demands are high.
A driving simulator study by Petermeijer et al. [19, 36] showed that drivers become quicker and more fluent in taking over control with increasing takeover experience. Payre et al. [48] recommended that drivers should be trained in how to operate their automated driving system. Accordingly, it may be argued that it is unreasonable to compare untrained drivers with the results of a path planner. However, it is noted that the differences between human behaviour and optimised behaviour were generally very large (see, e.g., Figure 7). It is unlikely that more experienced/trained drivers would perform on par with the trajectory planner in the urgent scenarios.
The 20 s time budgets were included to examine human driving behaviour in nonurgent takeover scenarios in relation to an optimised trajectory. The participants exhibited large individual differences, which can be attributed to personal preferences regarding when to make a lane change. These findings are in line with a meta-analysis by Zhang et al. [17] which concluded that participants do not take over control as quickly as possible, but may delay their steering or braking response according to their discretion. The acceleration measures and lane change duration of the trajectory planner in the 20 s condition were similar to those of the more time-critical 5 and 7 s time budget scenarios. In other words, the trajectory planner tended to finish the lane change manoeuvre relatively quickly, rather than decrease lateral accelerations with larger time budgets.
Figure 9 illustrates the fact that one should not rely on a single measure (peak deceleration) as an index of takeover quality. In the urgent scenarios of medium traffic density, the trajectory planner gently decelerated and let the vehicle in the left lane pass before executing a smooth lane change. Here, the trajectory planner adopted a deceleration of 4 m/s2, which was sufficient to let the vehicle in the left lane pass while retaining high comfort and forward progress. The human participants, on the other hand, showed large individual differences. Some participants did not decelerate at all and made an immediate lane change, as a result of which the vehicle in the left lane had to initiate an emergency brake; other participants initiated an emergency deceleration themselves in order to prevent colliding with the roadworks. In summary, it was shown that a low deceleration and an early lane change to avoid collision with the roadworks are not desirable in this scenario. What is desirable is to brake with an intermediate deceleration level, a type of behaviour not identified using traditional takeover measures, but captured by the trajectory planner.
The optimised trajectory was computed offline. Theoretically, this calculation could be performed in real time as the trajectory planner at Volkswagen Group Research is designed for real-time application as an integral part of an automated driving system. We do not claim that the trajectories generated by the trajectory planner are optimal. Instead, the trajectories are optimised for a specific set of variables, such as route progress, accelerations, and object proximity. Somewhat different conclusions may be obtained if selecting different weights for the trajectory planner. For example, it would be possible to assign a lower weight to acceleration and increase the weight for progress, leading the algorithm to drive with a more aggressive driving style.
In conclusion, this study showed benefits of assessing takeover quality in comparison with a reference behaviour. Humans deviate substantially from optimised acceleration values if the scenario is urgent. In other words, urgent scenarios are dangerous not only because of the impending collision (a physical factor) but also because humans have difficulty behaving optimally in cases of time pressure (a psychobehavioural factor). Additionally, we illustrated that takeover measures should not be used in isolation. Maximizing driver comfort (i.e., minimizing the deceleration) was found to be highly dangerous in scenarios of medium traffic density. The comparison with the trajectory planner showed that intermediate deceleration values are most desirable.
Recently, frameworks have been proposed that combine various measures into a single takeover quality score. The Takeover Controllability Rating (TOC) quantifies takeover quality through a coding sheet, which experts use to grade human takeover behaviour by inspection of video footage [49]. Similarly, the Takeover Performance Score (TOPS) combines three variables related to vehicle guidance, mental processing, and subjective rating. These three variables can be combined into a single quality score [10]. These approaches represent an improvement to traditional methods, but they rely on subjective judgments. The present trajectory-planning approach can contribute to a more objective quantification of takeover quality.
Data Availability
Raw data and scripts are available online at https://doi.org/10.4121/uuid:64ff102c-a2a7-4008-9fb1-088534c7a957.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
Supplementary Materials
The supplementary materials contain (1) the road layout for Scenarios 2–6, (2) the trajectory planner values for its relevant dependent variables, (3) statistical data for the dependent variables, and (4) raw data at the level of individual participants. (Supplementary Materials)