Abstract
Touchscreen interfaces are widely used in modern technology, from mobile devices to in-car infotainment systems. However, touchscreens impose significant visual workload demands on the user which have safety implications for use in cars. Previous studies indicate that the application of haptic feedback can improve both performance of and affective response to user interfaces. This paper reports on and extends the findings of a 2009 study conducted to evaluate the effects of different combinations of touchscreen visual, audible, and haptic feedback on driving and task performance, affective response, and subjective workload; the initial findings of which were originally published in (M. J. Pitts et al., 2009). A total of 48 non-expert users completed the study. A dual-task approach was applied, using the Lane Change Test as the driving task and realistic automotive use case touchscreen tasks. Results indicated that, while feedback type had no effect on driving or task performance, preference was expressed for multimodal feedback over visual alone. Issues relating to workload and cross-modal interaction were also identified.
1. Introduction
The touchscreen interface is synonymous with ubiquitous computing, being found in an ever-widening array of devices. This is due in part to the ease-of-use of the interface, with co-location of the input and display; and an interaction mode familiar to even novice users [1]. Having become established as the de facto standard interface for today’s multi-function smartphones [2], the emergence of the tablet computer has led to further entrenchment of the technology in the consumer market [3]. Touchscreens are also widely used in cars, where the flexibility of the interface also allows designers to create cleaner cockpit layouts free from the clutter of multiple pushbutton controls.
It is the direct nature of touchscreen interaction however that poses the largest challenge to automotive Human Machine Interface (HMI) designers. As visual attention must be directed to the touchscreen during use, the interface imposes significant levels of visual workload upon the user; over 70% of the time taken to complete an in-vehicle touchscreen task can be spent looking away from the road [4]. This has implications for safety: accident risk is correlated to both the duration and frequency of glances away from the forward roadway [5], and large-scale studies have found that up to 60% of crashes, near-crashes, and incidents can be attributed to visual distraction from the primary driving task [6]. This problem is exacerbated the lack of tactile and kinaesthetic feedback [7] that would normally be provided by a traditional mechanical control such as a pushbutton or dial.
This paper reports on the findings of a study into the effects of multimodal touchscreen feedback in an automotive context, conducted in 2009 and originally reported in [8]. The initial findings from the earlier publication are extended with a revised analysis of the subjective data and the addition of objective measures of task performance and driving behaviour, along with discussion of contemporary studies.
2. Touchscreens and Multimodal Interaction
The multiple resource theory model of workload [9] states that each sensory channel has a discrete and finite level of processing capacity, beyond which performance is degraded. By diverting some of the workload demands of HMI interaction from the visual to other senses, the overall level of distraction may be reduced. Given that touchscreen interaction employs vision and touch, the opportunity exists to improve performance through exploitation of the haptic channel.
Multimodal feedback has been the subject of numerous research studies. A review of 43 such studies [10] indicated that combining visual feedback with auditory or tactile feedback led to reduced reaction times and improved performance measures, although error rates were not improved. Studies on the effects of haptic feedback in handheld touchscreen devices indicated improvements in subjective workload [11] and reduced task completion times [12]; in both of these cases, improvements were also observed in error rates.
There have also been a number of relevant studies relating to automotive touchscreen use. Lee and Spence [13] used an automotive-themed attention task to assess dual-task performance using a touchscreen telephone secondary task. Their findings indicated that reaction times and task completion times were both improved when combined visual, audible, and haptic feedback was applied; this was complemented by a reduction in reported subjective workload. Serafin et al. [14] conducted on-bench and static vehicle evaluations of an automotive user interface on a touchscreen equipped with haptic feedback. Results showed that users showed strong preferences for trimodal feedback over visual alone; however, no evaluations were conducted under dynamic driving conditions.
Richter et al. [15] discussed the evaluation of a haptic touchscreen interface in an automotive scenario, using the Lane Change Test to create the driving environment. Their findings indicate improved error rates and task completion times with haptic feedback enabled. The authors’ own follow-up study [4] evaluated the effects of haptic feedback on the performance of an abstracted touchscreen task in an immersive simulated driving environment. Results showed reduced task completion time along with subjective preference for combined visual and haptic feedback.
In general, the studies described above provide evidence of the potential benefits, both objective and subjective, of haptic feedback to improve interaction with in-vehicle technology.
3. The Lane Change Test
The context in which a product is evaluated can have a significant influence on the user’s perception of usability [16]. It is therefore important to replicate the context of use when collecting user-derived data [17]. When evaluating automotive HMI technologies, this requires the use of an environment which represents the cognitive, visual, and physical workloads of the primary driving task. The use of driving simulators has become more popular in recent years as the cost of hardware has been reduced and capability increased [18], offering advantages over instrumented vehicle studies in terms of safety, cost, repeatability, and ease of data collection [19–21]. The term “driving simulator” however covers a wide range of systems with varying technical complexity. Hardware configurations can range from desktop-PC-based solutions with a single screen and a gaming controller through to fully immersive solutions utilising panoramic projections, real vehicle cabins, and full motion platforms. Clearly, the implications of cost and complexity vary accordingly, with an outlay of a few thousand pounds in the former case and several million in the latter.
The Lane Change Test (LCT) is a software-based approach to providing a standardised, low-cost method for the evaluation of in-vehicle technology. LCT was originally developed as part of the Advanced Driver Attention Metrics (ADAM) project investigating next-generation in-car user interfaces [22]. LCT provides a simple simulated driving task which represents the manual, visual, and cognitive demands of real-world driving. The driving task is performed in parallel with secondary user interface tasks, and variations in driver response are recorded and used to calculate quantitative measures of performance.
LCT is designed to facilitate the evaluation of any type of in-vehicle technology, both OEM and aftermarket, and to allow for all types of sensory interaction. It is, however, limited to domestic vehicle applications due to the vehicle dynamics model and assumptions of driver position. Driving inputs are made through a standard PC gaming wheel and pedals; while the LCT method can be used using a simple bench-top setup or in driving simulators with varying degrees of fidelity, lane change trajectories are improved with a driving simulator setup compared to a desktop computer [23]. LCT is the subject of ISO standard 26022 : 2010 [24], which outlines the experimental conditions and methodology required to conduct an LCT-based evaluation study, thus removing much of the variability often apparent in approaches to driving simulator studies [18].
Validation studies of LCT have indicated strong correlations in the measured effects of secondary task performance to both high-fidelity driving simulators [25] and real-world driving scenarios [22]. Recent research has concentrated on the development and validation of new metrics for use with LCT, adding extra measures of the effect of secondary tasks on aspects of driver performance [26].
LCT has been shown to be effective in identifying degradation in driving performance in the presence of secondary tasks [27, 28]. However, earlier studies using LCT highlighted differences in results, possibly attributable to variations in the application of the method between studies [28]. It is worthy of note that these earlier studies were conducted prior to the official publication of ISO 26022 : 2010 which specifies the experimental approach to the use of LCT.
4. Study Outline
It is clear that the potential exists for the performance of automotive touchscreen interfaces to be improved through the addition of haptic feedback. A study was therefore designed to investigate the effects of visual, audible, and haptic feedback on objective task performance and user response to touchscreen interaction in an automotive use scenario. The aim of the study was to evaluate the objective and subjective benefits of haptic feedback relative to the visual and auditory modalities commonly employed on touchscreen interfaces, when delivered in different combinations of feedback stimuli. The study sought to obtain data relevant to the real-world benefits of multimodal feedback through the use of participants who are all vehicle owners and have experience of touchscreen technology, along with realistic automotive touchscreen use cases.
4.1. Research Questions and Hypotheses
The research questions for the study were as follows.(i)Does the addition of haptic feedback affect driving performance when operating in-car HMI?(ii)Does the addition of haptic feedback affect task performance when operating in-car HMI?(iii)Does the addition of haptic feedback improve the user experience?(iv)Does the addition of haptic feedback make in-car technology easier to use?
Based on the benefits of haptic feedback described in Section 2, hypotheses were formed that (a) the addition of haptic feedback would improve driving and task performance, (b) the addition of haptic feedback would improve users’ affective response to the touchscreen interface, and (c) the addition of haptic feedback would reduce subjective workload.
5. Methodology
The following chapter describes the methodology used to test the hypotheses described above, outlining the experimental approach and evaluation setup.
5.1. Experiment Design
The study used a dual-task approach with users completing realistic automotive use case tasks (described in Section 5.5) while engaged in a driving task based on LCT. A within-subjects experiment design was employed, with feedback type as the independent variable. Four levels of feedback were employed: visual only (V), audible + visual (AV), haptic + visual (HV), and audible + haptic + visual (AHV). All participants experienced all four levels of feedback, and the study was counterbalanced for feedback presentation order, with the design perfectly balanced for multiples of 24 participants.
5.2. Participants
A total of 54 people were recruited with 48 completing the evaluation; three respondents were withdrawn from the study after exhibiting an adverse reaction to the simulated driving environment, two exhibiting poor driving performance, and a further one experiencing issues with the touchscreen tasks. All participants had at least one year experience of driving in the UK and also had experience of in-car touchscreen use. The demographic breakdown is given in Table 1.
5.3. Experiment Setup
The evaluation utilised the Lane Change Test software to create a simulated driving environment, as described above. To enhance the physical validity (fidelity) of the evaluation environment, a simple dash buck was employed, consisting of an aluminium frame supporting a vehicle instrument panel and centre console supplied by Jaguar Land Rover. A Logitech G25 gaming wheel was attached to the frame in the correct position, with the touchscreen mounted in the centre console in an approximation of its in-vehicle position. The participant was seated in front of the buck in a vehicle seat, thus replicating the ergonomic conditions of a real vehicle. The buck was situated in front of a rear-projected screen which displayed the driving simulation, as shown in Figure 1.
5.4. Evaluation Interface
The touchscreen hardware used for the study was an 8.4” TouchSense haptic touchscreen demonstrator unit from Immersion Corporation (Immersion Corporation: http://www.immersion.com/), which served as both the input surface and visual and haptic display. This consists of a resistive touchscreen fitted with Immersion’s proprietary electromechanical haptic feedback actuators and controller. The unit imparts haptic sensations to the user’s finger through a lateral displacement of the screen surface and is capable of reproducing a range of haptic effects; to maintain consistency a single effect was used throughout the study, selected on the basis of a preliminary study described in Section 5.6.
Given that context has a significant effect on the perception of usability of technology [16], the touchscreen task was also designed to provide an authentic user experience. This was achieved by using a direct replication of the graphical user interface from the Jaguar XF premium saloon car, as shown in Figure 2. The interface was programmed to perform logging of the task completion data, recording the start and end times of each touchscreen task. These values were then used to compute the task completion times reported in Section 6.2.
Graphics were supplied by Jaguar Land Rover in Adobe Flash format; the interface logic was then recoded in Actionscript 3.0 to incorporate the experimental controls and enable capture of task completion data. Visual feedback was provided by a change in colour of the button when pressed. Haptic feedback was produced on press and release of the button. The audible stimulus was the acknowledgement tone used on the Jaguar XF touchscreen interface, with a fundamental frequency of approximately 1 kHz and a duration of around 70 ms. This was delivered over headphones, providing a degree of acoustic isolation from the sound of the haptic touchscreen actuators.
5.5. Touchscreen Tasks
Touchscreen tasks were based on real life use case scenarios experienced in production vehicles and implemented on the interface described above. The selected use cases represented a range of functionality across the system, requiring different amounts of menu level navigation and button presses to complete the task. These are summarised in Table 2.
A mixed-task approach was adopted whereby participants completed all eight tasks during each drive, thus proving an aggregate measure of driving performance for each feedback condition. This approach overcomes difficulties of determining relative task workload for tasks of mixed duration [27], as used here. Tasks were modified between trials where possible to avoid repetition; for example, by requesting a different radio frequency or climate setting, without altering the required number of button presses. The order of presentation of use cases was varied for each feedback state, again to reduce potential learning effects.
5.6. Preliminary Study
It was important to ensure that the haptic effect used for the evaluation did not provoke a negative reaction from the participants, as this may bias opinions of the haptic touchscreen technology. A simple preliminary study was therefore conducted in order to ascertain which of the available preprogrammed haptic effects was most preferred by users. 34 participants were recruited, all within the automotive industry, with 50% experts in the design of touchscreen interfaces.
The study used a custom interface programmed in Adobe Flash to test preference for haptic feedback effects. Participants were presented with 20 different haptic effects in sets of 5, where each set was from one of the preprogrammed “click” effect groups: all of these stimuli were impulsive and designed to provide a haptic sensation reminiscent of a mechanical push switch but are differentiated by their duration and frequency characteristics.
Stimulus presentation order was randomised to avoid bias. Participants selected their most preferred effect from each group and were then asked to select an overall preference from the four preferred effects previously selected. Participants were asked to operate the touchscreen with their left hand as would be the case in a right-hand-drive car and wore ear defenders to avoid cross-modal interaction from the sound produced by the haptic touchscreen actuators.
While no overall preference for a specific haptic effect was found, one group of effects, “Crisp Click”, was more strongly preferred, with 16 of 34 users selecting effects from this group (Figure 3). A binomial test of this result showed statistical significance (, one-tailed). The effect used in the main study was therefore chosen from this group and consisted of an impulsive stimulus with overall duration of approximately 80 ms and fundamental frequency of 130 Hz, illustrated in Figure 4.
5.7. Driving Task
The LCT driving scenario involves driving down a straight, three-lane roadway at a limited constant speed of 60 km/h. Signs are positioned at regular intervals along the roadway which indicate the correct lane that the driver should occupy. At the appearance of each set of signs the driver is required to make a lane change manoeuvre such as to occupy the indicated lane. A screenshot of the LCT scenario is shown in Figure 5 [29].
The simulated track length is 3000 m, corresponding to 3 minutes of driving at a constant 60 km/h. Within this duration, 18 pairs of signs are displayed, corresponding to a lane change manoeuvre every 9 seconds.
The driver’s trajectory through the lane changes is compared to a normative model, the idealised path though along the track, which is identical for each participant. Deviations from the normative path are calculated using the LCT analysis tool software to provide a measure of mean deviation (MDev) for each drive. MDev values for the dual-task conditions are compared to those from a baseline measure, where the participant is engaged in the primary driving task only. Figure 6 shows an example of deviation from the normative path, an output from the LCT analysis tool [24].
5.8. Training Procedure
A five-stage training procedure was employed. Firstly, participants completed a demographic questionnaire about their experience of driving and touchscreen device use, along with a consent form. Participants were informed of the details of the study and also made aware of the potential symptoms of simulator sickness. It was made clear to participants that they were free to withdraw at any time and should stop immediately if they began to feel unwell.
The second phase of the training involved introducing the participants to the touchscreen and the different types of feedback that they would be experiencing. This was achieved using the simple interface shown in Figure 7, which consisted of four buttons, each of which produced one of the four combinations of feedback defined in Section 5.1. Audible feedback was delivered over headphones in order to provide acoustic isolation from the sound of the touchscreen’s haptic feedback actuators. As participants were non-expert users, the term “touch feedback” was used to refer to haptic feedback throughout the study.
The third element of the training involved familiarisation with the touchscreen evaluation interface. Each of the tasks involved in the study was demonstrated by the experimenter and repeated by the user until they were confident in using the interface. Once this stage had been reached participants were introduced to the driving task. Practice drives without secondary tasks were conducted until the participant was confident in performing the driving task; due to its simple nature, this stage was generally reached within a short period of time. This phase also served to provide baseline driving data for the single task condition, which was recorded once the participant was comfortable with the driving task. The final stage of the training was to include the secondary task in the practice drive; this was conducted using visual feedback only, with tasks selected at random by the experimenter.
5.9. Data Sources
Variation in driving performance was monitored using the mean deviation (MDev) parameter from LCT, as described in Section 5.7. This is calculated as an overall value for each drive (hence each feedback condition). Touchscreen task performance was measured by task completion time, from initiation of the task to its successful completion.
Subjective measures of users’ experience of the touchscreen interface were collected via questionnaire after each drive. A range of parameters were measured, including hedonic rating, confidence in button press, difficulty of the touchscreen task, interference with the driving task, and, when haptic feedback was enabled, the strength and realism of the haptic stimulus. Most measures employed a 9-point rating scale with semantic anchors at the end- and mid-points; hedonic rating was measured using the 9-point hedonic rating scale which features semantic anchors at every point [30].
An additional follow-up questionnaire was administered at the end of the study, which featured a most/least preferred feedback condition choice and two further questions to determine opinions on the effects of haptic feedback on pleasure and ease of use, using 5-point Likert scales.
5.10. Analysis
Driving and touchscreen task performance data was analysed using within-subjects ANOVA, with post hoc pairwise comparisons made using Tukey’s HSD test. Subjective data was analysed using the nonparametric Friedman’s test and post hoc paired Wilcoxon signed rank tests using the normal approximation. Statistical significance is determined at α = 0.05 (two-tailed) in all cases.
6. Results
Results from the objective and subjective measures employed are detailed below. For all figures error bars indicate the 95% confidence interval.
6.1. Driving Performance
Driving performance was measured by the mean deviation from the normative path: the average amount that the driver moved from the modelled path through the LCT scenario. One participant displayed difficulties with the driving task, resulting in consistently high mean and standard deviation values for lateral deviation which were well outside the expected range; data from this participant was therefore rejected. Mean values for each feedback state were then filtered for outliers then subjected to a one-way, within-subjects ANOVA, the output of which is shown in Figure 8. Outliers were identified based on the interquartile range method described by Tukey [29] and defined as occurring outside the range where is the median of the data set and and the 25th and 75th percentiles. 3 data points were removed, corresponding to 1.3% of the remaining data. Baseline driving performance was included in the analysis as a fifth level of the independent variable.
The difference due to the independent variable was shown to be statistically significant (, ). A Post hoc Tukey HSD test, corrected for familywise type I error as per Cicchetti [31] indicates that differences exist only between the baseline condition and other means (q(5,184) = 3.90, ). This indicates that, while there was no significant difference in driving performance due to feedback state, mean deviation increased from 1.03 m to 1.33 m when touchscreen tasks were introduced, an increase of 29%.
6.2. Task Performance
Mean task completion time (TCT) for each task under each experimental condition was calculated and data filtered for outliers as described above. A two-way, within-subjects ANOVA analysis was then performed, with feedback type and task as factors. The results are shown in Figure 9, with tasks ranked by task completion time and plotted against the required button presses for each task.
Mean TCT ranged from 5.00 seconds (climate control, AHV) to 21.64 seconds (CD task, HV). Task time roughly follows the number of button presses required to complete the task. No significant differences were observed between the feedback states (); however differences due to task were significant (, ). A Post hoc Tukey HSD test applied to mean TCT for each task across all feedback states indicates that the phone dialling and CD tasks were significantly different to all other tasks at the level, with overall mean TCT of 19.35 and 19.62 seconds, respectively (, ). The climate control task required the lowest task completion time of 7.27 seconds; this was significantly different to the phone, CD, FM, and fan speed tasks.
6.3. Subjective Data
The subjective response scores from the questionnaires were collated, filtered for outliers, and subjected to statistical analysis using the nonparametric Friedman’s test at the level. Post hoc tests of pairwise significance were conducted using the paired Wilcoxon signed rank test with a Šidák correction [32] maintaining familywise significance at . The outcomes from the analyses are detailed below.
6.3.1. Hedonic Rating
Figure 10 shows the mean hedonic rating for each feedback state. While there is a statistically significant increase in rating score with multimodal feedback (, ), post hoc tests indicate that introducing haptic feedback alone does not improve performance over visual feedback. However, audible + visual (, ) and audible + haptic + visual (, ) do offer a significant improvement over visual alone. Combined audible, haptic, and visual feedback also attracted a significantly higher rating than haptic + visual feedback (, ).
6.3.2. Confidence Rating
A similar pattern is observed for confidence rating, as shown in Figure 11. Differences across feedback states were again shown to be significant (χ2(3) = 32.97, ), with post hoc tests indicating significant improvements from visual feedback alone with the addition of audible (, ) and combined audible and haptic feedback (, ).
6.3.3. Touchscreen Task Difficulty
Figure 12 shows the rating scores for touchscreen task difficulty. The difference in rating score across feedback types is significant (, ), albeit smaller in magnitude than observed previously, with an increase of less than one scale point. Post hoc tests indicate significant differences between the “visual” and “audible + visual” conditions (, ) and between the “visual” and “audible + haptic + visual” conditions (, ). Note that the touchscreen task was rated as “more than moderately difficult” for all feedback conditions.
6.3.4. Driving Interference
The scores for driving task interference again follow a similar pattern (Figure 13). Difference across feedback conditions was found to be significant (, ), with improvements seen over visual feedback with the addition of auditory (, ) and auditory + haptic feedback (, ). Again, mean ratings for all conditions indicate a “more than moderate” level of interference with the driving task.
6.3.5. Haptic Feedback Perception
The subsequent two questions deal with user perception of the haptic effect itself. A Wilcoxon signed-rank test performed on mean ratings of the perceived strength of the haptic feedback stimulus showed significant differences with and without audible feedback (, ), indicating that the magnitude of the haptic effect was perceived to be higher in the presence of audible feedback (see Figure 14(a)), thus suggesting a multimodal interaction between the two stimuli. Without audible feedback, the mean haptic feedback strength rating of 3.51 indicates that participants perceived the effect to be weaker than optimal.
(a) Strength of haptic feedback
(b) Realism of haptic feedback
Users were also asked to rate the similarity of the feel of the touchscreen haptic feedback to that of a real switch. Again, a significantly higher rating was achieved when both audible and haptic feedback were present (, ; Figure 14(b)).
6.3.6. User Experience
Participants were asked if each of the feedback types presented enhanced the user experience. As with previous measures, a trend was seen indicating preference for enhanced feedback, with combined audible, haptic, and visual feedback attracted the highest rating, a mean of 4.04 corresponding to “agree” on the Likert scale. Differences across feedback types were significant (χ2(3) = 38.13, ), with significant pairwise differences seen between the visual/audible + visual (, ), visual/audible + haptic + visual (, ), and haptic + visual/audible + haptic + visual (, ) pairs. These results are shown in Figure 15.
Additional questions included in the follow-up questionnaire aimed to understand preference for feedback combination and the effect of haptic feedback specifically on the user experience. Figure 16 shows the most/least preferred choices for each feedback type, with “most preferred” choices shown as positive and “least preferred” as negative. Clearly, visual only feedback was strongly least favoured, with combined audible, haptic, and visual feedback most preferred, with 27 out of 48 choices. Participants were also asked to rate their agreement with the statements “Touch feedback makes the touchscreen easier to use” and “Touch feedback makes the touchscreen more pleasurable to use,” using 5-point Likert scales. The mean scores of 4.27 and 4.07 ( and 0.74, resp.) indicate that, on average, participants agreed with these statements.
6.3.7. Order Effects
Subjective rating data from the hedonic, confidence, and interference ratings were reanalysed to examine potential effects of presentation order. Results indicate that, while presentation order had no significant effect on hedonic rating (, ) or touchscreen task difficulty (, ), there was a significant effect on interference with the driving task over the duration of the study (, ), with the level of interference becoming lower as the study progressed.
7. Discussion
Results clearly show that lateral control performance is not significantly affected by the type of feedback presented. However, mean deviation increased significantly relative to a baseline when touchscreen tasks were introduced, highlighting the increase in workload experienced due to the inclusion of the secondary task. The application of multimodal feedback was not able to mitigate this increase. This concurs with the findings of Rydstrӧm et al. [33], who measured lateral deviation while engaged in haptic and visual secondary tasks, again using the LCT method.
Task performance also showed no significant differences due to feedback type, but there were large differences in task completion time across the different tasks. This is to be expected given the varying levels of task complexity; indeed, the pattern of task completion time roughly followed the minimum number of button presses required to complete the task. The debate into what tasks should be allowed while a vehicle is in motion is beyond the scope of this paper; however, the duration of most intensive task used, telephone dialling, sits approximately on the limit of the “15-Second Rule” for acceptable static task completion times defined under SAE J2364 [34].
The apparent lack of objective effect of multimodal feedback is contrary to findings in the literature which indicate that the addition of haptic stimuli to existing feedback sources will have benefits in terms of task performance and error rates. Given the specific nature of the sensory and workload demands imposed by the driving task, it is logical to consider these results in reference to studies that feature similar experimental conditions, that is, the evaluation of multisensory feedback using a touchscreen-based secondary task in a simulated automotive driving scenario.
In one such study, Lee and Spence [13] evaluated reaction time to driving and touchscreen tasks with the same combinations of feedback used in this study. Results indicated that a significant difference in reaction time existed between the visual only and trimodal (audible, tactile, and visual) combinations; as with this study, there was no significant difference between the visual only and bimodal visual + tactile condition. The touchscreen task required interaction with a telephone keypad, as opposed to the menu-based tasks employed in this study. A significant effect of bimodal (audible + tactile) feedback intensity on reaction times was also noted, indicating a workload-related perceptual threshold effect.
Richter et al. [15] conducted a study to evaluate a haptic touchscreen implementation, using the LCT to provide the driving context. Their secondary touchscreen task was also based on a telephone keypad. Improvements were shown in number entry error rates for small (but not large) buttons with the addition of haptic feedback, while the effect on task completion time was inconclusive. It should be noted that this study featured a small sample size (5 respondents) and no measures of significance were reported.
While not in total agreement, the differences to the findings of the above are not sufficient to invalidate this study. The influence of task complexity and perceived feedback magnitude on users’ subjective responses is discussed below.
7.1. Effects of Workload and Cross-Modal Interaction on Perception of Haptic Feedback Magnitude
The results from the rating of haptic feedback strength indicate that haptic feedback alone was not sufficiently strong to provide positive confirmation to the user. The effect employed in the study was chosen on the basis of the findings of the preliminary study described in Section 5.6, with the intention that this process would reject inappropriate effects, including those which were of insufficient magnitude. While all of the participants indicated that they could feel the haptic feedback in the training phase, a number reported difficulty in sensing the stimulus during the tasks. This suggests that the workload demands of the concurrent driving and touchscreen tasks reduced participants’ ability to detect the haptic stimulus; this agrees with the findings of Leung et al. [35] who observed a reduction in haptic sensitivity when participants were under cognitive workload.
The effects of cross-model interaction between tactile and auditory stimuli are well established [36–38], with significant links between the neural mechanisms for processing tactile and auditory information. It is therefore to be expected that the perception of the haptic stimulus will be effected by the presence of its auditory counterpart. Indeed, in this study, haptic feedback was perceived to be stronger when audible feedback was also present. Interestingly, the haptic feedback stimulus was also perceived to be more realistic (more like a real switch) in the presence of the audible stimulus; even though the latter was not representative of the sound made by a switch (a “beep” rather than a “click”), its presence supported the mental model of switch use during users’ interaction with the touchscreen.
7.2. The LCT and Workload
Measures of perceived touchscreen task difficulty and interference with the driving task also showed improvements with enhanced feedback, albeit with smaller differences. For both measures, scores indicated a “more than moderate” level of overall difficulty/interference. This suggests that the combined performance of the driving and touchscreen tasks placed significant workload demands on the participants. While the perceived level of interference was reduced in the presence of multimodal feedback, objective driving performance as measured by mean deviation was not improved.
Analysis of order effects indicated that, while presentation order had no effect on touchscreen task difficulty, there was a significant effect on interference with the driving task, with the level of interference becoming lower as the study progressed. As the perceived difficulty of the touchscreen task was constant throughout (no significant order effect), it is assumed that the perceived demands of concurrent performance of the driving and touchscreen tasks diminished as the study progressed.
The immediate implication is that participants did not receive sufficient training prior to commencing the evaluation. Petzoldt et al. [39] found that training, especially in the dual-task condition had a significant effect on mean deviation in an LCT-based study. As described in Section 5.8, training/practice was provided on the driving and touchscreen tasks both individually and concurrently. Clearly there is a limit to the extent to which participants can be trained within the practical constraints of the study, and Petzoldt et al. again identify that there are no guidelines within the ISO standard for training requirements for LCT. Furthermore, as the experiment design was counterbalanced, any order bias present should not introduce bias to the independent variable in this study.
It may be suggested that the use of tasks based on a real vehicle interface imposed a high level of cognitive and visual workload on the user, by compounding the demands of interacting with the interface and recalling the correct sequence from memory. Given that the purpose of the study was to evaluate the effectiveness of the feedback mechanism rather than the graphical user interface, an alternative, less demanding task may provide clearer results at the expense of physical validity. A subsequent study conducted by the authors [4] used an abstracted “search and select” touchscreen task requiring participants to locate and press a target button within a array. This task featured a lower training threshold, and it was found that subjective workload, when measured using the same scales applied in this study, was lower and order effects nonsignificant.
The nature of the workload experienced should also be considered. The LCT driving task is, in essence, very straightforward: there is no other traffic on the roadway and no requirement to moderate speed. Cognitive and physical demands must therefore be assumed to be low; by implication the operation of LCT requires significant visual attention. The use of an objective visual workload measure, such as eyes-off-the-road time, or a diagnostic subjective workload measure such as NASA-TLX [40] would provide evidence to support this hypothesis.
7.3. Haptic Feedback and User Experience
Hedonic rating scores indicate a preference for trimodal feedback, showing a trend across feedback states which is repeated for ratings of confidence. While combined audible, haptic, and visual feedback attracts the highest mean scores, no significant differences are shown for the addition of haptic feedback to other feedback states. This concurs with the findings of Serafin et al. [14], which indicated preference for “enhanced” (multimodal) feedback over visual alone, with trimodal feedback attracting the highest rating scores.
The trend in improved performance with multimodal feedback seen in the hedonic rating scores was repeated in the user experience questions. Participants “agreed” that trimodal audible, haptic, and visual feedback enhanced the user experience. Combined visual, audible, and haptic feedback was chosen as “most preferred” by 27 of 48 respondents. This, along with results indicating that users “agree” that haptic feedback makes the touchscreen both easier and more pleasurable to use indicate strong user acceptance of the technology.
The provision of feedback is an essential part of the user experience, informing the user of the state of the system under use and confirming that their expected outcomes have occurred. A failure to provide relevant and clear feedback can frustrate the user and negatively affect their perception of the usability of the system [41]. The results discussed above suggest that the benefits of the application of haptic feedback were, within the context of this study, subjective and affective. There is therefore potential to explore the technology solely in terms of its contribution to the user experience. Studies into the feel quality of mechanical switchgear [42, 43] have shown that vehicle users can differentiate between and express a preference for specific haptic characteristics. A system such as that tested which offers the potential for haptic feedback to be tailored to a user’s specific requirements would provide an opportunity to optimise affective response to interaction with the HMI; this would also allow users to select the stimulus magnitude required for their personal preference and tactile acuity. In the increasingly competitive automotive industry, features that delight users improve the emotional response to the product and help to provide differentiation in a crowded marketplace [44].
8. Conclusions
The findings of the study showed subjective benefits for multimodal feedback in automotive touchscreens but were unable to provide evidence of objective benefits in terms of driving or task performance. Feedback type was found to have no effect on lateral deviation or touchscreen task-completion time. In the latter case the time required to complete the task was, understandably, dominated by task complexity in terms of button presses and menu level navigations required. Driving performance was, however, degraded from a baseline measure when touchscreen tasks were introduced, highlighting the distraction caused by operation of in-car technology.
Measures of subjective user response indicate a preference for multimodal feedback over visual alone, with hedonic rating, confidence, difficulty, and driving interference all showing an improvement and combined visual, audible, and haptic feedback being most strongly preferred. While no specific benefit was demonstrated for haptic feedback over audible, users indicated that haptic feedback improved the usability and user experience of the touchscreen interface. Prior research indicates that haptic interaction can be used to improve users’ affective response to interfaces, suggesting that haptic touchscreen technology offers vehicle manufactures an opportunity to exploit this potential in the context of in-car technology.
The study also highlighted issues with dual-task workload and training when conducting HMI evaluation studies using the Lane Change Test. While LCT is designed to provide a measure of task workload through decrements in driving performance, it is difficult to diagnose the exact nature of the workload experienced by the user with the measures employed. The application of an objective measure of visual workload and/or a diagnostic subjective workload measure may help to clarify the findings from future studies in this respect.
This work could be extended by evaluating the performance of expert users in the dual-task scenario to understand the effect of training and learning effects throughout the study. Further evaluations could be conducted using measures of visual workload, as described above. Finally, correlation of the LCT-based results with repeat evaluations using an instrumented vehicle or high-fidelity simulation environment would help to determine the external validity of the approach.
Acknowledgments
The authors would like to thank Dr. Carl Pickering and colleagues at Jaguar Land Rover Research for their technical support and input into this study. This research was conducted as part of the Warwick Innovative Manufacturing Research Centre (WIMRC) Project: “Designing the Next-Generation of HMI for Future Intelligent Vehicles” under the Premium Vehicles Customer Interface Technologies (PVCIT) Centre of Excellence. The PVCIT is a collaborative research project between leading automotive companies and research partners, including Jaguar Land Rover and Tata Motors European Technical Centre. The £4.7 million project is funded by the Advantage West Midlands (AWM) and the European Regional Development Fund (ERDF).