Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature

Table 3

Speech interface performance statistics from selected bench-top studies.

System	CHAT [4]	CHAT [5]	CU Communicator [16]	CU Move [6]	SENECA [9]	VOIC [11]	Volkswagen [12]
Tasks	(1) NAV; (2) Restaurant finder (RF)	(1) MP3; (2) Restaurant finder (RF)	Phone for travel plan	NAV	(1) NAV; (2) Phone dialing; (3) Address book	(1) NAV; (2) Current time; (3) Tourism; (4) Fuel; (5) Car manual; (6) Hotel reservation; (7) Traffic information	(1) NAV; (2) Map control; (3) Phone
Completion time (s)			260.3		Average: 63	(1) 73; (2) 15; (3) 146; (4) 78; (5) 57; (6) 180; (7) 53
Completion rate	98%	MP3: 98%; RF: 94%	73.6%		Average: 79%
Turns^1	2.3^1	RF: 4.1^1	User: 19; Machine: 19
Word recognition Accuracy^2 (%)	NAV: 85.5%; RF: 85%	MP3: 90%; RF: 85%
Word error rate (%)			26%	30–65%
User satisfaction rating^3	1.98	MP3: 2.24; RF: 2.04

1Turn is defined as one user utterance to the system during a dialog exchange between the user and the system while attempting to perform the task.
²Word recognition accuracy (WA).
    WA=100(1−(Ws+Wi+Wd)/𝑊)%.
    𝑊: total number of words in reference.
   Ws: number of reference words which were substituted in output.
   Wi: number of reference words which were inserted in output.
   Wd: number of reference words which were deleted in output.
³User satisfaction rating: 1 = strong agreement; 5 = strong disagreement.