An Improved Transformer-Based Neural Machine Translation Strategy: Interacting-Head Attention

Table 6

Training time on IWSLT16 DE-EN training dataset.

Model	Number of heads
	2	4	8	16	32	64
Multihead attention	10 m 10 s	10 m 30 s	10 m 55 s	11 m 30 s	12 m 45 s	15 m 30 s
Multihead attention (head size = n)	9 m 20 s	9 m 40 s	10 m 10 s	11 m 05 s	12 m 55 s	16 m 55 s
Talking-head attention	10 m 30 s	10 m 40 s	11 m	11 m 40 s	15 m 50 s
Interacting-head attention	10 m 30 s	11 m 30 s	14 m 30 s	24 m 30 s	—	—

Note. The units of m and s stand for minute and second, respectively.