An Improved Transformer-Based Neural Machine Translation Strategy: Interacting-Head Attention

Table 7

Training time on WMT17 EN-DE training dataset.

Model	Number of heads/head size
	2/256	4/128	8/64	16/32	32/16	64/8
Multihead attention	9 h 55 m	11 h	11 h 30 m	12 h	12 h 30 m	14 h
Multihead attention (head size = n)	11 h 30 m	11 h 40 m	11 h 50 m	12 h 50 m	13 h 10 m	17 h 30 m
Talking-head attention	11 h 5 m	11 h 40 m	11 h 50 m	12 h 20 m	13 h 50 m	15 h
Interacting-head attention	10 h 40 m	11 h 30 m	12 h 50 m	20 h	—	—

Note. The units of h, m, and s stand for hour, minute, and second, respectively.