An Improved Transformer-Based Neural Machine Translation Strategy: Interacting-Head Attention

Table 4

Overall evaluation scores using four attention-based models on the WMT17 EN-DE evaluation set and test set.

Model	WMT17 subset	Number of heads/head size
		2/256	4/128	8/64	16/32	32/16	64/8
(a)
Multihead attention	dev	8.38	12.56	12.48	15.15	12.20	12.45
	newstest2014	8.77	11.31	12.85	15.73	11.61	12.28
	newstest2015	9.36	12.27	13.80	16.79	11.90	13.23
	newstest2016	11.40	14.83	16.79	20.97	15.17	16.16
	newstest2017	8.83	11.83	13.15	16.41	11.63	12.99
Multihead attention (head size = n)	dev	11.66	12.51	13.70	14.57	14.61	15.29
	newstest2014	11.64	13.26	14.93	15.42	14.96	15.00
	newstest2015	12.04	14.07	15.78	16.57	16.67	17.57
	newstest2016	14.71	17.57	19.77	20.33	19.80	20.08
	newstest2017	11.72	13.54	14.92	16.03	16.07	16.33
Talking-head attention	dev	10.90	13.89	15.31	15.37	15.23	15.50
	newstest2014	11.21	12.64	14.33	15.28	15.29	15.39
	newstest2015	11.85	12.81	14.61	15.95	15.55	15.36
	newstest2016	14.22	16.33	18.84	19.53	19.34	20.43
	newstest2017	11.15	12.81	14.35	15.49	14.44	15.36
Interacting-head attention	dev	13.07	14.84	14.53	17.46(+2.31)	—	—
	newstest2014	10.91	14.72	15.75	17.35(+1.62)	—	—
	newstest2015	11.99	15.43	16.90	18.40(+1.21)	—	—
	newstest2016	14.95	19.61	21.12	22.36(+1.39)	—	—
	newstest2017	11.09	15.00	17.30	17.67(+1.26)	—	—
(b)
Multihead attention	dev	39.23	37.80	28.23	15.57	32.93	22.47
	newstest2014	46.85	45.59	32.10	19.68	39.59	28.80
	newstest2015	45.22	44.31	32.64	22.68	42.78	28.58
	newstest2016	40.14	39.35	28.51	15.91	32.04	23.97
	newstest2017	43.85	42.74	32.12	19.07	38.38	26.50
Multihead attention (head size = n)	dev	39.37	36.73	25.11	22.97	16.67	13.42
	newstest2014	46.40	35.77	32.56	28.65	18.56	18.77
	newstest2015	46.93	38.45	33.16	28.03	23.13	20.07
	newstest2016	42.91	30.24	27.66	22.77	17.15	13.26
	newstest2017	44.65	34.60	29.82	26.78	19.75	16.52
Talking-head attention	dev	48.17	40.63	39.03	24.80	20.13	16.25
	newstest2014	48.19	48.02	52.98	33.93	27.58	20.32
	newstest2015	61.09	53.80	52.33	33.43	31.13	22.05
	newstest2016	47.02	47.28	47.42	31.08	23.16	14.82
	newstest2017	51.13	47.34	50.63	34.52	28.01	20.05
Interacting-head attention	dev	58.10	35.20	20.83	9.63(5.94)	—	—
	newstest2014	68.39	47.29	29.01	13.64(6.04)	—	—
	newstest2015	79.39	48.55	29.64	16.05(6.63)	—	—
	newstest2016	67.12	44.21	25.94	11.27(4.64)	—	—
	newstest2017	81.56	49.07	30.89	15.23(3.84)	—	—
(c)
Multihead attention	dev	19.36	19.54	20.31	23.16	20.56	20.53
	newstest2014	19.39	19.64	20.83	23.71	20.72	20.97
	newstest2015	19.36	19.60	21.15	24.11	20.70	21.06
	newstest2016	20.43	21.27	22.73	26.29	22.44	22.63
	newstest2017	19.52	19.85	21.05	23.96	20.70	21.11
Multihead attention (head size = n)	dev	19.47	20.96	21.04	23.01	23.32	23.79
	newstest2014	19.69	21.45	22.30	23.52	23.91	24.45
	newstest2015	19.64	21.54	22.61	23.70	24.02	24.46
	newstest2016	20.95	23.26	24.47	25.65	25.44	26.99
	newstest2017	19.68	21.50	22.25	23.73	24.05	24.49
Talking-head attention	dev	17.71	19.55	21.38	23.22	23.51	23.19
	newstest2014	17.73	19.59	21.87	23.65	23.73	23.75
	newstest2015	17.33	19.39	22.03	23.81	24.14	24.26
	newstest2016	18.60	20.52	23.19	25.40	26.02	25.88
	newstest2017	17.74	19.39	21.90	23.71	23.66	24.15
Interacting-head attention	dev	22.43	22.61	23.87	24.62(+1.46)	—	—
	newstest2014	22.67	22.66	24.35	25.10(+1.39)	—	—
	newstest2015	22.81	23.09	24.52	25.53(+1.42)	—	—
	newstest2016	24.69	25.00	26.68	27.27(+0.98)	—	—
	newstest2017	22.12	22.76	24.38	25.66(+1.70)	—	—
(d)
Multihead attention	dev	40.08	40.19	41.54	45.77	42.47	42.79
	newstest2014	36.65	37.81	40.22	49.04	40.40	41.18
	newstest2015	37.33	39.63	42.56	46.84	41.69	42.71
	newstest2016	39.88	41.98	44.40	44.88	44.52	45.13
	newstest2017	36.94	39.01	41.43	46.09	41.02	42.02
Multihead attention (head size = n)	dev	41.39	43.67	44.89	45.79	45.83	46.01
	newstest2014	39.75	42.35	43.62	44.69	45.07	47.25
	newstest2015	40.76	43.37	45.54	46.18	46.22	48.15
	newstest2016	43.18	46.51	48.21	49.34	48.96	50.08
	newstest2017	40.41	43.05	44.87	45.52	45.55	46.08
Talking-head attention	dev	38.42	40.20	42.03	43.99	44.67	45.33
	newstest2014	35.67	39.23	42.51	42.74	45.86	46.09
	newstest2015	37.94	38.61	41.00	44.15	44.92	46.67
	newstest2016	38.83	41.21	42.88	46.28	48.05	46.99
	newstest2017	37.29	39.58	40.93	43.30	45.23	45.65
Interacting-head attention	dev	45.47	46.03	46.54	47.12(+1.35)	—	—
	newstest2014	43.95	43.77	45.15	48.20	—	—
	newstest2015	45.33	45.83	46.61	47.31(+0.51)	—	—
	newstest2016	47.97	48.83	49.94	50.47(+5.59)	—	—
	newstest2017	43.61	44.58	45.91	45.99	—	—
(e)
Multihead attention	dev	1.26	1.30	1.37	1.64	1.37	1.38
	newstest2014	1.13	1.19	1.26	1.52	1.26	1.26
	newstest2015	1.24	1.31	1.42	1.63	1.36	1.36
	newstest2016	1.39	1.50	1.62	1.93	1.61	1.60
	newstest2017	1.17	1.22	1.35	1.61	1.29	1.33
Multihead attention (head size = n)	dev	1.23	1.38	1.45	1.63	1.63	1.66
	newstest2014	1.13	1.30	1.39	1.54	1.55	1.58
	newstest2015	1.19	1.37	1.45	1.67	1.68	1.65
	newstest2016	1.38	1.65	1.80	1.94	1.85	1.99
	newstest2017	1.16	1.35	1.43	1.60	1.71	1.70
Talking-head attention	dev	1.06	1.20	1.38	1.42	1.49	1.58
	newstest2014	0.96	1.11	1.35	1.38	1.46	1.51
	newstest2015	1.04	1.20	1.37	1.39	1.36	1.39
	newstest2016	1.16	1.40	1.67	1.45	1.48	1.62
	newstest2017	0.98	1.15	1.43	1.39	1.48	1.47
Interacting-head attention	dev	1.63	1.64	1.67	1.71(+0.07)	—	—
	newstest2014	1.43	1.51	1.57	1.63(+0.11)	—	—
	newstest2015	1.63	1.61	1.74	1.81(+0.18)	—	—
	newstest2016	1.89	1.88	2.02	2.17(+0.24)	—	—
	newstest2017	1.52	1.50	1.65	1.74(+0.13)	—	—
(f)
Multihead attention	dev	43.74	44.71	45.35	48.82	46.28	45.37
	newstest2014	41.50	42.57	43.80	47.84	44.44	43.77
	newstest2015	43.17	44.10	45.99	49.68	45.81	45.49
	newstest2016	44.90	45.96	47.37	52.29	47.86	47.13
	newstest2017	42.11	43.36	44.99	48.71	44.91	44.71
Multihead attention (head size = n)	dev	44.58	46.00	47.06	48.51	48.99	49.01
	newstest2014	43.09	44.63	45.63	47.40	48.04	47.99
	newstest2015	44.37	46.09	47.54	48.83	49.15	49.56
	newstest2016	46.24	48.30	49.65	51.20	51.30	51.06
	newstest2017	43.82	45.47	46.38	48.34	48.55	48.77
Talking-head attention	dev	38.11	39.63	40.73	44.07	46.14	48.60
	newstest2014	35.46	38.32	39.02	42.77	44.52	47.33
	newstest2015	37.48	38.71	40.35	44.35	46.84	49.52
	newstest2016	37.09	40.25	41.43	45.96	48.36	50.70
	newstest2017	36.20	38.93	39.47	43.75	45.85	48.05
Interacting-head attention	dev	48.03	48.23	48.78	49.15(+0.33)	—	—
	newstest2014	46.72	46.63	47.78	48.71(+0.87)	—	—
	newstest2015	48.11	48.52	49.09	50.20(+0.52)	—	—
	newstest2016	50.88	50.77	51.47	52.71(+0.42)	—	—
	newstest2017	47.73	48.48	49.83	50.01(+1.30)	—	—

Note. The units of the performances are (a) BLEU, (b) WER, (c) METEOR, (d) ROUGE_L, (e) CIDEr, and (f) YiSi.