Multimodal Fusion Method Based on Self-Attention Mechanism

Table 1

Results for emotion recognition on IEMPCAP, personality trait recognition on POM, and sentiment analysis on CMU-MOSI.

Data set	IEMOCAP				POM			CMU-MOSI
Metric	F1-happy	F1-sad	F1-angry	F1-neutral	MAE	Corr	Acc	MAE	Corr	Acc-2	F1	Acc-7
SVM	81.5	78.8	82.4	64.9	0.887	10.4	33.9	1.864	0.057	50.2	50.1	17.5
DF	81.0	81.2	65.4	44.0	0.869	14.4	34.1	1.143	0.518	72.3	72.1	26.8
BC-LSTM	81.7	81.7	84.2	64.1	0.840	27.8	34.8	1.079	0.581	73.9	73.9	28.7
MV-LSTM	81.3	74.0	84.3	66.7	0.891	27.0	34.6	1.019	0.601	73.9	74.0	33.2
LMF	85.2	85.8	87.4	71.7	0.837	32.3	36.8	1.071	0.571	73.3	73.3	30.0
HPFN	85.7	86.2	87.8	71.9	0.840	35.6	36.7	0.975	0.601	73.0	73.1	35.1
Our methods	85.7	86.3	87.6	71.9	0.834	52.3	36.7	1.032	0.610	74.1	73.0	30.5

Best results are italicized.