Bidirectional Language Modeling: A Systematic Literature Review

Table 5

Model type and sentence representation learning.

Pretraining objectives	Sentence representation learning
	NSP	SOP	None
Autoencoding (AE)	[14, 27, 29–32]	[33–35]	[10, 11, 15, 20, 36–49]
Autoregressive (AR)	—	—	[50]
Autoencoding and autoregressive	[51]	—	[28]
Autoencoding and partially autoregressive (PAR)	—	—	[52]