Research Article

Training Method and Device of Chemical Industry Chinese Language Model Based on Knowledge Distillation

Table 1

The description of previous research on language models.

Feature-based methodsFine-tuning methods

Context-independent word representationPretraining a language model on a large corpus with an unsupervised objective and then fine-tuning the model with in-domain labeled data
Sentence-level representation
Contextualized word representation