Research Article
Named Entity Recognition in Chinese Medical Literature Using Pretraining Models
Table 1
Parameters, pretraining tasks, and corpora of pretraining models.
| | Pretraining model | L | H | A | Pretraining task | Pretraining corpora |
| | BERT [18] | 12 | 768 | 12 | Masked Language Model, NSP | Books Corpus, Wikipedia | | BERT-WWM [20] | 12 | 768 | 12 | Whole Word Masking, NSP | Wikipedia | | BERT-WWM-EXT [20] | 12 | 768 | 12 | Whole Word Masking, NSP | General data (Baike, News, and QA), Wikipedia | | ERNIE [21] | 12 | 768 | 12 | Phrase-level and entity-level masking, NSP | Chinese Wikipedia, Baidu Baike, News, and Tieba | | ERNIE-tiny [22] | 3 | 1024 | 12 | Phrase-level and entity-level masking, NSP | Chinese Wikipedia, Baidu Baike, News, and Tieba | | RoBERTa [23] | 12 | 768 | 12 | Dynamic masking | Books Corpus, Wikipedia |
|
|
