BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2018 NAACL / arXiv 1810.04805 devlin2018-bert devlin2018-bert.txt, the project's staged copy 004b468db1da98e6f682985fbb328911fa1e8b60 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. language model pre-training for transfer to NLP tasks Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms masked language modeling: randomly mask tokens and train the model to predict them from both left and right context, giving a deep bidirectional representation that transfers to many tasks via fine-tuning Quotation withheld because this record is not cleared for redistribution. Masking creates a mismatch between pre-training and fine-tuning because [MASK] never appears downstream. what this limits: any account of what the pre-trained model has learned to expect: WE ARE CREATING A MISMATCH BETWEEN PRE-TRAINING AND FINE-TUNING, SINCE THE [MASK] TOKEN DOES NOT APPEAR DURING FINE-TUNING Quotation withheld because this record is not cleared for redistribution. The [CLS] vector is not usable as a sentence representation without fine-tuning. what this limits: any use of the [CLS] vector as a sentence embedding without fine-tuning: THE VECTOR C IS NOT A MEANINGFUL SENTENCE REPRESENTATION WITHOUT FINE-TUNING, SINCE IT WAS TRAINED WITH NSP Quotation withheld because this record is not cleared for redistribution. Fine-tuning the large model was unstable on small datasets and required multiple random restarts. what this limits: the reported small-dataset results: FOR BERT_LARGE WE FOUND THAT FINETUNING WAS SOMETIMES UNSTABLE ON SMALL DATASETS, SO WE RAN SEVERAL RANDOM RESTARTS AND SELECTED THE BEST MODEL ON THE DEV SET Quotation withheld because this record is not cleared for redistribution. The reported average is not the official score of the benchmark it is reported on. what this limits: the GLUE averages quoted from this table: the authors state the Average column is slightly different from the official GLUE score because the WNLI set is excluded, so it is not comparable with an official GLUE average Quotation withheld because this record is not cleared for redistribution. The pre-training objective is an existing task under a new name, which the authors state. what this limits: any claim of novelty for the masked LM objective: the authors note the same procedure is often referred to as a Cloze task in the literature, citing Taylor 1953 Quotation withheld because this record is not cleared for redistribution. The paper's central adjective is one the systems it is compared against also use. what this limits: any comparison of bidirectional models: the authors distinguish their deep bidirectional pre-training from a shallow concatenation of independently trained left-to-right and right-to-left language models, so the word alone does not separate the architectures Quotation withheld because this record is not cleared for redistribution. the Cloze task of Taylor 1953 - delete a word from a passage and ask a reader to supply it - made the pre-training objective of a transformer language model as the masked language model Quotation withheld because this record is not cleared for redistribution. the two pre-training approaches compared by applying them to the same CoNLL-2003 named entity recognition task eliminates: a difference between the approaches that is really a difference in the task each was tried on · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. masked LM masked LM Quotation withheld because this record is not cleared for redistribution. Cloze task Cloze task Quotation withheld because this record is not cleared for redistribution. deep bidirectional deep bidirectional Quotation withheld because this record is not cleared for redistribution. shallow concatenation shallow concatenation Quotation withheld because this record is not cleared for redistribution. GLUE GLUE Quotation withheld because this record is not cleared for redistribution. bidirectional - TWO ARCHITECTURES UNDER ONE WORD, SEPARATED BY THE AUTHORS: for ELMo it is the concatenation of two independently trained one-directional models, which this paper calls a shallow concatenation and elsewhere 'not deeply bidirectional'; for BERT it is a single model conditioned on both sides at once - so the adjective names a property of the training objective in one case and of the output vector in the other Quotation withheld because this record is not cleared for redistribution. masked LM - THE PAPER'S NAME FOR A TASK THE LITERATURE ALREADY NAMED: the authors say outright that the procedure is often referred to as a Cloze task, citing Taylor 1953, so the 2018 name and the 1953 name denote one procedure and the field kept the newer one Quotation withheld because this record is not cleared for redistribution. the GLUE Average - NOT THE OFFICIAL GLUE SCORE: the column excludes the WNLI set, which the caption states, so a number that carries the benchmark's name is computed over a different set of tasks than the benchmark's own average Quotation withheld because this record is not cleared for redistribution. builds-on → the Transformer architecture Quotation withheld because this record is not cleared for redistribution. extends → unidirectional pre-trained language models (e.g. GPT) Quotation withheld because this record is not cleared for redistribution. extends → Peters et al., 2018a and Radford et al., 2018 Quotation withheld because this record is not cleared for redistribution. extends → Radford et al. (2018) Quotation withheld because this record is not cleared for redistribution. builds-on → Vaswani et al. (2017) Quotation withheld because this record is not cleared for redistribution. builds-on → the Cloze task (Taylor, 1953) Quotation withheld because this record is not cleared for redistribution. builds-on → Jernite et al. (2017) and Logeswaran and Lee (2018) Quotation withheld because this record is not cleared for redistribution. masked LM —renamed-across-fields→ Cloze task masked LM Cloze task strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. deep bidirectional —contrasted-with→ shallow concatenation deep bidirectional shallow concatenation strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 80.5 — new GLUE score, +7.7 points absolute Quotation withheld because this record is not cleared for redistribution. 86.7 — MultiNLI accuracy, +4.6 points absolute Quotation withheld because this record is not cleared for redistribution. 93.2 — SQuAD v1.1 Test F1, +1.5 points Quotation withheld because this record is not cleared for redistribution. 83.1 — SQuAD v2.0 Test F1, +5.1 points Quotation withheld because this record is not cleared for redistribution. 11 — tasks on which BERT set state of the art Quotation withheld because this record is not cleared for redistribution. bidirectional pre-training is important for language representations Quotation withheld because this record is not cleared for redistribution. pre-trained representations reduce the need for task-specific architectures Quotation withheld because this record is not cleared for redistribution. a pre-trained BERT is fine-tuned with just one added output layer Quotation withheld because this record is not cleared for redistribution. the major limitation of prior models is unidirectionality Quotation withheld because this record is not cleared for redistribution. masked tokens masked tokens Quotation withheld because this record is not cleared for redistribution. left and right context left and right context Quotation withheld because this record is not cleared for redistribution. BERT BERT Quotation withheld because this record is not cleared for redistribution. masked language model masked language model Quotation withheld because this record is not cleared for redistribution. feature-based approach feature-based approach Quotation withheld because this record is not cleared for redistribution. Cloze task Cloze task Quotation withheld because this record is not cleared for redistribution. correlation correlation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1810.04805.