BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · NAACL / arXiv · 2018 · artificial-intelligence · record devlin2018-bert
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. method is not on the suggested list; the empirical work is benchmark fine-tuning plus ablations ('In this section, we perform ablation experiments over a number of facets of BERT in order to better understand their relative importance.'), so 'computational-experiment' was used. Note the verbatim quote for the Jernite/Logeswaran relation contains the PDF-extraction artefact 'representationlearning' (no space) — it is copied exactly as it appears in the article text. The paper also implicitly disputes Peters et al. (2018b) and Melamud et al. (2016) on model-size effects, but the sentence asserting its own novelty does not name them, so no 'contradicts' relation was recorded. 2026-08-14 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
fine-tuning 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 9 concept tags on this paper, 5 are grounded in a quoted sentence.

BERT Transformer pretraining language model masked LM Cloze task deep bidirectional shallow concatenation GLUE masked language modeling: randomly mask tokens and train the model to predict them from both left and right context, giving a deep bidirectional representation that transfers to many tasks via fine-tuning pre-training fine-tuning bidirectional masked language model language representation next sentence prediction word embeddings self-attention natural language processing feature-based approach masked tokens left and right context correlation Spearman system-or-tool-building

Method, as described: computational-experiment

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1810.04805. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 7 · claim_type 1 · concept_edges 2 · concepts 5 · content_tags 16 · controls 1 · limitations 6 · mechanisms 1 · method 1 · phenomenon 1 · senses 3 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 2 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	language model pre-training for transfer to NLP tasks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: masked language modeling: randomly mask tokens and train the model to predict them from both left and right context, giving a deep bidirectional representation that transfers to many tasks via fine-tuning	`mechanisms[0]`		the tagger's words
limitations	Masking creates a mismatch between pre-training and fine-tuning because [MASK] never appears downstream. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any account of what the pre-trained model has learned to expect: WE ARE CREATING A MISMATCH BETWEEN PRE-TRAINING AND FINE-TUNING, SINCE THE [MASK] TOKEN DOES NOT APPEAR DURING FINE-TUNING	`limitations[0]`		the tagger's words
limitations	The [CLS] vector is not usable as a sentence representation without fine-tuning. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any use of the [CLS] vector as a sentence embedding without fine-tuning: THE VECTOR C IS NOT A MEANINGFUL SENTENCE REPRESENTATION WITHOUT FINE-TUNING, SINCE IT WAS TRAINED WITH NSP	`limitations[1]`		the tagger's words
limitations	Fine-tuning the large model was unstable on small datasets and required multiple random restarts. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the reported small-dataset results: FOR BERT_LARGE WE FOUND THAT FINETUNING WAS SOMETIMES UNSTABLE ON SMALL DATASETS, SO WE RAN SEVERAL RANDOM RESTARTS AND SELECTED THE BEST MODEL ON THE DEV SET	`limitations[2]`		the tagger's words
limitations	The reported average is not the official score of the benchmark it is reported on. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the GLUE averages quoted from this table: the authors state the Average column is slightly different from the official GLUE score because the WNLI set is excluded, so it is not comparable with an official GLUE average	`limitations[3]`	1.0	the tagger's words
limitations	The pre-training objective is an existing task under a new name, which the authors state. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any claim of novelty for the masked LM objective: the authors note the same procedure is often referred to as a Cloze task in the literature, citing Taylor 1953	`limitations[4]`	1.0	the tagger's words
limitations	The paper's central adjective is one the systems it is compared against also use. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any comparison of bidirectional models: the authors distinguish their deep bidirectional pre-training from a shallow concatenation of independently trained left-to-right and right-to-left language models, so the word alone does not separate the architectures	`limitations[5]`	1.0	the tagger's words
transfers	the Cloze task of Taylor 1953 - delete a word from a passage and ask a reader to supply it - made the pre-training objective of a transformer language model as the masked language model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`	0.9	the tagger's words
controls	the two pre-training approaches compared by applying them to the same CoNLL-2003 named entity recognition task Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: a difference between the approaches that is really a difference in the task each was tried on · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.85	the tagger's words
concepts	masked LM Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: masked LM	`concepts[4]`		the paper's words
concepts	Cloze task Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Cloze task	`concepts[5]`		the paper's words
concepts	deep bidirectional Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: deep bidirectional	`concepts[6]`		the paper's words
concepts	shallow concatenation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: shallow concatenation	`concepts[7]`		the paper's words
concepts	GLUE Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: GLUE	`concepts[8]`		the paper's words
senses	bidirectional - TWO ARCHITECTURES UNDER ONE WORD, SEPARATED BY THE AUTHORS: for ELMo it is the concatenation of two independently trained one-directional models, which this paper calls a shallow concatenation and elsewhere 'not deeply bidirectional'; for BERT it is a single model conditioned on both sides at once - so the adjective names a property of the training objective in one case and of the output vector in the other Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	masked LM - THE PAPER'S NAME FOR A TASK THE LITERATURE ALREADY NAMED: the authors say outright that the procedure is often referred to as a Cloze task, citing Taylor 1953, so the 2018 name and the 1953 name denote one procedure and the field kept the newer one Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	the GLUE Average - NOT THE OFFICIAL GLUE SCORE: the column excludes the WNLI set, which the caption states, so a number that carries the benchmark's name is computed over a different set of tasks than the benchmark's own average Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → the Transformer architecture Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`	0.85	the tagger's words
claim_relations	extends → unidirectional pre-trained language models (e.g. GPT) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	extends → Peters et al., 2018a and Radford et al., 2018 Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → Radford et al. (2018) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	builds-on → Vaswani et al. (2017) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
claim_relations	builds-on → the Cloze task (Taylor, 1953) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[5]`		the tagger's words
claim_relations	builds-on → Jernite et al. (2017) and Logeswaran and Lee (2018) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[6]`		the tagger's words
concept_edges	masked LM —renamed-across-fields→ Cloze task Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: masked LM · Cloze task	`concept_edges[0]`		the tagger's words
concept_edges	deep bidirectional —contrasted-with→ shallow concatenation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: deep bidirectional · shallow concatenation	`concept_edges[1]`		the tagger's words
content_tags	80.5 — new GLUE score, +7.7 points absolute Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	86.7 — MultiNLI accuracy, +4.6 points absolute Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	93.2 — SQuAD v1.1 Test F1, +1.5 points Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	83.1 — SQuAD v2.0 Test F1, +5.1 points Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[3]`		the tagger's words
content_tags	11 — tasks on which BERT set state of the art Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[4]`		the tagger's words
content_tags	bidirectional pre-training is important for language representations Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	pre-trained representations reduce the need for task-specific architectures Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	a pre-trained BERT is fine-tuned with just one added output layer Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	the major limitation of prior models is unidirectionality Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	masked tokens Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: masked tokens	`content_tags.variables[0]`		the tagger's words
content_tags	left and right context Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: left and right context	`content_tags.variables[1]`		the tagger's words
content_tags	BERT Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: BERT	`content_tags.definitions[0]`		the tagger's words
content_tags	masked language model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: masked language model	`content_tags.definitions[1]`		the tagger's words
content_tags	feature-based approach Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: feature-based approach	`content_tags.definitions[2]`		the tagger's words
content_tags	Cloze task Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Cloze task	`content_tags.definitions[3]`		the tagger's words
content_tags	correlation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: correlation	`content_tags.statistics[0]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.