phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | language model pre-training for transfer to NLP tasks— “Language model pre-training has been shown to be effective for improving many natural language processing tasks (Dai and Le, 2015; Peters et al., 2018a; Radford et al., 2018; Howard and Ruder, 2018).” | |
| claim-type | method— “We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.” | |
| method — evidence | — “In this section, we present BERT fine-tuning results on 11 NLP tasks.” | |
| transfer | from psycholinguistics and reading research: the Cloze task of Taylor 1953 - delete a word from a passage and ask a reader to supply it - made the pre-training objective of a transformer language model as the masked language model— “BERT alleviates the previously mentioned unidirectionality constraint by using a “masked language model” (MLM) pre-training objective, inspired by the Cloze task (Taylor, 1953).” | transfers[0] |
| mechanism | masked language modeling: randomly mask tokens and train the model to predict them from both left and right context, giving a deep bidirectional representation that transfers to many tasks via fine-tuning— “The masked language model randomly masks some of the tokens from the input, and the objective is to predict the original vocabulary id of the masked word based only on its context” | mechanisms[0] |
| builds-on | the Transformer architecture— “We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers.” | claim_relations[0] |
| extends | unidirectional pre-trained language models (e.g. GPT)— “the BERT Transformer uses bidirectional self-attention” | claim_relations[1] |
| extends | Peters et al., 2018a and Radford et al., 2018— “Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.” | claim_relations[2] |
| extends | Radford et al. (2018)— “Unlike Radford et al. (2018), which uses unidirectional language models for pre-training, BERT uses masked language models to enable pretrained deep bidirectional representations.” | claim_relations[3] |
| builds-on | Vaswani et al. (2017)— “BERT’s model architecture is a multi-layer bidirectional Transformer encoder based on the original implementation described in Vaswani et al. (2017) and released in the tensor2tensor library.” | claim_relations[4] |
| builds-on | the Cloze task (Taylor, 1953)— “BERT alleviates the previously mentioned unidirectionality constraint by using a “masked language model” (MLM) pre-training objective, inspired by the Cloze task (Taylor, 1953).” | claim_relations[5] |
| builds-on | Jernite et al. (2017) and Logeswaran and Lee (2018)— “The NSP task is closely related to representationlearning objectives used in Jernite et al. (2017) and Logeswaran and Lee (2018).” | claim_relations[6] |
| sense | bidirectional - TWO ARCHITECTURES UNDER ONE WORD, SEPARATED BY THE AUTHORS: for ELMo it is the concatenation of two independently trained one-directional models, which this paper calls a shallow concatenation and elsewhere 'not deeply bidirectional'; for BERT it is a single model conditioned on both sides at once - so the adjective names a property of the training objective in one case and of the output vector in the other— “BERT uses masked language models to enable pretrained deep bidirectional representations. This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs” | senses[0] |
| sense | masked LM - THE PAPER'S NAME FOR A TASK THE LITERATURE ALREADY NAMED: the authors say outright that the procedure is often referred to as a Cloze task, citing Taylor 1953, so the 2018 name and the 1953 name denote one procedure and the field kept the newer one— “We refer to this procedure as a “masked LM” (MLM), although it is often referred to as a Cloze task in the literature” | senses[1] |
| sense | the GLUE Average - NOT THE OFFICIAL GLUE SCORE: the column excludes the WNLI set, which the caption states, so a number that carries the benchmark's name is computed over a different set of tasks than the benchmark's own average— “column is slightly different than the official GLUE score, since we exclude the problematic WNLI set” | senses[2] |
| control | the two pre-training approaches compared by applying them to the same CoNLL-2003 named entity recognition task — held — rules out a difference between the approaches that is really a difference in the task each was tried on— “In this section, we compare the two approaches by applying BERT to the CoNLL-2003 Named Entity Recognition (NER) task (Tjong Kim Sang and De Meulder, 2003).” | controls[0] |
| edge | masked LM —renamed-across-fields→ Cloze task— “We refer to this procedure as a “masked LM” (MLM), although it is often referred to as a Cloze task in the literature” | concept_edges[0] |
| edge | deep bidirectional —contrasted-with→ shallow concatenation— “BERT uses masked language models to enable pretrained deep bidirectional representations. This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs” | concept_edges[1] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — any account of what the pre-trained model has learned to expect: WE ARE CREATING A MISMATCH BETWEEN PRE-TRAINING AND FINE-TUNING, SINCE THE [MASK] TOKEN DOES NOT APPEAR DURING FINE-TUNING— “Although this allows us to obtain a bidirectional pre-trained model, a downside is that we are creating a mismatch between pre-training and fine-tuning, since the [MASK] token does not appear during fine-tuning.” | limitations[0] |
| for which use | limitation — any use of the [CLS] vector as a sentence embedding without fine-tuning: THE VECTOR C IS NOT A MEANINGFUL SENTENCE REPRESENTATION WITHOUT FINE-TUNING, SINCE IT WAS TRAINED WITH NSP— “The vector C is not a meaningful sentence representation without fine-tuning, since it was trained with NSP.” | limitations[1] |
| for which use | limitation — the reported small-dataset results: FOR BERT_LARGE WE FOUND THAT FINETUNING WAS SOMETIMES UNSTABLE ON SMALL DATASETS, SO WE RAN SEVERAL RANDOM RESTARTS AND SELECTED THE BEST MODEL ON THE DEV SET— “Additionally, for BERTLARGE we found that finetuning was sometimes unstable on small datasets, so we ran several random restarts and selected the best model on the Dev set.” | limitations[2] |
| for which use | limitation — the GLUE averages quoted from this table: the authors state the Average column is slightly different from the official GLUE score because the WNLI set is excluded, so it is not comparable with an official GLUE average— “column is slightly different than the official GLUE score, since we exclude the problematic WNLI set” | limitations[3] |
| for which use | limitation — any claim of novelty for the masked LM objective: the authors note the same procedure is often referred to as a Cloze task in the literature, citing Taylor 1953— “We refer to this procedure as a “masked LM” (MLM), although it is often referred to as a Cloze task in the literature” | limitations[4] |
| for which use | limitation — any comparison of bidirectional models: the authors distinguish their deep bidirectional pre-training from a shallow concatenation of independently trained left-to-right and right-to-left language models, so the word alone does not separate the architectures— “BERT uses masked language models to enable pretrained deep bidirectional representations. This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs” | limitations[5] |
| replication | untested | |
| limitation | Masking creates a mismatch between pre-training and fine-tuning because [MASK] never appears downstream.— “Although this allows us to obtain a bidirectional pre-trained model, a downside is that we are creating a mismatch between pre-training and fine-tuning, since the [MASK] token does not appear during fine-tuning.” | limitations[0] |
| limitation | The [CLS] vector is not usable as a sentence representation without fine-tuning.— “The vector C is not a meaningful sentence representation without fine-tuning, since it was trained with NSP.” | limitations[1] |
| limitation | Fine-tuning the large model was unstable on small datasets and required multiple random restarts.— “Additionally, for BERTLARGE we found that finetuning was sometimes unstable on small datasets, so we ran several random restarts and selected the best model on the Dev set.” | limitations[2] |
| limitation | The reported average is not the official score of the benchmark it is reported on.— “column is slightly different than the official GLUE score, since we exclude the problematic WNLI set” | limitations[3] |
| limitation | The pre-training objective is an existing task under a new name, which the authors state.— “We refer to this procedure as a “masked LM” (MLM), although it is often referred to as a Cloze task in the literature” | limitations[4] |
| limitation | The paper's central adjective is one the systems it is compared against also use.— “BERT uses masked language models to enable pretrained deep bidirectional… | limitations[5] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | bidirectional pre-training is important for language representations | content_tags.findings[0] |
| finding | pre-trained representations reduce the need for task-specific architectures | content_tags.findings[1] |
| finding | a pre-trained BERT is fine-tuned with just one added output layer | content_tags.findings[2] |
| finding | the major limitation of prior models is unidirectionality | content_tags.findings[3] |
| quantity | 80.5 percent GLUE — new GLUE score, +7.7 points absolute | content_tags.quantities[0] |
| quantity | 86.7 percent — MultiNLI accuracy, +4.6 points absolute | content_tags.quantities[1] |
| quantity | 93.2 F1 — SQuAD v1.1 Test F1, +1.5 points | content_tags.quantities[2] |
| quantity | 83.1 F1 — SQuAD v2.0 Test F1, +5.1 points Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[3] |
| quantity | 11 NLP tasks — tasks on which BERT set state of the art Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[4] |
| variable | masked tokens (randomly hidden inputs the model must predict) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.variables[0] |
| variable | left and right context (jointly conditioned on in all layers) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.variables[1] |
| definition | BERT Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[0] |
| definition | masked language model Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[1] |
| definition | feature-based approach Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[2] |
| definition | Cloze task Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[3] |
| key-terms | BERT, pre-training, fine-tuning, bidirectional, masked language model, Transformer, language representation, next sentence prediction, word embeddings, self-attention, natural language processing |
| concept | masked LM Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[4] |
| concept | Cloze task Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[5] |
| concept | deep bidirectional Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[6] |
| concept | shallow concatenation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[7] |
| concept | GLUE Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[8] |
| correlation r | as written: Spearman Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[0] |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824