phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | neural sequence transduction (machine translation)— “The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder.” | |
| claim-type | method— “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.” | |
| method — evidence | — “In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.” | |
| mechanism | replace recurrence and convolution with self-attention - every position attends to all others in parallel, capturing long-range dependencies and enabling fast parallel training— “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely” | mechanisms[0] |
| builds-on | attention mechanisms— “based solely on attention mechanisms” | claim_relations[0] |
| contradicts | recurrent and convolutional sequence models— “dispensing with recurrence and convolutions entirely” | claim_relations[1] |
| builds-on | residual connections [11] and layer normalization [1]— “We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1].” | claim_relations[2] |
| builds-on | dot-product (multiplicative) attention— “Dot-product attention is identical to our algorithm, except for the scaling factor” | claim_relations[3] |
| extends | RNN sequence-to-sequence parsing models of Vinyals & Kaiser et al. [37]— “In contrast to RNN sequence-to-sequence models [37], the Transformer outperforms the Berkeley- Parser [29] even when training only on the WSJ training set of 40K sentences.” | claim_relations[4] |
| sense | attention - AS A LOOKUP, NOT AS ANYTHING PSYCHOLOGICAL: an attention function maps a query and a set of key-value pairs to an output, all of them vectors, so the word names a weighted retrieval operation and carries none of its ordinary or cognitive meaning— “An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors” | senses[0] |
| sense | the field's name for what this became - ABSENT FROM IT: `large language model` occurs nowhere in this paper, nor do `foundation model` or `pretraining`; the architecture is introduced as a sequence transduction model evaluated on translation, and the vocabulary now used to describe its descendants postdates it entirely— “The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality” | senses[1] |
| sense | two attention functions, and the slower-but-better one was not taken: additive attention beats unscaled dot-product attention at larger key dimensions, and dot-product was adopted because it runs as optimised matrix multiplication - a choice made on speed with a correction bolted on— “The two most commonly used attention functions are additive attention [2], and dot-product (multiplicative) attention” | senses[2] |
| control | varying one component of the base Transformer at a time and measuring the change in BLEU (Model Variations, Table 3) — held— “In Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions, keeping the amount of computation constant, as described in Section 3.2.2.” — rules out that the Transformer's translation quality comes from the overall training setup rather than from the specific components varied— “To evaluate the importance of different components of the Transformer, we varied our base model in different ways, measuring the change in performance on English-to-German translation” | controls[0] |
| control | replacing the sinusoidal positional encoding with learned positional embeddings, everything else held fixed — held— “In row (E) we replace our sinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical results to the base model.” — rules out that the specific choice of positional encoding is responsible for the model's performance— “In row (E) we replace our sinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical results to the base model.” | controls[1] |
| edge | dot-product attention —is-a→ attention function— “The two most commonly used attention functions are additive attention [2], and dot-product (multiplicative) attention” | concept_edges[0] |
| edge | additive attention —distinct-from→ dot-product attention— “While for small values of dk the two mechanisms perform similarly, additive attention outperforms dot product attention without scaling for larger values of dk” | concept_edges[1] |
| outcome | control — held— “To evaluate the importance of different components of the Transformer, we varied our base model in different ways, measuring the change in performance on English-to-German translation” | controls[0] |
| outcome | control — held | controls[1] |
| for which use | limitation — the long-range dependency claim: the constant number of operations comes AT THE COST OF REDUCED EFFECTIVE RESOLUTION DUE TO AVERAGING ATTENTION-WEIGHTED POSITIONS | limitations[0] |
| for which use | limitation — the constituency-parsing result: it yields BETTER RESULTS THAN ALL PREVIOUSLY REPORTED MODELS WITH THE EXCE[ption of one] Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| for which use | limitation — the perplexity figures specifically: label smoothing HURTS PERPLEXITY, AS THE MODEL LEARNS TO BE MORE UNSURE, BUT IMPROVES ACCURACY AND BLEU SCORE Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| for which use | limitation — the parsing results: WE PERFORMED ONLY A SMALL NUMBER OF EXPERIMENTS TO SELECT THE DROPOUT ... LEARNING RATES AND BEAM SIZE Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| for which use | limitation — the scaled dot-product attention used here and since: the authors state they suspect that for large key dimensions the dot products grow large and push the softmax into regions of extremely small gradients, and scale to counteract that Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| for which use | limitation — the sinusoidal positional encoding as a design decision: the authors found the learned and sinusoidal versions produced nearly identical results and chose the sinusoidal one because it may allow extrapolation to longer sequences than those seen in training Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| for which use | limitation — attention distributions from this architecture used as explanations: the paper says self-attention could yield more interpretable models and that heads appear to exhibit behaviour related to syntactic and semantic structure Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[6] |
| for which use | limitation — the choice of dot-product over additive attention: the authors state additive attention outperforms unscaled dot-product attention for larger key dimensions, and adopt dot-product because it can be implemented with highly optimized matrix multiplication code Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[7] |
| replication | untested | |
| limitation | Reducing long-range dependencies to a constant number of operations costs effective resolution, which the authors only partly offset with multi-head attention. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| limitation | On the constituency-parsing generalisation test the Transformer is still beaten by one prior model. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| limitation | Label smoothing, which they use, degrades perplexity even as it helps BLEU. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| limitation | Hyperparameters for the parsing experiments were barely tuned. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| limitation | The scaling factor every later implementation copies is justified by a suspicion. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| limitation | The two candidate position encodings performed the same, and the one adopted was chosen on a conjecture about behaviour never tested here. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| limitation | The interpretability claim is hedged in both of its clauses. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[6] |
| limitation | The attention mechanism adopted is the one that loses in a stated regime, and it was chosen for implementation speed. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[7] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | The Transformer is a sequence-transduction architecture based SOLELY on attention, dispensing with recurrence and convolutions entirely — more parallelizable and faster to train. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[0] |
| finding | EVIDENCE CORRECTED (RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM) · It reaches 28.4 BLEU on WMT 2014 English-German, improving on the previous best results, including ensembles, by over 2 BLEU. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[1] |
| retracted | finding — RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM. The old evidence was verbatim, anchored and correctly cut, and it contained NONE of this claim's numbers (2014, 28.4). The paper states the claim in one sentence elsewhere and the tagger did not use it. the project, 2026-08-23: 'WE HAVE SPENT THE DAY VERIFYING THAT QUOTES ARE REAL AND NONE OF IT ASKS WHETHER THEY ARE RELEVANT.' Confirmed by reading the claim, the old span and the candidate together; the candidate carries every number AND the claim's own content words. | content_tags.findings[1] |
| quantity | 28.4 BLEU — WMT 2014 English-to-German translation score (new state of the art) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[0] |
| evidence status note | quantity — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.quantities[0] |
| definition | Transformer Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[0] |
| definition | self-attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[1] |
| definition | scaled dot-product attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[3] |
| definition | multi-head attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[4] |
| key-terms | attention, Transformer, encoder, decoder, recurrence, convolutions, sequence, translation, parallelizable |
| concept | self-attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[1] |
| concept | multi-head attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[2] |
| concept | attention function Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[5] |
| concept | additive attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[6] |
| concept | dot-product attention Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[7] |
| statistical-significance | as written: significantly Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[0] |
| variance σ² | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[1] |
| not machine-vouched | variance — not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside. | content_tags.statistics[1] |
| claim type | method — a second reading reaching the same value; recorded: method Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | alternatives.claim_type |
| method | system-or-tool-building — a second reading reaching the same value; recorded: system-or-tool-building Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | alternatives.method |
| phenomenon | sequence transduction without recurrence or convolution, using attention alone to model global dependencies — a second reading DIFFERING from the recorded value; recorded: neural sequence transduction (machine translation) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | alternatives.phenomenon |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824