Attention Is All You Need

Attention Is All You Need Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Illia Polosukhin

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Attention Is All You Need 2017 Advances in Neural Information Processing Systems (NeurIPS) 1706.03762 vaswani2017-attention vaswani2017-attention.txt, the project's staged copy ceecd93371a6f0d8313965aaf5e794617feb12d6 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. neural sequence transduction (machine translation) Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms replace recurrence and convolution with self-attention - every position attends to all others in parallel, capturing long-range dependencies and enabling fast parallel training Quotation withheld because this record is not cleared for redistribution. Reducing long-range dependencies to a constant number of operations costs effective resolution, which the authors only partly offset with multi-head attention. what this limits: the long-range dependency claim: the constant number of operations comes AT THE COST OF REDUCED EFFECTIVE RESOLUTION DUE TO AVERAGING ATTENTION-WEIGHTED POSITIONS Quotation withheld because this record is not cleared for redistribution. On the constituency-parsing generalisation test the Transformer is still beaten by one prior model. what this limits: the constituency-parsing result: it yields BETTER RESULTS THAN ALL PREVIOUSLY REPORTED MODELS WITH THE EXCE[ption of one] Quotation withheld because this record is not cleared for redistribution. Label smoothing, which they use, degrades perplexity even as it helps BLEU. what this limits: the perplexity figures specifically: label smoothing HURTS PERPLEXITY, AS THE MODEL LEARNS TO BE MORE UNSURE, BUT IMPROVES ACCURACY AND BLEU SCORE Quotation withheld because this record is not cleared for redistribution. Hyperparameters for the parsing experiments were barely tuned. what this limits: the parsing results: WE PERFORMED ONLY A SMALL NUMBER OF EXPERIMENTS TO SELECT THE DROPOUT ... LEARNING RATES AND BEAM SIZE Quotation withheld because this record is not cleared for redistribution. The scaling factor every later implementation copies is justified by a suspicion. what this limits: the scaled dot-product attention used here and since: the authors state they suspect that for large key dimensions the dot products grow large and push the softmax into regions of extremely small gradients, and scale to counteract that Quotation withheld because this record is not cleared for redistribution. The two candidate position encodings performed the same, and the one adopted was chosen on a conjecture about behaviour never tested here. what this limits: the sinusoidal positional encoding as a design decision: the authors found the learned and sinusoidal versions produced nearly identical results and chose the sinusoidal one because it may allow extrapolation to longer sequences than those seen in training Quotation withheld because this record is not cleared for redistribution. The interpretability claim is hedged in both of its clauses. what this limits: attention distributions from this architecture used as explanations: the paper says self-attention could yield more interpretable models and that heads appear to exhibit behaviour related to syntactic and semantic structure Quotation withheld because this record is not cleared for redistribution. The attention mechanism adopted is the one that loses in a stated regime, and it was chosen for implementation speed. what this limits: the choice of dot-product over additive attention: the authors state additive attention outperforms unscaled dot-product attention for larger key dimensions, and adopt dot-product because it can be implemented with highly optimized matrix multiplication code Quotation withheld because this record is not cleared for redistribution. varying one component of the base Transformer at a time and measuring the change in BLEU (Model Variations, Table 3) eliminates: that the Transformer's translation quality comes from the overall training setup rather than from the specific components varied · outcome: held Quotation withheld because this record is not cleared for redistribution. replacing the sinusoidal positional encoding with learned positional embeddings, everything else held fixed eliminates: that the specific choice of positional encoding is responsible for the model's performance · outcome: held Quotation withheld because this record is not cleared for redistribution. self-attention self-attention Quotation withheld because this record is not cleared for redistribution. multi-head attention multi-head attention Quotation withheld because this record is not cleared for redistribution. attention function attention function Quotation withheld because this record is not cleared for redistribution. additive attention additive attention Quotation withheld because this record is not cleared for redistribution. dot-product attention dot-product attention Quotation withheld because this record is not cleared for redistribution. attention - AS A LOOKUP, NOT AS ANYTHING PSYCHOLOGICAL: an attention function maps a query and a set of key-value pairs to an output, all of them vectors, so the word names a weighted retrieval operation and carries none of its ordinary or cognitive meaning Quotation withheld because this record is not cleared for redistribution. the field's name for what this became - ABSENT FROM IT: `large language model` occurs nowhere in this paper, nor do `foundation model` or `pretraining`; the architecture is introduced as a sequence transduction model evaluated on translation, and the vocabulary now used to describe its descendants postdates it entirely Quotation withheld because this record is not cleared for redistribution. two attention functions, and the slower-but-better one was not taken: additive attention beats unscaled dot-product attention at larger key dimensions, and dot-product was adopted because it runs as optimised matrix multiplication - a choice made on speed with a correction bolted on Quotation withheld because this record is not cleared for redistribution. claim_type: method a second reading reaching the same value. Recorded: method — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. method: system-or-tool-building a second reading reaching the same value. Recorded: system-or-tool-building — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. phenomenon: sequence transduction without recurrence or convolution, using attention alone to model global dependencies a second reading DIFFERING from the recorded value. Recorded: neural sequence transduction (machine translation) — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is)., Quotation withheld because this record is not cleared for redistribution. builds-on → attention mechanisms Quotation withheld because this record is not cleared for redistribution. contradicts → recurrent and convolutional sequence models Quotation withheld because this record is not cleared for redistribution. builds-on → residual connections [11] and layer normalization [1] Quotation withheld because this record is not cleared for redistribution. builds-on → dot-product (multiplicative) attention Quotation withheld because this record is not cleared for redistribution. extends → RNN sequence-to-sequence parsing models of Vinyals & Kaiser et al. [37] Quotation withheld because this record is not cleared for redistribution. dot-product attention —is-a→ attention function dot-product attention attention function strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. additive attention —distinct-from→ dot-product attention additive attention dot-product attention strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 28.4 — WMT 2014 English-to-German translation score (new state of the art) about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. The Transformer is a sequence-transduction architecture based SOLELY on attention, Quotation withheld — more parallelizable and faster to train. Quotation withheld because this record is not cleared for redistribution. EVIDENCE CORRECTED (RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM) · It reaches 28.4 BLEU on WMT 2014 English-German, improving on the previous best results, including ensembles, by over 2 BLEU. EVIDENCE CORRECTED: RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM. The old evidence was verbatim, anchored and correctly cut, and it contained NONE of this claim's numbers (2014, 28.4). The paper states the claim in one sentence elsewhere and the tagger did not use it. the project, 2026-08-23: 'WE HAVE SPENT THE DAY VERIFYING THAT QUOTES ARE REAL AND NONE OF IT ASKS WHETHER THEY ARE RELEVANT.' Confirmed by reading the claim, the old span and the candidate together; the candidate carries every number AND the claim's own content words. Quotation withheld because this record is not cleared for redistribution. Transformer Transformer Quotation withheld because this record is not cleared for redistribution. self-attention self-attention Quotation withheld because this record is not cleared for redistribution. scaled dot-product attention scaled dot-product attention Quotation withheld because this record is not cleared for redistribution. multi-head attention multi-head attention Quotation withheld because this record is not cleared for redistribution. statistical-significance statistical-significance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1706.03762.