Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin · Advances in Neural Information Processing Systems (NeurIPS) · 2017 · artificial-intelligence · record vaswani2017-attention
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. The paper has no limitations section; the quoted limits are scattered through §2, §5.4 and §6.3. The evidence for the dot-product-attention relation is a verbatim sentence prefix — the extracted PDF text puts a stray space before the sentence-final period ('scaling factor .') because the fraction 1/sqrt(dk) was lifted out of the line, so quoting the whole sentence would have risked a mismatch. The paper claims novelty ('the first transduction model relying entirely on self-attention') but names no specific prior work in that sentence, so it is not recorded as a claim_relation. 2026-08-14 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 1 quote(s) on this record are not found as one exact string in the current staged text - content_tags.statistics[1].evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row.

The quality of the text these quotes were cut from. symbol-font-not-extractable: 1 '(cid:NNN)' glyph-index markers remain in a staged text of 30,161 characters, where the PDF's embedded font had no ToUnicode map. THE QUOTES ON THIS RECORD ARE THE PAPER'S AND THE STAGED COPY IS THE DAMAGED ONE - which is the opposite of the usual case and is why it went unnoticed: one of its 50 live quotes is not an exact substring of this text, and it was located in it by aligning the quote's WORD SEQUENCE under exactly this damage - the words are there, the characters inside them are not. Not repaired in place: editing the staged text moves every offset after the edit and re-hashes every anchor on the record. Measured at 113aae60fefc3eb1, re-measured unchanged at 54b63482778ddf56.

1 quote on this page was cut from a different copy of this article. This project holds two extractions of this paper: the plain text every other quote here is anchored in, and the publisher’s own HTML edition, which we are not licensed to republish. It is verbatim in the second and cannot be found in the first — not because the tagging is wrong, but because the extraction damaged the passage: a page number dropped into the middle of a number, a formula moved out of its sentence, a ligature where two letters belong. The record names the copy for each one. The quote is the paper’s; the extraction is not.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
attention 36 fields · statistical-significance 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 8 concept tags on this paper, 5 are grounded in a quoted sentence.

Transformer self-attention multi-head attention positional encoding large language model attention function additive attention dot-product attention replace recurrence and convolution with self-attention - every position attends to all others in parallel, capturing long-range dependencies and enabling fast parallel training attention encoder decoder recurrence convolutions sequence translation parallelizable scaled dot-product attention statistical-significance variance system-or-tool-building

Legend

second reading	`alternatives`	an independent re-reading of a facet, recorded whether or not it agrees
relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1706.03762. The reading view exists for the papers whose licence permits it.

Every tag on this page

alternatives 3 · claim_relations 5 · claim_type 1 · concept_edges 2 · concepts 5 · content_tags 8 · controls 2 · limitations 8 · mechanisms 1 · method 1 · phenomenon 1 · senses 3

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 2 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	neural sequence transduction (machine translation) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: replace recurrence and convolution with self-attention - every position attends to all others in parallel, capturing long-range dependencies and enabling fast parallel training	`mechanisms[0]`		the tagger's words
limitations	Reducing long-range dependencies to a constant number of operations costs effective resolution, which the authors only partly offset with multi-head attention. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the long-range dependency claim: the constant number of operations comes AT THE COST OF REDUCED EFFECTIVE RESOLUTION DUE TO AVERAGING ATTENTION-WEIGHTED POSITIONS	`limitations[0]`		the tagger's words
limitations	On the constituency-parsing generalisation test the Transformer is still beaten by one prior model. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the constituency-parsing result: it yields BETTER RESULTS THAN ALL PREVIOUSLY REPORTED MODELS WITH THE EXCE[ption of one]	`limitations[1]`		the tagger's words
limitations	Label smoothing, which they use, degrades perplexity even as it helps BLEU. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the perplexity figures specifically: label smoothing HURTS PERPLEXITY, AS THE MODEL LEARNS TO BE MORE UNSURE, BUT IMPROVES ACCURACY AND BLEU SCORE	`limitations[2]`		the tagger's words
limitations	Hyperparameters for the parsing experiments were barely tuned. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the parsing results: WE PERFORMED ONLY A SMALL NUMBER OF EXPERIMENTS TO SELECT THE DROPOUT ... LEARNING RATES AND BEAM SIZE	`limitations[3]`		the tagger's words
limitations	The scaling factor every later implementation copies is justified by a suspicion. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the scaled dot-product attention used here and since: the authors state they suspect that for large key dimensions the dot products grow large and push the softmax into regions of extremely small gradients, and scale to counteract that	`limitations[4]`	1.0	the tagger's words
limitations	The two candidate position encodings performed the same, and the one adopted was chosen on a conjecture about behaviour never tested here. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the sinusoidal positional encoding as a design decision: the authors found the learned and sinusoidal versions produced nearly identical results and chose the sinusoidal one because it may allow extrapolation to longer sequences than those seen in training	`limitations[5]`	1.0	the tagger's words
limitations	The interpretability claim is hedged in both of its clauses. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: attention distributions from this architecture used as explanations: the paper says self-attention could yield more interpretable models and that heads appear to exhibit behaviour related to syntactic and semantic structure	`limitations[6]`	1.0	the tagger's words
limitations	The attention mechanism adopted is the one that loses in a stated regime, and it was chosen for implementation speed. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the choice of dot-product over additive attention: the authors state additive attention outperforms unscaled dot-product attention for larger key dimensions, and adopt dot-product because it can be implemented with highly optimized matrix multiplication code	`limitations[7]`	1.0	the tagger's words
controls	varying one component of the base Transformer at a time and measuring the change in BLEU (Model Variations, Table 3) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: that the Transformer's translation quality comes from the overall training setup rather than from the specific components varied · outcome: held	`controls[0]`	0.9	the tagger's words
controls	replacing the sinusoidal positional encoding with learned positional embeddings, everything else held fixed Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: that the specific choice of positional encoding is responsible for the model's performance · outcome: held	`controls[1]`	0.9	the tagger's words
concepts	self-attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: self-attention	`concepts[1]`		the paper's words
concepts	multi-head attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: multi-head attention	`concepts[2]`		the paper's words
concepts	attention function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: attention function	`concepts[5]`		the paper's words
concepts	additive attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: additive attention	`concepts[6]`		the paper's words
concepts	dot-product attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: dot-product attention	`concepts[7]`		the paper's words
senses	attention - AS A LOOKUP, NOT AS ANYTHING PSYCHOLOGICAL: an attention function maps a query and a set of key-value pairs to an output, all of them vectors, so the word names a weighted retrieval operation and carries none of its ordinary or cognitive meaning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	the field's name for what this became - ABSENT FROM IT: `large language model` occurs nowhere in this paper, nor do `foundation model` or `pretraining`; the architecture is introduced as a sequence transduction model evaluated on translation, and the vocabulary now used to describe its descendants postdates it entirely Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	two attention functions, and the slower-but-better one was not taken: additive attention beats unscaled dot-product attention at larger key dimensions, and dot-product was adopted because it runs as optimised matrix multiplication - a choice made on speed with a correction bolted on Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
alternatives	claim_type: method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. a second reading reaching the same value. Recorded: method — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is).,	`alternatives.claim_type`	0.9	the tagger's words
alternatives	method: system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. a second reading reaching the same value. Recorded: system-or-tool-building — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is).,	`alternatives.method`	0.8	the tagger's words
alternatives	phenomenon: sequence transduction without recurrence or convolution, using attention alone to model global dependencies Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. a second reading DIFFERING from the recorded value. Recorded: neural sequence transduction (machine translation) — note: A BLIND second reading by the project on 2026-08-25: the record's own claim type/method/phenomenon were not looked at until every value here was written and substring-verified. Not applied - a person decides. Sampling frame: the 504 staged records carrying no alternatives block, so this never re-measures the 202. All 11 records with 130+ tags were taken as a census and 19 more spread one per discipline across 22 disciplines - the dense non-neuroscience cell is the one the 202 do not contain (inside them, 94-100% of the densest bands are neuroscience; among these 11, one is).,	`alternatives.phenomenon`	0.85	the tagger's words
claim_relations	builds-on → attention mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	contradicts → recurrent and convolutional sequence models Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → residual connections [11] and layer normalization [1] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	builds-on → dot-product (multiplicative) attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	extends → RNN sequence-to-sequence parsing models of Vinyals & Kaiser et al. [37] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
concept_edges	dot-product attention —is-a→ attention function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: dot-product attention · attention function	`concept_edges[0]`		the tagger's words
concept_edges	additive attention —distinct-from→ dot-product attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: additive attention · dot-product attention	`concept_edges[1]`		the tagger's words
content_tags	28.4 — WMT 2014 English-to-German translation score (new state of the art) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted.	`content_tags.quantities[0]`		the tagger's words
content_tags	The Transformer is a sequence-transduction architecture based SOLELY on attention, Quotation withheld — more parallelizable and faster to train. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	EVIDENCE CORRECTED (RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM) · It reaches 28.4 BLEU on WMT 2014 English-German, improving on the previous best results, including ensembles, by over 2 BLEU. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. EVIDENCE CORRECTED: RE-GROUNDED - THE POINTER WAS WRONG, NOT THE CLAIM. The old evidence was verbatim, anchored and correctly cut, and it contained NONE of this claim's numbers (2014, 28.4). The paper states the claim in one sentence elsewhere and the tagger did not use it. the project, 2026-08-23: 'WE HAVE SPENT THE DAY VERIFYING THAT QUOTES ARE REAL AND NONE OF IT ASKS WHETHER THEY ARE RELEVANT.' Confirmed by reading the claim, the old span and the candidate together; the candidate carries every number AND the claim's own content words.	`content_tags.findings[1]`		the tagger's words
content_tags	Transformer Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Transformer	`content_tags.definitions[0]`		the tagger's words
content_tags	self-attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: self-attention	`content_tags.definitions[1]`		the tagger's words
content_tags	scaled dot-product attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: scaled dot-product attention	`content_tags.definitions[3]`		the tagger's words
content_tags	multi-head attention Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: multi-head attention	`content_tags.definitions[4]`		the tagger's words
content_tags	statistical-significance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: statistical-significance	`content_tags.statistics[0]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

content_tags.statistics[1] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
NOT MACHINE-VOUCHED: not-in-our-copy - The quote is in the original web page we fetched for this paper, but not in the text we tag against, which was extracted differently. Checked with ligatures, hyphens, spacing and punctuation set aside.

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.