Playing Atari with Deep Reinforcement Learning

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. · NeurIPS Deep Learning Workshop / arXiv · 2013 · artificial-intelligence · record mnih2013-dqn
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Workshop-length paper; all quotes taken from Abstract, Introduction, Related Work, Section 4 and Sections 5-6. Claim type is 'method' rather than 'existence' because the contribution is an algorithm (deep Q-learning with experience replay), even though the abstract's headline sentence is phrased as a first-demonstration claim. method recorded as system-or-tool-building; an equally defensible reading is 'simulation', since all evaluation happens inside the Atari emulator. No self-described limitations section exists, so the three limitation quotes were taken from asides in Sections 4, 5 and 5.3. Replication status is untested: the paper reports no attempt to reproduce prior results and makes no claim about others reproducing its own. 2026-08-14, 2026-08-15 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 1 quote(s) on this record are not found as one exact string in the current staged text - content_tags.findings[1].evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row. Update, 16 September 2026: EXCEPTION TO THE DATED PASS ABOVE, on this record only: this record carries 1 composite quote field - a quote assembled from two or more passages joined by an ellipsis. A COMPOSITE IS NOT AN EXACT CONTIGUOUS SUBSTRING of the staged text and is deliberately not anchored: each of its parts is verified separately at the offsets in evidence composite spans, and a part occurring more than once carries span null rather than a guessed offset. So the 'every quote verified as an exact substring' clause of the dated pass above DOES NOT HOLD for those fields, and never did - the parts were verified, the whole was not. Nothing is deleted, re-grounded or withdrawn: this writes down an exception that was always true of composites and had not been recorded, so a reader was told more than the pass had checked.

The quality of the text these quotes were cut from. symbol-font-not-extractable: 56 '(cid:NNN)' glyph-index markers remain in a staged text of 31,756 characters, where the PDF's embedded font had no ToUnicode map. THE QUOTES ON THIS RECORD ARE THE PAPER'S AND THE STAGED COPY IS THE DAMAGED ONE - which is the opposite of the usual case and is why it went unnoticed: one of its 48 live quotes is not an exact substring of this text, and it could NOT be located in it even by aligning the quote's WORD SEQUENCE under this damage, so the defect recorded here is present on the record but does not by itself account for the miss. Not repaired in place: editing the staged text moves every offset after the edit and re-hashes every anchor on the record. Measured at 113aae60fefc3eb1, re-measured unchanged at 54b63482778ddf56.

1 quote on this page was cut from a different copy of this article. This project holds two extractions of this paper: the plain text every other quote here is anchored in, and the publisher’s own HTML edition, which we are not licensed to republish. It is verbatim in the second and cannot be found in the first — not because the tagging is wrong, but because the extraction damaged the passage: a page number dropped into the middle of a number, a formula moved out of its sentence, a ligature where two letters belong. The record names the copy for each one. The quote is the paper’s; the extraction is not.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
reward 16 fields · deep learning 11 fields · learning 7 fields · value 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 9 concept tags on this paper, 6 are grounded in a quoted sentence.

reinforcement learning deep learning Q-learning experience replay perceptually aliased evaluation metric replay memory action-value function frame-skipping experience replay: store past transitions and train on random minibatches of them, breaking the correlations between consecutive experiences and stabilizing deep Q-learning reinforcement learning pixels reward value games convolutional policy deep reinforcement learning raw pixels value function mean average of variance system-or-tool-building

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1312.5602. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 5 · claim_type 1 · concept_edges 1 · concepts 6 · content_tags 9 · controls 1 · limitations 8 · mechanisms 1 · method 1 · phenomenon 1 · senses 3 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	reinforcement learning from raw visual input Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: experience replay: store past transitions and train on random minibatches of them, breaking the correlations between consecutive experiences and stabilizing deep Q-learning	`mechanisms[0]`		the tagger's words
limitations	reinforcement learning presents several challenges from a deep-learning perspective Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the transfer of supervised deep learning intuitions to this setting: REINFORCEMENT LEARNING PRESENTS SEVERAL CHALLENGES FROM A DEEP LEARNING PERSPECTIVE	`limitations[0]`		the tagger's words
limitations	The replay memory is uniformly sampled and finite, so it neither prioritises informative transitions nor retains old ones. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the sample efficiency and stability results: THE MEMORY BUFFER DOES NOT DIFFERENTIATE IMPORTANT TRANSITIONS AND ALWAYS OVERWRITES WITH RECENT TRANSITIONS DUE TO THE FINITE MEMORY SIZE N	`limitations[1]`		the tagger's words
limitations	Clipping all rewards to +1/-1 removes the agent's ability to distinguish rewards of different magnitude. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any task where reward magnitude matters: reward clipping means the agent CANNOT DIFFERENTIATE BETWEEN REWARDS OF DIFFERENT MAGNITUDE	`limitations[2]`		the tagger's words
limitations	Performance remains far below human level on games needing strategies over long time scales. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the human-comparison headline: THE GAMES Q*BERT, SEAQUEST, SPACE INVADERS, ON WHICH WE ARE FAR FROM HUMAN PERFORMANCE, ARE MORE CHALLENGING BECAUSE THEY REQUIRE ... A STRATEGY THAT EXTENDS OVER LONG TIME SCALES	`limitations[3]`		the tagger's words
limitations	The claim that one setting spans all the games is enabled by the reward change, in the sentence that follows it. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the claim that the same architecture and hyperparameters work across all seven games: the reward clipping is stated to limit the scale of the error derivatives and to make it easier to use the same learning rate across multiple games	`limitations[4]`	1.0	the tagger's words
limitations	The reward the agent learns from is not the reward it is scored on. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the scores reported for these agents: the agents were evaluated on the real and unmodified games while the reward structure was changed during training only	`limitations[5]`	1.0	the tagger's words
limitations	The agent cannot see enough of the game to know what state it is in, and the paper says so. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any description of this agent as learning from the game state: the task is partially observed, many emulator states are perceptually aliased, and the paper states it is impossible to fully understand the situation from the current screen alone	`limitations[6]`	1.0	the tagger's words
limitations	The primary evaluation metric is too noisy to display the result, so a second one is introduced. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the learning curves shown here: the averaged reward plots are described as quite noisy and as giving the impression that the algorithm is not making steady progress, which is why a more stable metric is also reported	`limitations[7]`	1.0	the tagger's words
transfers	a convolutional network taking raw pixels as input, used as the function approximator for the action-value function in reinforcement learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`	0.85	the tagger's words
controls	one set of hyperparameter values held constant across every game, with a single documented exception (k = 3, to make the lasers visible) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: per-game tuning - a score that reflects hyperparameters chosen for that game rather than a method that transfers across games · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.85	the tagger's words
concepts	experience replay Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: experience replay	`concepts[3]`		the paper's words
concepts	perceptually aliased Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: perceptually aliased	`concepts[4]`		the paper's words
concepts	evaluation metric Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: evaluation metric	`concepts[5]`		the paper's words
concepts	replay memory Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: replay memory	`concepts[6]`		the paper's words
concepts	action-value function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: action-value function	`concepts[7]`		the paper's words
concepts	frame-skipping Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: frame-skipping	`concepts[8]`		the paper's words
senses	the reward - TRANSFORMED IN TRAINING, INTACT IN EVALUATION: positive rewards are fixed to 1 and negative to -1 while learning, and the agents are scored on the real unmodified games, so the signal the agent optimises and the number reported are two different quantities Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	the state - PARTIALLY OBSERVED AND PERCEPTUALLY ALIASED: the agent sees only the current screen, many emulator states look identical to it, and the paper states outright that the situation cannot be fully understood from one screen Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	progress - MEASURED TWICE, AND THE SECOND MEASURE CHOSEN FOR LEGIBILITY: average total reward is the metric taken from the literature but is so noisy it suggests no steady progress, so the estimated action-value is reported as a more stable alternative Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → convolutional neural networks + Q-learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → Q-learning [26] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → the Arcade Learning Environment / Atari 2600 RL platform of [3] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → neural fitted Q-learning (NFQ) [20] and its autoencoder variant [12] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	extends → Q-learning combined with experience replay and a simple neural network [13] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
concept_edges	experience replay —operationalised-as→ replay memory Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 · note: RE-GROUNDS another row of this record: the old span named ONE endpoint. also in: experience replay · replay memory	`concept_edges[2]`	0.9	the tagger's words
content_tags	seven — number of games the single model was applied to Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted.	`content_tags.quantities[0]`		the tagger's words
content_tags	6 of 7 / 3 of 7 — games where DQN beat prior approaches / a human expert Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	The first deep-learning model to learn control policies directly from high-dimensional raw pixels via reinforcement learning: a CNN trained with a Q-learning variant whose output is a value function of future rewards. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted.	`content_tags.findings[0]`		the tagger's words
content_tags	raw pixels Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: raw pixels	`content_tags.variables[0]`		the tagger's words
content_tags	value function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: value function	`content_tags.variables[1]`		the tagger's words
content_tags	experience replay Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: experience replay	`content_tags.definitions[0]`		the tagger's words
content_tags	deep reinforcement learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: deep reinforcement learning	`content_tags.definitions[1]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean	`content_tags.statistics[2]`		the tagger's words
content_tags	variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variance	`content_tags.statistics[3]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

content_tags.findings[1] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
NOT MACHINE-VOUCHED: declared-ellipsis - An elided quote: its 2 pieces ('We find that it outperforms all previous' ... 'g distribution over many past behaviors.') are each in our copy of the paper, and the quote marks the cut with an ellipsis, so the joined string is in no copy. Located mechanically, piece by piece.

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.