phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | reinforcement learning from raw visual input— “Learning to control agents directly from high-dimensional sensory inputs like vision and speech is one of the long-standing challenges of reinforcement learning (RL).” | |
| claim-type | method— “We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning.” | |
| method — evidence | — “The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards.” | |
| transfer | from computer vision (convolutional neural networks): a convolutional network taking raw pixels as input, used as the function approximator for the action-value function in reinforcement learning— “The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards.” | transfers[0] |
| mechanism | experience replay: store past transitions and train on random minibatches of them, breaking the correlations between consecutive experiences and stabilizing deep Q-learning— “randomizing the samples breaks these correlations and therefore reduces the variance of the updates” | mechanisms[0] |
| builds-on | convolutional neural networks + Q-learning— “The model is a convolutional neural network, trained with a variant of Q-learning” | claim_relations[0] |
| builds-on | Q-learning [26]— “The network is trained with a variant of the Q-learning [26] algorithm, with stochastic gradient descent to update the weights.” | claim_relations[1] |
| builds-on | the Arcade Learning Environment / Atari 2600 RL platform of [3]— “The use of the Atari 2600 emulator as a reinforcement learning platform was introduced by [3], who applied standard reinforcement learning algorithms with linear function approximation and generic visual features.” | claim_relations[2] |
| extends | neural fitted Q-learning (NFQ) [20] and its autoencoder variant [12]— “In contrast our approach applies reinforcement learning end-to-end, directly from the visual inputs; as a result it may learn features that are directly relevant to discriminating action-values.” | claim_relations[3] |
| extends | Q-learning combined with experience replay and a simple neural network [13]— “Q-learning has also previously been combined with experience replay and a simple neural network [13], but again starting with a low-dimensional state rather than raw visual inputs.” | claim_relations[4] |
| sense | the reward - TRANSFORMED IN TRAINING, INTACT IN EVALUATION: positive rewards are fixed to 1 and negative to -1 while learning, and the agents are scored on the real unmodified games, so the signal the agent optimises and the number reported are two different quantities— “While we evaluated our agents on the real and unmodified games, we made one change to the reward structure of the games during training only. Since the scale of scores varies greatly from game to game, we fixed all positive rewards to be 1 and all negative rewards to be −1, leaving 0 rewards unchanged” | senses[0] |
| sense | the state - PARTIALLY OBSERVED AND PERCEPTUALLY ALIASED: the agent sees only the current screen, many emulator states look identical to it, and the paper states outright that the situation cannot be fully understood from one screen— “Since the agent only observes images of the current screen, the task is partially observed and many emulator states are perceptually aliased, i.e. it is impossible to fully understand the current situation from only the current screen” | senses[1] |
| sense | progress - MEASURED TWICE, AND THE SECOND MEASURE CHOSEN FOR LEGIBILITY: average total reward is the metric taken from the literature but is so noisy it suggests no steady progress, so the estimated action-value is reported as a more stable alternative— “Both averaged reward plots are indeed quite noisy, giving one the impression that the learning algorithm is not making steady progress. Another, more stable, metric is the policy’s estimated action-value function Q, which provides an estimate of how much discounted reward the agent can obtain by following its policy from any given state” | senses[2] |
| control | one set of hyperparameter values held constant across every game, with a single documented exception (k = 3, to make the lasers visible) — held — rules out per-game tuning - a score that reflects hyperparameters chosen for that game rather than a method that transfers across games— “We used k = 3 to make the lasers visible and this change was the only difference in hyperparameter values between any of the games.” | controls[0] |
| note | experience replay —operationalised-as→ replay memory — RE-GROUNDS another row of this record: the old span named ONE endpoint.— “In contrast to TD-Gammon and similar online approaches, we utilize a technique known as experience replay [13] where we store the agent’s experiences at each time-step, et = (st, at, rt, st+1) in a data-set D = e1, ..., eN , pooled over many episodes into a replay memory.” | concept_edges[2] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — the transfer of supervised deep learning intuitions to this setting: REINFORCEMENT LEARNING PRESENTS SEVERAL CHALLENGES FROM A DEEP LEARNING PERSPECTIVE | limitations[0] |
| for which use | limitation — the sample efficiency and stability results: THE MEMORY BUFFER DOES NOT DIFFERENTIATE IMPORTANT TRANSITIONS AND ALWAYS OVERWRITES WITH RECENT TRANSITIONS DUE TO THE FINITE MEMORY SIZE N | limitations[1] |
| for which use | limitation — any task where reward magnitude matters: reward clipping means the agent CANNOT DIFFERENTIATE BETWEEN REWARDS OF DIFFERENT MAGNITUDE Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| for which use | limitation — the human-comparison headline: THE GAMES Q*BERT, SEAQUEST, SPACE INVADERS, ON WHICH WE ARE FAR FROM HUMAN PERFORMANCE, ARE MORE CHALLENGING BECAUSE THEY REQUIRE ... A STRATEGY THAT EXTENDS OVER LONG TIME SCALES Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| for which use | limitation — the claim that the same architecture and hyperparameters work across all seven games: the reward clipping is stated to limit the scale of the error derivatives and to make it easier to use the same learning rate across multiple games Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| for which use | limitation — the scores reported for these agents: the agents were evaluated on the real and unmodified games while the reward structure was changed during training only Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| for which use | limitation — any description of this agent as learning from the game state: the task is partially observed, many emulator states are perceptually aliased, and the paper states it is impossible to fully understand the situation from the current screen alone Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[6] |
| for which use | limitation — the learning curves shown here: the averaged reward plots are described as quite noisy and as giving the impression that the algorithm is not making steady progress, which is why a more stable metric is also reported Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[7] |
| replication | untested | |
| limitation | reinforcement learning presents several challenges from a deep-learning perspective Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| limitation | The replay memory is uniformly sampled and finite, so it neither prioritises informative transitions nor retains old ones. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| limitation | Clipping all rewards to +1/-1 removes the agent's ability to distinguish rewards of different magnitude. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| limitation | Performance remains far below human level on games needing strategies over long time scales. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| limitation | The claim that one setting spans all the games is enabled by the reward change, in the sentence that follows it. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| limitation | The reward the agent learns from is not the reward it is scored on. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| limitation | The agent cannot see enough of the game to know what state it is in, and the paper says so. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[6] |
| limitation | The primary evaluation metric is too noisy to display the result, so a second one is introduced. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[7] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | The first deep-learning model to learn control policies directly from high-dimensional raw pixels via reinforcement learning: a CNN trained with a Q-learning variant whose output is a value function of future rewards. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[0] |
| evidence status note | finding — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.findings[0] |
| finding | It outperformed all previous approaches on six of the games and surpassed a human expert on three; experience replay randomly samples previous transitions to smooth the training distribution. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[1] |
| not machine-vouched | finding — declared-ellipsis - An elided quote: its 2 pieces ('We find that it outperforms all previous' ... 'g distribution over many past behaviors.') are each in our copy of the paper, and the quote marks the cut with an ellipsis, so the joined string is in no copy. Located mechanically, piece by piece. | content_tags.findings[1] |
| quantity | seven Atari 2600 games — number of games the single model was applied to Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[0] |
| evidence status note | quantity — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.quantities[0] |
| quantity | 6 of 7 / 3 of 7 — games where DQN beat prior approaches / a human expert Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.quantities[1] |
| variable | raw pixels (independent) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.variables[0] |
| variable | value function (dependent) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.variables[1] |
| definition | experience replay Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[0] |
| definition | deep reinforcement learning Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[1] |
| key-terms | reinforcement, learning, Q-learning, pixels, reward, value, games, convolutional, policy |
| concept | experience replay Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[3] |
| concept | perceptually aliased Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[4] |
| concept | evaluation metric Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[5] |
| concept | replay memory Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[6] |
| concept | action-value function Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[7] |
| concept | frame-skipping Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[8] |
| mean μ | as written: average of Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[2] |
| variance σ² | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[3] |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824