Playing Atari with Deep Reinforcement Learning

Playing Atari with Deep Reinforcement Learning Volodymyr Mnih Koray Kavukcuoglu David Silver et al.

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Playing Atari with Deep Reinforcement Learning 2013 NeurIPS Deep Learning Workshop / arXiv 1312.5602 mnih2013-dqn mnih2013-dqn.txt, the project's staged copy e11591c9e31dbd181981f09f705b3dceedf06db1 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. reinforcement learning from raw visual input Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms experience replay: store past transitions and train on random minibatches of them, breaking the correlations between consecutive experiences and stabilizing deep Q-learning Quotation withheld because this record is not cleared for redistribution. reinforcement learning presents several challenges from a deep-learning perspective what this limits: the transfer of supervised deep learning intuitions to this setting: REINFORCEMENT LEARNING PRESENTS SEVERAL CHALLENGES FROM A DEEP LEARNING PERSPECTIVE Quotation withheld because this record is not cleared for redistribution. The replay memory is uniformly sampled and finite, so it neither prioritises informative transitions nor retains old ones. what this limits: the sample efficiency and stability results: THE MEMORY BUFFER DOES NOT DIFFERENTIATE IMPORTANT TRANSITIONS AND ALWAYS OVERWRITES WITH RECENT TRANSITIONS DUE TO THE FINITE MEMORY SIZE N Quotation withheld because this record is not cleared for redistribution. Clipping all rewards to +1/-1 removes the agent's ability to distinguish rewards of different magnitude. what this limits: any task where reward magnitude matters: reward clipping means the agent CANNOT DIFFERENTIATE BETWEEN REWARDS OF DIFFERENT MAGNITUDE Quotation withheld because this record is not cleared for redistribution. Performance remains far below human level on games needing strategies over long time scales. what this limits: the human-comparison headline: THE GAMES Q*BERT, SEAQUEST, SPACE INVADERS, ON WHICH WE ARE FAR FROM HUMAN PERFORMANCE, ARE MORE CHALLENGING BECAUSE THEY REQUIRE ... A STRATEGY THAT EXTENDS OVER LONG TIME SCALES Quotation withheld because this record is not cleared for redistribution. The claim that one setting spans all the games is enabled by the reward change, in the sentence that follows it. what this limits: the claim that the same architecture and hyperparameters work across all seven games: the reward clipping is stated to limit the scale of the error derivatives and to make it easier to use the same learning rate across multiple games Quotation withheld because this record is not cleared for redistribution. The reward the agent learns from is not the reward it is scored on. what this limits: the scores reported for these agents: the agents were evaluated on the real and unmodified games while the reward structure was changed during training only Quotation withheld because this record is not cleared for redistribution. The agent cannot see enough of the game to know what state it is in, and the paper says so. what this limits: any description of this agent as learning from the game state: the task is partially observed, many emulator states are perceptually aliased, and the paper states it is impossible to fully understand the situation from the current screen alone Quotation withheld because this record is not cleared for redistribution. The primary evaluation metric is too noisy to display the result, so a second one is introduced. what this limits: the learning curves shown here: the averaged reward plots are described as quite noisy and as giving the impression that the algorithm is not making steady progress, which is why a more stable metric is also reported Quotation withheld because this record is not cleared for redistribution. a convolutional network taking raw pixels as input, used as the function approximator for the action-value function in reinforcement learning Quotation withheld because this record is not cleared for redistribution. one set of hyperparameter values held constant across every game, with a single documented exception (k = 3, to make the lasers visible) eliminates: per-game tuning - a score that reflects hyperparameters chosen for that game rather than a method that transfers across games · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. experience replay experience replay Quotation withheld because this record is not cleared for redistribution. perceptually aliased perceptually aliased Quotation withheld because this record is not cleared for redistribution. evaluation metric evaluation metric Quotation withheld because this record is not cleared for redistribution. replay memory replay memory Quotation withheld because this record is not cleared for redistribution. action-value function action-value function Quotation withheld because this record is not cleared for redistribution. frame-skipping frame-skipping Quotation withheld because this record is not cleared for redistribution. the reward - TRANSFORMED IN TRAINING, INTACT IN EVALUATION: positive rewards are fixed to 1 and negative to -1 while learning, and the agents are scored on the real unmodified games, so the signal the agent optimises and the number reported are two different quantities Quotation withheld because this record is not cleared for redistribution. the state - PARTIALLY OBSERVED AND PERCEPTUALLY ALIASED: the agent sees only the current screen, many emulator states look identical to it, and the paper states outright that the situation cannot be fully understood from one screen Quotation withheld because this record is not cleared for redistribution. progress - MEASURED TWICE, AND THE SECOND MEASURE CHOSEN FOR LEGIBILITY: average total reward is the metric taken from the literature but is so noisy it suggests no steady progress, so the estimated action-value is reported as a more stable alternative Quotation withheld because this record is not cleared for redistribution. builds-on → convolutional neural networks + Q-learning Quotation withheld because this record is not cleared for redistribution. builds-on → Q-learning [26] Quotation withheld because this record is not cleared for redistribution. builds-on → the Arcade Learning Environment / Atari 2600 RL platform of [3] Quotation withheld because this record is not cleared for redistribution. extends → neural fitted Q-learning (NFQ) [20] and its autoencoder variant [12] Quotation withheld because this record is not cleared for redistribution. extends → Q-learning combined with experience replay and a simple neural network [13] Quotation withheld because this record is not cleared for redistribution. experience replay —operationalised-as→ replay memory experience replay replay memory strength of the relation: 1 · note: RE-GROUNDS another row of this record: the old span named ONE endpoint. Quotation withheld because this record is not cleared for redistribution. seven — number of games the single model was applied to about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. 6 of 7 / 3 of 7 — games where DQN beat prior approaches / a human expert Quotation withheld because this record is not cleared for redistribution. The first deep-learning model to learn control policies directly from high-dimensional raw pixels via reinforcement learning: a CNN trained with a Q-learning variant whose output is a value function of future rewards. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. raw pixels raw pixels Quotation withheld because this record is not cleared for redistribution. value function value function Quotation withheld because this record is not cleared for redistribution. experience replay experience replay Quotation withheld because this record is not cleared for redistribution. deep reinforcement learning deep reinforcement learning Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1312.5602.