A Tour of Reinforcement Learning: The View from Continuous Control

A Tour of Reinforcement Learning: The View from Continuous Control Benjamin Recht

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

A Tour of Reinforcement Learning: The View from Continuous Control 2019 Annual Review of Control, Robotics, and Autonomous Systems / arXiv 10.1146/annurev-control-053018-023825 1806.09460 recht2019-continuous-control recht2019-continuous-control.txt, the project's staged copy baa933675a51fbdf0b7975b31a59ea3e11d33487 arxiv-open control-theory

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them critique Quotation withheld because this record is not cleared for redistribution. reinforcement learning for continuous control Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. mechanisms A DEFINITION IS OFFERED AND THE OBJECTION TO IT IS VOICED IN THE SAME BREATH, IN ANOTHER FIELD'S VOICE: a control engineer MIGHT BE PUZZLED by the definition and interject that this is PRECISELY THE SCOPE OF CONTROL THEORY - and the consequence named is that the two communities remaining PRACTICALLY DISJOINT has led to the CO-DEVELOPMENT OF VASTLY DIFFERENT APPROACHES TO THE SAME PROBLEMS Quotation withheld because this record is not cleared for redistribution. mechanisms THE SURVEY'S STATED PURPOSE IS TO SUPPLY A LANGUAGE, NOT A RESULT: it aims to provide A LANGUAGE for the two communities TO BEGIN COMMUNICATING, highlighting what each can learn from the other. The deliverable is a shared vocabulary between fields that already work on the same problems Quotation withheld because this record is not cleared for redistribution. mechanisms EACH FIELD IS CHARACTERISED IN A SINGLE CLAUSE AND THE CONTRAST IS PUT IN THE MODELS: controls is the theory of designing complex actions FROM WELL-SPECIFIED MODELS, while reinforcement learning often makes intricate MODEL-FREE predictions FROM DATA ALONE - and the sentence immediately after says that YET BOTH aim at systems doing the same three things Quotation withheld because this record is not cleared for redistribution. mechanisms ONE FIELD IS DEFINED AS ANOTHER FIELD'S PROBLEM WITH A SINGLE CONDITION CHANGED: the main view of the survey is of Quotation withheld - so the whole of one discipline is recast as a special case of another, and the difference between them is reduced to what is known about the system Quotation withheld because this record is not cleared for redistribution. mechanisms THE EVALUATION WORD IS DECLARED NOT CLEARLY DEFINED AND THE COMPETING OPERATIONALISATIONS ARE SPELLED OUT: is an algorithm BEST if it crosses some reward threshold in the fewest samples, or if it achieves the highest reward given a fixed budget of samples, OR MAYBE THERE IS A MIDDLE GROUND. The word by which methods are ranked is shown to have at least three readings that would rank them differently Quotation withheld because this record is not cleared for redistribution. mechanisms THE SAME OPERATION HAS A DIFFERENT NAME IN THE OTHER FIELD, AND THE DIFFERENCES THAT SURVIVE THE TRANSLATION ARE NAMED: estimation of dynamical systems IS CALLED SYSTEM IDENTIFICATION IN THE CONTROL COMMUNITY - and it DIFFERS from conventional estimation because the inputs must be chosen to excite various degrees of freedom and because dynamical outputs are correlated. The translation is given AND its residue is stated about this quote: EXTENDED, not trimmed: the first cut stopped one clause short of "and because dynamical outputs are correlated", which the tag depends on. Where the next clause carries the claim, widening the span beats weakening the tag. Quotation withheld because this record is not cleared for redistribution. mechanisms A SHORT NAME IS CHOSEN OVER THE COMMUNITY'S OWN, AND THE COMMUNITY'S IS CALLED VERBOSE: the estimated model is called THE NOMINAL MODEL and designing control as if it were true is called NOMINAL CONTROL - which is COMMONLY VERBOSELY REFERRED TO AS control under the principle of certainty equivalence. The author renames an existing concept for economy and says that is what he is doing Quotation withheld because this record is not cleared for redistribution. mechanisms A FIELD'S OWN CATEGORY TERM IS PUT IN SCARE QUOTES WHILE ITS CONTENTS ARE TESTED: the SO-CALLED MODEL-FREE methods popular in deep reinforcement learning are found CONSIDERABLY LESS EFFECTIVE in both theory and practice than simple model-based schemes on the baseline problem - so the label is held at arm's length in the same sentence that reports the result Quotation withheld because this record is not cleared for redistribution. mechanisms A FIELD'S CENTRAL DICHOTOMY IS ANNOUNCED AS SOMETHING TO BE DISSOLVED RATHER THAN ADJUDICATED: having laid out the split between model-free and model-based methodologies, the author states that he will argue the two perspectives CAN BE UNIFIED, COMBINING THEIR RELATIVE MERITS - the contribution is to remove the opposition, not to win it Quotation withheld because this record is not cleared for redistribution. mechanisms THE COMPARISON IS MADE POSSIBLE BY FORCING BOTH FIELDS ONTO ONE SHARED PROBLEM: the two sets of techniques are put ON THE SAME FOOTING through a case study of the linear quadratic regulator with unknown dynamics, and that baseline is what is said to ILLUMINATE THE VARIOUS TRADE-OFFS. A common benchmark is the instrument that makes two vocabularies commensurable Quotation withheld because this record is not cleared for redistribution. mechanisms THE ORACLE MODEL IS NAMED, AND ITS COMPLEXITY RELATIVE TO THE NEIGHBOURING FIELD IS STATED: the query protocol IS CALLED EPISODIC REINFORCEMENT LEARNING, and it is described as CONSIDERABLY MORE COMPLICATED than the oracle models typically considered in optimization, because each episode returns a complex feedback signal of states and rewards Quotation withheld because this record is not cleared for redistribution. mechanisms A WHOLE FIELD'S COLLOQUIAL MEANING IS NAMED AND THEN SET ASIDE: most of the time when the term MACHINE LEARNING is used colloquially it refers to PREDICTION, and from that perspective even niche topics are prediction tasks - after which the paper states what makes its own subject different, that there are TWO SPECIAL VARIABLES Quotation withheld because this record is not cleared for redistribution. significant challenges remain at the control-learning interface what this limits: the practical readiness of the whole area: significant CHALLENGES AT THE CONTROL-LEARNING INTERFACE remain Quotation withheld because this record is not cleared for redistribution. The survey deliberately stays at the level of conceptual ideas and omits the technical detail needed to make its statements precise. what this limits: the implementability of anything described: MY AIM WILL BE TO HIGHLIGHT THE MAIN CONCEPTUAL IDEAS ... AND TO AVOID EMBROILING MYSELF IN A THOROUGH DISCUSSION OF THE MYRIAD OF TECHNICAL DETAILS Quotation withheld because this record is not cleared for redistribution. The scope is restricted to episodic RL and excludes the harder adaptive-control setting. what this limits: the scope of every conclusion: THIS SURVEY HAS FOCUSED ON EPISODIC REINFORCEMENT LEARNING AND HAS STEERED CLEAR OF A MUCH HARDER PROBLEM: ADAPTIVE CONTROL Quotation withheld because this record is not cleared for redistribution. The theoretical comparison between value-function estimation and model estimation rests on worst-case upper bounds, which the author concedes is not a valid comparison. what this limits: the model-versus-value comparison: WHILE COMPARING WORST-CASE UPPER BOUNDS IS CERTAINLY NOT VALID, IT IS SUGGESTIVE THAT - declared invalid and offered as suggestive only Quotation withheld because this record is not cleared for redistribution. Function approximation, and deep RL specifically, is set aside rather than analysed. what this limits: any reading of the survey as covering deep RL: I HAVE SPENT NO TIME DISCUSSING DEEP REINFORCEMENT LEARNING Quotation withheld because this record is not cleared for redistribution. The empirical evidence rests on a small set of simulation benchmarks that invite overfitting. what this limits: the empirical evidence for every method surveyed: HAVING ONLY A FEW SIMULATION BENCHMARKS ENCOURAGES OVERFITTING TO THESE BENCHMARKS Quotation withheld because this record is not cleared for redistribution. The label model-free does not mean the method is free of modelling; the author says modelling is heavily built into its assumptions. about this quote: THE '7' IN THE QUOTE IS NOT A WORD OF THE SENTENCE - it is layout furniture (a page number or note mark) that this copy leaves between 'that' and 'modeling'. · what this limits: a comparison presented as model-free versus model-based: the author states that modelling is heavily built into the assumptions of model-free RL algorithms, so the label does not mark the absence of a model Quotation withheld because this record is not cleared for redistribution. A discounting device had to be introduced to obtain a tractable equation, and the author names that as troubling. what this limits: results that depend on the discounted formulation: the discount factor was introduced in order to obtain a simple Bellman equation, which the author flags as troubling rather than as a property of the problem Quotation withheld because this record is not cleared for redistribution. The scope claimed for the field is stated to be precisely the scope of a different, older field. what this limits: a definition of reinforcement learning as the study of learning to act: a control engineer would say this is precisely the scope of control theory, so the boundary between the two fields is a matter of vocabulary rather than of subject Quotation withheld because this record is not cleared for redistribution. framing and analyzing reinforcement learning with the tools of optimal control, using the Linear Quadratic Regulator as the baseline Quotation withheld because this record is not cleared for redistribution. reinforcement learning reinforcement learning Quotation withheld because this record is not cleared for redistribution. random search random search Quotation withheld because this record is not cleared for redistribution. Evolution Strategy Evolution Strategy Quotation withheld because this record is not cleared for redistribution. discount factor discount factor Quotation withheld because this record is not cleared for redistribution. Bellman equation Bellman equation Quotation withheld because this record is not cleared for redistribution. random search - THE SAME METHOD UNDER FOUR NAMES: pure random search in RL, a (mu,lambda) Evolution Strategy in evolutionary computation, and studied again under stochastic approximation and under bandits - four literatures, one procedure, and this paper is where they are listed Quotation withheld because this record is not cleared for redistribution. model-free - A LABEL THAT DENIES WHAT THE METHOD CARRIES: the name marks the absence of an explicit dynamics model while the modelling assumptions remain, built into the algorithm Quotation withheld because this record is not cleared for redistribution. state-action pair - TWO NOTATIONS FOR ONE OBJECT: (x, u) in control and (s, a) in reinforcement learning, reconciled here by the author choosing one on purpose and saying so Quotation withheld because this record is not cleared for redistribution. builds-on → reinforcement learning and optimal control theory Quotation withheld because this record is not cleared for redistribution. builds-on → the System Level Synthesis (SLS) framework [50, 84] Quotation withheld because this record is not cleared for redistribution. extends → the high-variance findings of Henderson et al. and Islam et al. for deep RL [35, 36] Quotation withheld because this record is not cleared for redistribution. random search —renamed-across-fields→ Evolution Strategy random search Evolution Strategy strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. discount factor —causes→ Bellman equation discount factor Bellman equation strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the discount factor had to be introduced IN ORDER TO get a simple Bellman equation; causes is loose. Quotation withheld because this record is not cleared for redistribution. 2 — the extra variables u and r distinguishing RL Quotation withheld because this record is not cleared for redistribution. N — horizon over which expected reward is maximized Quotation withheld because this record is not cleared for redistribution. mL — total cost of m queries at horizon length L Quotation withheld because this record is not cleared for redistribution. model-free RL is less effective than simple model-based schemes on LQR Quotation withheld because this record is not cleared for redistribution. RL is best viewed as optimal control when the dynamics are unknown Quotation withheld because this record is not cleared for redistribution. RL is harder than supervised learning but can be more valuable Quotation withheld because this record is not cleared for redistribution. model-free and model-based perspectives can be unified Quotation withheld because this record is not cleared for redistribution. policy policy Quotation withheld because this record is not cleared for redistribution. reward reward Quotation withheld because this record is not cleared for redistribution. horizon length L horizon length L Quotation withheld because this record is not cleared for redistribution. state of the system state of the system Quotation withheld because this record is not cleared for redistribution. control action control action Quotation withheld because this record is not cleared for redistribution. disturbance disturbance Quotation withheld because this record is not cleared for redistribution. reinforcement learning (RL) reinforcement learning (RL) Quotation withheld because this record is not cleared for redistribution. control policy control policy Quotation withheld because this record is not cleared for redistribution. trajectory trajectory Quotation withheld because this record is not cleared for redistribution. episodic reinforcement learning episodic reinforcement learning Quotation withheld because this record is not cleared for redistribution. bootstrap bootstrap Quotation withheld because this record is not cleared for redistribution. regression regression about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. variance variance about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1146/annurev-control-053018-023825.