A Tour of Reinforcement Learning: The View from Continuous Control

Benjamin Recht · Annual Review of Control, Robotics, and Autonomous Systems / arXiv · 2019 · control-theory · 10.1146/annurev-control-053018-023825 · record recht2019-continuous-control
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim type is 'critique' rather than 'review' (not an option) because the distinctive contribution is the case against model-free RL for continuous control, not the survey material itself. method is recorded as 'review' from the paper's self-description, but the paper also contributes original numerical comparisons on LQR instances: 'In this section, we implement these various methods and test them on some simple LQR instances to see how these theoretical predictions reflect practice.' discipline sits at the control/machine-learning boundary; 'control-theory' follows the paper's declared vantage point and its venue. A third candidate relation to Rajeswaran et al [59] was left out because the paper names the prior work ('In another piece of great work, Rajeswaran et al showed that Natural Policy Gradient could learn linear policies that could complete these benchmarks [59].') without stating in a single sentence how its own result stands to it. Replication status is 'untested': no external replication of the paper's own comparisons is reported. The extracted text contains PDF-joining artefacts such as 'offfrom' and 'trade-offthat'; quotes were chosen to avoid them. 2026-08-14 and 2026-08-17 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

Some quotes on this page contain (cid:NN). The PDF this text was extracted from encodes 13 characters in a way that names no letter, so the extractor could not say what they were. The quotes keep them because a quote is reproduced exactly or it proves nothing — the marks are the source’s, not the tagging’s.

This record’s own method is review, and the tags below follow from that. A review’s findings, quantities and statistics are things stated IN this paper, and some of them report results obtained elsewhere. No tag on this page names a different author. Measured across every review in this corpus, 81% of a review’s tags carry no citation at all — most of them are the review speaking. The corpus does not adjudicate the rest: read an unmarked tag as a claim this paper makes, not necessarily one it made first.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
reward 16 fields · bootstrap 2 fields · trajectory 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 9 concept tags on this paper, 5 are grounded in a quoted sentence.

reinforcement learning optimal control linear-quadratic regulator dynamical system policy optimization random search Evolution Strategy discount factor Bellman equation A DEFINITION IS OFFERED AND THE OBJECTION TO IT IS VOICED IN THE SAME BREATH, IN ANOTHER FIELD'S VOICE: a control engineer MIGHT BE PUZZLED by the definition and interject that this is PRECISELY THE SCOPE OF CONTROL THEORY - and the consequence named is that the two communities remaining PRACTICALLY DISJOINT has led to the CO-DEVELOPMENT OF VASTLY DIFFERENT APPROACHES TO THE SAME PROBLEMS THE SURVEY'S STATED PURPOSE IS TO SUPPLY A LANGUAGE, NOT A RESULT: it aims to provide A LANGUAGE for the two communities TO BEGIN COMMUNICATING, highlighting what each can learn from the other. The deliverable is a shared vocabulary between fields that already work on the same problems EACH FIELD IS CHARACTERISED IN A SINGLE CLAUSE AND THE CONTRAST IS PUT IN THE MODELS: controls is the theory of designing complex actions FROM WELL-SPECIFIED MODELS, while reinforcement learning often makes intricate MODEL-FREE predictions FROM DATA ALONE - and the sentence immediately after says that YET BOTH aim at systems doing the same three things ONE FIELD IS DEFINED AS ANOTHER FIELD'S PROBLEM WITH A SINGLE CONDITION CHANGED: the main view of the survey is of Quotation withheld - so the whole of one discipline is recast as a special case of another, and the difference between them is reduced to what is known about the system THE EVALUATION WORD IS DECLARED NOT CLEARLY DEFINED AND THE COMPETING OPERATIONALISATIONS ARE SPELLED OUT: is an algorithm BEST if it crosses some reward threshold in the fewest samples, or if it achieves the highest reward given a fixed budget of samples, OR MAYBE THERE IS A MIDDLE GROUND. The word by which methods are ranked is shown to have at least three readings that would rank them differently THE SAME OPERATION HAS A DIFFERENT NAME IN THE OTHER FIELD, AND THE DIFFERENCES THAT SURVIVE THE TRANSLATION ARE NAMED: estimation of dynamical systems IS CALLED SYSTEM IDENTIFICATION IN THE CONTROL COMMUNITY - and it DIFFERS from conventional estimation because the inputs must be chosen to excite various degrees of freedom and because dynamical outputs are correlated. The translation is given AND its residue is stated A SHORT NAME IS CHOSEN OVER THE COMMUNITY'S OWN, AND THE COMMUNITY'S IS CALLED VERBOSE: the estimated model is called THE NOMINAL MODEL and designing control as if it were true is called NOMINAL CONTROL - which is COMMONLY VERBOSELY REFERRED TO AS control under the principle of certainty equivalence. The author renames an existing concept for economy and says that is what he is doing A FIELD'S OWN CATEGORY TERM IS PUT IN SCARE QUOTES WHILE ITS CONTENTS ARE TESTED: the SO-CALLED MODEL-FREE methods popular in deep reinforcement learning are found CONSIDERABLY LESS EFFECTIVE in both theory and practice than simple model-based schemes on the baseline problem - so the label is held at arm's length in the same sentence that reports the result A FIELD'S CENTRAL DICHOTOMY IS ANNOUNCED AS SOMETHING TO BE DISSOLVED RATHER THAN ADJUDICATED: having laid out the split between model-free and model-based methodologies, the author states that he will argue the two perspectives CAN BE UNIFIED, COMBINING THEIR RELATIVE MERITS - the contribution is to remove the opposition, not to win it THE COMPARISON IS MADE POSSIBLE BY FORCING BOTH FIELDS ONTO ONE SHARED PROBLEM: the two sets of techniques are put ON THE SAME FOOTING through a case study of the linear quadratic regulator with unknown dynamics, and that baseline is what is said to ILLUMINATE THE VARIOUS TRADE-OFFS. A common benchmark is the instrument that makes two vocabularies commensurable THE ORACLE MODEL IS NAMED, AND ITS COMPLEXITY RELATIVE TO THE NEIGHBOURING FIELD IS STATED: the query protocol IS CALLED EPISODIC REINFORCEMENT LEARNING, and it is described as CONSIDERABLY MORE COMPLICATED than the oracle models typically considered in optimization, because each episode returns a complex feedback signal of states and rewards A WHOLE FIELD'S COLLOQUIAL MEANING IS NAMED AND THEN SET ASIDE: most of the time when the term MACHINE LEARNING is used colloquially it refers to PREDICTION, and from that perspective even niche topics are prediction tasks - after which the paper states what makes its own subject different, that there are TWO SPECIAL VARIABLES continuous control policy linear quadratic regulator model-free model-based trajectory reward control action reinforcement learning (RL) control policy episodic reinforcement learning horizon length L state of the system disturbance bootstrap regression least squares variance review

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.1146/annurev-control-053018-023825. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 3 · claim_type 1 · concept_edges 2 · concepts 5 · content_tags 20 · limitations 9 · mechanisms 12 · method 1 · phenomenon 1 · senses 3 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 2 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	critique Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	reinforcement learning for continuous control Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	review Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: review	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A DEFINITION IS OFFERED AND THE OBJECTION TO IT IS VOICED IN THE SAME BREATH, IN ANOTHER FIELD'S VOICE: a control engineer MIGHT BE PUZZLED by the definition and interject that this is PRECISELY THE SCOPE OF CONTROL THEORY - and the consequence named is that the two communities remaining PRACTICALLY DISJOINT has led to the CO-DEVELOPMENT OF VASTLY DIFFERENT APPROACHES TO THE SAME PROBLEMS	`mechanisms[0]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE SURVEY'S STATED PURPOSE IS TO SUPPLY A LANGUAGE, NOT A RESULT: it aims to provide A LANGUAGE for the two communities TO BEGIN COMMUNICATING, highlighting what each can learn from the other. The deliverable is a shared vocabulary between fields that already work on the same problems	`mechanisms[1]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: EACH FIELD IS CHARACTERISED IN A SINGLE CLAUSE AND THE CONTRAST IS PUT IN THE MODELS: controls is the theory of designing complex actions FROM WELL-SPECIFIED MODELS, while reinforcement learning often makes intricate MODEL-FREE predictions FROM DATA ALONE - and the sentence immediately after says that YET BOTH aim at systems doing the same three things	`mechanisms[2]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: ONE FIELD IS DEFINED AS ANOTHER FIELD'S PROBLEM WITH A SINGLE CONDITION CHANGED: the main view of the survey is of Quotation withheld - so the whole of one discipline is recast as a special case of another, and the difference between them is reduced to what is known about the system	`mechanisms[3]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE EVALUATION WORD IS DECLARED NOT CLEARLY DEFINED AND THE COMPETING OPERATIONALISATIONS ARE SPELLED OUT: is an algorithm BEST if it crosses some reward threshold in the fewest samples, or if it achieves the highest reward given a fixed budget of samples, OR MAYBE THERE IS A MIDDLE GROUND. The word by which methods are ranked is shown to have at least three readings that would rank them differently	`mechanisms[4]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: EXTENDED, not trimmed: the first cut stopped one clause short of "and because dynamical outputs are correlated", which the tag depends on. Where the next clause carries the claim, widening the span beats weakening the tag. also in: THE SAME OPERATION HAS A DIFFERENT NAME IN THE OTHER FIELD, AND THE DIFFERENCES THAT SURVIVE THE TRANSLATION ARE NAMED: estimation of dynamical systems IS CALLED SYSTEM IDENTIFICATION IN THE CONTROL COMMUNITY - and it DIFFERS from conventional estimation because the inputs must be chosen to excite various degrees of freedom and because dynamical outputs are correlated. The translation is given AND its residue is stated	`mechanisms[5]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A SHORT NAME IS CHOSEN OVER THE COMMUNITY'S OWN, AND THE COMMUNITY'S IS CALLED VERBOSE: the estimated model is called THE NOMINAL MODEL and designing control as if it were true is called NOMINAL CONTROL - which is COMMONLY VERBOSELY REFERRED TO AS control under the principle of certainty equivalence. The author renames an existing concept for economy and says that is what he is doing	`mechanisms[6]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A FIELD'S OWN CATEGORY TERM IS PUT IN SCARE QUOTES WHILE ITS CONTENTS ARE TESTED: the SO-CALLED MODEL-FREE methods popular in deep reinforcement learning are found CONSIDERABLY LESS EFFECTIVE in both theory and practice than simple model-based schemes on the baseline problem - so the label is held at arm's length in the same sentence that reports the result	`mechanisms[7]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A FIELD'S CENTRAL DICHOTOMY IS ANNOUNCED AS SOMETHING TO BE DISSOLVED RATHER THAN ADJUDICATED: having laid out the split between model-free and model-based methodologies, the author states that he will argue the two perspectives CAN BE UNIFIED, COMBINING THEIR RELATIVE MERITS - the contribution is to remove the opposition, not to win it	`mechanisms[8]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE COMPARISON IS MADE POSSIBLE BY FORCING BOTH FIELDS ONTO ONE SHARED PROBLEM: the two sets of techniques are put ON THE SAME FOOTING through a case study of the linear quadratic regulator with unknown dynamics, and that baseline is what is said to ILLUMINATE THE VARIOUS TRADE-OFFS. A common benchmark is the instrument that makes two vocabularies commensurable	`mechanisms[9]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE ORACLE MODEL IS NAMED, AND ITS COMPLEXITY RELATIVE TO THE NEIGHBOURING FIELD IS STATED: the query protocol IS CALLED EPISODIC REINFORCEMENT LEARNING, and it is described as CONSIDERABLY MORE COMPLICATED than the oracle models typically considered in optimization, because each episode returns a complex feedback signal of states and rewards	`mechanisms[10]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A WHOLE FIELD'S COLLOQUIAL MEANING IS NAMED AND THEN SET ASIDE: most of the time when the term MACHINE LEARNING is used colloquially it refers to PREDICTION, and from that perspective even niche topics are prediction tasks - after which the paper states what makes its own subject different, that there are TWO SPECIAL VARIABLES	`mechanisms[11]`	0.95	the tagger's words
limitations	significant challenges remain at the control-learning interface Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the practical readiness of the whole area: significant CHALLENGES AT THE CONTROL-LEARNING INTERFACE remain	`limitations[0]`		the tagger's words
limitations	The survey deliberately stays at the level of conceptual ideas and omits the technical detail needed to make its statements precise. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the implementability of anything described: MY AIM WILL BE TO HIGHLIGHT THE MAIN CONCEPTUAL IDEAS ... AND TO AVOID EMBROILING MYSELF IN A THOROUGH DISCUSSION OF THE MYRIAD OF TECHNICAL DETAILS	`limitations[1]`		the tagger's words
limitations	The scope is restricted to episodic RL and excludes the harder adaptive-control setting. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the scope of every conclusion: THIS SURVEY HAS FOCUSED ON EPISODIC REINFORCEMENT LEARNING AND HAS STEERED CLEAR OF A MUCH HARDER PROBLEM: ADAPTIVE CONTROL	`limitations[2]`		the tagger's words
limitations	The theoretical comparison between value-function estimation and model estimation rests on worst-case upper bounds, which the author concedes is not a valid comparison. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the model-versus-value comparison: WHILE COMPARING WORST-CASE UPPER BOUNDS IS CERTAINLY NOT VALID, IT IS SUGGESTIVE THAT - declared invalid and offered as suggestive only	`limitations[3]`		the tagger's words
limitations	Function approximation, and deep RL specifically, is set aside rather than analysed. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any reading of the survey as covering deep RL: I HAVE SPENT NO TIME DISCUSSING DEEP REINFORCEMENT LEARNING	`limitations[4]`		the tagger's words
limitations	The empirical evidence rests on a small set of simulation benchmarks that invite overfitting. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the empirical evidence for every method surveyed: HAVING ONLY A FEW SIMULATION BENCHMARKS ENCOURAGES OVERFITTING TO THESE BENCHMARKS	`limitations[5]`		the tagger's words
limitations	The label model-free does not mean the method is free of modelling; the author says modelling is heavily built into its assumptions. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: THE '7' IN THE QUOTE IS NOT A WORD OF THE SENTENCE - it is layout furniture (a page number or note mark) that this copy leaves between 'that' and 'modeling'. · what this limits: a comparison presented as model-free versus model-based: the author states that modelling is heavily built into the assumptions of model-free RL algorithms, so the label does not mark the absence of a model	`limitations[6]`	1.0	the tagger's words
limitations	A discounting device had to be introduced to obtain a tractable equation, and the author names that as troubling. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: results that depend on the discounted formulation: the discount factor was introduced in order to obtain a simple Bellman equation, which the author flags as troubling rather than as a property of the problem	`limitations[7]`	1.0	the tagger's words
limitations	The scope claimed for the field is stated to be precisely the scope of a different, older field. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: a definition of reinforcement learning as the study of learning to act: a control engineer would say this is precisely the scope of control theory, so the boundary between the two fields is a matter of vocabulary rather than of subject	`limitations[8]`	1.0	the tagger's words
transfers	framing and analyzing reinforcement learning with the tools of optimal control, using the Linear Quadratic Regulator as the baseline Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
concepts	reinforcement learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reinforcement learning	`concepts[0]`		the paper's words
concepts	random search Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: random search	`concepts[5]`		the paper's words
concepts	Evolution Strategy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Evolution Strategy	`concepts[6]`		the paper's words
concepts	discount factor Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: discount factor	`concepts[7]`		the paper's words
concepts	Bellman equation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Bellman equation	`concepts[8]`		the paper's words
senses	random search - THE SAME METHOD UNDER FOUR NAMES: pure random search in RL, a (mu,lambda) Evolution Strategy in evolutionary computation, and studied again under stochastic approximation and under bandits - four literatures, one procedure, and this paper is where they are listed Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	model-free - A LABEL THAT DENIES WHAT THE METHOD CARRIES: the name marks the absence of an explicit dynamics model while the modelling assumptions remain, built into the algorithm Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	state-action pair - TWO NOTATIONS FOR ONE OBJECT: (x, u) in control and (s, a) in reinforcement learning, reconciled here by the author choosing one on purpose and saying so Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → reinforcement learning and optimal control theory Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → the System Level Synthesis (SLS) framework [50, 84] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	extends → the high-variance findings of Henderson et al. and Islam et al. for deep RL [35, 36] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
concept_edges	random search —renamed-across-fields→ Evolution Strategy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: random search · Evolution Strategy	`concept_edges[0]`		the tagger's words
concept_edges	discount factor —causes→ Bellman equation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the discount factor had to be introduced IN ORDER TO get a simple Bellman equation; causes is loose. also in: discount factor · Bellman equation	`concept_edges[1]`		the tagger's words
content_tags	2 — the extra variables u and r distinguishing RL Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	N — horizon over which expected reward is maximized Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	mL — total cost of m queries at horizon length L Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	model-free RL is less effective than simple model-based schemes on LQR Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	RL is best viewed as optimal control when the dynamics are unknown Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	RL is harder than supervised learning but can be more valuable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	model-free and model-based perspectives can be unified Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	policy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: policy	`content_tags.variables[1]`		the tagger's words
content_tags	reward Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reward	`content_tags.variables[2]`		the tagger's words
content_tags	horizon length L Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: horizon length L	`content_tags.variables[3]`		the tagger's words
content_tags	state of the system Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: state of the system	`content_tags.variables[4]`		the tagger's words
content_tags	control action Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: control action	`content_tags.variables[5]`		the tagger's words
content_tags	disturbance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: disturbance	`content_tags.variables[6]`		the tagger's words
content_tags	reinforcement learning (RL) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reinforcement learning (RL)	`content_tags.definitions[0]`		the tagger's words
content_tags	control policy Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: control policy	`content_tags.definitions[1]`		the tagger's words
content_tags	trajectory Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: trajectory	`content_tags.definitions[2]`		the tagger's words
content_tags	episodic reinforcement learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: episodic reinforcement learning	`content_tags.definitions[3]`		the tagger's words
content_tags	bootstrap Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: bootstrap	`content_tags.statistics[1]`		the tagger's words
content_tags	regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: regression	`content_tags.statistics[2]`		the tagger's words
content_tags	variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: variance	`content_tags.statistics[3]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.