Reinforcement Learning in the Brain

Yael Niv · Journal of Mathematical Psychology · 2009 · neuroscience · 10.1016/j.jmp.2008.12.005 · record niv2009-rl-brain
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Replication status is set to 'replicated' on the paper's own statement about the finding it reviews: 'The above basic characteristics of phasic dopaminergic responding have since been replicated in many variants (eg. Hollerman & Schultz, 1998; Schultz, 1998; Tobler et al., 2003; Takikawa et al., 2004; Bayer & Glimcher, 2005).' Because this is a review, two of the recorded relations (extends Rescorla-Wagner, contradicts 'dopamine equals reward') relate the framework the review champions to named prior work rather than relating a novel result of the review itself; only the builds-on relation is stated in the first person. The limitations listed are limitations of the RL framework the review advocates, which the paper flags as its own closing section; it states no limitation of the review procedure as such. 2026-08-14 schema-0.3 pass: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 1 quote(s) on this record are not found as one exact string in the current staged text - content_tags.statistics[3].evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row. Update, 16 September 2026: EXCEPTION TO THE DATED PASS ABOVE, on this record only: this record carries 1 composite quote field - a quote assembled from two or more passages joined by an ellipsis. A COMPOSITE IS NOT AN EXACT CONTIGUOUS SUBSTRING of the staged text and is deliberately not anchored: each of its parts is verified separately at the offsets in evidence composite spans, and a part occurring more than once carries span null rather than a guessed offset. So the 'every quote verified as an exact substring' clause of the dated pass above DOES NOT HOLD for those fields, and never did - the parts were verified, the whole was not. Nothing is deleted, re-grounded or withdrawn: this writes down an exception that was always true of composites and had not been recorded, so a reader was told more than the pass had checked.

The quality of the text these quotes were cut from. symbol-font-not-extractable: 15 '(cid:NNN)' glyph-index markers remain in a staged text of 89,606 characters, where the PDF's embedded font had no ToUnicode map. THE QUOTES ON THIS RECORD ARE THE PAPER'S AND THE STAGED COPY IS THE DAMAGED ONE - which is the opposite of the usual case and is why it went unnoticed: one of its 54 live quotes is not an exact substring of this text, and it could NOT be located in it even by aligning the quote's WORD SEQUENCE under this damage, so the defect recorded here is present on the record but does not by itself account for the miss. Not repaired in place: editing the staged text moves every offset after the edit and re-hashes every anchor on the record. Measured at 113aae60fefc3eb1, re-measured unchanged at 54b63482778ddf56.

Some words in these quotes contain the wrong character. The PDF this text was extracted from substituted 1 character inside a word — a currency sign or a stray accent where a letter pair belongs, as in signi¢cantly for significantly, or a single ligature glyph, as in conﬁdence. They are valid characters in the wrong place, so they survive every encoding check. The quotes keep them for the same reason as above, and a search of this page will not find the affected words.

This record’s own method is review, and the tags below follow from that. A review’s findings, quantities and statistics are things stated IN this paper, and some of them report results obtained elsewhere. No tag on this page names a different author. Measured across every review in this corpus, 81% of a review’s tags carry no citation at all — most of them are the review speaking. The corpus does not adjudicate the rest: read an unmarked tag as a claim this paper makes, not necessarily one it made first.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
standard-deviation 3 fields · reward prediction error 2 fields · bootstrap 2 fields · linear regression 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 9 concept tags on this paper, 5 are grounded in a quoted sentence.

reinforcement learning reward prediction error dopamine temporal-difference learning BOLD measurements dopaminergic activity normative framework animal conditioning caveats the temporal-difference reward prediction error as the brain learning signal (carried by dopamine) decision-making conditioning temporal difference action selection basal ganglia instrumental conditioning Pavlovian conditioning normative model phasic signals scalar reinforcement signal bootstrap mean average of regression linear regression standard-deviation standard deviation review

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field
unrecordable relation	`unhoused_relations`	a relation the paper asserts that this vocabulary cannot express, with the reason

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.1016/j.jmp.2008.12.005. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 5 · claim_type 1 · concept_edges 1 · concepts 5 · content_tags 18 · limitations 8 · mechanisms 1 · method 1 · phenomenon 1 · senses 3 · transfers 1 · unhoused_relations 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	mechanism Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	reward prediction error signalling by midbrain dopamine neurons Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	review Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: review	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the temporal-difference reward prediction error as the brain learning signal (carried by dopamine)	`mechanisms[0]`		the tagger's words
limitations	the RL framework of learning has limitations and open questions Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the review's own scope: it sets out to discuss Quotation withheld rather than to defend it	`limitations[0]`		the tagger's words
limitations	The RL framework owes its success partly to being deliberately oversimplified. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any reading of the framework's explanatory success as evidence for its mechanisms: ONE OF THE REASONS THAT RL MODELS HAVE BEEN SUCCESSFUL IS THAT THEY ARE HIGHLY SIMPLIFIED MODELS ... ESCHEWING THE NECESSARY COMPLEXITIES	`limitations[1]`		the tagger's words
limitations	TD learning depends critically on temporal representations and breaks down under realistic timing noise. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any application of TD learning under realistic timing: SIMPLE VARIANTS OF TD LEARNING ARE EXTREMELY SENSITIVE TO TIMING NOISE, WITH EVEN VERY SMALL AMOUNTS OF NOISE DEVASTATING THE PREDICTIVE POWER OF THE MODEL	`limitations[2]`		the tagger's words
limitations	Basic conditioning phenomena such as extinction and conditioned inhibition still lack adequate models within the framework. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the coverage of the framework over basic conditioning: EVEN SIMPLE EXPERIMENTAL PARADIGMS SUCH AS EXTINCTION ... AND CONDITIONED INHIBITION ... DO NOT YET HAVE SATISFACTORY COMPUTATIONAL MODELS	`limitations[3]`		the tagger's words
limitations	The human fMRI evidence reviewed rests on a signal whose neural basis is not understood. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the human fMRI evidence reviewed: THE NEURAL UNDERPINNINGS OF THE MEASURED BLOOD OXYGEN LEVEL DEPENDENT (BOLD) SIGNAL ARE UNCLEAR	`limitations[4]`		the tagger's words
limitations	The paper names the inference its measurements do not license, not merely the uncertainty. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any reading of a BOLD prediction-error signal as dopamine: the authors state that one cannot interpret BOLD measurements as reflecting dopaminergic activity per se, despite dopamine's possible direct effect on the signal	`limitations[5]`	1.0	the tagger's words
limitations	The framework used to interpret animal conditioning is a normative one. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any claim that animal conditioning behaves as RL predicts: RL is described as providing a normative framework within which conditioning is analysed and interpreted	`limitations[6]`	1.0	the tagger's words
limitations	The imaging evidence is stated to have a ceiling, with the remedy named. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any conclusion about RL's neural basis drawn from fMRI alone: the authors state that fMRI results can reveal only so much	`limitations[7]`	1.0	the tagger's words
transfers	reinforcement learning used as a normative framework to model brain decision-making Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
concepts	BOLD measurements Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: BOLD measurements	`concepts[4]`		the paper's words
concepts	dopaminergic activity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: dopaminergic activity	`concepts[5]`		the paper's words
concepts	normative framework Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: normative framework	`concepts[6]`		the paper's words
concepts	animal conditioning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: animal conditioning	`concepts[7]`		the paper's words
concepts	caveats Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: caveats	`concepts[8]`		the paper's words
unhoused_relations	RL → animal conditioning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. the first is the NORMATIVE FRAMEWORK WITHIN WHICH the second is analysed and interpreted - a lens applied to a phenomenon. It is not 'operationalised-as' (the phenomenon is not measured as the framework), not 'causes', not 'is-a', and not 'part-of'; the relation is 'X is the standard by which Y is read' — note: STILL UNHOUSED, and the instance that shows the failure mode. Short of a type I wrote 'normative framework --operationalised-as--> animal conditioning', which is BACKWARDS, and it passed every automatic check; I caught it at the read-back and dropped the edge. kahneman-tversky1979 states the same structure for expected utility theory - accepted as a normative model, applied as a descriptive one - so a type for it would have at least two attestations in this corpus already.	`unhoused_relations[0]`		the tagger's words
senses	the BOLD signal, here - NOT A DOPAMINE MEASUREMENT: dopamine may affect BOLD directly, and the authors state one still cannot interpret BOLD as reflecting dopaminergic activity per se, so the prediction-error signal seen in human imaging and the dopaminergic signal recorded in animals are two different measurements that the review keeps apart Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	the caveat's symmetry - STATED IN BOTH DIRECTIONS: the same coupling that makes BOLD hard to interpret makes dopamine worth measuring directly, and the paper says so rather than reporting only the half that limits it Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	the RL framework - NORMATIVE, AND USED INTERPRETIVELY: it provides a framework for what an optimal agent should do, and is applied to analyse and interpret what animals actually did; the same structure kahneman-tversky1979 names for expected utility theory, which is accepted as a normative model and applied as a descriptive one Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → reinforcement learning (from AI/ML) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	extends → the Rescorla-Wagner model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → average reward reinforcement learning in a semi-Markov decision process (Schwartz, 1993; Doya, 2000; Daw & Touretzky, 2002) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → the Rescorla-Wagner model of Pavlovian conditioning (Rescorla & Wagner, 1972) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	contradicts → the "dopamine equals reward" hypothesis (Wise et al., 1978) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
concept_edges	BOLD measurements —distinct-from→ dopaminergic activity Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: BOLD measurements · dopaminergic activity	`concept_edges[0]`		the tagger's words
content_tags	3 — what RL models provide (predictions, method, computations) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	2 — Pavlovian and instrumental conditioned behavior Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	2 — why normative models appeal Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	dopamine conveys a reward prediction error influencing learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	the key RL neural signal is the temporal-difference reward prediction error Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	RL provides a normative framework for conditioned behavior Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	neural evidence discriminates between competing learning models Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	reward prediction error Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reward prediction error	`content_tags.variables[0]`		the tagger's words
content_tags	phasic signals Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: phasic signals	`content_tags.variables[1]`		the tagger's words
content_tags	scalar reinforcement signal Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: scalar reinforcement signal	`content_tags.variables[2]`		the tagger's words
content_tags	action selection Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: action selection	`content_tags.variables[3]`		the tagger's words
content_tags	reinforcement learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: reinforcement learning	`content_tags.definitions[0]`		the tagger's words
content_tags	Pavlovian conditioning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Pavlovian conditioning	`content_tags.definitions[1]`		the tagger's words
content_tags	instrumental conditioning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: instrumental conditioning	`content_tags.definitions[2]`		the tagger's words
content_tags	normative model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: normative model	`content_tags.definitions[3]`		the tagger's words
content_tags	bootstrap Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: bootstrap	`content_tags.statistics[0]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean	`content_tags.statistics[1]`		the tagger's words
content_tags	standard-deviation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: standard-deviation	`content_tags.statistics[4]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

content_tags.statistics[3] — quotation withheld — the page has already reproduced as much of this paper as the rights budget allows
this quote's status: quote-not-in-current-staged-text · about this quote: marked quote-not-in-current-staged-text - a DECLARED COMPOSITE (the composite-quote note): the whole string is not one span of the staged text; each part is verified there exactly once. · NOT MACHINE-VOUCHED: not-in-our-copy - The quote is in NEITHER copy we hold - not in the text we tag against and not in the page we fetched ('fulltext.html') - checked with ligatures, hyphens, spacing and punctuation set aside. This body carries 15 '(cid:NNN)' glyph ids, 5 detached diacritics. We do not know what it was taken from, and no class is claimed for it.

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.