Reinforcement Learning in the Brain

Reinforcement Learning in the Brain Yael Niv

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Reinforcement Learning in the Brain 2009 Journal of Mathematical Psychology 10.1016/j.jmp.2008.12.005 niv2009-rl-brain niv2009-rl-brain.txt, the project's staged copy 36fdb84c4abd81789f9ab6fa35d89fbe12e4fd1e open-access neuroscience

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them mechanism Quotation withheld because this record is not cleared for redistribution. reward prediction error signalling by midbrain dopamine neurons Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. mechanisms the temporal-difference reward prediction error as the brain learning signal (carried by dopamine) Quotation withheld because this record is not cleared for redistribution. the RL framework of learning has limitations and open questions what this limits: the review's own scope: it sets out to discuss Quotation withheld rather than to defend it Quotation withheld because this record is not cleared for redistribution. The RL framework owes its success partly to being deliberately oversimplified. what this limits: any reading of the framework's explanatory success as evidence for its mechanisms: ONE OF THE REASONS THAT RL MODELS HAVE BEEN SUCCESSFUL IS THAT THEY ARE HIGHLY SIMPLIFIED MODELS ... ESCHEWING THE NECESSARY COMPLEXITIES Quotation withheld because this record is not cleared for redistribution. TD learning depends critically on temporal representations and breaks down under realistic timing noise. what this limits: any application of TD learning under realistic timing: SIMPLE VARIANTS OF TD LEARNING ARE EXTREMELY SENSITIVE TO TIMING NOISE, WITH EVEN VERY SMALL AMOUNTS OF NOISE DEVASTATING THE PREDICTIVE POWER OF THE MODEL Quotation withheld because this record is not cleared for redistribution. Basic conditioning phenomena such as extinction and conditioned inhibition still lack adequate models within the framework. what this limits: the coverage of the framework over basic conditioning: EVEN SIMPLE EXPERIMENTAL PARADIGMS SUCH AS EXTINCTION ... AND CONDITIONED INHIBITION ... DO NOT YET HAVE SATISFACTORY COMPUTATIONAL MODELS Quotation withheld because this record is not cleared for redistribution. The human fMRI evidence reviewed rests on a signal whose neural basis is not understood. what this limits: the human fMRI evidence reviewed: THE NEURAL UNDERPINNINGS OF THE MEASURED BLOOD OXYGEN LEVEL DEPENDENT (BOLD) SIGNAL ARE UNCLEAR Quotation withheld because this record is not cleared for redistribution. The paper names the inference its measurements do not license, not merely the uncertainty. what this limits: any reading of a BOLD prediction-error signal as dopamine: the authors state that one cannot interpret BOLD measurements as reflecting dopaminergic activity per se, despite dopamine's possible direct effect on the signal Quotation withheld because this record is not cleared for redistribution. The framework used to interpret animal conditioning is a normative one. what this limits: any claim that animal conditioning behaves as RL predicts: RL is described as providing a normative framework within which conditioning is analysed and interpreted Quotation withheld because this record is not cleared for redistribution. The imaging evidence is stated to have a ceiling, with the remedy named. what this limits: any conclusion about RL's neural basis drawn from fMRI alone: the authors state that fMRI results can reveal only so much Quotation withheld because this record is not cleared for redistribution. reinforcement learning used as a normative framework to model brain decision-making Quotation withheld because this record is not cleared for redistribution. BOLD measurements BOLD measurements Quotation withheld because this record is not cleared for redistribution. dopaminergic activity dopaminergic activity Quotation withheld because this record is not cleared for redistribution. normative framework normative framework Quotation withheld because this record is not cleared for redistribution. animal conditioning animal conditioning Quotation withheld because this record is not cleared for redistribution. caveats caveats Quotation withheld because this record is not cleared for redistribution. RL → animal conditioning the first is the NORMATIVE FRAMEWORK WITHIN WHICH the second is analysed and interpreted - a lens applied to a phenomenon. It is not 'operationalised-as' (the phenomenon is not measured as the framework), not 'causes', not 'is-a', and not 'part-of'; the relation is 'X is the standard by which Y is read' — note: STILL UNHOUSED, and the instance that shows the failure mode. Short of a type I wrote 'normative framework --operationalised-as--> animal conditioning', which is BACKWARDS, and it passed every automatic check; I caught it at the read-back and dropped the edge. kahneman-tversky1979 states the same structure for expected utility theory - accepted as a normative model, applied as a descriptive one - so a type for it would have at least two attestations in this corpus already. Quotation withheld because this record is not cleared for redistribution. the BOLD signal, here - NOT A DOPAMINE MEASUREMENT: dopamine may affect BOLD directly, and the authors state one still cannot interpret BOLD as reflecting dopaminergic activity per se, so the prediction-error signal seen in human imaging and the dopaminergic signal recorded in animals are two different measurements that the review keeps apart Quotation withheld because this record is not cleared for redistribution. the caveat's symmetry - STATED IN BOTH DIRECTIONS: the same coupling that makes BOLD hard to interpret makes dopamine worth measuring directly, and the paper says so rather than reporting only the half that limits it Quotation withheld because this record is not cleared for redistribution. the RL framework - NORMATIVE, AND USED INTERPRETIVELY: it provides a framework for what an optimal agent should do, and is applied to analyse and interpret what animals actually did; the same structure kahneman-tversky1979 names for expected utility theory, which is accepted as a normative model and applied as a descriptive one Quotation withheld because this record is not cleared for redistribution. builds-on → reinforcement learning (from AI/ML) Quotation withheld because this record is not cleared for redistribution. extends → the Rescorla-Wagner model Quotation withheld because this record is not cleared for redistribution. builds-on → average reward reinforcement learning in a semi-Markov decision process (Schwartz, 1993; Doya, 2000; Daw & Touretzky, 2002) Quotation withheld because this record is not cleared for redistribution. extends → the Rescorla-Wagner model of Pavlovian conditioning (Rescorla & Wagner, 1972) Quotation withheld because this record is not cleared for redistribution. contradicts → the "dopamine equals reward" hypothesis (Wise et al., 1978) Quotation withheld because this record is not cleared for redistribution. BOLD measurements —distinct-from→ dopaminergic activity BOLD measurements dopaminergic activity strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 3 — what RL models provide (predictions, method, computations) Quotation withheld because this record is not cleared for redistribution. 2 — Pavlovian and instrumental conditioned behavior Quotation withheld because this record is not cleared for redistribution. 2 — why normative models appeal Quotation withheld because this record is not cleared for redistribution. dopamine conveys a reward prediction error influencing learning Quotation withheld because this record is not cleared for redistribution. the key RL neural signal is the temporal-difference reward prediction error Quotation withheld because this record is not cleared for redistribution. RL provides a normative framework for conditioned behavior Quotation withheld because this record is not cleared for redistribution. neural evidence discriminates between competing learning models Quotation withheld because this record is not cleared for redistribution. reward prediction error reward prediction error Quotation withheld because this record is not cleared for redistribution. phasic signals phasic signals Quotation withheld because this record is not cleared for redistribution. scalar reinforcement signal scalar reinforcement signal Quotation withheld because this record is not cleared for redistribution. action selection action selection Quotation withheld because this record is not cleared for redistribution. reinforcement learning reinforcement learning Quotation withheld because this record is not cleared for redistribution. Pavlovian conditioning Pavlovian conditioning Quotation withheld because this record is not cleared for redistribution. instrumental conditioning instrumental conditioning Quotation withheld because this record is not cleared for redistribution. normative model normative model Quotation withheld because this record is not cleared for redistribution. bootstrap bootstrap Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1016/j.jmp.2008.12.005.