| phenomenon | reward prediction error signalling by midbrain dopamine neurons— “Specifically, extracellular recordings in behaving animals and functional imaging of human decision-making have revealed in the brain the existence of a key RL signal, the temporal difference reward prediction error.” |
| claim-type | mechanism— “The prediction error theory of dopamine is a computationally precise theory of how phasic dopaminergic firing patterns are generated.” |
| method — evidence | — “In this review we will focus on these links between the theory of reinforcement learning and its implementation in animal and human neural processing.” |
| transfer | from artificial intelligence (reinforcement learning): reinforcement learning used as a normative framework to model brain decision-making— “analysis at the computational level has drawn on ideas from reinforcement” | transfers[0] |
| mechanism | the temporal-difference reward prediction error as the brain learning signal (carried by dopamine)— “a key reinforcement learning signal, the temporal difference reward prediction error” | mechanisms[0] |
| builds-on | reinforcement learning (from AI/ML)— “evidence linking reinforcement learning to the function of dopaminergic neurons” | claim_relations[0] |
| extends | the Rescorla-Wagner model— “it suffers from two major shortcomings” | claim_relations[1] |
| builds-on | average reward reinforcement learning in a semi-Markov decision process (Schwartz, 1993; Doya, 2000; Daw & Touretzky, 2002)— “Building on this theoretical framework, we have recently proposed an RL model of optimal rates of responding (Niv et al., 2005).” | claim_relations[2] |
| extends | the Rescorla-Wagner model of Pavlovian conditioning (Rescorla & Wagner, 1972)— “Temporal-difference (TD) learning is an extension of the Rescorla-Wagner model that also takes into account the timing of different events.” | claim_relations[3] |
| contradicts | the "dopamine equals reward" hypothesis (Wise et al., 1978)— “Contrary to the “dopamine equals reward” hypothesis, the disappearance of the dopaminergic response to reward delivery did not accompany extinction, but rather it followed acquisition of the conditioning relationship – as the cells ceased to respond to rewards the monkeys began showing conditioned responses of anticipatory licking and arm… | claim_relations[4] |
| sense | the BOLD signal, here - NOT A DOPAMINE MEASUREMENT: dopamine may affect BOLD directly, and the authors state one still cannot interpret BOLD as reflecting dopaminergic activity per se, so the prediction-error signal seen in human imaging and the dopaminergic signal recorded in animals are two different measurements that the review keeps apart— “However, despite its possible direct effect on BOLD signals, one can not interpret BOLD measurements as reflecting dopaminergic activity per se” | senses[0] |
| sense | the caveat's symmetry - STATED IN BOTH DIRECTIONS: the same coupling that makes BOLD hard to interpret makes dopamine worth measuring directly, and the paper says so rather than reporting only the half that limits it— “Indeed, these caveats go both ways: since dopamine is a major neuromodulator of interest in RL, its direct measurement is actually of major interest” | senses[1] |
| sense | the RL framework - NORMATIVE, AND USED INTERPRETIVELY: it provides a framework for what an optimal agent should do, and is applied to analyse and interpret what animals actually did; the same structure kahneman-tversky1979 names for expected utility theory, which is accepted as a normative model and applied as a descriptive one— “Importantly, RL provides a normative framework within which to analyze and interpret animal conditioning” | senses[2] |
| edge | BOLD measurements —distinct-from→ dopaminergic activity— “However, despite its possible direct effect on BOLD signals, one can not interpret BOLD measurements as reflecting dopaminergic activity per se” | concept_edges[0] |
| for which use | limitation — the review's own scope: it sets out to discuss SOME OF THE LIMITATIONS OF THE RL FRAMEWORK OF LEARNING rather than to defend it— “some of the limitations of the RL framework of learning” | limitations[0] |
| for which use | limitation — any reading of the framework's explanatory success as evidence for its mechanisms: ONE OF THE REASONS THAT RL MODELS HAVE BEEN SUCCESSFUL IS THAT THEY ARE HIGHLY SIMPLIFIED MODELS ... ESCHEWING THE NECESSARY COMPLEXITIES— “However, one of the reasons that RL models have been successful is that they are highly simplified models, accounting for fundamental phenomena while eschewing the necessary complexities that accompany more detailed explanations.” | limitations[1] |
| for which use | limitation — any application of TD learning under realistic timing: SIMPLE VARIANTS OF TD LEARNING ARE EXTREMELY SENSITIVE TO TIMING NOISE, WITH EVEN VERY SMALL AMOUNTS OF NOISE DEVASTATING THE PREDICTIVE POWER OF THE MODEL— “Simple variants of TD learning are extremely sensitive to timing noise, with even very small amounts of noise devastating the predictive power of the model, and resulting in pronounced prediction errors at the time of reward delivery, even in thoroughly learned tasks.” | limitations[2] |
| for which use | limitation — the coverage of the framework over basic conditioning: EVEN SIMPLE EXPERIMENTAL PARADIGMS SUCH AS EXTINCTION ... AND CONDITIONED INHIBITION ... DO NOT YET HAVE SATISFACTORY COMPUTATIONAL MODELS— “Even simple experimental paradigms such as extinction (in which a once predicted reward ceases to appear) and conditioned inhibition (in which a cue predicts that an otherwise expected reward will not occur) do not yet have satisfactory computational models.” | limitations[3] |
| for which use | limitation — the human fMRI evidence reviewed: THE NEURAL UNDERPINNINGS OF THE MEASURED BLOOD OXYGEN LEVEL DEPENDENT (BOLD) SIGNAL ARE UNCLEAR— “A more pervasive disadvantage of fMRI investigations, however, is that the neural underpinnings of the measured blood oxygen level dependent (BOLD) signal are unclear.” | limitations[4] |
| for which use | limitation — any reading of a BOLD prediction-error signal as dopamine: the authors state that one cannot interpret BOLD measurements as reflecting dopaminergic activity per se, despite dopamine's possible direct effect on the signal— “However, despite its possible direct effect on BOLD signals, one can not interpret BOLD measurements as reflecting dopaminergic activity per se” | limitations[5] |
| for which use | limitation — any claim that animal conditioning behaves as RL predicts: RL is described as providing a normative framework within which conditioning is analysed and interpreted— “Importantly, RL provides a normative framework within which to analyze and interpret animal conditioning” | limitations[6] |
| for which use | limitation — any conclusion about RL's neural basis drawn from fMRI alone: the authors state that fMRI results can reveal only so much— “While outlining the contribution of fMRI to elucidating the neural underpinnings of RL, it is clear from the above that fMRI results can reveal only so much” | limitations[7] |
| note | RL → animal conditioning — STILL UNHOUSED, and the instance that shows the failure mode. Short of a type I wrote 'normative framework --operationalised-as--> animal conditioning', which is BACKWARDS, and it passed every automatic check; I caught it at the read-back and dropped the edge. kahneman-tversky1979 states the same structure for expected utility theory - accepted as a normative model, applied as a descriptive one - so a type for it would have at least two attestations in this corpus already.— “Importantly, RL provides a normative framework within which to analyze and interpret animal conditioning” | unhoused_relations[0] |
| replication | replicated— “multiple lines of evidence linking reinforcement learning to the function of dopaminergic neurons” |
| no type for this | RL → animal conditioning — the first is the NORMATIVE FRAMEWORK WITHIN WHICH the second is analysed and interpreted - a lens applied to a phenomenon. It is not 'operationalised-as' (the phenomenon is not measured as the framework), not 'causes', not 'is-a', and not 'part-of'; the relation is 'X is the standard by which Y is read'— “Importantly, RL provides a normative framework within which to analyze and interpret animal conditioning” | unhoused_relations[0] |
| limitation | the RL framework of learning has limitations and open questions— “some of the limitations of the RL framework of learning” | limitations[0] |
| limitation | The RL framework owes its success partly to being deliberately oversimplified.— “However, one of the reasons that RL models have been successful is that they are highly simplified models, accounting for fundamental phenomena while eschewing the necessary complexities that accompany more detailed explanations.” | limitations[1] |
| limitation | TD learning depends critically on temporal representations and breaks down under realistic timing noise.— “Simple variants of TD learning are extremely sensitive to timing noise, with even very small amounts of noise devastating the predictive power of the model, and resulting in pronounced prediction errors at the time of reward delivery, even in thoroughly learned tasks.” | limitations[2] |
| limitation | Basic conditioning phenomena such as extinction and conditioned inhibition still lack adequate models within the framework.— “Even simple experimental paradigms such as extinction (in which a once predicted reward ceases to appear) and conditioned inhibition (in which a cue predicts that an otherwise expected reward will not occur) do not yet have satisfactory computational models.” | limitations[3] |
| limitation | The human fMRI evidence reviewed rests on a signal whose neural basis is not understood.— “A more pervasive disadvantage of fMRI investigations, however, is that the neural underpinnings of the measured blood oxygen level dependent (BOLD) signal are unclear.” | limitations[4] |
| limitation | The paper names the inference its measurements do not license, not merely the uncertainty.— “However, despite its possible direct effect on BOLD signals, one can not interpret BOLD measurements as reflecting dopaminergic activity per se” | limitations[5] |
| limitation | The framework used to interpret animal conditioning is a normative one.— “Importantly, RL provides a normative framework within which to analyze and interpret animal conditioning” | limitations[6] |
| limitation | The imaging evidence is stated to have a ceiling, with the remedy named.— “While outlining the contribution of fMRI to elucidating the neural underpinnings of RL, it is clear from the above that fMRI results can reveal only so much” | limitations[7] |