Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network Wei‐Xing Pan Robert Schmidt Jeffery R. Wickens Brian I. Hyland

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network 2005 Journal of Neuroscience 10.1523/jneurosci.1478-05.2005 pan2005-dopamine-cells-respond-to-predicte pan2005-dopamine-cells-respond-to-predicte.txt, the project's staged copy e5954dac87e4c29baf955d55da69c4f257b7b70d open-access neuroscience

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them how the dopamine response moves from reward to predictive cue during learning, and whether the temporal difference algorithm can produce that time course Quotation withheld because this record is not cleared for redistribution. whether dopamine cells show the stepwise latency migration that published TD models predict, and if not, which parameter settings would reproduce what they do show Quotation withheld because this record is not cleared for redistribution. shows cue and reward responses coexist rather than trading off, then finds the pattern is reproduced only by long eligibility traces and a low learning rate - turning physiology into a constraint on the algorithm's parameters Quotation withheld because this record is not cleared for redistribution. mechanism Quotation withheld because this record is not cleared for redistribution. dopamine reward-prediction-error signalling during classical conditioning Quotation withheld because this record is not cleared for redistribution. electrophysiology electrophysiology Quotation withheld because this record is not cleared for redistribution. mechanisms A long-lasting eligibility trace keeps the weights of an earlier sensory signal open to modification for many time steps, so the reward-driven prediction error can be assigned back to the cue in one step instead of migrating one time step per trial Quotation withheld because this record is not cleared for redistribution. mechanisms A very low learning rate combined with a high eligibility-trace decay makes each trial change the weights only slightly, so responses to the cue and to the predicted reward coexist for a prolonged period before the reward response is suppressed Quotation withheld because this record is not cleared for redistribution. mechanisms Sustained firing in reverberating circuits, or biochemical processes at the synapse, could hold the seconds-long eligibility trace the model requires Quotation withheld because this record is not cleared for redistribution. an eligibility trace keeps earlier time steps modifiable, so a cue several steps back can be updated in one trial instead of waiting for the error to migrate Quotation withheld because this record is not cleared for redistribution. with a low learning rate each trial moves the weights only slightly, which is what lets cue and reward responses coexist for hundreds of trials Quotation withheld because this record is not cleared for redistribution. under TD(0) the error must arrive one step earlier per trial, so reaching a cue k steps back takes at least k trials Quotation withheld because this record is not cleared for redistribution. candidate substrates for a seconds-long trace are reverberating circuits and synaptic biochemistry Quotation withheld because this record is not cleared for redistribution. long traces with slow learning is the combination machine learning already found to give early but stable learning, which may be why biology uses it Quotation withheld because this record is not cleared for redistribution. The fitted parameter values are not claimed to be fixed neural constants, since a different model structure could yield different best-fitting values. what this limits: the fitted parameter values: OUR RESULTS SHOULD NOT BE UNDERSTOOD AS PROVIDING EXPLICIT FIXED VALUES FOR NEURAL TD LEARNING PARAMETERS, BECAUSE THE SPECIFIC VALUES ... MAY BE DIFFERENT IF DETAILS OF THE MODEL STRUCTURE WERE CHANGED Quotation withheld because this record is not cleared for redistribution. The stimulus representation assumed by the model (serial compound) is not known to be how the brain represents stimuli for reward learning. what this limits: the timing predictions in particular: IT IS NOT KNOWN HOW SENSORY STIMULI ARE REPRESENTED IN THE BRAIN FOR REWARD LEARNING Quotation withheld because this record is not cleared for redistribution. the paper refuses the strong reading of its own result: these are not fixed values for neural parameters, because a different model structure would shift them what this limits: the same scope as limitations[0] - the values are a property of this model structure Quotation withheld because this record is not cleared for redistribution. how the brain represents stimuli in time is unknown; the serial compound representation is a convention inherited from earlier models what this limits: the same scope as limitations[1], with the convention named: WE FOLLOWED MANY PREVIOUS STUDIES IN USING A SERIAL COMPOUND REPRESENTATION OF CONDITIONED STIMULI Quotation withheld because this record is not cleared for redistribution. the animals had already had hundreds of solenoid-reward pairings before recording, so the click was not a neutral event at the start of conditioning what this limits: any reading of the click as a neutral cue: ANIMALS WERE THEREFORE EXPOSED TO HUNDREDS OF SOLENOID-REWARD PAIRINGS BY THE TIME THE RECORDINGS WERE MADE Quotation withheld because this record is not cleared for redistribution. the negative prediction error signal is small and hard to measure against an already low baseline rate what this limits: the negative prediction-error signal: THE AMPLITUDE OF INHIBITIONS WAS SMALL, CONSISTENT WITH THE FACT THAT INHIBITORY RESPONSES CAN BE DIFFICULT TO DETECT AND QUANTIFY IN NEURONS THAT ALREADY HAVE A LOW BASELINE RATE Quotation withheld because this record is not cleared for redistribution. 21 of 45 electrophysiologically plausible cells failed the pharmacological test and were discarded what this limits: what a 'dopamine neuron' means here, and in every study that skips the drug test: FROM 45 NEURONS THAT WERE POSSIBLY DOPAMINERGIC ON ELECTROPHYSIOLOGICAL CRITERIA, 24 ... WERE CONFIRMED DA NEURONS BASED ON HAVING A CLEAR INHIBITORY RESPONSE TO DOPAMINE AGONIST DRUGS Quotation withheld because this record is not cleared for redistribution. The temporal difference algorithm and its prediction-error signal, taken from machine learning, are used as the model of midbrain dopamine cell activity. Quotation withheld because this record is not cleared for redistribution. the temporal difference algorithm, imported as a candidate account of what dopamine cells compute Quotation withheld because this record is not cleared for redistribution. the century-old requirement that some trace of the cue survive until the reward arrives, here identified with the eligibility trace Quotation withheld because this record is not cleared for redistribution. the known trade-off that high lambda with low alpha buys stable early learning, used here as an adaptive argument Quotation withheld because this record is not cleared for redistribution. electrode track positions confirmed histologically after the experiments were completed eliminates: attributing the responses to dopamine cells that were not where the recordings were thought to be - the confirmation comes after, from the tissue, not from the coordinates · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. presumed DA cells are a doubly filtered category: rate below 10 Hz and spike at least 1 ms, THEN more than 50% inhibition by a dopamine agonist. Cell identity is pharmacological here, not genetic or anatomical Quotation withheld because this record is not cleared for redistribution. TD(0), TD(1) and TD(lambda) are three different models, not one - the value of lambda changes which weights an error can touch, and the published DA models were TD(0) Quotation withheld because this record is not cleared for redistribution. prediction error here is the scalar difference between actual reward and the temporal difference of successive predictions, with the NEGATIVE side deliberately clipped to match dopamine cells' floor Quotation withheld because this record is not cleared for redistribution. early and late training are defined by block count - 6 or fewer against 10 or more - and the paper's central result holds only early Quotation withheld because this record is not cleared for redistribution. the recorded population pools substantia nigra pars compacta, ventral tegmental area and supramammillary nucleus, justified by a null on regional differences Quotation withheld because this record is not cleared for redistribution. claim_type: mechanism a second reading reaching the same value. Recorded: mechanism — note: a second reading reaching the same value from a different sentence. Not applied - a person decides. Quotation withheld because this record is not cleared for redistribution. method: extracellular single-unit recording from pharmacologically identified midbrain dopamine cells during first acquisition, paired with a systematic sweep of the TD algorithm's parameter space a second reading DIFFERING from the recorded value. Recorded: electrophysiology — note: a second reading, verbatim-verified, disagreeing with the value on the record. Not applied - a person decides. Quotation withheld because this record is not cleared for redistribution. phenomenon: the shift of dopamine cell firing from unexpected reward to the cue that predicts it a second reading DIFFERING from the recorded value. Recorded: dopamine reward-prediction-error signalling during classical conditioning — note: a second reading, verbatim-verified, disagreeing with the value on the record. Not applied - a person decides. Quotation withheld because this record is not cleared for redistribution. builds-on → Montague et al. (1996) temporal-difference model of dopamine activity Quotation withheld because this record is not cleared for redistribution. contradicts → previous TD models predicting a gradual stepwise latency shift (Montague et al., 1996; Schultz et al., 1997) differs in: the time course of the latency shift: previous TD models PREDICT A GRADUAL STEPWISE SHIFT IN LATENCY WITH RESPONSES TO REWARDS LOST BEFORE RESPONSES DEVELOP TO THE CONDITIONED CUE Quotation withheld because this record is not cleared for redistribution. extends → Miller et al., 1981; Kosobud et al., 1994; Kiyatkin and Rebec, 2001; Hyland et al., 2002 (rat DA cell responses to reward-predicting cues) Quotation withheld because this record is not cleared for redistribution. contradicts → the stepwise latency migration predicted by TD(0) models of dopamine activity differs in: the latency profile of the new response: NEW CONDITIONED RESPONSES MADE THEIR FIRST APPEARANCE AT A CONSTANT, SHORT LATENCY AFTER PREDICTIVE CUES, and DEVELOPED RAPIDLY, WITHIN A FEW TRIALS - not the gradual migration the models expect Quotation withheld because this record is not cleared for redistribution. contradicts → the prediction that a redundant intermediate cue draws no response Quotation withheld because this record is not cleared for redistribution. contradicts → the prediction that reward responses vanish before cue responses appear Quotation withheld because this record is not cleared for redistribution. reconciles → the temporal difference framework with the recorded time course, by relocating the disagreement to parameter choice differs in: the model's free parameters: THESE APPARENT DISPARITIES BETWEEN TD MODEL PREDICTION ERRORS AND DA CELL ACTIVITY ARE ATTRIBUTABLE TO THE PARTICULAR CHOICE OF PARAMETERS USED IN THE MODEL - the settings for ALPHA and LAMBDA, not the framework Quotation withheld because this record is not cleared for redistribution. critiques → published primate datasets, which on re-inspection never supported gradual migration either differs in: gradual migration against simultaneous responding, and the stage of learning: the existing primate data PROVIDES NO SUPPORT FOR THE GRADUAL MIGRATION OF A RESPONSE DURING LEARNING AND SUGGESTS THAT DA CELLS SHOW SIMULTANEOUS RESPONDING TO BOTH CUE AND REWARD, AT LEAST EARLY IN LEARNING Quotation withheld because this record is not cleared for redistribution. confirms → earlier reports that reward responses are diminished in highly trained animals Quotation withheld because this record is not cleared for redistribution. extends → rat studies showing dopamine responses to reward-predicting cues, by showing they arise during acquisition Quotation withheld because this record is not cleared for redistribution. prediction error —operationalised-as→ DA cells prediction error DA cells strength of the relation: 1 · note: the identification is asymmetric by construction: firing can rise to 100 Hz but only fall to zero from a 5 Hz baseline Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says DA cells' baseline spiking can be equated to zero prediction error; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. TD(0) —is-a→ temporal difference TD(0) temporal difference strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'TD(0) model'; 'temporal difference' is an abbreviation variant. Quotation withheld because this record is not cleared for redistribution. TD(1) —is-a→ temporal difference TD(1) temporal difference strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'TD(1) model'; 'temporal difference' is an abbreviation variant. Quotation withheld because this record is not cleared for redistribution. apomorphine —operationalised-as→ DA cells apomorphine DA cells strength of the relation: 1 · note: direction is test-to-category: more than 50% inhibition by the agonist is what makes a recorded cell count as dopaminergic Audit 2026-09-28: DOUBTFUL - read against its span and kept, but apomorphine inhibition was an inclusion criterion for DA cells; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. serial compound —operationalised-as→ state vector serial compound state vector strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. omission —operationalised-as→ prediction error omission prediction error strength of the relation: 0.8 · note: the negative side of the signal is measured by withholding what was predicted Audit 2026-09-28: DOUBTFUL - read against its span and kept, but responses to cue omission were analysed; operationalised-as 'prediction error' is interpretive. Quotation withheld because this record is not cleared for redistribution. t-test t-test Quotation withheld because this record is not cleared for redistribution. standard deviation standard deviation Quotation withheld because this record is not cleared for redistribution. standard error of the mean standard error of the mean Quotation withheld because this record is not cleared for redistribution. standard deviation threshold criterion standard deviation threshold criterion Quotation withheld because this record is not cleared for redistribution. histogram histogram Quotation withheld because this record is not cleared for redistribution. paired t test paired t test Quotation withheld because this record is not cleared for redistribution. modulation index modulation index Quotation withheld because this record is not cleared for redistribution. SD SD Quotation withheld because this record is not cleared for redistribution. Dopamine cells kept responding to reward long after they developed responses to the predictive cue. Quotation withheld because this record is not cleared for redistribution. New conditioned responses appeared straight away at their final short latency after the cue, with no gradual backward drift of latency from reward to cue. Quotation withheld because this record is not cleared for redistribution. Every cell trained with two cues in sequence responded clearly to the second, seemingly redundant cue, and that response persisted for the whole experiment. Quotation withheld because this record is not cleared for redistribution. Cells recorded after many sessions of cue-reward pairing no longer responded to a predictable reward, unlike the cells recorded early in training. Quotation withheld because this record is not cleared for redistribution. Acquiring a cue response and losing the reward response run on different time courses rather than being two sides of one shift. Quotation withheld because this record is not cleared for redistribution. Omitting an expected reward significantly reduced firing rate. Quotation withheld because this record is not cleared for redistribution. Omitting the second cue restored reward responses to the level seen with random reward. Quotation withheld because this record is not cleared for redistribution. With the chosen parameter values, the TD model's prediction error reproduced key elements of the recorded dopamine pattern. Quotation withheld because this record is not cleared for redistribution. A high eligibility-trace decay parameter was the one setting that let the model learn cue significance without stepwise trial-by-trial migration of the response from reward to cue. Quotation withheld because this record is not cleared for redistribution. Once the eligibility-trace decay fell below about 0.6, stepwise migration of the prediction error reappeared no matter what learning rate was used. Quotation withheld because this record is not cleared for redistribution. conditioned responses appeared at their final short latency within a few trials, with no gradual migration Quotation withheld because this record is not cleared for redistribution. the redundant intermediate cue drew a clear response in every two-cue cell, which persisted Quotation withheld because this record is not cleared for redistribution. responses to predicted reward persisted after cue responses had developed, in every cell recorded early in training Quotation withheld because this record is not cleared for redistribution. responses to predicted reward are lost completely only after training beyond what first establishes cue responses, so the two changes have different time courses Quotation withheld because this record is not cleared for redistribution. those late-training cells still responded robustly to reward delivered without cues Quotation withheld because this record is not cleared for redistribution. omitting the second cue restored the reward response to its uncued level, so the redundant cue was doing predictive work Quotation withheld because this record is not cleared for redistribution. TD with eligibility zero and a fast learning rate reproduced the published pattern, with no second-cue response and no overlap Quotation withheld because this record is not cleared for redistribution. TD with lambda 0.9 and alpha 0.005 reproduced the recorded pattern, including the coexistence and the intermediate cue response Quotation withheld because this record is not cleared for redistribution. lambda below about 0.6 brought stepwise migration back for every value of alpha, making it the critical parameter Quotation withheld because this record is not cleared for redistribution. omitted rewards produced a significant firing decrease but omitted cues did not Quotation withheld because this record is not cleared for redistribution. CONTROL NULL: no regional difference in response pattern, which is what licensed pooling the three midbrain sites Quotation withheld because this record is not cleared for redistribution. BUILT IN, NOT PREDICTED: the model's small negative errors follow from the clipping imposed to match dopamine cells' floor, as the authors state Quotation withheld because this record is not cleared for redistribution. re-inspection of published primate datasets found no support for gradual migration there either Quotation withheld because this record is not cleared for redistribution. eligibility trace eligibility trace Quotation withheld because this record is not cleared for redistribution. temporal difference temporal difference Quotation withheld because this record is not cleared for redistribution. learning rate parameter learning rate parameter Quotation withheld because this record is not cleared for redistribution. complete serial compound stimulus complete serial compound stimulus Quotation withheld because this record is not cleared for redistribution. temporal difference temporal difference Quotation withheld because this record is not cleared for redistribution. prediction error prediction error Quotation withheld because this record is not cleared for redistribution. eligibility trace eligibility trace Quotation withheld because this record is not cleared for redistribution. serial compound serial compound Quotation withheld because this record is not cleared for redistribution. modulation index modulation index Quotation withheld because this record is not cleared for redistribution. TD(0) TD(0) Quotation withheld because this record is not cleared for redistribution. TD(1) model TD(1) model Quotation withheld because this record is not cleared for redistribution. TD(0) model TD(0) model Quotation withheld because this record is not cleared for redistribution. oddball sequences oddball sequences Quotation withheld because this record is not cleared for redistribution. decay parameter decay parameter Quotation withheld because this record is not cleared for redistribution. firing rate firing rate Quotation withheld because this record is not cleared for redistribution. discount factor discount factor Quotation withheld because this record is not cleared for redistribution. intertrial interval intertrial interval Quotation withheld because this record is not cleared for redistribution. control paradigms control paradigms Quotation withheld because this record is not cleared for redistribution. learning rate learning rate Quotation withheld because this record is not cleared for redistribution. eligibility trace eligibility trace Quotation withheld because this record is not cleared for redistribution. latency latency Quotation withheld because this record is not cleared for redistribution. omission omission Quotation withheld because this record is not cleared for redistribution. 24 of 45, from 11 rats — cells that passed pharmacological confirmation as dopamine neurons out of those meeting the electrophysiological criteria Quotation withheld because this record is not cleared for redistribution. 5.7 ± 3.5 Hz — average baseline firing rate of the confirmed dopamine cells Quotation withheld because this record is not cleared for redistribution. 8 of 11 — cells tested in which conditioned responses to the cue developed during recording Quotation withheld because this record is not cleared for redistribution. 74 ± 33 ms; 53 ± 9 ms — latencies of the newly conditioned excitatory and inhibitory responses to the cue Quotation withheld because this record is not cleared for redistribution. -0.12 ± 0.17 — average modulation index at the time of an omitted second cue, a nonsignificant departure from baseline Quotation withheld because this record is not cleared for redistribution. -0.21 ± 0.20; p < 0.01; n = 11 — average modulation index when an expected reward was omitted, a significant reduction from baseline Quotation withheld because this record is not cleared for redistribution. ∼0.6 — threshold value of the eligibility-trace decay parameter below which stepwise migration of the prediction error reappears Quotation withheld because this record is not cleared for redistribution. -0.05 — ceiling imposed on the model's negative prediction error, to match the limited range over which dopamine cells can be inhibited Quotation withheld because this record is not cleared for redistribution. 0.9 and 0.005 — the lambda and alpha that reproduced the recorded dopamine pattern Quotation withheld because this record is not cleared for redistribution. about 0.6 — the lambda threshold below which stepwise migration reappears for every alpha Quotation withheld because this record is not cleared for redistribution. -0.05 — the imposed floor on negative prediction error, matching dopamine cells' limited inhibitory range Quotation withheld because this record is not cleared for redistribution. 24 of 45 — electrophysiologically plausible cells that passed the pharmacological identification Quotation withheld because this record is not cleared for redistribution. 5.7 Hz, 1.5 ms — mean firing rate and spike duration of the accepted dopamine cells Quotation withheld because this record is not cleared for redistribution. -0.21 vs -0.12 — modulation index for omitted reward (significant) against omitted cue (not) Quotation withheld because this record is not cleared for redistribution. 74 ms and 53 ms — latency of the new conditioned excitations and inhibitions Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1523/jneurosci.1478-05.2005.