Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks

Brenden M. Lake, Marco Baroni · ICML / arXiv · 2018 · cognitive-psychology · record lake2018-generalization-systematicity
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim type was set to null-result because the headline contribution is a failure to observe systematic composition; the paper also makes a method contribution (the SCAN dataset) and a positive existence claim (strong mix-and-match zero-shot generalisation, 99.8% on the random split), which the single-valued facet cannot hold. Method is tagged simulation (training and testing 180 network architectures); it is equally a benchmark-building paper. Discipline is machine-learning although the framing is the cognitive-science systematicity debate (Fodor & Pylyshyn). The paper never claims replication of an earlier study, hence replication status untested. 2026-08-14, 2026-08-17 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
simulation 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 10 concept tags on this paper, 5 are grounded in a quoted sentence.

compositional generalization systematicity sequence-to-sequence neural network natural language processing SCAN CommAI Navigation tasks systematic compositionality algebraic mind mix-and-match A STIPULATED EQUIVALENCE CUTS THE TEST SPACE: commands are declared distributionally identical - defined as occurring in exactly the same composed commands - and that declaration is what makes testing the rest redundant. The sameness asserted is a property of the training distribution, not of the actions THE DIFFERENCE BETWEEN TWO ITEMS IS EXPLAINED BY EXPOSURE RATHER THAN CAPACITY, and hedged twice while doing it: one primitive's action is seen many times inside other directed actions, which APPARENTLY suffices, PROBABLY because the model gets direct evidence of how it is used in context - while the other action occurs only with its own primitive command and does not generalise from that minimal context A NEAR-PERFECT SUCCESS IS DISQUALIFIED ON THE SHAPE OF ITS ERRORS RATHER THAN THEIR NUMBER: all 45 errors are conjunctions containing one of just two components (22 and 23 cases), both of which the model handles correctly on their own - and the authors say it is HARD TO CONCEIVE OF someone who understood the longer command but not the shorter one inside it. The error DISTRIBUTION, not the error rate, is what is read as measuring the capacity A DELIBERATELY UNREALISTIC MANIPULATION IS ADDED AS A DIAGNOSTIC, WITH BOTH OUTCOMES NAMED IN ADVANCE: an oracle supplies the correct output length, and the authors state before running it that a large improvement would mean the decoder was merely terminating early while a small one would mean the failure is deeper. The control is designed to be uninformative about performance and informative about cause A RIVAL EXPLANATION OF THE FAILURE IS ELIMINATED BY A SEPARATE MEASUREMENT: the network preferred its own output to the target by log-likelihood, so the errors are not search failures in the decoder - the model is not failing to FIND the right answer, it does not prefer it THE SUFFICIENCY OF THE TRAINING DATA IS ARGUED SEPARATELY FROM THE RESULT AND MARKED AS IMPORTANT: the authors state that the training data of both experiments provide enough evidence to learn the composition rules that would afford the correct generalisations, and then spell out, per experiment, what was demonstrated during training THE DEFLATIONARY READING OF THE RESULT IS STATED BY THE AUTHORS THEMSELVES AND THEN ANSWERED: the finding can be seen as a trivial restatement of the principle that train and test should come from the same distribution - and the reply is that human learners DO generalise to data governed by rules they have learned, so the triviality does not cover the case AN ALTERNATIVE EXPLANATION OF A NULL IS ELIMINATED BY GRADED DOSING RATHER THAN ARGUMENT: composed examples of the new primitive are stepped 8 -> 16 -> 32 and accuracy climbs 38.3 -> 77.8 -> 88.4, appearing GRADUALLY instead of switching on, which rules out the reading that the network had simply lacked evidence THE CAPACITY IS OPERATIONALISED AS A BEHAVIOURAL SIGNATURE, NOT AS AN ACCURACY: what is expected of a systematically compositional device is ALL-OR-NOTHING rule-based behaviour, so a network that generalises powerfully but needs more positive examples for it to emerge fails the definition WHILE SUCCEEDING AT THE TASK THE AUTHORS NAME THEIR OWN TEST'S POSSIBLE UNFAIRNESS BEFORE ANYONE ELSE CAN: they say they tested systematicity in its purest form, then concede the setup is arguably too opaque for a computational model which could lack evidence that the two actions are the same sort of thing - and the concession is what motivates the follow-up experiment THE CONTROL ITEM IS MATCHED ON THE TEST AND DELIBERATELY UNMATCHED ON THE TRAINING: a real adjective seen in 80 different constructions is tested on exactly the same eight constructions as the invented one and scores 8 of 8 against 1 of 8 - so the comparison isolates breadth of training context while holding the test fixed THE POSITIVE HALF OF A NEGATIVE RESULT IS INSISTED ON RATHER THAN CONCEDED: the authors explicitly stress that the high zero-shot accuracy is an important positive result in controlled experiments, and add that understanding the mechanisms behind it is future work - refusing to let the headline failure absorb the finding that sits beside it compositional generalization RNNs commands recurrent zero-shot networks SCAN domain mean average value standard-error simulation

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1711.00350. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 4 · claim_type 1 · concepts 5 · content_tags 6 · controls 1 · limitations 6 · mechanisms 12 · method 1 · phenomenon 1 · senses 3 · transfers 1

This record holds no relations between concepts. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	null-result Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	systematic compositional generalization in sequence-to-sequence networks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	simulation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: simulation	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A STIPULATED EQUIVALENCE CUTS THE TEST SPACE: commands are declared distributionally identical - defined as occurring in exactly the same composed commands - and that declaration is what makes testing the rest redundant. The sameness asserted is a property of the training distribution, not of the actions	`mechanisms[0]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE DIFFERENCE BETWEEN TWO ITEMS IS EXPLAINED BY EXPOSURE RATHER THAN CAPACITY, and hedged twice while doing it: one primitive's action is seen many times inside other directed actions, which APPARENTLY suffices, PROBABLY because the model gets direct evidence of how it is used in context - while the other action occurs only with its own primitive command and does not generalise from that minimal context	`mechanisms[1]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: A NEAR-PERFECT SUCCESS IS DISQUALIFIED ON THE SHAPE OF ITS ERRORS RATHER THAN THEIR NUMBER: all 45 errors are conjunctions containing one of just two components (22 and 23 cases), both of which the model handles correctly on their own - and the authors say it is HARD TO CONCEIVE OF someone who understood the longer command but not the shorter one inside it. The error DISTRIBUTION, not the error rate, is what is read as measuring the capacity	`mechanisms[2]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: The span contains a LOST-HYPHEN FUSION - 'overallbest' for 'overall-best'. A line-break hyphen was dropped by the extraction and the two halves fused. The quote is an exact substring and anchors correctly; the damage is in what a reader of the tag sees. None of the four shapes an internal tool looks for covers this, so it is annotated by hand. Seven such fusions are in this file. also in: A DELIBERATELY UNREALISTIC MANIPULATION IS ADDED AS A DIAGNOSTIC, WITH BOTH OUTCOMES NAMED IN ADVANCE: an oracle supplies the correct output length, and the authors state before running it that a large improvement would mean the decoder was merely terminating early while a small one would mean the failure is deeper. The control is designed to be uninformative about performance and informative about cause	`mechanisms[3]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: The span contains a LOST-HYPHEN FUSION - 'selfgenerated' for 'self-generated'. A line-break hyphen was dropped by the extraction and the two halves fused. The quote is an exact substring and anchors correctly; the damage is in what a reader of the tag sees. None of the four shapes an internal tool looks for covers this, so it is annotated by hand. Seven such fusions are in this file. also in: A RIVAL EXPLANATION OF THE FAILURE IS ELIMINATED BY A SEPARATE MEASUREMENT: the network preferred its own output to the target by log-likelihood, so the errors are not search failures in the decoder - the model is not failing to FIND the right answer, it does not prefer it	`mechanisms[4]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE SUFFICIENCY OF THE TRAINING DATA IS ARGUED SEPARATELY FROM THE RESULT AND MARKED AS IMPORTANT: the authors state that the training data of both experiments provide enough evidence to learn the composition rules that would afford the correct generalisations, and then spell out, per experiment, what was demonstrated during training	`mechanisms[5]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE DEFLATIONARY READING OF THE RESULT IS STATED BY THE AUTHORS THEMSELVES AND THEN ANSWERED: the finding can be seen as a trivial restatement of the principle that train and test should come from the same distribution - and the reply is that human learners DO generalise to data governed by rules they have learned, so the triviality does not cover the case	`mechanisms[6]`	0.95	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: AN ALTERNATIVE EXPLANATION OF A NULL IS ELIMINATED BY GRADED DOSING RATHER THAN ARGUMENT: composed examples of the new primitive are stepped 8 -> 16 -> 32 and accuracy climbs 38.3 -> 77.8 -> 88.4, appearing GRADUALLY instead of switching on, which rules out the reading that the network had simply lacked evidence	`mechanisms[7]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE CAPACITY IS OPERATIONALISED AS A BEHAVIOURAL SIGNATURE, NOT AS AN ACCURACY: what is expected of a systematically compositional device is ALL-OR-NOTHING rule-based behaviour, so a network that generalises powerfully but needs more positive examples for it to emerge fails the definition WHILE SUCCEEDING AT THE TASK	`mechanisms[8]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE AUTHORS NAME THEIR OWN TEST'S POSSIBLE UNFAIRNESS BEFORE ANYONE ELSE CAN: they say they tested systematicity in its purest form, then concede the setup is arguably too opaque for a computational model which could lack evidence that the two actions are the same sort of thing - and the concession is what motivates the follow-up experiment	`mechanisms[9]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE CONTROL ITEM IS MATCHED ON THE TEST AND DELIBERATELY UNMATCHED ON THE TRAINING: a real adjective seen in 80 different constructions is tested on exactly the same eight constructions as the invented one and scores 8 of 8 against 1 of 8 - so the comparison isolates breadth of training context while holding the test fixed	`mechanisms[10]`	0.85	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: THE POSITIVE HALF OF A NEGATIVE RESULT IS INSISTED ON RATHER THAN CONCEDED: the authors explicitly stress that the high zero-shot accuracy is an important positive result in controlled experiments, and add that understanding the mechanisms behind it is future work - refusing to let the headline failure absorb the finding that sits beside it	`mechanisms[11]`	0.95	the tagger's words
limitations	the SCAN benchmark grammar lacks recursion and generates only a finite command set Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[0]`		the tagger's words
limitations	The machine-translation demonstration is small-scale and its result is described by the authors as preliminary. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[1]`		the tagger's words
limitations	The authors concede the Experiment 3 setup (a primitive seen only in isolation) may be unfairly opaque for a computational model that has no evidence the new action belongs to the same class as the others. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[2]`		the tagger's words
limitations	The benchmark's name is an acronym whose first letter is its own admission of simplification. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any result reported as SCAN accuracy: the authors state the data set is called SCAN because it is Quotation withheld	`limitations[3]`	1.0	the tagger's words
limitations	The criterion the models are measured against comes from an argument the authors call controversial. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the claim that these networks lack systematicity: the standard comes from a set of papers the authors describe as influential and controversial, in which neural networks are argued to be associative devices that cannot capture systematic compositionality	`limitations[4]`	1.0	the tagger's words
limitations	Failure is defined by what an idealised agent would find trivial, not by a measured human baseline. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the description of these failures as catastrophic: the differences are said to be of the sort that would be trivial for an agent equipped with an algebraic mind	`limitations[5]`	1.0	the tagger's words
transfers	the systematicity failure is reframed using machine learning's own assumption that training and test data come from the same distribution Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`	0.9	the tagger's words
controls	the network re-trained with varying numbers of DISTINCT examples while the number of training presentations was held constant at 100K Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: diversity being confounded with volume - more distinct examples otherwise means more gradient steps, and the two cannot be separated · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.85	the tagger's words
concepts	SCAN Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: SCAN	`concepts[5]`		the paper's words
concepts	CommAI Navigation tasks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: CommAI Navigation tasks	`concepts[6]`		the paper's words
concepts	systematic compositionality Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: systematic compositionality	`concepts[7]`		the paper's words
concepts	algebraic mind Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: algebraic mind	`concepts[8]`		the paper's words
concepts	mix-and-match Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mix-and-match	`concepts[9]`		the paper's words
senses	SCAN - AN ACRONYM THAT ADMITS ITS OWN REDUCTION: the S is for Simplified, because the data set is Quotation withheld; the expansion carries the scope limitation and the four letters that get cited do not Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	systematic compositionality - A CONTESTED PHILOSOPHICAL CRITERION USED AS A BENCHMARK: it comes from papers the authors themselves call influential and controversial, in which neural networks are argued to be associative devices that cannot capture it, so a SCAN score inherits an argument it cannot display Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	a catastrophic failure - RELATIVE TO AN ALGEBRAIC MIND: the models' errors are called catastrophic because the differences involved would be trivial for an agent equipped with an algebraic mind, which is a construct rather than a measured comparison group Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	builds-on → sequence-to-sequence learning methods Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → the CommAI Navigation tasks (Mikolov et al., 2016) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → the attentional sequence-to-sequence model of Bahdanau et al. (2015) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → three decades of prior tests of compositional generalization in neural networks in the systematicity debate Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
content_tags	RNNs generalize zero-shot when training and test commands differ only slightly, using mix-and-match strategies, but fail spectacularly when generalization needs systematic compositional skills. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted.	`content_tags.findings[0]`		the tagger's words
content_tags	A proof-of-concept in neural machine translation suggests lack of systematicity partly explains data-hungriness. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted.	`content_tags.findings[1]`		the tagger's words
content_tags	SCAN domain Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: SCAN domain	`content_tags.definitions[0]`		the tagger's words
content_tags	systematic compositionality Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: systematic compositionality	`content_tags.definitions[1]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean	`content_tags.statistics[1]`		the tagger's words
content_tags	standard-error Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: standard-error	`content_tags.statistics[2]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.