phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | systematic compositional generalization in sequence-to-sequence networks— “We thus use it to test a wide range of modern recurrent network architectures in terms of their compositional abilities.” | |
| claim-type | null-result— “However, when generalization requires systematic compositional skills (as in the “dax” example above), RNNs fail spectacularly.” | |
| method — evidence | — “Using the seq2seq framework, we tested a range of standard recurrent neural network models from the literature: simple recurrent networks (SRNs; Elman, 1990), long short-term memory networks (LSTMs; Hochreiter & Schmidhuber, 1997), and gated recurrent units (GRUs; Chung et al., 2014).” | |
| transfer | from statistical machine learning (the train/test distribution assumption): the systematicity failure is reframed using machine learning's own assumption that training and test data come from the same distribution— “This can be seen as a trivial confirmation of the basic principle of statistical machine learning that your training and test data should come from the same distribution.” | transfers[0] |
| mechanism | A STIPULATED EQUIVALENCE CUTS THE TEST SPACE: commands are declared distributionally identical - defined as occurring in exactly the same composed commands - and that declaration is what makes testing the rest redundant. The sameness asserted is a property of the training distribution, not of the actions— “Since “turn right” is distributionally identical to “turn left” (in the sense that it occurs in exactly the same composed commands) and “walk”, “run” and “look” are distributionally identical to “jump”, it is redundant to test all commands.” | mechanisms[0] |
| mechanism | THE DIFFERENCE BETWEEN TWO ITEMS IS EXPLAINED BY EXPOSURE RATHER THAN CAPACITY, and hedged twice while doing it: one primitive's action is seen many times inside other directed actions, which APPARENTLY suffices, PROBABLY because the model gets direct evidence of how it is used in context - while the other action occurs only with its own primitive command and does not generalise from that minimal context— “In the case of “turn left”, although models are only exposed to the primitive command during training, they will see the action it denotes (LTURN) many times, as it is used to accomplish many directed actions. For example, a training item is: “walk left and jump left”, with ground-truth interpretation: LTURN WALK LTURN JUMP. Apparently… | mechanisms[1] |
| mechanism | A NEAR-PERFECT SUCCESS IS DISQUALIFIED ON THE SHAPE OF ITS ERRORS RATHER THAN THEIR NUMBER: all 45 errors are conjunctions containing one of just two components (22 and 23 cases), both of which the model handles correctly on their own - and the authors say it is HARD TO CONCEIVE OF someone who understood the longer command but not the shorter one inside it. The error DISTRIBUTION, not the error rate, is what is read as measuring the capacity— “One would expect such errors to be randomly distributed, or perhaps to pertain to the longest commands or action sequences. Instead, all 45 errors made by the model are conjunctions where one of the components is simple “turn left” (22 cases) or “turn left thrice” (23 cases). This is particularly striking because the network produced the… | mechanisms[2] |
| mechanism | A DELIBERATELY UNREALISTIC MANIPULATION IS ADDED AS A DIAGNOSTIC, WITH BOTH OUTCOMES NAMED IN ADVANCE: an oracle supplies the correct output length, and the authors state before running it that a large improvement would mean the decoder was merely terminating early while a small one would mean the failure is deeper. The control is designed to be uninformative about performance and informative about cause— “If this difficulty is a relatively straightforward issue of the decoder terminating too early, then this should provide an (unrealistic) fix. If this difficulty is symptomatic of deeper problems with generalization, then this change will have only a small effect. With the oracle, the overallbest network performance improved from 13.8% to… | mechanisms[3] |
| mechanism | A RIVAL EXPLANATION OF THE FAILURE IS ELIMINATED BY A SEPARATE MEASUREMENT: the network preferred its own output to the target by log-likelihood, so the errors are not search failures in the decoder - the model is not failing to FIND the right answer, it does not prefer it— “First, we examined the greedy decoder for search-related errors. We confirmed that, for almost every error, the network preferred its selfgenerated output sequence to the target output sequence (as measured by log-likelihood).” | mechanisms[4] |
| mechanism | THE SUFFICIENCY OF THE TRAINING DATA IS ARGUED SEPARATELY FROM THE RESULT AND MARKED AS IMPORTANT: the authors state that the training data of both experiments provide enough evidence to learn the composition rules that would afford the correct generalisations, and then spell out, per experiment, what was demonstrated during training— “Importantly, the training data of experiments 2 and 3 provide enough evidence to learn composition rules affording the correct generalizations. In Experiment 2, the training data contain examples of all modifiers and connectives that are needed at test time for producing longer action sequences. In Experiment 3, the usage of modifiers and… | mechanisms[5] |
| mechanism | THE DEFLATIONARY READING OF THE RESULT IS STATED BY THE AUTHORS THEMSELVES AND THEN ANSWERED: the finding can be seen as a trivial restatement of the principle that train and test should come from the same distribution - and the reply is that human learners DO generalise to data governed by rules they have learned, so the triviality does not cover the case— “This can be seen as a trivial confirmation of the basic principle of statistical machine learning that your training and test data should come from the same distribution. But our results also point to an important difference in how humans and current seq2seq models generalize, since there is no doubt that human learners can generalize to… | mechanisms[6] |
| mechanism | AN ALTERNATIVE EXPLANATION OF A NULL IS ELIMINATED BY GRADED DOSING RATHER THAN ARGUMENT: composed examples of the new primitive are stepped 8 -> 16 -> 32 and accuracy climbs 38.3 -> 77.8 -> 88.4, appearing GRADUALLY instead of switching on, which rules out the reading that the network had simply lacked evidence— “Here, even when shown 8 different composed commands with “jump” at training time, the network only weakly generalizes to other composed commands (38.3% correct). Significant generalization (still far from systematic) shows up when the training set contains 16 and especially 32 distinct composed commands (77.8% and 88.4%, respectively). We… | mechanisms[7] |
| mechanism | THE CAPACITY IS OPERATIONALISED AS A BEHAVIOURAL SIGNATURE, NOT AS AN ACCURACY: what is expected of a systematically compositional device is ALL-OR-NOTHING rule-based behaviour, so a network that generalises powerfully but needs more positive examples for it to emerge fails the definition WHILE SUCCEEDING AT THE TASK— “On the other hand, the runs with more composed examples confirm that, as we found in Experiment 1, the network does display powerful generalization abilities. Simply, they do not conform to the “all-or-nothing” rule-based behaviour we would expect from a systematically compositional device–and, as a consequence, they require more positive… | mechanisms[8] |
| mechanism | THE AUTHORS NAME THEIR OWN TEST'S POSSIBLE UNFAIRNESS BEFORE ANYONE ELSE CAN: they say they tested systematicity in its purest form, then concede the setup is arguably too opaque for a computational model which could lack evidence that the two actions are the same sort of thing - and the concession is what motivates the follow-up experiment— “We tested here systematicity in its purest form: the model was only exposed to “jump” in isolation, and asked to bootstrap to its compositional paradigm based on the behaviour… | mechanisms[9] |
| mechanism | THE CONTROL ITEM IS MATCHED ON THE TEST AND DELIBERATELY UNMATCHED ON THE TRAINING: a real adjective seen in 80 different constructions is tested on exactly the same eight constructions as the invented one and scores 8 of 8 against 1 of 8 - so the comparison isolates breadth of training context while holding the test fixed Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | mechanisms[10] |
| mechanism | THE POSITIVE HALF OF A NEGATIVE RESULT IS INSISTED ON RATHER THAN CONCEDED: the authors explicitly stress that the high zero-shot accuracy is an important positive result in controlled experiments, and add that understanding the mechanisms behind it is future work - refusing to let the headline failure absorb the finding that sits beside it Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | mechanisms[11] |
| builds-on | sequence-to-sequence learning methods Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | claim_relations[0] |
| builds-on | the CommAI Navigation tasks (Mikolov et al., 2016) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | claim_relations[1] |
| builds-on | the attentional sequence-to-sequence model of Bahdanau et al. (2015) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | claim_relations[2] |
| extends | three decades of prior tests of compositional generalization in neural networks in the systematicity debate Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | claim_relations[3] |
| sense | SCAN - AN ACRONYM THAT ADMITS ITS OWN REDUCTION: the S is for Simplified, because the data set is a simplified version of the CommAI Navigation tasks; the expansion carries the scope limitation and the four letters that get cited do not Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[0] |
| sense | systematic compositionality - A CONTESTED PHILOSOPHICAL CRITERION USED AS A BENCHMARK: it comes from papers the authors themselves call influential and controversial, in which neural networks are argued to be associative devices that cannot capture it, so a SCAN score inherits an argument it cannot display Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[1] |
| sense | a catastrophic failure - RELATIVE TO AN ALGEBRAIC MIND: the models' errors are called catastrophic because the differences involved would be trivial for an agent equipped with an algebraic mind, which is a construct rather than a measured comparison group Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | senses[2] |
| control | the network re-trained with varying numbers of DISTINCT examples while the number of training presentations was held constant at 100K — held — rules out diversity being confounded with volume - more distinct examples otherwise means more gradient steps, and the two cannot be separated Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | controls[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — any result reported as SCAN accuracy: the authors state the data set is called SCAN because it is a simplified version of the CommAI Navigation tasks Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| for which use | limitation — the claim that these networks lack systematicity: the standard comes from a set of papers the authors describe as influential and controversial, in which neural networks are argued to be associative devices that cannot capture systematic compositionality Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| for which use | limitation — the description of these failures as catastrophic: the differences are said to be of the sort that would be trivial for an agent equipped with an algebraic mind Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
| evidence status note | mechanism — The span contains a LOST-HYPHEN FUSION - 'overallbest' for 'overall-best'. A line-break hyphen was dropped by the extraction and the two halves fused. The quote is an exact substring and anchors correctly; the damage is in what a reader of the tag sees. None of the four shapes an internal tool looks for covers this, so it is annotated by hand. Seven such fusions are in this file. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | mechanisms[3] |
| evidence status note | mechanism — The span contains a LOST-HYPHEN FUSION - 'selfgenerated' for 'self-generated'. A line-break hyphen was dropped by the extraction and the two halves fused. The quote is an exact substring and anchors correctly; the damage is in what a reader of the tag sees. None of the four shapes an internal tool looks for covers this, so it is annotated by hand. Seven such fusions are in this file. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | mechanisms[4] |
| replication | untested | |
| limitation | the SCAN benchmark grammar lacks recursion and generates only a finite command set Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| limitation | The machine-translation demonstration is small-scale and its result is described by the authors as preliminary. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| limitation | The authors concede the Experiment 3 setup (a primitive seen only in isolation) may be unfairly opaque for a computational model that has no evidence the new action belongs to the same class as the others. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| limitation | The benchmark's name is an acronym whose first letter is its own admission of simplification. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| limitation | The criterion the models are measured against comes from an argument the authors call controversial. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| limitation | Failure is defined by what an idealised agent would find trivial, not by a measured human baseline. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[5] |
finding · quantity · variable · definition · key-term — grounded in the full text| finding | RNNs generalize zero-shot when training and test commands differ only slightly, using mix-and-match strategies, but fail spectacularly when generalization needs systematic compositional skills. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[0] |
| evidence status note | finding — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.findings[0] |
| finding | A proof-of-concept in neural machine translation suggests lack of systematicity partly explains data-hungriness. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.findings[1] |
| evidence status note | finding — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.findings[1] |
| definition | SCAN domain Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[0] |
| evidence status note | definition — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.definitions[0] |
| definition | systematic compositionality Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.definitions[1] |
| evidence status note | definition — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.definitions[1] |
| key-terms | compositional, generalization, systematicity, RNNs, SCAN, commands, recurrent, zero-shot, networks |
| concept | SCAN Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[5] |
| concept | CommAI Navigation tasks Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[6] |
| concept | systematic compositionality Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[7] |
| concept | algebraic mind Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[8] |
| concept | mix-and-match Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[9] |
| mean μ | as written: average value Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[1] |
| standard-error SE | as written: SEM Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[2] |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824