| phenomenon | in-context (few-shot) learning in large language models— “Recent work [RWC+19] attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by… |
| claim-type | existence— “Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.” |
| method — evidence | — “In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call GPT-3, and measuring its in-context learning abilities.” |
| transfer | from educational psychometrics: SAT analogy problems - 374 of them, an instrument built to test American secondary-school pupils - collected and used as a few-shot benchmark for a language model— “To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of 374 “SAT analogy” problems [TLBS03].” | transfers[0] |
| mechanism | in-context (few-shot) learning: a large pretrained language model performs a new task from a few demonstrations placed in its prompt, with no gradient updates - an ability that emerges from scale— “GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model” | mechanisms[0] |
| contradicts | the fine-tuning paradigm (task-specific fine-tuning)— “GPT-3 is applied without any gradient updates or fine-tuning” | claim_relations[0] |
| builds-on | GPT-2 [RWC+19] and the Sparse Transformer [CGRS19]— “We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [CGRS19].” | claim_relations[1] |
| extends | the in-context learning of [RWC+19] (GPT-2)— “Our use of in-context learning is also similar to [RWC+19], but in this work we systematically explore different settings for learning within the context.” | claim_relations[2] |
| contradicts | [BHT+20]'s argument that continuing to expand hardware and data sizes is not the path forward— “We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.” | claim_relations[3] |
| replicates | the power-law scaling of language-model loss reported by [KMH+20]— “The power-law behavior observed in [KMH+20] continues for an additional two orders of magnitude with only small deviations from the predicted curve.” | claim_relations[4] |
| extends | the closed-book QA result of [RRS20]— “Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3.” | claim_relations[5] |
| sense | few-shot - K demonstrations supplied at inference time as conditioning, with no weight updates; K typically 10 to 100— “Few-Shot (FS) is the term we will use in this work to refer to the setting where the model is given a few demonstrations of the task at inference time as conditioning [RWC+19], but no weight updates are allowed.” | senses[0] |
| sense | one-shot - exactly one demonstration, distinguished not on performance grounds but because it matches how tasks are communicated to people— “One-Shot (1S) is the same as few-shot except that only one demonstration is allowed, in addition to a natural language description of the task, as shown in Figure 1. The reason to distinguish one-shot from few-shot and zero-shot (below) is that it most closely matches the way in which some tasks are communicated to humans.” | senses[1] |
| sense | zero-shot - no demonstrations at all, only a natural language instruction; described as the most convenient and the most challenging setting— “Zero-Shot (0S) is the same as one-shot except that no demonstrations are allowed, and the model is only given a natural language instruction describing the task. This method provides maximum convenience, potential for robustness, and avoidance of spurious correlations (unless they occur very broadly across the large corpus of pre-training… | senses[2] |
| sense | in-context learning - a term the authors introduce for the inner loop that occurs within a single forward pass, not for weight change— “We use the term “in-context learning” to describe the inner loop of this process, which occurs within the forward-pass upon each sequence.” | senses[3] |
| sense | `zero-shot transfer` - A TERM RETIRED FOR A NAMED AMBIGUITY, WITH THE TWO READINGS SEPARATED: `the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time demonstrations to the model, so is not truly learning from zero examples.` TWO DIFFERENT ZEROS - zero weight updates and zero examples - collapsed into one word. The repair is a SPLIT: `meta-learning` for the outer structure and `in context-learning` for the inner loop, then `zero-`/`one-`/`few-shot` graded by demonstration count. NOT a redefinition; the old term is abandoned and its two senses are given separate names.— “In the context of language models this has sometimes been called “zero-shot transfer”, but this term is potentially ambiguous: the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time demonstrations to the model, so is not truly learning from zero examples. To avoid this… | senses[4] |
| sense | THE VOCABULARY IS DELIBERATELY UNDER-SPECIFIED, AND THE AUTHORS SAY SO: `These terms are intended to remain agnostic on the question of whether the model learns new tasks from scratch at inference time or simply recognizes patterns seen during training.` A NAMING SCHEME BUILT NOT TO DECIDE A LIVE THEORETICAL DISPUTE. The word `learner` is in the paper's own title, and this footnote withdraws the reading that title invites. MOST DEFINITIONAL WORK IN THIS CORPUS NARROWS A TERM; THIS ONE WIDENS IT ON PURPOSE, so that adopting the vocabulary does not commit a reader to either answer.— “These terms are intended to remain agnostic on the question of whether the model learns new tasks from scratch at inference time or simply recognizes patterns seen during training” | senses[5] |
| sense | `in-context learning` - THE COINAGE IS ATTACHED TO SOMEBODY ELSE'S WORK. `Recent work [RWC+19] attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification.` The naming and the doing are separated: the prior paper did it, this paper named it. `what we call` is doing that work explicitly, and the name is the one the field kept.— “attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task” | senses[6] |
| sense | 'zero-shot' / 'one-shot' / 'few-shot' - THE PAPER REFUSES THE COMPARATIVE FRAME ITS OWN NAMES INVITE. 'with the aim of comparing them not as competing alternatives, but as different problem settings which offer a varying trade-off between performance on specific benchmarks and sample efficiency.' A graded series of names - zero, one, few - reads as a ladder with a best rung, and the paper says in advance that it is not one. Compare mechanism 26 ('brown2020' again): there the vocabulary is agnostic on a live dispute; here the vocabulary is agnostic about its own ordering. NO EDGE PROPOSED, AND I WROTE ONE BEFORE DELETING IT. I had 'in-context learning --part-of--> zero-shot' with no relation evidence, which passes the gate because the type-specific cue test only fires when that field is present. Twenty minutes earlier I refused to do exactly that on 'cragg2017' and called it the same shape. The cases are not even parallel in my favour: cragg2017's paper DOES state its partition ('divided into three main components') and only the cue list cannot see it, whereas this paper states the containment nowhere - which is the same reason the 'gurnee2026' edge was dropped. An edge that survives only because I left out the field that would have tested it is not evidence.— “we focus on zero-shot, one-shot and few-shot, with the aim of comparing them not as competing alternatives, but as different problem settings which offer a varying trade-off between performance on specific benchmarks and sample efficiency” | senses[7] |
| control | a parallel experiment in the same format using intentionally BAD model-generated articles — held — rules out inattentive or low-effort raters as the explanation of the headline result - if participants cannot detect deliberately bad output either, their failure to detect good output means nothing— “However, we also ran an experiment to control for participant effort and attention that followed the same format but involved intentionally bad model generated articles.” | controls[0] |
| edge | few-shot learning —distinct-from→ fine-tuning— “In this work we do not fine-tune GPT-3 because our focus is on task-agnostic performance, but GPT-3 can be fine-tuned in principle and this is a promising direction for future work. • Few-Shot (FS) is the term we will use in this work to refer to the setting where the model is given a few demonstrations of the task at inference time as… | concept_edges[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — the human-comparison framing: one-shot and zero-shot are the fairer comparisons and remain IMPORTANT TARGETS FOR FUTURE WORK, not settled here— “important targets for future work” | limitations[0] |
| for which use | limitation — the text-synthesis results specifically: GPT-3 SAMPLES STILL SOMETIMES REPEAT THEMSELVES SEMANTICALLY AT THE DOCUMENT LEVEL, START TO LOSE COHERENCE over long passages— “On text synthesis, although the overall quality is high, GPT-3 samples still sometimes repeat themselves semantically at the document level, start to lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs.” | limitations[1] |
| for which use | limitation — the scope of every architectural conclusion: OUR EXPERIMENTS DO NOT INCLUDE ANY BIDIRECTIONAL ARCHITECTURES OR OTHER TRAINING OBJECTIVES SUCH AS DENOISING— “As a result our experiments do not include any bidirectional architectures or other training objectives such as denoising.” | limitations[2] |
| for which use | limitation — any comparison with human learning: POOR SAMPLE EFFICIENCY DURING PRE-TRAINING is shared by language models broadly— “Another limitation broadly shared by language models is poor sample efficiency during pre-training.” | limitations[3] |
| for which use | limitation — the interpretation of few-shot learning itself: AMBIGUITY ABOUT WHETHER FEW-SHOT LEARNING ACTUALLY LEARNS NEW TASKS FROM SCRATCH AT INFERENCE time, or recognises tasks seen in training— “A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot learning actually learns new tasks “from scratch” at inference time, or if it simply recognizes and identifies tasks that it has learned during training.” | limitations[4] |
| for which use | limitation — any deployment or calibration use: ITS DECISIONS ARE NOT EASILY INTERPRETABLE, IT IS NOT NECESSARILY WELL-CALIBRATED, and it retains its training data's biases— “Finally, GPT-3 shares some limitations common to most deep learning systems – its decisions are not easily interpretable, it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in performance than humans on standard benchmarks, and it retains the biases of the data it has been trained… | limitations[5] |
| for which use | limitation — every benchmark number in the paper: A BUG IN THE FILTERING CAUSED US TO IGNORE SOME OVERLAPS, AND DUE TO THE COST OF TRAINING IT WAS NOT FEASIBLE TO RETRAIN THE MODEL - the contamination is known and was left in— “Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.” | limitations[6] |
| for which use | limitation — the contamination correction specifically: WE CANNOT BE SURE THAT THE CLEAN SUBSET IS DRAWN FROM THE SAME DISTRIBUTION AS THE ORIGINAL DATASET - the remedy is unvalidated— “An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the same distribution as the original dataset.” | limitations[7] |
| for which use | limitation — the bias findings only: THIS IS A PRELIMINARY ANALYSIS AND DOES NOT REFLECT ALL OF THE MODEL'S BIASES EVEN WITHIN THE STUDIED CATEGORIES— “This is a preliminary analysis and does not reflect all of the model’s biases even within the studied categories.” | limitations[8] |
| replication | untested |
| limitation | one-shot/zero-shot are fairer human comparisons and remain targets for future work— “important targets for future work” | limitations[0] |
| limitation | Text synthesis still degrades over long passages: repetition, loss of coherence, self-contradiction, non-sequiturs.— “On text synthesis, although the overall quality is high, GPT-3 samples still sometimes repeat themselves semantically at the document level, start to lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs.” | limitations[1] |
| limitation | The study covers only autoregressive models; no bidirectional architectures or denoising objectives were tested.— “As a result our experiments do not include any bidirectional architectures or other training objectives such as denoising.” | limitations[2] |
| limitation | Pre-training sample efficiency is poor compared with human learning.— “Another limitation broadly shared by language models is poor sample efficiency during pre-training.” | limitations[3] |
| limitation | It is unclear whether few-shot learning genuinely learns new tasks at inference time or merely recognises tasks seen in training.— “A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot learning actually learns new tasks “from scratch” at inference time, or if it simply recognizes and identifies tasks that it has learned during training.” | limitations[4] |
| limitation | The model is uninterpretable, poorly calibrated on novel inputs, and retains its training data's biases.— “Finally, GPT-3 shares some limitations common to most deep learning systems – its decisions are not easily interpretable, it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in performance than humans on standard benchmarks, and it retains the biases of the data it has been trained… | limitations[5] |
| limitation | A filtering bug left benchmark data in the training set and retraining was unaffordable.— “Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.” | limitations[6] |
| limitation | The contamination analysis cannot guarantee the 'clean' subsets are distributionally comparable to the full benchmarks.— “An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the same distribution as the original dataset.” | limitations[7] |
| limitation | The bias analysis is preliminary and incomplete even for gender, race and religion.— “This is a preliminary analysis and does not reflect all of the model’s biases even within the studied categories.” | limitations[8] |