Language Models are Few-Shot Learners

Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah et al. (OpenAI)

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Language Models are Few-Shot Learners 2020 Advances in Neural Information Processing Systems (NeurIPS) 2005.14165 brown2020-gpt3 brown2020-gpt3.txt, the project's staged copy bb619f4c9b8dabec21567c9aaf14059d9a5baf92 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them existence Quotation withheld because this record is not cleared for redistribution. in-context (few-shot) learning in large language models Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms in-context (few-shot) learning: a large pretrained language model performs a new task from a few demonstrations placed in its prompt, with no gradient updates - an ability that emerges from scale Quotation withheld because this record is not cleared for redistribution. one-shot/zero-shot are fairer human comparisons and remain targets for future work what this limits: the human-comparison framing: one-shot and zero-shot are the fairer comparisons and remain IMPORTANT TARGETS FOR FUTURE WORK, not settled here Quotation withheld because this record is not cleared for redistribution. Text synthesis still degrades over long passages: repetition, loss of coherence, self-contradiction, non-sequiturs. what this limits: the text-synthesis results specifically: GPT-3 SAMPLES STILL SOMETIMES REPEAT THEMSELVES SEMANTICALLY AT THE DOCUMENT LEVEL, START TO LOSE COHERENCE over long passages Quotation withheld because this record is not cleared for redistribution. The study covers only autoregressive models; no bidirectional architectures or denoising objectives were tested. what this limits: the scope of every architectural conclusion: OUR EXPERIMENTS DO NOT INCLUDE ANY BIDIRECTIONAL ARCHITECTURES OR OTHER TRAINING OBJECTIVES SUCH AS DENOISING Quotation withheld because this record is not cleared for redistribution. Pre-training sample efficiency is poor compared with human learning. what this limits: any comparison with human learning: POOR SAMPLE EFFICIENCY DURING PRE-TRAINING is shared by language models broadly Quotation withheld because this record is not cleared for redistribution. It is unclear whether few-shot learning genuinely learns new tasks at inference time or merely recognises tasks seen in training. what this limits: the interpretation of few-shot learning itself: AMBIGUITY ABOUT WHETHER FEW-SHOT LEARNING ACTUALLY LEARNS NEW TASKS FROM SCRATCH AT INFERENCE time, or recognises tasks seen in training Quotation withheld because this record is not cleared for redistribution. The model is uninterpretable, poorly calibrated on novel inputs, and retains its training data's biases. what this limits: any deployment or calibration use: ITS DECISIONS ARE NOT EASILY INTERPRETABLE, IT IS NOT NECESSARILY WELL-CALIBRATED, and it retains its training data's biases Quotation withheld because this record is not cleared for redistribution. A filtering bug left benchmark data in the training set and retraining was unaffordable. what this limits: every benchmark number in the paper: A BUG IN THE FILTERING CAUSED US TO IGNORE SOME OVERLAPS, AND DUE TO THE COST OF TRAINING IT WAS NOT FEASIBLE TO RETRAIN THE MODEL - the contamination is known and was left in Quotation withheld because this record is not cleared for redistribution. The contamination analysis cannot guarantee the 'clean' subsets are distributionally comparable to the full benchmarks. what this limits: the contamination correction specifically: WE CANNOT BE SURE THAT THE CLEAN SUBSET IS DRAWN FROM THE SAME DISTRIBUTION AS THE ORIGINAL DATASET - the remedy is unvalidated Quotation withheld because this record is not cleared for redistribution. The bias analysis is preliminary and incomplete even for gender, race and religion. what this limits: the bias findings only: THIS IS A PRELIMINARY ANALYSIS AND DOES NOT REFLECT ALL OF THE MODEL'S BIASES EVEN WITHIN THE STUDIED CATEGORIES Quotation withheld because this record is not cleared for redistribution. SAT analogy problems - 374 of them, an instrument built to test American secondary-school pupils - collected and used as a few-shot benchmark for a language model Quotation withheld because this record is not cleared for redistribution. a parallel experiment in the same format using intentionally BAD model-generated articles eliminates: inattentive or low-effort raters as the explanation of the headline result - if participants cannot detect deliberately bad output either, their failure to detect good output means nothing · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. in-context learning in-context learning Quotation withheld because this record is not cleared for redistribution. few-shot learning few-shot learning Quotation withheld because this record is not cleared for redistribution. fine-tuning fine-tuning Quotation withheld because this record is not cleared for redistribution. zero-shot transfer zero-shot transfer Quotation withheld because this record is not cleared for redistribution. meta-learning meta-learning Quotation withheld because this record is not cleared for redistribution. zero-shot zero-shot Quotation withheld because this record is not cleared for redistribution. one-shot one-shot Quotation withheld because this record is not cleared for redistribution. few-shot - K demonstrations supplied at inference time as conditioning, with no weight updates; K typically 10 to 100 Quotation withheld because this record is not cleared for redistribution. one-shot - exactly one demonstration, distinguished not on performance grounds but because it matches how tasks are communicated to people Quotation withheld because this record is not cleared for redistribution. zero-shot - no demonstrations at all, only a natural language instruction; described as the most convenient and the most challenging setting Quotation withheld because this record is not cleared for redistribution. in-context learning - a term the authors introduce for the inner loop that occurs within a single forward pass, not for weight change Quotation withheld because this record is not cleared for redistribution. `zero-shot transfer` - A TERM RETIRED FOR A NAMED AMBIGUITY, WITH THE TWO READINGS SEPARATED: `the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time demonstrations to the model, so is not truly learning from zero examples.` TWO DIFFERENT ZEROS - zero weight updates and zero examples - collapsed into one word. The repair is a SPLIT: `meta-learning` for the outer structure and `in context-learning` for the inner loop, then `zero-`/`one-`/`few-shot` graded by demonstration count. NOT a redefinition; the old term is abandoned and its two senses are given separate names. Quotation withheld because this record is not cleared for redistribution. THE VOCABULARY IS DELIBERATELY UNDER-SPECIFIED, AND THE AUTHORS SAY SO: `Quotation withheld.` A NAMING SCHEME BUILT NOT TO DECIDE A LIVE THEORETICAL DISPUTE. The word `learner` is in the paper's own title, and this footnote withdraws the reading that title invites. MOST DEFINITIONAL WORK IN THIS CORPUS NARROWS A TERM; THIS ONE WIDENS IT ON PURPOSE, so that adopting the vocabulary does not commit a reader to either answer. Quotation withheld because this record is not cleared for redistribution. `in-context learning` - THE COINAGE IS ATTACHED TO SOMEBODY ELSE'S WORK. `Recent work [RWC+19] attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification.` The naming and the doing are separated: the prior paper did it, this paper named it. `what we call` is doing that work explicitly, and the name is the one the field kept. Quotation withheld because this record is not cleared for redistribution. 'zero-shot' / 'one-shot' / 'few-shot' - THE PAPER REFUSES THE COMPARATIVE FRAME ITS OWN NAMES INVITE. 'with the aim of comparing them not as competing alternatives, but as different problem settings which offer a varying trade-off between performance on specific benchmarks and sample efficiency.' A graded series of names - zero, one, few - reads as a ladder with a best rung, and the paper says in advance that it is not one. Compare mechanism 26 ('brown2020' again): there the vocabulary is agnostic on a live dispute; here the vocabulary is agnostic about its own ordering. NO EDGE PROPOSED, AND I WROTE ONE BEFORE DELETING IT. I had 'in-context learning --part-of--> zero-shot' with no relation evidence, which passes the gate because the type-specific cue test only fires when that field is present. Twenty minutes earlier I refused to do exactly that on 'cragg2017' and called it the same shape. The cases are not even parallel in my favour: cragg2017's paper DOES state its partition ('divided into three main components') and only the cue list cannot see it, whereas this paper states the containment nowhere - which is the same reason the 'gurnee2026' edge was dropped. An edge that survives only because I left out the field that would have tested it is not evidence. Quotation withheld because this record is not cleared for redistribution. contradicts → the fine-tuning paradigm (task-specific fine-tuning) Quotation withheld because this record is not cleared for redistribution. builds-on → GPT-2 [RWC+19] and the Sparse Transformer [CGRS19] Quotation withheld because this record is not cleared for redistribution. extends → the in-context learning of [RWC+19] (GPT-2) Quotation withheld because this record is not cleared for redistribution. contradicts → [BHT+20]'s argument that continuing to expand hardware and data sizes is not the path forward Quotation withheld because this record is not cleared for redistribution. replicates → the power-law scaling of language-model loss reported by [KMH+20] Quotation withheld because this record is not cleared for redistribution. extends → the closed-book QA result of [RRS20] Quotation withheld because this record is not cleared for redistribution. few-shot learning —distinct-from→ fine-tuning few-shot learning fine-tuning strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 175 — size of the GPT-3 model Quotation withheld because this record is not cleared for redistribution. 300 — total training tokens for every model Quotation withheld because this record is not cleared for redistribution. 2048 — context window shared by all models Quotation withheld because this record is not cleared for redistribution. 8 — models spanning three orders of magnitude Quotation withheld because this record is not cleared for redistribution. 570 — filtered Common Crawl text size (from 45TB raw) Quotation withheld because this record is not cleared for redistribution. 93 — share of training data that is English Quotation withheld because this record is not cleared for redistribution. scaling up language models greatly improves task-agnostic performance Quotation withheld because this record is not cleared for redistribution. GPT-3 is evaluated with no gradient updates or fine-tuning Quotation withheld because this record is not cleared for redistribution. GPT-3's few-shot result of 71.2% is reported as state-of-the-art Quotation withheld because this record is not cleared for redistribution. GPT-3 can produce news articles humans struggle to distinguish from real ones Quotation withheld because this record is not cleared for redistribution. measured data contamination has minimal effect on most benchmark results Quotation withheld because this record is not cleared for redistribution. model size model size Quotation withheld because this record is not cleared for redistribution. cross-entropy validation loss cross-entropy validation loss Quotation withheld because this record is not cleared for redistribution. learning rate learning rate Quotation withheld because this record is not cleared for redistribution. in-context learning in-context learning Quotation withheld because this record is not cleared for redistribution. few-shot few-shot Quotation withheld because this record is not cleared for redistribution. zero-shot zero-shot Quotation withheld because this record is not cleared for redistribution. meta-learning meta-learning Quotation withheld because this record is not cleared for redistribution. data contamination data contamination Quotation withheld because this record is not cleared for redistribution. effect-size effect-size Quotation withheld because this record is not cleared for redistribution. hypothesis-testing hypothesis-testing Quotation withheld because this record is not cleared for redistribution. mean mean about this quote: QUOTE REPAIRED: the extractor split a unicode and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. p-value p-value Quotation withheld because this record is not cleared for redistribution. regression regression Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation Quotation withheld because this record is not cleared for redistribution. t-test t-test Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/2005.14165.