Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. (OpenAI) · Advances in Neural Information Processing Systems (NeurIPS) · 2020 · artificial-intelligence · record brown2020-gpt3
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim type is 'existence' rather than 'method' because the headline claim is that a capability (task-agnostic few-shot performance) appears and grows with scale, not that a new training technique is introduced — GPT-3 is a scaled GPT-2, and the paper says so explicitly. The method facet is 'system-or-tool-building' as the closest fit; the paper is really large-scale benchmark evaluation of a system the authors built, and it additionally contains a genuine human-subjects experiment (621 participants judging whether news articles were machine-generated) that the single-value vocabulary cannot express. Limitations are unusually well served here because §5 is an explicit self-limitations section; those eight are a selection, not the full set. The 'contradicts' relation targets a stated argument by [BHT+20], not their empirical result. 2026-08-13, 2026-08-14 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 15 September 2026: EXCEPTION, measured with the merge gate's own predicate (Paper.locate): 1 quote(s) on this record are not found as one exact string in the current staged text - content_tags.statistics[3].evidence. The sentence above does not hold for them as written. Where a row records why - the quote is grounded in another copy of the paper, joins two passages with an ellipsis, or differs from the staged text by extraction noise - that note stays on the row.

Some words in these quotes contain the wrong character. The PDF this text was extracted from substituted 4 characters inside a word — a currency sign or a stray accent where a letter pair belongs, as in signi¢cantly for significantly, or a single ligature glyph, as in conﬁdence. They are valid characters in the wrong place, so they survive every encoding check. The quotes keep them for the same reason as above, and a search of this page will not find the affected words.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
logistic regression 3 fields · standard-deviation 3 fields · fine-tuning 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 10 concept tags on this paper, 7 are grounded in a quoted sentence. 4 carry the tagger's gloss of what the term means in this paper, which is not something the article can license.

large language model in-context learning few-shot learning prompting scaling fine-tuning zero-shot transfer meta-learning zero-shot one-shot in-context (few-shot) learning: a large pretrained language model performs a new task from a few demonstrations placed in its prompt, with no gradient updates - an ability that emerges from scale language models few-shot autoregressive pre-training Common Crawl task-agnostic demonstrations data contamination model size cross-entropy validation loss learning rate effect-size effect size hypothesis-testing mean sample mean p-value regression logistic regression standard-deviation standard deviation t-test variance system-or-tool-building

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/2005.14165. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 6 · claim_type 1 · concept_edges 1 · concepts 7 · content_tags 27 · controls 1 · limitations 9 · mechanisms 1 · method 1 · phenomenon 1 · senses 8 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	existence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	in-context (few-shot) learning in large language models Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: in-context (few-shot) learning: a large pretrained language model performs a new task from a few demonstrations placed in its prompt, with no gradient updates - an ability that emerges from scale	`mechanisms[0]`		the tagger's words
limitations	one-shot/zero-shot are fairer human comparisons and remain targets for future work Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the human-comparison framing: one-shot and zero-shot are the fairer comparisons and remain IMPORTANT TARGETS FOR FUTURE WORK, not settled here	`limitations[0]`		the tagger's words
limitations	Text synthesis still degrades over long passages: repetition, loss of coherence, self-contradiction, non-sequiturs. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the text-synthesis results specifically: GPT-3 SAMPLES STILL SOMETIMES REPEAT THEMSELVES SEMANTICALLY AT THE DOCUMENT LEVEL, START TO LOSE COHERENCE over long passages	`limitations[1]`		the tagger's words
limitations	The study covers only autoregressive models; no bidirectional architectures or denoising objectives were tested. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the scope of every architectural conclusion: OUR EXPERIMENTS DO NOT INCLUDE ANY BIDIRECTIONAL ARCHITECTURES OR OTHER TRAINING OBJECTIVES SUCH AS DENOISING	`limitations[2]`		the tagger's words
limitations	Pre-training sample efficiency is poor compared with human learning. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any comparison with human learning: POOR SAMPLE EFFICIENCY DURING PRE-TRAINING is shared by language models broadly	`limitations[3]`		the tagger's words
limitations	It is unclear whether few-shot learning genuinely learns new tasks at inference time or merely recognises tasks seen in training. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the interpretation of few-shot learning itself: AMBIGUITY ABOUT WHETHER FEW-SHOT LEARNING ACTUALLY LEARNS NEW TASKS FROM SCRATCH AT INFERENCE time, or recognises tasks seen in training	`limitations[4]`		the tagger's words
limitations	The model is uninterpretable, poorly calibrated on novel inputs, and retains its training data's biases. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any deployment or calibration use: ITS DECISIONS ARE NOT EASILY INTERPRETABLE, IT IS NOT NECESSARILY WELL-CALIBRATED, and it retains its training data's biases	`limitations[5]`		the tagger's words
limitations	A filtering bug left benchmark data in the training set and retraining was unaffordable. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: every benchmark number in the paper: A BUG IN THE FILTERING CAUSED US TO IGNORE SOME OVERLAPS, AND DUE TO THE COST OF TRAINING IT WAS NOT FEASIBLE TO RETRAIN THE MODEL - the contamination is known and was left in	`limitations[6]`		the tagger's words
limitations	The contamination analysis cannot guarantee the 'clean' subsets are distributionally comparable to the full benchmarks. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the contamination correction specifically: WE CANNOT BE SURE THAT THE CLEAN SUBSET IS DRAWN FROM THE SAME DISTRIBUTION AS THE ORIGINAL DATASET - the remedy is unvalidated	`limitations[7]`		the tagger's words
limitations	The bias analysis is preliminary and incomplete even for gender, race and religion. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the bias findings only: THIS IS A PRELIMINARY ANALYSIS AND DOES NOT REFLECT ALL OF THE MODEL'S BIASES EVEN WITHIN THE STUDIED CATEGORIES	`limitations[8]`		the tagger's words
transfers	SAT analogy problems - 374 of them, an instrument built to test American secondary-school pupils - collected and used as a few-shot benchmark for a language model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`	0.85	the tagger's words
controls	a parallel experiment in the same format using intentionally BAD model-generated articles Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: inattentive or low-effort raters as the explanation of the headline result - if participants cannot detect deliberately bad output either, their failure to detect good output means nothing · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.95	the tagger's words
concepts	in-context learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: in-context learning	`concepts[1]`		the paper's words
concepts	few-shot learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: few-shot learning	`concepts[2]`		the paper's words
concepts	fine-tuning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: fine-tuning	`concepts[5]`		the paper's words
concepts	zero-shot transfer Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: zero-shot transfer	`concepts[6]`	1.0	the paper's words
concepts	meta-learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: meta-learning	`concepts[7]`	1.0	the paper's words
concepts	zero-shot Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: zero-shot	`concepts[8]`	0.95	the paper's words
concepts	one-shot Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: one-shot	`concepts[9]`	0.95	the paper's words
senses	few-shot - K demonstrations supplied at inference time as conditioning, with no weight updates; K typically 10 to 100 Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	one-shot - exactly one demonstration, distinguished not on performance grounds but because it matches how tasks are communicated to people Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	zero-shot - no demonstrations at all, only a natural language instruction; described as the most convenient and the most challenging setting Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
senses	in-context learning - a term the authors introduce for the inner loop that occurs within a single forward pass, not for weight change Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	1.0	the tagger's words
senses	`zero-shot transfer` - A TERM RETIRED FOR A NAMED AMBIGUITY, WITH THE TWO READINGS SEPARATED: `the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time demonstrations to the model, so is not truly learning from zero examples.` TWO DIFFERENT ZEROS - zero weight updates and zero examples - collapsed into one word. The repair is a SPLIT: `meta-learning` for the outer structure and `in context-learning` for the inner loop, then `zero-`/`one-`/`few-shot` graded by demonstration count. NOT a redefinition; the old term is abandoned and its two senses are given separate names. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[4]`	1.0	the tagger's words
senses	THE VOCABULARY IS DELIBERATELY UNDER-SPECIFIED, AND THE AUTHORS SAY SO: `Quotation withheld.` A NAMING SCHEME BUILT NOT TO DECIDE A LIVE THEORETICAL DISPUTE. The word `learner` is in the paper's own title, and this footnote withdraws the reading that title invites. MOST DEFINITIONAL WORK IN THIS CORPUS NARROWS A TERM; THIS ONE WIDENS IT ON PURPOSE, so that adopting the vocabulary does not commit a reader to either answer. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[5]`	1.0	the tagger's words
senses	`in-context learning` - THE COINAGE IS ATTACHED TO SOMEBODY ELSE'S WORK. `Recent work [RWC+19] attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification.` The naming and the doing are separated: the prior paper did it, this paper named it. `what we call` is doing that work explicitly, and the name is the one the field kept. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[6]`	0.95	the tagger's words
senses	'zero-shot' / 'one-shot' / 'few-shot' - THE PAPER REFUSES THE COMPARATIVE FRAME ITS OWN NAMES INVITE. 'with the aim of comparing them not as competing alternatives, but as different problem settings which offer a varying trade-off between performance on specific benchmarks and sample efficiency.' A graded series of names - zero, one, few - reads as a ladder with a best rung, and the paper says in advance that it is not one. Compare mechanism 26 ('brown2020' again): there the vocabulary is agnostic on a live dispute; here the vocabulary is agnostic about its own ordering. NO EDGE PROPOSED, AND I WROTE ONE BEFORE DELETING IT. I had 'in-context learning --part-of--> zero-shot' with no relation evidence, which passes the gate because the type-specific cue test only fires when that field is present. Twenty minutes earlier I refused to do exactly that on 'cragg2017' and called it the same shape. The cases are not even parallel in my favour: cragg2017's paper DOES state its partition ('divided into three main components') and only the cue list cannot see it, whereas this paper states the containment nowhere - which is the same reason the 'gurnee2026' edge was dropped. An edge that survives only because I left out the field that would have tested it is not evidence. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[7]`	0.9	the tagger's words
claim_relations	contradicts → the fine-tuning paradigm (task-specific fine-tuning) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	builds-on → GPT-2 [RWC+19] and the Sparse Transformer [CGRS19] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	extends → the in-context learning of [RWC+19] (GPT-2) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	contradicts → [BHT+20]'s argument that continuing to expand hardware and data sizes is not the path forward Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	replicates → the power-law scaling of language-model loss reported by [KMH+20] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
claim_relations	extends → the closed-book QA result of [RRS20] Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[5]`		the tagger's words
concept_edges	few-shot learning —distinct-from→ fine-tuning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: few-shot learning · fine-tuning	`concept_edges[0]`		the tagger's words
content_tags	175 — size of the GPT-3 model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	300 — total training tokens for every model Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	2048 — context window shared by all models Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	8 — models spanning three orders of magnitude Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[3]`		the tagger's words
content_tags	570 — filtered Common Crawl text size (from 45TB raw) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[4]`		the tagger's words
content_tags	93 — share of training data that is English Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[5]`		the tagger's words
content_tags	scaling up language models greatly improves task-agnostic performance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	GPT-3 is evaluated with no gradient updates or fine-tuning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	GPT-3's few-shot result of 71.2% is reported as state-of-the-art Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	GPT-3 can produce news articles humans struggle to distinguish from real ones Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	measured data contamination has minimal effect on most benchmark results Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[4]`		the tagger's words
content_tags	model size Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: model size	`content_tags.variables[0]`		the tagger's words
content_tags	cross-entropy validation loss Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: cross-entropy validation loss	`content_tags.variables[2]`		the tagger's words
content_tags	learning rate Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: learning rate	`content_tags.variables[3]`		the tagger's words
content_tags	in-context learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: in-context learning	`content_tags.definitions[0]`		the tagger's words
content_tags	few-shot Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: few-shot	`content_tags.definitions[1]`		the tagger's words
content_tags	zero-shot Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: zero-shot	`content_tags.definitions[2]`		the tagger's words
content_tags	meta-learning Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: meta-learning	`content_tags.definitions[3]`		the tagger's words
content_tags	data contamination Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: data contamination	`content_tags.definitions[4]`		the tagger's words
content_tags	effect-size Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: effect-size	`content_tags.statistics[0]`		the tagger's words
content_tags	hypothesis-testing Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: hypothesis-testing	`content_tags.statistics[1]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a unicode and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: mean	`content_tags.statistics[2]`		the tagger's words
content_tags	p-value Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: p-value	`content_tags.statistics[3]`		the tagger's words
content_tags	regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: regression	`content_tags.statistics[4]`		the tagger's words
content_tags	standard-deviation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: standard-deviation	`content_tags.statistics[5]`		the tagger's words
content_tags	t-test Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: t-test	`content_tags.statistics[6]`		the tagger's words
content_tags	variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variance	`content_tags.statistics[7]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.