Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov · Journal of Machine Learning Research · 2014 · artificial-intelligence · record srivastava2014-dropout
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Extraction is clean apart from figure-axis text and inline math fragments; all quotes are from running prose. Claim type is 'method' because the contribution is the training technique itself, but the paper also advances a mechanistic hypothesis for why it works ('We hypothesize that for each hidden unit, dropout prevents co-adaptation by making the presence of other hidden units unreliable.'), which the single-valued facet cannot hold alongside the method claim. Method label is 'computational-experiment' - benchmark training runs across seven data sets - since none of the suggested labels fits a machine-learning empirical evaluation. Only one claim_relation is recorded: the related-work section mentions van der Maaten et al. (2013), Wang and Manning (2013), Chen et al. (2012), Globerson and Roweis (2006) and others, but those are descriptions of neighbouring work rather than statements of how this paper stands to them, so per the rule they are left out. Replication status is 'untested' in the sense of independent replication; the paper's own repeated evaluations across data sets are internal. 2026-08-14 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
deep learning 11 fields · overfitting 6 fields · training 3 fields · neural networks 3 fields · standard-deviation 3 fields · linear regression 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 8 concept tags on this paper, 5 are grounded in a quoted sentence.

dropout regularization overfitting neural network geometric mean sexual reproduction thinned model combination randomly drop units during training so neurons cannot co-adapt; this approximates averaging an exponential ensemble of thinned networks, regularizing the model neural networks thinned networks co-adapting deep learning training hidden units thinned network Bayesian mean mean of regression linear regression standard-deviation standard deviation system-or-tool-building

Method, as described: computational-experiment

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://jmlr.org/papers/v15/srivastava14a.html. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 2 · claim_type 1 · concept_edges 1 · concepts 5 · content_tags 16 · controls 1 · limitations 8 · mechanisms 1 · method 1 · phenomenon 1 · senses 3 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	overfitting in deep neural networks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: randomly drop units during training so neurons cannot co-adapt; this approximates averaging an exponential ensemble of thinned networks, regularizing the model	`mechanisms[0]`		the tagger's words
limitations	dropout increases training time Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the cost side of adopting dropout: ONE OF THE DRAWBACKS OF DROPOUT IS THAT IT INCREASES TRAINING TIME	`limitations[0]`		the tagger's words
limitations	Dropout makes training substantially slower. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. duplicate of: limitations[0] · what this limits: the same scope as limitations[0] - training is substantially slower	`limitations[1]`		the tagger's words
limitations	On very small training sets dropout gives no benefit at all. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: small-data applications, where the benefit is measured at zero: FOR EXTREMELY SMALL DATA SETS (100, 500) DROPOUT DOES NOT GIVE ANY IMPROVEMENTS	`limitations[2]`		the tagger's words
limitations	The gain on the Reuters text data set was far smaller than on vision and speech data. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: text applications specifically: on Reuters THE IMPROVEMENT WAS MUCH SMALLER COMPARED TO THAT FOR THE VISION AND SPEECH DATA SETS	`limitations[3]`		the tagger's words
limitations	The equivalent deterministic regularizer is only derived for linear regression; for richer models it is unknown. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any theoretical account of dropout beyond linear regression: FOR MORE COMPLICATED MODELS, IT IS NOT OBVIOUS HOW TO OBTAIN AN EQUIVALENT REGULARIZER	`limitations[4]`		the tagger's words
limitations	The method's stated motivation is an analogy to sexual reproduction, not a derivation. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any account of why dropout works: the authors give as motivation a theory of the role of sex in evolution, in which reproduction combines half the genes of each parent with a small amount of random mutation	`limitations[5]`	1.0	the tagger's words
limitations	A second, separate motivation is offered by analogy to conspiracies. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any account of why dropout works: the authors offer a second motivation, described as closely related but slightly different, from thinking about successful conspiracies	`limitations[6]`	1.0	the tagger's words
limitations	The test-time procedure is an approximation to averaging an exponential number of networks. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the weight-scaling rule used at test time: the authors describe it as	`limitations[7]`	1.0	the tagger's words
transfers	dropout as a neural-net regularizer inspired by the evolutionary role of sex - breaking co-adaptations, analogous to genetic recombination Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
controls	the same architectures trained with and without dropout, differing only in whether a thinned network is sampled for each training case in a mini-batch Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: an improvement owed to the architecture, the data or the training schedule rather than to dropout itself · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.85	the tagger's words
concepts	dropout Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: dropout	`concepts[0]`		the paper's words
concepts	geometric mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: geometric mean	`concepts[4]`		the paper's words
concepts	sexual reproduction Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: sexual reproduction	`concepts[5]`		the paper's words
concepts	thinned Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: thinned	`concepts[6]`		the paper's words
concepts	model combination Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: model combination	`concepts[7]`		the paper's words
senses	the motivation for dropout - AN ANALOGY, FLAGGED AS ONE: a theory of the role of sex in evolution, in which combining half of each parent's genes beats copying a good genome directly; the paper presents this as a motivation and not as a derivation, and its only proof of an equivalent regularizer is for linear regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	the test-time network - A STAND-IN FOR AN ENSEMBLE THAT IS NEVER BUILT: a single unthinned network with scaled weights approximates an equally weighted geometric mean over an exponential number of thinned networks, so the rule everyone implements is a cheap approximation to a computation nobody performs Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	a thinned network - ONE SAMPLE FROM 2^n: applying dropout amounts to sampling a thinned network, and a net with n units is described as a collection of that many possible thinned networks all sharing weights, so 'a dropout network' names a distribution over architectures rather than one architecture Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
claim_relations	extends → prior regularization methods for neural networks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	extends → denoising autoencoders (Vincent et al., 2008, 2010), where noise is added to the input units Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
concept_edges	dropout —operationalised-as→ thinned Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says applying dropout AMOUNTS TO sampling a 'thinned' network; operationalised-as is loose. also in: dropout · thinned	`concept_edges[0]`		the tagger's words
content_tags	0.5 — typical unit-retention probability Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	2 — example net in Figure 1 Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[2]`		the tagger's words
content_tags	~1 — input units are retained with probability near 1 Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[3]`		the tagger's words
content_tags	dropout prevents units from co-adapting too much Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	training samples from exponentially many thinned networks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted.	`content_tags.findings[1]`		the tagger's words
content_tags	dropout reduces overfitting and beats other regularizers Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	dropout achieves state-of-the-art on many benchmarks Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	overfitting Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: overfitting	`content_tags.variables[3]`		the tagger's words
content_tags	dropout Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: dropout	`content_tags.definitions[0]`		the tagger's words
content_tags	thinned network Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: thinned network	`content_tags.definitions[1]`		the tagger's words
content_tags	overfitting Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: overfitting	`content_tags.definitions[2]`		the tagger's words
content_tags	model combination Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: model combination	`content_tags.definitions[3]`		the tagger's words
content_tags	Bayesian Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Bayesian	`content_tags.statistics[0]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean	`content_tags.statistics[1]`		the tagger's words
content_tags	regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: regression	`content_tags.statistics[3]`		the tagger's words
content_tags	standard-deviation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: standard-deviation	`content_tags.statistics[4]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.