Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Dropout: A Simple Way to Prevent Neural Networks from Overfitting Nitish Srivastava Geoffrey Hinton Alex Krizhevsky Ilya Sutskever Ruslan Salakhutdinov

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Dropout: A Simple Way to Prevent Neural Networks from Overfitting 2014 Journal of Machine Learning Research srivastava2014-dropout srivastava2014-dropout.txt, the project's staged copy 55ab7c0eaa57cc1f8242d1d7a15cf5ab7c0f4c02 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. overfitting in deep neural networks Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms randomly drop units during training so neurons cannot co-adapt; this approximates averaging an exponential ensemble of thinned networks, regularizing the model Quotation withheld because this record is not cleared for redistribution. dropout increases training time what this limits: the cost side of adopting dropout: ONE OF THE DRAWBACKS OF DROPOUT IS THAT IT INCREASES TRAINING TIME Quotation withheld because this record is not cleared for redistribution. Dropout makes training substantially slower. duplicate of: limitations[0] · what this limits: the same scope as limitations[0] - training is substantially slower Quotation withheld because this record is not cleared for redistribution. On very small training sets dropout gives no benefit at all. what this limits: small-data applications, where the benefit is measured at zero: FOR EXTREMELY SMALL DATA SETS (100, 500) DROPOUT DOES NOT GIVE ANY IMPROVEMENTS Quotation withheld because this record is not cleared for redistribution. The gain on the Reuters text data set was far smaller than on vision and speech data. what this limits: text applications specifically: on Reuters THE IMPROVEMENT WAS MUCH SMALLER COMPARED TO THAT FOR THE VISION AND SPEECH DATA SETS Quotation withheld because this record is not cleared for redistribution. The equivalent deterministic regularizer is only derived for linear regression; for richer models it is unknown. what this limits: any theoretical account of dropout beyond linear regression: FOR MORE COMPLICATED MODELS, IT IS NOT OBVIOUS HOW TO OBTAIN AN EQUIVALENT REGULARIZER Quotation withheld because this record is not cleared for redistribution. The method's stated motivation is an analogy to sexual reproduction, not a derivation. what this limits: any account of why dropout works: the authors give as motivation a theory of the role of sex in evolution, in which reproduction combines half the genes of each parent with a small amount of random mutation Quotation withheld because this record is not cleared for redistribution. A second, separate motivation is offered by analogy to conspiracies. what this limits: any account of why dropout works: the authors offer a second motivation, described as closely related but slightly different, from thinking about successful conspiracies Quotation withheld because this record is not cleared for redistribution. The test-time procedure is an approximation to averaging an exponential number of networks. what this limits: the weight-scaling rule used at test time: the authors describe it as Quotation withheld because this record is not cleared for redistribution. dropout as a neural-net regularizer inspired by the evolutionary role of sex - breaking co-adaptations, analogous to genetic recombination Quotation withheld because this record is not cleared for redistribution. the same architectures trained with and without dropout, differing only in whether a thinned network is sampled for each training case in a mini-batch eliminates: an improvement owed to the architecture, the data or the training schedule rather than to dropout itself · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. dropout dropout Quotation withheld because this record is not cleared for redistribution. geometric mean geometric mean Quotation withheld because this record is not cleared for redistribution. sexual reproduction sexual reproduction Quotation withheld because this record is not cleared for redistribution. thinned thinned Quotation withheld because this record is not cleared for redistribution. model combination model combination Quotation withheld because this record is not cleared for redistribution. the motivation for dropout - AN ANALOGY, FLAGGED AS ONE: a theory of the role of sex in evolution, in which combining half of each parent's genes beats copying a good genome directly; the paper presents this as a motivation and not as a derivation, and its only proof of an equivalent regularizer is for linear regression Quotation withheld because this record is not cleared for redistribution. the test-time network - A STAND-IN FOR AN ENSEMBLE THAT IS NEVER BUILT: a single unthinned network with scaled weights approximates an equally weighted geometric mean over an exponential number of thinned networks, so the rule everyone implements is a cheap approximation to a computation nobody performs Quotation withheld because this record is not cleared for redistribution. a thinned network - ONE SAMPLE FROM 2^n: applying dropout amounts to sampling a thinned network, and a net with n units is described as a collection of that many possible thinned networks all sharing weights, so 'a dropout network' names a distribution over architectures rather than one architecture Quotation withheld because this record is not cleared for redistribution. extends → prior regularization methods for neural networks Quotation withheld because this record is not cleared for redistribution. extends → denoising autoencoders (Vincent et al., 2008, 2010), where noise is added to the input units Quotation withheld because this record is not cleared for redistribution. dropout —operationalised-as→ thinned dropout thinned strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says applying dropout AMOUNTS TO sampling a 'thinned' network; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. 0.5 — typical unit-retention probability Quotation withheld because this record is not cleared for redistribution. 2 — example net in Figure 1 Quotation withheld because this record is not cleared for redistribution. ~1 — input units are retained with probability near 1 Quotation withheld because this record is not cleared for redistribution. dropout prevents units from co-adapting too much Quotation withheld because this record is not cleared for redistribution. training samples from exponentially many thinned networks about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. dropout reduces overfitting and beats other regularizers Quotation withheld because this record is not cleared for redistribution. dropout achieves state-of-the-art on many benchmarks Quotation withheld because this record is not cleared for redistribution. overfitting overfitting Quotation withheld because this record is not cleared for redistribution. dropout dropout about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. thinned network thinned network Quotation withheld because this record is not cleared for redistribution. overfitting overfitting Quotation withheld because this record is not cleared for redistribution. model combination model combination Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. regression regression about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. standard-deviation standard-deviation Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://jmlr.org/papers/v15/srivastava14a.html.