Adam: A Method for Stochastic Optimization

Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Adam: A Method for Stochastic Optimization 2014 ICLR / arXiv 1412.6980 kingma2014-adam kingma2014-adam.txt, the project's staged copy 685c8f034ec860c572d0d9d8b467427c5617ea28 arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. stochastic gradient-based optimization Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. The moment estimates start at zero, so early estimates are biased toward zero; the correction divides by one minus the decay rate raised to the timestep. initialisation bias from zero-initialised moving averages Quotation withheld because this record is not cleared for redistribution. As an optimum is approached the ratio falls, so the effective step shrinks with no schedule imposed. automatic annealing via the signal-to-noise ratio Quotation withheld because this record is not cleared for redistribution. Rescaling the gradient by c scales the first moment by c and the second by c squared, and the two cancel in the ratio. invariance of the update to gradient rescaling Quotation withheld because this record is not cleared for redistribution. The convergence analysis covers only convex objectives, not the non-convex problems on which Adam is mostly used. what this limits: the convergence guarantee, which does not cover the setting the method is used in: Quotation withheld Quotation withheld because this record is not cleared for redistribution. The Lp-norm generalisation of the update rule is numerically unstable except in the infinity-norm limit. what this limits: the Lp-norm generalisation of the update rule: SUCH VARIANTS BECOME NUMERICALLY UNSTABLE FOR LARGE p, leaving the infinity-norm limit as the usable case Quotation withheld because this record is not cleared for redistribution. In convolutional nets the second moment estimate vanishes and poorly approximates the cost surface. what this limits: convolutional networks specifically: THE SECOND MOMENT ESTIMATE IS THEREFORE A POOR APPROXIMATION TO THE GEOMETRY OF THE COST FUNCTION IN CNNs Quotation withheld because this record is not cleared for redistribution. On the CNN experiment the gain over SGD with momentum is only marginal. what this limits: the size of the advantage on the CNN experiment: THOUGH ADAM SHOWS MARGINAL IMPROVEMENT OVER SGD WITH MOMENTUM, IT ADAPTS LEARNING RATE SCALE FOR DIFFERENT LAYERS Quotation withheld because this record is not cleared for redistribution. The convergence guarantee is proved only for convex objectives, and the paper says so in the same sentence in which it reports the non-convex results. Quotation withheld because this record is not cleared for redistribution. the optimiser comparison run both with and without dropout noise eliminates: an advantage that holds only in the absence of the regularisation practitioners actually use · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. AdaGrad AdaGrad Quotation withheld because this record is not cleared for redistribution. RMSProp RMSProp Quotation withheld because this record is not cleared for redistribution. Sum-of-Functions Optimizer (SFO) Sum-of-Functions Optimizer (SFO) Quotation withheld because this record is not cleared for redistribution. signal-to-noise ratio signal-to-noise ratio Quotation withheld because this record is not cleared for redistribution. trust region trust region Quotation withheld because this record is not cleared for redistribution. regret regret Quotation withheld because this record is not cleared for redistribution. second raw moment second raw moment Quotation withheld because this record is not cleared for redistribution. diagonal of the Fisher information matrix diagonal of the Fisher information matrix Quotation withheld because this record is not cleared for redistribution. adaptive moment estimation adaptive moment estimation Quotation withheld because this record is not cleared for redistribution. effective stepsize effective stepsize Quotation withheld because this record is not cleared for redistribution. initialization bias correction initialization bias correction Quotation withheld because this record is not cleared for redistribution. signal-to-noise ratio - here the ratio of the two bias-corrected moment estimates, used as confidence in the gradient DIRECTION, not a power ratio Quotation withheld because this record is not cleared for redistribution. regret - the summed gap between the online prediction and the best fixed parameter in hindsight, over the whole sequence Quotation withheld because this record is not cleared for redistribution. second raw moment - the UNCENTERED variance of the gradient, not the variance Quotation withheld because this record is not cleared for redistribution. trust region - the region within which the current gradient estimate is taken to carry enough information, bounded here by the stepsize hyper-parameter Quotation withheld because this record is not cleared for redistribution. builds-on → AdaGrad (Duchi et al., 2011) and RMSProp (Tieleman & Hinton, 2012) Quotation withheld because this record is not cleared for redistribution. extends → RMSProp (Tieleman & Hinton, 2012) Quotation withheld because this record is not cleared for redistribution. builds-on → the online learning framework of (Zinkevich, 2003) Quotation withheld because this record is not cleared for redistribution. AdaGrad —is-a→ Adam AdaGrad Adam strength of the relation: 0.7 · note: A LIMITING-CASE MEMBERSHIP, typed 'is-a' not 'same-as' because the span says 'corresponds to a version of', not that the two are the same thing. The paper scopes it itself one sentence later: the correspondence 'does not hold when removing…' the bias-correction terms. Schema 0.3 has nowhere to put that scope - holds for in an internal file. Filing it in differs in would mis-state a scope as an axis. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but AdaGrad 'corresponds to a version of Adam' under limiting settings; is-a is loose. Quotation withheld because this record is not cleared for redistribution. effective stepsize —is-a→ trust region effective stepsize trust region strength of the relation: 0.6 — the record marks this weak · note: Typed 'is-a', not 'renamed-across-fields': 'trust region' is optimisation vocabulary in an optimisation paper, so nothing crosses a field boundary. The paper hedges - 'can be understood as establishing a trust region' - hence 0.7. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the stepsize 'can be understood as establishing a trust region'; is-a is loose. Quotation withheld because this record is not cleared for redistribution. Adam —contrasted-with→ RMSProp Adam RMSProp differs in: whether a bias-correction term is present, and the paper names the regime in which it decides the outcome: beta2 close to 1, which is what sparse gradients require · strength of the relation: 0.8 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'RMSProp also lacks' continues the list of differences between RMSProp and Adam (previous sentence). Quotation withheld because this record is not cleared for redistribution. Adam —contrasted-with→ Sum-of-Functions Optimizer (SFO) Adam Sum-of-Functions Optimizer (SFO) differs in: memory scaling: SFO's requirement is linear in the number of minibatch partitions · strength of the relation: 0.8 Quotation withheld because this record is not cleared for redistribution. Adam —same-as→ adaptive moment estimation Adam adaptive moment estimation strength of the relation: 0.5 — the record marks this weak · note: An acronym and its expansion, so 'same-as'. NOT derived from name: in this corpus that field marks compound names whose head noun is the parent concept ('radical reductionism is-a reductionism', 11 of 18 uses). Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. regression regression about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1412.6980.