Adam: A Method for Stochastic Optimization

Diederik P. Kingma, Jimmy Ba · ICLR / arXiv · 2014 · artificial-intelligence · record kingma2014-adam
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. method is not on the suggested list: the paper combines a theoretical regret analysis with benchmark optimisation experiments on MNIST, IMDB and CIFAR-10, so 'computational-experiment' was chosen for the empirical core; the convergence proof (Theorem 4.1, Corollary 4.2) is a second, analytical strand not captured by a single method tag. The AdaMax variant and temporal averaging extension are additional method contributions inside the same paper and are not separately faceted. 2026-08-15 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

The quality of the text these quotes were cut from. residual-ligature-artifacts: some fi/fl were lost in an earlier cleanup and the source could not be re-fetched cleanly (personal copy / paywalled / corrupt text layer). Text is complete and readable but not pristine; not enriched until re-sourced.

Some words in these quotes contain the wrong character. The PDF this text was extracted from substituted 8 characters inside a word — a currency sign or a stray accent where a letter pair belongs, as in signi¢cantly for significantly, or a single ligature glyph, as in conﬁdence. They are valid characters in the wrong place, so they survive every encoding check. The quotes keep them for the same reason as above, and a search of this page will not find the affected words.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
signal-to-noise ratio 4 fields · logistic regression 3 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 15 concept tags on this paper, 11 are grounded in a quoted sentence.

Adam stochastic optimization adaptive learning rate gradient descent AdaGrad RMSProp Sum-of-Functions Optimizer (SFO) signal-to-noise ratio trust region regret second raw moment diagonal of the Fisher information matrix adaptive moment estimation effective stepsize initialization bias correction initialisation bias from zero-initialised moving averages automatic annealing via the signal-to-noise ratio invariance of the update to gradient rescaling Bayesian prior distribution mean average of regression logistic regression variance system-or-tool-building

Method, as described: computational-experiment

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
control	`controls`	an experiment run to eliminate a rival explanation
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1412.6980. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 3 · claim_type 1 · concept_edges 5 · concepts 11 · content_tags 4 · controls 1 · limitations 5 · mechanisms 3 · method 1 · phenomenon 1 · senses 4

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 5 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	stochastic gradient-based optimization Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	system-or-tool-building Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: system-or-tool-building	`method`		the paper's words
mechanisms	The moment estimates start at zero, so early estimates are biased toward zero; the correction divides by one minus the decay rate raised to the timestep. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: initialisation bias from zero-initialised moving averages	`mechanisms[0]`	0.9	the tagger's words
mechanisms	As an optimum is approached the ratio falls, so the effective step shrinks with no schedule imposed. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: automatic annealing via the signal-to-noise ratio	`mechanisms[1]`	0.85	the tagger's words
mechanisms	Rescaling the gradient by c scales the first moment by c and the second by c squared, and the two cancel in the ratio. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: invariance of the update to gradient rescaling	`mechanisms[2]`	0.9	the tagger's words
limitations	The convergence analysis covers only convex objectives, not the non-convex problems on which Adam is mostly used. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the convergence guarantee, which does not cover the setting the method is used in: Quotation withheld	`limitations[0]`		the tagger's words
limitations	The Lp-norm generalisation of the update rule is numerically unstable except in the infinity-norm limit. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the Lp-norm generalisation of the update rule: SUCH VARIANTS BECOME NUMERICALLY UNSTABLE FOR LARGE p, leaving the infinity-norm limit as the usable case	`limitations[1]`		the tagger's words
limitations	In convolutional nets the second moment estimate vanishes and poorly approximates the cost surface. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: convolutional networks specifically: THE SECOND MOMENT ESTIMATE IS THEREFORE A POOR APPROXIMATION TO THE GEOMETRY OF THE COST FUNCTION IN CNNs	`limitations[2]`		the tagger's words
limitations	On the CNN experiment the gain over SGD with momentum is only marginal. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the size of the advantage on the CNN experiment: THOUGH ADAM SHOWS MARGINAL IMPROVEMENT OVER SGD WITH MOMENTUM, IT ADAPTS LEARNING RATE SCALE FOR DIFFERENT LAYERS	`limitations[3]`		the tagger's words
limitations	The convergence guarantee is proved only for convex objectives, and the paper says so in the same sentence in which it reports the non-convex results. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`limitations[4]`	0.9	the tagger's words
controls	the optimiser comparison run both with and without dropout noise Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. eliminates: an advantage that holds only in the absence of the regularisation practitioners actually use · outcome (stated by the tagger, no quotation): held	`controls[0]`	0.85	the tagger's words
concepts	AdaGrad Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: AdaGrad	`concepts[4]`	0.9	the paper's words
concepts	RMSProp Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: RMSProp	`concepts[5]`	0.9	the paper's words
concepts	Sum-of-Functions Optimizer (SFO) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Sum-of-Functions Optimizer (SFO)	`concepts[6]`	0.9	the paper's words
concepts	signal-to-noise ratio Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: signal-to-noise ratio	`concepts[7]`	0.9	the paper's words
concepts	trust region Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: trust region	`concepts[8]`	0.9	the paper's words
concepts	regret Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: regret	`concepts[9]`	0.9	the paper's words
concepts	second raw moment Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: second raw moment	`concepts[10]`	0.9	the paper's words
concepts	diagonal of the Fisher information matrix Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: diagonal of the Fisher information matrix	`concepts[11]`	0.9	the paper's words
concepts	adaptive moment estimation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: adaptive moment estimation	`concepts[12]`	0.9	the paper's words
concepts	effective stepsize Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: effective stepsize	`concepts[13]`	0.9	the paper's words
concepts	initialization bias correction Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: initialization bias correction	`concepts[14]`	0.9	the paper's words
senses	signal-to-noise ratio - here the ratio of the two bias-corrected moment estimates, used as confidence in the gradient DIRECTION, not a power ratio Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	0.9	the tagger's words
senses	regret - the summed gap between the online prediction and the best fixed parameter in hindsight, over the whole sequence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	0.9	the tagger's words
senses	second raw moment - the UNCENTERED variance of the gradient, not the variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	0.95	the tagger's words
senses	trust region - the region within which the current gradient estimate is taken to carry enough information, bounded here by the stepsize hyper-parameter Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	0.8	the tagger's words
claim_relations	builds-on → AdaGrad (Duchi et al., 2011) and RMSProp (Tieleman & Hinton, 2012) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	extends → RMSProp (Tieleman & Hinton, 2012) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → the online learning framework of (Zinkevich, 2003) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
concept_edges	AdaGrad —is-a→ Adam Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.7 · note: A LIMITING-CASE MEMBERSHIP, typed 'is-a' not 'same-as' because the span says 'corresponds to a version of', not that the two are the same thing. The paper scopes it itself one sentence later: the correspondence 'does not hold when removing…' the bias-correction terms. Schema 0.3 has nowhere to put that scope - holds for in an internal file. Filing it in differs in would mis-state a scope as an axis. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but AdaGrad 'corresponds to a version of Adam' under limiting settings; is-a is loose. also in: AdaGrad · Adam	`concept_edges[1]`	0.85	the tagger's words
concept_edges	effective stepsize —is-a→ trust region Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.6 — the record marks this weak · note: Typed 'is-a', not 'renamed-across-fields': 'trust region' is optimisation vocabulary in an optimisation paper, so nothing crosses a field boundary. The paper hedges - 'can be understood as establishing a trust region' - hence 0.7. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the stepsize 'can be understood as establishing a trust region'; is-a is loose. also in: effective stepsize · trust region	`concept_edges[3]`	0.7	the tagger's words
concept_edges	Adam —contrasted-with→ RMSProp Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. differs in: whether a bias-correction term is present, and the paper names the regime in which it decides the outcome: beta2 close to 1, which is what sparse gradients require · strength of the relation: 0.8 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'RMSProp also lacks' continues the list of differences between RMSProp and Adam (previous sentence). also in: Adam · RMSProp	`concept_edges[4]`	0.9	the tagger's words
concept_edges	Adam —contrasted-with→ Sum-of-Functions Optimizer (SFO) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. differs in: memory scaling: SFO's requirement is linear in the number of minibatch partitions · strength of the relation: 0.8 also in: Adam · Sum-of-Functions Optimizer (SFO)	`concept_edges[5]`	0.9	the tagger's words
concept_edges	Adam —same-as→ adaptive moment estimation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 0.5 — the record marks this weak · note: An acronym and its expansion, so 'same-as'. NOT derived from name: in this corpus that field marks compound names whose head noun is the parent concept ('radical reductionism is-a reductionism', 11 of 18 uses). also in: Adam · adaptive moment estimation	`concept_edges[6]`	0.95	the tagger's words
content_tags	Bayesian Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Bayesian	`content_tags.statistics[0]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean	`content_tags.statistics[1]`		the tagger's words
content_tags	regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: regression	`content_tags.statistics[2]`		the tagger's words
content_tags	variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variance	`content_tags.statistics[3]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.