phenomenon · claim-type · cross-domain-transfer · mechanism · claim-relation · replication-status · limitation — grounded in the full text| phenomenon | stochastic gradient-based optimization— “Stochastic gradient-based optimization is of core practical importance in many fields of science and engineering.” | |
| claim-type | method— “We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.” | |
| method — evidence | — “To empirically evaluate the proposed method, we investigated different popular machine learning models, including logistic regression, multilayer fully connected neural networks and deep convolutional neural networks.” | |
| mechanism | initialisation bias from zero-initialised moving averages The moment estimates start at zero, so early estimates are biased toward zero; the correction divides by one minus the decay rate raised to the timestep.— “these moving averages are initialized as (vectors of) 0’s, leading to moment estimates that are biased towards zero, especially during the initial timesteps” | mechanisms[0] |
| mechanism | automatic annealing via the signal-to-noise ratio As an optimum is approached the ratio falls, so the effective step shrinks with no schedule imposed.— “the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps in parameter space: a form of automatic annealing” | mechanisms[1] |
| mechanism | invariance of the update to gradient rescaling Rescaling the gradient by c scales the first moment by c and the second by c squared, and the two cancel in the ratio.— “The effective stepsize ∆t is also invariant to the scale of the gradients; rescaling the gradientsg with factorc will scaleˆmt with a factorc andˆvt with a factorc2, which cancel out” | mechanisms[2] |
| builds-on | AdaGrad (Duchi et al., 2011) and RMSProp (Tieleman & Hinton, 2012)— “Our method is designed to combine the advantages of two recently popular methods: AdaGrad (Duchi et al., 2011), which works well with sparse gradients, and RMSProp (Tieleman & Hinton, 2012), which works well in on-line and non-stationary settings; important connections to these and other stochastic optimization methods are clarified in… | claim_relations[0] |
| extends | RMSProp (Tieleman & Hinton, 2012)— “RMSProp also lacks a bias-correction term; this matters most in case of a value of β2 close to 1 (required in case of sparse gradients), since in that case not correcting the bias leads to very large stepsizes and often divergence, as we also empirically demonstrate in section 6.4.” | claim_relations[1] |
| builds-on | the online learning framework of (Zinkevich, 2003)— “We analyze the convergence of Adam using the online learning framework proposed in (Zinkevich, 2003).” | claim_relations[2] |
| sense | signal-to-noise ratio - here the ratio of the two bias-corrected moment estimates, used as confidence in the gradient DIRECTION, not a power ratio— “With a slight abuse of terminology, we will call the ratio ˆmt/√ˆvt the signal-to-noise ratio (SNR )” | senses[0] |
| sense | regret - the summed gap between the online prediction and the best fixed parameter in hindsight, over the whole sequence— “the regret, that is the sum of all the previous difference between the online prediction ft(θt) and the best fixed point parameterft(θ∗) from a feasible setX for all the previous steps” | senses[1] |
| sense | second raw moment - the UNCENTERED variance of the gradient, not the variance— “The moving averages themselves are estimates of the 1 st moment (the mean) and the 2nd raw moment (the uncentered variance) of the gradient” | senses[2] |
| sense | trust region - the region within which the current gradient estimate is taken to carry enough information, bounded here by the stepsize hyper-parameter— “This can be understood as establishing a trust region around the current parameter value, beyond which the current gradient estimate does not provide sufficient information” | senses[3] |
| control | the optimiser comparison run both with and without dropout noise — held — rules out an advantage that holds only in the absence of the regularisation practitioners actually use— “In figure 1, Adagrad outperforms SGD with Nesterov momentum by a large margin both with and without dropout noise.” | controls[0] |
| audit | AdaGrad —is-a→ Adam — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but AdaGrad 'corresponds to a version of Adam' under limiting settings; is-a is loose.— “AdaGrad corresponds to a version of Adam withβ1 = 0, infinitesimal (1−β2) and a replacement ofα by an annealed version” | concept_edges[1] |
| note | AdaGrad —is-a→ Adam — A LIMITING-CASE MEMBERSHIP, typed 'is-a' not 'same-as' because the span says 'corresponds to a version of', not that the two are the same thing. The paper scopes it itself one sentence later: the correspondence 'does not hold when removing…' the bias-correction terms. Schema 0.3 has nowhere to put that scope - holds for in an internal file. Filing it in differs in would mis-state a scope as an axis.— “AdaGrad corresponds to a version of Adam withβ1 = 0, infinitesimal (1−β2) and a replacement ofα by an annealed version” | concept_edges[1] |
| weight | AdaGrad —is-a→ Adam — 0.7— “AdaGrad corresponds to a version of Adam withβ1 = 0, infinitesimal (1−β2) and a replacement ofα by an annealed version” | concept_edges[1] |
| audit | effective stepsize —is-a→ trust region — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the stepsize 'can be understood as establishing a trust region'; is-a is loose. | concept_edges[3] |
| note | effective stepsize —is-a→ trust region — Typed 'is-a', not 'renamed-across-fields': 'trust region' is optimisation vocabulary in an optimisation paper, so nothing crosses a field boundary. The paper hedges - 'can be understood as establishing a trust region' - hence 0.7. | concept_edges[3] |
| weight | effective stepsize —is-a→ trust region — 0.6 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[3] |
| audit | Adam —contrasted-with→ RMSProp — Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'RMSProp also lacks' continues the list of differences between RMSProp and Adam (previous sentence). Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[4] |
| differs in | Adam —contrasted-with→ RMSProp — whether a bias-correction term is present, and the paper names the regime in which it decides the outcome: beta2 close to 1, which is what sparse gradients require Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[4] |
| weight | Adam —contrasted-with→ RMSProp — 0.8 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[4] |
| differs in | Adam —contrasted-with→ Sum-of-Functions Optimizer (SFO) — memory scaling: SFO's requirement is linear in the number of minibatch partitions Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[5] |
| weight | Adam —contrasted-with→ Sum-of-Functions Optimizer (SFO) — 0.8 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[5] |
| note | Adam —same-as→ adaptive moment estimation — An acronym and its expansion, so 'same-as'. NOT derived from name: in this corpus that field marks compound names whose head noun is the parent concept ('radical reductionism is-a reductionism', 11 of 18 uses). Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[6] |
| weight | Adam —same-as→ adaptive moment estimation — 0.5 Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concept_edges[6] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — the convergence guarantee, which does not cover the setting the method is used in: ALTHOUGH OUR CONVERGENCE ANALYSIS DOES NOT APPLY TO NON-CONVEX PROBLEMS, WE EMPIRICALLY FOUND THAT ADAM OFTEN OUTPERFORMS OTHER METHODS IN SUCH CASES Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| for which use | limitation — the Lp-norm generalisation of the update rule: SUCH VARIANTS BECOME NUMERICALLY UNSTABLE FOR LARGE p, leaving the infinity-norm limit as the usable case Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| for which use | limitation — convolutional networks specifically: THE SECOND MOMENT ESTIMATE IS THEREFORE A POOR APPROXIMATION TO THE GEOMETRY OF THE COST FUNCTION IN CNNs Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| for which use | limitation — the size of the advantage on the CNN experiment: THOUGH ADAM SHOWS MARGINAL IMPROVEMENT OVER SGD WITH MOMENTUM, IT ADAPTS LEARNING RATE SCALE FOR DIFFERENT LAYERS Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| replication | untested | |
| limitation | The convergence analysis covers only convex objectives, not the non-convex problems on which Adam is mostly used. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[0] |
| limitation | The Lp-norm generalisation of the update rule is numerically unstable except in the infinity-norm limit. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[1] |
| limitation | In convolutional nets the second moment estimate vanishes and poorly approximates the cost surface. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[2] |
| limitation | On the CNN experiment the gain over SGD with momentum is only marginal. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[3] |
| limitation | The convergence guarantee is proved only for convex objectives, and the paper says so in the same sentence in which it reports the non-convex results. Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | limitations[4] |
| concept | AdaGrad Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[4] |
| concept | RMSProp Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[5] |
| concept | Sum-of-Functions Optimizer (SFO) Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[6] |
| concept | signal-to-noise ratio Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[7] |
| concept | trust region Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[8] |
| concept | regret Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[9] |
| concept | second raw moment Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[10] |
| concept | diagonal of the Fisher information matrix Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[11] |
| concept | adaptive moment estimation Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[12] |
| concept | effective stepsize Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[13] |
| concept | initialization bias correction Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | concepts[14] |
| Bayesian | as written: prior distribution Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[0] |
| mean μ | as written: average of Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[1] |
| regression β | as written: logistic regression Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[2] |
| evidence status note | regression — QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. | content_tags.statistics[2] |
| variance σ² | Quotation withheld — this paper is not ours to reproduce, and the page has already quoted as much of it as the rights layer allows. The tag, its field and its confidence stand; the sentence is at the publisher. | content_tags.statistics[3] |
When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Tag layer © the Meta-Tagging Project, CC BY 4.0. The paper text is © its authors/publisher
and is linked, not reproduced. Excerpts shown are short quotations for identification and verification.
· back to index
· concepts · authors
· עברית
© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824