Stochastic Variational Inference

Matthew D. Hoffman, David M. Blei, Chong Wang, John Paisley · Journal of Machine Learning Research / arXiv · 2013 · statistics · record hoffman2013-svi
tag layer · index · concepts · authors · עברית
Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Method is tagged system-or-tool-building for the algorithm derivation; the paper also contains a substantial empirical study on Nature, New York Times and Wikipedia corpora that yields a secondary comparative finding between models ("The HDP gives consistently better performance."), which the single claim type facet cannot hold. Discipline is machine-learning; the venue is JMLR and the framing is statistical. 2026-08-14, 2026-08-15, 2026-08-17 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.
How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.
Other fields define these differently the same word, defined another way in another discipline
sample-size 5 fields · linear regression 2 fields · posterior distribution 2 fields · scalability 2 fields · topic 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 10 concept tags on this paper, 5 are grounded in a quoted sentence.
variational inference Bayesian inference stochastic optimization topic model scalability topic topic proportions evidence lower bound vocabulary natural gradient THREE SOURCES FOR A MODEL'S ASSUMPTIONS ARE NAMED, AND WHAT THEY ASSERT IS DECLARED UNOBSERVABLE IN THE SAME CLAUSE: we have assumptions to make - FROM SCIENCE, INTUITION, OR OTHER DATA ANALYSES - that involve STRUCTURES WE BELIEVE EXIST IN THE DATA BUT THAT WE CANNOT DIRECTLY OBSERVE. The provenance of the assumptions and the inaccessibility of their objects are stated together A FORMALISM IS DESCRIBED AS A LANGUAGE, AND THE INFERENCE PROCEDURE AS ITS READING: Quotation withheld ASSUMPTIONS about data and its HIDDEN STRUCTURE, and the corresponding algorithms LET US ANALYZE DATA UNDER THOSE ASSUMPTIONS, INFERRING THE HIDDEN STRUCTURE THAT BEST EXPLAINS OUR OBSERVATIONS. The notation is cast as a vocabulary and the computation as its interpretation THE PAPER'S OWN TASKS ARE SPLIT INTO TWO KINDS WITH DIFFERENT WARRANTS, AND THE UNOBSERVED PART IS PUT IN SCARE QUOTES: in DESCRIPTIVE tasks the models help EXPLORE the data with the hidden structure probabilistically FILLED IN; in PREDICTIVE tasks they FORM PREDICTIONS ABOUT NEW OBSERVATIONS. The same machinery serves two purposes that would be judged by different evidence A MODEL CLASS IS DEFINED BY WHAT IT COSTS RATHER THAN BY WHAT IT CONTAINS: NONCONJUGATE MODELS are MODELS WHERE A RICHER PRIOR IS USED AT THE EXPENSE OF MATHEMATICAL CONVENIENCE - and they HAVE EXPANDED THE SUITE of tools available. The category's definition is a trade-off, with the thing traded away named explicitly ONE ALGORITHM IS RE-DESCRIBED AS ANOTHER, AND THE EQUIVALENCE IS STATED AS AN IDENTITY OF STEPS: the classical coordinate ascent algorithm CAN THUS BE INTERPRETED AS a projected natural gradient algorithm - and updating a parameter by taking a natural gradient step OF LENGTH ONE IS EQUIVALENT TO PERFORMING A COORDINATE UPDATE. Two names for one procedure, with the exchange rate given A FORMAL FACT IS GIVEN AN INTUITIVE ROUTE ALONGSIDE THE FORMAL ONE, AND BOTH ARE LABELLED: FORMALLY this is most easily seen via other perspectives, THOUGH IT CAN BE SEEN INTUITIVELY WITH the stick-breaking construction - and then THE INTUITION IS THAT more of the stick is absorbed in the first break locations. The paper marks which of its two explanations is which THE EVALUATION CRITERION IS OPERATIONALISED AS A PROBABILITY ASSIGNED TO WITHHELD WORDS: under the predictive distribution, A BETTER MODEL WILL ASSIGN HIGHER PROBABILITY TO THE HELD-OUT WORDS - and the procedure divides each document into observed and held-out words, KEEPING THE SETS OF UNIQUE WORDS DISJOINT. What BETTER means is fixed by a number and a partition, and the partition rule is stated A PROCEDURE IS NAMED BY COMPOSING TWO EXISTING NAMES: interleaving the coordinate ascent updates with an update that increases the bound IS CALLED VARIATIONAL EXPECTATION-MAXIMIZATION - and the stochastic version is then described by what it does DIFFERENTLY, updating the two simultaneously. The name records its parentage and the variant is defined against it stochastic variational inference posterior probabilistic models topic models latent Dirichlet allocation Bayesian massive data scalable probabilistic modeling probabilistic graphical models posterior distribution mean average of regression linear regression sample-size variance system-or-tool-building

Legend

relation to other workclaim_relationshow this paper stands to another: agrees, contradicts, extends, qualifies
claim typeclaim_typethe shape of the claim: existence, mechanism, method, critique, prediction, null
concept edgeconcept_edgesan asserted relation between two concepts, with the sentence asserting it
conceptconceptsan idea the paper works with, quoted where the paper uses it
content tagcontent_tagsa finding, definition, quantity, statistic or variable named in the paper
controlcontrolsan experiment run to eliminate a rival explanation
limitationlimitationswhat the paper says its evidence cannot support
mechanismmechanismshow the reported effect is said to come about
methodmethodhow the paper produced its evidence
phenomenonphenomenonthe effect or regularity under study
sensesensesa distinct use of a term, quoted where the paper fixes it
transfertransfersa method or concept carried in from another field
unrecordable relationunhoused_relationsa relation the paper asserts that this vocabulary cannot express, with the reason

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/1206.7051. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 5 · claim_type 1 · concept_edges 1 · concepts 5 · content_tags 18 · controls 1 · limitations 7 · mechanisms 8 · method 1 · phenomenon 1 · senses 3 · transfers 2 · unhoused_relations 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

facettagfieldconf.whose words
claim_typemethod
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_typethe paper's words
phenomenonscalable approximate posterior inference in hierarchical Bayesian models
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
phenomenonthe paper's words
methodsystem-or-tool-building
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
methodthe paper's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[0]0.95the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[1]0.95the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[2]0.95the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[3]0.95the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[4]0.95the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
about this quote: RE-SLICED off an UNCLOSED OPENING BRACKET. The first cut ended "...drawn from Beta(1", truncated inside a distribution's argument list. An internal tool did NOT catch it: that tool looks for a CLOSER with nothing to close, and this is the mirror case - an OPENER with nothing closing it. A real gap in the guard, logged as its own question.
mechanisms[5]0.85the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
mechanisms[6]0.95the tagger's words
mechanismsmechanisms
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
about this quote: RE-SLICED: the first cut ended "we update alpha simultaneously with", dangling in front of a symbol the span does not include.
mechanisms[7]0.85the tagger's words
limitationsThe general algorithm is derived only for conjugate exponential-family models and does not cover nonconjugate models such as correlated or dynamic topic models.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: the reach of the derivation: WE DEVELOPED OUR ALGORITHM WITH CONJUGATE EXPONENTIAL FAMILY MODELS, so nonconjugate models are outside it
limitations[0]the tagger's words
limitationsConvergence is only to a local optimum, and very small minibatches can lead to a poor one.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: any solution the algorithm returns: IT IS GUARANTEED TO CONVERGE TO A LOCAL OPTIMUM BUT TAKING LARGE STEPS ON THE BASIS OF VERY FEW DATA POINTS MAY LEAD TO A POOR ONE
limitations[1]the tagger's words
limitationsThe quality of the solution and speed of convergence depend on how the forgetting rate, delay and minibatch size are set.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: the reported convergence behaviour: THE QUALITY OF THIS STATIONARY POINT AND THE SPEED OF CONVERGENCE MAY DEPEND ON HOW THESE PARAMETERS ARE SET
limitations[2]the tagger's words
limitationsWhat is optimised is a bound standing in for a quantity the method cannot compute.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: any model comparison made on this objective: the marginal probability cannot be computed exactly, so what is maximised is a fitted variational lower bound on it
limitations[3]1.0the tagger's words
limitationsA prior is made exchangeable for simplicity, and the paper notes the conditional then is not.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: results depending on the Dirichlet prior here: the parameters are all set to the same value for simplicity, and the paper notes that conditioning makes the resulting conditional non-exchangeable
limitations[4]1.0the tagger's words
limitationsThe evaluation is built from a split of each test document's words, constructed to be disjoint.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: the held-out probability figures reported here: each test document's words are divided into observed and held-out sets, with the sets of unique words kept disjoint
limitations[5]1.0the tagger's words
limitationsHow many topics a collection needs is answered by the model's own posterior.
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
what this limits: the number of topics reported for a collection: it is determined by the posterior distribution of the hidden structure under this model, not by anything independent of it
limitations[6]1.0the tagger's words
transfersstochastic optimization following noisy gradient estimates, applied to the variational objective so inference need not pass over the whole data set each iteration
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
transfers[0]0.9the tagger's words
transfersthe natural gradient, which rescales the step by the Fisher information of the variational family rather than using the Euclidean gradient
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
transfers[1]0.85the tagger's words
controlsa test set of 10,000 documents set aside per collection and explicitly withheld from the algorithms during training
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
eliminates: a fitness figure earned on documents the model was fitted to, with the paper stating the withholding rather than leaving it implied · outcome (stated by the tagger, no quotation): held
controls[0]0.85the tagger's words
conceptstopic
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: topic
concepts[5]the paper's words
conceptstopic proportions
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
concepts[6]the paper's words
conceptsevidence lower bound
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
concepts[7]the paper's words
conceptsvocabulary
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: vocabulary
concepts[8]the paper's words
conceptsnatural gradient
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
concepts[9]the paper's words
unhoused_relationsoptimized distribution → posterior
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
THE INDEXING / PROXY RELATION - 'from' STANDS IN FOR 'to'. We have no type for it: 'causes' asserts production, and 'operationalised-as' says the authors CHOSE a measure, whereas a proxy claim says THIS STANDS IN FOR WHAT WE ACTUALLY CARE ABOUT AND WE KNOW IT IS NOT THE SAME THING. Measured at 8% of papers. The substitution at the centre of variational inference, stated in one line: 'we use the optimized distribution as a proxy for the posterior'.
unhoused_relations[0]the tagger's words
sensesa topic - A WEIGHTED DISTRIBUTION OVER THE VOCABULARY: not a subject, a theme or an aboutness, but a probability vector over words, shown to a reader by printing its most frequent members - which is why a multi-word name has no way to stay together inside one
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
senses[0]1.0the tagger's words
sensestopic proportions - A DISTRIBUTION OVER TOPICS, so a distribution over distributions: the same word does duty at two levels in one sentence, and a document's 'topics' are weights on objects that are themselves weights on words
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
senses[1]1.0the tagger's words
sensesthe ELBO - A LOWER BOUND SUBSTITUTED FOR THE THING WANTED: the algorithm maximises a bound on the log marginal probability rather than that probability, and the gap between them is not reported because computing it would require the quantity the bound exists to avoid
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
senses[2]1.0the tagger's words
claim_relationsextends → variational inference (Jordan et al.)
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_relations[0]the tagger's words
claim_relationsbuilds-on → latent Dirichlet allocation (LDA)
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_relations[1]the tagger's words
claim_relationsbuilds-on → Sato (2001), online/natural-gradient variational Bayes for EM-fittable models
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_relations[2]the tagger's words
claim_relationsbuilds-on → the Robbins and Monro (1951) stochastic approximation method
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_relations[3]the tagger's words
claim_relationsextends → the online LDA algorithm of Hoffman et al. (2010a)
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
claim_relations[4]the tagger's words
concept_edgestopic —operationalised-as→ vocabulary
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span says each topic is a distribution over the vocabulary; operationalised-as is loose.
also in: topic · vocabulary
concept_edges[0]the tagger's words
content_tags300K / 1.8M / 3.8M — Nature, NYT, Wikipedia corpora analyzed
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted.
content_tags.quantities[0]the tagger's words
content_tags2 million — scale of an example text archive
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.quantities[1]the tagger's words
content_tags2 — models used for demonstration (LDA, HDP)
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.quantities[2]the tagger's words
content_tagsSVI is a scalable algorithm for approximating posteriors
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[0]the tagger's words
content_tagsSVI handles large data and outperforms traditional VI
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[1]the tagger's words
content_tagsthe Bayesian nonparametric model beats its parametric counterpart
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[2]the tagger's words
content_tagsSVI enables complex Bayesian models on massive data sets
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.findings[3]the tagger's words
content_tagsposterior distribution
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.variables[0]the tagger's words
content_tagstopic model
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: topic model
content_tags.variables[2]the tagger's words
content_tagsprobabilistic graphical models
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.variables[3]the tagger's words
content_tagsstochastic variational inference
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.definitions[0]the tagger's words
content_tagsprobabilistic modeling
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.definitions[1]the tagger's words
content_tagsprobabilistic graphical models
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
content_tags.definitions[2]the tagger's words
content_tagsBayesian
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: Bayesian
content_tags.statistics[0]the tagger's words
content_tagsmean
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: mean
content_tags.statistics[2]the tagger's words
content_tagsregression
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted.
also in: regression
content_tags.statistics[3]the tagger's words
content_tagssample-size
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: sample-size
content_tags.statistics[4]the tagger's words
content_tagsvariance
Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.
also in: variance
content_tags.statistics[5]the tagger's words

Interpretive choices

Tags that could not be placed in the text:

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.

© 2026 Shir Sivroni · ORCID 0009-0000-2597-2824