Stochastic Variational Inference

Stochastic Variational Inference Matthew D. Hoffman David M. Blei Chong Wang John Paisley

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Stochastic Variational Inference 2013 Journal of Machine Learning Research / arXiv 1206.7051 hoffman2013-svi hoffman2013-svi.txt, the project's staged copy 897dee8eaabeb6c2be3a635a9a4e6941e89a27b6 arxiv-open statistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. scalable approximate posterior inference in hierarchical Bayesian models Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms THREE SOURCES FOR A MODEL'S ASSUMPTIONS ARE NAMED, AND WHAT THEY ASSERT IS DECLARED UNOBSERVABLE IN THE SAME CLAUSE: we have assumptions to make - FROM SCIENCE, INTUITION, OR OTHER DATA ANALYSES - that involve STRUCTURES WE BELIEVE EXIST IN THE DATA BUT THAT WE CANNOT DIRECTLY OBSERVE. The provenance of the assumptions and the inaccessibility of their objects are stated together Quotation withheld because this record is not cleared for redistribution. mechanisms A FORMALISM IS DESCRIBED AS A LANGUAGE, AND THE INFERENCE PROCEDURE AS ITS READING: Quotation withheld ASSUMPTIONS about data and its HIDDEN STRUCTURE, and the corresponding algorithms LET US ANALYZE DATA UNDER THOSE ASSUMPTIONS, INFERRING THE HIDDEN STRUCTURE THAT BEST EXPLAINS OUR OBSERVATIONS. The notation is cast as a vocabulary and the computation as its interpretation Quotation withheld because this record is not cleared for redistribution. mechanisms THE PAPER'S OWN TASKS ARE SPLIT INTO TWO KINDS WITH DIFFERENT WARRANTS, AND THE UNOBSERVED PART IS PUT IN SCARE QUOTES: in DESCRIPTIVE tasks the models help EXPLORE the data with the hidden structure probabilistically FILLED IN; in PREDICTIVE tasks they FORM PREDICTIONS ABOUT NEW OBSERVATIONS. The same machinery serves two purposes that would be judged by different evidence Quotation withheld because this record is not cleared for redistribution. mechanisms A MODEL CLASS IS DEFINED BY WHAT IT COSTS RATHER THAN BY WHAT IT CONTAINS: NONCONJUGATE MODELS are MODELS WHERE A RICHER PRIOR IS USED AT THE EXPENSE OF MATHEMATICAL CONVENIENCE - and they HAVE EXPANDED THE SUITE of tools available. The category's definition is a trade-off, with the thing traded away named explicitly Quotation withheld because this record is not cleared for redistribution. mechanisms ONE ALGORITHM IS RE-DESCRIBED AS ANOTHER, AND THE EQUIVALENCE IS STATED AS AN IDENTITY OF STEPS: the classical coordinate ascent algorithm CAN THUS BE INTERPRETED AS a projected natural gradient algorithm - and updating a parameter by taking a natural gradient step OF LENGTH ONE IS EQUIVALENT TO PERFORMING A COORDINATE UPDATE. Two names for one procedure, with the exchange rate given Quotation withheld because this record is not cleared for redistribution. mechanisms A FORMAL FACT IS GIVEN AN INTUITIVE ROUTE ALONGSIDE THE FORMAL ONE, AND BOTH ARE LABELLED: FORMALLY this is most easily seen via other perspectives, THOUGH IT CAN BE SEEN INTUITIVELY WITH the stick-breaking construction - and then THE INTUITION IS THAT more of the stick is absorbed in the first break locations. The paper marks which of its two explanations is which about this quote: RE-SLICED off an UNCLOSED OPENING BRACKET. The first cut ended "...drawn from Beta(1", truncated inside a distribution's argument list. An internal tool did NOT catch it: that tool looks for a CLOSER with nothing to close, and this is the mirror case - an OPENER with nothing closing it. A real gap in the guard, logged as its own question. Quotation withheld because this record is not cleared for redistribution. mechanisms THE EVALUATION CRITERION IS OPERATIONALISED AS A PROBABILITY ASSIGNED TO WITHHELD WORDS: under the predictive distribution, A BETTER MODEL WILL ASSIGN HIGHER PROBABILITY TO THE HELD-OUT WORDS - and the procedure divides each document into observed and held-out words, KEEPING THE SETS OF UNIQUE WORDS DISJOINT. What BETTER means is fixed by a number and a partition, and the partition rule is stated Quotation withheld because this record is not cleared for redistribution. mechanisms A PROCEDURE IS NAMED BY COMPOSING TWO EXISTING NAMES: interleaving the coordinate ascent updates with an update that increases the bound IS CALLED VARIATIONAL EXPECTATION-MAXIMIZATION - and the stochastic version is then described by what it does DIFFERENTLY, updating the two simultaneously. The name records its parentage and the variant is defined against it about this quote: RE-SLICED: the first cut ended "we update alpha simultaneously with", dangling in front of a symbol the span does not include. Quotation withheld because this record is not cleared for redistribution. The general algorithm is derived only for conjugate exponential-family models and does not cover nonconjugate models such as correlated or dynamic topic models. what this limits: the reach of the derivation: WE DEVELOPED OUR ALGORITHM WITH CONJUGATE EXPONENTIAL FAMILY MODELS, so nonconjugate models are outside it Quotation withheld because this record is not cleared for redistribution. Convergence is only to a local optimum, and very small minibatches can lead to a poor one. what this limits: any solution the algorithm returns: IT IS GUARANTEED TO CONVERGE TO A LOCAL OPTIMUM BUT TAKING LARGE STEPS ON THE BASIS OF VERY FEW DATA POINTS MAY LEAD TO A POOR ONE Quotation withheld because this record is not cleared for redistribution. The quality of the solution and speed of convergence depend on how the forgetting rate, delay and minibatch size are set. what this limits: the reported convergence behaviour: THE QUALITY OF THIS STATIONARY POINT AND THE SPEED OF CONVERGENCE MAY DEPEND ON HOW THESE PARAMETERS ARE SET Quotation withheld because this record is not cleared for redistribution. What is optimised is a bound standing in for a quantity the method cannot compute. what this limits: any model comparison made on this objective: the marginal probability cannot be computed exactly, so what is maximised is a fitted variational lower bound on it Quotation withheld because this record is not cleared for redistribution. A prior is made exchangeable for simplicity, and the paper notes the conditional then is not. what this limits: results depending on the Dirichlet prior here: the parameters are all set to the same value for simplicity, and the paper notes that conditioning makes the resulting conditional non-exchangeable Quotation withheld because this record is not cleared for redistribution. The evaluation is built from a split of each test document's words, constructed to be disjoint. what this limits: the held-out probability figures reported here: each test document's words are divided into observed and held-out sets, with the sets of unique words kept disjoint Quotation withheld because this record is not cleared for redistribution. How many topics a collection needs is answered by the model's own posterior. what this limits: the number of topics reported for a collection: it is determined by the posterior distribution of the hidden structure under this model, not by anything independent of it Quotation withheld because this record is not cleared for redistribution. stochastic optimization following noisy gradient estimates, applied to the variational objective so inference need not pass over the whole data set each iteration Quotation withheld because this record is not cleared for redistribution. the natural gradient, which rescales the step by the Fisher information of the variational family rather than using the Euclidean gradient Quotation withheld because this record is not cleared for redistribution. a test set of 10,000 documents set aside per collection and explicitly withheld from the algorithms during training eliminates: a fitness figure earned on documents the model was fitted to, with the paper stating the withholding rather than leaving it implied · outcome (stated by the tagger, no quotation): held Quotation withheld because this record is not cleared for redistribution. topic topic Quotation withheld because this record is not cleared for redistribution. topic proportions topic proportions Quotation withheld because this record is not cleared for redistribution. evidence lower bound evidence lower bound Quotation withheld because this record is not cleared for redistribution. vocabulary vocabulary Quotation withheld because this record is not cleared for redistribution. natural gradient natural gradient Quotation withheld because this record is not cleared for redistribution. optimized distribution → posterior THE INDEXING / PROXY RELATION - 'from' STANDS IN FOR 'to'. We have no type for it: 'causes' asserts production, and 'operationalised-as' says the authors CHOSE a measure, whereas a proxy claim says THIS STANDS IN FOR WHAT WE ACTUALLY CARE ABOUT AND WE KNOW IT IS NOT THE SAME THING. Measured at 8% of papers. The substitution at the centre of variational inference, stated in one line: 'we use the optimized distribution as a proxy for the posterior'. Quotation withheld because this record is not cleared for redistribution. a topic - A WEIGHTED DISTRIBUTION OVER THE VOCABULARY: not a subject, a theme or an aboutness, but a probability vector over words, shown to a reader by printing its most frequent members - which is why a multi-word name has no way to stay together inside one Quotation withheld because this record is not cleared for redistribution. topic proportions - A DISTRIBUTION OVER TOPICS, so a distribution over distributions: the same word does duty at two levels in one sentence, and a document's 'topics' are weights on objects that are themselves weights on words Quotation withheld because this record is not cleared for redistribution. the ELBO - A LOWER BOUND SUBSTITUTED FOR THE THING WANTED: the algorithm maximises a bound on the log marginal probability rather than that probability, and the gap between them is not reported because computing it would require the quantity the bound exists to avoid Quotation withheld because this record is not cleared for redistribution. extends → variational inference (Jordan et al.) Quotation withheld because this record is not cleared for redistribution. builds-on → latent Dirichlet allocation (LDA) Quotation withheld because this record is not cleared for redistribution. builds-on → Sato (2001), online/natural-gradient variational Bayes for EM-fittable models Quotation withheld because this record is not cleared for redistribution. builds-on → the Robbins and Monro (1951) stochastic approximation method Quotation withheld because this record is not cleared for redistribution. extends → the online LDA algorithm of Hoffman et al. (2010a) Quotation withheld because this record is not cleared for redistribution. topic —operationalised-as→ vocabulary topic vocabulary strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span says each topic is a distribution over the vocabulary; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. 300K / 1.8M / 3.8M — Nature, NYT, Wikipedia corpora analyzed about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. 2 million — scale of an example text archive Quotation withheld because this record is not cleared for redistribution. 2 — models used for demonstration (LDA, HDP) Quotation withheld because this record is not cleared for redistribution. SVI is a scalable algorithm for approximating posteriors Quotation withheld because this record is not cleared for redistribution. SVI handles large data and outperforms traditional VI Quotation withheld because this record is not cleared for redistribution. the Bayesian nonparametric model beats its parametric counterpart Quotation withheld because this record is not cleared for redistribution. SVI enables complex Bayesian models on massive data sets Quotation withheld because this record is not cleared for redistribution. posterior distribution posterior distribution Quotation withheld because this record is not cleared for redistribution. topic model topic model Quotation withheld because this record is not cleared for redistribution. probabilistic graphical models probabilistic graphical models Quotation withheld because this record is not cleared for redistribution. stochastic variational inference stochastic variational inference Quotation withheld because this record is not cleared for redistribution. probabilistic modeling probabilistic modeling Quotation withheld because this record is not cleared for redistribution. probabilistic graphical models probabilistic graphical models Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. regression regression about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. sample-size sample-size Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1206.7051.