| phenomenon | scalable approximate posterior inference in hierarchical Bayesian models— “Inference algorithms of the 1990s and 2000s used to be considered scalable, but they cannot easily handle the amount of data that we described in the four examples above.” |
| claim-type | method— “We develop stochastic variational inference, a scalable algorithm for approximating posterior distributions.” |
| method — evidence | — “We develop this technique for a large class of probabilistic models and we demonstrate it with two probabilistic topic models, latent Dirichlet allocation and the hierarchical Dirichlet process topic model.” |
| transfer | from stochastic approximation (Robbins and Monro 1951): stochastic optimization following noisy gradient estimates, applied to the variational objective so inference need not pass over the whole data set each iteration— “In this paper we derive a more efficient algorithm by using stochastic optimization (Robbins and Monro, 1951), a technique that follows noisy estimates of the gradient of the objective.” | transfers[0] |
| transfer | from information geometry (Amari 1998): the natural gradient, which rescales the step by the Fisher information of the variational family rather than using the Euclidean gradient— “With one more detail—the idea of a natural gradient (Amari, 1998)—stochastic variational inference has an attractive form: 1.” | transfers[1] |
| mechanism | THREE SOURCES FOR A MODEL'S ASSUMPTIONS ARE NAMED, AND WHAT THEY ASSERT IS DECLARED UNOBSERVABLE IN THE SAME CLAUSE: we have assumptions to make - FROM SCIENCE, INTUITION, OR OTHER DATA ANALYSES - that involve STRUCTURES WE BELIEVE EXIST IN THE DATA BUT THAT WE CANNOT DIRECTLY OBSERVE. The provenance of the assumptions and the inaccessibility of their objects are stated together— “Our data are complex and high-dimensional; we have assumptions to make—from science, intuition, or other data analyses—that involve structures we believe exist in the data but that we cannot directly observe; and finally our data sets are large, possibly even arriving in a never-ending stream.” | mechanisms[0] |
| mechanism | A FORMALISM IS DESCRIBED AS A LANGUAGE, AND THE INFERENCE PROCEDURE AS ITS READING: probabilistic graphical models GIVE US A VISUAL LANGUAGE FOR EXPRESSING ASSUMPTIONS about data and its HIDDEN STRUCTURE, and the corresponding algorithms LET US ANALYZE DATA UNDER THOSE ASSUMPTIONS, INFERRING THE HIDDEN STRUCTURE THAT BEST EXPLAINS OUR OBSERVATIONS. The notation is cast as a vocabulary and the computation as its interpretation— “In particular, probabilistic graphical models give us a visual language for expressing assumptions about data and its hidden structure. The corresponding posterior inference algorithms let us analyze data under those assumptions, inferring the hidden structure that best explains our observations.” | mechanisms[1] |
| mechanism | THE PAPER'S OWN TASKS ARE SPLIT INTO TWO KINDS WITH DIFFERENT WARRANTS, AND THE UNOBSERVED PART IS PUT IN SCARE QUOTES: in DESCRIPTIVE tasks the models help EXPLORE the data with the hidden structure probabilistically FILLED IN; in PREDICTIVE tasks they FORM PREDICTIONS ABOUT NEW OBSERVATIONS. The same machinery serves two purposes that would be judged by different evidence— “In descriptive tasks, like problems #1 and #4 above, graphical models help us explore the data— the organization of books or the connections between genes and traits—with the hidden structure probabilistically “filled in.” In predictive tasks, like problems #2 and #3, we use models to form predictions about new observations. For example, we… | mechanisms[2] |
| mechanism | A MODEL CLASS IS DEFINED BY WHAT IT COSTS RATHER THAN BY WHAT IT CONTAINS: NONCONJUGATE MODELS are MODELS WHERE A RICHER PRIOR IS USED AT THE EXPENSE OF MATHEMATICAL CONVENIENCE - and they HAVE EXPANDED THE SUITE of tools available. The category's definition is a trade-off, with the thing traded away named explicitly— “This class of models is expressive, but nonconjugate models—models where a richer prior is used at the expense of mathematical convenience—have expanded the suite of probabilistic tools at our disposal.” | mechanisms[3] |
| mechanism | ONE ALGORITHM IS RE-DESCRIBED AS ANOTHER, AND THE EQUIVALENCE IS STATED AS AN IDENTITY OF STEPS: the classical coordinate ascent algorithm CAN THUS BE INTERPRETED AS a projected natural gradient algorithm - and updating a parameter by taking a natural gradient step OF LENGTH ONE IS EQUIVALENT TO PERFORMING A COORDINATE UPDATE. Two names for one procedure, with the exchange rate given— “The classical coordinate ascent algorithm can thus be interpreted as a projected natural gradient algorithm (Sato, 2001). Updating a parameter by taking a natural gradient step of length one is equivalent to performing a coordinate update.” | mechanisms[4] |
| mechanism | A FORMAL FACT IS GIVEN AN INTUITIVE ROUTE ALONGSIDE THE FORMAL ONE, AND BOTH ARE LABELLED: FORMALLY this is most easily seen via other perspectives, THOUGH IT CAN BE SEEN INTUITIVELY WITH the stick-breaking construction - and then THE INTUITION IS THAT more of the stick is absorbed in the first break locations. The paper marks which of its two explanations is which— “Formally, this is most easily seen via other perspectives on the DP (Ferguson, 1973; Blackwell and MacQueen, 1973; Pitman, 2002), though it can be seen intuitively with the stick-breaking construction. The intuition is that as α gets smaller more of the stick is absorbed in the first break locations” | mechanisms[5] |
| mechanism | THE EVALUATION CRITERION IS OPERATIONALISED AS A PROBABILITY ASSIGNED TO WITHHELD WORDS: under the predictive distribution, A BETTER MODEL WILL ASSIGN HIGHER PROBABILITY TO THE HELD-OUT WORDS - and the procedure divides each document into observed and held-out words, KEEPING THE SETS OF UNIQUE WORDS DISJOINT. What BETTER means is fixed by a number and a partition, and the partition rule is stated— “Under this predictive distribution, a better model will assign higher probability to the held-out words. In more detail, we divide each test document’s words w into a set of observed words wobs and held-out words who, keeping the sets of unique words in wobs and who disjoint.” | mechanisms[6] |
| mechanism | A PROCEDURE IS NAMED BY COMPOSING TWO EXISTING NAMES: interleaving the coordinate ascent updates with an update that increases the bound IS CALLED VARIATIONAL EXPECTATION-MAXIMIZATION - and the stochastic version is then described by what it does DIFFERENTLY, updating the two simultaneously. The name records its parentage and the variant is defined against it— “In the non-stochastic setting, α can be optimized by interleaving the coordinate ascent updates in Figure 3 with an update for α that increases the ELBO. This is called variational expectation-maximization.” | mechanisms[7] |
| extends | variational inference (Jordan et al.)— “variational inference, a method that transforms complex inference problems into high-dimensional optimization problems” | claim_relations[0] |
| builds-on | latent Dirichlet allocation (LDA)— “we demonstrate it with two probabilistic topic models, latent Dirichlet allocation” | claim_relations[1] |
| builds-on | Sato (2001), online/natural-gradient variational Bayes for EM-fittable models— “Our algorithm builds on the earlier approach of Sato (2001), whose algorithm only applies to the limited set of models that can be fit with the EM algorithm (Dempster et al., 1977).” | claim_relations[2] |
| builds-on | the Robbins and Monro (1951) stochastic approximation method— “In this paper, we develop scalable methods for generic Bayesian inference by solving the variational inference problem with stochastic optimization (Robbins and Monro, 1951).” | claim_relations[3] |
| extends | the online LDA algorithm of Hoffman et al. (2010a)— “The algorithm we present was originally developed in Hoffman et al. (2010a), which is a special case of the stochastic variational inference algorithm we developed in Section 2.” | claim_relations[4] |
| sense | a topic - A WEIGHTED DISTRIBUTION OVER THE VOCABULARY: not a subject, a theme or an aboutness, but a probability vector over words, shown to a reader by printing its most frequent members - which is why a multi-word name has no way to stay together inside one— “Each topic is a weighted distribution over the vocabulary and each topic’s plot illustrates its most frequent words” | senses[0] |
| sense | topic proportions - A DISTRIBUTION OVER TOPICS, so a distribution over distributions: the same word does duty at two levels in one sentence, and a document's 'topics' are weights on objects that are themselves weights on words— “Each document in the collection is associated with a vector of topic proportions θd, which is a distribution over topics” | senses[1] |
| sense | the ELBO - A LOWER BOUND SUBSTITUTED FOR THE THING WANTED: the algorithm maximises a bound on the log marginal probability rather than that probability, and the gap between them is not reported because computing it would require the quantity the bound exists to avoid— “It maximizes the evidence lower bound (ELBO), a lower bound on the logarithm of the marginal probability of the observations” | senses[2] |
| control | a test set of 10,000 documents set aside per collection and explicitly withheld from the algorithms during training — held — rules out a fitness figure earned on documents the model was fitted to, with the paper stating the withholding rather than leaving it implied— “For each collection, we set aside a test set of 10, 000 documents for evaluating model fitness; these test sets were not given to the algorithms for training.” | controls[0] |
| audit | topic —operationalised-as→ vocabulary — Audit 2026-09-28: DOUBTFUL - read against its span and kept, but the span says each topic is a distribution over the vocabulary; operationalised-as is loose.— “Each topic is a weighted distribution over the vocabulary and each topic’s plot illustrates its most frequent words” | concept_edges[0] |
| outcome (stated by the tagger, no quotation) | control — held | controls[0] |
| for which use | limitation — the reach of the derivation: WE DEVELOPED OUR ALGORITHM WITH CONJUGATE EXPONENTIAL FAMILY MODELS, so nonconjugate models are outside it— “We developed our algorithm with conjugate exponential family models.” | limitations[0] |
| for which use | limitation — any solution the algorithm returns: IT IS GUARANTEED TO CONVERGE TO A LOCAL OPTIMUM BUT TAKING LARGE STEPS ON THE BASIS OF VERY FEW DATA POINTS MAY LEAD TO A POOR ONE— “Stochastic variational inference is guaranteed to converge to a local optimum but taking large steps on the basis of very few data points may lead to a poor one.” | limitations[1] |
| for which use | limitation — the reported convergence behaviour: THE QUALITY OF THIS STATIONARY POINT AND THE SPEED OF CONVERGENCE MAY DEPEND ON HOW THESE PARAMETERS ARE SET— “Although stochastic variational inference algorithm converges to a stationary point for any valid κ, τ , and S, the quality of this stationary point and the speed of convergence may depend on how these parameters are set.” | limitations[2] |
| for which use | limitation — any model comparison made on this objective: the marginal probability cannot be computed exactly, so what is maximised is a fitted variational lower bound on it— “Since we cannot compute p(x | α) exactly, an approximate approach is to maximize the fitted variational lower bound” | limitations[3] |
| for which use | limitation — results depending on the Dirichlet prior here: the parameters are all set to the same value for simplicity, and the paper notes that conditioning makes the resulting conditional non-exchangeable— “but for simplicity we assume exchangeable Dirichlet priors; that is, we require that all of these parameters are set to the same value” | limitations[4] |
| for which use | limitation — the held-out probability figures reported here: each test document's words are divided into observed and held-out sets, with the sets of unique words kept disjoint— “we divide each test document’s words w into a set of observed words wobs and held-out words who, keeping the sets of unique words in wobs and who disjoint” | limitations[5] |
| for which use | limitation — the number of topics reported for a collection: it is determined by the posterior distribution of the hidden structure under this model, not by anything independent of it— “Given a collection of documents, the posterior distribution of the hidden structure determines how many topics are needed to describe them” | limitations[6] |
| evidence status note | mechanism — RE-SLICED off an UNCLOSED OPENING BRACKET. The first cut ended "...drawn from Beta(1", truncated inside a distribution's argument list. An internal tool did NOT catch it: that tool looks for a CLOSER with nothing to close, and this is the mirror case - an OPENER with nothing closing it. A real gap in the guard, logged as its own question.— “Formally, this is most easily seen via other perspectives on the DP (Ferguson, 1973; Blackwell and MacQueen, 1973; Pitman, 2002), though it can be seen intuitively with the stick-breaking construction. The intuition is that as α gets smaller more of the stick is absorbed in the first break locations” | mechanisms[5] |
| evidence status note | mechanism — RE-SLICED: the first cut ended "we update alpha simultaneously with", dangling in front of a symbol the span does not include.— “In the non-stochastic setting, α can be optimized by interleaving the coordinate ascent updates in Figure 3 with an update for α that increases the ELBO. This is called variational expectation-maximization.” | mechanisms[7] |
| replication | untested |
| no type for this | optimized distribution → posterior — THE INDEXING / PROXY RELATION - 'from' STANDS IN FOR 'to'. We have no type for it: 'causes' asserts production, and 'operationalised-as' says the authors CHOSE a measure, whereas a proxy claim says THIS STANDS IN FOR WHAT WE ACTUALLY CARE ABOUT AND WE KNOW IT IS NOT THE SAME THING. Measured at 8% of papers. The substitution at the centre of variational inference, stated in one line: 'we use the optimized distribution as a proxy for the posterior'.— “Finally, we use the optimized distribution as a proxy for the posterior.” | unhoused_relations[0] |
| limitation | The general algorithm is derived only for conjugate exponential-family models and does not cover nonconjugate models such as correlated or dynamic topic models.— “We developed our algorithm with conjugate exponential family models.” | limitations[0] |
| limitation | Convergence is only to a local optimum, and very small minibatches can lead to a poor one.— “Stochastic variational inference is guaranteed to converge to a local optimum but taking large steps on the basis of very few data points may lead to a poor one.” | limitations[1] |
| limitation | The quality of the solution and speed of convergence depend on how the forgetting rate, delay and minibatch size are set.— “Although stochastic variational inference algorithm converges to a stationary point for any valid κ, τ , and S, the quality of this stationary point and the speed of convergence may depend on how these parameters are set.” | limitations[2] |
| limitation | What is optimised is a bound standing in for a quantity the method cannot compute.— “Since we cannot compute p(x | α) exactly, an approximate approach is to maximize the fitted variational lower bound” | limitations[3] |
| limitation | A prior is made exchangeable for simplicity, and the paper notes the conditional then is not.— “but for simplicity we assume exchangeable Dirichlet priors; that is, we require that all of these parameters are set to the same value” | limitations[4] |
| limitation | The evaluation is built from a split of each test document's words, constructed to be disjoint.— “we divide each test document’s words w into a set of observed words wobs and held-out words who, keeping the sets of unique words in wobs and who disjoint” | limitations[5] |
| limitation | How many topics a collection needs is answered by the model's own posterior.— “Given a collection of documents, the posterior distribution of the hidden structure determines how many topics are needed to describe them” | limitations[6] |