Variational Inference: A Review for Statisticians

Variational Inference: A Review for Statisticians David M. Blei Alp Kucukelbir Jon D. McAuliffe

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Variational Inference: A Review for Statisticians 2017 Journal of the American Statistical Association / arXiv 10.1080/01621459.2017.1285773 1601.00670 blei2017-variational-inference blei2017-variational-inference.txt, the project's staged copy feb9cb16d975cd88cfb14e6d318a63a72d32f0d4 arxiv-open statistics

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. variational inference for approximate posterior densities Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. mechanisms recast posterior inference as optimization: choose the distribution in a simple (mean-field) family that is closest in KL divergence to the true posterior, via coordinate ascent Quotation withheld because this record is not cleared for redistribution. VI has been studied less rigorously than MCMC and its statistical properties are less well understood what this limits: any statistical guarantee: Quotation withheld AND ITS STATISTICAL PROPERTIES ARE LESS WELL UNDERSTOOD Quotation withheld because this record is not cleared for redistribution. The authors state that variational inference is not yet theoretically well understood. what this limits: any guarantee about a variational solution: VI IS POWERFUL, BUT IT IS NOT YET WELL UNDERSTOOD Quotation withheld because this record is not cleared for redistribution. The paper's comparison against MCMC is not definitive; other samplers might perform better. what this limits: the paper's own VI-versus-MCMC comparison: THIS IS NOT A DEFINITIVE COMPARISON BETWEEN VARIATIONAL INFERENCE AND MCMC Quotation withheld because this record is not cleared for redistribution. Collapsed variational inference is explicitly excluded from the review's scope. what this limits: the review's coverage of collapsed variational inference and related speed-ups: THESE IDEAS ARE OUTSIDE THE SCOPE OF OUR REVIEW Quotation withheld because this record is not cleared for redistribution. The review deliberately restricts itself to the mean-field family, excluding richer variational families. what this limits: every result in the review, which assumes independence: IN THIS REVIEW WE FOCUS ON THE MEAN-FIELD VARIATIONAL FAMILY, WHERE THE LATENT VARIABLES ARE MUTUALLY INDEPENDENT AND EACH GOVERNED BY A DISTINCT FACTOR IN THE VARIATIONAL DENSITY Quotation withheld because this record is not cleared for redistribution. Collapsed variational inference and related speed-ups are excluded from the review's coverage. duplicate of: limitations[3] · what this limits: the same scope as limitations[3] - collapsed VI is excluded, so the review is not a survey of what is fastest in practice Quotation withheld because this record is not cleared for redistribution. The survey of applications is not exhaustive. what this limits: the survey of applications: THIS DISCUSSION IS NOT EXHAUSTIVE; OUR INTENTION IS TO OUTLINE THE DIVERSITY OF APPLICATIONS OF VARIATIONAL INFERENCE Quotation withheld because this record is not cleared for redistribution. The accuracy of variational inference relative to MCMC is not known, and VI is known to underestimate posterior variance. what this limits: any accuracy claim against MCMC: THE RELATIVE ACCURACY OF VARIATIONAL INFERENCE AND MCMC IS STILL UNKNOWN Quotation withheld because this record is not cleared for redistribution. Variational inference systematically underestimates posterior variance because of its objective function. what this limits: any use of the posterior variance, with the direction and the cause: VARIATIONAL INFERENCE GENERALLY UNDERESTIMATES THE VARIANCE OF THE POSTERIOR DENSITY; THIS IS A CONSEQUENCE OF ITS OBJECTIVE FUNCTION Quotation withheld because this record is not cleared for redistribution. The paper's empirical CAVI-versus-Hamiltonian-Monte-Carlo timing comparison is not a definitive comparison of the two paradigms. what this limits: the empirical timing comparison: OTHER SAMPLERS, SUCH AS A COLLAPSED GIBBS SAMPLER, MAY PERFORM BETTER THAN HAMILTONIAN MONTE CARLO SAMPLING Quotation withheld because this record is not cleared for redistribution. The statistical theory of variational inference remains undeveloped, which the paper flags as an open problem rather than something it settles. what this limits: the same scope as limitations[0]: THE STATISTICAL PROPERTIES OF VARIATIONAL INFERENCE ARE NOT YET WELL UNDERSTOOD, ESPECIALLY IN CONTRAST TO THE WEALTH OF ANALYSIS OF MCMC TECHNIQUES Quotation withheld because this record is not cleared for redistribution. The review states that its own subject is not yet well understood, and that its statistical properties have been studied less rigorously than the alternative it is compared to. what this limits: choosing variational inference on the strength of its literature: the review states it has been studied less rigorously than MCMC and that its statistical properties are less well understood Quotation withheld because this record is not cleared for redistribution. The defect is known, directional, and declared possibly acceptable — the acceptability depending on the task rather than on the method. what this limits: any posterior variance, credible interval or uncertainty estimate taken from a variational posterior: the review states VI generally underestimates posterior variance as a consequence of its objective function, and that whether this is acceptable depends on the task Quotation withheld because this record is not cleared for redistribution. How accurate the method is relative to the alternative is stated to be unknown. what this limits: a claim that variational inference is or is not as accurate as MCMC for a given problem: the review states the relative accuracy is still unknown Quotation withheld because this record is not cleared for redistribution. The objective actually optimised is a substitute for the one wanted, adopted because the wanted one cannot be computed. what this limits: reading an ELBO value as a measure of divergence from the posterior: the review states the KL cannot be computed and that the ELBO is an alternative objective equal to it up to an added constant Quotation withheld because this record is not cleared for redistribution. The optimisation reaches a local optimum only, the objective is generally non-convex, and different initialisations were observed to reach different values. what this limits: a single variational fit as THE variational approximation: the review states the ELBO is generally non-convex, that CAVI guarantees only a local optimum sensitive to initialisation, and that ten random initialisations each reached a different value Quotation withheld because this record is not cleared for redistribution. The monitoring quantity the review recommends in practice lacks the guarantee that belongs to the quantity it stands in for. what this limits: using held-out predictive probability to decide that CAVI has converged: the review recommends it as a proxy while stating it is not guaranteed to increase monotonically, unlike the full ELBO Quotation withheld because this record is not cleared for redistribution. variational inference used to approximate difficult Bayesian posterior densities in statistics Quotation withheld because this record is not cleared for redistribution. variational inference variational inference Quotation withheld because this record is not cleared for redistribution. Kullback-Leibler divergence Kullback-Leibler divergence Quotation withheld because this record is not cleared for redistribution. closeness closeness Quotation withheld because this record is not cleared for redistribution. evidence lower bound evidence lower bound Quotation withheld because this record is not cleared for redistribution. coordinate ascent variational inference coordinate ascent variational inference Quotation withheld because this record is not cleared for redistribution. MCMC MCMC Quotation withheld because this record is not cleared for redistribution. accuracy of variational inference - AS POSTERIOR VARIANCE: known to be underestimated, and stated to be a consequence of the objective function rather than of any implementation Quotation withheld because this record is not cleared for redistribution. accuracy of variational inference - AS POSTERIOR PREDICTIVE PERFORMANCE: reported empirically NOT to suffer, which is the opposite verdict on the same method Quotation withheld because this record is not cleared for redistribution. closeness - stipulated to be measured by Kullback-Leibler divergence, which is the choice that makes the whole method definite Quotation withheld because this record is not cleared for redistribution. local optimum - not treated as a defect throughout: the review states different initialisations reaching different optima is NOT ALWAYS A DISADVANTAGE Quotation withheld because this record is not cleared for redistribution. extends → MCMC sampling for approximate Bayesian inference Quotation withheld because this record is not cleared for redistribution. extends → Bishop (2006) Quotation withheld because this record is not cleared for redistribution. builds-on → Hoffman et al. (2013) Quotation withheld because this record is not cleared for redistribution. builds-on → Drugowitsch (2013) Quotation withheld because this record is not cleared for redistribution. closeness —operationalised-as→ Kullback-Leibler divergence closeness Kullback-Leibler divergence strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. coordinate ascent variational inference —is-a→ variational inference coordinate ascent variational inference variational inference strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'CAVI' = coordinate ascent variational inference, introduced in the previous sentence as an algorithm for the variational optimisation problem; is-a is loose. Quotation withheld because this record is not cleared for redistribution. MCMC —distinct-from→ variational inference MCMC variational inference strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. 1953 — Metropolis-Hastings algorithm landmark Quotation withheld because this record is not cleared for redistribution. 1984 — Gibbs sampler introduced Quotation withheld because this record is not cleared for redistribution. variational inference tends to be faster than MCMC and scales to large data Quotation withheld because this record is not cleared for redistribution. VI approximates densities through optimization rather than sampling Quotation withheld because this record is not cleared for redistribution. VI is less rigorously studied than MCMC Quotation withheld because this record is not cleared for redistribution. closeness to the target is measured by KL divergence Quotation withheld because this record is not cleared for redistribution. latent variables z latent variables z Quotation withheld because this record is not cleared for redistribution. posterior density posterior density Quotation withheld because this record is not cleared for redistribution. variational family variational family Quotation withheld because this record is not cleared for redistribution. KL divergence KL divergence Quotation withheld because this record is not cleared for redistribution. variational inference (VI) variational inference (VI) Quotation withheld because this record is not cleared for redistribution. mean-field variational inference mean-field variational inference Quotation withheld because this record is not cleared for redistribution. MCMC MCMC Quotation withheld because this record is not cleared for redistribution. posterior posterior Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian Quotation withheld because this record is not cleared for redistribution. mean mean Quotation withheld because this record is not cleared for redistribution. regression regression Quotation withheld because this record is not cleared for redistribution. standard-error standard-error Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1080/01621459.2017.1285773.