Variational Inference: A Review for Statisticians

David M. Blei, Alp Kucukelbir, Jon D. McAuliffe · Journal of the American Statistical Association / arXiv · 2017 · statistics · 10.1080/01621459.2017.1285773 · record blei2017-variational-inference
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdf); every evidence string verified as an exact substring of that text before merging. The extracted file is the Taylor & Francis landing page, not the full article: it contains the abstract, keywords, endnotes 1-5 and the endnote reference list only. All facets are therefore grounded in the abstract and endnotes. Claim relations left empty: the endnotes cite Wainwright and Jordan (2008), Barber (2012), Minka (2001) and others, but no single verbatim sentence in the available text relates this paper's own claim to a named prior work without stitching two sentences together, so per the rules nothing was recorded. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Review/tutorial paper, so 'method' is the shape of the contribution (a way of doing approximate inference) rather than a new empirical finding; the paper also contains a small simulation and image-clustering demonstration, but its stated purpose is review. The limitations listed are all scope/known-weakness statements the paper makes about itself and about the method it reviews; several are drawn from Section 5.4 'Open problems'. Extraction is heavy with (cid:...) artifacts and mangled equations, so all quotes were taken from clean prose passages and substring-checked against the source file. Claim relations are restricted to the three places where the paper explicitly says its own derivation is based on / differs from a named prior source; the many bare citations in the applications and theory surveys were not turned into relations. 2026-08-14 schema-0.3 pass: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node. Update, 18 September 2026: the full article has since been staged (2026-08-22, 91,948 characters) and this record's quotes now come from it - all 75 are in it, anchored as deep as character 87,838; the sentences above describe the landing page it was first read from.

This record’s own method is review, and the tags below follow from that. A review’s findings, quantities and statistics are things stated IN this paper, and some of them report results obtained elsewhere. No tag on this page names a different author. Measured across every review in this corpus, 81% of a review’s tags carry no citation at all — most of them are the review speaking. The corpus does not adjudicate the rest: read an unmarked tag as a claim this paper makes, not necessarily one it made first.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
latent variables 2 fields · linear regression 2 fields · posterior distribution 2 fields · variance 2 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 9 concept tags on this paper, 6 are grounded in a quoted sentence.

variational inference Bayesian inference approximate inference posterior distribution Kullback-Leibler divergence closeness evidence lower bound coordinate ascent variational inference MCMC recast posterior inference as optimization: choose the distribution in a simple (mean-field) family that is closest in KL divergence to the true posterior, via coordinate ascent Bayesian statistics posterior probability densities optimization Markov chain Monte Carlo latent variables mean-field variational inference (VI) mean-field variational inference latent variables z posterior density variational family KL divergence Bayesian mean mean of regression linear regression standard-error standard errors variance review

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://doi.org/10.1080/01621459.2017.1285773. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 4 · claim_type 1 · concept_edges 3 · concepts 6 · content_tags 19 · limitations 17 · mechanisms 1 · method 1 · phenomenon 1 · senses 4 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 3 relations here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	variational inference for approximate posterior densities Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	review Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: review	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: recast posterior inference as optimization: choose the distribution in a simple (mean-field) family that is closest in KL divergence to the true posterior, via coordinate ascent	`mechanisms[0]`		the tagger's words
limitations	VI has been studied less rigorously than MCMC and its statistical properties are less well understood Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any statistical guarantee: Quotation withheld AND ITS STATISTICAL PROPERTIES ARE LESS WELL UNDERSTOOD	`limitations[0]`		the tagger's words
limitations	The authors state that variational inference is not yet theoretically well understood. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any guarantee about a variational solution: VI IS POWERFUL, BUT IT IS NOT YET WELL UNDERSTOOD	`limitations[1]`		the tagger's words
limitations	The paper's comparison against MCMC is not definitive; other samplers might perform better. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the paper's own VI-versus-MCMC comparison: THIS IS NOT A DEFINITIVE COMPARISON BETWEEN VARIATIONAL INFERENCE AND MCMC	`limitations[2]`		the tagger's words
limitations	Collapsed variational inference is explicitly excluded from the review's scope. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the review's coverage of collapsed variational inference and related speed-ups: THESE IDEAS ARE OUTSIDE THE SCOPE OF OUR REVIEW	`limitations[3]`		the tagger's words
limitations	The review deliberately restricts itself to the mean-field family, excluding richer variational families. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: every result in the review, which assumes independence: IN THIS REVIEW WE FOCUS ON THE MEAN-FIELD VARIATIONAL FAMILY, WHERE THE LATENT VARIABLES ARE MUTUALLY INDEPENDENT AND EACH GOVERNED BY A DISTINCT FACTOR IN THE VARIATIONAL DENSITY	`limitations[4]`		the tagger's words
limitations	Collapsed variational inference and related speed-ups are excluded from the review's coverage. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. duplicate of: limitations[3] · what this limits: the same scope as limitations[3] - collapsed VI is excluded, so the review is not a survey of what is fastest in practice	`limitations[5]`		the tagger's words
limitations	The survey of applications is not exhaustive. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the survey of applications: THIS DISCUSSION IS NOT EXHAUSTIVE; OUR INTENTION IS TO OUTLINE THE DIVERSITY OF APPLICATIONS OF VARIATIONAL INFERENCE	`limitations[6]`		the tagger's words
limitations	The accuracy of variational inference relative to MCMC is not known, and VI is known to underestimate posterior variance. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any accuracy claim against MCMC: THE RELATIVE ACCURACY OF VARIATIONAL INFERENCE AND MCMC IS STILL UNKNOWN	`limitations[7]`		the tagger's words
limitations	Variational inference systematically underestimates posterior variance because of its objective function. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any use of the posterior variance, with the direction and the cause: VARIATIONAL INFERENCE GENERALLY UNDERESTIMATES THE VARIANCE OF THE POSTERIOR DENSITY; THIS IS A CONSEQUENCE OF ITS OBJECTIVE FUNCTION	`limitations[8]`		the tagger's words
limitations	The paper's empirical CAVI-versus-Hamiltonian-Monte-Carlo timing comparison is not a definitive comparison of the two paradigms. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the empirical timing comparison: OTHER SAMPLERS, SUCH AS A COLLAPSED GIBBS SAMPLER, MAY PERFORM BETTER THAN HAMILTONIAN MONTE CARLO SAMPLING	`limitations[9]`		the tagger's words
limitations	The statistical theory of variational inference remains undeveloped, which the paper flags as an open problem rather than something it settles. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the same scope as limitations[0]: THE STATISTICAL PROPERTIES OF VARIATIONAL INFERENCE ARE NOT YET WELL UNDERSTOOD, ESPECIALLY IN CONTRAST TO THE WEALTH OF ANALYSIS OF MCMC TECHNIQUES	`limitations[10]`		the tagger's words
limitations	The review states that its own subject is not yet well understood, and that its statistical properties have been studied less rigorously than the alternative it is compared to. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: choosing variational inference on the strength of its literature: the review states it has been studied less rigorously than MCMC and that its statistical properties are less well understood	`limitations[11]`	1.0	the tagger's words
limitations	The defect is known, directional, and declared possibly acceptable — the acceptability depending on the task rather than on the method. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any posterior variance, credible interval or uncertainty estimate taken from a variational posterior: the review states VI generally underestimates posterior variance as a consequence of its objective function, and that whether this is acceptable depends on the task	`limitations[12]`	1.0	the tagger's words
limitations	How accurate the method is relative to the alternative is stated to be unknown. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: a claim that variational inference is or is not as accurate as MCMC for a given problem: the review states the relative accuracy is still unknown	`limitations[13]`	1.0	the tagger's words
limitations	The objective actually optimised is a substitute for the one wanted, adopted because the wanted one cannot be computed. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: reading an ELBO value as a measure of divergence from the posterior: the review states the KL cannot be computed and that the ELBO is an alternative objective equal to it up to an added constant	`limitations[14]`	1.0	the tagger's words
limitations	The optimisation reaches a local optimum only, the objective is generally non-convex, and different initialisations were observed to reach different values. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: a single variational fit as THE variational approximation: the review states the ELBO is generally non-convex, that CAVI guarantees only a local optimum sensitive to initialisation, and that ten random initialisations each reached a different value	`limitations[15]`	1.0	the tagger's words
limitations	The monitoring quantity the review recommends in practice lacks the guarantee that belongs to the quantity it stands in for. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: using held-out predictive probability to decide that CAVI has converged: the review recommends it as a proxy while stating it is not guaranteed to increase monotonically, unlike the full ELBO	`limitations[16]`	1.0	the tagger's words
transfers	variational inference used to approximate difficult Bayesian posterior densities in statistics Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`		the tagger's words
concepts	variational inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variational inference	`concepts[0]`		the paper's words
concepts	Kullback-Leibler divergence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Kullback-Leibler divergence	`concepts[4]`		the paper's words
concepts	closeness Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: closeness	`concepts[5]`		the paper's words
concepts	evidence lower bound Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: evidence lower bound	`concepts[6]`		the paper's words
concepts	coordinate ascent variational inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: coordinate ascent variational inference	`concepts[7]`		the paper's words
concepts	MCMC Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: MCMC	`concepts[8]`		the paper's words
senses	accuracy of variational inference - AS POSTERIOR VARIANCE: known to be underestimated, and stated to be a consequence of the objective function rather than of any implementation Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	accuracy of variational inference - AS POSTERIOR PREDICTIVE PERFORMANCE: reported empirically NOT to suffer, which is the opposite verdict on the same method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	closeness - stipulated to be measured by Kullback-Leibler divergence, which is the choice that makes the whole method definite Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
senses	local optimum - not treated as a defect throughout: the review states different initialisations reaching different optima is NOT ALWAYS A DISADVANTAGE Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	1.0	the tagger's words
claim_relations	extends → MCMC sampling for approximate Bayesian inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	extends → Bishop (2006) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	builds-on → Hoffman et al. (2013) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	builds-on → Drugowitsch (2013) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
concept_edges	closeness —operationalised-as→ Kullback-Leibler divergence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: closeness · Kullback-Leibler divergence	`concept_edges[0]`		the tagger's words
concept_edges	coordinate ascent variational inference —is-a→ variational inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'CAVI' = coordinate ascent variational inference, introduced in the previous sentence as an algorithm for the variational optimisation problem; is-a is loose. also in: coordinate ascent variational inference · variational inference	`concept_edges[1]`		the tagger's words
concept_edges	MCMC —distinct-from→ variational inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 also in: MCMC · variational inference	`concept_edges[2]`		the tagger's words
content_tags	1953 — Metropolis-Hastings algorithm landmark Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[0]`		the tagger's words
content_tags	1984 — Gibbs sampler introduced Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.quantities[1]`		the tagger's words
content_tags	variational inference tends to be faster than MCMC and scales to large data Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[0]`		the tagger's words
content_tags	VI approximates densities through optimization rather than sampling Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[1]`		the tagger's words
content_tags	VI is less rigorously studied than MCMC Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[2]`		the tagger's words
content_tags	closeness to the target is measured by KL divergence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`content_tags.findings[3]`		the tagger's words
content_tags	latent variables z Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: latent variables z	`content_tags.variables[0]`		the tagger's words
content_tags	posterior density Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: posterior density	`content_tags.variables[1]`		the tagger's words
content_tags	variational family Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variational family	`content_tags.variables[2]`		the tagger's words
content_tags	KL divergence Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: KL divergence	`content_tags.variables[3]`		the tagger's words
content_tags	variational inference (VI) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variational inference (VI)	`content_tags.definitions[0]`		the tagger's words
content_tags	mean-field variational inference Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean-field variational inference	`content_tags.definitions[1]`		the tagger's words
content_tags	MCMC Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: MCMC	`content_tags.definitions[2]`		the tagger's words
content_tags	posterior Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: posterior	`content_tags.definitions[3]`		the tagger's words
content_tags	Bayesian Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: Bayesian	`content_tags.statistics[0]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mean	`content_tags.statistics[2]`		the tagger's words
content_tags	regression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: regression	`content_tags.statistics[3]`		the tagger's words
content_tags	standard-error Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: standard-error	`content_tags.statistics[4]`		the tagger's words
content_tags	variance Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: variance	`content_tags.statistics[5]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.