Auto-Encoding Variational Bayes

Auto-Encoding Variational Bayes Diederik P. Kingma Max Welling

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

Auto-Encoding Variational Bayes 2014 ICLR / arXiv 1312.6114 kingma2013-vae kingma2013-vae.txt, the project's staged copy 3f44d5463c80141708178f33e90bb65708ccf79a arxiv-open artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. approximate posterior inference with continuous latent variables Quotation withheld because this record is not cleared for redistribution. system-or-tool-building system-or-tool-building Quotation withheld because this record is not cleared for redistribution. mechanisms the reparameterization trick: rewrite latent-variable sampling as a deterministic function of noise so the variational lower bound (ELBO) becomes differentiable and trainable by stochastic gradient descent Quotation withheld because this record is not cleared for redistribution. The derivation assumes a fixed dataset for simplicity rather than the online or non-stationary case. what this limits: the derivation as presented: OUR METHOD CAN BE APPLIED TO ONLINE, NON-STATIONARY SETTINGS ... BUT HERE WE ASSUME A FIXED DATASET FOR SIMPLICITY Quotation withheld because this record is not cleared for redistribution. The marginal-likelihood evaluation only works for very low-dimensional latent spaces; estimates degrade beyond three latent variables. what this limits: the marginal-likelihood numbers reported: they use 3 LATENT VARIABLES, because FOR HIGHER DIMENSIONAL LATENT SPACE THE ESTIMATES BECAME UNRELIABLE Quotation withheld because this record is not cleared for redistribution. The approximate posterior is assumed Gaussian with diagonal covariance, and the paper argues in advance that this is a choice rather than a limitation. what this limits: results obtained with this method: the approximate posterior is assumed to take an approximately Gaussian form with approximately diagonal covariance, which the authors call a simplifying choice and not a limitation of the method Quotation withheld because this record is not cleared for redistribution. The reconstruction criterion that gives the method its auto-encoding name is stated to be insufficient on its own for learning useful representations. what this limits: an auto-encoding objective read as a representation-learning objective: the paper states it is well known that the Quotation withheld because this record is not cleared for redistribution. The guarantee is conditional on differentiability, which is assumed to hold almost everywhere rather than shown. what this limits: the claim that the estimator works in the intractable case: it holds under mild differentiability conditions, with the prior and likelihood assumed to have PDFs differentiable almost everywhere Quotation withheld because this record is not cleared for redistribution. variational Bayes / approximate posterior inference combined with neural networks (a recognition/encoder model) to learn deep generative models Quotation withheld because this record is not cleared for redistribution. comparison against the wake-sleep algorithm using the same encoder and the same parameters eliminates: that AEVB's advantage over wake-sleep comes from the network architecture or parameter count rather than from the learning method · outcome: held Quotation withheld because this record is not cleared for redistribution. AEVB AEVB Quotation withheld because this record is not cleared for redistribution. recognition model recognition model Quotation withheld because this record is not cleared for redistribution. approximate inference model approximate inference model Quotation withheld because this record is not cleared for redistribution. reparameterization reparameterization Quotation withheld because this record is not cleared for redistribution. lower bound estimator lower bound estimator Quotation withheld because this record is not cleared for redistribution. the algorithm's name - AEVB IN THE PAPER, VAE IN THE FIELD: the authors name their algorithm Auto-Encoding VB and use that name throughout; the term the field kept appears once, as what you arrive at when the recognition model happens to be a neural network Quotation withheld because this record is not cleared for redistribution. the variational auto-encoder - AS A SPECIAL CASE, NOT THE SUBJECT: it is what AEVB becomes when a neural network is used for the recognition model, stated in a single consequence clause Quotation withheld because this record is not cleared for redistribution. the recognition model - TWO NAMES IN ONE PARENTHESIS: an approximate inference model, also called a recognition model, fitted to a posterior that cannot be computed Quotation withheld because this record is not cleared for redistribution. extends → variational inference (variational Bayes) Quotation withheld because this record is not cleared for redistribution. builds-on → predictive sparse decomposition (PSD) [KRL08] Quotation withheld because this record is not cleared for redistribution. extends → the wake-sleep algorithm [HDFN95] Quotation withheld because this record is not cleared for redistribution. approximate inference model —same-as→ recognition model approximate inference model recognition model strength of the relation: 1 Quotation withheld because this record is not cleared for redistribution. reparameterization —causes→ lower bound estimator reparameterization lower bound estimator the relation itself is separately evidenced · strength of the relation: 1 · the relation stated: the reparameterisation YIELDS an estimator that ordinary SGD can optimise Quotation withheld because this record is not cleared for redistribution. A reparameterization of the variational lower bound yields an estimator that can be optimized with standard stochastic gradient methods. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Posterior inference is made efficient by fitting an approximate inference (recognition) model to the intractable posterior using the proposed lower-bound estimator. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. continuous latent variables continuous latent variables Quotation withheld because this record is not cleared for redistribution. variational lower bound variational lower bound Quotation withheld because this record is not cleared for redistribution. SGVB (Stochastic Gradient Variational Bayes) estimator SGVB (Stochastic Gradient Variational Bayes) estimator about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. recognition model recognition model about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian Quotation withheld because this record is not cleared for redistribution. chi-square chi-square Quotation withheld because this record is not cleared for redistribution. variance variance Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/1312.6114.