The Information Bottleneck Method

The Information Bottleneck Method Naftali Tishby Fernando C. Pereira William Bialek

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

The Information Bottleneck Method 1999 Allerton Conference / arXiv physics/0004057 tishby1999-information-bottleneck tishby1999-information-bottleneck.txt, the project's staged copy 8520923960fceb55c1b08fe27fbe705c9b5dfdb6 arxiv-open information-theory

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them method Quotation withheld because this record is not cleared for redistribution. relevant information extraction under lossy compression Quotation withheld because this record is not cleared for redistribution. mathematical-proof mathematical-proof Quotation withheld because this record is not cleared for redistribution. mechanisms relevance is supplied by a SECOND VARIABLE with positive mutual information, in place of a chosen distortion function, and the joint distribution of the two is assumed available Quotation withheld because this record is not cleared for redistribution. mechanisms one Lagrange multiplier carries the whole tradeoff, and its two limits are stated: at zero everything collapses to a single codeword, and as it grows the quantization becomes arbitrarily fine Quotation withheld because this record is not cleared for redistribution. mechanisms the Markov condition is stated NOT to be a modelling assumption - the compressed variable is not a hidden variable in a model of the data - and the paper names the different condition a model would have required instead Quotation withheld because this record is not cleared for redistribution. mechanisms the compressed representation and its representatives are solved for TOGETHER, which is what separates this from rate distortion theory, where choosing representatives is a separate problem Quotation withheld because this record is not cleared for redistribution. that the alternation runs over THREE convex sets is stated in Theorem 5, a different span, not in this one convergence of the alternating iterations is argued from each step minimising the same functional independently, that functional being bounded below as a sum of two non-negative terms Quotation withheld because this record is not cleared for redistribution. mechanisms raising the multiplier traces convex curves in the information plane, and pairs of curves separate at a finite critical value through a second order phase transition, producing a hierarchy of quantizations Quotation withheld because this record is not cleared for redistribution. The convergence proof does not establish that the solution is unique, since the functional is convex in each distribution separately but not jointly. what this limits: any claim that the method finds the solution: THUS OUR CONVERGENCE PROOF DOES NOT IMPLY UNIQUENESS OF THE SOLUTION Quotation withheld because this record is not cleared for redistribution. The solution is formal: the conditional distribution appearing in the exponent is defined only implicitly in terms of the mapping being solved for. what this limits: the solution as something to compute with: IT IS A FORMAL SOLUTION SINCE p(y|x~) IN THE EXPONENTIAL IS DEFINED IMPLICITLY IN TERMS OF THE ASSIGNMENT MAPPING Quotation withheld because this record is not cleared for redistribution. The structure and cardinality of the compressed representation still have to be specified externally and are not determined by the theory. what this limits: the claim that the method determines the representation: THE FORMAL SOLUTION ... STILL REQUIRES A SPECIFICATION OF THE STRUCTURE AND CARDINALITY OF X~, AS IN RATE DISTORTION THEORY Quotation withheld because this record is not cleared for redistribution. The method assumes access to the joint distribution; how to obtain an adequate sample of it is left outside the paper. what this limits: every application to finite data: THE PROBLEM OF ACTUALLY OBTAINING A GOOD ENOUGH SAMPLE OF THIS DISTRIBUTION IS AN INTERESTING ISSUE IN LEARNING THEORY, BUT IS BEYOND THE SCOPE OF THIS PAPER Quotation withheld because this record is not cleared for redistribution. The convergence proof for the bottleneck iterations is only sketched in this paper for reasons of length. what this limits: the convergence result as a proved theorem in this paper: FOR LACK OF SPACE WE CAN ONLY OUTLINE THE PROOF Quotation withheld because this record is not cleared for redistribution. The problem the paper addresses is stated to be ill-posed until someone supplies a definition of relevance, which is a choice and not a measurement. what this limits: any claim that a compressed representation keeps the relevant information: the problem is not well posed without a suitable definition of relevance, so the definition is supplied by the analyst before any result exists Quotation withheld because this record is not cleared for redistribution. Choosing the distortion function is choosing which features matter, and the paper says the correct one is virtually impossible to obtain for a real signal. what this limits: a rate-distortion result as a statement about which parts of a signal matter: the distortion function is an implicit feature selection, and the paper states it is virtually impossible to come up with the correct one for acoustic signals Quotation withheld because this record is not cleared for redistribution. The trade-off the method exposes has no uniquely right setting; the balance point is a parameter the analyst turns. what this limits: any single information-bottleneck solution: compression is traded against preserved meaningful information, there is no single right solution for that trade-off, and the balance is set by the Lagrange multiplier Quotation withheld because this record is not cleared for redistribution. the rate-distortion trade-off curve and its cardinality specification, taken from lossy source coding and re-drawn in the information plane as the bottleneck curve Quotation withheld because this record is not cleared for redistribution. mutual information mutual information Quotation withheld because this record is not cleared for redistribution. relevant information relevant information Quotation withheld because this record is not cleared for redistribution. distortion function distortion function Quotation withheld because this record is not cleared for redistribution. relevance variable relevance variable Quotation withheld because this record is not cleared for redistribution. meaningful information meaningful information Quotation withheld because this record is not cleared for redistribution. relevant information - DEFINED BY A CHOSEN SECOND VARIABLE: what a signal tells you about another signal Y, so relevance is never a property of the signal alone and the choice of Y fixes which features count Quotation withheld because this record is not cleared for redistribution. relevance - AS SET BY THE ANALYST'S PURPOSE: the same speech signal has different relevant features depending on whether the task is recognition or speaker identification, and the paper lists the choice per domain Quotation withheld because this record is not cleared for redistribution. meaning - WHAT SHANNON DELIBERATELY LEFT OUT: information theory was formulated about transmitting information rather than judging its value to the recipient, and this paper returns to the excluded question Quotation withheld because this record is not cleared for redistribution. the distortion measure - EMERGENT, NOT ASSUMED: the KL divergence falls out of the variational principle rather than being chosen in advance, which is the paper's answer to the arbitrariness it identifies Quotation withheld because this record is not cleared for redistribution. extends → rate-distortion theory (Shannon) Quotation withheld because this record is not cleared for redistribution. contradicts → the arbitrary choice of distortion function in rate-distortion theory Quotation withheld because this record is not cleared for redistribution. extends → rate distortion theory Quotation withheld because this record is not cleared for redistribution. extends → the Blahut–Arimoto algorithm Quotation withheld because this record is not cleared for redistribution. builds-on → the convergence lemma of Csiszár and Tusnády Quotation withheld because this record is not cleared for redistribution. relevant information —operationalised-as→ distortion function relevant information distortion function strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the distortion function IMPLICITLY specifies the relevant aspects; operationalised-as is loose. Quotation withheld because this record is not cleared for redistribution. The information X gives about Y is squeezed through a bottleneck of codewords, a constrained optimization that generalizes rate-distortion theory with the distortion measure emerging from the joint statistics of X and Y. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. The method yields an exact set of self-consistent equations solvable by a convergent re-estimation generalizing the Blahut–Arimoto algorithm. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. information about Y information about Y Quotation withheld because this record is not cleared for redistribution. relevant information relevant information about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. information bottleneck information bottleneck about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. Bayesian Bayesian about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution. mean mean about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://arxiv.org/abs/physics/0004057.