The Information Bottleneck Method

Naftali Tishby, Fernando C. Pereira, William Bialek · Allerton Conference / arXiv · 1999 · information-theory · record tishby1999-information-bottleneck
tag layer · index · concepts · authors · עברית

Where this record's text came from. faceted 2026-08 by reading the locally held text (pdftext); every evidence string verified as an exact substring of that text before merging. Claim type is 'method' because the contribution is a variational principle plus an algorithm, even though the paper's supporting work is theorem-and-proof; 'method' captures the shape of the contribution and the method facet records that the work is carried out as mathematical derivation and proof rather than experiment or simulation. The paper positions itself against the view that information theory is irrelevant to meaning and mentions Shannon's original formulation, but that passage attributes the omission to a research tradition rather than contradicting a specific named result, so no 'contradicts' relation was recorded. No empirical evaluation is reported here (applications are only listed as existing elsewhere), so replication status is 'untested'. The extraction interleaves displayed equations with running prose, so quotes were restricted to passages that survive intact; all were substring-verified. 2026-08-14, 2026-08-16 and 2026-08-25 schema-0.3 passes: every quote verified as an exact substring of the staged text and anchored at merge time; concept terms verified present; every edge endpoint an existing node.

Some words in these quotes contain the wrong character. The PDF this text was extracted from substituted 2 characters inside a word — a currency sign or a stray accent where a letter pair belongs, as in signi¢cantly for significantly, or a single ligature glyph, as in conﬁdence. They are valid characters in the wrong place, so they survive every encoding check. The quotes keep them for the same reason as above, and a search of this page will not find the affected words.

How to read this page. The article itself is not reproduced here — this paper is not cleared for redistribution, so what follows is the tag layer alone. The evidence sentences are withheld with the article. The table at the foot lists every tag the record carries; each row names the facet it belongs to, and the whose words column says whether the tag records the paper's own wording or the tagger's sentence about the paper. Every tag links to each other paper in the corpus carrying it. The machine-readable edition of the same thing is the TEI file beside this one, where the tags live in standOff, and the article is at its publisher via the link below. When we have changed our mind. Some entries in the table begin with a word in capitals and the reason for it. EVIDENCE CORRECTED — the finding still stands, but we had quoted the wrong sentence from the paper, and the quotation has been replaced with the right one. RETRACTED — the entry itself was wrong; it stays on the page, marked, with the reason, rather than being deleted. WITHDRAWN AND REPLACED — the entry was wrong, and a corrected version of it appears in the entries below. Each mark carries its own reason, in brackets, because the reasons differ from one entry to the next. An entry we have WITHDRAWN is removed from this page and kept in the record, with its reason. A line reading Audit and a date is a later re-reading of the entry against its whole sentence: DOUBTFUL means it was kept, and the reason says what is loose about it. Quotation marks inside the tagger's sentence. Where a tag is the tagger's own sentence, a phrase in quotation marks inside it is borrowed from the paper — the tagger's wording around it is not. The paper's own sentence is the evidence quote for that row; on this page it is withheld with the article, and the article is at its publisher. Nothing inside quotation marks anywhere on this page is ours: if it is quoted, it is the paper's. Whose words a concept carries. A concept entry records a term the paper itself uses, so the term, the sense given for it, and its evidence are the paper's own words, quoted where the paper fixes them. This is the opposite of a sense entry in the tag table, which is the tagger's sentence about how the paper uses a term. Where a concept entry could not be grounded in the paper's words, the entry says so on its own row.

Other fields define these differently the same word, defined another way in another discipline
information 4 fields

Tags on this paper

Every tag the record carries, each linking to every other paper in the corpus that carries it. A tag appears here whether or not its words occur in the article — the spans below are only the ones a sentence in this paper licenses. Of the 7 concept tags on this paper, 5 are grounded in a quoted sentence.

information bottleneck mutual information rate-distortion relevant information distortion function relevance variable meaningful information relevance is supplied by a SECOND VARIABLE with positive mutual information, in place of a chosen distortion function, and the joint distribution of the two is assumed available one Lagrange multiplier carries the whole tradeoff, and its two limits are stated: at zero everything collapses to a single codeword, and as it grows the quantization becomes arbitrarily fine the Markov condition is stated NOT to be a modelling assumption - the compressed variable is not a hidden variable in a model of the data - and the paper names the different condition a model would have required instead the compressed representation and its representatives are solved for TOGETHER, which is what separates this from rate distortion theory, where choosing representatives is a separate problem convergence of the alternating iterations is argued from each step minimising the same functional independently, that functional being bounded below as a sum of two non-negative terms raising the multiplier traces convex curves in the information plane, and pairs of curves separate at a finite critical value through a second order phase transition, producing a hierarchy of quantizations information signal distortion compression bottleneck rate relevant quantization information about Y Bayesian mean average of mathematical-proof

Legend

relation to other work	`claim_relations`	how this paper stands to another: agrees, contradicts, extends, qualifies
claim type	`claim_type`	the shape of the claim: existence, mechanism, method, critique, prediction, null
concept edge	`concept_edges`	an asserted relation between two concepts, with the sentence asserting it
concept	`concepts`	an idea the paper works with, quoted where the paper uses it
content tag	`content_tags`	a finding, definition, quantity, statistic or variable named in the paper
limitation	`limitations`	what the paper says its evidence cannot support
mechanism	`mechanisms`	how the reported effect is said to come about
method	`method`	how the paper produced its evidence
phenomenon	`phenomenon`	the effect or regularity under study
sense	`senses`	a distinct use of a term, quoted where the paper fixes it
transfer	`transfers`	a method or concept carried in from another field

The paper

The full text is not reproduced here. This article is not cleared for redistribution, so the page carries the tag layer only. Its evidence sentences are withheld; the grounding status remains with each tag, and the article itself is at https://arxiv.org/abs/physics/0004057. The reading view exists for the papers whose licence permits it.

Every tag on this page

claim_relations 5 · claim_type 1 · concept_edges 1 · concepts 5 · content_tags 7 · limitations 8 · mechanisms 6 · method 1 · phenomenon 1 · senses 4 · transfers 1

Relations on this page carry a quotation for the claim. Where a relation was separately evidenced — and so tested against the vocabulary for its type — the row says so; none of 1 relation here carry that mark. The rest are the tagger’s reading of the sentence quoted beside them.

This record holds no controls. A layer is empty when the tagger recorded nothing in it — which is not the same as the paper having nothing to record.

facet	tag	field	conf.	whose words
claim_type	method Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_type`		the paper's words
phenomenon	relevant information extraction under lossy compression Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`phenomenon`		the paper's words
method	mathematical-proof Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mathematical-proof	`method`		the paper's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: relevance is supplied by a SECOND VARIABLE with positive mutual information, in place of a chosen distortion function, and the joint distribution of the two is assumed available	`mechanisms[0]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: one Lagrange multiplier carries the whole tradeoff, and its two limits are stated: at zero everything collapses to a single codeword, and as it grows the quantization becomes arbitrarily fine	`mechanisms[1]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the Markov condition is stated NOT to be a modelling assumption - the compressed variable is not a hidden variable in a model of the data - and the paper names the different condition a model would have required instead	`mechanisms[2]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: the compressed representation and its representatives are solved for TOGETHER, which is what separates this from rate distortion theory, where choosing representatives is a separate problem	`mechanisms[3]`	0.9	the tagger's words
mechanisms	that the alternation runs over THREE convex sets is stated in Theorem 5, a different span, not in this one Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: convergence of the alternating iterations is argued from each step minimising the same functional independently, that functional being bounded below as a sum of two non-negative terms	`mechanisms[4]`	0.9	the tagger's words
mechanisms	mechanisms Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: raising the multiplier traces convex curves in the information plane, and pairs of curves separate at a finite critical value through a second order phase transition, producing a hierarchy of quantizations	`mechanisms[5]`	0.9	the tagger's words
limitations	The convergence proof does not establish that the solution is unique, since the functional is convex in each distribution separately but not jointly. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any claim that the method finds the solution: THUS OUR CONVERGENCE PROOF DOES NOT IMPLY UNIQUENESS OF THE SOLUTION	`limitations[0]`		the tagger's words
limitations	The solution is formal: the conditional distribution appearing in the exponent is defined only implicitly in terms of the mapping being solved for. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the solution as something to compute with: IT IS A FORMAL SOLUTION SINCE p(y\|x~) IN THE EXPONENTIAL IS DEFINED IMPLICITLY IN TERMS OF THE ASSIGNMENT MAPPING	`limitations[1]`		the tagger's words
limitations	The structure and cardinality of the compressed representation still have to be specified externally and are not determined by the theory. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the claim that the method determines the representation: THE FORMAL SOLUTION ... STILL REQUIRES A SPECIFICATION OF THE STRUCTURE AND CARDINALITY OF X~, AS IN RATE DISTORTION THEORY	`limitations[2]`		the tagger's words
limitations	The method assumes access to the joint distribution; how to obtain an adequate sample of it is left outside the paper. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: every application to finite data: THE PROBLEM OF ACTUALLY OBTAINING A GOOD ENOUGH SAMPLE OF THIS DISTRIBUTION IS AN INTERESTING ISSUE IN LEARNING THEORY, BUT IS BEYOND THE SCOPE OF THIS PAPER	`limitations[3]`		the tagger's words
limitations	The convergence proof for the bottleneck iterations is only sketched in this paper for reasons of length. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: the convergence result as a proved theorem in this paper: FOR LACK OF SPACE WE CAN ONLY OUTLINE THE PROOF	`limitations[4]`		the tagger's words
limitations	The problem the paper addresses is stated to be ill-posed until someone supplies a definition of relevance, which is a choice and not a measurement. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any claim that a compressed representation keeps the relevant information: the problem is not well posed without a suitable definition of relevance, so the definition is supplied by the analyst before any result exists	`limitations[5]`	1.0	the tagger's words
limitations	Choosing the distortion function is choosing which features matter, and the paper says the correct one is virtually impossible to obtain for a real signal. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: a rate-distortion result as a statement about which parts of a signal matter: the distortion function is an implicit feature selection, and the paper states it is virtually impossible to come up with the correct one for acoustic signals	`limitations[6]`	1.0	the tagger's words
limitations	The trade-off the method exposes has no uniquely right setting; the balance point is a parameter the analyst turns. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. what this limits: any single information-bottleneck solution: compression is traded against preserved meaningful information, there is no single right solution for that trade-off, and the balance is set by the Lagrange multiplier	`limitations[7]`	1.0	the tagger's words
transfers	the rate-distortion trade-off curve and its cardinality specification, taken from lossy source coding and re-drawn in the information plane as the bottleneck curve Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`transfers[0]`	0.85	the tagger's words
concepts	mutual information Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: mutual information	`concepts[1]`		the paper's words
concepts	relevant information Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: relevant information	`concepts[4]`		the paper's words
concepts	distortion function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: distortion function	`concepts[5]`		the paper's words
concepts	relevance variable Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: relevance variable	`concepts[6]`		the paper's words
concepts	meaningful information Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: meaningful information	`concepts[7]`		the paper's words
senses	relevant information - DEFINED BY A CHOSEN SECOND VARIABLE: what a signal tells you about another signal Y, so relevance is never a property of the signal alone and the choice of Y fixes which features count Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[0]`	1.0	the tagger's words
senses	relevance - AS SET BY THE ANALYST'S PURPOSE: the same speech signal has different relevant features depending on whether the task is recognition or speaker identification, and the paper lists the choice per domain Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[1]`	1.0	the tagger's words
senses	meaning - WHAT SHANNON DELIBERATELY LEFT OUT: information theory was formulated about transmitting information rather than judging its value to the recipient, and this paper returns to the excluded question Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[2]`	1.0	the tagger's words
senses	the distortion measure - EMERGENT, NOT ASSUMED: the KL divergence falls out of the variational principle rather than being chosen in advance, which is the paper's answer to the arbitrariness it identifies Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`senses[3]`	1.0	the tagger's words
claim_relations	extends → rate-distortion theory (Shannon) Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[0]`		the tagger's words
claim_relations	contradicts → the arbitrary choice of distortion function in rate-distortion theory Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[1]`		the tagger's words
claim_relations	extends → rate distortion theory Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[2]`		the tagger's words
claim_relations	extends → the Blahut–Arimoto algorithm Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[3]`		the tagger's words
claim_relations	builds-on → the convergence lemma of Csiszár and Tusnády Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above.	`claim_relations[4]`		the tagger's words
concept_edges	relevant information —operationalised-as→ distortion function Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. strength of the relation: 1 Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span says the distortion function IMPLICITLY specifies the relevant aspects; operationalised-as is loose. also in: relevant information · distortion function	`concept_edges[0]`		the tagger's words
content_tags	The information X gives about Y is squeezed through a bottleneck of codewords, a constrained optimization that generalizes rate-distortion theory with the distortion measure emerging from the joint statistics of X and Y. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted.	`content_tags.findings[0]`		the tagger's words
content_tags	The method yields an exact set of self-consistent equations solvable by a convergent re-estimation generalizing the Blahut–Arimoto algorithm. Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted.	`content_tags.findings[1]`		the tagger's words
content_tags	information about Y Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. also in: information about Y	`content_tags.variables[1]`		the tagger's words
content_tags	relevant information Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE REPAIRED: the extractor split a hyphen-space and the quote was recorded faithfully with the damage in it, which made it unanchorable. The repaired string is an EXACT SUBSTRING of the staged text - that is why it was accepted, not because a rule produced it. The text as originally recorded is kept in evidence as extracted. also in: relevant information	`content_tags.definitions[0]`		the tagger's words
content_tags	information bottleneck Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: information bottleneck	`content_tags.definitions[1]`		the tagger's words
content_tags	Bayesian Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: Bayesian	`content_tags.statistics[0]`		the tagger's words
content_tags	mean Quotation withheld — this record is not cleared for redistribution. The sentence is in the paper at the link above. about this quote: QUOTE RECOVERED FROM THE PAPER'S OWN CHARACTERS. The recorded quote was LETTERS-ONLY IDENTICAL to this stretch of the staged text but not an exact substring of it - spaces, hyphens, ligature glyphs or mathematical operators differed, so it could never anchor. This is the file's own text between the same two letters: not produced by a rule about the damage, copied out of the file. Round-trip checked (the recovered span is letters-only equal to the original) and length-checked. The text as originally recorded is kept in evidence as extracted. also in: mean	`content_tags.statistics[1]`		the tagger's words

Interpretive choices

Tags that could not be placed in the text:

none — every tag was located in the text

What was deliberately not tagged, and why, is recorded per paper by the tagger. Where a paper has no such record, this page says so rather than implying that nothing was declined.