A theory of learning from different domains

A theory of learning from different domains Shai Ben-David John Blitzer Koby Crammer Alex Kulesza Fernando Pereira Jennifer Wortman Vaughan

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

A theory of learning from different domains 2009 Machine Learning 10.1007/s10994-009-5152-4 ben-david2009-a-theory-of-learning-from-differen ben-david2009-a-theory-of-learning-from-differen.txt, the project's staged copy ee2dc8a228ee7f96387e55dfb71f7172a97dcb45 cc-by-nc machine-learning

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them when a classifier trained on one distribution can be expected to work on another, and how to weight source against target data when a little labelled target data exists Quotation withheld because this record is not cleared for redistribution. what quantity, estimable from unlabeled samples, bounds the target error of a source-trained classifier Quotation withheld because this record is not cleared for redistribution. a classifier-induced divergence that can be estimated from finite unlabeled samples, and a bound in which the target error is the source error plus that divergence plus the error of the ideal joint hypothesis Quotation withheld because this record is not cleared for redistribution. mechanism Quotation withheld because this record is not cleared for redistribution. domain adaptation - training where the labels are and deploying where they are not, with a theory whose usable half is the half that can be computed without labels Quotation withheld because this record is not cleared for redistribution. mathematical-proof mathematical-proof Quotation withheld because this record is not cleared for redistribution. the L1 divergence takes a supremum over all measurable subsets and is rejected as unestimable and over-strict; the H-divergence takes it over the hypothesis class. So the distance between two FIXED distributions is a function of H - a richer class sees the domains as further apart, and the bound loosens because the model got more expressive rather than because the data moved a distance defined by the classifier you were going to use Quotation withheld because this record is not cleared for redistribution. label every unlabeled source instance 0 and every unlabeled target instance 1, train a classifier to tell them apart, and read the divergence off its error. If the domains are indistinguishable to H the divergence is zero. The measurement IS the adversary, which is why the whole domain-adversarial literature follows from this paragraph measure the difference by trying to detect it Quotation withheld because this record is not cleared for redistribution. target error is bounded by source error plus divergence plus lambda. The first two come from unlabeled data. Lambda is the error of the best joint hypothesis and needs labelled target data to evaluate - the thing the setting is defined by not having. THE BOUND IS USABLE EXACTLY WHEN ITS UNCHECKABLE TERM IS SMALL the third term is the assumption Quotation withheld because this record is not cleared for redistribution. two domains can differ in their labeling functions as well as in their input distributions. One clause - `since we expect the labeling function difference to be small in practice` - removes the first from the theory. Everything downstream measures distributional shift and inherits the assumption without restating it the half of the difference that is assumed away Quotation withheld because this record is not cleared for redistribution. computing the H-divergence needs the minimum-error hypothesis for the source-versus-target problem, which is intractable for most classes. What is used instead is the error of a hypothesis minimising a convex upper bound - so the reported divergence is an approximation of an approximation, and the paper says so an intractable quantity approximated by a surrogate Quotation withheld because this record is not cleared for redistribution. the bound is only useful when the ideal joint hypothesis has small error, and that error cannot be estimated without labelled target data Quotation withheld because this record is not cleared for redistribution. the difference between the two domains' labeling functions is assumed small rather than measured Quotation withheld because this record is not cleared for redistribution. minimising error to compute the H-divergence is computationally intractable for most hypothesis classes, so a convex surrogate is used Quotation withheld because this record is not cleared for redistribution. common divergences such as L1 and Kullback-Leibler cannot be accurately estimated from finite samples in a distribution-free setting Quotation withheld because this record is not cleared for redistribution. with enough labelled target data the source data is never preferred, no matter how much of it there is - the method's value is bounded by target-label scarcity Quotation withheld because this record is not cleared for redistribution. the practical recommendation holds only given a reasonable measure of complexity Quotation withheld because this record is not cleared for redistribution. a divergence built for change detection in data streams, taken into statistical learning theory as the domain-distance term of a generalisation bound Quotation withheld because this record is not cleared for redistribution. binary classification itself used as a measuring instrument - a discriminator trained to separate two unlabeled samples, with its error read as a distance Quotation withheld because this record is not cleared for redistribution. the H-divergence checked against the L1 divergence on both counts that matter - estimability from finite samples and tightness of the resulting bound eliminates: a new measure adopted for convenience - it is shown never to be larger than the one it replaces, so the bound cannot have been loosened to make it work · outcome: held Quotation withheld because this record is not cleared for redistribution. the bound's behaviour examined at both extremes of lambda rather than only where it is convenient eliminates: a bound that is quietly vacuous when adaptation is impossible - the paper states plainly that a large lambda means no classifier serves both domains · outcome: held Quotation withheld because this record is not cleared for redistribution. the intractability of computing the H-divergence stated, with the convex surrogate named as an approximation rather than presented as the quantity eliminates: a reported divergence being read as the defined one - the gap between them is declared at the point the substitution is made · outcome: held Quotation withheld because this record is not cleared for redistribution. source domains at varying distance compared against target-only training as labelled target data increases eliminates: the claim that source data always helps - with enough target labels it never does, whatever its volume · outcome: failed Quotation withheld because this record is not cleared for redistribution. the relation to Crammer et al. (2008) worked out, with the conditions under which the two measures coincide eliminates: that the two divergence measures differ in the cases that matter - the conditions under which they coincide are what would settle it · outcome (stated by the tagger, no quotation): absent-stated-about-other Quotation withheld because this record is not cleared for redistribution. H-divergence H-divergence Quotation withheld because this record is not cleared for redistribution. L1 divergence L1 divergence Quotation withheld because this record is not cleared for redistribution. ideal joint hypothesis ideal joint hypothesis Quotation withheld because this record is not cleared for redistribution. adaptability adaptability Quotation withheld because this record is not cleared for redistribution. labeling function labeling function Quotation withheld because this record is not cleared for redistribution. distribution divergence distribution divergence Quotation withheld because this record is not cleared for redistribution. VC dimension VC dimension Quotation withheld because this record is not cleared for redistribution. source domain source domain Quotation withheld because this record is not cleared for redistribution. target domain target domain Quotation withheld because this record is not cleared for redistribution. hypothesis classes hypothesis classes Quotation withheld because this record is not cleared for redistribution. divergence between domains - AS L1: a supremum over all measurable subsets, which cannot be accurately estimated from finite samples and unnecessarily inflates the bound Quotation withheld because this record is not cleared for redistribution. divergence between domains - AS H-DIVERGENCE: a supremum over the hypothesis class only, estimable from finite unlabeled samples and never larger than L1 Quotation withheld because this record is not cleared for redistribution. the domain difference - THE PART MEASURED: the divergence between the two marginal distributions Quotation withheld because this record is not cleared for redistribution. the domain difference - THE PART ASSUMED AWAY: the difference between the two labeling functions, expected to be small in practice and not measured Quotation withheld because this record is not cleared for redistribution. lambda - AS A TERM IN THE BOUND: the combined error of the hypothesis that minimises source plus target error Quotation withheld because this record is not cleared for redistribution. lambda - AS AN ASSUMPTION: adaptability itself. If it is large there is no classifier good on both domains and no hope from source training; if small, the H-divergence carries the bound - and which of the two holds cannot be checked without labelled target data Quotation withheld because this record is not cleared for redistribution. critiques → the use of L1 and Kullback-Leibler divergence in domain adaptation, on the grounds that neither can be accurately estimated from finite samples in a distribution-free setting Quotation withheld because this record is not cleared for redistribution. extends → Kifer et al. 2004's divergence for change detection, carried into a learning bound Quotation withheld because this record is not cleared for redistribution. distinguishes-from → Crammer et al. 2008, whose bound is less general and does not handle positive H-delta-H divergence or non-uniform weighting Quotation withheld because this record is not cleared for redistribution. qualifies → its own bound, by stating that it is informative only when the ideal joint hypothesis has small error - a quantity the setting cannot supply Quotation withheld because this record is not cleared for redistribution. H-divergence —distinct-from→ L1 divergence H-divergence L1 divergence differs in: what set the supremum runs over, and therefore whether the distance can be estimated · the relation itself is separately evidenced · strength of the relation: 0.95 · differs in (stated by the tagger, no quotation): what set the supremum runs over, and therefore whether the distance can be estimated · the relation stated: the H-divergence is estimable from finite samples for classes of finite VC dimension and is never larger than the L1 divergence, because its supremum runs over the hypothesis class rather than over all measurable subsets Quotation withheld because this record is not cleared for redistribution. labeling function —distinct-from→ distribution divergence labeling function distribution divergence differs in: which of the two ways a domain can shift the theory actually measures · the relation itself is separately evidenced · strength of the relation: 0.95 · differs in (stated by the tagger, no quotation): which of the two ways a domain can shift the theory actually measures · the relation stated: two domains can differ in their labeling functions or in their distributions; the labeling difference is expected to be small in practice and the paper measures the other one Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span sets the labeling-function difference aside to focus on distribution divergence; distinct-from is loose. Quotation withheld because this record is not cleared for redistribution. A classifier-induced divergence estimable from unlabeled samples, with the empirical source error, characterizes a source-trained classifier's target error, assuming some hypothesis performs well in both domains. A classifier-induced divergence estimable from unlabeled samples, with the empirical source error, characterizes a source-trained classifier's target error, assuming some hypothesis performs well in both domains. Quotation withheld because this record is not cleared for redistribution. when the ideal joint hypothesis has large error, no classifier performs well on both domains and source training cannot help when the ideal joint hypothesis has large error, no classifier performs well on both domains and source training cannot help Quotation withheld because this record is not cleared for redistribution. the H-divergence is computed by training a classifier to distinguish source from target instances and reading it off the error the H-divergence is computed by training a classifier to distinguish source from target instances and reading it off the error Quotation withheld because this record is not cleared for redistribution. the H-divergence is never larger than the L1 divergence and is generally smaller when H has finite VC dimension the H-divergence is never larger than the L1 divergence and is generally smaller when H has finite VC dimension Quotation withheld because this record is not cleared for redistribution. the difference between the two domains' labeling functions is assumed small rather than measured the difference between the two domains' labeling functions is assumed small rather than measured Quotation withheld because this record is not cleared for redistribution. computing the H-divergence exactly is intractable for most hypothesis classes; convex surrogates are used to approximate it computing the H-divergence exactly is intractable for most hypothesis classes; convex surrogates are used to approximate it Quotation withheld because this record is not cleared for redistribution. with enough labelled target data, target-only training is always preferred no matter how much source data is available with enough labelled target data, target-only training is always preferred no matter how much source data is available Quotation withheld because this record is not cleared for redistribution. a divergence whose supremum runs over the hypothesis class rather than over all measurable subsets - estimable from finite unlabeled samples when H has finite VC dimension, and never larger than L1 H-divergence Quotation withheld because this record is not cleared for redistribution. what the ideal joint hypothesis embodies - whether any single classifier can perform well on both domains at once adaptability Quotation withheld because this record is not cleared for redistribution. VC dimension VC dimension Quotation withheld because this record is not cleared for redistribution. distribution divergence distribution divergence Quotation withheld because this record is not cleared for redistribution. labeling function labeling function Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1007/s10994-009-5152-4.