A survey on semi-supervised learning

A survey on semi-supervised learning Jesper E. van Engelen Holger H. Hoos

Tag layer of the DHSS Meta-Tagging Project. The article text is reproduced from the staged copy named in sourceDesc and is not altered; the tags are held separately in standOff so that overlapping and nested spans are preserved.

A survey on semi-supervised learning 2019 Machine Learning 10.1007/s10994-019-05855-6 engelen2019-a-survey-on-semi-supervised-learni engelen2019-a-survey-on-semi-supervised-learni.txt, the project's staged copy 4d1e90293a2c88dc14f2fe75d881c61f8307aff9 cc-by artificial-intelligence

the paper's own words

the tag layer; a sentence written by the tagger about the paper, always anchored to a quotation but not itself a quotation

0 spans located by searching the stored text for the quote's exact words, allowing any whitespace between them semi-supervised classification - using unlabelled data to beat a learner trained on the labelled data alone - and the assumptions that have to hold for it to be possible at all Quotation withheld because this record is not cleared for redistribution. what the field's methods assume, how those assumptions relate to one another, and when unlabelled data actually helps Quotation withheld because this record is not cleared for redistribution. a taxonomy of semi-supervised classifiers, and the argument that the field's three named assumptions are one assumption under three definitions of 'similar' Quotation withheld because this record is not cleared for redistribution. critique Quotation withheld because this record is not cleared for redistribution. unlabelled data carrying information about the label distribution - when it does, and the fact that nobody can tell in advance whether it does Quotation withheld because this record is not cleared for redistribution. review review Quotation withheld because this record is not cleared for redistribution. THE CATEGORY IS DECIDED BY A CHOICE THE PRACTITIONER USUALLY DOES NOT KNOW THEY ARE MAKING: `The concept of similarity we choose, often implicitly, dictates what constitutes a cluster` Quotation withheld because this record is not cleared for redistribution. and the field's three named assumptions differ only in what they mean by `close` - smoothness says near in input space, low-density says in the same dense region, manifold says on the same low-dimensional surface Quotation withheld because this record is not cleared for redistribution. so all three are instances of the cluster assumption, which is itself the necessary condition for the whole enterprise: that p(x) carries information about p(y|x) Quotation withheld because this record is not cleared for redistribution. A TAXONOMY PAPER THAT DIAGNOSES WHAT IS WRONG WITH TAXONOMIES: `Existing categorizations ... are typically relatively flat, thereby failing to capture similarities between different groups of methods. Furthermore, the categorizations are often fine-tuned towards existing work, making them less suited for the inclusion of new approaches.` Two named failure modes - too shallow to show kinship, and overfitted to the papers that existed when it was drawn. Quotation withheld because this record is not cleared for redistribution. AND THE PROPOSED REPAIR IS STATED AS A DESIGN GOAL, NOT A RESULT: `we propose a new way to represent the spectrum ... we ATTEMPT to group them in a clear, FUTURE-PROOF way`. `attempt` and `future-proof` in one sentence - the second is a property no survey can demonstrate about itself at publication. Quotation withheld because this record is not cleared for redistribution. THE FIELD'S OWN NAME FOR KNOWINGLY USING THE WRONG WORD: `in a slight abuse of notation`, used twice in this paper for two different acts - once to call lists `sets`, once to reassign X and x to random variables. Mathematics has a standard phrase for declaring a deliberate terminological error and continuing, and this survey uses it in a footnote and in body text without further comment. Quotation withheld because this record is not cleared for redistribution. THE NECESSARY CONDITION CANNOT BE CHECKED. `if the data points ... cannot be meaningfully clustered, it is impossible for a semi-supervised learning method to improve on a supervised learning method` - and whether they can is exactly what cannot be established in advance Quotation withheld because this record is not cleared for redistribution. stated twice over, at both levels: it is difficult to DEFINE the conditions under which an algorithm works, and `rarely straightforward to evaluate to what extent these conditions are satisfied` Quotation withheld because this record is not cleared for redistribution. IT IS IMPOSSIBLE TO GUARANTEE THAT UNLABELLED DATA WILL NOT DEGRADE PERFORMANCE - the method can do worse than ignoring the extra data, and no test tells you beforehand Quotation withheld because this record is not cleared for redistribution. and no method exists to choose the algorithm a priori, exactly as in supervised learning - so the practitioner picks the assumption and finds out afterwards Quotation withheld because this record is not cleared for redistribution. THE CENTRAL CLAIM ABOUT THE TAXONOMY IS UNFALSIFIABLE AT TIME OF WRITING: `allowing researchers and practitioners alike to gain insight into the way semi-supervised learning methods relate to each other`, built in a `future-proof way`. The paper's stated advantage over prior categorizations is that new methods will fit it - a claim only later work can test, and the same claim the superseded categorizations could have made. Quotation withheld because this record is not cleared for redistribution. the matching problem - find a subset of edges sharing no vertices - is taken from graph theory and used to pair unlabelled samples in a semi-supervised learner Quotation withheld because this record is not cleared for redistribution. the published empirical comparisons are surveyed rather than a single one cited - eight data sets in one study, several neural methods on two image problems in another eliminates: a field-level claim resting on the studies that happened to work; the answer reported is that NO ALGORITHM UNIFORMLY OUTPERFORMED THE OTHERS and relative performance varied with the amount of unlabelled data · outcome: failed Quotation withheld because this record is not cleared for redistribution. where degradation was observed, the condition producing it is isolated - a mismatch between the classes in the labelled and the unlabelled data eliminates: degradation being treated as unexplained noise; here it has a named cause that a practitioner can check for · outcome: held Quotation withheld because this record is not cleared for redistribution. whether the conditions for a given method to work can be evaluated on a given dataset eliminates: nothing - it is 'rarely straightforward to evaluate to what extent these conditions are satisfied', which is why the necessary condition is not a test · outcome: failed Quotation withheld because this record is not cleared for redistribution. cluster assumption cluster assumption Quotation withheld because this record is not cleared for redistribution. smoothness smoothness Quotation withheld because this record is not cleared for redistribution. low-density low-density Quotation withheld because this record is not cleared for redistribution. manifold manifold Quotation withheld because this record is not cleared for redistribution. local similarity measure local similarity measure Quotation withheld because this record is not cleared for redistribution. taxonomy taxonomy Quotation withheld because this record is not cleared for redistribution. inductive inductive about this quote: THE TERM IS NOT IN THE QUOTE - the quote's former and latter terms are inductive and transductive methods, named in the sentence just before it ('between inductive and transductive methods'). That sentence is not quoted because reference-list text is spliced into its citation. Quotation withheld because this record is not cleared for redistribution. transductive transductive about this quote: THE TERM IS NOT IN THE QUOTE - the quote's former and latter terms are inductive and transductive methods, named in the sentence just before it ('between inductive and transductive methods'). That sentence is not quoted because reference-list text is spliced into its citation. Quotation withheld because this record is not cleared for redistribution. self-training self-training Quotation withheld because this record is not cleared for redistribution. self-learning self-learning Quotation withheld because this record is not cleared for redistribution. sets sets Quotation withheld because this record is not cleared for redistribution. margin margin Quotation withheld because this record is not cleared for redistribution. intrinsically semi-supervised intrinsically semi-supervised Quotation withheld because this record is not cleared for redistribution. Transductive graph-based methods Transductive graph-based methods Quotation withheld because this record is not cleared for redistribution. graph construction graph construction Quotation withheld because this record is not cleared for redistribution. graph weighting graph weighting Quotation withheld because this record is not cleared for redistribution. pseudo-labelled data pseudo-labelled data Quotation withheld because this record is not cleared for redistribution. Semi-supervised learning Semi-supervised learning Quotation withheld because this record is not cleared for redistribution. wrapper methods wrapper methods Quotation withheld because this record is not cleared for redistribution. unsupervised preprocessing methods unsupervised preprocessing methods Quotation withheld because this record is not cleared for redistribution. co-training co-training Quotation withheld because this record is not cleared for redistribution. pseudo-labelled boosting methods pseudo-labelled boosting methods Quotation withheld because this record is not cleared for redistribution. inference inference Quotation withheld because this record is not cleared for redistribution. two alternating steps of _training two alternating steps of _training Quotation withheld because this record is not cleared for redistribution. pseudo-labelling pseudo-labelling Quotation withheld because this record is not cleared for redistribution. co-training → self-training The paper says co-training extends self-training to multiple classifiers. Extension is not is-a, part-of, same-as, distinct-from, contrasted-with, operationalised-as, causes, or renamed-across-fields; the concept-edge schema has no extends type. Quotation withheld because this record is not cleared for redistribution. `similar` - THE LOAD-BEARING WORD OF THE WHOLE FIELD AND IT IS A CHOICE. Smoothness, low-density and manifold are three definitions of it, and the survey's contribution is showing they are the same assumption differently parameterised. WHICH ONE YOU PICKED IS USUALLY IMPLICIT Quotation withheld because this record is not cleared for redistribution. the cluster assumption - AS RE-READ HERE: not one assumption among several but the genus of which the other three are species, and equivalent to the necessary condition that p(x) carries information about p(y|x) Quotation withheld because this record is not cleared for redistribution. `semi-supervised learning works` - AS THE SURVEY IS FORCED TO QUALIFY IT: it can help, it can do nothing, and it can DEGRADE performance, with no way to know which in advance. The literature's record of the third case is described as UNDER-REPORTED Quotation withheld because this record is not cleared for redistribution. `sets` - A DECLARED TYPE ERROR, KEPT ON PURPOSE: `the collections of data points referred to here are technically lists. However, following common usage, in this survey, we refer to them as ‘sets’ and, in a slight abuse of notation, apply standard set-theoretic concepts to them.` The authors know the word is wrong, say so in a footnote, and use it anyway because the field does. Quotation withheld because this record is not cleared for redistribution. `abuse of notation` IS USED TWICE IN THIS PAPER FOR TWO DIFFERENT ACTS. The second is symbol reuse: `we use X and x to denote random variables and their realizations, respectively, rather than data points` - the same symbols carry a new meaning from that point on. Quotation withheld because this record is not cleared for redistribution. `inductive` / `transductive` - the split is defined by WHAT THE METHOD RETURNS, not by how it works: `the former attempt to find a classification model, whereas the latter are solely concerned with obtaining label predictions for the given unlabelled data points`. Quotation withheld because this record is not cleared for redistribution. 'margin' - ONE TERM, TWO DIFFERENT KINDS OF OBJECT, AND THE SURVEY RECORDS BOTH WITHOUT CHOOSING: 'the _margin_, which is defined as the distance between the decision boundary and the data points closest to it. The term is also commonly used to describe the area extruding from the decision boundary in which no data points lie.' A SCALAR AND A REGION. Not two readings of one quantity - two different types, so 'maximise the margin' and 'lies within the margin' are grammatical with different referents, and only one of them can be a number. This record already carries the survey's declared type error over 'sets', kept on purpose, and its two senses of 'abuse of notation'; this is the same survey's third type-level observation, and the only one it does not flag as its own. Quotation withheld because this record is not cleared for redistribution. `pseudo-labelled data` - A NAME WHOSE PREFIX CARRIES THE ONLY WARNING. `Quotation withheld` `pseudo-` is doing all the epistemic work: the data is treated exactly like labelled data by the algorithm, and only the name records that the labels are the model's own guesses. Drop the prefix in a citation and the caution is gone with it. Quotation withheld because this record is not cleared for redistribution. Semi-supervised learning - a machine-learning branch that performs tasks using both labelled and unlabelled data Quotation withheld because this record is not cleared for redistribution. Wrapper methods - pseudo-label unlabelled data externally while the supervised learner remains unaware of label provenance Quotation withheld because this record is not cleared for redistribution. Unsupervised preprocessing methods - use unlabelled data for features, clustering, or initial parameters but give the classifier only original labels Quotation withheld because this record is not cleared for redistribution. Intrinsically semi-supervised - no intermediate step and no supervised base learner Quotation withheld because this record is not cleared for redistribution. Co-training - self-training extended to multiple classifiers that learn from one another's most confident predictions Quotation withheld because this record is not cleared for redistribution. Pseudo-labelled boosting methods - sequential ensembles whose classifiers use labels plus earlier classifiers' confident unlabelled predictions Quotation withheld because this record is not cleared for redistribution. Inference - the final transductive graph-based phase, where the graph assigns labels to unlabelled data points Quotation withheld because this record is not cleared for redistribution. Training in wrapper methods - fitting classifiers on labelled and possibly prior pseudo-labelled data Quotation withheld because this record is not cleared for redistribution. Pseudo-labelling in wrapper methods - turning confident predictions for unlabelled objects into labels for the next iteration Quotation withheld because this record is not cleared for redistribution. critiques → the presentation of smoothness, low-density and manifold as separate assumptions, when they are one assumption under three definitions of similarity Quotation withheld because this record is not cleared for redistribution. critiques → the semi-supervised literature's record of when its methods fail, whose prevalence is described as likely under-reported Quotation withheld because this record is not cleared for redistribution. supports → Oliver et al.'s finding that degradation appeared only under a class mismatch between the labelled and unlabelled sets Quotation withheld because this record is not cleared for redistribution. smoothness —is-a→ cluster assumption smoothness cluster assumption Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but endpoint by anaphora: 'the semi-supervised learning assumptions' = the smoothness, low-density and manifold assumptions of the previous sentence; is-a is loose. Quotation withheld because this record is not cleared for redistribution. manifold —is-a→ cluster assumption manifold cluster assumption Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span introduces the cluster assumption as an additional assumption beside manifolds; is-a is loose. Quotation withheld because this record is not cleared for redistribution. self-training —same-as→ self-learning self-training self-learning the relation itself is separately evidenced · strength of the relation: 0.9 · note: Given parenthetically and without preference, in a sentence that cites a survey whose own title uses a THIRD name - 'Self-labeled techniques for semi-supervised learning…'. Three names for one method inside one sentence, and the paper treats the multiplicity as unremarkable. · the relation stated: one method, two names, offered parenthetically and without preference Quotation withheld because this record is not cleared for redistribution. inductive —distinct-from→ transductive inductive transductive the relation itself is separately evidenced · strength of the relation: 0.95 · note: The top-level split of the taxonomy, and the differentiating property is the OUTPUT TYPE - a model versus a label vector. · the relation stated: the taxonomy's top-level split, and the differentiating property is the OUTPUT TYPE - a model versus a label vector Quotation withheld because this record is not cleared for redistribution. graph construction —part-of→ Transductive graph-based methods graph construction Transductive graph-based methods the relation itself is separately evidenced · strength of the relation: 0.9 · note: 'consist of' is in the containment cue list, so this edge is cue-tested. Weight 0.9 rather than 1.0 - 'GENERALLY consist of' is the survey hedging across the family it is describing. Quotation withheld because this record is not cleared for redistribution. graph weighting —part-of→ Transductive graph-based methods graph weighting Transductive graph-based methods the relation itself is separately evidenced · strength of the relation: 0.9 · note: Second of the three steps, same sentence, same hedge. Quotation withheld because this record is not cleared for redistribution. inference —part-of→ Transductive graph-based methods inference Transductive graph-based methods the relation itself is separately evidenced · strength of the relation: 0.9 · note: The survey says these methods generally have the three steps, so the paper-level hedge is 0.9. Quotation withheld because this record is not cleared for redistribution. pseudo-labelling —part-of→ wrapper methods pseudo-labelling wrapper methods the relation itself is separately evidenced · strength of the relation: 0.9 · note: The procedure usually has these two alternating steps; weight 0.9 preserves that hedge. Audit 2026-09-28: DOUBTFUL - read against its full span and kept, but the span writes 'The procedure ... pseudo-labelling step'; 'wrapper methods' is named only in the preceding text. Quotation withheld because this record is not cleared for redistribution. unsupervised preprocessing methods —distinct-from→ wrapper methods unsupervised preprocessing methods wrapper methods the relation itself is separately evidenced · strength of the relation: 1 · note: The paper explicitly contrasts what the supervised classifier receives under the two method families. Quotation withheld because this record is not cleared for redistribution. the concept of similarity chosen, often implicitly, dictates what counts as a cluster the concept of similarity chosen, often implicitly, dictates what counts as a cluster Quotation withheld because this record is not cleared for redistribution. THREE NAMES, ONE ASSUMPTION: smoothness, low-density and manifold boil down to different definitions of the similarity between points THREE NAMES, ONE ASSUMPTION: smoothness, low-density and manifold boil down to different definitions of the similarity between points Quotation withheld because this record is not cleared for redistribution. if the data cannot be meaningfully clustered, semi-supervised learning cannot improve on supervised learning if the data cannot be meaningfully clustered, semi-supervised learning cannot improve on supervised learning Quotation withheld because this record is not cleared for redistribution. it is impossible to guarantee that adding unlabelled data will not degrade performance it is impossible to guarantee that adding unlabelled data will not degrade performance Quotation withheld because this record is not cleared for redistribution. no algorithm uniformly outperformed the others, and relative performance varied with the amount of unlabelled data no algorithm uniformly outperformed the others, and relative performance varied with the amount of unlabelled data Quotation withheld because this record is not cleared for redistribution. where degradation was observed it had a named condition - a mismatch between the classes present in the labelled data and those in the unlabelled data where degradation was observed it had a named condition - a mismatch between the classes present in the labelled data and those in the unlabelled data Quotation withheld because this record is not cleared for redistribution. that similar points tend to belong to the same group - and the survey argues the field's other assumptions are more specific instances of it cluster assumption Quotation withheld because this record is not cleared for redistribution. similarity similarity Quotation withheld because this record is not cleared for redistribution. unlabelled data unlabelled data Quotation withheld because this record is not cleared for redistribution.

The full text of this article is not redistributable, so it is not included here. The evidence sentences are withheld with it; the tag layer and its grounding status remain. The article itself is at https://doi.org/10.1007/s10994-019-05855-6.