Semi-Supervised Learning: A Few Labels, Many Unlabeled Examples

A small labeled set anchors the task while a larger unlabeled pool provides information about input structure. The labeled anchor defines the classes, fixes what the target means, and supplies the evaluation everything else is judged against. The unlabeled pool contributes only what the inputs look like. Between them sits an assumption — that nearby, consistent, or low-density structure in that pool relates to the target — and a protected validation set whose job is to find out whether it did.

The anchor is the part the whole arrangement treats as solid, so it is worth seeing one that was not. ChestX-ray8 is 108,948 frontal chest X-rays from 32,717 patients, collected between 1992 and 2015 and released by the NIH Clinical Center in 2017. A later release, ChestXray14, expanded it to 14 classes and 112,120 images. The disease labels were not drawn by radiologists looking at the images. They were text-mined from free-text reports by NLP, and the authors scored that mining at precision 0.90, recall 0.91 and F1 0.90 against the separate OpenI dataset. Only 983 of the images carry expert-drawn bounding boxes, 1,600 boxes in total.

Then a board-certified radiologist sat down with about 700 of those images and looked at them. Luke Oakden-Rayner reported what he saw in Academic Radiology in 2020: “The ChestXray14 labels did not accurately reflect the visual content of the images, with positive predictive values mostly between 10% and 30% lower than the values presented in the original documentation.” Emphysema came out at 14% under a permissive rule and 10% under a clinical one. Pneumothorax looked healthy at 90%. Then the images containing chest drains — the treatment, not the untreated finding — were excluded, and it was 60%. He also found that 13,302 patients, 43% of them, contributed more than one study, and that those patients account for 84% of the images.

Every method in this lesson would have taken those labels as the definition of the task.

The goal remains a supervised task, such as classification or regression, and unlabeled examples influence representation, boundaries, or pseudo-targets while labeled examples define the task.

This differs from unsupervised exploration, where no immediate target is supplied. It also differs from self-supervised pretraining, where the training targets are constructed from the data itself before a downstream task is chosen.

Pseudo-labeling turns confident predictions into temporary targets. Consistency training asks for stable predictions under perturbations that are supposed to preserve meaning. Graph and neighborhood methods propagate information between similar examples. Each mechanism relies on an assumption, and those assumptions may fail in a mismatched or ambiguous pool. The mechanisms are also separable enough to measure, which is the useful part: put two of them in one system and you can see what each was buying.

FixMatch is that experiment. Google Research took a pseudo-label from a weakly augmented copy of an unlabeled image, kept it only when the model was confident about it, and then required the model to reproduce that label on a strongly augmented copy of the same image. Pseudo-labeling and consistency training, doing one job. The 2020 paper reports what that bought: “Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 – just 4 labels per class.”

The size of the contribution is clearest in someone else’s code. The FlexMatch team, at Tokyo Institute of Technology and Microsoft, re-ran both methods in 2021 inside their own TorchSSL codebase. FixMatch: 4.86% ± 0.05 error with 250 CIFAR-10 labels, 7.47% ± 0.28 with 40. Fully supervised training on all 50,000 labels, in the same codebase: 4.62% ± 0.05. Two hundred and fifty labels come within a quarter of a point of fifty thousand. Plain pseudo-labeling, same codebase, same 250 labels: 46.49% ± 2.20, and 74.61% ± 0.26 at 40 labels. The confidence filter and the perturbation are not decoration on the idea of self-training. On this benchmark they are most of it.

A hospital's unlabeled pool is not curated, and one benchmark uses it the way hospitals actually hold it. Heart2Heart, built at Tufts University and Tufts Medical Center in 2023, trains echocardiogram view classifiers on 56 labelled studies from TMED-2 plus a genuinely uncurated unlabeled set of 353,500 images from 5,486 routine patient-studies collected in Boston. Nobody filtered that pool to the labelled task first, because in a hospital nobody does.

The paper that introduced it, Fix-A-Step, is blunt about what happens next: “Unfortunately, modern deep SSL often makes accuracy worse when given uncurated unlabeled data.”

Compare populations, strengthen the anchor, state the assumption you are relying on, track the pseudo-labels, run the ablations, inspect the amplified errors. Step five is the one that gets skipped, and there is now a tally of what it finds when it is not.

USB, the Unified Semi-supervised Learning Benchmark, evaluated 14 SSL algorithms on 15 tasks across vision, language and audio in 2022. Twenty-two authors built it, at Microsoft Research Asia, Westlake University, Carnegie Mellon and several other institutions. Just as importantly, they made the comparison affordable. Evaluating one algorithm fell from about 335 GPU days under TorchSSL's 5 vision tasks to 39 GPU days on a single NVIDIA V100. Microsoft states the same reduction as 7,000 GPU hours to 900.

Table 9 of that paper counts how often each algorithm was worse than plain supervised training on the labelled data alone. The Π-Model: worse in 9 of 10 NLP settings and 7 of 9 audio settings. Pseudo-Labeling: worse in 7 of 10 NLP settings. Mean Teacher: 3 of 9 vision settings and 6 of 9 audio settings. MixMatch: 4 of 9 vision settings. These are published methods, run by people trying to make them work. The paper's own summary, in its robustness section, is one line: “SSL algorithms cannot always outperform supervised training especially when labeled data is scarce.” Of its medical dataset, TissueMNIST, it adds that “the biased pseudo-labels might produce a destructive effect that impedes training”.

The labelled-only baseline is therefore not a formality to be reported after the decision. On this evidence it is a live candidate, and in several modalities it is the favourite.

A large unlabeled pool from another language, device, market, or time period may distort the decision boundary, and even in-domain data can overrepresent ordinary cases while adding little information about rare classes.

This is not a defect of deep learning, and the demonstration predates deep SSL by seventeen years. Cozman and Cohen made it in 2002, in one line: “We show that unlabeled data can degrade the performance of a classifier when there are discrepancies between modeling assumptions used to build the classifier and the actual model that generates the data; our analysis of this situation explains several seemingly disparate results in the literature.”

Their worked example is a Naive Bayes classifier on two Gaussian features whose true dependence the model does not represent. 50 labelled records give about 7.8% classification error. 100 labelled records, about 7.2%. Adding 100 unlabelled records raises it to about 8.2%. Adding 9,800 more raises it to about 16.5% — more than twice the error of the 50-label starting point, produced entirely by data that cost nothing to collect. Each plotted point averages 100 trials, tested on 10,000 labelled records drawn from the correct model. The unlabelled data were not corrupt. The model was wrong, and more data made it more precisely wrong.

The modern version was measured in 2018, by five authors who reimplemented the major methods against one shared model so that the comparison meant something. On mismatch, they classified CIFAR-10's six animal classes from 2,400 labels and varied how many of the four unlabelled classes were among the labelled six. They found that “adding unlabeled data from a mismatched set of classes can actually hurt performance compared to not using any unlabeled data at all.”

Then the alternative nobody was running. With 4,000 CIFAR-10 labels the best semi-supervised method in their reimplementation reached 13.13% error. Simply fine-tuning an ImageNet-pretrained model of the same architecture reached 12.09%, and 12.91% even after the ImageNet classes resembling CIFAR-10 were removed.

Report the marginal value of the unlabeled pool by slice. Before paying for the pool, price what it is competing against.