Semi-Supervised Learning: Pseudo-Labels, Consistency, and Confirmation Bias

A semi-supervised system pairs a small labeled set with a much larger pool that carries no task labels. The unlabeled examples reveal structure in the input. They do not say which distinctions the product should make. The central risk is circularity. The model predicts labels for data it does not understand, then trains on those predictions as though they were evidence. If early mistakes are accepted confidently, the learner reinforces its own blind spots.

That failure is not a worry someone raised. It is the subject of a peer-reviewed paper, published in 2020, whose abstract states the result and the remedy in one sentence: “We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it.”

Two things follow. First, the circularity is measurable. It shows up as overfitting to the model's own wrong labels, not as a vague drift. Second, the countermeasures that worked were structural rather than a better score cut-off: mixup, and a floor on how many genuinely labeled samples appear in every mini-batch, so the trusted signal never vanishes from the gradient. Semi-supervised learning is not supervised learning with more rows. It is a mechanism for deciding when model-generated targets are trustworthy enough to influence training.

The methods differ in what target they generate and what assumptions make that target useful.

FixMatch combines the first two families. A pseudo-label is kept only when a weakly augmented view is predicted confidently. The model then trains to predict that label from a strongly augmented copy. Kihyuk Sohn and colleagues report “94.93% accuracy on CIFAR-10 with 250 labels”. With 40 labels, four per class, it still reached 88.61%.

The third family has a canonical member, and it comes with numbers rather than adjectives. Mean Teacher, published by Antti Tarvainen and Harri Valpola in 2017, averages model weights rather than label predictions. The teacher is a running average of the student's own parameters, not an ensemble of its outputs. Their abstract reports that “Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels.” The same method took CIFAR-10 with 4000 labels from 10.55% to 6.28% error. It took ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

Read that last pair slowly, because it is what the third column is claiming. Change only how the target is generated — average the weights instead of trusting the fast-moving student — and ImageNet 2012 error moves from 35.24% to 9.11% at a fixed 10% label budget. Stabilising the target generator is worth a large fraction of the labels nobody collected. It is also still the same model's bias, averaged. A smoother teacher makes systematic errors more consistent, not less systematic. That is why the column ends where it does, at independent labeled evaluation.

A robust loop tracks both target quality and the amount of unlabeled data admitted.

The loop below is not a whiteboard exercise. It has been run at a scale that makes every step visible. Noisy Student Training, which Google published in 2020, describes the first two steps exactly as the diagram does: “On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images.” The student trained on those 300 million generated targets reached 88.4% top-1 on ImageNet. That is 2.0% above a model trained on 3.5 billion weakly labeled Instagram images.

The refresh step is where the design differs from a naive loop. The student is promoted to teacher and the cycle repeats, and noise is injected into the student rather than into the target generator. Targets are made as clean as possible, and learning from them as hard as possible. The gains that matter for this lesson are not the headline. ImageNet-A top-1 rose from 61.0% to 83.7%. ImageNet-C mean corruption error fell from 45.7 to 28.3. Those are the robustness slices, measured on labels the loop never touched. They are the last step of the diagram, and the only step that can tell you whether the previous five did anything real.

Aggregate accuracy can hide the exact places where generated supervision becomes unsafe.

The first slice is the best documented of the five. Train under an imbalanced class distribution and the accepted pseudo-labels tilt toward the majority classes. Jaehyung Kim and colleagues reported at NeurIPS 2020 that “these SSL algorithms trained under imbalanced class distributions can severely suffer when generalizing to a balanced testing criterion, since they utilize biased pseudo-labels of unlabeled data toward majority classes”. Their remedy, DARP, refines the pseudo-labels by solving a convex optimisation problem. It does not move a threshold. Register that next to the previous section. When the bias sits in the class distribution of the accepted targets, the fix acts on the distribution, and threshold sweeps are working on the wrong object.

The experiment should isolate the value of unlabeled data from extra compute and tuning.

Put the main methods in one codebase and the picture changes. Avital Oliver and colleagues did exactly that, and found “the performance of simple baselines which do not use unlabeled data is often underreported”, and that methods differ in sensitivity to the amount of labeled and unlabeled data. That is what a matched ablation exists to expose. The baseline is the experiment.

Step 2 is where the ablation usually dies, and the reason has a price tag. The Unified SSL Benchmark, published in 2022, covers 15 classification tasks across vision, NLP and audio, and its authors reported what the older way of doing this cost: “on a single NVIDIA V100, only 39 GPU days are required to evaluate FixMatch on 15 tasks in USB while 335 GPU days (279 GPU days on 4 CV datasets except for ImageNet) are needed on 5 CV tasks with TorchSSL.” Three hundred and thirty-five GPU days to evaluate one method on five vision tasks explains, without excusing, why the matched-compute supervised baseline is the run that gets dropped. Starting from pretrained models brought that to 39 GPU days across all 15 tasks. The ablation this section prescribes now has a named, public suite to run in, spanning three modalities rather than the single benchmark domain most headline results come from.

Unlabeled data is not automatically an asset. It may come from a different task, contain duplicated users, violate consent, or encode a changed label policy. In those cases, adding it can worsen both performance and governance.

The task-mismatch case has been quantified, and then contested, which is the more useful thing to know. Lan-Zhe Guo and colleagues reported at ICML 2020 that existing deep SSL methods drop below plain supervised learning once a large share of the pool comes from classes the task never had: “In the experiment of benchmark data, unlike the existing deep SSL methods which are no longer as good as supervised learning in 40% of unseen-class unlabeled data, the new method can still achieve performance gain in more than 60% of unseen-class unlabeled data.” A 40% contamination rate is not exotic in a pool assembled from whatever logs were available.

Then the measurement itself was challenged. Rundong He and colleagues re-examined it at ICLR 2025 and argued that the standard protocol violates the control-of-variables principle, concluding that “unseen classes in unlabeled data do not necessarily impair the performance of SSL models; in fact, under certain conditions, unseen classes may even enhance them”. Two competent groups, opposite readings, one experimental design between them. This lesson's own claim — that the protocol decides the answer — is being demonstrated on the question of whether the pool helps at all.

A strong negative result is valuable. If a supervised baseline matches the semi-supervised method after fair tuning, keep the simpler system. The extra loop is justified only when it improves target evidence without creating an unmanageable feedback process.