Semi-Supervised Regression with Heteroscedastic Pseudo-Labels

AI summary

Overview

Research area: Semi-supervised learning, specifically semi-supervised regression (SSR), with a focus on uncertainty-aware pseudo-labeling and bi-level optimization.

Technical level: Advanced. The paper assumes familiarity with pseudo-labeling, maximum-likelihood estimation of heteroscedastic noise, bilevel/meta-learning optimization, and gradient-based approximations of second-order derivatives.

Scope in one sentence: The paper proposes a bi-level framework in which a lightweight "uncertainty-learner" network assigns a per-sample log-variance to each pseudo-label, so that unreliable pseudo-labels are automatically downweighted during semi-supervised regression training.

What This Paper Is About

Pseudo-labeling works well in semi-supervised classification, where pseudo-labels are discrete and can be filtered by confidence, but it is much harder to apply to semi-supervised regression, where outputs are continuous and pseudo-label errors vary in size from sample to sample. The authors argue that this error variability is heteroscedastic (it depends on the input), so a single global noise level or a naive joint training of a variance network cannot tell the difference between a "hard but correct" sample and an "easy but incorrect" one, both of which produce a large squared error. The goal is to learn which pseudo-labels to trust and by how much, using the labeled data itself as the signal for what "trustworthy" means.

Key Contributions

1. A bi-level SSR framework with an explicit uncertainty-learner. The framework separates the regression network update (inner loop) from the uncertainty-learner update (outer loop), with the uncertainty-learner optimized to improve the updated model's performance on a separate batch of labeled data.

2. A gradient-alignment theoretical result. Theorem 1 shows that the bi-level problem is equivalent to minimizing the negative inner product between the inner-loop and outer-loop gradients with respect to the regression parameters, giving a principled account of why the method resists overfitting to incorrect pseudo-labels.

3. Heteroscedastic pseudo-label modeling. Each pseudo-label is modeled as a Gaussian draw around the model prediction with an input-dependent variance, and the unsupervised loss is written as the negative log-likelihood of that observation (a precision-weighted squared error plus a log-variance penalty), which reduces to standard MSE when the variance is fixed at 1.

4. Empirical validation across three benchmarks and three label ratios. The method is evaluated on UTKFace, IMDB-WIKI, and STS-B at 5%, 10%, and 20% labeled data, with six runs per setting.

Main Findings

• Consistent gains on age estimation (UTKFace). At 5% labeled data the method reaches MAE 5.639 ± 0.035 and R² 0.523 ± 0.009, beating the second-best RankUp by 1.4% in MAE and 5.7% in R². Compared with the Supervised lower bound (MAE 6.135 ± 0.046, R² 0.454 ± 0.008), it cuts MAE by up to 8.1% and raises R² by up to 15.2%.

• Strongest results on IMDB-WIKI. At 5% labeled data the method achieves MAE 9.177 ± 0.061 and R² 0.664 ± 0.003, improving over the second-best Mean Teacher (MAE 9.492 ± 0.051, R² 0.647 ± 0.002) by about 3.3% in MAE and 2.6% in R². At 20% labeled data it reaches MAE 8.166 ± 0.071 and R² 0.712 ± 0.002, approaching the Fully-Supervised upper bound of 7.974 ± 0.043 and 0.724 ± 0.002.

• Largest relative gains on text similarity (STS-B) when labels are scarce. At 5% labeled data, MSE is 1.540 ± 0.006 versus 1.606 ± 0.057 for the second-best SSDKL (a 4.1% reduction), and R² is 0.270 ± 0.003 versus 0.239 ± 0.027 (13.0% higher). At 20% labeled data the method (MSE 1.246 ± 0.015, R² 0.409 ± 0.008) is comparable to SSDKL (MSE 1.211 ± 0.031, R² 0.426 ± 0.015).

• Gains shrink as labels increase. On UTKFace, the authors state that at 20% labeled data the improvement over UCVME and RankUp is slightly lower, which they attribute to stronger supervision improving pseudo-label quality and stability.

• Ablation shows both components are necessary. On IMDB-WIKI at 5% labeled data, the baseline (fixed σ² = 1) gives MAE 9.512 and R² 0.651; adding the uncertainty-learner alone degrades this to MAE 9.914 and R² 0.630; adding bi-level optimization as well gives MAE 9.177 and R² 0.664. At 10% the same ordering is 8.864/0.683, 9.562/0.651, and 8.539/0.695.

• Estimated uncertainty tracks prediction error. On IMDB-WIKI unlabeled samples under age 40 at 10% labeled data, the estimated variance σ² increases with absolute prediction error, and the relationship is not simply linear, which the authors use as evidence that a learned uncertainty-learner is preferable to a fixed rule.

• Better pseudo-labels across demographic subgroups. Comparing pseudo-labels from UCVME and the proposed method against ground truth on UTKFace, the proposed method produces more accurate pseudo-labels globally, including on elder age ranges that are much less represented.

• Low computational overhead. On UTKFace at 10% labeled data, average training time per iteration is 91.6 ms with 5116 MB GPU memory, close to the baseline (83.0 ms, 5099 MB) and much cheaper than UCVME (257.2 ms, 10057 MB) and SimRegMatch (548.1 ms, 7419 MB).

Methodology in Plain English

The authors start from a simple probabilistic idea: a pseudo-label is a noisy version of the true output, and the amount of noise differs from sample to sample. Under a Gaussian assumption, the negative log-likelihood of a pseudo-label is a squared error divided by its variance, plus the log of that variance. Dividing by a large variance shrinks the sample's influence on training, so incorrect pseudo-labels can be automatically suppressed.

The catch is deciding the variances. If you just train a variance network end-to-end with the regression model, it cannot tell "the pseudo-label is right but my model is still wrong" apart from "the pseudo-label is wrong but my model happens to be near the truth" — both look like a large squared error, and both get a large variance, which wrongly suppresses genuinely difficult but valid samples.

To fix this, the authors split training into two nested loops. In the inner loop, the regression network is updated on labeled data plus pseudo-labeled data, with each pseudo-label weighted by the uncertainty-learner's output. In the outer loop, the uncertainty-learner is updated by checking how well the freshly updated regression model performs on a different batch of labeled data — so the uncertainty network is rewarded for producing variance estimates that lead to a model that generalizes to trustworthy labels. The two labeled batches must differ, which prevents the uncertainty-learner from trivially memorizing.

Practically, the uncertainty-learner is a small MLP with a single hidden layer that takes the regression prediction and the pseudo-label as input and outputs a log-variance (log is used instead of the raw variance for numerical stability). Following FixMatch, the pseudo-label comes from a weakly augmented view of the unlabeled input, while the prediction comes from a strongly augmented view. Because the outer-loop gradient involves a second-order derivative through the whole network, the authors approximate it by only unrolling the second-order derivative of the regression head (a single fully connected layer), which keeps training cheap.

The theory (Theorem 1) recasts the whole procedure as maximizing the inner product between the inner-loop and outer-loop gradients with respect to the regression parameters — that is, the uncertainty-learner is trained to make the pseudo-labeled gradient agree with the labeled-data gradient.

Why This Matters

Impact on research. Semi-supervised regression has lagged behind semi-supervised classification because confidence thresholding has no natural analogue for continuous outputs. This work shows that learned, input-dependent uncertainty can serve as that analogue, and the gradient-alignment framing connects bi-level meta-learning to pseudo-label robustness in a way that could transfer to other noisy-label settings.

Real-world applications:

• Medical imaging, where labeled scans are scarce and expensive but unlabeled ones are plentiful.
• Video analysis, where frame-by-frame annotation is labor-intensive.
• Facial age estimation from photographs, one of the paper's own benchmarks.
• Semantic similarity scoring for sentence pairs, the paper's text benchmark and a component of retrieval and evaluation pipelines.

Industry relevance. The method adds only a small MLP and roughly 9 ms per iteration over a plain semi-supervised baseline on the reported hardware, while avoiding the much heavier cost of co-training or multiple-forward-pass uncertainty estimation. That makes it a practical drop-in for production regression pipelines that already have a large pool of unlabeled data and a small labeled set.

Future Directions

• Scaling the second-order approximation. The current implementation only unrolls the regression head; whether unrolling more of the network yields better uncertainty estimates without prohibitive cost is untested here.
• Extending beyond Gaussian heteroscedastic noise. The likelihood assumes Gaussian pseudo-label noise, which may not hold for skewed or heavy-tailed regression targets.
• Broadening the benchmark set. Evaluation covers three datasets (UTKFace, IMDB-WIKI, STS-B) with ResNet-50 and BiLSTM backbones; other modalities and architectures are not reported.
• Clarifying the ablation anomaly. Adding the uncertainty-learner without bi-level optimization degrades performance below the baseline on IMDB-WIKI, which raises the question of how sensitive the method is to the outer-loop batch design and the choice of λ.

Target Audience

Researchers and graduate students working on semi-supervised learning, uncertainty estimation, or meta-learning and bi-level optimization, particularly those who need regression rather than classification. It is also relevant to practitioners in medical imaging, video analysis, and age estimation who have large unlabeled datasets and limited labeled ones, provided they are comfortable with the bilevel training machinery.

Authors’ abstract

Pseudo-labeling is a commonly used paradigm in semi-supervised learning, yet its application to semi-supervised regression (SSR) remains relatively under-explored. Unlike classification, where pseudo-labels are discrete and confidence-based filtering is effective, SSR involves continuous outputs with heteroscedastic noise, making it challenging to assess pseudo-label reliability. As a result, naive pseudo-labeling can lead to error accumulation and overfitting to incorrect labels. To address this, we propose an uncertainty-aware pseudo-labeling framework that dynamically adjusts pseudo-label influence from a bi-level optimization perspective. By jointly minimizing empirical risk over all data and optimizing uncertainty estimates to enhance generalization on labeled data, our method effectively mitigates the impact of unreliable pseudo-labels. We provide theoretical insights and extensive experiments to validate our approach across various benchmark SSR datasets, and the results demonstrate superior robustness and performance compared to existing methods. Our code is available at https://github.com/sxq/Heteroscedastic-Pseudo-Labels.