Weak-to-Strong Generalization under Distribution Shifts

AI summary

Overview

• Research area: Machine learning alignment and supervision, specifically weak-to-strong (W2S) generalization combined with learning under distribution shift.
• Technical level: Intermediate. The paper assumes familiarity with supervised fine-tuning, ensembles, cross-entropy losses, and preference optimization (DPO), but the core idea is explained conceptually.
• Scope: The paper shows that weak-to-strong generalization breaks down when the weak supervisor is trained on a different distribution from the strong model's fine-tuning data, and proposes RAVEN, a robust framework that dynamically reweights an ensemble of weak models to recover strong-model performance under such shifts.

What This Paper Is About

Weak-to-strong generalization asks whether a small, weak model can supervise a large, strong model well enough that the strong model surpasses its weak supervisor. The authors identify a failure mode the original framing assumes away: when the weak model is trained on a distribution that differs from the strong model's fine-tuning data ("weak and misaligned supervision"), the strong model can end up performing worse than the weak supervisor. The goal is to design a framework that stays effective under these distribution shifts.

Key Contributions

1. A new problem setting. The authors extend the standard W2S generalization formulation to include distribution shift between the source distribution $P_{src}$ and the fine-tuning distribution $P_{tuning}$, formalized through the generalization gap $\Delta R(f) = R_{tuning}(f) - R_{src}(f)$.
2. An empirical demonstration that naive W2S fails under shift. On 11 out of 19 ImageNet-C corruption cases, W2S generalization is infeasible—the strong model performs worse than the weak model, giving negative performance gap recovered (PGR).
3. RAVEN (Robust AdaptiVe wEightiNg). A framework in which the strong model jointly learns its own parameters and trainable weights over an ensemble of weak models, combined with an "easy-sample guided initialization" warm-up phase where weights are fixed to $1/M$.
4. Evaluation across three task families. Image classification (iWildCam, Camelyon17, fMoW, plus ImageNet for in-distribution), text classification (Amazon-Wilds, MedMCQA/MedQA), and preference alignment (HH-RLHF, OpenAI Summarize from Feedback, Human-Like DPO), reporting improvements of 55%, 57%, and 33% respectively over the best alternative baselines, and over 30% on out-of-distribution tasks in the abstract.

Main Findings

• Naive W2S collapses under distribution shift: In the ImageNet to ImageNet-C study, PGR is much lower out-of-distribution (OOD) than in-distribution (InD), and on 11 of 19 corruption cases it is negative, meaning the strong model is worse than its weak supervisor.
• Weak models vary more OOD than InD: Using 20 weak models trained with different random seeds on WILDS benchmark datasets, the authors observe substantially higher performance variance in OOD scenarios than in InD scenarios.
• RAVEN leads on image classification: Average accuracy 51.54 with PGR 16.09 for RAVEN(3), and 53.23 / 19.27 for RAVEN(>>3), versus the best alternative ensemble (Ens(3)) at 49.70 / 10.39 (average accuracy / PGR). In-distribution ImageNet accuracy reaches 67.90 (PGR 66.33) for RAVEN(3) and 68.35 (PGR 68.60) for RAVEN(>>3).
• RAVEN leads on text classification: In the OOD setting, RAVEN improves PGR by 57% and accuracy by 1.4% on average over the best alternative baseline; in the InD setting, RAVEN improves PGR by 27% on average. Bayes underperforms here and can fall below individual weak models.
• RAVEN leads on preference alignment: Average improvements of 2.7% (win rate) and 32.8% (PGR) in the OOD settings, and 1.2% and 25.6% in the InD setting. For Summarization to Human-like, RAVEN reaches 70.37 WR / 50.47 PGR versus Bayes at 68.18 / 41.88.
• Every component helps (ablation): Accumulating ensemble, easy-sample guided initialization, and adaptive weighting moves average accuracy from 46.78 to 51.97 and average PGR from 1.80 to 16.09.
• The scheduling choice matters: Comparing All, Easy, Easy-All, Easy-All+AW, and the full RAVEN schedule, RAVEN achieves the best result (51.97 accuracy, 16.09 PGR), supporting the combination of easy-sample initialization followed by adaptive weighting.
• Robust to weak-model diversity: With different weak architectures (AlexNet, ResNet18, SqueezeNet), RAVEN reaches 46.03 accuracy / 9.65 PGR, an 81% PGR improvement over the best alternative baseline Bayes (5.33). With different data sources (seven AlexNets on camera location for iWildCam, five on hospital for Camelyon17, seven on time for fMoW), RAVEN reaches 51.17 / 10.49, a 24% PGR improvement over Bayes (7.76).
• Automatic identification of the best weak model: Without access to ground-truth labels or any information about weak model performance, RAVEN typically assigns the highest weight to the best-performing weak model on the target distribution, and high correlation is observed between a weak model's target accuracy and its W2S accuracy.
• More weak models help, then saturate: Performance improves as the number of weak models $M$ increases, with gains diminishing after a large number are included.

Methodology in Plain English

The researchers set up a three-part data split: a source distribution on which weak models are trained with ground-truth labels, a fine-tuning distribution on which the strong model is trained with pseudo-labels, and a target distribution used for evaluation. They deliberately make the source and fine-tuning distributions differ.

Instead of using a single weak model, RAVEN trains multiple weak models (differing by random seed, and in some experiments by architecture or training data source) and freezes them. The strong model is a pretrained, frozen backbone with a trainable linear classifier on top. The core mechanism is a set of trainable weights over the weak models, constrained to sum to one, which are updated alternately with the strong model's classifier parameters. Rather than treating all weak models equally, the strong model learns which ones to trust.

To prevent the classifier from shortcutting early in training—collapsing onto whichever weak model resembles its own poorly optimized initial predictions—RAVEN starts with a warm-up on "easy" samples where all weak models agree, holding the weights fixed at $1/M$. Only afterward does adaptive weighting begin across the full dataset. For the generation tasks, the authors introduce a DPO-based objective called DPO-R to integrate this scheme into preference alignment.

Why This Matters

• Research impact: The paper reframes weak-to-strong generalization as a problem that must account for distribution shift, and shows that standard remedies (confidence loss, bootstrapping, Bayesian weakS-to-strong, co-supervised learning) are insufficient. It also connects the W2S literature with the distribution-shift literature, treating the combination as a distinct research problem.
• Real-world applications:
  • Medical imaging: The paper's motivating example is a radiologist misreading scans from a different machine or mislabeling rare diseases outside their usual practice—matching the Camelyon17 hospital-domain shift setup.
  • Clinical text: Using MedMCQA (Indian medical examinations) as source and MedQA (U.S. medical board questions) as target, simulating annotation by a model unfamiliar with a different examination system.
  • Sentiment analysis across reviewers: Amazon-Wilds, where domains correspond to individual reviewers and both domain and subpopulation shifts occur.
  • Preference alignment of language models: Aligning a strong model using feedback from a weaker one, which the authors note could apply to aligning a model like GPT-5 using only GPT-4-level supervision.
• Industry relevance: As frontier models grow more capable, human or weaker-model supervision becomes less reliable on unfamiliar inputs. RAVEN suggests that aggregating multiple imperfect supervisors and letting the strong model learn which to trust can recover robustness without needing ground-truth labels on the target domain.

Future Directions

1. Richer weak-model diversity. The authors note that the weak models used in RAVEN differ mainly by random seeds, and question whether this captures the diversity of human annotators in expertise-heavy fields.
2. Better weighting granularity. Model-wise weights beat the (model, sample)-wise variant in their tests, even though model-wise weights are a strict subset of (model, sample)-wise weights. Improving optimization for per-sample weighting is flagged as a route to a stronger RAVEN.
3. Scaling analysis. The paper references Appendix H.11 as a scaling analysis using diverse (weak, strong) model pairs, implying further exploration of how the approach behaves across larger model combinations.
4. Wider task coverage. Preference alignment was previously studied only narrowly (e.g., slot filling), so extending the evaluation to other generation and alignment tasks remains open.

Target Audience

Researchers and practitioners working on model alignment, scalable oversight, and trustworthy weak supervision—particularly those who fine-tune large models with imperfect labels or on data that differs from the annotation source. It is also relevant to applied teams in healthcare, biology, and other specialized domains where annotation quality degrades on unfamiliar data. Readers should be comfortable with supervised fine-tuning, ensemble methods, and preference optimization to get the most out of the methodological details.

Authors’ abstract

As future superhuman models become increasingly complex, accurately supervising their behavior may exceed human capabilities. Recent works have demonstrated that in such scenarios, weak models can effectively supervise strong models, a phenomenon known as weak-to-strong generalization. However, we find that naive weak-to-strong generalization fails under distribution shifts, often leading to worse performance of the strong model than its weak supervisors. To address this, we propose RAVEN, a robust weak-to-strong generalization framework that dynamically learns the optimal combinations of weak models in addition to parameters of the strong model. We demonstrate the effectiveness of RAVEN on image classification, text classification, and preference alignment tasks. RAVEN outperforms alternative baselines by over 30% on out-of-distribution tasks while matching or surpassing existing methods on in-distribution tasks. Moreover, our results show that RAVEN assigns higher weights to more accurate weak models, demonstrating its ability to automatically identify trustworthy supervision.