Self-Soupervision: Cooking Model Soups without Labels

AI summary

Overview

Research area: Machine learning — model soups, parameter merging, and self-supervised learning (SSL), evaluated on image classification and transfer benchmarks.

Technical level: Advanced. The paper assumes familiarity with fine-tuning, pre-training, linear mode connectivity, k-nearest-neighbor classification, and self-supervised loss families.

Scope: The paper generalizes model soups by replacing the labeled data needed to "prepare" soup ingredients with self-supervised training, and demonstrates this on ImageNet (plus seven shifted ImageNet variants), LAION-C, and the 21-dataset VTAB collection.

What This Paper Is About

Model soups take one pre-trained model (the "stock"), fine-tune it into several models (the "ingredients"), and average their parameters back into a single model (the "soup") that predicts better than any single ingredient at no extra inference cost. Every known soup recipe, however, requires labels for training the ingredients. This paper introduces Self-Souping (styled "Self-Soup ervision"), which lets ingredients be prepared by self-supervised learning on unlabeled data — including data from the target task or from a shift — before the usual supervised fine-tuning and mixing.

Key Contributions

1. A general definition of Self-Soups. The authors formalize a soup as a weighted average over M self-supervised inter-trainings, each followed by N supervised fine-tunings, so that the intermediate training stage requires no labels (Y_i not required). Any soup whose ingredients differ in their self-supervised training runs counts as a Self-Soup.

2. First demonstration that ingredients can differ in their SSL algorithms, not just hyperparameters. They cook soups from MAE (reconstruction), MoCoV3 (instance-contrastive), MMCR and LeJEPA (dimension-contrastive) ingredients, and show linear mode connectivity (LMC) can hold between them — a novel finding.

3. Shift-aware soups. They inter-train on unlabeled corrupted test data (ImageNet-C and LAION-C) and then fine-tune back on the labeled training distribution, gaining +3.5% on ImageNet-C and +7% on LAION-C.

4. Self-Seasoning. A fully unsupervised mixing method that learns the soup mixture coefficients by gradient descent on the entropy of k-nearest-neighbor predictions, removing the need for the few-shot labeled data that standard "seasoning" requires.

The code is released at https://github.com/antofuller/self_soupervision.

Main Findings

• Self-Souping is possible and productive. Inter-training four models for 5 epochs at a learning rate of 1e-5 with MAE, MoCoV3, MMCR and LeJEPA, then fine-tuning each for 10 epochs at 8e-5, produces 4 ingredients. Across 64 uniformly distributed convex combinations evaluated on 8 ImageNet test sets, the best models are always combinations of ingredients and the worst are always the ingredients alone, with the largest gains of +3% on corrupted data (ImageNet-C and LAION-C) and roughly equal-weight mixtures performing best.

• Self-Soups beat supervised soups on corruptions. Uniform mixing of Self-Soup ingredients reaches 32.34% on ImageNet-C and 22.92% on LAION-C, versus 30.91% and 22.05% for the supervised soup and 31.44% and 21.50% for the continued SSL + supervised soup baseline — gains of +0.9% on both benchmarks. Self-Soups also beat the logit-averaging ensemble by 0.9% on ImageNet-C and 2.1% on LAION-C.

• Inter-training on the test distribution is the largest single win. Using even-indexed corrupted samples for MAE inter-training (100K steps, learning rates 1e-5, 2e-5, 3e-5, 4e-5) and odd-indexed samples for testing, the Self-Soup on ImageNet-C reaches 35.72% on ImageNet-C and the Self-Soup on LAION-C reaches 29.48% on LAION-C, relative to 30.91% and 22.05% for the supervised soup on ImageNet-Train: +3.5% and +7%. Inter-training on even-indexed samples used also for testing (the "joint" setting) adds +0.3% on ImageNet-C and +0.8% on LAION-C.

• Gains survive test-time adaptation. With SAR applied, the supervised soup on ImageNet-Train goes from 30.95% to 32.67% on ImageNet-C, while the Self-Soup on ImageNet-C goes from 35.72% to 37.50%. The Self-Soup without TTA already beats the train-distribution soup with TTA, and the two strategies remain complementary.

• Transfer gains on VTAB. Averaged across 336 test sets (21 datasets × 16 clean/corrupted conditions), the Self-Soup with greedy search reaches 67.5% top-1 on clean data (next best 66.2%) and 35.1% under uniform mixing on corrupted data (next best 34.6%). Gains are largest on the 8 natural datasets: +1% over the continued SSL + supervised soup and +1.4% over the supervised-only soup. The method is most robust to blur, weather and digital corruptions (+2%) but handles noise worse (-1% versus the supervised-only soup).

• Self-Souping matches Model Ratatouille without extra labels. Preparing ingredients by varying SSL algorithms on ImageNet-1K yields 79.09% on ImageNet-Val, 32.34% on ImageNet-C and 22.92% on LAION-C, essentially matching Ratatouille's 79.05%, 32.20% and 22.89% — which requires four auxiliary labeled datasets.

• Self-Seasoning is competitive with labeled seasoning. On VTAB natural tasks, both seasoning methods — which learn only the mixture coefficients — outperform full fine-tuning when fewer than 500 training samples are available. The result holds for the natural, specialized, and structured VTAB task groups.

• The approach is not tied to one stock. Using the Franca ViT-B stock instead of the MAE ViT-B pre-trained for 1600 epochs on ImageNet-1K, ingredients trained independently with MAE and MoCoV3 still show LMC and comparable gains.

Methodology in Plain English

The idea rests on a simple two-stage pipeline. First, a single pre-trained "stock" model is copied and each copy is trained further with a different self-supervised objective on different unlabeled data — this is the "inter-training" stage, and because the objectives are self-supervised no labels are needed. Second, each of those models is fine-tuned with labels on the task (or, in the fully unsupervised setting, not fine-tuned at all), and the resulting parameter sets are averaged, either uniformly or via a searched mixture.

To test this, the authors compare three things on the same total search budget: an ordinary supervised soup, a "continued SSL + supervised soup" (SSL inter-training followed by several supervised fine-tunings that all branch from one inter-training), and their Self-Soup (where the fine-tunings branch from many different inter-trainings). They measure accuracy on ImageNet and its shifted variants, and on VTAB plus a corrupted version they build called mini-VTAB-C (1K-sample train and test subsets with 15 corruption types at the highest severity).

For the unsupervised mixing method, Self-Seasoning, they keep the ingredients purely self-supervised, predict via nearest neighbors on the learned representations, and tune the mixture coefficients by gradient descent to minimize the entropy of the neighbor-based predictive distribution — a signal that needs no labels.

Why This Matters

Impact on research. The paper removes a hard constraint from the model-soup literature: labels were previously required to create ingredients. It also broadens the source of ingredient diversity from hyperparameters and labeled datasets to self-supervised losses, algorithms, and unlabeled data distributions, and reports a new case where linear mode connectivity holds. It connects soups to the much larger SSL literature, since "countless" self-supervised algorithms can now be used as ingredient recipes. It also introduces two reusable baselines and a corrupted VTAB variant for future comparison.

Real-world applications:

• Robustness to distribution shift in deployed vision systems, where unlabeled data from the deployment environment can be collected far more cheaply than labels.
• Transfer learning to niche domains (medical, industrial, scientific imaging) that have many images but few annotations.
• Low-label settings, where Self-Seasoning outperforms fine-tuning below roughly 500 training samples.
• Deployment-constrained systems, since the soup is a single model with the same inference cost as the original.

Industry relevance. The paper targets settings where inference and deployment cost are prioritized over development cost, because an averaged model needs no more computation than the original. It also matters where labels are expensive but unlabeled in-domain data is abundant.

Future Directions

• Better mixing rules. The authors note that their Self-Soups supply more ingredients that could be fed to more sophisticated mixing methods such as layer-wise re-weighting (Model Stock) or merging approaches like TIES-Merging and EMR-Merging.
• Combining with multi-task SSL. Because multi-task SSL optimizes several self-supervised losses jointly while Self-Souping runs them independently, mixing multi-task SSL ingredients into a soup is an open, by-definition extension.
• More stocks, algorithms, and modalities. The paper demonstrates a second stock (Franca) but states that the framework admits endless instantiations; only MAE, MoCoV3, MMCR, and LeJEPA are tested here.
• Understanding when noise robustness regresses. Self-Soups are +2% against blur, weather, and digital corruptions but -1% against noise (versus the supervised-only soup), which the paper reports without resolving.

Target Audience

Researchers and practitioners working on model merging, model soups, and self-supervised learning; engineers who need robust vision models under distribution shift without additional labeling budget; and readers interested in low-shot transfer, since the paper reports gains for both large-scale ImageNet robustness and small-sample VTAB transfer.

Authors’ abstract

Model soups are strange and strangely effective combinations of parameters. They take a model (the stock), fine-tune it into multiple models (the ingredients), and then mix their parameters back into one model (the soup) to improve predictions. While all known soups require supervised learning, and optimize the same loss on labeled data, our recipes for Self-Soupervision generalize soups to self-supervised learning (SSL). Our Self-Souping lets us flavor ingredients on new data sources, e.g. from unlabeled data from a task for transfer or from a shift for robustness. We show that Self-Souping on corrupted test data, then fine-tuning back on uncorrupted train data, boosts robustness by +3.5% (ImageNet-C) and +7% (LAION-C). Self-Soupervision also unlocks countless SSL algorithms to cook the diverse ingredients needed for more robust soups. We show for the first time that ingredients can differ in their SSL hyperparameters -- and more surprisingly, in their SSL algorithms. We cook soups of MAE, MoCoV3, MMCR, and LeJEPA ingredients that are more accurate than any single SSL ingredient.