Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations

AI summary

Overview

• Research area: Machine learning; out-of-distribution (OOD) generalization, spurious correlations, and benchmark evaluation for distribution shift (DomainBed and WILDS benchmarks).
• Technical level: Advanced. The paper assumes familiarity with distribution shift, domain generalization, Pearson/Spearman correlation, probit transforms, and gradient-based optimization.
• Scope: The paper argues that the widely reported "accuracy-on-the-line" (AoTL) pattern—a strong positive correlation between in-distribution (ID) and OOD accuracy across models—is an artifact of aggregating heterogeneous OOD examples, and it introduces a method, OODSelect, that finds large OOD subsets where higher ID accuracy predicts lower OOD accuracy.

What This Paper Is About

Benchmarks for OOD generalization typically report that models which do well in-distribution also do well out-of-distribution, a trend called accuracy-on-the-line, which is often read as evidence that spurious correlations are rare. The authors show this impression is produced by aggregating OOD data: when the same OOD set is split into semantically coherent subsets, large portions of it show the opposite pattern, which they name accuracy-on-the-inverse-line (AoTIL). Their goal is to identify such subsets across standard benchmarks and demonstrate that aggregate metrics hide real robustness failures.

Key Contributions

1. The authors demonstrate that state-of-the-art domain generalization benchmarks contain large, semantically meaningful OOD subsets—sometimes over half of the data—where the ID–OOD correlation drops as low as -0.9 Pearson R. The familiar AoTL trend appears only when these subsets are aggregated back with the rest of the data.
2. They show the subsets are not arbitrary. In chest X-ray diagnosis tasks, models that improved overall performance performed worse on patients with pleural conditions and enlarged cardiomediastinum.
3. They propose OODSelect, described as a simple yet effective gradient-based selection procedure for identifying such subsets across datasets, when they exist.
4. They release the identified subsets for benchmarks including DomainBed and WILDS, along with code, to facilitate further research.

Main Findings

• AoTIL exists in standard benchmarks: Using OODSelect, the authors uncover subsets where higher ID accuracy predicts lower OOD accuracy. In one case, reported for Chest X-rays (CXR), this holds for over 70% of the usual OOD dataset.
• Aggregation masks the effect: For CXR No Finding, the full OOD set has a strong positive ID–OOD correlation of 0.86, while an OODSelect subset of 55000 examples (reported as 75% of the full dataset in Table 1 and 77% in Figure 3) has a negative correlation of -0.60.
• Terra Incognita: The full OOD set has a strong positive correlation, but a 30% slice from OODSelect has a notable negative correlation. In Figure 1, with ID data from locations L100, L38, and L43 and OOD data from L46, the original OOD data gives a Pearson correlation of 0.89 across models, while OODSelect finds up to 1000 (approximately 16%) examples with an ID–OOD correlation of -0.77.
• Effect sizes vary by dataset: For PACS, a small OODSelect size making up 8% of the full dataset has a correlation of -0.33; at 60% the correlation is already negligible at 0.01, and it becomes strongly positive as the subset grows, with the full dataset at 0.81. In Table 1, the largest OODSelect subset with AoTIL below -0.3 ranges from 250 examples (6% of the full OOD set) for PACS to 60000 examples (40%) for WILDSCamelyon-H5 and 55000 (75%) for Chest X-rays.
• The selection is not just picking hard examples: The most misclassified examples have an ID–OOD correlation near 0 and rarely invert the correlation the way OODSelect examples do. Random selection consistently preserves a strong positive correlation similar to the full dataset. This indicates OODSelect does not conflate spuriousness with general difficulty such as uniform label noise.
• Direction of correlation flips via relative performance changes: For VLCS, models with relatively low ID accuracy performed better on the OODSelect set than on the full OOD set, while models with relatively high ID accuracy performed better on the full OOD set. On some OODSelect sets (Terra Incognita) some models still perform near or below chance, while on others (WILDSCamelyon) all models are above chance.
• Not driven by outliers: Spearman rank correlation, which is more robust to outliers than Pearson R, leaves the conclusions unchanged.
• Not driven by architecture confounds: When train/test models come from disjoint architecture families (e.g., ResNets versus ViTs), the strength of the correlation changes but the conclusions do not. Sampling from an entirely disjoint population of architectures mitigates the observed strength of spurious correlations learned by model families.
• Subsets are semantically coherent where metadata exists: For CXR, prevalence comparisons between the selected subset and the full OOD pool show statistically significant shifts in sex, race, Pleural Other, Support Devices, and Sex-Ethnicity, assessed via bootstrapping with 1000 resamples. No Finding prevalence was 39% in the OODSelect samples and 38% in the full dataset, while "Pleural Other" prevalence was 61% in the full OOD set and 68% in the OODSelect subset.
• Subsets are consistent: Measured with a normalized Jaccard index, the selected subsets are more consistent across selection sizes than arbitrary selection (the lower bound) would produce, even though each subset is selected independently and nesting is not enforced.
• Vision-language model behavior: Zero-shot VLM performance shows strongly positive correlations between ID accuracy and accuracy on OODSelect examples. The weakest correlations reported are PACS (0.78), VLCS (0.62), TerraIncognita (0.84), WILDSCamelyon (1.00), and CXR (0.94). The authors caution this should not be read as evidence of VLM robustness to spurious correlations, since for the VLMs both ID and OODSelect examples are out of distribution (or, because many datasets are public, may all be in distribution).
• Distance-based selection fails: A baseline that greedily selects OOD examples farthest from ID examples in CLIP embedding space often yields positive ID–OOD correlations (for example, R = 0.52 on PACS with N = 10), sometimes stronger than random selection, and does not recover the negative correlations OODSelect identifies (the paper cites R = -0.92 on VLCS with N = 10).

Methodology in Plain English

The researchers first fix, for each dataset, the standard ID/OOD domain split the community uses (for example, Sketch as OOD for PACS, LabelMe for VLCS, L46 for Terra Incognita, MIMIC-CXR as OOD for the chest X-ray task). They then train a large, diverse population of models on the same ID training data—up to 4200 models per dataset—varying architecture, pretraining weights, initialization, and hyperparameters, spanning vision architectures from ResNets and DenseNets to ViTs and Inception, and text models from BERT to GPT-2. These models are randomly split into train, validation, and test model sets (60/20/20), so the models used to learn the selection, to validate it, and to report final results do not overlap.

For each model, they record whether it classifies each OOD example correctly, forming a binary matrix of models by OOD examples, and pair this with each model's held-out ID accuracy. Finding a subset of OOD examples is cast as an optimization: choose a fixed number S of OOD examples so that the Pearson correlation between ID accuracy and accuracy on the selected subset is minimized. This objective is relaxed to allow soft selections (produced by a sigmoid in practice) with a penalty term that pushes the subset size toward S, and it is solved with the Adam optimizer and a cosine annealing schedule for the learning rate and penalty parameter.

The authors note that their objective is non-convex and non-submodular, so global optimization is intractable, but it is Lipschitz-continuous, which gives stable gradients and bounded progress under descent toward near-binary stationary points approximating the discrete optima. They also bound how much adding one model or one OOD example can change Pearson R. Statistical uncertainty is handled with Fisher z confidence intervals (95%), and correlation strength is classified as AoTL above 0.3, AoTIL below -0.3, and weak in between. Finally, they check that the subsets are coherent using metadata where available (CXR demographics and clinical attributes), a normalized Jaccard index where metadata is absent, and a model-generated semantic comparison pipeline using Qwen2.5-32B-Instruct for captions, AIMV2-large-patch14-224-lit to propose candidate descriptions, and CLIP to score and rank them.

Why This Matters

The paper argues that spurious correlations do not disappear in the real world; current benchmarks and aggregate metrics simply hide them. If aggregate OOD performance looks robust while a large, coherent part of the OOD population is systematically failing, then reported robustness numbers can give a false sense of safety. The authors also observe that spurious associations reflecting historical or structural bias—such as occupation and gender, which can bias recommendation system outputs—may be preserved in both training and test environments of naturally collected benchmarks, so models relying on them can keep "performing well OOD" while degrading on the affected subsets.

Real-world applications affected:

• Medical diagnosis across hospitals: A model trained on data from one set of hospitals may underperform on a different hospital's patients (for example, chest X-ray models that improve overall performance performing worse on patients with pleural conditions or enlarged cardiomediastinum).
• Histopathology and cancer detection: WILDS-Camelyon shifts across hospitals; the authors find OODSelect subsets up to 60000 examples (40% of the OOD set) with AoTIL.
• Content moderation: WILDS-CivilComments classifies comments as toxic or non-toxic across demographic subgroups such as gender, religion, and race, where a subset with hidden failures means some groups bear the cost.
• Wildlife monitoring and camera traps: Terra Incognita involves wildlife recognition across camera trap locations, where spurious cues such as daylight or occlusion can drive errors.

Industry relevance: teams that report a single aggregate OOD or domain-generalization number can be masking deployment risk. The released subsets and code offer a robustness check that companies can apply to models before deployment, and the paper recommends treating identified large OODSelect subsets as first-class evaluation targets and designing methods that improve both average and subset robustness.

Future Directions

1. Adopting the OODSelect selection protocol as a standard robustness check when creating any new OOD benchmark, and treating identified large subsets as first-class evaluation targets.
2. Designing training methods that improve both average and subset robustness simultaneously, rather than trading one for the other.
3. Making semantic explanation of the selected subsets more reliable; the model-generated descriptions (for example, "frost", "motion blur", "extreme weather conditions" for TerraIncognita) were found to be neither consistent nor robust, and many spurious features may be imperceptible to humans, as in chest X-rays or WILDSCamelyon tissue slides.
4. Reducing the computational cost: the analysis requires training up to 4200 models per dataset and optimizing a selection objective over up to around 146000 elements, though this is described as a one-time cost per dataset that is mitigated by releasing the selections.

Target Audience

Researchers and practitioners working on OOD generalization, domain generalization, and robustness evaluation; benchmark designers for both vision and language tasks; and applied machine learning teams in high-stakes domains such as medical imaging and content moderation who rely on aggregate OOD metrics to make deployment decisions. It is most useful to readers already comfortable with distribution shift terminology and quantitative correlation analysis.

Authors’ abstract

Benchmarks for out-of-distribution (OOD) generalization frequently show a strong positive correlation between in-distribution (ID) and OOD accuracy across models, termed "accuracy-on-the-line." This pattern is often taken to imply that spurious correlations - correlations that improve ID but reduce OOD performance - are rare in practice. We find that this positive correlation is often an artifact of aggregating heterogeneous OOD examples. Using a simple gradient-based method, OODSelect, we identify semantically coherent OOD subsets where accuracy on the line does not hold. Across widely used distribution shift benchmarks, the OODSelect uncovers subsets, sometimes over half of the standard OOD set, where higher ID accuracy predicts lower OOD accuracy. Our findings indicate that aggregate metrics can obscure important failure modes of OOD robustness. We release code and the identified subsets to facilitate further research.