Scalable Evaluation and Neural Models for Compositional Generalization

AI summary

Overview

Research area: Machine learning — compositional generalization, out-of-distribution generalization, disentangled representation learning, and vision architecture design. Technical level: Advanced. Scope: The paper proposes a scalable evaluation framework (orthotopic evaluation) for measuring compositional generalization in supervised vision models, and introduces Attribute Invariant Networks (AINs), an architecture class that trades a small parameter overhead for large gains in compositional generalization.

What This Paper Is About

Compositional generalization is the ability to correctly predict novel combinations of concepts that were individually seen during training (for example, predicting "apple, green" after only seeing "apple, yellow" and "banana, green"). The paper argues that progress in this area is stalled both because there is no standardized evaluation protocol — existing benchmarks trade rigor for efficiency — and because standard vision backbones lack the inductive biases needed for compositionality. It therefore supplies a unified, computationally cheap evaluation method and a new model family designed to be compositional by construction.

Key Contributions

1. A universal evaluation framework for compositional generalization in supervised learning that unifies prior strategies, reduces computational complexity from combinatorial (Θ(I^c)) to constant (Θ(1)), and adds a controllable degree of freedom — the compositional similarity index c — that produces a principled ladder of evaluation difficulty.
2. A large-scale empirical study validating the framework, training more than 5000 SOTA vision models — described by the authors as the most extensive and up-to-date evaluation of compositional generalization for supervised models.
3. Attribute Invariant Networks (AINs), a new class of architectures enforcing attribute invariance in gradient updates by construction, achieving a reported 23.43% average test-accuracy improvement over monolithic baselines while limiting parameter overhead to 6.4%–16%, versus 300%–600% for explicitly disentangled (ED) models.
4. A new Pareto frontier in the scalability–generalization tradeoff, supported by comparisons across monolithic, AIN, and ED architectures on multiple datasets.

Main Findings

• The index c strongly controls measured performance. Across all studied model families and datasets, changing c distinctly changes measured compositional generalization, even though training data size and concept-combination variety are held fixed. The authors argue prior results must be reinterpreted in light of the c value used.
• The difficulty ladder is empirically supported. No model generalized properly in the extrapolation regime (c = 0), while (almost) every model reached perfect accuracy in-distribution (c = I). Between those extremes, c and compositional generalization were positively correlated in the majority of cases.
• Disentangled vs. entangled regimes diverge on some datasets. On datasets such as dSprites and Shapes3D, most models reached roughly 100% test accuracy in the entangled compositional regime (1 < c < I) but struggled at c = 1. This gap was less pronounced on other datasets.
• Orthotopic evaluation is stricter than pair-wise evaluation. Test accuracy under orthotopic evaluation was consistently lower — 11.6% on average — than under pair-wise evaluation, expected because orthotopic excludes more training data (40% vs. 10%) and can involve far more task-relevant factors (I−1 vs. 2).
• Overall trends match between frameworks. MLPs (no architectural inductive bias) were worst; pre-trained models generally performed worse than models trained from scratch; ConvNeXt was consistently strong; and ED architectures significantly outperformed monolithic models.
• Prior-work consistency. Results on dSprites and Shapes3D were similar to Schott et al. [5], while performance on MPI3D-real was significantly better. The remaining datasets could not be compared directly, as this is the first work to use them for compositional generalization.
• Negative results are reported. Increasing optimization steps up to 240× (to probe grokking) and replacing activation functions with learnable ones did not improve outcomes.
• ED models are the strongest baseline but poorly scalable. ED architectures improved over the strongest baseline (ConvNeXt) by an average of 21.38%, but require a separate set of weights per attribute.
• AINs close much of the gap cheaply. In Table 1 (ResNet-18 backbone), AINs outperformed the monolithic RN-18 across datasets — for example 54.02 ±0.24 on MPI3D and 66.23 ±2.26 on I-RAVEN versus 41.59 ±2.16 and 11.30 ±2.80 for RN-18 — with an overhead of 6.4%–16%, compared with 300%–600% for ED. On Shapes3D, ED (96.09 ±2.65) still exceeded AIN (85.26 ±0.63).
• Only minimal architecture changes were needed. Disentangling just the first convolutional layer was, in preliminary experiments, the minimum additional weighting sufficient to improve attribute invariance.

Methodology in Plain English

The authors formalize data as being generated by discrete factors, split into task-relevant factors G and task-irrelevant factors O. Compositional generalization is defined as the case where the training and test distributions over combinations of task-relevant factors have disjoint support.

They then generalize this definition by introducing the compositional similarity index c, which sets how many task-relevant factors a test sample can share with its closest training sample. This turns a vague definition into a tunable knob: c = 0 is extrapolation, 1 ≤ c ≤ I−1 is compositional generalization (c = 1 "disentangled", 2 ≤ c ≤ I−1 "entangled"), and c = I is in-distribution generalization.

To build test splits efficiently, their orthotopic evaluation algorithm iteratively projects the training data onto c-dimensional attribute subspaces and prunes "orthotopes" (hyper-rectangular volumes) from the training set using dynamically computed thresholds. Instead of training one model per attribute subspace — which scales as Θ(I^c) — a single training run per model suffices, giving Θ(1) complexity. Excluded attribute-pairs were held at roughly 60% across datasets.

They then evaluate many model families (MLPs, ResNets, DenseNets, Wide ResNets, ConvNeXt, ViT, Swin Transformer, plus ImageNet-1k pre-trained RN-101, RN-152, DN-121) using exact-match multi-label accuracy, with cross-entropy loss on all attribute labels and model selection on a 10% held-out split. They also implement an improved version of the Shared Architecture as an explicitly disentangled baseline, using kernel-based initialization to give concept representations a similarity-preserving structure.

Finally, they design AINs. Each attribute i has its own encoder and classification head, but a shared "meta-model" sits between them, so that gradients from attribute j do not flow into the encoder for attribute i (Theorem 4.2). This yields invariance to transformations of other attributes at near-constant parameter cost.

Why This Matters

Impact on research: The paper argues that the field has been comparing incomparable results because prior work used different, unstated values of the similarity constraint — some using κ = 1, others κ = I−1. A unified, single-training-run protocol with an explicit difficulty parameter makes cross-paper comparison possible and gives researchers a way to report results along a full difficulty spectrum rather than at one arbitrary point.

Real-world applications (as implied by the paper's framing):

• Robust vision systems that must handle novel combinations of object properties not present together in training data.
• Reducing brittleness under distribution shift in deployed perception models.
• Attribute-based recognition and tagging pipelines where separate attributes (shape, color, size) must be predicted jointly.
• Model design in resource-constrained settings, where AINs' small overhead is preferable to fully disentangled ensembles.

Industry relevance: The parameter-overhead comparison matters directly for deployment: ED models' 300%–600% overhead versus AINs' 6.4%–16% is framed as the difference between an approach that is impractical in real-world scenarios and one that stays close to standard model size. The released code (github.com/IBM/scalable-compositional-generalization) supports adoption as a benchmarking standard.

Future Directions

• The paper's own limitations section notes that both orthotopic evaluation and AINs assume that at least a subset of generative factors is accessible and labeled — analogous to requiring ground-truth labels for supervised training. Extending to unlabeled or partially labeled settings is left open.
• The provided content is truncated mid-sentence in the limitations discussion of how AINs scale linearly with the number of attributes; the full set of proposed future work and the complete limitations discussion are not reported in the available text.
• The paper notes that c should be systematically included in future empirical investigations, implying broader re-evaluation of prior results under the new protocol.
• Additional degrees of freedom in the evaluation — the size and position of the excluded volumes — are identified but held fixed in this work, leaving them as open design choices.
• The discrepancy in MPI3D-real results relative to Schott et al. [5] is attributed to differences in problem formulation (classification vs. regression), preprocessing, and models, and is not fully resolved.

Target Audience

Researchers and practitioners working on out-of-distribution generalization, compositional generalization, disentangled and attribute-based representation learning, and robust vision architectures — particularly those who need a standardized benchmark, or who are building models that must handle novel combinations of known concepts. The paper is also relevant to engineers weighing the accuracy-versus-model-size tradeoff of disentangled approaches against standard backbones.

Authors’ abstract

Compositional generalization-a key open challenge in modern machine learning-requires models to predict unknown combinations of known concepts. However, assessing compositional generalization remains a fundamental challenge due to the lack of standardized evaluation protocols and the limitations of current benchmarks, which often favor efficiency over rigor. At the same time, general-purpose vision architectures lack the necessary inductive biases, and existing approaches to endow them compromise scalability. As a remedy, this paper introduces: 1) a rigorous evaluation framework that unifies and extends previous approaches while reducing computational requirements from combinatorial to constant; 2) an extensive and modern evaluation on the status of compositional generalization in supervised vision backbones, training more than 5000 models; 3) Attribute Invariant Networks, a class of models establishing a new Pareto frontier in compositional generalization, achieving a 23.43% accuracy improvement over baselines while reducing parameter overhead from 600% to 16% compared to fully disentangled counterparts. Our code is available at https://github.com/IBM/scalable-compositional-generalization.