To Align or Not To Align: Check Your COMPASS Before You Train

AI summary

Overview

• Research area: Transfer learning and representation learning, specifically the selection of pretrained model pairs for cross-modal alignment and weight-space model merging.
• Technical level: Intermediate. The core diagnostic is simple (nearest-neighbor overlap on a probe set), but the motivation draws on total variation distance, KL divergence, and neighborhood distributions.
• Scope in one sentence: The paper introduces COMPASS, a cheap pre-training diagnostic that predicts whether two pretrained models will combine well, and validates it on 72 cross-modal alignment pairs and 392 task-arithmetic model merges.

What This Paper Is About

Combining pretrained models, either by aligning a vision encoder with a language model or by merging fine-tuned checkpoints, is a common way to build AI systems, but the right pair of models is usually identified only after running the expensive alignment or merging procedure. The authors propose COMPASS, a diagnostic that scores candidate pairs beforehand using only forward passes over a small probe set, so that compute is spent on pairs likely to succeed. The paper supplies a theoretical motivation for why local neighborhood agreement is predictive and then tests that claim across two transfer settings.

Key Contributions

1. The COMPASS diagnostic. A model-pair compatibility score built from two cheap-to-compute axes: a pair-specific ease score (the k-NN overlap between two encoders at the start of the relevant dynamics) and a paradigm-level ceiling score (the maximum overlap their training paradigms allow). The continuous score is the product of the two.
2. Theoretical motivation. An analysis (Theorem 4.1) linking per-sample start-condition overlap to finite-budget convergence, and a bound (Theorem 4.2) showing that target discrepancy between two training paradigms caps achievable per-sample overlap at 1 − γ_i/(1 − ε) even under perfect optimization.
3. Empirical validation across two applications. COMPASS is evaluated on 72 cross-modal alignment experiments and 392 task-arithmetic model merges, with average Pearson correlation of 0.91 on cross-modal alignment tasks and 0.70 on model merging (abstract figures).
4. An action-guiding phase diagram. Thresholding both axes at their medians assigns each pair to one of four regimes — act now, budget-limited, ceiling-limited, or incompatible — each prescribing a different response (proceed, invest more compute, switch paradigm, or deprioritize).

Main Findings

• Ease and ceiling together beat global similarity measures. In cross-modal alignment over 72 vision-language pairs, the product ease × ceiling reaches |r| = 0.90 with Flickr30k I2T R@1 and |r| = 0.91 with COCO I2T R@1, versus 0.63 and 0.68 for global CKA. In task-arithmetic merging over 392 merges, the product reaches |r| = 0.70 with mean retention, versus 0.25 for global CKA.
• Each axis contributes separately. Ease alone reaches 0.84/0.83 (Flickr30k/COCO) and ceiling alone 0.85/0.83; for merging, ease alone is 0.67 and ceiling alone 0.64.
• The advantage survives at the level of rankings. COMPASS reaches Spearman ρ = 0.91/0.93 on Flickr30k/COCO and 0.70 for merging. Under budgeted evaluation it finds the optimal alignment pair immediately and the optimal merge within the first 5 of 392 candidates.
• The four regimes separate outcomes. On Flickr30k, mean I2T R@1 is 0.46 ± 0.09 for act now (n = 28), 0.34 ± 0.05 for ceiling-limited (n = 8), 0.30 ± 0.06 for budget-limited (n = 10), and 0.21 ± 0.11 for incompatible (n = 26), with the same ordering on COCO (Kruskal–Wallis H = 45.3, p < 10⁻⁹ for Flickr30k; H = 51.5, p < 10⁻¹⁰ for COCO).
• Merge regimes separate similarly. Mean retention is 0.84 ± 0.16 for act now (n = 129), 0.57 ± 0.26 for budget-limited (n = 67), 0.56 ± 0.23 for ceiling-limited (n = 67), and 0.35 ± 0.17 for incompatible (n = 129), with Kruskal–Wallis H = 198.2, p < 10⁻⁴².
• Budget-limited and ceiling-limited pairs look similar but require different actions. Their mean retention is nearly identical (0.57 vs. 0.56), yet the paper argues the first calls for more optimization or a stronger partner while the second calls for a different pretraining paradigm.
• Ceiling values track pretraining objective. Cross-modal ceiling scores are 0.140 for CLIP, 0.134 for SigLIP, 0.112 for CLIP fine-tuned, 0.109 for self-distillation, 0.100 for discriminative, and 0.058 for reconstructive paradigms.
• The same paradigm can be good in one application and bad in another. Contrastive vision-language models populate act now for cross-modal alignment but concentrate in the ceiling-limited and incompatible regimes for task-arithmetic merging, while discriminative, self-distillation, and reconstructive paradigms show the reverse pattern.
• The signal is not just family-level. Holding the vision family fixed, ease correlates positively with R@1 across text partners with a median Spearman ρ = 0.80 across the six vision families.
• Ceiling reflects achievable performance, not marginal budget gains. Across 18 alignment pairs, its correlation with downstream performance rises from ρ = 0.79 after 10 epochs to 0.92 after 300 epochs.
• Unimodal settings show the same pattern. Initial random k-NN overlap predicts final pretrained overlap across n = 21 vision family pairs (r = 0.69, p < 10⁻⁴) and n = 42 text family pairs (r = 0.91, p < 10⁻¹⁷); pretrained overlap predicts Flickr30k I2T R@1 across n = 72 vision-language pairs (r = 0.84, p < 10⁻²⁰), and pre-merge overlap predicts merge quality across n = 392 merges (r = 0.67, p < 10⁻⁵²).
• Results are robust to library composition and ceiling estimation. Leaving out individual families or paradigms preserves correlations of 0.85–0.93 for alignment and 0.60–0.72 for merging; replacing pair-specific ceiling estimates with a single shared value per paradigm, or perturbing them with moderate noise, leaves the ranking largely unchanged, while shuffling the paradigm ordering removes the benefit.
• Results are not specific to the CSA adapter. On 24 representative pairs, COMPASS remains predictive under trained Linear, MLP, and LoRA-style adapters and outperforms global CKA in each case.
• Results hold across neighborhood sizes. The qualitative pattern is unchanged at k ∈ {20, 50, 100}, while the paper fixes k = 10 throughout.

Methodology in Plain English

The authors treat both cross-modal alignment and weight merging as instances of one question: how much do two pretrained models agree? They measure agreement locally rather than globally. For each sample in a small probe set, they take the k nearest other samples in each model's embedding space and count how many neighbors the two models share; averaging this over the probe set gives the k-NN overlap. Prior work (the Aristotelian Representation Hypothesis) showed this local agreement is what is empirically observable across architectures and modalities, whereas global geometry differs.

From this single measurement they construct two scores. Ease is the overlap measured at the start of the relevant dynamics — the frozen pretrained encoders for alignment, the fine-tuned checkpoints for merging. Ceiling is a paradigm-level average: how much overlap models from a given pair of training paradigms can support in general. Both are computed with forward passes only, so scoring scales with the number of models rather than the number of pairs; for the 19 × 10 alignment grid, COMPASS needs 29 probe passes instead of 190 downstream alignment runs, translating to more than two orders of magnitude lower wall-clock cost in their setup. Probe sets used are 1024 image-text pairs from WIT for alignment and a 2048-sample probe pooled from one of the eight fine-tuning tasks (here MNIST) for merging.

Two theoretical results motivate the decomposition. The first says that when two models start with more shared neighbors, the probability of them reaching a common neighborhood structure within a fixed training budget is higher. The second says that even with perfect optimization, the overlap two paradigms can achieve is capped by how differently their training targets define neighborhoods. The authors then validate the score by correlating it with downstream outcomes: image-to-text R@1 on Flickr30k and COCO after training a CSA adapter on MS-COCO for alignment, and mean retention across tasks after averaging weights for task-arithmetic merging.

Why This Matters

Impact on research. The paper shifts representational convergence from an observation to a decision tool. Instead of aligning or merging every candidate combination and measuring the result, researchers can rank pairs before committing compute, and the two-axis view explains why a model family that is excellent in one transfer operation can be poor in another. It also supplies theoretical scaffolding (finite-budget convergence and a target-discrepancy ceiling) for why local overlap is the right quantity rather than global CKA.

Real-world applications.

• Selecting which vision encoder to pair with a language model for image-text retrieval, where the paper reports up to 63% Flickr R@1 for act-now pairs.
• Choosing which fine-tuned checkpoints to merge for multi-task deployment, where the paper reports 392 candidate merges from 14 ViT backbones and 8 classification tasks.
• Allocating a fixed compute budget across a large grid of candidate combinations, since scoring scales with models rather than pairs.
• Deciding whether to change pretraining paradigm rather than spend more on data or optimization, which the ceiling-limited regime is designed to flag.

Industry relevance. Model libraries containing many pretrained encoders and fine-tuned checkpoints make pair selection combinatorially expensive. A diagnostic that costs forward passes over roughly 1024 or 2048 samples and returns a pre-training ranking directly addresses that bottleneck, and the paper reports that it reduces wall-clock cost by more than two orders of magnitude relative to running every alignment.

Future Directions

• Extending COMPASS to other settings that involve interactions between two learned representations or models, which the authors explicitly leave to future work.
• Defining a more general ceiling proxy, since the authors state that their ceiling is instantiated separately for cross-modal alignment and for task-arithmetic merging and cannot be expressed as a single universal formula.
• Relaxing or broadening the dynamical assumptions behind Theorem 4.1, which the authors describe as sufficient rather than necessary conditions that may not hold for all training regimes.
• Further testing whether the ranking generalizes beyond the CSA adapter and the 24 representative pairs already evaluated with Linear, MLP, and LoRA-style adapters, and beyond the specific model library used here.

Target Audience

Researchers and practitioners working on transfer learning, multimodal alignment, and weight-space model merging who need to choose among many candidate pretrained models. It is also relevant to engineers managing large checkpoint libraries who want to reduce the number of expensive training or merging runs, and to readers interested in the empirical and theoretical relationship between local representational agreement and downstream transfer performance.

Authors’ abstract

Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$-NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to $72$ cross-modal alignment experiments and $392$ task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of $0.91$ on cross-modal alignment tasks and $0.70$ on model merging.