CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection

AI summary

Overview

• Research area: Data-centric domain adaptation of vision-language models, specifically continual pre-training (CPT) of CLIP for vertical/medical domains via data selection.
• Technical level: Advanced (influence functions, second-order curvature, Newton-style preconditioning, Johnson-Lindenstrauss sketching, with formal bounds).
• Scope (one sentence): The paper proposes CHIPS, a CLIP-specific data selector that scores each image-text pair by a curvature-aware, InfoNCE-aware, learnability- and relevance-modulated utility, and shows it reaches strong medical-domain performance using only a fraction of large-scale continual pre-training data while better preserving general-domain ability.

What This Paper Is About

Adapting a general-purpose CLIP model to a specialized domain such as medical imaging is usually done either by changing the training procedure or by continuing to pre-train on enormous domain-specific image-text collections (millions to hundreds of millions of pairs). The authors ask whether the data itself can be chosen more intelligently, so that effective adaptation does not require extreme scale. CHIPS answers this by assigning every candidate image-text pair a utility score that predicts how useful that sample would be for a one-step update, then selecting the top-n pairs for continual pre-training.

Key Contributions

1. A curvature-aware alignment score in CLIP's "end-point" subspace. The score is computed on the projection heads and the temperature parameter, using a Newton-style preconditioned inner product, and is shown by Theorem 1 to lower-bound its correlation with full-parameter alignment.
2. An InfoNCE-aware curvature estimator with Johnson-Lindenstrauss (JL) sketching. The estimator mixes self (positive) and cross (negative-pair) gradient moments to capture the cross-example coupling that symmetric InfoNCE creates, and Theorem 2 characterizes the resulting variance-bias trade-off between an O(1/k) projection variance term and a curvature bias term.
3. A selection-aware relevance weight combined with learnability. Two multiplicative weights—one favoring decision-boundary samples (based on correctness probability and hardest-negative margin) and one softly favoring target-domain-relevant samples—balance target adaptation against general-domain retention.
4. An empirical demonstration across 48 tasks (17 medical benchmarks, 31 general-domain benchmarks). CHIPS is reported as state-of-the-art among selection baselines on the medical suite, matching full-dataset CPT with 30% of the data and outperforming half-dataset CPT with only 10%, while showing the least general-domain drop under all tested retention ratios.

Main Findings

• Best medical average among selectors. On MetaCLIP-B16-400M with subsets of BIOMEDICA, CHIPS reaches Medical Avg scores of 27.03, 28.20, and 29.96 at retention ratios r = 10%, 20%, and 30%, exceeding the second-best method by +0.57, +1.57, and +3.68 points respectively.
• Small data rivals much larger data. With only 10% of BIOMEDICA, CHIPS (27.03) outperforms a 50% random subset (26.26); with 30% it reaches 95.1% of the full-dataset performance (29.96 vs. 31.51).
• Competitive with medical-specific CLIP models. At r = 30%, CHIPS slightly surpasses BMCLIP (29.96 vs. 29.86) and consistently outperforms PubMedCLIP (average 26.21) and BioMedCLIP (average 21.45) across retention ratios.
• Better general-domain retention than the previous SOTA selector. For classification, CHIPS retains 90.1%, 89.3%, and 87.2% of the vanilla model at r = 10%, 20%, and 30% (versus 88.9%, 88.0%, and 87.0% for TracIn); for retrieval, R@1 is slightly lower at r = 10% but higher at r ≥ 20%, giving a higher average across retention ratios.
• Scores transfer across backbones and pre-training scales. In a seven-setting generalization experiment (B32/B16/L14/H14 backbones at 400M and CC = 2.5B pre-training scale), CHIPS attains the best Medical performance in every setting, outperforming TracIn by 0.20 to 2.65 points, while typically ranking second (behind Random) on general-domain metrics, with a single retrieval exception on B16-400M.
• Lower or equal scoring cost than influence baselines. CHIPS requires 50.9475 × 10^15 FLOPs, 3.1% lower than TracIn (52.5891 × 10^15) and effectively on par with TRAK (50.9458 × 10^15).
• Ablation supports the multiplicative design. Progressively adding components, CHIPS surpasses the strongest ablation by +1.05, +0.28, and +1.46 points at r = 10%, 20%, and 30%, with general-domain performance within ≤ 0.53 (classification) and ≤ 0.99 (retrieval) of the best ablation.
• End-point subspace scores preserve full-parameter ordering. On a 100K subset of BIOMEDICA, the Spearman correlation between proxy and full alignment scores is 0.83.
• Text projection matters most. Ablating the end-point geometry, Text-only nearly matches All (98.6–99.7% across methods); for CHIPS the scores are 27.03 (All), 26.95 (Text-only, 99.7%), 26.67 (Visual-only, 98.7%), and 23.83 (Logit-only, -3.20). CHIPS has the smallest text-visual gap (0.28 vs. 2.43/3.77/1.05 for Dot/TracIn/TRAK).
• Bigger evaluation sets help mainly the target domain. Increasing samples per task from 50 to 200 raises Medical CLS from 26.25 to 27.03 (+0.78), then saturates at 250 (27.12, +0.09); General CLS stays essentially flat (48.41 at 50 to 47.88 at 200, range ≤ 0.55). The default is 200 samples per task.
• Moderate curvature mixing is best. Medical CLS peaks at α = 0.6 (27.05) and is essentially tied at 0.8 (27.03), while 0.2 (25.92), 0.4 (25.66), and 1.0 (26.07) underperform by roughly 1–1.4 points; General CLS is nearly flat (48.36–48.46). The recommended range is α ∈ [0.6, 0.8].
• Sparse random projections are most effective. Over k ∈ {2048, 4096, 8192, 16384}, Sparse peaks at 28.31 (16k) and stays strong at 2k (26.56); CountSketch improves from 24.40 to 27.64 between 2k and 16k (small dip at 8k, 26.12; 27.03 already at 4k); SRHT peaks at 4k (26.42) and degrades at 16k (24.61). Sparse-16k is best, +0.67 over CountSketch-16k and +1.89 over SRHT-4k; CountSketch-4k offers a cost-efficient alternative within about 1.3 points of the best.
• The β analysis is the point where the provided text ends. The truncated content stops mid-sentence in Section 4.2 while introducing the β parameter in the domain-relevance weight; the reported outcomes of that analysis are not available in the supplied text.

Methodology in Plain English

The core idea is to treat data selection as a prediction problem: which training samples, if used for a single gradient step, would most reduce the loss on a held-out target evaluation set?

1. Define the score. For each image-text pair, CHIPS computes a gradient, then multiplies it by a curvature matrix and compares it to the average gradient of the target evaluation set. A high score means the sample points in a direction that would help the target domain.
2. Compute curvature where it matters. The authors argue that for CLIP, the projection heads and the temperature parameter drive early changes in similarity structure, so they compute the score only there rather than over all parameters. They also account for the fact that InfoNCE's softmax normalizer couples each positive pair with many negatives, by mixing "self" gradient moments with "cross" gradient moments using a mixing weight α and a small ridge term λ

Authors’ abstract

Adapting CLIP to vertical domains is typically approached by novel fine-tuning strategies or by continual pre-training (CPT) on large domain-specific datasets. Yet, data itself remains an underexplored factor in this process. We revisit this task from a data-centric perspective: Can effective data selection substitute for large-scale datasets in CPT? We introduce CHIPS (Curvature-aware Hybrid Influence in Projection Subspace), which assigns each image-text pair a utility score that integrates three complementary factors aligned with three goals: faithfulness via a curvature-aware and Newton-style alignment computed in CLIP's end-point subspace; scalability via an InfoNCE-aware curvature estimator with Johnson-Lindenstrauss (JL) sketching; and retention via a selection-aware relevance weight combined with learnability to balance target adaptation against general-domain preservation. We justify this design theoretically by proving a lower-bound guarantee on the proxy's correlation with full-parameter alignment and by characterizing the bias-variance trade-offs introduced by curvature mixing and JL sketching. We evaluate CHIPS empirically across various settings: 1) CHIPS attains state-of-the-art performance among selection baselines on 17 medical benchmarks, matches full-dataset CPT with 30% of the data, and outperforms half-dataset CPT using only 10%; 2) on 31 general-domain benchmarks, CHIPS yields the least performance drop under all retention ratios.