Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models

AI summary

Overview

Research area: Robust fine-tuning of contrastive pre-trained vision-language models (CLIP-style models) for computer vision.

Technical level: Intermediate to Advanced. The paper assumes familiarity with contrastive pre-training, embedding spaces, and fine-tuning baselines such as FLYP, ARF, LP-FT, and SnD.

Scope: The paper proposes Difference Vector Equalization (DiVE), a fine-tuning method that constrains the difference vectors between pre-trained and fine-tuned embeddings to stay equal across samples, in order to preserve the geometric structure of the embedding space while fine-tuning on in-distribution data.

What This Paper Is About

Vision-language models like CLIP generalize well zero-shot, but fine-tuning them on a target dataset (for example ImageNet) improves in-distribution accuracy while damaging out-of-distribution and zero-shot performance. The authors observe that existing robust fine-tuning methods distort the geometric structure of the pre-trained embeddings — the relative positions of embeddings that encode semantic similarity — and that this distortion limits OOD and zero-shot results. DiVE's goal is to fine-tune on in-distribution data while keeping that geometry intact.

Key Contributions

1. Identification of the problem: The authors measure the geometric structure of the embedding space using a representation similarity analysis (RSA) correlation score and report that existing robust fine-tuning methods (FLYP, "FLYP + replay" as a proxy for ARF, and SnD) score only in the 0.825 to 0.850 range, indicating structural distortion.

2. The DiVE method: Two new losses built on FLYP's contrastive fine-tuning — average vector loss (AVL) and pairwise vector loss (PVL). Both operate on difference vectors, defined as the fine-tuned embedding minus the pre-trained embedding for the same sample.

3. Empirical demonstration of structural preservation: DiVE reaches an RSA correlation score of 0.981, compared with 0.978 for FLYP + AVL alone and 0.976 for FLYP + PVL alone, versus 0.850 for FLYP + replay.

4. Strong results across ID, OOD, and zero-shot metrics: Verified on ImageNet, iWildCam, and FMoW with CLIP ViT-B/16, and additionally on ImageNet with CLIP ViT-L/14, plus ablations over AVL/PVL, the α hyper-parameter, reference datasets, and weight ensembling.

Main Findings

• Geometric preservation: On the RSA correlation score evaluated with Flickr8K, DiVE scores 0.981, ahead of FLYP (0.825), FLYP + replay (0.850), SnD (0.847), FLYP + AVL (0.978), and FLYP + PVL (0.976).

• ImageNet ID and OOD (CLIP ViT-B/16): DiVE achieves 82.5 ID accuracy and 63.2 average OOD accuracy across ImageNet-V2, ImageNet-R, ImageNet-A, ImageNet-Sketch, and ObjectNet. It is second-best on ID (below ARF at 82.7) but state of the art on four of the five OOD datasets and on the OOD average. Per-dataset DiVE values: ImageNet-V2 73.8, ImageNet-R 77.3, ImageNet-A 54.9, ImageNet-Sketch 52.9, ObjectNet 56.9.

• Zero-shot results: DiVE reaches a 63.7% average over ten zero-shot datasets, versus 55.6% for ARF, 54.4% for SnD, 49.5% for FLYP, 51.0% for LP-FT, 35.7% for vanilla FT, and 66.0% for the pre-trained model. It is state of the art on nine of the ten datasets, with the best per-dataset scores on Flowers (66.0), Food (84.3), SUN397 (64.7), DTD (47.0), Aircraft (22.1), StanfordCars (55.5), OxfordPets (88.4), EuroSAT (51.4), and UCF-101 (68.9).

• Trade-off quantification: Relative to pre-trained CLIP ViT-B/16, DiVE's zero-shot performance drops by only 2.3 points while ID improves by 14.2 points and OOD improves by 4.5 points.

• Beyond ImageNet: On iWildCam, DiVE records 53.1 ID (macro F1), 37.2 OOD (macro F1), and 65.3 zero-shot average. On FMoW, it records 69.9 ID (top-1), 42.3 OOD (worst-region top-1), and 65.1 zero-shot average.

• Ablation of the two losses: Starting from FLYP (82.2 ID, 59.5 OOD average, 49.5 zero-shot), AVL alone gives 82.4 / 62.9 (+3.4) / 62.9 (+13.4), PVL alone gives 82.4 / 62.6 (+3.1) / 62.7 (+13.2), and both together give 82.5 / 63.2 (+3.7) / 63.7 (+14.2).

• Effect of α: With α = 0, OOD and zero-shot average are 61.7 and 61.4; α = 0.5 gives 62.6 and 62.4; α = 0.9 gives 62.8 and 62.9; α = 0.99 (the default) gives 62.9 and 62.9.

• Difference vectors beat cosine constraints: Replacing the vector constraint with a CyCLIP-style cosine similarity constraint yields RSA 0.949, 82.4 ID, 62.3 OOD average, and 61.7 zero-shot, all below DiVE's 0.981, 82.5, 63.2, and 63.7.

• Scaling and reference data: With CLIP ViT-L/14, DiVE reaches 86.1 ID, 74.5 OOD average, and 70.1 zero-shot, versus 86.0 / 73.2 / 61.2 for SnD and 86.0 / 71.5 / 55.4 for FLYP. Performance improves with reference dataset size: no reference (FLYP, 0 samples) gives 82.2 / 59.5 / 49.5; Flickr8K (8K) gives 82.2 / 60.6 / 55.2; COCO Captions (118K) gives 82.4 / 62.3 / 62.2; CC3M (3M, default) gives 82.5 / 63.2 / 63.7. Even with COCO Captions, DiVE outperforms ARF, which uses CC3M.

• Weight ensembling helps further: FLYP + ensemble gives 82.5 / 62.3 / 57.3; DiVE alone gives 82.5 / 63.2 / 63.7; DiVE + ensemble gives 82.6 / 63.5 / 64.6.

• Non-zero difference vectors are better than zero: SnD, which forces the image-embedding difference to zero, underperforms DiVE across ImageNet, iWildCam, and FMoW, and its RSA score is 0.847. The authors attribute this to the constraint being too strong for adapting to target data.

Methodology in Plain English

The approach starts from FLYP, which fine-tunes both the image and text encoders of a vision-language model using the same contrastive loss used during pre-training, applied to target data with class-name prompts.

On top of that, the authors add a reference image-caption dataset that resembles the pre-training data. For each reference image and caption, they compute a difference vector: the embedding from the fine-tuned model minus the embedding from the frozen pre-trained model. The core idea is that if all these difference vectors are the same, the embedding space has effectively been shifted rather than warped, so relative positions — and therefore semantic relationships — are preserved. The paper supports this by noting that the norm of the difference vectors is very small on average (10⁻³ to 10⁻⁴), so curvature of the hypersphere can be ignored in practice.

Two losses enforce this. AVL computes an exponential moving average of difference vectors across batches (with α controlling how much weight the previous average retains; α = 0.99 by default) and pulls every difference vector toward that average. PVL instead pulls the difference vector of each reference image toward the difference vector of its matching caption, keeping the local image-text alignment consistent. The total objective is the contrastive loss plus λ times (AVL + PVL), with λ = 1000 selected from {100, 500, 1000, 2500, 5000} on the ImageNet validation set.

Training reuses FLYP's codebase and most of its hyper-parameters: AdamW, a cosine learning rate scheduler, batch size 512 for ImageNet and 256 for other datasets, B' = B unless noted (reduced to 256 for ViT-L/14 due to GPU memory), early stopping on ID validation performance, and results averaged over three seeds. The base model is CLIP ViT-B/16 pre-trained on LAION-400M, with CLIP ViT-L/14 also tested. Inference uses only the final fine-tuned model, so there is no extra computational cost at inference relative to a standard vision-language model.

Why This Matters

Impact on research. The paper reframes robust fine-tuning as a geometry-preservation problem and provides evidence that the relative positions of embeddings, not just their alignment, drive generalization. It also draws a clear distinction from SnD (which forces difference vectors to zero) and from cosine-similarity constraints like CyCLIP's, arguing that both are insufficient. The authors state that the official ARF implementation was not publicly available, so they used FLYP with an auxiliary contrastive loss on CC3M as a proxy — a detail worth noting when interpreting the comparisons.

Real-world applications (grounded in the benchmarks the paper uses):

• Self-driving and other dynamic-environment perception systems, which the paper explicitly cites as motivation.
• Satellite imagery classification, as tested with the FMoW dataset's 62 land and building classes.
• Wildlife monitoring with camera traps, as tested with iWildCam's 182 animal species and its camera/background/illumination distribution shifts.
• Fine-grained and domain-shifted image recognition in settings such as sketch recognition, food, flowers, pets, cars, aircraft, and remote sensing, all of which appear in the zero-shot evaluation.

Industry relevance. The method adds no inference-time cost, reuses an existing codebase and hyper-parameters, and works with a larger backbone (ViT-L/14). It also composes with weight ensembling, which is already a widely used technique. Those properties make it comparatively easy to slot into existing fine-tuning pipelines.

Future Directions

• Theoretical grounding. The authors explicitly list deepening the theoretical understanding of DiVE and clarifying the principles behind its effectiveness as future work.
• Where the structural account of generalization breaks down. SnD's failure to preserve geometry even at a score of 0.847 despite constraining difference vectors toward zero suggests the relationship between constraint strength, structural preservation, and adaptation is not fully characterized.
• Reference-data dependence. Performance rises with reference dataset size (8K to 118K to 3M), raising the question of how to obtain strong results without a large in-domain-like reference corpus.
• Hyper-parameter and architecture generality. λ was tuned on ImageNet validation and reused across datasets; whether that transfers to substantially different target domains or architectures beyond ViT-B/16 and ViT-L/14 is not established here.

Target Audience

Researchers and practitioners working on vision-language models, transfer learning, and robust fine-tuning — particularly those who care about retaining zero-shot and out-of-distribution performance after adapting a CLIP-style model to a target dataset. It is also relevant to readers interested in representation geometry in contrastive learning. Some prior familiarity with CLIP-style contrastive pre-training and baseline methods like FLYP and ARF will make the paper considerably easier to follow.

Authors’ abstract

Contrastive pre-trained vision-language models, such as CLIP, demonstrate strong generalization abilities in zero-shot classification by leveraging embeddings extracted from image and text encoders. This paper aims to robustly fine-tune these vision-language models on in-distribution (ID) data without compromising their generalization abilities in out-of-distribution (OOD) and zero-shot settings. Current robust fine-tuning methods tackle this challenge by reusing contrastive learning, which was used in pre-training, for fine-tuning. However, we found that these methods distort the geometric structure of the embeddings, which plays a crucial role in the generalization of vision-language models, resulting in limited OOD and zero-shot performance. To address this, we propose Difference Vector Equalization (DiVE), which preserves the geometric structure during fine-tuning. The idea behind DiVE is to constrain difference vectors, each of which is obtained by subtracting the embeddings extracted from the pre-trained and fine-tuning models for the same data sample. By constraining the difference vectors to be equal across various data samples, we effectively preserve the geometric structure. Therefore, we introduce two losses: average vector loss (AVL) and pairwise vector loss (PVL). AVL preserves the geometric structure globally by constraining difference vectors to be equal to their weighted average. PVL preserves the geometric structure locally by ensuring a consistent multimodal alignment. Our experiments demonstrate that DiVE effectively preserves the geometric structure, achieving strong results across ID, OOD, and zero-shot metrics.