Data or Language Supervision: What Makes CLIP Better than DINO?

AI summary

Overview

Research area: computer vision, specifically vision encoder design for vision-language models (VLMs). Technical level: Intermediate. Scope: a controlled experimental study that trains CLIP and DINO under identical architecture, dataset, and training configuration to isolate whether CLIP's advantage as a VLM vision encoder comes from language supervision or from its much larger training data.

What This Paper Is About

CLIP is widely reported to be a better vision encoder than DINO for building vision-language models, but the two models differ in two ways at once: CLIP uses image-text contrastive learning while DINO uses image-only self-supervision, and CLIP-family models (including SigLIP) are trained on datasets up to 100 times larger than DINO's. The paper asks which factor actually drives CLIP's downstream advantage by removing the data-scale difference entirely and comparing the two supervision styles head to head.

Key Contributions

1. Controlled pretraining of both paradigms. The authors train CLIP and DINO from scratch with the same ViT-B/16 backbone, the same 10M-image subset of DataComp, and the same training configuration (20 epochs, AdamW, learning rate 1e-3, cosine decay, 4 A100 GPUs over 3 days), reaching comparable ImageNet linear probing accuracy (CLIP 65.8%, DINO 66.4%).

2. Embedding-level analysis of what language supervision changes. Using image pairs where the two models disagree strongly, the paper shows CLIP organizes its embedding space around high-level semantics (object category, embedded text) while DINO is more responsive to low-level visual attributes (color, style), with quantitative cosine-similarity experiments in the appendix supporting the qualitative examples.

3. Downstream VLM comparison on 20 benchmarks. Both encoders are plugged into LLaVA-1.5 and evaluated on the 20 subsets of VMCBench, revealing that CLIP's benefit is concentrated in text-intensive tasks rather than being uniform across the board.

4. Tests of alternative language supervision. Replacing the contrastive loss with a sigmoid-based SigLIP loss, or replacing CLIP's randomly initialized text encoder with a frozen pretrained Vicuna-7B, both fail to improve results.

Main Findings

• Matched ImageNet accuracy under matched conditions. Trained identically except for the supervision signal, controlled CLIP and controlled DINO score 65.8% and 66.4% on ImageNet linear probing. For contrast, the official pretrained models score 79.5% (CLIP) and 76.1% (DINO), showing that the official gap cannot be attributed purely to supervision type.

• Language supervision helps most on fine-grained recognition. CLIP reaches 74.7% on Stanford Cars versus DINO's 54.1% (+20.6%) and 52.3% on CUB versus 43.0% (+9.3%). On general benchmarks the two are close (CIFAR-10: 90.7% vs 92.1%; Flowers: 78.7% vs 80.7%) and on robustness benchmarks they are comparable (ImageNetV2: 53.0% vs 53.5%; CIFAR10.1: 82.8% vs 86.0%; controlled DINO is higher on the latter two).

• Different embedding geometry, not just different accuracy. CLIP embeds semantically distinct symbols closer to separation than DINO does; in an experiment with images of unique letters and numbers, average pairwise cosine similarity was 0.713 for CLIP versus 0.877 for DINO. For images of simple repeating visual patterns with minimal semantic content, the trend reversed (CLIP 0.497, DINO 0.478).

• Downstream VLM gains are task-dependent. On VMCBench, CLIP and DINO are nearly tied on reasoning (DINO 41.5% vs CLIP 41.2%), general VQA (CLIP 46.2% vs DINO 46.0%), and document/chart understanding (CLIP 33.2% vs DINO 33.1%). The large gap appears in OCR: LLaVA-CLIP averages 47.5% on OCRVQA and TextVQA versus 40.0% for LLaVA-DINO, a 7.5 percentage-point difference.

• Stronger or different language supervision does not help. In the VLM evaluation, baseline CLIP averages 41.4, SigLIP loss 40.8, and the pretrained Vicuna-based language encoder 40.5, so neither variant beats the standard CLIP setup.

• The LLM backbone changes the picture. With Vicuna-7B, CLIP and DINO are essentially tied on general VQA (46.23% vs 46.20%) and average 41.44 versus 40.71 overall. With Qwen2-7B, CLIP leads general VQA at 57.90% versus 54.02% (a 3.88 percentage-point gain) and averages 49.69 versus 47.72.

Methodology in Plain English

The authors remove the confound that makes existing comparisons unfair. Instead of studying released CLIP and DINO checkpoints—which differ in architecture, data, and training recipe—they train both models from scratch on exactly the same 10M-image DataComp subset with the same ViT-B/16 backbone, the same optimizer settings, and the same 20-epoch schedule. The only difference is that CLIP also receives the paired captions as supervision while DINO receives images alone.

They then verify the comparison is fair by checking that the two models are equally good at a generic task (ImageNet linear probing), and only then move to more diagnostic measurements. First, they mine image pairs from the training pool where the two models' similarity scores strongly disagree, defined by thresholds (CLIP similarity above 0.8 while DINO is below 0.5, and the reverse), and inspect what those pairs have in common. They back this up quantitatively by measuring average pairwise cosine similarity on two small purpose-built image sets: one of distinct letters and digits, one of simple repeating visual patterns.

Finally, they swap each encoder into LLaVA-1.5, train identically, save checkpoints every 500 steps, and score them on the 20 datasets of VMCBench. Because test labels are not public, they select the best checkpoint by validation performance and report validation results. A further experiment substitutes the sigmoid SigLIP loss and a frozen pretrained Vicuna-7B text encoder to test whether other forms of language supervision do better.

Why This Matters

The paper separates two explanations that the literature usually conflates—supervision objective versus data scale—and shows that the often-cited CLIP-over-DINO gap is not simply a property of language supervision. This gives vision encoder designers a more precise target: language supervision buys semantic abstraction and text handling, while data scale drives general classification and robustness. It also implies that adding more language-side machinery (SigLIP loss, pretrained text encoders) is not automatically beneficial.

Real-world applications:

• Document and OCR-heavy systems, such as receipt, invoice, and form processing, where the encoder must read text embedded in images.
• Fine-grained visual categorization, such as identifying car models, bird species, or product variants that differ only in subtle details.
• Multimodal assistants and visual question answering, where the choice of vision encoder and LLM backbone jointly determines accuracy on scene understanding.
• Encoder selection for constrained training budgets, since the study shows what can be expected from image-only pretraining when large caption datasets are unavailable.

Industry relevance: companies building VLMs must pick a vision encoder and decide where to spend compute—on more image-text pairs or on a better training objective. This paper's controlled results suggest the data budget matters most for broad capability, while language supervision matters most for text-in-image and fine-grained tasks, which informs those trade-offs directly.

Future Directions

• Scale the controlled comparison beyond the 10M-image subset to billion-image datasets, which the authors explicitly name as a crucial next step; they note a concurrent work (Fan et al., 2025) that scaled DINO and CLIP to 7B parameters on 8B data.
• Develop hybrid approaches that strategically combine self-supervised and language-supervised signals rather than choosing between them.
• Investigate why a stronger LLM backbone such as Qwen2-7B extracts more benefit from CLIP's high-level semantic representations than Vicuna-7B does on general VQA.
• Determine why alternative language supervision forms (SigLIP loss, pretrained language encoders) do not translate into VLM gains, since the paper reports the negative result without a full mechanistic explanation.

Target Audience

Researchers and engineers working on vision-language models, multimodal representation learning, and vision encoder design will benefit most. The paper is also useful for practitioners choosing between CLIP-style and self-supervised encoders for a specific downstream application, and for students who want a clear example of how to design a controlled experiment that disentangles two confounded factors. Some familiarity with contrastive learning and VLM architecture helps, but the framing and results are accessible without deep expertise.

Authors’ abstract

CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP's language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings -- using the same architecture, dataset, and training configuration -- achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance.