Revisiting Multi-Task Visual Representation Learning

AI summary

Overview

• Research area: Computer vision, specifically large-scale visual pretraining and multi-task representation learning for general-purpose visual encoders.
• Technical level: Advanced. The paper assumes familiarity with vision transformers, contrastive vision-language pretraining (CLIP/SigLIP), and self-supervised methods (DINO, MAE), though the core ideas are explained here in plain language.
• Scope: The paper proposes MTV, a single pretraining framework that jointly optimizes vision-language contrastive, self-supervised, and dense spatial (grounding plus depth) objectives on a shared backbone, and then systematically studies how those objectives interact and scale. (arXiv:2601.13886v1, cs.CV, published 20 Jan 2026, CC BY 4.0; authors Shangzhe Di and Weidi Xie of SAI, Shanghai Jiao Tong University, and Zhonghua Zhai of ByteDance Seed; code at github.com/Becomebright/MTV.)

What This Paper Is About

Visual pretraining is currently split between two families of methods: vision-language models such as CLIP, which align whole images with text and excel at global semantics but are weak at spatial precision, and self-supervised methods such as MAE and DINO, which capture fine local structure but lack high-level semantic context. The paper argues these paradigms are complementary and asks whether they can be combined, along with dense spatial supervision, into one principled multi-task framework. The goal is to build a single visual encoder that keeps strong semantic understanding while gaining fine-grained spatial and geometric reasoning, using pseudo-labels synthesized by large expert models instead of manual annotation.

Key Contributions

1. A unified multi-task pretraining framework (MTV). A shared ViT backbone is optimized jointly over three supervision types: vision-language contrastive learning (sigmoid-based, following SigLIP), self-supervised learning (local-to-global self-distillation plus masked feature prediction plus a KoLeo regularizer), and dense structured supervision (region-level grounding plus per-pixel depth regression), with uniform loss weighting and no exhaustive hyperparameter tuning.
2. Scaling dense supervision with expert pseudo-labels. Rather than human labels, the pipeline uses RAM++ to extract salient entity names, OWLv2-Base to localize the corresponding regions, and Depth-Anything-V2-Large to produce relative depth maps, yielding 100M "fully-labeled" samples each carrying a caption, region-text pairs, and a depth map.
3. A systematic study of multi-task mechanics. The paper quantifies (i) the marginal gain of each objective, (ii) task synergy versus interference via a dedicated Synergy metric, and (iii) scaling behavior across data scales (10M, 50M, 100M) and model sizes (ViT-B/16, ViT-L/16).
4. Empirical "best-of-both-worlds" evidence. Multi-task training improves spatial reasoning without sacrificing global semantics, and the ViT-Base model trained on 100M samples outperforms CLIP-Base trained on 400M samples on ImageNet zero-shot accuracy.

Main Findings

• Every added objective helps. Along the expansion path VL → +SSL → +Grounding → +Depth, performance improves almost monotonically at every step across all four model/data settings. The overall gain over the VL-only baseline (Absolute Gain Δ) is +13.5 ImageNet top-1 for ViT-B/16 at 10M, +7.2 for ViT-B/16 at 50M, +19.3 for ViT-L/16 at 10M, and +7.1 for ViT-L/16 at 50M.
• SSL is the largest single lever. Adding SSL lifts ViT-L/10M by +14.4% on ImageNet-1k and by +10.4 mIoU on ADE20k, which the authors read as local structural regularities providing a geometric scaffold for global semantics. They caution that part of this jump reflects the chosen expansion order and the spatial sparsity of the VL-only baseline.
• Grounding bridges semantics and space. Because grounding is a fine-grained vision-language signal, it benefits spatial tasks and global multimodal understanding alike, including a +3.8 R@1 gain on COCO image-to-text retrieval in the ViT-L/10M setting.
• Depth refines pixel-level geometry. Depth supervision gives the final refinement for pixel-level tasks, improving NYUv2 by up to 0.051 RMSE in the ViT-L/50M setting.
• Tasks are synergistic, not adversarial. Using Synergy(A,B) = (ΔAB − max(ΔA, ΔB)) / max(ΔA, ΔB), all task pairs show positive average synergy across data and model scales, typically ranging from 20% to 50%. Pairs involving grounding are particularly robust. Individual cells can be negative, for example −10.0 for SSL+Depth on NAVI at ViT-B/16 10M and −34.7 for SSL+Depth on SPair at ViT-L/16 50M.
• Larger capacity leaves less relative headroom. Synergy scores are generally lower for ViT-L/16 than for ViT-B/16, which the authors attribute to higher-capacity VL baselines already capturing more of the signal, even as absolute performance keeps rising.
• Multitask supervision is more data-efficient than pure VL. In the paper's reported comparison, the ViT-Base model trained on 100M samples beats CLIP-Base trained on 400M samples on ImageNet zero-shot accuracy. The introduction states 69.4% for this model; Table 3 and Table 5 report 69.5% for MTV B/16 at 100M versus 68.3% for CLIP B/16 at 400M.
• Correspondence tasks do not scale naively. Geometric and semantic correspondence saturate and begin to decline once the number of seen samples exceeds 1B, suggesting these tasks need targeted architecture or supervision rather than more data alone.
• Results against large-scale VLMs. MTV at 100M training samples beats CLIP (400M) on global tasks and decisively beats CLIP, SigLIP, and SigLIP2 on correspondence and depth. It lags SigLIP and SigLIP2, which are trained on 10B samples (100× more data), and it trails them on semantic segmentation. For example, MTV L/16 (100M) reaches 75.2 ImageNet top-1, 45.0 NAVI recall, 30.1 SPair recall, and 0.414 NYUv2 RMSE, versus SigLIP2 L/16 (10B) at 82.5, 43.2, 29.9, and 0.484 respectively.
• Modest training overhead. For ViT-B/16, going from VL alone (203.2M trainable parameters, 1.0× time) to VL+SSL (232.6M, +29.4M, 1.2×) to adding grounding (239.7M, +7.1M, 1.5×) to adding depth (250.3M, +10.6M, 1.7×) roughly doubles training time in total, with the frozen EMA teacher excluded from the trainable count.

Methodology in Plain English

The researchers start from a corpus of web image-text pairs drawn from DataComp (1.1B pairs in the source; they crawled 200M images over roughly 40 days, about 21 TB). They filter out images whose shorter side is under 224 pixels and downsample high-resolution images to a maximum dimension of 1024 pixels while keeping aspect ratios, partly so the expert teacher models operate in their optimal range. Then they attach three kinds of automatically generated labels to each surviving image: the original web caption, region-text pairs, and a relative depth map. Only 100M samples end up "fully labeled" and are used afterward.

The shared image encoder is a ViT (ViT-B/16-224 and ViT-L/16-256) with a SigLIP-style text encoder and tokenizer. Three losses are summed with equal weight:

• A vision-language contrastive loss aligns pooled image embeddings with text embeddings using the sigmoid pairwise objective from SigLIP (logit scale and bias initialized from log 10 and −10). Text embeddings are synchronized across devices with a differentiable all_gather rather than SigLIP's cyclic rotation protocol, because the communication cost is negligible relative to the multi-task backbone.
• A self-supervised loss uses a student-teacher setup in which the teacher is an exponential moving average of the student (momentum 0.994) with 32k-dimensional prototype heads. It combines DINO-style local-to-global self-distillation over M = 6 local crops with masked feature prediction over 50% randomly masked patches (using masked-patch KL terms), plus a KoLeo regularizer weighted at 0.1 and a mask loss weighted at 2. Centering (momentum 0.9) and sharpening (student temperature τ_s = 0.1; teacher temperature warmed from τ_t = 0.04 to 0.07) prevent collapse.
• A dense structured loss contains region-level grounding and pixel-level depth. For grounding, up to four region-text pseudo-labels per image are encoded by a lightweight Transformer "Prompter" (the CLOC approach) that conditions on bounding-box location embeddings and a learnable query token, producing soft region features aligned with text via the same sigmoid contrastive objective. For depth, a lightweight DPT head fuses features from four intermediate ViT layers ({3, 6, 9, 12} for ViT-B/16 and {6, 12, 18, 24} for ViT-L/16) and is supervised with the MiDaS scale- and shift-invariant loss plus a multi-scale gradient loss. A top-K denoising strategy discards the highest 10% of per-pixel losses within each sample to protect against pseudo-label artifacts.

Training uses NVIDIA H200 GPUs, AdamW at a learning rate of 5×10⁻⁴, no weight decay, linear warmup over the first 1k steps, and BF16 mixed precision except the DPT depth head, which stays in FP32 for numerical stability. The 10M and 50M scales are used mainly for ablations (20 and 10 epochs, global batch size 4k); the full 100M model trains for 32 epochs at global batch size 8k, following the standard CLIP protocol.

Evaluation covers zero-shot ImageNet-1k classification, COCO retrieval (Recall@1 in both directions), vision-centric VQA through a frozen encoder plugged into TinyLLaVA with a Qwen2.5-3B language model (trained on LLaVA-558K then LLaVA-NEXT-779K, evaluated on MMVP, CVBench, and RealWorldQA), ADE20k semantic segmentation with a frozen encoder and linear head (40k steps, batch 32, learning rate 4e-5, 576×576 input), geometric correspondence on NAVI and semantic correspondence on SPair via the Probe3D framework, and monocular depth on NYUv2 using a linear head over 256 discretized bins.

Why This Matters

The paper reframes multi-task visual pretraining from an engineering trick into a design principle: heterogeneous objectives, even when supervised only by imperfect pseudo-labels, push toward a unified multi-granularity representation space. For research, it provides a controlled setting and a quantitative Synergy metric showing that global semantic and dense spatial objectives cooperate rather than compete, and it shows that pseudo-labeled supervision can substitute for scarce human annotation at scale. It also reports a specific, reproducible counterpoint to pure vision-language scaling — 100M multitask samples beating 400M VL samples on classification — while being candid that some geometric tasks stop scaling.

Real-world applications implied by the evaluated capabilities:

• Cross-modal search and retrieval: stronger image-to-text and text-to-image recall on COCO for photo and media libraries.
• Depth-aware perception: improved monocular depth on NYUv2 for robotics, AR, and 3D scene understanding.
• Dense scene parsing: ADE20k semantic segmentation for autonomous driving and image editing pipelines.
• Visual question answering: vision-centric VQA scores that feed into multimodal assistants built on an LLM backbone.
• 3D correspondence: NAVI and SPair matching for pose estimation, 3D reconstruction, and one-shot part transfer.

Industry relevance: the pipeline is deliberately cheap to scale. Depth generation runs at about 120K images per GPU-hour (13 hours for 100M images on 64 GPUs) and grounding at about 18K images per GPU-hour (86 hours for 100M images on 64 GPUs), so pseudo-label generation is a tractable cost. Trainable-parameter overhead is small (a 250.3M-parameter ViT-B/16 configuration for the full task set versus 203.2M for VL alone), making the recipe attractive to teams that already have a vision-language training stack and want spatial capabilities without rebuilding.

Future Directions

• Fix the correspondence bottleneck. Geometric and semantic correspondence saturate past 1B seen samples, so the authors call for more targeted architectural designs or supervision rather than naive data scaling.
• Close the semantic segmentation gap. MTV trails SigLIP and SigLIP2 on ADE20k; understanding whether this is a data-scale issue (100M versus 10B) or an objective-design issue is left open.
• Push toward billion-scale pseudo-labeling. Both the depth and grounding pipelines are described as fast enough to support billion-scale production, but the paper only reports results up to 100M fully-labeled samples.
• Better teachers and richer tasks. The framework's gains depend on expert models (Depth Anything V2, OWLv2); stronger or additional open-vocabulary teachers, and additional task types beyond grounding and depth, are natural extensions.
• Move beyond uniform loss weighting. Uniform weighting worked well here with no hyperparameter tuning, but whether learned or scheduled task weighting improves the synergy-interference trade-off at larger scales is untested.

Target Audience

This paper is most useful to machine learning researchers and engineers working on visual foundation models, particularly those already familiar with CLIP/SigLIP-style pretraining and DINO/MAE-style self-supervision who want to understand how to combine them. It is also relevant to practitioners building multimodal or spatial perception systems who need a data-efficient alternative to pure vision-language scaling, and to students studying multi-task learning trade-offs, since much of the value is in the ablation tables and the Synergy metric rather than the framework alone. Readers without a background in transformer-based pretraining will find the loss formulations dense, though the experimental conclusions are stated in accessible terms.

Authors’ abstract

Current visual representation learning remains bifurcated: vision-language models (e.g., CLIP) excel at global semantic alignment but lack spatial precision, while self-supervised methods (e.g., MAE, DINO) capture intricate local structures yet struggle with high-level semantic context. We argue that these paradigms are fundamentally complementary and can be integrated into a principled multi-task framework, further enhanced by dense spatial supervision. We introduce MTV, a multi-task visual pretraining framework that jointly optimizes a shared backbone across vision-language contrastive, self-supervised, and dense spatial objectives. To mitigate the need for manual annotations, we leverage high-capacity "expert" models -- such as Depth Anything V2 and OWLv2 -- to synthesize dense, structured pseudo-labels at scale. Beyond the framework, we provide a systematic investigation into the mechanics of multi-task visual learning, analyzing: (i) the marginal gain of each objective, (ii) task synergies versus interference, and (iii) scaling behavior across varying data and model scales. Our results demonstrate that MTV achieves "best-of-both-worlds" performance, significantly enhancing fine-grained spatial reasoning without compromising global semantic understanding. Our findings suggest that multi-task learning, fueled by high-quality pseudo-supervision, is a scalable path toward more general visual encoders.