VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models

AI summary

Overview

• Research area: Self-supervised learning and transfer learning for visual foundation models (VFMs), specifically label-free domain adaptation using object-centric video.
• Technical level: Intermediate. The paper assumes familiarity with self-supervised distillation (DINO), Vision Transformers, and parameter-efficient fine-tuning (LoRA), but the core idea is explained clearly in the text.
• Scope (one sentence): The paper proposes VESSA, a method that fine-tunes a pretrained vision foundation model to a new domain using only short, unlabeled, multi-view object-centric videos, and evaluates it on three foundation models and two datasets.

What This Paper Is About

Vision foundation models are pretrained on large image datasets, but they can underperform when applied to specialized domains whose visual statistics differ from the pretraining data. Supervised fine-tuning fixes this but requires labels, which are often expensive or impossible to collect. VESSA adapts a pretrained model to a target domain with no annotations at all, by exploiting the multiple views of an object that naturally appear across frames of a short object-centric video.

Key Contributions

1. A new formulation of self-supervised fine-tuning for vision foundation models. The paper frames domain adaptation as a fine-tuning problem (not building a new foundation model), where the model is adapted to a target domain without any annotations, using only short multi-view object-centric videos.
2. A video-based positive-pair construction. Instead of treating two augmentations of the same image as a positive pair (as DINO does), VESSA samples two temporally separated frames from the same video, so the model learns robustness to genuinely different capture conditions and viewpoints rather than only synthetic image transformations.
3. An Uncertainty-Weighted Self-Distillation (UWSD) loss plus a staged unfreezing schedule. The loss reweights each sample by the entropy of the teacher's output distribution, and the training procedure first trains only the projection head, then gradually unfreezes the network — applying LoRA to the first H layers (Query, Key, Value projections) while keeping normalization layers trainable, and fully unfreezing the last L layers.
4. An empirical demonstration that naive self-supervised continued training on images degrades performance. The paper isolates this "Static-baseline" and shows that straightforward image-based continued fine-tuning generally does not help, whereas the video-based version does.

Main Findings

• Video beats images for the same adaptation recipe. On MVImageNet with ViT-S/16 (k-NN, k=1), DINO trained from scratch reached 33.86% with images but 39.39% with video (a 5.53 percentage-point gain). Switching from image input with local crops (88.54%) to video input without local crops (90.53%) gave a gain of 1.99 p.p.
• Training the prediction head is the single most important design choice. Using video input with local crops but without training the head gave 80.87%; enabling head training in the same setup raised accuracy to 91.87%, a difference of approximately 10 p.p.
• Unfreezing the right number of layers matters. In the ablation, unfreezing the last 1, 2, 3, and 4 layers yielded 87.14%, 91.87%, 90.80%, and 90.55% respectively; the optimal setting was the last 2 layers.
• The uncertainty weighting contributes. Removing the UWSD loss from the full configuration gave 90.92% versus 91.87% for the complete method.
• Full VESSA improves over the pretrained baseline. The best configuration reached 91.87% on MVImageNet, 2.18 p.p. above the pretrained DINO baseline (89.69%).
• Randomized temporal gaps help. On CO3D, the best frame-distance strategy was sampling δ randomly from [5, 10], giving 85.03% with DINO and 91.85% with DINOv2, compared with, for example, δ=20 (84.82% / 91.23%) and random [10, 30] (82.46% / 91.52%).
• VESSA leads on CO3D, with statistical significance. With DINOv2, VESSA achieved 91.85% ± 0.56 versus ExPLoRA + video at 89.64% ± 0.47 — a 2.21 p.p. gap that the paper reports as statistically significant.
• On MVImageNet the top result is a statistical tie. VESSA reached 96.01% ± 1.08 with DINOv2, while ExPLoRA + video reached 96.15% ± 0.87; the difference is reported as not statistically significant. For DINO, VESSA outperformed ExPLoRA (92.51% ± 1.11 versus 87.74% ± 1.03) and also improved over the pretrained DINOv2 for TIPS (80.54% vs 78.65% pretrained TIPS).
• Image-based continued training is unreliable. The Static-baseline reached 80.31% / 81.60% / 55.59% on CO3D (DINO/DINOv2/TIPS) and 89.39% / 92.53% / 76.05% on MVImageNet, below or near the pretrained models in several cases.
• Synthetic motion augmentation does not reproduce the video benefit. Adding translations up to 10% of image dimensions, rotations up to 10 degrees, scaling up to 5%, brightness shifts of 0.1 and contrast in 0.9–1.1 gave 80.60% (DINO) and 81.49% (DINOv2) on CO3D, barely different from the Static-baseline.
• Qualitative retrieval shows object focus. The paper reports that pretrained DINOv2 embeddings tend to retrieve by background similarity, while VESSA attends more to the object of interest, even when texture or color differs.

Methodology in Plain English

VESSA takes a pretrained foundation model and adapts it to a target domain using only short videos of single objects.

1. Frame selection. From each video, the method samples pairs of frames. A starting frame index is drawn uniformly, and a temporal gap δ is also drawn uniformly, so the two frames in a pair span a variable amount of time and therefore show the object from different angles or under different lighting.
2. Preprocessing and augmentation. Each frame in a pair gets its own random global augmentation, and the pair is also used to produce local crops — small crops sampled in pairs, one from each frame, so the temporal linkage is preserved at the local level.
3. Self-distillation. Two identical networks (student and teacher) are initialized from the same pretrained model, with LoRA modules inserted for parameter-efficient updates, and the teacher is updated as an exponential moving average of the student. The student sees one frame, local crops, and the teacher sees the other frame; the student is trained to match the teacher's output distributions using the DINO cross-entropy objective.
4. Uncertainty weighting. Each sample's contribution to the loss is scaled by 1 + γ·H(q), where H(q) is the entropy of the teacher's prediction; the hyperparameter γ was set to 1. This gives harder, more uncertain examples more weight.
5. Careful optimization. Because starting from a pretrained model with a randomly initialized projection head can destabilize training, the backbone is frozen initially and only the head is trained. Then the network is unfrozen in stages: the first H layers get LoRA updates on the Query, Key, and Value attention projections (with normalization layers trainable), and the last L layers are fully updated.

Experiments used ViT-Base backbones, 10 training epochs for the projection-head adaptation and 10 for the full model training, a batch size of 256, input resolution 224×224, and 3 frame pairs sampled per video. Training ran on TPU v3-8 (8 cores, 128 GB memory) using the scenic library in JAX. Evaluation was k-Nearest Neighbors with k=1 on held-out splits (each class split 75%/25%), and statistical significance was checked with an unpaired Student's t-test at 90% confidence over three independent runs.

Why This Matters

• Impact on research. The paper shows that the NLP-style recipe of continued self-supervised pretraining does not transfer automatically to vision encoders — a naive version degrades the model — and that the missing ingredient is multi-view video data plus careful optimization. This reframes unsupervised adaptation for vision as a viable research direction rather than a dead end.
• Real-world applications (areas where labeled data are scarce and distribution shift is common, as identified in the paper):
  • Remote sensing, where imagery differs sharply from natural-image pretraining data.
  • Medical imaging, where annotations require expert time and cost.
  • Place recognition, where capture conditions vary widely.
  • Any deployment where a practitioner can record short object videos but cannot label them.
• Industry relevance. Because the method uses LoRA for parameter-efficient updates and needs no annotations, adaptation becomes cheap in storage and label cost. The video data required can be captured with an ordinary camera, and the resulting model is reported to improve downstream k-NN classification without any labeled fine-tuning stage. The code is publicly available at the stated repository URL.

Future Directions

• Reducing catastrophic forgetting. The authors explicitly name the tendency to forget previously acquired knowledge during fine-tuning as a limitation, and the supplementary material includes a study of how far general-purpose performance degrades.
• Removing the multi-view video requirement. The paper notes that datasets offering multiple viewpoints of the same object are not commonly available, which limits applicability; finding ways to obtain comparable view diversity from ordinary data is an open question.
• Understanding why video helps. The experiment with synthetic motion-inspired augmentations (translations, rotations, scaling, brightness, contrast) did not reproduce the gains, suggesting the benefit comes from cues beyond simple geometric or photometric variation. Identifying those cues remains unresolved.
• Broadening the evaluation. The paper evaluates 3 foundation models on 2 datasets and notes that most prior video-based methods target pixel-level tasks like segmentation and detection while showing limited gains for frame-level classification; extending VESSA to those dense prediction tasks is a natural next step.

Target Audience

Researchers and practitioners working on self-supervised learning, transfer learning, or domain adaptation for vision models; engineers who need to specialize a pretrained model in a setting where labels are unavailable but short videos can be collected; and readers interested in the contrast between how adaptation is done in NLP versus computer vision.

Authors’ abstract

Foundation models have advanced computer vision by enabling strong performance across diverse tasks through large-scale pretraining and supervised fine-tuning. However, they may underperform in domains with distribution shifts and scarce labels, where supervised fine-tuning may be infeasible. While continued self-supervised learning for model adaptation is common for generative language models, this strategy has not proven effective for vision-centric encoder models. To address this challenge, we introduce a novel formulation of self-supervised fine-tuning for vision foundation models, where the model is adapted to a new domain without requiring annotations, leveraging only short multi-view object-centric videos. Our method is referred to as VESSA: Video-based objEct-centric Self-Supervised Adaptation for visual foundation models. VESSA's training technique is based on a self-distillation paradigm, where it is critical to carefully tune prediction heads and deploy parameter-efficient adaptation techniques - otherwise, the model may quickly forget its pretrained knowledge and reach a degraded state. VESSA benefits significantly from multi-view object observations sourced from different frames in an object-centric video, efficiently learning robustness to varied capture conditions, without the need of annotations. Through comprehensive experiments with 3 vision foundation models on 2 datasets, VESSA demonstrates consistent improvements in downstream classification tasks, compared to the base models and previous adaptation methods. Code is publicly available at https://github.com/jesimonbarreto/VESSA.