VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

AI summary

Overview

• Research area: Video domain adaptation for large Vision Language Models (VLMs), spanning computer vision and multimodal learning.
• Technical level: Advanced — familiarity with vision encoders, cross-attention, LoRA, and autoregressive VLM training is assumed.
• Scope: The paper introduces VisCoP (Vision Contextualized Probing), a lightweight mechanism that adds learnable visual probes to a frozen vision encoder so a VLM can specialize to new visual domains while retaining its pretrained abilities.

What This Paper Is About

VLMs perform well on general visual reasoning but degrade sharply when deployed in domains that differ from pretraining, such as egocentric instead of exocentric viewpoints, depth instead of RGB, or robot control instead of human action understanding. The standard fix — finetuning parts of the VLM — forces a trade-off: freezing the vision encoder preserves pretrained knowledge but blocks domain-specific visual learning, while training the encoder enables specialization at the cost of catastrophic forgetting. VisCoP aims to break that trade-off by learning domain-specific visual features through a separate pathway that never modifies the pretrained vision encoder.

Key Contributions

1. VisCoP, a video domain adaptation strategy that learns domain-specific visual representations via layer-wise probing of a frozen vision encoder, enabling domain transfer while preventing catastrophic forgetting of pretrained multimodal capabilities.
2. A comprehensive evaluation setting for VLM domain adaptation spanning three target domains — cross-view (exocentric to egocentric), cross-modality (RGB to depth), and cross-task (action understanding to robotic control) — along with standardized metrics (Δ_target and Δ_source) for measuring improvement and retention.
3. Empirical demonstration that VLMs trained with VisCoP outperform alternative adaptation strategies across diverse target domains while retaining more source-domain knowledge.
4. Diagnostic analysis of design choices — number of probes, interaction module placement, and interaction scope (spatial, temporal, or spatio-temporal) — plus t-SNE and attention visualizations of the learned representations.

Main Findings

• Egocentric adaptation: VisCoP achieves Δ_target = +3.53 and Δ_source = +1.77, the strongest result in Table 1. Training only the vision-language connector gives Δ_target = -0.74 and Δ_source = +0.31; connector plus LLM LoRA gives +1.24 / +0.18; connector plus vision encoder gives +0.57 / -0.50; all three modules gives -0.68 / +0.68.
• Positive transfer on some source benchmarks: VisCoP not only avoids forgetting but improves some source benchmarks (e.g., ADL-X), which the authors attribute to overlapping action distributions across the viewpoint shift.
• Depth adaptation: VisCoP reaches Δ_target = +19.27 with Δ_source = +1.84, while jointly updating the vision encoder and connector yields Δ_target = +15.57 but severe forgetting (Δ_source = -3.87). Connector-only gives +16.44 / -1.42, and connector plus LLM LoRA gives +5.23 / -2.08.
• Robotic control (simulation): The base VLM scores 0% on all three VIMA-Bench levels. VisCoP reaches an average of 67.82 (Δ_target = +67.82) with Δ_source = -4.58, the best source retention among robot control experts; vision-encoder-trained experts score averages of 65.13 and 65.10 with Δ_source = -8.87 and -7.84.
• Robotic control (real world): With VIMA-Bench training only, VisCoP averages 43.33 (T1 40.00, T2 70.00, T3 20.00) versus 40.00 for the baseline, with Δ_source = -4.58 versus -8.87. Joint training on VIMA-Bench + xArm-Det raises VisCoP to 96.67 average (T1 100.00, T2 100.00, T3 90.00) versus 80.00, with Δ_source = -11.00 versus -11.04.
• Comparisons against alternatives (egocentric target): VisCoP (73.96 target / 76.19 source, +3.53 / +1.77) beats VP (65.57 / 75.05, -4.86 / +0.62), LoRA Full (69.85 / 75.35, -0.59 / +0.92), Last-4 (70.46 / 72.62, +0.02 / -1.80), VPT (71.36 / 74.97, +0.93 / +0.55), QFormer Style (70.99 / 75.03, +0.56 / +0.61), and Model Tailor (70.27 / 75.29, -0.16 / +0.86). Several alternatives underperform even the frozen base VLM (70.43 / 74.42).
• Interaction scope matters: Spatio-temporal probing (+3.53 / +1.77) outperforms spatial-only (+1.83 / +1.70) and temporal-only (-0.30 / +1.10).
• Modest overhead: VisCoP uses 27.8GB max VRAM versus 24.4GB for the base VLM, 8.21B parameters versus 8.04B (about 2% more), 0.417s full-model latency versus 0.767s, and 0.069s versus 0.056s for visual feature extraction. The authors attribute the lower full-model latency to the base VLM generating longer, less focused responses.

Methodology in Plain English

The starting model is VideoLLaMA3's architecture: a SigLIP vision encoder (embedding dimension 1152), a Qwen 2.5 LLM (embedding dimension 3584), and a 2-layer MLP vision-language connector. Instead of finetuning the vision encoder, VisCoP adds a small set of learnable tokens — 16 visual probes by default, initialized from N(0, 0.02) — plus an "interaction module" at every layer of the encoder. Each interaction module is a multi-head cross-attention whose weights are initialized from the vision encoder's self-attention weights at that layer, and it lets the probes pull domain-relevant information from the encoder's intermediate representations. Because the probes attend across all frames, they can capture spatio-temporal relationships that the encoder's intra-frame self-attention misses; in some settings such as robotic control, cross-attention is restricted to spatial tokens only. After the final layer, the updated probes are projected into the language embedding space by a dedicated connector and used as extra conditioning for autoregressive answer generation. During adaptation only the probes, interaction modules, vision-language connectors, and LoRA parameters (r = 16) in the LLM are updated, while the vision encoder stays frozen. The LLM learning rate is 1e-5 and the vision encoder rate is 2e-6 when trainable; training runs for 3 epochs on video domains and 2 epochs on robotic control on 4 NVIDIA H200 GPUs. Training data: 24,688 EgoExo4D videos expanded into 74,064 instruction pairs (45,888 egocentric and 28,176 depth pairs generated with DepthAnythingV2); VIMA-Bench with 17 manipulation tasks and LLaRA instructions yielding 13,922 pairs over 7,995 trajectories; and the real-world xArm-Det dataset of 1,007 instruction pairs collected with a 6-DoF xArm 7, for 14,929 pairs when trained jointly. Evaluation uses Ego-in-Exo Perception (3,881 video-QA pairs in four categories) and EgoSchema (5,031 pairs) as targets, and NeXTQA (8,564 QA), VideoMME (2,700 QA), and ADL-X (10,561 multiple-choice and 1,862 description questions) as source benchmarks. A t-SNE analysis with 95% confidence ellipses compares source and target feature distributions using Bhattacharyya distance and per-sample distance, and attention maps compare visual-probe attention with language-token attention to visual embeddings. Results of the ablations on probe count and interaction module placement (Figure 4) are not specified in the available text.

Why This Matters

• Research impact: The paper reframes VLM domain adaptation as a pathway problem rather than a which-components-to-train problem, showing that intermediate encoder representations can be exploited for specialization without overwriting pretrained weights. The Δ_target / Δ_source metric pair also offers a standardized way to report the adaptation-versus-forgetting trade-off.
• Real-world applications:
  • Wearable and smart-glasses assistants that reason over egocentric video while the same model still handles standard web video.
  • Depth-sensing robots and AR/VR devices that need RGB-trained perception to transfer to depth input.
  • Robot manipulation policies that inherit broad human-activity understanding instead of being trained from scratch.
  • Deployment pipelines that must keep a single shared VLM backbone across multiple specialized domains without maintaining separate finetuned copies that lose general capability.
• Industry relevance: The 2% parameter increase and the lower observed full-model inference latency mean adaptation adds little serving cost, and because the pretrained encoder is never modified, domain experts can be trained and swapped as additive modules on a shared backbone.

Future Directions

• Extending beyond the three studied shifts: The paper covers cross-view, cross-modal, and cross-task adaptation; whether the same probe pathway works for other distribution shifts (lighting, sensor noise, language shifts) is not reported.
• Scaling and tuning the probe design: The available text truncates the ablations on probe count and interaction module placement, so the sensitivity of VisCoP to these choices remains an open question.
• Reducing source-domain forgetting further: Source retention is still negative in the robotic control settings (Δ_source = -4.58 for simulation, -11.00 under joint training), leaving room to close the remaining gap.
• Handling joint shifts: The authors note the multi-axis nature of the egocentric shift; a natural next step is testing VisCoP when viewpoint, modality, and task change simultaneously rather than one at a time.

Target Audience

Researchers and engineers working on multimodal models, video understanding, and domain adaptation, especially those who need to specialize VLMs to new visual domains without losing general capability. It is also relevant to practitioners deploying VLMs on robots, wearables, or depth-sensing hardware, and to readers interested in parameter-efficient adaptation methods such as probing, prompt tuning, and LoRA.

Authors’ abstract

Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.