TRANSPORTER: Transferring Visual Semantics from VLM Manifolds

AI summary

Overview

Research area: Computer vision and multimodal AI — specifically interpretability of Vision Language Models (VLMs) applied to video, combined with text-to-video (T2V) generative modeling.

Technical level: Advanced. The paper assumes familiarity with flow matching, variational autoencoders, optimal transport, and transformer/MLP-Mixer architectures.

Scope: The paper introduces a new task called logits-to-video (L2V) and a model named TRANSPORTER that generates videos representing what a VLM's token logits predict, evaluated across three VLMs on visual-quality and semantic-alignment metrics.

What This Paper Is About

Current Vision Language Models can answer questions about complex video scenes, but researchers lack good ways to see why a model produced a particular answer. Existing explanation methods produce text descriptions, saliency maps, or abstract shapes that are sensitive to small changes, short, or unfaithful to the model's internal state. This paper argues that since VLMs answer as probability distributions over a token dictionary, the most direct explanation is a video that visually reflects those logits. The goal is to build a system that takes the logit difference between two captions (for example, "young" versus "old", or "swift" versus "slow") and generates a video in which only the corresponding visual attribute changes while the rest of the scene stays intact.

Key Contributions

1. L2V task formulation: A new controlled video generation task for VLM visual explanations, where the conditioning signal is the divergence between VLM logit distributions rather than text alone.
2. TRANSPORTER model: A VLM-independent method that learns optimal transport (OT) couplings between a generative model's high-fidelity visual latent space and a VLM's high-semantic embedding space, so that VLM logits can define embedding directions for video generation.
3. Learnable concept bank: A set of concept vectors Q = {q_o : o ∈ O} trained to encode latent directions corresponding to contrastive token pairs (such as "baseball hit" versus "baseball miss"), supervised by the logit divergence between those pairs.
4. Learnable ρ-OT module: An entropic optimal-transport module with P learnable projection vectors that combines globally projected embeddings and local token-structure-preserving embeddings, using partial double-stochastic iterations rather than solving the full doubly-constrained problem.
5. Empirical evaluation: Quantitative and qualitative comparisons against adapted attribution and concept-based baselines across three VLMs, plus ablations on coupling components, projection count, and divergence metrics.

Main Findings

• Large gains in condition alignment: On VideoLLaMA 3, CLIP^v scores reach 35.44 for TRANSPORTER versus 16.74 for the best Activation-Maximization baseline — described in the paper as a two-fold improvement. Aesthetic scores rise from 2.56 to 4.28, and the divergence score Δ rises from 2.28 to 12.62.
• Gemma 3 conditioning is slightly stronger: TRANSPORTER reaches a CLIP^v score of 36.18, aesthetic 4.21, and Δ of 11.56, against a baseline of 17.41, 2.67, and 1.82.
• Phi 4 MM shows the largest reported gains: CLIP^v 35.71 versus 15.06, aesthetic 4.18 versus 2.24, and Δ 11.35 versus 1.73.
• Video fidelity improves substantially: FVD drops from 5.18e3 to 1.25e2 for VideoLLaMA 3, from 2.42e3 to 1.05e2 for Gemma 3 (reported as a −2.31e3 decrease relative to the best baseline), and from 2.58e3 to 1.42e2 for Phi 4 MM. LPIPS^v falls to 1.67, 1.43, and 1.54 respectively, and is reported as consistently reduced by −5.29 for VideoLLaMA 3 relative to the ω⁺ and ω⁻+Δω AM settings.
• Embedding distributions match real videos far better: Against 1K VidChapters7M videos, cosine similarity improves from 6.45e-8 to 3.28e-2 (VideoLLaMA 3), 2.31e-8 to 1.58e-2 (Gemma 3), and 1.31e-8 to 1.25e-2 (Phi 4 MM). KL divergence falls from 56.31 to 1.67, 73.00 to 4.00, and 45.39 to 3.66.
• Baselines fail to produce distinct frames: In the qualitative comparison on the token "walk", the AM baseline generates a video containing only an abstract shape, while L2V produces detailed full scenes that capture target attributes and can interpolate between "walk" and "run".
• Attribute types differ in how strongly they are encoded: General attributes such as object count ("two" versus "one") exhibit greater variance in logit divergence, while finer details such as cutting "thin" versus "thick" strips appear only for a fraction of Δω values.
• Modulation timing matters: Large distribution shifts appear immediately in generation, while finer details emerge in later steps and become progressively less visible as modulations are added downstream.
• Partial conditioning works: Modulating only a fraction of frames shows how a spray bottle blends into a "wipe" to "spray" transition, indicating that VLMs bind objects to adjacent actions across time.
• Ablation — coupling components: Using Φ_Ω1 and Φ_Ω2 together (mean⟨Φ_Ω1, Ω2⟩) gives cosine similarity 0.23 on VideoLLaMA 3 versus 0.19 for decode-and-re-encode, 0.16 for Φ_Ω1 only, and 0.21 for Φ_Ω2 only. A Sliced Gromov-Wasserstein heuristic drops to 0.14.
• Ablation — projection count: Increasing P shows marginal gains: P=50 gives 0.24, P=200 gives 0.26, and P=400 gives 0.26 on VideoLLaMA 3, with added computational cost.
• Ablation — divergence metric: The Hellinger distance used by TRANSPORTER yields 0.26 cosine similarity on VideoLLaMA 3, against 0.22 for Jaccard Similarity and 0.19 for KL divergence, which is unbounded.
• Inference-only settings are insufficient: The paper reports that decode-and-re-encode inference cannot reflect token modulations across logit differences or preserve scene aspects the way learned TRANSPORTER modulations can.

Methodology in Plain English

The work starts from a practical observation: a text-to-video generator has a latent space that encodes how a video looks, and a VLM has a latent space that encodes what a video means. If those two spaces can be linked, a change in the VLM's probability distribution can be turned into a change in a generated video.

TRANSPORTER builds that link in stages:

1. A coupling network takes a video's generator embedding and translates it into something that approximates the VLM's embedding for the same content. One part of the network handles the global projection (matching the overall embedding), and a second part preserves the relationships between tokens by matching Gram matrices, so the local geometry of the representation is not lost.
2. A learnable optimal transport module then merges these two views. It uses several learnable projection vectors to collapse tokens into scalars, computes transport costs between them, and solves an entropy-regularized transport problem using an efficient partial double-stochastic approximation rather than the full expensive formulation. Sinkhorn iterations are used with a temperature parameter τ and regularization λ.
3. A concept bank stores direction vectors, one per contrastive token pair. For a pair like "happy" → "sad", the model computes the difference between the generator's predicted velocity fields for the two conditions (Δv). That difference is supervised by the VLM's logit change (Δω), measured as a Hellinger distance between the two logit distributions.
4. At inference, a user picks a concept vector and a strength δ, adds it to the text condition, and the generator produces a video whose frames decode from 16 × 90² latents up to 16 × 720² resolution. The same coupling can also produce a caption for the generated video.

Generator weights and VLM weights stay frozen throughout; only the coupling network and concept bank are trained. The coupling network uses a 24-layer MLPMixer projector and a 12-layer Transformer, trained with AdamW for 100K iterations at a learning rate of 1e-3, batch size 8, with gradients accumulated every 8 steps. The concept bank is trained in a second stage for 1K iterations per vector at 1e-4. The ρ-OT module uses 100 projections. The base generator is Wan2.2, and coupling training uses videos from VATEX, LAVIB, and Ego4D.

Why This Matters

Impact on research. Interpretability for multimodal models has largely relied on text probes, saliency maps, or concept activation vectors. This paper offers a different modality entirely — generated video — and claims it as a fidelity-rich direction that has not been previously explored. If the approach generalizes, it gives researchers a way to inspect what a model's token predictions correspond to visually, not just what words correlate with them. The paper also demonstrates that a single method works across three different VLMs (VideoLLaMA 3, Gemma 3, Phi 4 MM), supporting the claim of model independence.

Real-world applications (as supported by the paper's demonstrations):

• Model debugging and safety review: Inspecting which visual attributes a model associates with tokens like "young" versus "old", or action attributes like "swift" versus "slow", to surface potentially biased or unintended associations.
• Video content creation and editing: Using learned concept vectors to modulate a single attribute (for example, "red" to "blue" bowling ball) while preserving camera view, object motion, and scene dynamics.
• Robotics and action understanding: Understanding how models bind objects to actions and to adjacent actions over time, shown in the "wipe" to "spray" transition.
• Educational and interactive exploration: A generative tool that lets a user ask what a VLM's prediction for a token looks like in motion, including intermediate steps between two concepts such as walk and run.

Industry relevance. Organizations deploying VLMs in video pipelines — surveillance, media production, autonomous systems, content moderation — need ways to audit model reasoning that go beyond text. A model-independent interpretability layer that works with frozen proprietary models could be valuable, since it does not require access to internal weights or retraining of the underlying VLM.

Future Directions

• Scaling the concept bank and projection count: The paper notes that increasing P offers only marginal improvement at additional computational cost (detailed further in §9 of the supplementary material), leaving the trade-off between cost and fidelity unresolved.
• Understanding attribute granularity: The finding that general attributes (object count) produce larger logit divergence than fine details (slice thickness) raises the question of how to reliably modulate subtle attributes, which currently appear only for a fraction of Δω values.
• Temporal control: The partial-conditioning experiments and the observation that large modulations appear early while fine details appear late suggest more work is needed on precisely when and where in a video a modulation should be applied.
• Extension beyond the three tested VLMs and the single base generator: The paper tests VideoLLaMA 3 (7B), Gemma 3 (12B), and Phi 4 MM (5B) with Wan2.2, and does not report how the coupling transfers to other generator families or larger VLMs.

Target Audience

Researchers and graduate students working on multimodal interpretability, video generation, or vision-language modeling will benefit most, particularly those familiar with optimal transport, flow matching, and latent diffusion or flow-based generators. Practitioners building auditing or debugging tools for deployed VLM systems may also find the L2V framing useful, though the method's technical requirements (training a coupling network against a frozen VLM and generator) make it more suited to research settings than to quick deployment. Readers looking for a beginner-level introduction to VLM interpretability should look elsewhere first; this paper assumes substantial background.

Authors’ abstract

How do video understanding models acquire their answers? Although current Vision Language Models (VLMs) reason over complex scenes with diverse objects, action performances, and scene dynamics, understanding and controlling their internal processes remains an open challenge. Motivated by recent advancements in text-to-video (T2V) generative models, this paper introduces a logits-to-video (L2V) task alongside a model-independent approach, TRANSPORTER, to generate videos that capture the underlying rules behind VLMs' predictions. Given the high-visual-fidelity produced by T2V models, TRANSPORTER learns an optimal transport coupling to VLM's high-semantic embedding spaces. In turn, logit scores define embedding directions for conditional video generation. TRANSPORTER generates videos that reflect caption changes over diverse object attributes, action adverbs, and scene context. Quantitative and qualitative evaluations across VLMs demonstrate that L2V can provide a fidelity-rich, novel direction for model interpretability that has not been previously explored.