Meta Inverse Prompting for Video Generative Models

AI summary

Overview

Research area: Computer vision — inverse prompting (recovering text prompts from media) for video generative models, combining vision-language models (VLMs) with gradient-based meta-learning.

Technical level: Intermediate (the meta-learning math is denser than the rest; the core idea is easy to grasp).

Scope (one sentence): The paper introduces MVIP (Meta Video Inverse Prompting), a framework that meta-learns a compact soft-prompt initialization inside a frozen VLM so that a frozen first-last-frame video generator can reproduce a target video from a recovered textual prompt, without per-video optimization at test time.

What This Paper Is About

Text-to-video, image-to-video, and reference-guided models let users turn conditions into videos, but the mapping is one-directional: given a finished video, there is no easy way to recover the prompt that would reproduce it. Prior inverse-prompting work targets images and relies on slow per-image gradient optimization or on captioning that produces fluent but reconstruction-poor text; video makes both worse because prompts must also encode motion, action sequences, and camera movement, and because each optimization step requires running an expensive video generator. The goal is a prompt that both faithfully reconstructs the target video and can be edited textually to produce controlled variations.

Key Contributions

1. First practical framework for video inverse prompting (MVIP). The authors formulate recovering a prompt from a video as an inverse problem and argue this is the first practical framework for the video domain, to their knowledge.

2. A meta-learning formulation of inverse prompting. Instead of optimizing a prompt from scratch for each video, MVIP learns a shared soft-prompt initialization (in the space of a frozen VLM's instruction prompt) across many videos, so the inversion cost is amortized. The design follows the first-order Reptile update rather than MAML, avoiding second-order differentiation through a costly, partly discrete pipeline.

3. Reconstruction-guided inner adaptation. For each video, the meta-prompt is adapted by sampling N candidate prompts from the VLM, scoring each with a frozen black-box video generator, and using the best-scoring candidate as a pseudo-target for updating the soft prompt — a way to get gradient-style optimization benefits while never touching generator internals.

4. A broad empirical study including unseen datasets, cross-generator transfer, runtime, ablations, and a downstream editing demo. Baselines are constructed from image inverse prompting (PEZ, ARPO-style iterative refinement) and VLM captioning, since no established video inverse-prompting benchmark or baseline exists.

Main Findings

• Better reconstruction on held-out real and synthetic video. On the real-video split (ActivityNet and MSR-VTT sources) MVIP reaches a ViCLIP V-V score of 0.8328, versus 0.8047 for the 3-step iterative refinement baseline ("Captioning + Refine"), 0.7977 for Qwen2.5-VL captioning, 0.7883 for PEZ, and 0.6992 for first-last-frame-only (FLF-Only) conditioning with no text. On the synthetic split MVIP scores 0.9460 ViCLIP V-V versus 0.9424 for FLF-Only, 0.9381 for PEZ, 0.9327 for Captioning + Refine, and 0.9284 for captioning.

• Quality gains are not limited to a single metric family. MVIP also leads on the visual and image-alignment columns reported: e.g., on real videos DINOv2 0.5848 (vs. 0.5710 PEZ) and CLIP-I 0.7825 (vs. 0.7712 Captioning + Refine), and on synthetic videos VideoMAE 0.9905, DINOv2 0.8785, DreamSim 0.0636, CLIP-I 0.9335.

• Generalizes to unseen video distributions. Without changing the learned meta-prompt or the training procedure, MVIP is best on most metrics on Kinetics-400 (ViCLIP V-V 0.9186 vs. 0.9007 captioning and 0.8963 PEZ) and DAVIS-2017 (ViCLIP V-V 0.9568, VideoMAE 0.9790, CLIP-I 0.9234).

• Efficiency is a central result. At K = 0 (no test-time adaptation), MVIP runs at 273.1 ± 1.0 s per video-second versus 261.9 ± 1.2 for plain captioning, 342.7 ± 1.9 for PEZ, and 1060.0 ± 3.7 for Captioning + Refine — i.e., iterative refinement costs 3.88× MVIP's runtime while still scoring lower (0.8047 vs. 0.8328 ViCLIP V-V).

• Test-time adaptation helps only slightly and then saturates. Starting from the same meta-learned prompt, K = 0 already beats all baselines (0.8328 ViCLIP V-V / 0.7825 CLIP-I); K = 1 is best (0.8392 / 0.7861); K = 2 drops back to 0.8322 / 0.7808 while runtime grows to 1893.4 ± 5.9 s/video-second.

• The meta-prompt transfers across generators. Replacing Wan2.2 with CogVideoX at test time, with no further optimization, MVIP scores 0.8709 ViCLIP V-V, 0.7044 DINOv2, and 0.8517 CLIP-I, ahead of captioning (0.8222 / 0.6615 / 0.8307), FLF-Only, and PEZ.

• Stronger captioning does not close the gap. On MSR-VTT, swapping Qwen2.5-VL-7B for Qwen3.5-9B raises captioning to 0.8312 ViCLIP V-V / 0.8063 CLIP-I, and adding a more detailed instruction gives 0.8123 / 0.7932; MVIP remains best at 0.8434 / 0.8091, with DINOv2 0.6516 versus 0.6214 and 0.6207 for the captioning variants.

• Both components of the method matter. Removing the meta-update (keeping Best-of-N) yields 0.8314 / 0.6303 / 0.8057; removing Best-of-N (keeping meta-learning) yields 0.8389 / 0.6174 / 0.8081; both are above plain captioning (0.8021 / 0.6214 / 0.7872) but below full MVIP (0.8434 / 0.6516 / 0.8091) on MSR-VTT.

• Prompt design choices: 16 soft tokens, natural-text initialization, one inner step. On a held-out validation subset, 16 tokens beats 4 (ViCLIP V-V 0.9193 vs. 0.9129) and 8 (0.9045); initializing soft tokens from the embeddings of "Describe objects, actions, camera motion, and temporal changes" beats random Gaussian initialization of the same length (0.9193 vs. 0.9064, with DreamSim 0.0917 vs. 0.0986); and K = 3 during meta-training (0.9174 / 0.0971 / 0.8992) does not beat K = 1 while taking roughly 3× longer per outer update.

• Recovered prompts work as editing interfaces. On IVEBench subject-motion editing, with GPT-4o minimally rewriting the recovered prompt and only the first frame used for generation, MVIP leads on Instruction Satisfaction (3.25 vs. RF-Edit 3.09 and Wan-Edit 2.09) and Content Fidelity (3.95 vs. 3.81 and 3.06), while Subject Consistency (0.92), Background Consistency (0.95), Temporal Flickering (0.97), Motion Smoothness (0.99), and Semantic Fidelity (0.96) are competitive — with no editing-specific training.

Methodology in Plain English

The setup: a video generator takes a text prompt plus the first and last frames of the target video, and produces a video. The first and last frames pin down where the video starts and ends, so the search space is only the text in between. A big frozen VLM (Qwen2.5-VL-7B-Instruct) writes candidate prompts from sampled frames, guided by a short learned "soft prompt" prepended to a fixed instruction. The generator and the VLM are never modified.

Training works in two nested loops. The inner loop takes one video: it samples N candidate prompts from the VLM, generates a video for each with the frozen generator, keeps the candidate whose output scores best against the target, and nudges the soft prompt to make that winning caption more likely. Doing this costs only forward passes — no gradients through the video model. The outer loop then averages the adapted soft prompts from a batch of videos and moves the shared starting point toward that average, in the spirit of Reptile-style first-order meta-learning. Over many videos, this starting point absorbs the general regularities of "what makes a caption reconstruct a video" — which details about appearance, action, and camera motion actually matter.

At test time, the learned starting point is dropped into the VLM for an unseen video and produces a reconstruction-oriented prompt in roughly captioning time; optional extra adaptation steps can be run, but the paper shows they buy little. Long videos are split into scene-consistent segments with PySceneDetect and each segment is inverted separately. Evaluation uses video-level similarity (ViCLIP V-V, VideoMAE, RAFT distance), visual similarity (DINOv2, DreamSim), and image alignment (CLIP-I).

Why This Matters

For research: The paper reframes inverse prompting from "optimize a prompt per sample" to "meta-learn a prompt prior," and shows that a compact set of soft tokens inside a frozen VLM can absorb reconstruction-relevant knowledge. It also provides a baseline suite (FLF-Only, PEZ, captioning, iterative refinement) for a task that previously had no established benchmarks, and demonstrates that black-box, gradient-free feedback can substitute for differentiable optimization when the generator is expensive.

Real-world applications:

• Media production and post-production: recovering an editable prompt from existing footage so creators can change lighting, motion, or action while preserving scene and composition, instead of re-engineering prompts by trial and error.
• Prompt reuse and template building: turning a library of finished clips into reusable prompts for consistent stylistic or narrative generation across shots.
• Controllable video editing in creative tools: using the recovered prompt as the control interface, as demonstrated with IVEBench subject-motion edits driven by a small GPT-4o rewrite.
• Simulation and synthetic data pipelines: reproducing or systematically varying generated clips for training data, with prompts that can be audited and modified rather than hidden.

Industry relevance: The efficiency numbers matter commercially — inverse prompting that costs roughly captioning time rather than 4× to 7× more makes it plausible inside interactive editing tools. The cross-generator transfer result (a meta-prompt trained with Wan2.2 still working with CogVideoX) suggests the learned prior is not locked to one vendor's model, which is important for pipelines that mix generators. The paper also flags misuse concerns: recovering prompts from video could enable unauthorized reproduction or manipulation, and may raise intellectual-property questions when applied to proprietary content.

Future Directions

• Removing the first-last-frame crutch. The authors explicitly name fully text-only video inverse prompting as an unsolved problem, blocked by the severe ambiguity of prompt–video mappings; the current framework assumes boundary frames are available.
• Learning structured, editable caption schemas. The paper suggests that meta-learning richer prompt structures could improve controllability and make downstream editing more flexible than minimally rewriting a free-form caption.
• Better use of test-time compute. Adaptation beyond a single step did not improve results (K = 2 was worse than K = 1 at much higher cost), leaving open how to spend additional inference budget productively.
• Deployment and safeguards for real workflows. The broader-impact discussion calls for responsible deployment, access control, and misuse monitoring for inverse prompting systems, plus further work before real media-production deployment.

Target Audience

Researchers and graduate students working on controllable video generation, prompt optimization, and meta-learning; practitioners building editing or content-creation tools on top of diffusion/flow-matching video models who need prompt recovery rather than prompt engineering; and readers interested in amortizing expensive per-sample optimization into a learned initialization. Some familiarity with diffusion/flow-based video generators, VLMs, and the MAML/Reptile family helps, though the central idea is accessible to anyone who knows text-to-video models.

Authors’ abstract

Recent advances in video generative models have enabled flexible conditional generation and editing pipelines, such as text-to-video, image-to-video, and reference-guided synthesis. However, these workflows are inherently one-directional, mapping conditions to videos without providing a mechanism to invert this process. In complex media production settings, the ability to recover controllable textual conditions from a given video is highly desirable, yet remains underexplored. While inverse prompting has been studied for image generation, directly extending these approaches to video is challenging due to the need to capture temporal dynamics beyond static appearance, as well as the substantial computational cost of per-video optimization. In this work, we propose the first practical framework for efficient video inverse prompting. Specifically, we formulate inverse prompting as a meta-learning problem and introduce Meta Video Inverse Prompting (MVIP), which learns a meta-prompt optimized over temporal dimensions to enable fast and effective prompt inference. Our approach amortizes the inversion process, eliminating the need for costly iterative optimization at test time. Experimental results demonstrate that MVIP recovers higher-quality inverse prompts with significantly improved reconstruction fidelity compared to image-based inversion and iterative captioning baselines, while achieving substantial gains in efficiency. Code and video results are available at https://ueriniuno.github.io/mvip-page/.