Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation

AI summary

Overview

Research area: Computer Vision — cinematic video generation, specifically conditioning Video Diffusion Transformers (DiTs) on multimodal large language model (MLLM) representations.

Technical level: Advanced. The paper assumes familiarity with flow matching, DiT architectures, MLLM hidden states, connector/projection layers, and classifier-free guidance.

Scope: The paper introduces Factor-Aware Predictive Learning (FAPL), a three-stage training procedure that forces a Video DiT to actually rely on MLLM latents, teaches it to respond to localized factor-level latent edits, and then trains the MLLM pathway itself through a frozen generator — evaluated on sparse-to-video generation, factor-aware editing, and multi-shot generation.

What This Paper Is About

Systems that connect an MLLM to a video generator often let the generator cheat: it can reduce its training loss by leaning on the familiar text-encoder and VAE-reference pathways and largely ignore the MLLM features. The paper argues that this "MLLM feature bypass" is the real bottleneck, and that ordinary video-level supervision also never tells the model which factor (character, scene, action, camera) an edit is supposed to change and which content must survive. FAPL's goal is to make MLLM latents both used and selectively controllable, so a model can infer a plausible shot from sparse inputs (a script plus a few references) while keeping the four cinematic factors independently editable.

Key Contributions

1. A diagnosis plus remedy for MLLM feature bypass. The authors identify that MLLM latents are routinely bypassed by native text and VAE-reference pathways, and introduce Stage I "latent-only generator grounding," where the DiT is trained with MLLM latents as its only content condition (text-encoder and VAE-reference conditions disabled), across both text-only and text-plus-complete-video observations.

2. Factor-aware supervision via latent span interventions. Structured four-field prompts (character, scene, action, camera) locate the token spans of each factor in the projected latent sequence. Paired videos that differ in exactly one factor are used to swap that factor's span after the connector and before the DiT, so the generator learns the change-and-preserve relationship without any assumption that the spans are fully disentangled.

3. Predictive learning through a frozen generator. Stage III freezes the trained DiT and jointly optimizes the MLLM and connector, using text-only and text-plus-complete-video inputs that share the same video target, so the MLLM learns to produce generation-compatible representations from limited evidence while complete-video reconstruction remains a complementary training mode.

4. Evidence across three cinematic tasks. FAPL is built on UniVideo (13B DiT) and compared against Omni-Video 2 (1.3B and 14B) and UniVideo (13B) on sparse-to-video generation, factor-aware editing, and multi-shot generation, with stage-wise ablations isolating each component.

Main Findings

• All three tasks improve over the baselines. On sparse-to-video generation FAPL reaches Motion Order 61%, Translation Error 0.0586 and Rotation Error 1.02 rad, versus 48%, 0.0763 and 1.54 for UniVideo (13B), 43%, 0.0817 and 1.49 for Omni-Video 2 (14B), and 33%, 0.1084 and 1.87 for Omni-Video 2 (1.3B).

• Factor-aware editing accuracy rises substantially. FAPL achieves ∩cap-Acc of 58% (FiVE's metric, requiring both yes/no and multiple-choice checks to pass), against 42% for UniVideo (13B), 37% for Omni-Video 2 (14B), and 28% for Omni-Video 2 (1.3B).

• Cross-shot consistency improves. On the 100 ShotVerse-Bench sequences, FAPL scores Subject Consistency 0.516, Scene Consistency 0.471, and FaceSim-Arc 0.523, compared with 0.462 / 0.427 / 0.451 for UniVideo (13B), 0.476 / 0.423 / 0.468 for Omni-Video 2 (14B), and 0.435 / 0.373 / 0.412 for Omni-Video 2 (1.3B).

• Latent-only grounding alone already helps. Stage I lifts Motion Order from the pretrained 48% to 54%, Translation Error from 0.0763 to 0.0708, Rotation Error from 1.54 to 1.38 rad, and ∩cap-Acc from 42% to 44%. The paper also reports that after Stage I, MLLM-only generation improves on Omni-Video 2 (1.3B and 14B) as well as UniVideo (13B), i.e. the effect generalizes across MLLM–DiT models.

• Factor swapping is what buys editing control, at a small cost elsewhere. Under the same paired data and optimization budget, training with factor swapping reaches 57% ∩cap-Acc versus 48% without it. Motion Order (53% vs 54%) and camera errors (0.0671 / 1.32 vs 0.0654 / 1.30) do not improve over that control, which the authors interpret as the benefit being concentrated in factor-aware editing.

• Mixed-observation predictive learning adds temporal gains. Compared with text-only Stage III training, mixed-observation training raises Motion Order from 57% to 61% and lowers Translation Error from 0.0629 to 0.0586 and Rotation Error from 1.26 to 1.02 rad, while editing accuracy stays comparable (57% to 58%).

• Qualitative failure modes of baselines are documented. Under MLLM-only conditioning, Omni-Video 2 (1.3B) produces noise, while Omni-Video 2 (14B) and UniVideo produce artifacts and render one person's successive actions as two people; under conflicting conditions the VAE references dominate the output. Qualitatively, UniVideo misses details such as hair length, hand gestures, or a requested camera orbit, and its edits leak into non-target factors.

Methodology in Plain English

The starting point is a standard MLLM-conditioned video model: an MLLM reads the input, a connector projects its hidden states into the DiT's conditioning space, and the DiT (trained with flow matching) also receives text-encoder features and VAE-encoded reference images.

Stage I removes the crutches. The MLLM and connector are frozen, and the DiT is trained with only the projected MLLM latents as content conditions, split 1:1 between text-only inputs and inputs that include the complete target video. Since nothing else is available, the DiT is forced to learn how to read MLLM latents. The complete-video mode grounds the DiT in dense visual and temporal information; the text-only mode keeps it compatible with sparse-input generation.

Stage II adds factor control. Captions are rewritten into four ordered fields — character, scene, action, camera — and edits are created by changing only one field. Because the field boundaries are known, the corresponding spans in the projected latent sequence can be located. For a pair that differs only in factor k, the k-span of one example is cut and pasted into the other (and vice versa), and the DiT must predict the corresponding video. Swapping happens after the connector and before the DiT, and spans need not have equal length. This teaches the generator what a localized change in its conditioning should do to the output, and what should stay put, without requiring cleanly separated features.

Stage III flips the direction of adaptation. The DiT is now frozen at its post-Stage-II weights, and the MLLM and connector are trained through it: gradients pass through the frozen generator back into the language model. Text-only and text-plus-complete-video inputs are both supervised against the same target video, so the MLLM learns to produce representations from which the fixed generator can reconstruct semantics, motion and timing from sparse evidence.

At inference the same architecture serves script-plus-reference generation, factor editing (re-encode the edited prompt, splice only that factor's span into the original latents), and multi-shot generation (shots generated independently from their own descriptions plus shared character and scene references). No complete target video is needed at test time.

Why This Matters

For research: The paper reframes a commonly reported weakness — MLLM features "not helping" in multimodal generators — as a supervision and interface problem rather than an architecture problem, and shows sizable gains without adding new modules. The factor-span intervention is a cheap way to convert paired editing data into structured supervision, and the frozen-generator predictive stage offers a template for training an encoder-like pathway against a fixed decoder.

Real-world applications:

• Previsualization and previz for film and advertising, where a script plus a few references must become a plausible shot before physical production.
• Iterative shot editing, where a director changes one character, prop, action, or camera move and expects everything else to hold.
• Multi-shot sequence generation for storyboards and short-form narrative content, keeping a character and set consistent across cuts.
• Controllable asset generation for games and virtual production, where camera and action need to be specified independently of scene appearance.

Industry relevance: The method is built on released backbones (UniVideo 13B, Omni-Video 2) and uses 100K quality-filtered source–target pairs (25K per factor) drawn from CoinVE-200K and OmniVBench training data — a data budget that fits within typical studio fine-tuning pipelines rather than requiring pretraining-scale resources.

Future Directions

• Stronger MLLMs for spatial and camera reasoning. The authors state that current MLLMs struggle to interpret 3D blockouts or reason about complex camera motion for DiT control; upgrading the reasoning model is a direct next step.
• Stronger generators for fidelity and acting. The pretrained DiT is named as the limit on visual fidelity and on complex factor realization such as realistic acting, so pairing FAPL with a more capable video generator is the natural follow-up.
• Beyond four factors and beyond single-factor edits. The current factor set is fixed to character, scene, action, and camera, and editing cases modify one field while the other three are held; compositional multi-factor edits and finer-grained attributes are untested here.
• Rethinking disentanglement and span design. The paper deliberately does not assume the contextualized spans are fully disentangled; whether better factor localization, learned span boundaries, or explicit separation objectives would add further gains is left open.

Target Audience

Researchers and engineers working on controllable video generation and on MLLM–diffusion coupling, particularly those building or fine-tuning MLLM-conditioned Video DiTs and those interested in turning paired editing data into structured supervision. It is also relevant to practitioners in creative tooling for film, advertising, and virtual production who need independent control over character, scene, action, and camera, and to readers following predictive representation learning who want to see it applied with a frozen high-fidelity video decoder. Readers without background in flow matching and diffusion transformers will find the method sections demanding.

Authors’ abstract

Cinematic video generation requires both temporal inference from limited inputs (i.e., sparse rather than frame-by-frame guidance) and factor-aware control of character, scene, action, and camera. Multimodal large language models (MLLMs) offer useful priors for both capabilities, but their representations are not automatically actionable for video generation: a Video Diffusion Transformer (DiT) may bypass them through native conditioning pathways, and using them does not ensure either factor-aware control or generation-compatible temporal prediction. We propose Factor-Aware Predictive Learning (FAPL), which establishes effective MLLM latent use as a shared prerequisite and then targets the two capabilities with complementary learning signals. First, latent-only generator grounding trains the DiT with a frozen MLLM and connector, strengthening the DiT's use of MLLM latents for generation. Second, for factor-aware control, paired videos specify a target-factor change and non-target preservation, and factor-span replacement trains the DiT to realize this relationship in its output. Third, for temporal inference, text-only and text-plus-complete-video observations share a video target, and the trained DiT is then frozen while the MLLM and connector are optimized through it. This final stage trains limited-input representations for generation while retaining complete-observation reconstruction as a complementary training mode. Extensive experiments demonstrate our consistently better performance on three cinematic video generation tasks (i.e., sparse-to-video, factor-aware editing, and multi-shot generation) over baselines and validate FAPL's contributions to MLLM feature use, temporal control, and factor coordination.