OmniXRef: A Unified Framework for Multimodal Multi-Reference Video Editing

AI summary

Overview

• Research area: Computer vision / generative video editing, specifically reference-guided video editing with multimodal conditioning (text, image, video) built on a diffusion transformer (DiT) plus a multimodal large language model (MLLM).
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching transformers, RoPE positional encodings, VAE latents, patchification, and classifier-free guidance.
• Scope (one sentence): The paper formulates, trains, benchmarks, and ablates a unified "reference-bank" system that lets users combine a variable number of text, image, and video references to edit different attributes of a source video in one pass.

What This Paper Is About

Existing video editing models mostly accept one kind of guidance at a time — a text prompt with an optional image, several images, or a single video reference for one narrow task such as camera control or visual effects. Creative intent often needs several references at once (one for subject motion, one for camera movement, one for style), and the paper argues that a systematic formulation of this "multimodal multi-reference video editing" problem is largely missing. OmniXRef's goal is a single model that ingests any mix of text, image, and video references, each tied to its own editing task, and applies all of them to the source video while leaving everything else untouched.

Key Contributions

1. A unified formulation and interface. OmniXRef defines editing along four dimensions — number of references (1, …, K), reference modality (text, image, video), reference attribute (Camera, Motion, Content, Style, VFX), and composition (text-image, text-video, image-video, text-image-video) — and organizes all inputs through a common reference-bank interface, where each bank holds a task definition, an edit instruction, and an optional visual reference.
2. OmniXRef-Model with two architectural devices. A frozen MLLM (Qwen3-VL-8B-Instruct) encodes an interleaved sequence of system prompt, source video, and reference banks into semantic context; a single-stream DiT then jointly denoises source, reference, and target tokens. Mix-Patchify uses task-specific patch granularity per reference to cut token counts, and R²-RoPE (RoPE with Reserved Reference Slots) reserves a pool of positional slots so the model can handle more references at inference than it saw in training.
3. OmniXRef-Data. Roughly 10M training examples with synthesis pipelines for five single-video-reference tasks (Camera, Motion, Content, Style, VFX) plus compositional data combining two visual references, assembled with text-only and text-image data.
4. OmniXRef-Bench. A 125-case benchmark across Camera, Motion, Content, Style, and VFX, scoring Reference Adherence (RA), Visual Quality (VQ), and Source Preservation (SP) with a VLM judge, plus geometry-based camera trajectory metrics.

Main Findings

• Best overall on the new benchmark: On OmniXRef-Bench, OmniXRef-Model posts the best overall average Score (3.78), RA (3.67), and SP (4.68), compared with Seedance 2.0 (Score 3.12), Seedance 2.5 (2.73), and MiniMax-H3 (1.45). It leads on Motion (3.30), Content (4.49), and Style (3.75).
• Acknowledged visual-quality gap: OmniXRef-Model's overall VQ (3.68) trails the commercial systems (MiniMax-H3 3.84, Seedance 2.0 3.83, Seedance 2.5 3.70). The authors attribute the remaining VQ gap to differences in pretrained backbone capacity.
• The formulation transfers to another backbone: Fine-tuning MiniMax-H3 on OmniXRef-Data from its Ref2VA checkpoint — without architecture changes — yields "OmniXRef (MiniMax-H3)," which improves on original MiniMax-H3 in all five categories and beats OmniXRef-Model on Camera (3.99 vs 3.81) and VFX (3.83 vs 3.55), with overall Score 3.59.
• Strongest camera trajectories: Geometry-based camera errors (summed over 31 frames, RotErr in radians, estimated with VGGT-Ω on 32 aligned frames) are lowest for OmniXRef: RotErr 0.874, TransErr 3.961, CamMC 4.437, versus CamCloneMaster (2.086 / 9.329 / 10.332) and Seedance 2.0 and 2.5, which show RotErr around 4.4 and TransErr/CamMC around 22–24.
• Task-specific experts are clearly beaten in their own domains: RefVFX scores 1.98 (Content), 2.24 (Style), 2.49 (VFX); MotionFollower scores 2.14 on Motion; CamCloneMaster scores 3.20 on Camera, all below OmniXRef-Model.
• Ablations confirm each component (overall average Score, full model 3.77): removing Mix-Patchify drops to 2.86, reference-bank dropout to 2.83, removing R²-RoPE to 3.18, and removing curriculum learning to 3.13. R²-RoPE removal causes the smallest overall degradation; the authors trace it to an indexing conflict where the noise tokens' fourth-axis index shifts with the number of references.
• The model uses the visual reference, not just the instruction: Holding the source and the instruction fixed and swapping only the reference video reproduces the distinct sunset lighting of each reference.
• Robust to reference order: Swapping the two reference banks between R²-RoPE slots still transfers camera motion and replaces the table, indicating slots are not tied to particular attributes.
• Generalizes beyond the training mix: With three video references (motion, camera, style) — exceeding the maximum of two visual references in any training sample — OmniXRef-Model follows all three while Seedance 2.5 largely fails on motion and camera and only partially reproduces style.
• Cost savings from Mix-Patchify: Measured on a single NVIDIA A100 GPU at 480P with 121 frames, more references and finer patches raise latency and compute, and Mix-Patchify reduces both, with a larger advantage as reference count grows.

Methodology in Plain English

The team treats a video edit as a short conversation. First, an off-the-shelf multimodal language model reads a system prompt, the source video, and a stack of "reference banks" — each bank being a sentence saying what kind of edit this is, a free-form instruction, and optionally a reference image or video. The language model's output is the semantic plan. Second, a diffusion transformer receives that plan together with the compressed (VAE) version of the source video, the references, and a noisy copy of the target video, and denoises everything in one shared attention sequence.

Two tricks make multi-reference conditioning practical. Instead of compressing every reference at the finest patch granularity, Mix-Patchify assigns coarser patches to references whose contribution is global or dynamic (Camera, Motion, Style, VFX) and keeps fine patches where appearance detail matters (Content), while always keeping temporal patches at 1 to preserve motion detail. To let the model handle more references than the training data covers, R²-RoPE gives each reference bank a slot from a reserved pool of eight and randomly assigns banks to slots during training, so every slot has seen a reference and extra references at inference stay within the positional range seen during training.

Training data are synthesized rather than collected, because source–reference–target triplets are scarce. For VFX and Style, the pipeline starts from roughly 2M candidate target videos found in a text-to-video collection, uses GPT Image 2 to remove the effect from the first frame and a propagation model to produce the source, then generates a matching reference from a different scene with GPT Image 2 and Wan2.2-TI2V-5B. For Camera, two distinct videos are "reshot" with Vista4D along the same sampled camera path and swapped into cross-pairs. For Motion, paired videos from MotionPair-60K are perturbed with Wan2.2-TI2V-5B to create alternate motions. Content draws on RefVIE triplets. Compositional data are made by applying the Camera reshooting pipeline to Content and Motion data, and an MLLM filters quality throughout.

Training runs in three curriculum stages: single-reference adaptation at 272P (learning rate 1.5×10⁻⁵ decaying to 1×10⁻⁵ over the first 5,000 steps), then multi-reference composition at 272P, then everything at 480P. Evaluation uses a VLM judge (Gemini 3.8 Flash) with per-category prompts scoring RA, VQ, and SP, supplemented by pose-based camera metrics, and the paper also reports ReCo-Bench and VEFX-Bench to check that text-only editing ability is retained (those specific numbers appear in the paper's Appendix D and are not in the provided content).

Why This Matters

• Research impact: The paper reframes reference-guided editing as a composable, multi-bank problem rather than a set of single-modality tasks, and supplies both a training corpus and a benchmark for video-as-control-modality — a combination that RefVIE, EditVerseBench, TIDE-Bench, and others in the paper's related work do not cover, since none systematically evaluates transferring dynamic attributes from a reference video while preserving the rest of the source.
• Real-world applications:
  • Advertising and social content, where a brand asset, a camera move, and a look must be applied consistently across footage.
  • Film and post-production, restaging shots with a reference camera path or adding practical effects like rain, smoke, or fire taken from reference plates.
  • Motion transfer and character animation, retiming a performer's motion onto a source subject.
  • Stylization and relighting of existing footage, applying lighting or a look from a reference clip.
• Industry relevance: The use of a frozen public MLLM plus a pretrained DiT, and demonstration that the data recipe transfers to a third-party backbone (MiniMax-H3), suggests the approach is portable to existing commercial pipelines rather than requiring a new base model. Mix-Patchify and R²-RoPE address two concrete deployment costs — inference latency with multiple video references and the need for training data at every reference count.

Future Directions

• Closing the visual-quality gap: The authors explicitly note that the remaining VQ deficit relative to commercial systems may stem from backbone capacity, and that more tailored adaptation strategies for the MiniMax-H3 variant "could yield further gains."
• Scaling reference count and composition: Training data covers at most two visual references while R²-RoPE reserves eight slots; how far the model scales, and how to build data for higher-order reference combinations, remains open.
• Richer attribute and modality coverage: The five categories (Camera, Motion, Content, Style, VFX) and three modalities define the current scope; extending to further attributes and to joint audio or other signals is a natural extension.
• Positional encoding for variable-length context: The ablation identifies that sequential asset indexing produces conflicting positional signals when reference counts vary, suggesting further work on RoPE schemes for interleaved multimodal sequences.

Target Audience

Researchers and engineers working on controllable video generation and editing, particularly those building diffusion-transformer systems with multimodal conditioning; practitioners who need to evaluate reference adherence and source preservation rather than generic video quality; and teams interested in data synthesis pipelines for scarcity-limited video tasks. Readers should be comfortable with diffusion, transformer attention, and positional encoding to get full value from the method sections.

Authors’ abstract

Creative intent in video editing can be expressed through text instructions (e.g., "remove the people"), image references (e.g., subject appearance), video references (e.g., camera movement), or a combination of the above. We introduce OmniXRef, a unified framework for multimodal multi-reference video editing that allows users to combine variable numbers of text, image, and video references to control diverse attributes of a source video. We first devise a unified input design with a reference-bank interface and introduce $R^2$-RoPE to support generalization to more references. Then our model, OmniXRef-Model, uses a single-stream diffusion transformer to jointly process the source video and multimodal context from an MLLM in an interleaved sequence. To address the scarcity of training data for video-reference editing, we construct OmniXRef-Data using synthesis pipelines for different editing tasks and combinations of references. We use curriculum learning to progressively increase task complexity and resolution. To support systematic evaluation of video as a control modality, we introduce OmniXRef-Bench, a benchmark for video-reference attribute transfer across Camera, Motion, Content, Style, and Visual Effects (VFX). Comprehensive evaluations show that our OmniXRef-Model outperforms task-specific baselines and commercial models.