LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows

AI summary

Overview

Research area: Generative computer vision — video diffusion models, native multimodal conditioning, agentic visual creation pipelines, and inference-time acceleration (few-step distillation, quantization, lightweight decoders).

Technical level: Advanced. The paper is a technical report for a 32B-parameter diffusion transformer system. It assumes familiarity with diffusion transformers, flow matching, distribution-matching distillation, VAE latents, and quantization. The plain-language sections below are written so a non-specialist can follow the argument even without that background.

Scope in one sentence: The paper describes LynnReal-Omni, a single 32B multimodal diffusion transformer (plus a 27B real-time "Flash" variant) that unifies text-to-video, image conditioning, reference-guided and structural control, editing, restoration, and long-video generation, together with the data pipeline, training recipe, decoder distillation, and a new 100-prompt evaluation protocol built to support agent-driven visual workflows.

What This Paper Is About

Video diffusion models produce visually impressive results but are stochastic and hard to steer: getting an exact scene often requires many samples, and long clips drift in appearance, object identity, and physical consistency. Meanwhile, agentic visual creation — where an LLM agent writes a game, builds an editable 3D scene, or supplies reference images — provides explicit, repeatable structure but not high-fidelity rendering on its own. The paper's goal is to combine the two: a single model that natively accepts heterogeneous agent-produced conditions (references, frame-aligned controls, 3D renders, game recordings, causal history) and renders stable, coherent, optionally audio-synchronized video quickly enough to sit inside an interactive agent loop.

Key Contributions

1. A unified native multimodal backbone. LynnReal-Omni replaces a stack of task-specific pipelines (text-to-video, image-conditioned generation, reference-guided generation, structural control, editing, degraded-video restoration, long-video generation) with one shared multimodal diffusion transformer and a task-specific input layout. A separate 27B Flash variant is trained for real-time rendering.

2. A native task representation. Modality identity, modality-specific row indices, 3D rotary positions, per-row noise levels, and output targets are encoded explicitly, so appearance references, frame-aligned controls, editable 3D renders, game recordings, and causal history each retain a distinct role. Aligned controls are held at a constant positive temporal offset from target rows (Δ > 0) so reference and generated tokens occupy separate domains.

3. A systematic data and annotation pipeline. Shot segmentation, OCR-based watermark/border removal, scene grouping, cross-shot subject discovery and linking, retrieval and deduplication, pose tracking, and diarized audio alignment yield a curated corpus of multi-shot audiovisual clips. Roughly 0.6% of the original storage footprint survives processing, and every retained clip is at most one minute. A separate large-scale video-editing dataset was also constructed.

4. Efficiency and evaluation infrastructure. A distilled lightweight VAE decoder, a bounded-history long-video scheme, W4A8 quantization, and operator fusion bring warm 22-frame 540p generation and decoding to sub-second latency on a single H100. The MSAVP protocol (100 prompts, six metric families) keeps instruction following, plausibility, visual quality, temporal behavior, controllability, and audio coordination separate rather than collapsing them into one number.

Main Findings

• One model, many tasks: Text-to-video, image conditioning, reference-guided generation, structural control, editing, restoration, and long-video generation run through a single shared denoiser with task-specific token ordering, modality labels, and temporal positions, rather than through separate models and ad hoc interfaces.

• Two explicit image-conditioning contracts: A unified reference layout places supplied images in ordered picture-reference segments (with an opening-frame instruction treated semantically), while a native keyframe layout encodes first and optional last images into the target's keyframe partition. The release selects between them through the model bundle's conditioning contract; both are available to the standard DiT and to Flash.

• Distinct noise schedules for video and audio: Although both modalities share four denoising evaluations, they are evaluated at different clean-time points — video at approximately (0, 0.0270, 0.0769, 0.2000) and audio at approximately (0, 0.1000, 0.2500, 0.5000) under the native 768p configuration. Missing audio is not treated as supervised silence.

• Depth and token reduction in Flash: Flash keeps 42 of 50 transformer blocks and uses three denoising evaluations. The first two blocks run at full token resolution for early multimodal integration; the next 26 subsample video tokens with stride 2 in both spatial dimensions (about a quarter of the tokens, boundary rows and columns retained), with a residual path restoring full-resolution features before the final fourteen blocks.

• Bounded-history long video: Long videos are generated in fixed-length chunks producing 17 new RGB frames each. The previous chunk's final frame is reused as a shared boundary frame, its latent held fixed while five future latents are predicted and jointly decoded. Temporal context is token-budgeted — one full-resolution earliest frame, two recent frames at stride 2, up to eight earlier frames at stride 4 — so attention cost stays independent of video length, and storage is capped by discarding intermediate entries.

• Lightweight decoder distillation: The 36-block teacher decoder is distilled to 26 blocks at hidden dimension 2,048, initialized from corresponding teacher blocks and trained against teacher reconstructions, feature and perceptual losses, latent consistency through a frozen encoder, and paired source pixels when available. Latents are drawn 60% from encoded data and 40% from the generative model, with Gaussian perturbation on 25% of samples for robustness to distribution shift. An image-specialized decoder is used adaptively as teacher for image latents when it reconstructs better.

• Reported latency: On one H100, warm generation and decoding of a 22-frame 540p video is reported as 843 ms for LynnReal-Omni and 377 ms for Flash in the abstract; the introduction states 909 ms and 591 ms for the same setting. The paper does not reconcile the two figures. Similarly, MSAVP is described as a 20-metric design in the abstract and a 25-metric benchmark in the introduction, though both describe the same six metric families.

• Data quality discipline: Optical flow, sharpness, brightness, and frame-change statistics only prioritize candidates for review and never decide quality, because editing cuts, flashing lights, subtitles, and camera shake can all produce large frame differences without real subject motion. Assets are explicitly labeled available, not applicable, or blocked so that missing outputs are not mistaken for valid ones.

Methodology in Plain English

The core idea. Instead of chaining several specialized models, the authors train one large transformer that accepts a mixed bag of inputs — text, reference images, on-screen controls tied to specific frames, 3D renders, game footage, and the model's own earlier output — and learns which parts of the input are conditions and which parts it must generate. A small "task representation" tells the model what each incoming token is for, where it sits in time, and how noisy it currently is.

How conditions stay separate from targets. Frame-aligned controls (pose, depth, edits, game renders) are given a constant positive time offset relative to the frames the model must produce. This prevents the model from confusing "this is what you were told" with "this is what you made."

Training. The model is trained as a flow-matching denoiser with a shifted logistic-normal noise distribution, plus a coordinated noise schedule for audio so both modalities share a random draw while following their own clocks. To make inference cheap, the authors use few-step distillation: a frozen teacher and a trained critic guide a student that reaches good outputs in three or four denoising evaluations instead of many. A dynamic visual reward reweights the video term to improve motion dynamics; the audio term keeps a fixed weight because the reward model is not audio-aware.

Flash. The fast variant is shallower, drops three-quarters of the video tokens inside the middle of the network, and is quantized to INT4 weights and FP8 activations, with operator fusion to cut memory traffic. These optimizations are ablated separately so their individual effects on latency and quality can be told apart.

Data. Raw public video goes through shot detection with visual verification, OCR-based cleaning of watermarks and subtitles, scene grouping by a large multimodal model over frame mosaics, cross-shot subject discovery with two-stage bounding-box localization, curation and deduplication, pose and background asset preparation, and diarized audio transcription with separate handling of environmental sounds and music. The design principle is a division of labor: specialist models supply measurable evidence (cuts, text regions, speech times, poses), the multimodal model supplies interpretation (scenes, identities, events, captions), and deterministic checks ensure all outputs refer to the same video, timeline, and subjects.

Evaluation. MSAVP uses 100 prompts across several metric families, retaining per-metric applicability counts and original scoring units alongside a hierarchical aggregate, so dimension-specific failures stay visible instead of being hidden by averaging across incompatible scales.

Why This Matters

Impact on research. The paper argues that the practical bottleneck in video generation has moved. Once few-step distillation collapses denoising cost, the VAE decoder becomes the dominant inference expense — which is why a distilled decoder is treated as a first-class contribution rather than an afterthought. The paper also pushes back on task fragmentation, showing that a single backbone with a well-designed task representation can cover conditioning, editing, restoration, and long video without separate weights. The MSAVP design is a methodological argument as much as a metric: aggregate scores hide failures, so per-dimension evidence should be reported.

Real-world applications.

• Game prototyping and previsualization: An agent writes a low-poly playable scene or prototype; the model renders detailed, coherent visuals from the game state, collapsing the geometry, materials, rigging, and rendering pipeline.
• Interactive content and streaming: Sub-second warm generation at 540p on a single H100 opens the door to real-time streaming generation rather than offline batch rendering.
• Film and advertising preproduction: Editable 3D scenes plus appearance references give directors repeatable, adjustable shots instead of re-rolling a stochastic sampler until something usable appears.
• Video post-production and restoration: Editing, text-conditioned control, and degraded-video restoration live in the same model, so cleanup and modification share one conditioning interface.

Industry relevance. The release includes open weights on Hugging Face (flash beta, standard beta, and a light VAE) alongside a public demo and code repository, which positions the work as an inspectable, deployable foundation rather than a closed benchmark result. The latency numbers, quantization choices, and single-GPU deployment target speak directly to serving-cost economics for anyone building agentic video products.

Future Directions

• Reconciling the reported numbers. The abstract and introduction disagree on both latency (843/377 ms versus 909/591 ms) and metric count (20 versus 25). Clarifying these, and publishing a full per-dimension MSAVP breakdown, would make the efficiency and quality claims directly checkable.

• Audio-aware distillation and rewards. The dynamic reward model is visual only, so the audio term keeps a fixed weight. Extending reward-based distribution matching to audio could improve synchronization and sound-event fidelity rather than holding audio to a static objective.

• Scaling bounded-history generation. The current scheme fixes the conditioning token budget so attention cost is independent of length, but the retention policy discards intermediate frames. Whether longer or open-ended streaming generation preserves identity and photometry over minutes rather than chunks remains open.

• Closing the agent loop. The paper positions the model as a component in agentic workflows, but the reported evaluation is prompt-based. Measuring end-to-end agent performance — where the agent inspects output, revises the 3D scene or game, and regenerates — would test the central claim that combining agentic control with diffusion actually reduces repeated sampling.

Target Audience

• Generative video and diffusion researchers interested in unified multitask architectures, few-step distillation, and decoder-side efficiency.
• Applied ML and infrastructure engineers building video generation services, especially those weighing single-GPU latency, quantization, and decoder cost against quality.
• Agent and tooling developers who need a controllable rendering backend for LLM-driven creative or simulation workflows.
• Data pipeline and dataset engineers who will find the five-stage curation design, the 0.6% retention figure, and the specialist-model-versus-multimodal-model division of labor directly reusable.
• Evaluation and benchmark researchers working on multi-dimensional video metrics that resist aggregation into a single misleading score.

Authors’ abstract

Video diffusion models are stochastic and hard to control: precise content often requires repeated sampling without guaranteed success, and long-horizon scenes drift in appearance, interactions, and temporal coherence. Agentic visual creation provides explicit references, editable 3D scenes, or executable game states for stable control, but does not by itself guarantee high object or character fidelity. Combining the two can enable stable, high-quality generation. To realize this combination, we present LynnReal-Omni, a native multimodal video generation framework built on a 32B shared multimodal diffusion transformer that unifies text-to-video, image-conditioned generation, reference-guided generation, structural control, editing, degraded video restoration, and long-video generation. It accepts heterogeneous visual inputs, including appearance references, editable 3D renders, and game recordings, allowing agents to compose visual conditions within a unified model. We also train a dedicated 27B Flash shared multimodal diffusion transformer for real-time rendering. We build a systematic data pipeline for video cleaning, subject association, multimodal annotation, and aligned control construction, yielding a curated corpus of multi-shot audiovisual segments, and introduce MSAVP, a 100-prompt, 20-metric evaluation design that separates instruction following, generating plausibility, visual quality, temporal behavior, and audio coordination. LynnReal-Omni-Flash further reduces inference cost through model and decoding acceleration, including a lightweight VAE decoder; on one H100, warm generation and decoding of a 22-frame 540p video take 843 ms with LynnReal-Omni and 377 ms with Flash. These results provide a foundation for real-time streaming video generation, making LynnReal-Omni a unified, controllable, and efficient basis for agentic visual creation.