Hy-Nomas: Towards Free-form Interactive World Generation

AI summary

Overview

Research area: Computer vision and generative AI — specifically video world models, i.e. models that render an interactive virtual environment in response to user commands rather than a single fixed prompt.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT/MMDiT), flow matching, KV caching, sequence parallelism, and video autoencoders.

Scope (one sentence): The paper describes Hy-Nomas, an 8B-parameter autoregressive diffusion world model from the Tencent Hy Visual Generation Team that accepts interleaved free-form text instructions and continuous camera control across multiple interaction turns, together with its data pipeline, action-aware attention design, training curriculum, and distributed training infrastructure.

What This Paper Is About

Conventional video generators take one prompt and produce one short clip. An interactive world model has to do something harder: the user gives a command mid-session, the model renders the resulting world state, and that output becomes the history the next command builds on. The core problem is that simply swapping full attention for a causal mask is not enough — the model must not leak future information, must keep persistent scene context available, and must apply each action instruction only during the time window it was meant for. Hy-Nomas addresses this by co-designing structured video captions, action-aware causal attention, camera-trajectory conditioning, a multi-stage training curriculum, and the infrastructure needed to train on very long packed video sequences.

Key Contributions

1. An interactive 8B world model. Hy-Nomas supports multi-turn free-form interaction and continuous camera navigation within a single autoregressive DiT, built on HunyuanVideo 1.5, generating variable-length segments conditioned on current inputs plus cached history.

2. Aligned supervision and action conditioning. The team combines structured chronological captions (persistent subject profiles, timestamped actions, scene changes) and estimated camera trajectories with action-aware causal attention, which distinguishes enduring descriptive context from temporally localized action instructions.

3. A scalable training recipe and open-weight release. A progressive curriculum (segment causality → localized action conditioning → camera control → training on model-generated histories → diffusion distillation), bounded-cache inference, a LoRA workflow that transfers adapters from the base model to its distilled counterpart, and released weights, code, benchmarks, and an adaptation workflow.

4. A world-model evaluation suite. An assessment of multi-turn text interaction, camera navigation, interleaved control, and spatial memory, with all benchmarks released to support reproducible research.

Main Findings

• Two concrete failure modes of naive segment-causal masking. The authors report that segment-wise causal attention alone produced (1) within-segment action misalignment, where actions inside a segment occur in the wrong temporal order because query tokens can see the whole segment caption, and (2) cross-segment action repetition, where past actions recur in later segments because earlier action descriptions stay attendable.

• Action-aware causal attention reduces both failures. Splitting each segment's text into a descriptive head (environment, kept accessible to later segments) and action tails (restricted to their originating segment and time window) substantially reduces action-order errors and cross-segment repetition when applied in later training stages.

• Continuous, finer-grained camera control than comparable systems. Hy-Nomas encodes frame-to-frame relative camera poses (both translation and rotation continuous), in contrast to recent interactive world models that quantize camera translation and rotation into fixed directional commands, or keep rotation continuous but restrict translation to discrete keyboard actions.

• Camera conditioning can be toggled per segment. Randomly disabling camera conditioning per segment during training — and removing camera-motion sentences from captions for those segments — lets the model alternate between camera-controlled and text-controlled segments and accept keyboard input (W/S/A/D, mouse or arrow keys) at inference, with speed, scale, and duration adjustable without retraining.

• Human evaluation favors Hy-Nomas. The abstract states that in human evaluation Hy-Nomas shows superior performance over existing world models. The specific evaluation protocol, benchmark names, and numeric scores are not reported in the portion of the paper available here — the results section is truncated after the infrastructure description.

• Training at 360K-token scale is feasible with the described stack. The combination of Ulysses sequence parallelism with N_SP = 8, data parallelism, FSDP2 hybrid sharding, mask-flexible attention kernels, FLOPs-balanced packing, a distributed Muon optimizer, and stage-dependent memory management supports packed sequences of up to 360K tokens, with roughly 45K tokens per rank at maximum sequence length.

• Stage-dependent memory policy matters. During segment forcing, retaining attention outputs increased memory by approximately 1.6 GB per rank without a meaningful speed improvement, so ordinary attention recomputation with streaming backward was used instead.

Methodology in Plain English

Data. The team starts from a large corpus of real and virtual videos. Videos are cut into 5-minute chunks, split again at shot boundaries (using a detector based on OmniShotCut, fine-tuned on high-quality annotations), transcoded to one format, and filtered on properties and quality metrics. Near-duplicates are removed by embedding each video (Qwen3-VL-Embedding), hierarchically clustering, and applying SemDeDup inside each cluster. A VLM is fine-tuned to label every video against a hand-built taxonomy of content and visual attributes, and sampling weights are shifted between training stages toward whatever the previous stage did poorly.

Camera trajectories. ViPE estimates camera motion on a subset of clips, with upgraded calibration and segmentation modules, a modularized pipeline, adaptive spatiotemporal downsampling, and retuned SLAM windows. A separate video-based inverse dynamics model, initialized from a VPT backbone, predicts discrete camera motion directly from video and helps filter clips with unreliable trajectories. A four-step filtering pass rejects static scenes, abrupt motion spikes, extreme point displacement (above 20% or below 30% of max(H, W)), bad focal-length behavior, and degenerate trajectories.

Captions. Captions are structured JSON. Video-level fields hold perspective, style, and a subject list in which every subject gets an ID and a static profile anchored to its earliest clear appearance; segment-level fields hold scene descriptions with camera-motion sentences wrapped in explicit tags, and a separate list of timestamped, non-overlapping actions referencing subject IDs. Annotation is done multi-turn by a VLM: segments are first split by V-JEPA 2 feature changes, then described one at a time with prior segments and annotations kept in context, and finally reviewed in a text-only turn that validates structure and produces aligned English and Chinese versions. Cheaper models (Gemma 4, Qwen3.5-122B-A10B) handle data-intensive stages; Kimi K3 is used for the quality-focused annealing stage.

Model. Video is compressed by a DCAE-V autoencoder (32× spatial, 4× temporal, 32 channels, 1×1×1 patches), text is encoded by a frozen Qwen2.5-VL-7B and refined by a timestep-aware token refiner, and 54 dual-stream MMDiT blocks do the denoising with flow matching. Each block carries an extra cross-attention module where visual tokens query frame-aligned camera tokens. Camera poses are converted to frame-to-frame relative transforms normalised by the 99th-percentile translation magnitude, padded to a multiple of four, and flattened in groups of four 4×4 matrices into 64-dimensional vectors.

Attention. Each segment can attend bidirectionally within itself and causally to earlier segments. Action-aware masking adds a finer split within that: action windows may only see their own action text plus the segment's descriptive head, and action text never crosses segment boundaries.

Training. Starting from the bidirectional base model, training proceeds through segment causality, localized action conditioning, camera control, high-quality annealing, and segment forcing on the model's own generated histories, followed by diffusion distillation. Video and text tokens for both text-only and camera-conditioned segments are packed together with block-diagonal attention, and packs are balanced by estimated FLOPs rather than by token count, since masked attention makes cost depend on the number of unmasked query–key pairs.

Inference. A vision-language Prompt Enhancer turns the user request plus observed history into a structured prompt and a duration for the next segment; the model then denoises that segment with 100 Euler steps using classifier-free text guidance and camera guidance, reusing a bounded KV cache across turns.

Why This Matters

Impact on research. The paper reframes a masking detail as a modelling problem: distinguishing descriptive context that should persist from action instructions that should expire. That framing, plus the released benchmarks and open weights, gives the field a concrete artifact for studying multi-turn world models rather than one-shot video generators. The infrastructure description (360K-token packing, heterogeneous mask kernels, FLOPs-balanced batching) is also a useful blueprint for others training autoregressive video models.

Real-world applications:

• Interactive entertainment and game prototyping — a designer types "the character picks up the crate" and steers the camera to inspect the result, iterating in one continuous session rather than re-prompting from scratch.
• Embodied AI and robotics simulation — using the model as a learned, controllable environment for generating plausible action-to-observation rollouts.
• Film and previsualization — exploring scene changes and camera moves together, with the ability to change speed, scale, and duration of camera motion without retraining.
• Spatial memory and scene-return testing — checking whether a system preserves identities, layouts, and state changes when previously seen content comes back into view, which matters for any long-horizon virtual environment.

Industry relevance. The release of model weights, code, benchmarks, and a lightweight LoRA adaptation workflow lowers the barrier for downstream customization, and the separation of a base model from a distilled counterpart with transferable adapters is a practical deployment pattern for expensive video generators.

Future Directions

• Physical causality beyond temporal causality. The authors explicitly note that temporally causal information flow alone does not establish physical causality, and that learning plausible action-to-state transitions requires data linking actions to observable consequences plus evaluation across successive interactions.

• Compounding error over long rollouts. Because each generated segment becomes evidence for the next, small errors accumulate into appearance drift, implausible motion, and scene breakdown. Segment forcing is the authors' mitigation, but the paper frames this as an ongoing challenge rather than a solved one.

• Richer action and control interfaces. The current interface covers discrete keyboard-style camera motion plus free-form text; extending toward more expressive or more physically grounded action spaces is a natural next step.

• Standardized evaluation of interactive behaviour. The paper's own evaluation suite, which probes multi-turn text interaction, camera navigation, interleaved control, and spatial memory, is presented as a starting point, with remaining challenges in world modeling explicitly left open.

Target Audience

This paper is most useful to advanced researchers and engineers building generative video or interactive world models — particularly those working on autoregressive diffusion, long-context video training, or controllable video generation. It is also valuable to infrastructure engineers interested in sequence packing, heterogeneous attention masks, and memory strategies for long-video training, and to applied teams who want an open-weight, LoRA-adaptable base for interactive content tools. Readers seeking an entry-level introduction to video diffusion will find the paper dense, since much of its argument depends on the mechanics of MMDiT attention and flow matching.

Authors’ abstract

Interactive world generation requires models to follow successive commands while preserving relevant history and action consequences. We present Hy-Nomas, a video world model with an 8B diffusion backbone that supports free-form text instructions and continuous camera control across multiple interaction turns. It generates variable-length segments conditioned on current inputs and cached history. Our central design aligns structured video captions with action-aware causal attention: descriptive context remains accessible across segments, while action instructions are restricted to their intended temporal windows. A vision-language Prompt Enhancer (PE) translates user requests and multimodal interaction histories into structured conditions for the next segment. A multi-stage curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories, followed by diffusion distillation. In human evaluation, Hy-Nomas shows superior performance over existing world models. We release model weights, code and benchmarks, together with a lightweight adaptation workflow, to support reproducible research on interactive world generation.