Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory

AI summary

Overview

Research area: Computer vision and generative modeling — specifically interactive world models built on video diffusion, combining long-horizon memory, explicit camera geometry, dynamic-subject identity preservation, and real-time distillation.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, 3D rotary position embeddings (RoPE), camera intrinsics/extrinsics and projection matrices, VAE latents, autoregressive streaming generation, and distribution-matching distillation (DMD).

Scope in one sentence: The paper describes Matrix-Game 3.5, a real-time interactive world model that unifies a geometry-aware patch-level memory, a pose-aware sequence representation called tiled PRoPE, a static-dynamic disentangled memory, and a two-stage progressive distillation pipeline to produce stable, camera-controllable, minute-long autoregressive world simulation at 720p up to 20 FPS on a single GPU.

Note on completeness: the provided content is truncated inside Section 3.1 (Data Infrastructure). The experimental sections (including Section 4, referenced for the per-target memory budget and for real-time cost) and the appendices are not included, so no quantitative benchmark tables, dataset sizes, or ablation numbers can be reported here.

What This Paper Is About

Interactive world models generate a persistent virtual world that keeps evolving in response to a user's camera movements, actions, and text prompts, rather than producing a fixed short clip. The core problem is that autoregressive generation feeds its own predictions back as context, so visual and geometric errors accumulate, causing scene drift, identity inconsistency, and loss of controllability, while the limited context window makes it hard to recall scenes the camera revisits. Matrix-Game 3.5's goal is to make long-horizon interactive generation stable, geometrically consistent, and fast enough to run in real time, by changing how past observations are stored (as 3D-lifted image patches rather than whole frames or latent tokens) and by distilling a bidirectional diffusion model into a few-step causal generator.

Key Contributions

1. A geometry-aware persistent world representation with no extra learnable parameters. The paper introduces a unified geometry-aware memory framework combining patch memory (originally from MosaicMem) and tiled PRoPE, a video adaptation of PRoPE. Patch memory lifts historical latent patches into a persistent 3D space and retrieves only regions visible from the target viewpoint, while tiled PRoPE establishes explicit geometric correspondence between retrieved memories and target-view attention. Tiled PRoPE tiles the camera projection over all head channels and multiplies it onto the full spatiotemporal RoPE (frame axis included), rather than carving out a disjoint subspace as native PRoPE does.

2. A static-dynamic disentangled world representation. Static scene geometry is stored in patch memory, while dynamic subjects are represented by lightweight multi-view reference tokens. A motion-aware object filter (using YOLO segmentation and tracking) decides by motion rather than semantic category which regions belong to static memory. Combined with motion-aware filtering and leakage-resistant subject training, this is intended to suppress ghosting and identity drift.

3. A progressive long-horizon real-time distillation framework. A two-stage pipeline converts a bidirectional diffusion world model into a three-step fully causal generator: first a teacher-forced causal adaptation stage using Perceptual Flow Matching, then curriculum-based Self-Rollout Distribution Matching (DMD) that optimizes the student's own autoregressive trajectory.

4. A unified training corpus and data infrastructure. The model is trained on data spanning Unreal simulation environments, open-world games, and internet videos, augmented with geometry, language, and object-level annotations plus quality-aware curation. The model files are released as Matrix-Game-3.5-Base and Matrix-Game-3.5-Distilled, with a project page and GitHub repository.

Main Findings

• Camera and memory geometry can be unified without new parameters: The paper states that the patch-memory and tiled-PRoPE components introduce no additional learnable parameters on top of the backbone. The only learnable additions the framework makes are the additive reference-index, type, and local spatial embeddings for reference patch tokens.

• Tiled PRoPE is proposed as a necessary redesign for video. Native PRoPE is designed for novel-view synthesis with unordered views: it splits the attention head dimension into disjoint blocks, with half for camera projection and two quarters for 2D row and column position, so camera and position never multiply and there is no time axis. The authors argue this cannot be transplanted to video because it would take channels away from the pretrained 3D spatiotemporal RoPE whose frame axis encodes temporal ordering, and because a disjoint split leaves no way for a single token to carry both a camera pose and a spatiotemporal position when memory, anchor, and target tokens are mixed.

• A single softmax carries both time and pose. After the standard spatiotemporal RoPE rotation, queries are multiplied by tile(Pᵗᵒᵖ), keys and values by tile(P⁻¹), with the attention output mapped back by P, so one softmax carries the relative time Δt = i − j and the relative pose M_ij = P_i P_j⁻¹.

• Numerical stabilization for large camera motion: All camera poses are recentered with respect to the first target frame of the current generation window, and a direction-preserving logarithmic compression is applied only to the metric-scale translation: t̃ = log(1 + ‖t‖)/(4‖t‖) · t. This maps translation magnitude to log(1 + ‖t‖)/4 while preserving direction, compressing large translations to prevent instability from distant historical viewpoints.

• Patch-level memory is positioned as an intermediate granularity between explicit and implicit memory. Whole frames are described as indivisible memory units retrievable only in their entirety and from their original viewpoint, while explicit point/splat reconstructions impose strong geometric constraints but offer limited flexibility for synthesizing unobserved content. Patch memory is described as combining accurate geometric correspondence for scene recall with generative flexibility, supporting sparse retrieval, localized editing, and robust long-horizon memory updates without accumulating global reconstruction errors.

• Z-buffer selection and hole dropping control memory cost. Retrieved patches are resolved by a z-buffer under the target viewpoint, keeping only the surface closest to the target camera. Regions that are occluded or never observed are dropped from the token sequence rather than kept as placeholder tokens, so the memory track does not double the sequence length.

• Coverage-aware frame selection instead of nearest-frame selection. Historical frames are chosen greedily to maximize additional coverage of the target latent grid: f_k = argmax over f in H∖S of |C(f) ∖ ∪_{s∈S} C(s)|. Selection is per target latent frame and stops when marginal coverage gain vanishes or a per-target budget of five frames (as given in Section 4) is reached.

• Sub-grid positional addressing for memory tokens. Retrieved patches take the RoPE timestamp of the target frame they support and the floating-point spatial coordinate where their source content lands in the target view; valid projections index a higher-resolution RoPE frequency table for sub-grid precision (similar to PE-Field), with a fallback to the nearest native integer-grid coordinate.

• Motion, not category, defines static-memory validity. The motion-aware object filter computes a directional overlap score s_{i→j} from depth-based unprojection and reprojection, and a symmetric static-consistency score s_ij = min(s_{i→j}, s_{j→i}). Tracks with sufficient temporal support and consistently high s_ij are retained as static; low-scoring tracks are excluded from patch memory; tracks with fewer than N_min valid projected samples are marked uncertain and kept outside the static branch. The same criterion is used in training and inference; at inference the filter runs once per completed rollout segment, not per generated frame.

• Four reference images with a fixed-gap timing scheme. Four reference images are VAE-encoded and mapped to a compact latent token budget, each using roughly one quarter of a single frame's token budget, so the four-reference prefix is comparable in length to one additional context frame. The r-th reference is placed at RoPE time t_r = −rΔ_ref before the video timeline, and all reference tokens inherit the camera transform of the current clean anchor rather than a fixed identity camera, keeping the relative transform local to the current generation window.

• Leakage-resistant subject training. Subject regions in anchor and context frames are replaced with background patches sampled from the same scene rather than zeros or blurred noise, so conditioning statistics match between training and inference and direct appearance leakage is prevented. Training adds stronger random mask dilation and perturbation plus a subject-region auxiliary diffusion loss L = L_full + λ_sub · (weighted subject-region velocity error), used with the same flow-matching velocity parameterization as the base model and distillation stages.

• Prefix tokens instead of cross-attention. The paper states that a cross-attention formulation would introduce a separate alignment problem between reference space and video space and add a parameterized conditioning path that could be bypassed via shortcut solutions, so reference tokens stay inside the original transformer token space.

• Two-stage distillation detail. Causal adaptation reformulates generation as chunk-wise causal denoising with a teacher-forced condition H_i^gt = (x_{<i}^gt, p_{≤i}, m_i^gt, r_i^gt, c), constrains flow matching in a pretrained perceptual feature space rather than only in VAE latent space (Perceptual Flow Matching), and thereby learns causal denoising and few-step generation with a single objective. Self-rollout DMD then removes teacher forcing, retrieves patch memory and context frames online from causally visible generated history, and matches the student distribution to the bidirectional teacher under a shared condition. Only stable external conditions (initial memory, anchor frame, text prompt, camera trajectory) are shared with the scorers, while the student updates its own memory and scorer memory stays fixed. A condition curriculum distills classifier-free guidance and camera control first, then gradually introduces patch memory and context frames; following HiAR, the autoregressive prefix and chunk-local context are kept at the next noise level during each denoising substep while retaining a clean anchor.

• Reported real-time target: Figure 1's caption states the system achieves single-GPU 720p real-time generation at up to 20 FPS, and the paper describes a three-step causal generator that preserves quality and control over minute-long real-time interactive streams.

• Qualitative claims: The paper claims strong performance in long-horizon scene recall, precise and geometry-consistent camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction. No numerical benchmark scores, dataset sizes, or comparison tables appear in the supplied excerpt.

Methodology in Plain English

The authors start from a pretrained video diffusion model and reorganize how it sees its own history.

One shared sequence. Instead of routing different kinds of information through separate branches, every token type — an anchor frame, a few clean past context frames, retrieved memory patches, reference images of the controllable character, and the noisy frames currently being generated — is placed into a single sequence. Each token carries a time coordinate on the RoPE timeline and a camera pose. The text prompt is the only thing injected outside this sequence, via the backbone's native cross-attention.

Camera geometry baked into attention. For each latent frame the method builds a world-to-image projection matrix P from the camera intrinsics and the inverse of the camera-to-world pose. Rather than reserving part of the attention head dimension for camera information, it tiles that projection across all channels and overlays it on the existing spatiotemporal RoPE, so a single attention computation encodes both how far apart two frames are in time and how they relate geometrically. Translation magnitudes are compressed logarithmically to keep distant history numerically well behaved.

Memory as 3D-lifted patches, not whole frames. Each stored memory entry is a VAE latent patch. Using the camera pose, intrinsics, and a metric depth map, each historical patch is lifted into a shared 3D space and reprojected into the target camera. A z-buffer under the target view keeps only the closest surface when several patches land on the same target location; anything occluded or never seen is simply dropped, leaving the diffusion model to synthesize those areas from the current latent, the text prompt, and surrounding context. Which historical frames to use is chosen greedily for maximum new coverage of the target view, up to a budget of five frames per target latent frame.

Separating what should persist from what should move. Because a moving character observed at different times would confuse a purely spatial memory, a motion-based filter (built on YOLO segmentation and tracking) decides which tracked regions behave like static scene and should enter patch memory, and which are dynamic and should be masked out. Dynamic subjects instead get their identity from four reference images carried as prefix tokens with fixed negative time coordinates. During training, subject pixels in the conditioning frames are replaced with background from the same scene so the model cannot cheat by copying appearance from the context, and an extra loss term upweights gradient on subject regions.

Making it fast. Because the base model is bidirectional and multi-step, the authors distill it. Stage one teaches causal, chunk-by-chunk denoising with a teacher-forced setup and a perceptual (feature-space) flow-matching objective, producing a few-step causal initializer in one training signal. Stage two drops teacher forcing and lets the model roll out on its own generated history, matching its distribution to the teacher's with a DMD objective, while a curriculum first distills guidance and camera control and only later introduces the patch memory and context frames. The result is a three-step causal generator.

Data. Because raw video lacks the needed supervision, the authors build an offline pipeline that annotates videos with geometry (poses, depth, intrinsics), language, and object-level information, then curates by quality. Training data spans Unreal simulation environments, open-world games, and internet videos.

Why This Matters

Impact on research. The paper reframes long-horizon interactive world modeling as a single joint problem of memory, geometry, dynamics, and efficient autoregressive generation, rather than four separately solved issues. Its most transferable idea is the memory granularity argument: whole frames are indivisible and tied to their original viewpoint, explicit 3D reconstructions are rigid, and latent-token memories are implicit — patches sit in between and, notably, add no learnable parameters. Showing that camera geometry can be overlaid onto an existing spatiotemporal RoPE without a dedicated subspace is also a reusable trick for any video model that needs pose conditioning.

Real-world applications:

• Open-world games and interactive content creation: persistent environments that remember previously visited locations and let players drive the camera while the world keeps evolving.
• Extended reality (XR): real-time, camera-controllable scene generation that a user can look around in.
• Embodied agents and robotics simulation: a world model that maintains scene geometry and object identity as an agent acts, providing a stable environment for training and evaluation.
• Prompt-driven scene editing and prototyping: because memory is patch-level rather than globally reconstructed, the paper notes it supports sparse retrieval and localized editing, useful for quickly iterating on virtual spaces.

Industry relevance. The headline engineering claim is single-GPU 720p generation at up to 20 FPS with a three-step causal model, which moves world models from offline clip synthesis toward deployable interactive systems. The release of both a Base and a Distilled checkpoint, a project page, and a GitHub repository suggests an intended role as a practical base for further research and product integration.

Future Directions

• Full quantitative characterization. The supplied content omits the experimental section, so the relative gains from patch memory, tiled PRoPE, static-dynamic disentanglement, and each distillation stage over Matrix-Game 3.0 and prior memory paradigms (RELIC, Context as Memory, WorldMem, VMem) remain to be examined in detail.

• Memory scaling and robustness over very long rollouts. The paper notes that patch-level memory avoids accumulating global reconstruction errors over repeated revisits; how the 3D patch cloud should be pruned, updated, or compressed over hour-scale sessions — and how depth errors propagate into retrieved memory — is an open question.

• Beyond third-person game settings. The paper explicitly states that "in the current third-person game setting, the dynamic subject refers to the controllable character," leaving open how the reference-token design generalizes to multiple dynamic subjects, first-person views, or deforming and articulated objects.

• Dependence on the geometry annotation pipeline. The whole approach rests on camera poses, metric depth, and intrinsics supplied by offline annotation; the sensitivity of tiled PRoPE, patch retrieval, and the motion filter to annotation error, and whether geometry can instead be estimated online during rollout, are natural next steps.

Target Audience

This paper is for researchers and engineers working on video diffusion, interactive world models, and generative simulation — particularly those already familiar with diffusion transformers, RoPE variants, camera parameterization, and distillation-based acceleration. It should also interest graphics and game-engine practitioners evaluating real-time neural world simulation, and embodied-AI or robotics researchers who need a long-horizon, camera-controllable environment model. Readers seeking a beginner-level introduction to world models will find the paper dense, since its derivations assume comfort with projection matrices, attention mechanics, and distribution-matching distillation.

Authors’ abstract

Interactive world models extend video generation from offline clip synthesis toward persistent simulation of interactive virtual worlds, enabling applications in games, robotics, embodied agents, and XR. Achieving stable long-horizon interactive generation, however, remains challenging, as the model must simultaneously preserve scene geometry, dynamic consistency, and camera control while supporting real-time autoregressive generation. Building upon Matrix-Game 3.0, we present Matrix-Game 3.5, as shown in Figure 1, which advances real-time interactive world generation toward geometry-aware and long-horizon consistent simulation through three key improvements. First, we propose a unified geometry-aware memory framework, whose patch-memory and tiled-PRoPE components introduce no additional learnable parameters, combining explicit 3D patch retrieval with projective camera conditioning to enable geometry-consistent camera control and faithful long-horizon scene recall. Second, we introduce a static-dynamic disentangled world representation that separately models static scene geometry and dynamic subjects, preserving both geometric consistency and subject identity throughout long-horizon generation. Third, we develop a two-stage progressive real-time distillation framework that converts a bidirectional diffusion model into a few-step causal generator through Perceptual Flow Matching and curriculum based Self-Rollout DMD, enabling minute-long real-time interactive generation. Extensive experiments demonstrate that, with a unified training corpus spanning Unreal simulation environments, open-world games, and internet videos, MatrixGame 3.5 achieves strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.