SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models

AI summary

Overview

Research area: Computer vision — interactive video world models, controllable video generation, and large-scale multimodal data engineering.

Technical level: Advanced. The paper combines diffusion/flow-matching training, causal autoregressive distillation, SLAM-based camera annotation, and WebDataset-scale data infrastructure; readers are assumed to be familiar with latent video generators, attention masking, and distillation objectives.

Scope: SolarWM is a fully open release — data engine, processed corpus, training recipes, four adapted model weights, and inference code — for turning pretrained bidirectional video generators into camera-controllable, long-horizon autoregressive world models.

What This Paper Is About

Interactive world models need to generate long, coherent video that responds to camera motion, but the field is fragmented: existing datasets differ in temporal scale, camera conventions, quality and captioning style, while video generators differ in architecture and latent representation. Naively mixing data or building a separate training pipeline per model produces inconsistent supervision and results that are hard to reproduce or compare.

SolarWM's goal is to fix this coupling with one open foundation: a reconfigurable multi-source data engine that normalizes heterogeneous clips into a shared frame-aligned contract, plus a backbone-native adaptation framework with shared camera-conditioning, training, and inference interfaces that is instantiated across four different video backbones.

Key Contributions

1. A fully open, unified foundation. SolarWM integrates a reconfigurable multi-source data infrastructure with a scalable backbone-native adaptation framework spanning data construction, model training, and long-horizon inference, with commitments to release data, pipeline, recipes, weights, and framework.

2. A reconfigurable multi-source data engine. Approximately 1.43 million canonical clips from 10 datasets covering real-world, synthetic, and game environments, organized into 14 independently addressable "dataset owners," with source preprocessing decoupled from training-mixture construction so users can change filters, weights, or temporal views without rerunning expensive preprocessing.

3. A scalable backbone-native model family. Four models — SolarWM-wan2.2-5B, SolarWM-wan2.2-14B, SolarWM-ltx-2.5-22B, and SolarWM-minimax-h3-33B — built on Wan2.2, LTX-2.5, and MiniMax-H3, trained with one unified three-stage recipe while preserving each backbone's native representations and objectives.

4. A simple, efficient three-stage training recipe. Bidirectional camera-conditioned adaptation, teacher-forced AnyFlow autoregressive initialization, and DMD-based causal training; the paper reports that this avoids the separate Causal ODE and Causal Consistency Distillation initialization stages used by prior causal-forcing approaches.

Main Findings

• Long horizons from short training clips. Models trained solely on 5s sequences are reported to support open-ended rollouts over minutes-to-hours horizons without additional long-sequence fine-tuning or attention-sink mechanisms.

• No specialized initialization is needed. The authors report achieving strong results without specialized ODE or consistency-distillation (CD) initialization, because the teacher-forced AnyFlow stage directly produces a few-step autoregressive initializer.

• Training effort concentrates in the first stage. The paper states that most optimization should be performed during bidirectional training; AR adaptation then converges rapidly, while DMD requires even fewer optimization steps.

• Corpus retention and storage. Of the 1.4M canonical clips, 876k are retained — 471k in the high tier and 404k in xhigh — with the remaining 549k fully processed clips published in a rejected partition. The complete corpus comprises 29k shards and approximately 25.85 TB of physical storage.

• Clean-plate derivatives are a substantial separate resource. 543k clean clips are produced: 298k SpatialVID-Clean (73k at 81 frames and 224k at 160 frames), 135k MiraData-Clean, and 109k Sekai-Walking-Clean. These are independent recipe owners rather than silent replacements for the original datasets.

• Recipe-level reproducibility via indices. The 81f short recipe spans ten owners and contains 600,320 physical training rows; source balancing applies a repeat factor of six to ABOT, MiraData, and Sekai-Game, producing 870,210 virtual occurrences per epoch, with a held-out test view of 1,000 rows. Independent 100-sample test views are published per owner (1,400 rows total).

• Quantitative benchmark scores are not reported in the available content. The truncated text states that the models achieve "state-of-the-art performance" but does not provide benchmark names, metric values, or head-to-head evaluation numbers. Any comparison figures would need to be read from the full paper.

Methodology in Plain English

Data first, as an engine rather than a list. Every clip from every source is processed and annotated before any training-time selection happens. Clips that fail the default recipe are kept in a rejected partition with their annotations and machine-readable rejection reasons, so thresholds can be changed later without redoing camera estimation, captioning, or quality assessment.

One unified sample contract. Each sample is stored as video, metric camera-to-world transforms (N×4×4), per-frame intrinsics (N×4, as fx, fy, cx, cy), a dense caption, source metadata, a full metric record, and provenance. Video, poses, and intrinsics must share exactly the same frame indices. Three namespaces are kept separate: the physical corpus, the logical recipe (splits, tiers, weights, repeat factors), and the model view (backbone-specific windows or latents).

Camera annotation. For video-only sources, Pi3X estimates temporally consistent but scale-ambiguous structure while MoGe-2 supplies per-frame metric depth anchors; the depths are fused and passed to a modified VIPE SLAM backend, which estimates a 6-DoF trajectory with bundle adjustment and independently optimizable per-frame intrinsics initialized by GeoCalib. Where ground-truth or COLMAP poses exist, the supplied trajectory is preserved and Pi3X only connects predicted structure to the trajectory's metric gauge via a robust Umeyama Sim(3) fit re-estimated from the lowest-residual 80% of frames.

Clean plates. A separate LTX-2.3 Clean Plate IC-LoRA pipeline removes people and vehicles while preserving scene layout and camera motion, using eight denoising steps at strength 1.0, processing at 704×1248 and saving at 1280×720 at 16 fps. Captions, visual metrics, semantic metrics, and camera diagnostics are recomputed for every output.

Captioning and annotation. All 1.43M clips are captioned by one Kimi-K2.6 pipeline that sees the entire clip at 1 fps (at most 64 frames, max image edge 768 pixels, JPEG qscale 3, no audio) and returns a strict JSON object with six fields. The caption itself is a 60–150 word English paragraph describing only persistent environment content; people, animals, actions, camera motion, shot terminology, and speculation are excluded so text does not leak camera-control information. Only the caption becomes a text condition; the other fields (entity density, quality, reject flags, scene type, scene transition) are annotations. Quality and motion metrics include DOVER technical and aesthetic scores, FFmpeg VMAF Motion, UniMatch/GMFlow correspondence magnitude, PySceneDetect cut counts, saturation, and camera-integrity checks.

Source-aware filtering. Each owner gets a frozen versioned policy producing three disjoint labels: xhigh (kept rule plus stricter promotion rule), high (kept rule only), and rejected. Policies are deliberately asymmetric — for example Sekai-Game preserves high-motion game trajectories without generic camera-geometry or saturation gates, while clean owners require post-transformation geometry checks.

Three-stage training. Stage one trains the pretrained generator under bidirectional attention on camera-conditioned data, with camera control injected through fused-PRoPE so projective rotations act directly on query, key, and value tensors in the existing self-attention path — no extra control branch or attention pass. Stage two switches to causal attention and combines teacher forcing (clean ground-truth history, future blocks hidden) with the AnyFlow loss over arbitrary noise-level pairs, yielding a few-step autoregressive initializer. Stage three applies DMD: a causal student generates trajectories under the inference-time temporal rule with a detached rollout-and-replay procedure to allow KV Cache gradients, a frozen bidirectional teacher supplies the real score, and a trainable fake-distribution model tracks the student's evolving distribution; their difference gives the student its update direction.

Why This Matters

Impact on research. Most released world-model systems ship checkpoints or demos, and even those that release training code often omit processed data, selection records, exact mixture recipes, or checkpoint-matched configurations. SolarWM's release matrix (verified from official artifacts as of August 18, 2026) argues that multi-backbone releases typically cover only two or three backbone families, leaving it unclear whether a training design transfers broadly or depends on model-specific engineering. By releasing a recipe-complete, four-backbone stack, SolarWM makes controlled cross-backbone comparison and end-to-end reproduction possible.

Real-world applications:

• Simulation and embodied learning. Camera-controllable, hour-scale rollouts provide trainable environments for agents that need to navigate and act in visually realistic settings.
• Interactive content creation. Creators can explore generated scenes by moving a virtual camera rather than generating isolated clips, which matters for previz, virtual production, and game prototyping.
• Robotics and autonomous-driving data augmentation. Metric-scale camera trajectories and clean-plate derivatives (dynamic agents removed) give a source of geometrically consistent, controllable video for perception and planning experiments.
• Data-tooling and dataset engineering. The open pipeline (Kimi-K2.6 captioning contract, Pi3X/MoGe-2/VIPE camera annotation, DOVER/VMAF/UniMatch quality metrics) is reusable for building new video datasets with auditable provenance.

Industry relevance. The finding that long-horizon capability can emerge from 5s training clips, without long-video training or attention-sink mechanisms, suggests substantially lower training cost for interactive video products. The decoupling of source preprocessing from mixture construction also means organizations can adapt the released corpus to their own domain without paying for repeated annotation.

Future Directions

• Publish and verify quantitative evaluation. The available content asserts state-of-the-art performance but reports no benchmark names or metric values; head-to-head numbers against the systems in the release matrix would substantiate the claim.
• Test recipe reconfiguration at scale. The engine is explicitly designed so filters, source weights, temporal views, and model readers can be swapped; how much these choices move quality for world models (versus general video generation) is an open empirical question.
• Broaden backbone coverage further. Four models across three backbone families test transferability, but it remains unknown whether the three-stage recipe holds for backbones with fundamentally different conditioning or latent structures.
• Extend beyond camera control. The current recipe conditions on camera motion, text, and image; the introduction also mentions player actions and semantic instructions, which the released recipe does not yet cover.
• Study what long-horizon rollout actually degrades on. Since training and DMD use short windows, characterizing drift, memory retention, and failure modes over hour-scale rollouts is a natural follow-up.

Target Audience

Researchers and engineers building interactive video world models, video diffusion or flow-matching practitioners interested in causal autoregressive conversion and distillation, and dataset/infrastructure teams who need an auditable, reconfigurable multi-source video pipeline. It is most valuable to readers who want to reproduce or extend a complete training stack rather than only consume a demo — the paper is written at an advanced level and assumes familiarity with attention masking, flow matching, DMD, and SLAM-style camera estimation.

Authors’ abstract

We introduce SolarWM, a fully open foundation for building interactive video world models from data preparation through long-horizon inference. Training across heterogeneous data sources and video backbones is challenging: datasets differ in temporal scale, camera geometry, visual quality, motion, and captioning styles, while video generators use distinct representations and architectures. Naive data mixing and model-specific implementations therefore produce inconsistent supervision and make results difficult to reproduce and compare. SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework. The engine converts 1.43 million canonical clips from 10 datasets into a unified, frame-aligned contract covering visual observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance, while decoupling source processing from mixture construction. Under shared camera-conditioning, training, and inference interfaces, we instantiate four 5B--33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives. A unified three-stage recipe combines bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation. The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences. By releasing the resulting data, pipeline, recipes, weights, and framework, SolarWM provides a reproducible and extensible foundation for interactive world-model research.