ROWBench: Do Video Models Render What the Program Specifies?

AI summary

Overview

Research area: Generative video and world models — specifically the evaluation of how faithfully video generation models render the entities, rules, and interactions defined by an executable program.

Technical level: Advanced. The paper assumes familiarity with video world models, diffusion-based video generation, structured conditioning (depth, bounding boxes, segmentation), and VLM-based evaluation metrics.

Scope: The paper introduces PROWBench, a benchmark of 170 programmatically constructed episodes and 600 proxy videos that checks generated video against replayable engine world records rather than against human judgments of plausibility. (The title field of this submission reads "ROWBench," but the paper content consistently names the benchmark PROWBench; arXiv:2610.02205v1 [cs.CV], dated 01 Oct 2026, license CC BY 4.0.)

What This Paper Is About

Programmable world models separate the world's state evolution and rules (maintained by code) from the visual generation that draws what that state looks like. A program can execute perfectly while the video model still renders something visually plausible but wrong — breaking a rule, missing an interaction, or placing an entity in the wrong spot. PROWBench exists to test that gap: it records entity states, timestamped events, and camera trajectories from a game-style engine, then asks whether generated videos actually show the consequences the program executed.

Key Contributions

1. Benchmark. PROWBench comprises 170 programmatically constructed episodes and 600 synchronized proxy videos spanning multiple perspectives, visual representations, and interactions, all sharing world-state and action records.
2. Metrics. The authors evaluate entity control and long-horizon memory against recorded world states, and propose two VLM-based metrics — Logic-Render Alignment and Interaction Success Rate — to test whether generated videos follow the prescribed timeline and visually realize engine-recorded interaction events.
3. Extensible framework. A reusable programmatic generation framework separates scene construction, behavior control, state recording, and rendering, supporting extensions to scene layouts, actions, and visual representations, and enabling aligned multi-view and multi-representation data from shared state records.
4. Diagnostic dataset design. Because each recorded state is rendered into paired proxy representations (coarse 3D, semantic proxy, colored oriented bounding boxes) under identical camera parameters, the benchmark isolates representation and viewpoint effects while holding scene evolution fixed — something the paper states prior benchmarks do not do.

Main Findings

• Proxy conditioning beats camera conditioning on entity following. In the main track, class-mean BBox IoU is higher for proxy-conditioned classes in 38 of 40 Verified-FF scenes and 87 of 89 Unverified-FF scenes (median +0.14, p<10⁻⁸). Class means in Verified-FF are 0.450 (task-specific renderers) and 0.465 (general-purpose video models) against 0.322 for camera-conditioned world models; in Unverified-FF, 0.376 and 0.373 against 0.235.
• General-purpose video models are competitive without task-specific training. In Verified-FF, LynnReal-Omni, MiniMax-H3 and CWM reach BBox IoU of 0.518, 0.494 and 0.454, while the four camera-conditioned world models reach only 0.303–0.351. In Unverified-FF, LynnReal-Omni and MiniMax-H3 reach 0.457 and 0.409, versus 0.204–0.255 for the camera-conditioned models.
• High interaction success does not require high spatial alignment. LingBot 2.0 has the highest Interaction Success Rate in Verified-FF (0.733) despite the third-lowest BBox IoU (0.327). Ranking changes under IoU weighting: MiniMax-H3 achieves the highest gISR in Verified-FF (0.334) followed by LynnReal-Omni (0.324), while LynnReal-Omni leads gLRA (0.429) followed by MiniMax-H3 (0.368). LingBot 2.0 falls to fifth in gISR (0.241).
• Dense geometric conditioning does not guarantee behavioral fidelity. Cosmos-Transfer2.5 receives engine depth and class-level segmentation, ranks fourth and third in BBox IoU, yet has the lowest Interaction Success Rate and Logic-Render Alignment in both main-track settings.
• Timeline adherence is led by LynnReal-Omni and Seedance 2.5. They achieve the highest Logic-Render Alignment in both settings: 0.832 and 0.817 in Verified-FF, and 0.852 and 0.829 in Unverified-FF. In Unverified-FF, LynnReal-Omni also leads gISR (0.379) and gLRA (0.395), followed by MiniMax-H3 (0.326 and 0.337).
• Camera control is nearly saturated. In Verified-FF, rotation error for most methods is 1.7–3.4°, comparable to the pose estimator on the engine renders (1.1–2.7°); only AlayaWorld v1.1 and C2R are clearly worse. LynnReal-Omni has the lowest CamMC in both settings.
• Settings rank methods consistently. Spearman rank correlations between per-method means in the two main-track settings are 0.98 for BBox IoU, 0.93 for translation error, 0.95 for CamMC, and 0.91 for Logic-Render Alignment.
• Long-horizon generation remains hard. Over 30 seconds, BBox IoU is only 0.175–0.280. Reappearance IoU for entities that leave and re-enter is only 0.053–0.131. State Persistence ranks methods differently from Reappearance IoU: CWM has the lowest Reappearance IoU but a State Persistence of 0.886. The authors caution that state scores cover only 17–21 events per method from an uncalibrated judge, and that window-chaining protocols differ (Seedance 2.5 additionally receives the last frame of the previous window), so this measures end-to-end preservation under each method's own protocol rather than isolated internal memory.
• Multi-view generation is not yet satisfactory. C2R obtains the lowest MEt3R point estimate (0.397) yet achieves only 32.2% Appearance Compliance and 25.5% Cross-view Consistency. Methods whose inputs explicitly distinguish participants reach 82–94% Appearance Compliance and 81–90% Cross-view Consistency on judgeable observations. In the multi-view table, MiniMax-H3 leads entity control (IoU 0.524), gISR (0.428), gLRA (0.489) and Appearance Compliance (94.0%), while CWM reaches 92.9% compliance and 88.1% consistency and LynnReal-Omni has the lowest rotation error (0.4°).
• Proxy design is an open problem. Outputs sometimes follow the input geometry too closely and inherit the coarse shapes of low-poly proxies, while box proxies leave open which described appearance belongs to which entity.
• Global similarity metrics can mislead. Figure 1 shows CLIP assigning a higher score to a clearly inferior case, whereas the proposed interaction metric correctly reflects whether the prescribed interaction was executed.
• Coverage limitation of this summary. The provided paper content is truncated mid-sentence in the discussion section, so takeaways T2 through T7 (announced as seven takeaways) are not reported here.

Methodology in Plain English

The authors built a game-engine-style pipeline in three stages. World synthesis creates dynamic worlds from configurations specifying environment, entity layout, actors, task controllers, random seeds, and duration, using procedural routines for terrain, buildings, roads, interiors, and interaction objects. Entities carry persistent identifiers, semantic categories, hierarchies, and attributes, and specialized controllers drive behaviors such as carrying (approaching, grasping, lifting, transporting, releasing), animal activity, simplified flight and swimming, and closed-loop vehicle driving. Every execution produces a structured episode record of transforms, articulated poses, visibility, appearance changes, controller events, and explicit mappings between world execution, saved states, and video frames.

Control compilation turns those records into conditioning inputs. Third-person cameras follow task-dependent trajectories; first-person cameras derive from recorded head positions, character or vehicle orientations, and task-specific gaze. The observing character's own geometry is hidden in its first-person view. Rendering adapters then map the same saved state and camera into three representations: coarse 3D with neutral materials, a semantic proxy combining semantically colored environments with simplified entity proxies, and colored oriented bounding boxes with stable instance colors and a semi-transparent white front face. Because all representations for a view share camera parameters and state, comparisons across them are controlled.

Final process exports scene and motion records, synchronized proxy videos with metric depth, and text and interaction labels, then validates them through task and timeline checks, geometric checks (penetration, support, contact), replay and alignment checks, and visual and media quality checks. Failed attempts are retained separately from accepted episodes with provenance hashes.

Evaluation covers five capabilities: controllability, visual and temporal quality, logic and state alignment, multi-view consistency, and long-horizon memory. Generated entities are localized with SAM 3 from semantic-label prompts and matched to engine-projected references by label-compatible Hungarian matching; BBox IoU measures spatial alignment with missed detections assigned zero, and Center Error measures normalized center displacement conditional on detection. For Interaction Success Rate, Qwen3.6-27B receives eight frames sampled around each registered event (full-frame plus an engine-localized crop) and judges the prescribed action, expected end state, and participant presence; an event counts as successful only when both action and end state are judged present. Logic-Render Alignment gives the judge twelve frames per annotated timeline segment plus neighboring descriptions. The IoU-weighted variants gISR and gLRA multiply each sample's behavioral score by its BBox IoU. For long-horizon memory, an absence event is defined when an entity becomes invisible and later re-enters; Reappearance IoU measures spatial recovery and State Persistence checks whether a returning person shows the prescribed walking or standing state.

Eleven models are evaluated in three families: camera-conditioned world models (EchoWM, SANA-WM, LingBot-World 2.0, AlayaWorld v1.1), proxy-video-to-RGB models (Cosmos-Transfer2.5, C2R, CWM, LynnReal-Omni, and PWM, which is trained only on gunfight scenes and evaluated only on that set), and video foundation models used without adaptation (MiniMax-H3, Seedance 2.5). Each method receives the representation its native interface expects. The main track uses five-second clips in two settings with disjoint scenes — Verified-FF (40 scene-camera pairs: 31 third-person follow, 9 first-person), where the RGB first frame is generated with MiniMax-H3 from the colored-OBB proxy and checked against the recorded state, and Unverified-FF (90 pairs: 78 third-person follow, 12 first-person), where appearance must be inferred from text and control signals. The challenge track adds ten 30-second single-camera recordings and ten four-player scenes with four synchronized third-person follow views each (40 clips per method), generated independently. Methods needing an RGB image receive frame 0 of the MiniMax-H3 output for the same scene and camera; the paper notes this frame can transmit MiniMax-H3's appearance assignments, and that the curation of the Unverified-FF set used MiniMax-H3 outputs, so results involving MiniMax-H3 should be read with that potential bias in mind.

Why This Matters

Impact on research. Prior video benchmarks assess quality, controllability, instruction following, or physical adherence, but the paper argues they cannot judge a generated video against what a program actually executed, because they lack replayable records of entity states and timestamped events. PROWBench supplies those records and paired proxy representations, so failures can be attributed to rendering fidelity rather than to confounded errors in action understanding, state prediction, or memory. The finding that untrained general-purpose video models follow entity trajectories and event logic on par with camera-conditioned world models — and that proxy conditioning beats camera conditioning on entity placement — reframes what explicit camera control and task-specific training actually buy.

Real-world applications:

• Game engines and interactive entertainment. Programmable world models are pitched as a foundation for next-generation engines; this benchmark measures whether a generative renderer can be trusted to depict the engine's authoritative state.
• Simulation and training data. Recorded state plus synchronized proxy representations let teams generate aligned multi-view data for robotics, driving, or embodied agents while keeping ground truth intact.
• Quality assurance for generative pipelines. The logic and interaction metrics provide automated checks that a rendered video obeys a scripted event timeline, useful wherever generated footage must match a specification.
• Agentic content creation. Situations where a coding agent maintains world state and a video model renders it require knowing when plausibility diverges from correctness.

Industry relevance. The benchmark's two-track structure mirrors production realities: short controlled clips versus long-horizon persistence, and single-camera versus multi-player views that must agree on who is where and what they look like. The result that direct multi-view generation still yields unsatisfactory results, and that long-video generation struggles with both camera following and entity persistence, identifies where current systems are not yet deployable.

Future Directions

• Designing better proxies. Outputs sometimes inherit the coarse shapes of low-poly proxies, while box proxies leave ambiguous which described appearance belongs to which entity; the paper calls a good proxy design an open problem.
• Long-horizon memory and appearance persistence. With 30-second BBox IoU at 0.175–0.280 and Reappearance IoU at 0.053–0.131, maintaining entity identity and state across generation windows, and across videos longer than five seconds, remains unresolved.
• Multi-view generation. With the best MEt3R point estimate (0.397) coming from a method that achieves only 32.2% Appearance Compliance and 25.5% Cross-view Consistency, generating independent views that agree on participants, identities, and scene structure is an open challenge.
• Task- or domain-specific training and metric calibration. The authors report that task- or domain-specific training can improve entity placement and realization of domain-specific events, and that the State Persistence judge is uncalibrated and covers only 17–21 events per method, indicating a need for more reliable behavioral judging.

Target Audience

Researchers and engineers working on video world models, generative rendering, and programmable or code-driven world models; benchmark designers who need evaluation grounded in engine records rather than human plausibility judgments; and practitioners building game engines, simulators, or agentic content pipelines who must verify that generated video matches a specification. Readers need a working understanding of video diffusion models and structured conditioning to follow the quantitative comparisons.

Authors’ abstract

Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However, their visual adherence to explicit rules and interactions remains insufficiently evaluated. Existing benchmarks assess visual quality, controllability, and instruction or physical adherence, but rarely test fidelity to fine-grained, program-specified world events. We introduce PROWBench, comprising 170 programmatically constructed episodes and 600 proxy videos covering diverse scenes and interactions. PROWBench logs entity states and timestamped events, including those outside the camera's field of view, as replayable world records, from which it renders synchronized views and proxy representations. This enables generated videos to be checked against the observable consequences of program execution. An extensible framework constructs scenes, controls behaviors, and can render each camera view in different representations, such as coarse 3D, and bounding boxes. The benchmark covers first- and third-person perspectives, with synchronized multi-view observations available for a subset of episodes. Grounded in these records, PROWBench evaluates entity control, long-horizon memory, and, with two VLM-based metrics, Logic-Render Alignment and Interaction Success Rate, adherence to the prescribed timeline and the visual realization of timestamped engine-recorded events.