World Editing: Intervening on Executable Worlds at Increasing Depth

AI summary

Overview

Research area: Artificial intelligence / world models, agentic coding, and game modding. The paper sits at the intersection of interactive world models, repository-level software engineering agents, and executable game environments.

Technical level: Advanced. The core ideas are explained accessibly, but the work presupposes familiarity with coding agents, benchmark design, and perceptual similarity metrics.

Scope: The paper formulates "world editing" as intervening on an existing executable world while preserving what should not change, and introduces the IGMWorld environment plus the 110-task IGMBench benchmark across Minecraft and Terraria to measure how reliably frontier agents perform such edits at four increasing depths.

What This Paper Is About

Most AI research on interactive worlds studies either generating worlds or acting inside them. Far less studied is whether an AI system can deliberately edit an existing world — changing an entity, a rule, or an entire coupled system — while leaving unrelated properties intact. The paper defines this capability as "world editing," organizes it along an axis called "intervention depth," and builds an executable testbed (IGMWorld and IGMBench) in commercial games so that edits are judged by whether the resulting world actually behaves as requested rather than by the code alone.

Key Contributions

1. A formulation of world editing as intervention. The authors formalize editing as transforming a world W = (S, T, R, …) — its entities and state space, transition dynamics, and governing rules and systems — via an intervention Δ_k of type k, producing W′ that realizes the intervention while preserving unrelated properties.

2. Intervention depth as a four-level taxonomy. Property (L1), entity (L2), dynamics (L3), and system (L4) interventions, mapped respectively to parameter editing, content editing, mechanic editing, and system editing in game-modding terms. Depth is defined by how strongly an edit couples world components, not by how much code it requires.

3. IGMWorld and IGMBench. An execution environment plus a benchmark of 110 tasks (57 Minecraft, 53 Terraria) and over 1.1K executable state and behavioral criteria (1,161 total), with deterministic state checks, behavioral checks, regression checks, and visual validation.

4. A characterization of current frontier agents on this capability. Results show reliability falls with intervention depth, most failures occur after the world is already executable, and visual consistency is a separate, much weaker capability.

Main Findings

• Frontier agents already edit worlds substantially, but task-level success lags criterion-level success. Across the seven evaluated configurations, World-Editing Success Rate (WSR, all criteria of a task satisfied) ranges from 31.8% to 78.2%, with GPT-5.6 Sol reaching 78.2%. Criterion Pass Rate (CPR) is much higher, reaching 94.8% for GPT-5.6 Sol.

• Reliability decreases with intervention depth. Every evaluated configuration achieves its highest WSR at L1. GPT-5.6 Sol drops from 96.3% WSR at L1 to 48.1% at L4; Claude Opus 4.8 drops from 92.6% to 55.6%. The relative ordering of L3 and L4 varies across models and games.

• Depth is not reducible to the number of requirements. Deeper tasks contain more criteria (median criteria rise from 8 at L1 to 15 at L4), so the authors stratify by criterion count. Within comparable criterion-count ranges, L1 remains substantially more reliable and L3/L4 generally remain below shallower interventions.

• Most failures happen after the edit builds and loads. Behavioral failures dominate unsuccessful runs at every level, while build and load failures are comparatively uncommon. Even at L4, where build failures become more frequent, behavioral correctness remains the dominant failure stage. The bottleneck is realizing requested behavior, not producing compiling code.

• The composition of behavioral failures shifts with depth. L1 failures are almost entirely mechanic semantics (a wrong value or formula). From L2 onward, interaction/progression failures — new content misbehaving when in contact with existing systems — become the largest category and rise to 43% at L4. Resource/registration failures stay at 13–24% and do not grow with depth.

• Preservation is only partially tested. 225 regression checks across 30 tasks show few unintended changes on the tasks where checks exist, but coverage is concentrated at L1, so preservation under deeper interventions remains a benchmark limitation.

• Visual consistency is a distinct bottleneck. Native game assets achieve joint style-and-semantic pass rates of 0.80 in Minecraft and 0.78 in Terraria, whereas every evaluated agent configuration remains below 0.50. The strongest joint pass rates reach 0.47 in Minecraft and 0.29 in Terraria. Visual performance does not follow the ranking for functional editing.

• Grounding strategies differ by harness. GPT-5.6 Sol, GPT-5.6 Luna, and Claude Opus 4.8 inspect existing game sources in 91–99% of runs, while Gemini relies more heavily on online search. Rebuilding after an initial implementation is common across configurations.

Methodology in Plain English

Each benchmark task packages a natural-language edit request with a scaffold mod repository, an executable environment, and a hidden validator. The agent works through an edit → build → run → observe loop: it can read and modify source and visual assets, invoke the native build toolchain, launch the game server, and use runtime logs to revise its implementation. The agent sees only the request and a natural-language list of state and behavioral requirements; the executable grading logic is withheld. Every run has a fixed wall-clock budget of T = 3600 seconds and is evaluated once per model–task pair.

Tasks were authored by 11 annotators with prior game-modding experience, given level definitions and examples but asked to write novel requests; 110 tasks survived review for clarity, feasibility, and level consistency. The benchmark uses native ecosystems — Fabric for Minecraft (pinned to Minecraft 1.21.1 with Fabric and Java 21) and tModLoader with its bundled .NET runtime for Terraria — with fixed-seed worlds and shared runtimes per game.

Evaluation proceeds in three stages. First, executability: the submission must compile into a valid mod package and load without crashing, as strict gates. Second, world-edit correctness: task-specific state checks query running-game properties (entity attributes, item statistics, recipes, registrations) and behavioral checks execute controlled in-game actions and verify the resulting state transitions, with paired regression checks comparing modified and original worlds where available. Validators judge observable behavior, so different implementations can satisfy the same intervention. Third, visual consistency: for introduced or modified assets, a text-conditioned consistency score S_f(x;c) averages TPIPS similarity between an asset and its k nearest category-matched references, with thresholds calibrated from the α-quantile of leave-one-out scores over native assets; semantic consistency uses the requested object identity as the text factor and style consistency uses "art style."

Seven agent configurations were evaluated: GPT-5.6 Sol and GPT-5.6 Luna (Codex-CLI), Gemini 3.5 Flash (Gemini-CLI), Claude Opus 4.8 (Claude Code), and DeepSeek-V4-Pro, Kimi-K3, and GLM-5.3 all sharing Hermes Agent as a harness. All configurations had Internet search and an optional image-generation tool — gpt-image for Codex-based configurations, nano-banana for the others — plus the same generic modding skills and optional sprite post-processing pipeline, with no task-specific or evaluator information.

Why This Matters

Impact on research. The paper argues that world editing is a distinct capability from world generation and world interaction, and provides a way to construct controlled variants of an executable world while preserving its surrounding structure. That makes intervention depth a natural axis for organizing increasingly coupled changes, and the benchmark a testbed where correctness is judged on runtime world behavior rather than code structure. It also positions world editing as complementary to code-based world models: rather than constructing or inferring a world representation, it asks whether an agent can intervene on one that already exists.

Real-world applications:

• Content and mod creation for existing commercial games, where an agent modifies entities, mechanics, or whole subsystems instead of authoring everything from scratch.
• Generating controlled world variants for training and evaluating future world models and agents, since edits preserve much of the original world while varying specific properties.
• Robustness and generalization studies for coding agents operating on large, unfamiliar repositories with real toolchains and runtime feedback.
• A testbed for multimodal agents that must coordinate code, game data, runtime logs, and visual assets in a single task.

Industry relevance. The tasks are drawn from industry-grade game modding ecosystems (Fabric and tModLoader), and the evaluation covers models and harnesses from OpenAI, Google DeepMind, Anthropic, DeepSeek, Moonshot/Kimi, and GLM. The finding that most failures are behavioral rather than build-related, and that visual integration lags far behind functional correctness, is directly relevant to anyone deploying agents on real software repositories or shipping generated assets into an established art pipeline. The authors note that the formulation is not specific to these games; they show qualitative property, entity, and dynamics interventions in PEAK, Palworld, and Starbound, though these are not part of the benchmark or its quantitative evaluation.

Future Directions

• Stronger and broader preservation testing. The current regression checks (225 across 30 tasks) are concentrated at L1, so preservation under deeper interventions is only partially tested. The authors call for stronger regression testing and more extensive runtime coverage, including long-horizon and multiplayer scenarios for interaction-dependent failures.

• Generalizing beyond two games. IGMBench is instantiated in Minecraft and Terraria only. Other games expose different APIs, asset pipelines, runtime interfaces, and modding constraints, and each environment requires ecosystem-specific execution and validation infrastructure.

• Extending evaluation beyond state, behavior, and visual assets. Audio, animation, and narrative are not explicitly evaluated; the authors note such modalities are less naturally captured by deterministic state checks and would require modality-specific infrastructure and validation protocols.

• Repeated evaluation for reliability estimates. Each model–task pair was evaluated once due to the cost of native game execution and multimodal validation, so the results characterize observed configuration-level performance rather than per-task success probabilities across repeated stochastic runs.

Target Audience

Researchers working on world models, interactive environment generation, and code-based world representations; agent and coding-agent researchers interested in repository-level tasks with runtime verification; game AI and procedural content generation researchers; and practitioners building or auditing agents that modify complex, executable software systems. Benchmark designers will also find the taxonomy, evaluation staging (executability, correctness, preservation, visual consistency), and the documented limitations useful as a template.

Authors’ abstract

Interactive world models are increasingly capable of generating environments and acting within them, yet deliberately editing an existing executable world remains underexplored. We formulate world editing as intervening on an existing world while preserving properties that should remain unchanged, and introduce intervention depth as an axis describing how strongly an edit couples world entities, dynamics, and systems. We instantiate this capability through industry-grade game modding and introduce IGMWorld, together with IGMBench, a benchmark of 110 tasks and over 1.1K executable state and behavioral criteria across Minecraft and Terraria. The tasks span property, entity, dynamics, and system interventions and are evaluated through deterministic executability, behavioral, preservation, and visual checks. Frontier coding agents already exhibit substantial world-editing capability: the strongest configuration solves 78.2% of tasks under a strict task-level criterion, while criterion-level performance reaches 94.8%. Reliability generally decreases with intervention depth, and this pattern persists even among tasks with similar numbers of evaluation criteria. Most failed edits still build and load successfully, suggesting that the main difficulty is making the edited world behave as requested. Visual consistency remains a separate weakness, with all evaluated configurations below 50% joint visual pass rate. These results show that world editing is a distinct capability from world generation and interaction, and that executable games provide a practical testbed for studying it.