AgentGarten: Code Worlds for Evolving Agents

AI summary

Overview

Research area: Interactive virtual worlds for embodied agent learning, combining executable simulation environments ("code worlds") with real-time neural video rendering.

Technical level: Advanced. The paper assumes familiarity with autoregressive video generation, distribution matching distillation, adversarial training, attention kernels, and KV caching.

Scope: The paper introduces AgentGarten, a framework that pairs simulator/game-engine world state with a single shared real-time neural renderer, and studies pretrained agents that improve across rounds of interaction by writing playbooks.

What This Paper Is About

Simulators and game engines give agents faithful, programmable state and rules, but expanding their visual diversity requires substantial 3D asset creation and complex rendering pipelines. Video world models instead synthesize rich observations from data, but their task-relevant state and transition rules stay implicit inside generated histories, so they cannot be inspected, edited, or tested. AgentGarten addresses this trade-off by letting each scene run as a program in an engine that maintains explicit state and interaction rules, while one shared neural renderer turns the engine's structured conditions (colorized depth or surface normals) into the agent's visual observations.

Key Contributions

1. A framework for code worlds. It couples persistent, editable environment state to a shared real-time neural renderer through structured conditions, enabling programmable interaction and diverse visual observations across compatible engines without per-scene visual asset authoring. State transitions take no rendered observation as input, so rendering errors cannot accumulate in the state.

2. Adversarial Forcing. A distillation method for a few-step, block-causal neural renderer. It combines distribution matching on self-rollouts with exact bitwise replay for history gradients, alongside a real-data adversarial objective whose exact R1/R2 regularization avoids double-backward passes through fused attention kernels.

3. An empirical study of evolving agents. Pretrained agents that observe exclusively through neural rendering rapidly ground semantic tool knowledge and improve outcomes across rounds through written playbooks in hide-and-seek and several further distinct worlds.

4. Authoring worlds from image or text. A coding agent builds an executable scene program from a single image or a text description, using perception and generation models as tools (instance masks, monocular depth and camera intrinsics, 3D bounding boxes, extracted object meshes), then repairs the scene through an interactive feedback loop that inspects test rollouts and engine verification queries (contact, collision clearance, occlusion).

Main Findings

• Fewer rounds than millions of episodes. In hide-and-seek, hiders used panels to build a shelter by round 4, and seekers used a ramp to cross walls by round 10. The paper compares this with self-play reinforcement learning in the 2019 study, where shelter construction emerged after roughly 25 million episodes and ramp use after another 75 million (listed as approximately 100 million in the milestone table). The authors state these numbers reflect two fundamentally different learning paradigms rather than a direct sample-efficiency ratio, since the pretrained agents already possess abstract knowledge from web-scale pretraining and must ground it into closed-loop execution.

• Adversarial Forcing beats Self Forcing qualitatively. In 30-second rollouts on the same condition trajectories, renderers trained with Self Forcing develop repetitive surface patterns and lose scene detail as rollouts grow longer, while Adversarial Forcing retains natural textures and fine detail throughout.

• Exact replay is bitwise identical. Changing only the attention execution, a full-sequence FlexAttention replay differs from the cached rollout by 3.99% relative L2 error, while block-by-block SDPA replay is bitwise identical. Forward time decreases by 5.4% (2.95 s to 2.79 s) and peak allocated memory changes by 0.2% (50.83 GiB to 50.92 GiB). Measured on one H100 with 36 layers, 480×832 resolution, 61 latent frames, and BF16; times are medians of three warmed-up runs of the replay forward pass and exclude backward and optimizer updates.

• Real-time rendering on one GPU. The renderer generates 480×832 video at over 35 frames per second on one NVIDIA H100, including condition encoding, four denoising steps, decoding, and host transfer. One served block of sixteen frames breaks down as 10.6 ms condition encoding, 414.4 ms for the Transformer (four denoising steps and cache publication), and 8.8 ms for the tiny decoder and host transfer, totaling 438.8 ms wall time and 36.5 frames per second. The measurement uses five sink and 44 recent latent frames in the cache plus four current latent frames, so each four-frame block attends to 49 frames of history; it is taken at steady state (blocks 17 to 48 of a 48-block, 769-frame rollout, averaged over two rollouts), with dense attention and the tiny decoder in FP16. The distilled tiny decoder reconstructs pixel frames in under 10 ms.

• The same practice loop works across worlds. Across four additional worlds, each run for four rounds: the companion-dog engagement score rose from 13 in round 1 to 19 in rounds 3 and 4 (13, 14, 19, 19); on the one-lane bridge, seconds until both cars arrive fell from 71 to 41 (71, 68, 45, 41); in herding, the score out of 100 went 60, 90.1, 87.6, 88.3, with sheep penned going from 3 of 4 in round 1 to 4 of 4 in every later round; in the quarry, the loader scored 30, 0, 30, and 90.9, clearing the rocks, delivering one, and parking in round 4 in 329 of its 360 seconds.

• Agents see only rendered frames. In all worlds, agents act directly on first-person observations synthesized in real time by the neural renderer, without access to internal simulation state or raw engine geometry buffers. In hide-and-seek they submit short Python programs for movement and object interaction and never receive object coordinates, hidden world state, or the opponent's private observations.

Methodology in Plain English

Separating state from appearance. Each code world is a scene program plus an engine plus a neural renderer. The engine updates state from the agent's actions and exports a structured condition map from the camera; the renderer produces the observation from that condition, an appearance reference image, a text description, and the visual history so far. Because the state transition never reads rendered pixels, a recorded state trajectory can be re-rendered under a different appearance reference or camera without altering what happened.

Conditions. Depth or surface normals are used as structured conditions, encoded into three channels so the pretrained video encoder and Transformer can be reused. Real videos get depth from ViPE (with Depth Anything 3 as its depth backend) and normals from NormalCrafter; simulated scenes supply these maps directly from geometry. For robustness, the authors randomly keep depth or normals, sometimes drop both (dropped inputs are black), suppress fine texture by down- and upsampling, perturb alignment with a smooth spatial warp anchored to the first frame, and add noise to condition latents.

Training the renderer. Starting from Cosmos 3-Nano with its understanding and generation towers separated, a frozen Wan video autoencoder maps RGB and geometry videos to latents and the appearance reference is passed as a clean frame with no diffusion timestep. Training proceeds in three stages: geometry conditioning with geometry and RGB tokens concatenated along the sequence dimension and processed by shared Transformer layers (no channel concatenation or control branches); teacher-forcing adaptation to blockwise autoregressive generation, where each block is generated from aligned geometry and previously completed blocks with a block-causal mask; and Adversarial Forcing.

Adversarial Forcing. The student generates each block in a few denoising steps from its own preceding outputs and is matched to the frozen bidirectional teacher by distribution matching distillation, with an auxiliary fake-score model. Because standard Self Forcing detaches the KV cache, later losses never supervise the history-prefill computation, so the authors use a no-gradient rollout to record each block's input and clean output, then a differentiable replay that recomputes history and predictions with the teacher-forcing visibility pattern. Unlike Self Gradient Forcing, which does the second pass with full-sequence FlexAttention, this replay runs attention block by block with scaled dot-product attention using the same key–value order and call shapes. On top of distribution matching, a trainable head aggregates features from the frozen teacher backbone into a discriminator logit with a relativistic objective and R1/R2 penalties computed exactly: with the backbone frozen, one vector–Jacobian product and one Jacobian–vector product give the penalty and its exact head-parameter gradient without materializing the Jacobian or differentiating through a backward kernel.

Serving. Inference keeps a bounded KV cache with a permanent sink prefix (including the reference) and a sliding window of recent history; for rollouts beyond the training horizon it applies top-aligned rotary position remapping. Custom Triton kernels fuse elementwise operations around attention, CUDA graph capture removes launch overhead, and the tiny decoder replaces the standard VAE decoder.

Agent practice loop. A round begins with a task file stating the goal, available actions, and limits but no solution. Agents play in parallel within a fixed budget, seeing only rendered camera frames, then each writes a playbook recording what it tried, observed, doubted, and plans to test. Playbooks are archived and the next round's agents start in fresh conversations from the task file and every earlier playbook. Each role keeps its own playbook organized as a library of short skill files.

Why This Matters

The work targets a bottleneck in embodied learning: environments must be both faithful (consistent state, rules, dynamics) and realistic (observations following real-world visual distributions), and achieving both across diverse worlds has been difficult. By making state and rules explicit and editable while delegating appearance to a shared renderer, the framework allows environments to scale in number and difficulty alongside their agents, and allows recorded rollouts to be re-shot in a different visual style or from a different camera without changing what happened.

Real-world applications suggested by the environments and deployment described:

• Interactive browser games and simulations. The interactive deployment adds browser games whose rules live entirely in code, including bowling, a penalty shootout, and a crate-vault puzzle, where keyboard, mouse, or gamepad input advances the code world and the rendered stream is the only view of it, streamed over WebRTC.
• Robotics and manipulation. The environment library spans navigation, manipulation, tool use, and multi-agent interaction, with benchmarks such as Habitat 2.0, ManiSkill2, and ProcTHOR cited as prior executable-environment substrates.
• Automated content and scene authoring. A coding agent can build a scene program from a single image or a text description, and refine it by inspecting probe-camera rollouts and engine queries on contact, collision clearance, and occlusion.
• Agent training and evaluation infrastructure. Agents can practice in many distinct worlds (companion dog, one-lane bridge, herding, quarry loader, hide-and-seek) using the same renderer and the same task-file-and-playbook procedure.

Industry relevance: The renderer runs at over 35 frames per second on a single NVIDIA H100 with a distilled tiny decoder, custom Triton kernels, and CUDA graph capture, which matters for serving interactive environments rather than offline video generation. The ability to reuse one renderer across compatible engines reduces the per-scene asset authoring cost that currently limits environment scaling.

Future Directions

• Extending the environment library and difficulty. The framework is designed so new worlds can be written as code and rendered through the same interface; scaling environments in number and difficulty alongside agents is framed as a step toward agents that keep evolving through interactive experience.
• Beyond the settings studied. The reported agent study covers hide-and-seek plus four further worlds (companion dog, one-lane bridge, herding, quarry loader), each run for four rounds; the paper does not report results beyond these.
• Strengthening long-horizon rendering. Adversarial Forcing addresses visual degradation over long rollouts that score distillation alone exhibits; continued work on consistency over extended rollouts is implied by the 30-second comparison and the streaming cache design.
• Broader replay and memory trade-offs. Exact replay eliminates the 3.99% relative L2 discrepancy of full-sequence FlexAttention replay at a 0.2% memory change and 5.4% forward-time reduction in the reported setting; how this scales to other configurations is not reported.

Target Audience

Researchers and engineers working on embodied agents, world models, interactive simulation, and real-time video generation. It will be most useful to readers already comfortable with diffusion and flow-matching video models, distillation (distribution matching, Self Forcing, DMD2), adversarial training with R1/R2 regularization, and attention-kernel-level optimization, as well as to practitioners building agent training environments who care about separating faithful state from realistic appearance.

Authors’ abstract

Interactive virtual worlds allow agents to learn through exploration and interaction. What agents can learn is bounded by the environments they practice in, which must be faithful, with consistent state, rules, and dynamics, and realistic, with observations that follow the real-world visual distributions. Achieving both across diverse worlds remains a bottleneck. We introduce AgentGarten, a framework that couples simulators and game engines with a shared neural renderer to build real-time interactive environments. Its simulation backends maintain persistent world state and execute program-defined interaction rules, while the renderer generates visual observations from structured conditions exported through a common interface. To build the neural renderer, we adapt a pretrained video model to geometry conditions, distill it with our proposed Adversarial Forcing, and optimize inference for real-time interaction. Adversarial Forcing makes history prefilling differentiable through exact replay, so that losses on later predictions update how the renderer encodes prior observations, and adds real-data adversarial supervision to improve its visual quality. In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine. Our empirical study demonstrates a substantial gain in learning efficiency, with agents learning from just 4 rounds compared with millions for a conventional reinforcement learning counterpart. As new worlds can be written as code and rendered through the same interface, environments can scale in both number and difficulty alongside their agents, a step toward agents that keep evolving through interactive experience.