The Pulse
MirroS Puts AgentGarten’s Code-Run Worlds to the Test
MirroS introduced AgentGarten on October 8, pairing executable simulation worlds with a neural renderer that gives AI agents visual feedback as they act. Its reported trials span hide-and-seek, sheep herding and vehicle tasks, while the pub
AI.info Team · Written with AI ·
Executable simulations give AI agents explicit rules and persistent state; video-generated worlds can offer richer-looking scenes but do not necessarily expose the underlying objects and mechanics for direct inspection. MirroS says AgentGarten joins the two: code governs what happens, while a neural renderer turns exported geometry into the agent’s next visual observation. The team announced the project on October 8, with experiments in which agents learn tasks by acting, reviewing their attempts and passing written playbooks to later rounds.
The claim that agents can learn useful behavior in a handful of rounds sets up the project’s central comparison: MirroS measures its pretrained visual agents against strategies that emerged in a much longer reinforcement-learning study. But the work is not yet a complete public environment toolkit. The team’s repository says the renderer’s training and inference code is available, while the code worlds and agent practice loop remain to be released.
Code controls the world; the renderer supplies its appearance
AgentGarten uses simulators or game engines to maintain the world state and execute program-defined rules. Those environments export structured conditions—such as depth or surface normals—to a shared neural renderer, which generates the images an agent sees. The agent acts on those images, and the world updates before the next observation.
That split gives the system a different interface from a video model that generates a sequence from conditions supplied in advance. MirroS’s renderer generates observations in short blocks, so the agent can inspect an action’s consequences before choosing what to do next. The team says the same interface supports navigation, object manipulation and multi-agent settings without requiring every world to have its own detailed visual-asset pipeline.
“In AgentGarten, agents perceive the world through visual observations, interact with it in real time, and improve by distilling each round of experience into playbooks that subsequent agents inherit and refine.”
Jiawei Chi et al., “AgentGarten: Code Worlds for Evolving Agents”
Four rounds for a dog; ten for a ramp
MirroS’s headline comparison revisits hide-and-seek. In the team’s version, pretrained agents see first-person rendered frames and issue short Python programs for movement and object interaction. They do not receive object coordinates or hidden world state, according to the project’s account. After each round, agents review their own attempts and add lessons to role-specific playbooks.
The hiders built cover by moving a panel by round four; seekers used a ramp to cross a wall by round ten. MirroS contrasts those results with the 2019 OpenAI hide-and-seek study, which it says required about 25 million training episodes for shelter-building and about 100 million for ramp use. The comparison is not a like-for-like training benchmark: AgentGarten starts with pretrained agents and counts rounds of play and review, while the cited earlier work trains policies through self-play reinforcement learning.
The team also describes four-round trials in four other worlds. In a companion-dog task, an agent had to sustain a willing 60-second interaction; in a one-lane bridge task, two cars had to swap ends; in herding, two dogs had to guide four sheep into a pen; and a wheel loader had to move rocks, deliver one behind a wall and park. MirroS reports that the fourth-round herding attempts penned all four sheep without startling them, while the loader completed more of its sequence than in round one, ending with 31 seconds left.
The renderer is public; the practice loop is not
The project’s technical challenge is to make generated observations both responsive and consistent over longer interactions. MirroS adapts a pretrained video model to geometry conditions, then uses a method it calls Adversarial Forcing to train a block-by-block renderer. The paper says exact replay lets later prediction losses affect how the model encodes its earlier generated history, while real-video adversarial training supplies another signal for visual quality.
The public repository provides renderer training recipes and streaming inference, with instructions calling for Python 3.11 or later, PyTorch 2.9 or later and a CUDA host. It also lists the renderer checkpoints, real-time rendering engine, code worlds and agent practice loop among items still to come. That limits what outside researchers can currently reproduce: they can inspect and run parts of the renderer, but the full play-review-and-playbook experiments depend on components the repository does not yet offer.
MirroS identifies limits in geometry-only observations
The project’s own account outlines a boundary to the approach. Depth and surface normals do not reveal every relevant property: geometry alone, for example, may not show a painted mark rotating on a symmetrical object, and appearance details can be lost after long occlusions. Tasks that depend on those details may need a richer way to pass world state to the renderer.
MirroS says it built the demonstrated worlds one at a time and wants coding agents to create and revise new ones. The team also argues that those generated worlds need checks for whether a task is solvable from the available observations and whether it rewards useful learning. For now, the concrete public release is narrower: renderer code is available, while the worlds and the agent practice loop that produced the headline results are still pending.