Dreaming in Code for Curriculum Learning in Open-Ended Worlds

AI summary

Overview

Research area: Reinforcement learning, unsupervised environment design (UED), automatic curriculum learning, and large language model (LLM) driven program synthesis for environment generation.

Technical level: Advanced. The paper assumes familiarity with partially observable Markov decision processes, regret-based UED objectives, Prioritized Level Replay, and PPO-style policy optimization.

One-sentence scope: The paper introduces Dreaming in Code (DiCode), a framework in which a foundation model writes executable environment code that serves as a stepping-stone curriculum toward mastering a complex open-ended target environment (Craftax).

What This Paper Is About

Agents in open-ended worlds tend to improve quickly and then plateau, because fixed environments eventually stop offering a useful learning signal. Prior UED methods can generate new environments, but they mostly operate over low-dimensional parameter spaces (such as random seeds) and treat environments as isolated challenges rather than as an ordered progression. This paper asks whether an LLM can generate executable environment code — one that changes initial states, game mechanics, and goals — to build a curriculum that carries an agent from basic survival to late-game skills.

Key Contributions

1. The DiCode framework. A UED method in which a foundation model synthesizes executable environment code (initial state distribution, transition dynamics, and goal) that is run by a fixed world engine, so all generated experience obeys valid game mechanics.
2. Demonstrated scaffolding of otherwise unattainable behaviors. DiCode achieves non-zero success on late-game combat tasks where prior methods score 0%, while improving mean return by 17% over the strongest baseline.
3. Qualitative evidence of teacher-like behavior. The foundation model spontaneously removes resource and workstation scaffolding and expands objectives as agent competence rises, holding the average success rate on active training levels near 0.5 (a zone of proximal development).
4. A four-axis ablation. Parent selection, environment reshaping, foundation model capability, and closed-loop grounding are each isolated, showing that environment reshaping is the primary driver and that curriculum quality scales with FM reasoning capability.

Main Findings

• Aggregate performance gain. DiCode reaches a final mean return of 48.55 on the held-out test set, against 41.59 for the strongest baseline — a relative improvement of ~17%. The lead is statistically significant early in training and is maintained across the full 2×10^9 environment step budget.
• Late-game combat becomes solvable. Baselines collapse to 0% on Defeat Gnome Warrior and Defeat Gnome Archer; DiCode achieves 15% and 11% respectively (reported as 14.7% and 11.2% in the ablation comparison).
• Instrumental milestones improve sharply. On Make Iron Armour, DiCode reaches 47% success versus 15% for the best baseline, and it enters the Gnomish Mines (Floor 2) in 32% of episodes versus 9% for the strongest baseline.
• Deep hierarchical dependencies are learned. On Make Diamond Sword, DiCode doubles baseline success (6% versus 3%).
• Curriculum stays in the learnable zone. The average success rate across active training levels remains stable around 0.5 from iteration 15 to 100, indicating difficulty is continuously matched to capability.
• Parent selection matters for depth, not breadth. Random parent sampling drops mean return only modestly (46.44 ± 0.41) but collapses late-game combat to near-zero (Gnome Warrior 0.6%, Gnome Archer 0.1%), showing principled selection is essential for reaching deep objectives.
• Environment reshaping is the main driver. Restricting the model to goal selection alone ("Goal Only") drops performance to 42.82 ± 0.39, near baseline.
• Curriculum quality scales with model capability. Holding prompts, API, and hyperparameters fixed, mean return degrades monotonically as the generator shrinks: Qwen3-235B → 80B → 30B gives 48.55 → 45.88 → 44.38.
• Closed-loop grounding is necessary. Removing the parent level description and the agent's performance profiles ("Open-Loop", 41.12 ± 0.77) performs comparably to the plain PPO-GTrXL baseline (41.59), confirming that generation must be steered toward the agent's learning frontier.

Methodology in Plain English

DiCode keeps a fixed, hand-written game engine — in this case Craftax — and lets a language model write only the code that configures a level: how the world is laid out, what the agent starts with, how game rules such as combat formulas and progression conditions behave, and what counts as the level's goal. Because the real engine executes that code, no generated level can violate the game's physics or mechanics.

Training proceeds in a loop. Levels live in an archive structured as a directed graph, where nodes are levels and edges record parent–offspring relationships. Each level carries the agent's recent success rate, mapped to a status: A at 0.75 or above, B in [0.50, 0.75), C in [0.25, 0.50), and D otherwise. A parent is selected only from candidates in status A or B whose offspring are not all status D, with sampling probability proportional to the learnability score, defined as p(1−p) where p is the success rate. The language model is then given the parent's description and performance profile, the target environment's performance profile, domain context, and mutation instructions; it first writes a natural-language description of the new level, then writes the executable Python for it. A compilation check plus a short agent rollout filters out code that fails to compile or throws runtime errors; the authors deliberately skip self-correction because the compute cost outweighed the gain.

Each training batch mixes three sources: the target environment always gets a fixed 20% of the simulation budget, with the rest split between newly generated levels and archived levels replayed using the PLR prioritization scheme. New levels arrive every v = 2 iterations, and generation runs asynchronously so training only blocks if v−1 cycles pass without a valid batch.

Generated levels use the target environment's reward and termination structure, except that completing a level's goal ends the episode with a bonus B_t = max(d, 2 × R_{t−1}), where R_{t−1} is the agent's mean return on the target environment in the previous cycle and d is a minimum floor. The policy is also conditioned on a multi-hot encoding of the level's active achievements.

The evaluation uses PPO with Gated Transformer-XL as the shared agent across all methods, 8 random seeds for DiCode (5 for ablations), 2×10^9 environment steps, Qwen3-235B as the generator, 50 uniformly spaced checkpoints, and a fixed held-out test set of 1024 unseen procedurally generated Craftax instances. Baselines are PLR, SFL, DR, and a non-curriculum PPO-GTrXL reference.

Why This Matters

Impact on research. The paper reframes UED from parameter search over a fixed, smooth design space to program synthesis over an unbounded one, and it treats the foundation model not as a supervisor or a world model but as an architect of experience. It also supplies an ablation-based account of which ingredient carries the benefit: reshaping environments, not just choosing goals, and keeping the generator inside a closed loop with the agent's measured competence.

Real-world applications.

• Training robot control policies by generating staged simulation tasks in a physics engine such as MuJoCo, where the engine guarantees valid physics.
• Designing training curricula for video game AI and automated playtesting, where level logic and progression rules can be varied programmatically.
• Building adaptive training simulators for sequential decision-making, such as logistics routing under evolving constraints.
• Producing diagnostic evaluation suites that step an agent through prerequisite skills before testing end-to-end competence.

Industry relevance. Adopting the approach requires no proprietary model — the experiments use open-weights Qwen3-235B — and the generator runs asynchronously alongside training, which matters for keeping expensive GPU training loops saturated. The cost is a one-time domain-specific interface layer per environment.

Future Directions

• Escaping the fixed engine. The world engine guarantees valid mechanics but also bounds invention; the model can configure the world, not invent new physical laws. Extending generation to new mechanics is left open.
• Fixing the learnability blind spot. The learnability score inherited from prior UED work is vulnerable to stochastic levels whose success rate hovers near 0.5 indefinitely without real learning, causing persistent oversampling. The authors suggest supplementing it with a learning-progress signal measuring the rate of change in success rate.
• Reducing per-domain engineering. Applying DiCode to a new domain currently requires implementing a domain-specific interface layer; a practitioner guide is provided, but automating that porting remains an open problem.
• Scaling with model capability. The monotonic degradation from Qwen3-235B to 80B to 30B suggests headroom if stronger reasoning models are used, and raises the question of how far curriculum quality can be pushed by model improvements alone.

Target Audience

Researchers and graduate students working on reinforcement learning, unsupervised environment design, and automatic curriculum learning; practitioners who build training environments and want to automate curriculum construction; and engineers applying LLMs to simulation code generation for games, robotics, or embodied agents. Readers need a working background in RL, POMDPs, and priority-based replay to follow the methodology and ablation reasoning.

Authors’ abstract

Open-ended learning frames intelligence as emerging from continual interaction with an ever-expanding space of environments. While recent advances have utilized foundation models to programmatically generate diverse environments, these approaches often focus on discovering isolated behaviors rather than orchestrating sustained progression. In complex open-ended worlds, the large combinatorial space of possible challenges makes it difficult for agents to discover sequences of experiences that remain consistently learnable. To address this, we propose Dreaming in Code (DiCode), an unsupervised environment design (UED) framework in which foundation models (large language models) synthesize executable environment code to scaffold learning toward increasing competence. In DiCode, "dreaming" takes the form of materializing code-level variations of the world. We instantiate DiCode in Craftax, a challenging open-ended reinforcement learning benchmark characterized by rich mechanics and long-horizon progression. Empirically, DiCode enables agents to acquire long-horizon skills, achieving a 17% improvement in mean return over the strongest baseline and non-zero success on late-game combat tasks where prior methods fail. Our results suggest that code-level environment design provides a practical mechanism for curriculum control, enabling the construction of intermediate environments that bridge competence gaps in open-ended worlds.