One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Exploration

AI summary

Overview

Research area: Artificial Intelligence / symbolic world modeling, program synthesis, probabilistic programming, and reinforcement learning environments.

Technical level: Intermediate. The paper is readable without deep mathematical background, but familiarity with world models, transition functions, and probabilistic inference helps.

Scope: The paper introduces OneLife, a framework that infers an executable, probabilistic symbolic world model of a complex, stochastic environment from a single unguided episode of exploration, evaluated on Crafter-OO with new state ranking and state fidelity metrics.

What This Paper Is About

An agent dropped into a complex, dangerous, randomly behaving world needs to figure out how that world works, and it usually has no map, no rewards, and no second chances. Prior symbolic world modeling work gets around this by assuming abundant interaction data, mostly deterministic mechanics, and human-provided goals or rewards. This paper asks how an agent can reverse-engineer the laws of a stochastic, hostile world given only a limited interaction budget, no human-specified goals, and no rewards.

Key Contributions

1. OneLife, a probabilistic symbolic world model that represents environment dynamics as a mixture of modular programmatic laws, each with a precondition and an effect, and learns from stochastic, hostile environments with minimal interaction and no human-defined rewards. It outperforms prior work in predicting true environment dynamics.
2. Crafter-OO, a reimplementation of Crafter (Hafner, 2022) that exposes a structured, object-oriented symbolic state and a pure transition function operating on that state alone, providing a testbed more complex than gridworlds and with more diverse mechanics than Atari.
3. An evaluation suite for world modeling within Crafter / Crafter-OO with 30+ executable scenarios covering all core mechanics and a pool of mutators that programmatically generate illegal distractor states, plus new state fidelity and state ranking metrics for complex, stochastic environments.
4. A demonstration that the inferred world model is useful for planning, where rollouts simulated entirely within the model identify superior strategies in multi-step goal-oriented tasks.

Main Findings

• OneLife beats the strongest baseline on most scenarios: OneLife outperforms the baseline on 16 out of 23 scenarios tested (as reported in the abstract and Figure 1), measured by Mean Reciprocal Rank of the true next state under the world model's likelihood.
• Discriminative accuracy improves: OneLife achieves Rank@1 of 18.7% and MRR of 0.479, versus PoE-World's 10.8% and 0.351 — a gain of 7.9 percentage points and 0.128 respectively. Comparisons are averaged over ten trials.
• Generative fidelity is competitive: OneLife's raw edit distance is 8.764 and normalized edit distance is 0.058, versus PoE-World's 10.634 and 0.071.
• Learnable law parameters matter: Ablating the learnable law weights ("No Inference") gives Rank@1 of 13.0%, MRR of 0.429, raw edit distance 8.540, and normalized edit distance 0.057 — better fidelity than full OneLife but worse ranking.
• Deterministic monolithic synthesis fails here: WorldCoder obtains Rank@1 of 0.0%, MRR of 0.264, raw edit distance 27.180, and normalized edit distance 0.181, consistent with its assumption of determinism being violated by Crafter-OO's stochastic dynamics.
• Random baseline reference points: The random world model scores 8.5% Rank@1, 0.322 MRR, 121.538 raw edit distance, and 0.809 normalized edit distance.
• Ranking and fidelity measure different things: The paper argues that optimizing generative fidelity alone does not yield a better world model for guiding an agent — PoE-World reduces edit distance by roughly a factor of 10 relative to random while improving Rank@1 only about 2% over random.
• The model captures stochastic NPC behavior without supervision: For a zombie that chases the player with stochastic movement, the learned model predicts a discrete distribution over zombie.position attributes without explicit supervision.
• Planning in imagination works: Simulated rollouts within the learned world model successfully distinguish effective from ineffective strategies in multi-step goal-oriented tasks. Specific numeric planning results are not reported in the provided content.

Methodology in Plain English

The environment is treated as a pure but potentially stochastic transition function that maps a state and action to a distribution over next states. Instead of writing one monolithic program, OneLife represents the world as a mixture of small programs called laws. Each law has a precondition deciding when it applies and an effect describing a probability distribution over the attributes it governs. A law stays silent on attributes it does not model, so only the relevant laws contribute to any given prediction.

An observable extractor maps the complex hierarchical state into a vector of primitive-valued observables, letting the system compare predicted and true changes tractably. For each observable, the active laws that have an opinion on it are combined by a weighted product, with learnable weights serving as model selection over candidate laws and allowing multiple plausible laws to vote.

Exploration is unguided: an LLM-driven policy (built using the Balrog agent scaffolding) is given only general genre priors about open-world survival games — hostile entities, resource collection, crafting — and is deliberately withheld from environment-specific rules such as "zombies chase players." It maintains a rolling window of recent state-action history and a transient summary of its hypotheses. A single LLM-based law synthesizer then iterates through every transition, compares the object-oriented state across timesteps to find changed attributes, and asks the LLM to output a Python class with precondition and effect methods for each change, yielding atomic laws.

Inference maximizes log-likelihood over the collected transitions. Because observables are treated as conditionally independent given the state and action, the loss decomposes into a sum per observable, and gradients are routed only to laws active for that observable — a dynamic computation graph that gives precise credit assignment. Optimization uses L-BFGS. To generate future states, the model forms a predictive distribution per observable, samples a value, and reconstructs a full symbolic state.

Evaluation uses two axes. State ranking checks whether the model ranks the true next state above distractors generated by mutators that make rule-breaking changes. It reports Rank@1 and Mean Reciprocal Rank. State fidelity measures raw edit distance (the number of atomic JSON Patch operations needed to turn the predicted state into the true state) and normalized edit distance (raw edit distance divided by the total number of elements in the state representation). Evaluation trajectories come from scripted scenarios covering every achievement in the game's achievement tree, and candidate set sizes range from N=7 to N=11 per transition.

Why This Matters

Impact on research: The paper shifts symbolic world modeling from deterministic, data-rich, human-guided settings toward stochastic, hostile, unguided ones, and argues that state ranking metrics — not just generative accuracy — are the right way to evaluate whether a world model can guide an agent. Crafter-OO fills a gap the authors identify: the lack of an environment with exposed object-oriented state that is more complex than gridworlds or more mechanically diverse than Atari.

Real-world applications:

• Autonomous agents that must survive and learn in unfamiliar, hazardous settings where mistakes are costly and a single episode is all that is available.
• Scientific discovery and system identification, where an agent reverse-engineers the rules of an unknown system from observation alone.
• Robotics or industrial control, where an interpretable, editable program describing dynamics can be inspected and audited by humans, unlike a neural latent model.
• Game AI and simulation tooling, where inferred symbolic rules can be reused, modified, and verified.

Industry relevance: Because the world model is code rather than a black-box latent state, it is interpretable, editable, and verifiable — properties that matter for deployment and auditing. The approach also targets a hard practical constraint: learning useful dynamics under a tight interaction budget rather than from massive offline datasets.

Future Directions

• Extending the framework beyond the currently implemented categorical and discrete distributions to continuous distributions, which the authors note the inference algorithm supports in principle since it only requires querying the likelihood of an observed data point.
• Investigating why better generative fidelity did not translate into better ranking (as seen in the "No Inference" ablation and the PoE-World comparison), and how to optimize both axes jointly.
• Scaling symbolic world modeling to environments with even more diverse mechanics, hierarchical state, and interacting stochastic agents beyond Crafter-OO.
• Using the inferred world model for downstream decision-making more broadly — the paper demonstrates planning via simulated rollouts, but the full extent of planning quality and its numeric results are not detailed in the provided content.

Target Audience

Researchers and practitioners working on world models, model-based reinforcement learning, program synthesis, neurosymbolic AI, and open-ended exploration. It is also relevant to engineers building interpretable or verifiable agent components and to anyone evaluating how well a learned model can rank plausible futures rather than merely reproduce them.

Authors’ abstract

Symbolic world modeling requires inferring and representing an environment's transitional dynamics as an executable program. Prior work has focused on largely deterministic environments with abundant interaction data, simple mechanics, and human guidance. We address a more realistic and challenging setting, learning in a complex, stochastic environment where the agent has only "one life" to explore a hostile environment without human guidance. We introduce OneLife, a framework that models world dynamics through conditionally-activated programmatic laws within a probabilistic programming framework. Each law operates through a precondition-effect structure, activating in relevant world states. This creates a dynamic computation graph that routes inference and optimization only through relevant laws, avoiding scaling challenges when all laws contribute to predictions about a complex, hierarchical state, and enabling the learning of stochastic dynamics even with sparse rule activation. To evaluate our approach under these demanding constraints, we introduce a new evaluation protocol that measures (a) state ranking, the ability to distinguish plausible future states from implausible ones, and (b) state fidelity, the ability to generate future states that closely resemble reality. We develop and evaluate our framework on Crafter-OO, our reimplementation of the Crafter environment that exposes a structured, object-oriented symbolic state and a pure transition function that operates on that state alone. OneLife can successfully learn key environment dynamics from minimal, unguided interaction, outperforming a strong baseline on 16 out of 23 scenarios tested. We also test OneLife's planning ability, with simulated rollouts successfully identifying superior strategies. Our work establishes a foundation for autonomously constructing programmatic world models of unknown, complex environments.