Neuro-Symbolic Synergy for World Modeling

AI summary

Overview

• Research area: Neuro-symbolic AI for world modeling in LLM-based sequential decision-making agents, spanning natural language processing, symbolic reasoning, and reinforcement-learning-style environment interaction.
• Technical level: Advanced. The paper assumes familiarity with partially observable Markov decision processes, LLM fine-tuning, candidate reranking, and symbolic rule systems.
• Scope: The paper proposes Neuro-Symbolic Synergy (NeSyS), a framework that combines an LLM world model with a weighted set of executable Python rules through candidate-level log-linear reranking, and evaluates it on ScienceWorld, WebShop, and PlanCraft.

What This Paper Is About

LLM-based world models are good at general reasoning but hallucinate and break deterministic transition rules, especially in rare corner cases, while purely symbolic world models obey rules but cannot handle rich natural-language semantics. NeSyS aims to get both properties at once: the LLM supplies semantic priors and the symbolic rule set supplies hard, executable judgments, with their scores merged directly rather than by asking the LLM to read rule text. The framework also trains the two components in alternation so that each one focuses on the cases the other explains poorly.

Key Contributions

1. NeSyS framework. A method that integrates an LLM world model with executable rules at the score level via log-linear reranking, so rules can impose strong penalties without depending on the model's instruction-following ability.
2. Reciprocal refinement training paradigm. A two-phase pipeline in which neural training prioritizes transitions weakly covered by symbolic rules, and rule induction targets the neural model's residual errors.
3. Rule-guided data selection. A sampling scheme that retains all steps with zero active rules and samples covered steps with probability proportional to 1/k, where k is the number of active rules on the gold candidate.
4. Empirical demonstration across environments and scales. Consistent gains in world-model prediction accuracy and data efficiency across ScienceWorld, WebShop, and PlanCraft with Llama-3.2-1B-Instruct and Qwen3-4B backbones, plus an open-ended WebShop lookahead study with agent reward improvement. Models, rules, and code are released at https://github.com/tianyi-lab/NeSyS.

Main Findings

• Fewer training pairs beat full-data SFT. Rule-guided sampling uses 35–60% of the training pairs while outperforming full-data supervised fine-tuning in five of six settings. On ScienceWorld, Llama-1B NeSyS reaches 68.3% with 45% of the data versus 64.4% for SFT on 100% of the data, and Qwen-4B NeSyS reaches 71.0% versus SFT's 68.3%.
• Reciprocal refinement improves both world models. Both the neural and symbolic world models improve from Phase 1 to Phase 2, and the combined NeSyS surpasses each component in isolation after refinement. WebShop Phase 2 with 60% of the data yields 92.2% (Llama) and 92.6% (Qwen); PlanCraft Phase 2 with 35% of the data yields 87.7% (Llama) and 88.4% (Qwen).
• Score-level integration beats context injection. WALL-E-1.0 style rule injection in Phase 1 gives only marginal gains over the standalone neural world model (Llama: 38.3% vs. 32.3%; Qwen: 50.8% vs. 46.9%), while NeSyS reaches 48.8% and 58.6%. After Phase 2 fine-tuning, WALL-E improves to 59.5% and 68.1% but still trails NeSyS by 8.8 and 2.9 percentage points. A WALL-E-2.0-style executable-feedback baseline averages only 49.0/56.5 across the three environments.
• Symbolic rules dominate on structural tasks. On WebShop, aside from GPT-5-mini, all neural models score zero on "Search" and perform random guessing on "Decision", whereas the Symbolic WM achieves 100% and over 66% respectively even in Phase 1.
• Synergy tolerates a weak neural component. In Llama WebShop Phase 1, NeSyS (80.9%) slightly underperforms Symbolic WM alone (83.4%) because the initialized neural model is very weak (22.2%), but once the neural model reaches 45.9% in Phase 2, the combined framework (92.2%) overtakes the symbolic model (91.5%).
• Explicit rules preserve transition logic that fine-tuning can lose. For Llama on PlanCraft, SFT drops Smelt accuracy from 54.7% to 31.3%, while NeSyS achieves 98.4% on Smelt for Llama and 92.2% for Qwen.
• Medium-difficulty tasks benefit most. PlanCraft "Move-Medium" requires both recipe knowledge and planning; NeSyS reaches 96.6% for Qwen, ahead of Neural WM (93.1%) and Symbolic WM (86.8%).
• Phase 2 exceeds GPT-5-mini in all six settings. Phase 1 can surpass GPT-5-mini without neural training in some settings (both backbones on PlanCraft) but not universally. The comparison is described as conservative because GPT-5-mini answers each test input directly, while NeSyS uses it only for offline rule induction and then freezes the rules.
• Adaptive scaling matters. A fixed γ = 1 averages 70.1 (Llama-1B) and 73.9 (Qwen-4B) across environments, versus 82.7 and 83.7 for the candidate-adaptive gap-based scale. Linear interpolation is comparable (82.4/84.5), confirming the algebraic combiner is not the source of novelty.
• Rule generation does not strictly require the strongest teacher. With GPT-5-nano rules, NeSyS still averages 78.8 (Llama-1B) and 79.3 (Qwen-4B), above full SFT in four of six settings. Using Qwen3-4B to generate rules for its own world model reaches 87.1% on PlanCraft and 80.3% on WebShop.
• Open-ended WebShop lookahead improves reward. Rule-guided one-step lookahead raises average reward from 0.2755 to 0.3276 over 100 sessions with a 15-step cap; a same-protocol no-rule lookahead control did not improve over the base agent.
• Low-data point on PlanCraft. With only 100 PlanCraft pairs drawn from the stricter rule-uncovered pool of 1,349 candidate rows, Llama-1B NeSyS reaches 86.1% test accuracy, close to 87.7% from the full pipeline using 35% of the 6,083 pairs. The paper explicitly limits this to PlanCraft rather than a general few-shot claim.
• Cost profile. The six final GPT-5-mini rule files contain 216 rules and 729,103 bytes of Python code (3,375 bytes per rule on average), adding no neural parameters. One PlanCraft run with Qwen3-4B made 150 GPT-5-mini calls at about 2,700 tokens per call and cost less than USD 1; on four 48GB A6000 GPUs, full SFT took 2.3 wall-clock hours versus 0.8 hours for NeSyS's reduced-data neural training stage.
• Weight calibration. Scalar rule weights are tuned by coordinate descent on the development set; the gain over uniform weights averages 2.6 points with a range of 0.3–4.8.

Methodology in Plain English

The framework keeps two world models side by side. The neural world model is a pretrained LLM that assigns a likelihood to each candidate next state and reward. The symbolic world model is a weighted set of Python functions that each read the belief state, action, and a candidate outcome and return a real-valued score — typically zero when the rule is irrelevant, positive when the candidate looks right, negative when it violates a rule. Rule scores are combined into a single scalar, and that scalar multiplies the LLM's probability through a log-linear reranking step, p̃ᵢ = pᵢ exp(γ Eᵢ), where γ adapts to the spread of the candidate log-likelihoods. The candidate with the largest modified score wins. Because the rules act on candidate scores directly, the LLM never has to read or interpret rule text.

Training happens in two phases. In the initialization phase, the pretrained LLM is evaluated on a development set, its errors are clustered by similarity, and gpt-5-mini is prompted to write a Python rule for each cluster; a rule is kept only if it improves development accuracy, and failed rules get up to 3 reflection iterations. In the reciprocal refinement phase, training trajectories are scored by the current rules: steps with zero active rules are all kept, and covered steps are sampled with probability proportional to 1/k, so harder-to-cover examples appear more often. The neural model is then fine-tuned on this subset. Rules that now hurt performance are discarded, new rules are induced for new residual errors, and weights are recalibrated. All rules and weights are frozen before a single run on the unseen test set.

Why This Matters

• Impact on research. The paper separates two ideas that are often conflated: giving an LLM rules as text versus applying rules as numerical scores. It shows that the second is markedly stronger for small or fine-tuned backbones (1B and 4B in this study), and it reframes data selection as a complementarity problem rather than a quality-filtering problem. It also provides a concrete signal that much of a fine-tuning set can be redundant once deterministic dynamics are extracted into code.
• Real-world applications.
  • Web and e-commerce agents that must follow rigid navigation and page-transition protocols while still understanding product text (demonstrated on WebShop).
  • Game and crafting assistants where recipes and inventory rules are exact but goals require flexible planning (demonstrated on PlanCraft).
  • Interactive scientific or educational simulations where physical reasoning is required alongside a fixed set of environment rules (demonstrated on ScienceWorld).
  • Agent controllers that use a learned world model for one-step lookahead advice, as shown with a WebShop-7B-SFT agent.
• Industry relevance. The results suggest that practitioner teams can shrink fine-tuning datasets and add auditable, versionable Python rules instead of a larger neural training budget, with rules that can be inspected, edited, and frozen before deployment. The reported cost profile — negligible neural parameter overhead, offline rule induction under USD 1 for one run, and roughly a third of the fine-tuning wall-clock time — is relevant to teams constrained by GPU hours or by a need for deterministic behavior on corner cases.

Future Directions

• Extending evaluation beyond one-step lookahead on one environment to longer-horizon closed-loop planning.
• Adding a second held-out split for rule selection, since the current protocol selects and calibrates rules on a single development set and may overfit it.
• Reducing dependence on the rule-generator model: the paper notes sensitivity to generator capability, with same-backbone Qwen3-4B generation accepted only 6 rules and reaching 80.3% on WebShop, and the same-backbone control was not run on ScienceWorld or with Llama3.2-1B.
• Generalizing beyond environment-specific rules, since the induced rules are intentionally environment-specific (though they transfer across tasks within an environment), and extending the low-data result beyond the single PlanCraft setting where it was measured.

Target Audience

Researchers and engineers working on LLM agents, world models, and neuro-symbolic systems, particularly those dealing with environments that have deterministic transition rules and long-tail corner cases. It is also relevant to practitioners who fine-tune small or mid-sized models for sequential decision-making and want to reduce training data or add auditable rule-based guarantees. Readers need a working familiarity with LLM likelihoods, fine-tuning, and symbolic rule systems to follow the method sections.

Authors’ abstract

Large language models (LLMs) exhibit strong general-purpose reasoning capabilities, yet they frequently hallucinate when used as world models (WMs), where strict compliance with deterministic transition rules--particularly in corner cases--is essential. In contrast, Symbolic WMs provide logical consistency but lack semantic expressivity. To bridge this gap, we propose Neuro-Symbolic Synergy (NeSyS), a framework that integrates the probabilistic semantic priors of LLMs with executable symbolic rules to achieve both expressivity and robustness. NeSyS alternates training between the two models using trajectories inadequately explained by the other. Unlike rule-based prompting, the symbolic WM contributes candidate-level scores through log-linear reranking, without requiring the LLM to interpret rule text. Rule-guided sampling prioritizes transitions that are weakly covered by symbolic rules, using 35--60% of the training pairs while outperforming full-data supervised fine-tuning in five of six settings. Experiments on ScienceWorld, WebShop, and PlanCraft demonstrate consistent gains in WM prediction accuracy and data efficiency; one-step lookahead on open-ended WebShop also improves agent reward. Our models, rules, and code are available at https://github.com/tianyi-lab/NeSyS.