ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs

AI summary

Overview

Research area: Machine learning / large language model agents for software engineering, specifically reinforcement learning (RL) for coding agents and learned "world models" that predict action outcomes.

Technical level: Intermediate. The paper formalizes the agent as a partially-observable Markov decision process (POMDP) and includes formal propositions, but its central idea — have the agent state a prediction before acting, then check that prediction against real execution — is intuitive and accessible.

Scope (one sentence): The paper introduces ANTE, an agentic RL framework in which a coding agent writes an explicit "belief state" (a state summary, a forecast of what its next command will produce, and a trust/abstain decision) before every tool call, has that forecast verified by real execution, and converts the verified forecasts into per-step training credit and test-time action selection.

What This Paper Is About

Coding agents inside a container observe tool results and pick their next action, but most existing agents never look ahead: they do not anticipate what an edit will do, and the final task reward barely attributes a failure back to the step that caused it. This means an ill-judged edit can corrupt the repository irreversibly before any learning signal arrives. ANTE's goal is to train a code world model that knows what has happened, what holds now, and what the next action will achieve — and to make that foresight improve the agent's actual actions, both during RL and at inference time.

Key Contributions

1. Verified forecasting during RL. To the authors' knowledge, ANTE is the first agentic RL framework in the code domain that checks the policy's explicit pre-action forecasts against real execution throughout training, turning those verified forecasts into step-level supervision for an internal world model that lives inside the policy itself.
2. Belief-Anchored Policy Optimization (BAPO). A credit-assignment method that groups steps across rollouts of the same task by their predicted outcome, then compares their execution feedback — so individual steps receive credit beyond the single terminal task reward.
3. ANTE-TTO (test-time optimization). At inference, when the policy abstains or predicts failure, ANTE-TTO samples alternative actions, ranks candidates using the policy's own forecasts (no external scorer), and executes only the highest-ranked one. At 7B scale it improves task success with approximately 31% additional inference tokens over a single greedy attempt.
4. Empirical validation across harnesses and domains. ANTE-32B reaches 57.2% on SWE-bench Verified, described as outperforming all compared open-weight agents above 30B; ANTE-9B reaches 24.7% on Terminal-Bench 2.0; and across eleven out-of-distribution benchmarks ANTE-32B improves average performance by 6.2 percentage points over a matched RL baseline trained without the belief state.

Main Findings

• Large gains on SWE-bench Verified. ANTE raises Qwen3-32B from 25.0% to 57.2% pass@1. Relative to a matched GRPO baseline trained without the belief state, the abstract reports a 6.2% improvement; the main table reports ANTE leading the Reactive baseline by 2.8, 4.4, and 6.2 points at 7B, 14B, and 32B respectively.
• Consistent lead over both controlled baselines. On SWE-bench Verified, ANTE leads Reactive by 2.8/4.4/6.2 points and Retrospective by 2.0/2.8/4.8 points at 7B/14B/32B; on terminal tasks it leads Reactive by 6.7 to 10.0 points and Retrospective by 2.0 to 5.5 points across TB-Dev, TB-Lite and Terminal-Bench 2.0.
• The world model's accuracy rises during RL. The paper reports the world model's prediction accuracy climbing from 44% to 69% over the course of RL, with resolve rate and prediction hit rate rising together.
• BAPO supplies a substantial share of the gain. Training with BAPO adds 1.8, 2.6, and 3.0 points on top of the belief state at 7B, 14B, and 32B (about half or more of ANTE's lead over Reactive), and 1.4 to 6.0 points on the terminal tasks.
• The belief state alone already helps. Without BAPO, the belief state improves over Reactive by 1.0 to 3.2 points on SWE-bench Verified and 4.0 to 5.5 points on the terminal tasks, matching or exceeding Retrospective except on TB-Lite.
• Terminal-harness results. ANTE scores 42.5 on TB-Dev, 45 on TB-Lite, and 24.7 on Terminal-Bench 2.0 with a Qwen3.5-9B backbone, versus the base model's 32.9, 32, and 16.9.
• Foresight transfers out of distribution. On eleven benchmarks spanning execution prediction (CruxEval-O, CruxEval-I, LCB-exec), state tracking (BBH-hard, BBH-state), mathematics (MATH500, AIME'25, AMC23, Minerva) and science (MMLU-Pro, SciBench), ANTE-32B leads Reactive-RL on every dataset, with a mean of 83.3 versus 77.1 (+6.2). The largest single gain is on AIME'25 (63.3 to 80.0, +16.7).
• Test-time scaling is cheap. ANTE-TTO adds 1.4/2.4/0.6 points on SWE-bench Verified at 7B/14B/32B. Measured from call counts, it runs at 1.31× the inference tokens of a single greedy attempt at 7B versus 4.42× for TTS-4, and 1.6× versus 4.1× averaged over its four settings.
• Competitive with published agents. Above 30B on SWE-bench Verified the paper reports ANTE ahead of SWE-World, FrogBoss and CWM, and of every 72B method compared; on Terminal-Bench 2.0, ANTE-9B leads every compared terminal agent below 20B, including ECHO.

Methodology in Plain English

The belief state. Before each tool call, the agent writes three things: a one-sentence summary of what is established and what is broken; a forecast of the outcome its next command will produce (the result of the check it runs — pass, fail, or not applicable — plus whether the command exits cleanly); and a decision to either trust the forecast or abstain from it. Abstaining still writes a forecast and still takes the action, it just does not commit. The decision exists because rewarding prediction accuracy alone would encourage guessing when the repository state is genuinely underdetermined.

Settling forecasts by execution. A deterministic action-type map classifies each step as "epistemic" (it runs a check the verifier can evaluate, like a target test or reproduction script) or "pragmatic." After execution a verifier reads the real outcome. A step is verifiable when its belief block parses and the verifier returns a value, and conclusive when the check field is specifically pass or fail. On repository issues these conclusive steps are the ones that carry credit.

Stage I: supervised fine-tuning on BeliefTraj. The policy is first trained by maximum likelihood on a corpus of solving trajectories collected by a capable agent in the execution environment, where each step is annotated with a belief state whose forecast is the outcome the verifier read from the logged next observation. The paper proves a bound (Proposition 1) showing that minimizing this loss brings the fine-tuned policy close to the demonstration success rate, up to a remainder term from hindsight-written labels — a mismatch Stage II is designed to address.

Stage II: RL with BAPO. The policy generates N rollouts per training task. Three signals are combined: (i) an execution-grounded prediction loss, a cross-entropy on the forecast tokens against the realized outcome of every verifiable step; (ii) a decision reward of +α for a trusted forecast that is correct in both fields with a clean exit, −γ for an incorrect trusted forecast, and a fixed cost −κ for abstaining, with γ set to αq*/(1−q*) so that trusting pays off only above a target accuracy q*; and (iii) the BAPO step credit. Because an individual step needs something to be compared against, BAPO uses the policy's own predicted outcome as a recurring grouping key: steps predicting the same outcome across rollouts of the same task are pooled into a "belief group," and further split by trust/abstain into "belief-decision groups." The decision return of a step is compared against the mean return of its belief group and of its belief-decision group, producing belief-relative and decision-relative advantages that are combined with weight ω = 1/2, standardized within the task, clipped, and added to the episode-level advantage. Pragmatic steps receive only the episode-level term. The summary tokens receive no gradient, since nothing checks a summary.

ANTE-TTO at inference. When the policy abstains or predicts failure, it samples K = 3 alternative messages at temperature 1.0, scores all candidates (including the original) by their own beliefs using φ = 3, 1, 0 for pass, na, and fail plus a bonus for a predicted clean exit, and executes only the highest-ranked one, keeping the original on ties. No weights change and no external scorer or extra executions are needed — a regret bound (Proposition 2) ties selection quality to how divergent the forecasts are from the executed outcome laws.

Setup. Training and evaluation use two harnesses: repository issues under R2E-Gym at 7B, 14B and 32B (Qwen2.5-Coder-Instruct backbones for 7B/14B, Qwen3-32B for 32B), and terminal tasks at 9B with the Tmax agent on Qwen3.5-9B. SFT uses 12,500 verified successful trajectories over 6,729 repository tasks and 30,161 trajectories over 10,823 terminal tasks from the tmax-15k pool; RL uses matched pools of 6,566 repository tasks and 3,558 terminal tasks. Baselines are a Reactive agent (thinking plus tool calls) and a Retrospective agent (Reactive plus a memory summary), each trained from its own SFT checkpoint with the belief components removed, so every matched pair isolates the belief.

Why This Matters

Impact on research. The paper closes a specific gap: existing code world models are either trained and then frozen before RL, or reduced to an auxiliary next-token loss on environment outputs; and existing belief-rewarding methods rely on language-model judges or embedding similarity, which only approximate correctness and are prone to reward hacking. ANTE replaces those proxies with ground truth from real execution and shows the resulting forecasts can be converted into per-step policy credit — a template that extends beyond code.

Real-world applications:

• Autonomous issue resolution: agents that fix a reported bug without silently breaking previously passing tests, which is exactly the failure mode the paper illustrates for reactive and memory-only agents.
• CI and regression safety: forecasts of which checks a command will trigger let an agent avoid edits with destructive side effects before they are committed to a repository.
• Terminal and DevOps automation: the same belief schema and advantage are applied unchanged to terminal tasks, suggesting portability to operational workflows.
• Cost-aware deployment: ANTE-TTO reaches better task success with far fewer extra inference tokens than best-of-n sampling (1.31× versus 4.42× at 7B), which matters for production budgets.

Industry relevance. All authors are affiliated with AWS AI Labs (the first author's work was done during an internship there), and the method is evaluated on two distinct production-style harnesses with open-weight models from 7B to 32B. The result that foresight transfers to out-of-distribution reasoning benchmarks (+6.2 points on average) is directly relevant to teams that want one training investment to improve general agent reliability, not just one benchmark.

Future Directions

• Pushing world-model accuracy further. Prediction accuracy reaches 69% by the end of RL; how much of the remaining error is reducible, and whether more or better-verified training signal closes it, is left open.
• Broadening beyond code and terminal harnesses. The paper states the belief schema and advantage are the same across both harnesses while the outcome space, anchor key and decision reward are set per harness — a natural question is how much per-harness engineering is needed for a new domain.
• Cheaper or more aggressive test-time foresight. ANTE-TTO already costs far less than best-of-n, but the trade-off between K, trigger frequency, and gain (for example the modest +0.6 points at 32B) leaves room for better scheduling of when to resample.
• Tighter theory and broader credit assignment. The paper bounds selection regret and imitation error, but the belief-group construction depends on predicted outcomes recurring across rollouts; how BAPO behaves when groups are sparse or degenerate (the paper falls back to the task mean) is a question worth pursuing.

Target Audience

Researchers and engineers working on LLM agents, agentic reinforcement learning, and code generation who want agents that reason about consequences rather than only reacting to observations. It is also useful for practitioners deploying coding or terminal agents in production who care about regression safety and inference cost, and for readers interested in world models and credit assignment in sparse-reward, long-horizon settings. A reader with basic familiarity with policy-gradient RL and LLM fine-tuning will get the most from the formal sections; the core intuition is accessible without them.

Authors’ abstract

Rapid advances in large language models have made software engineering agents capable of tackling complex coding tasks. Within a containerized environment, a coding agent receives tool responses or execution results, and determines its later actions by those earlier results. However, most existing coding agents never look ahead: their policies do not anticipate the consequences of actions, and the terminal reward barely attributes a failure to certain earlier steps, so an ill-judged edit can corrupt the repository state irreversibly before any signal arrives. Recent code world models aim to address these challenges, but they are either trained and then frozen before reinforcement learning (RL), or reduced to an auxiliary next-token loss on the environment's outputs. We present ANTE (Anticipate, Then Execute), an agentic RL framework that verifies the policy's pre-action predictions by real execution and turns them into per-step credit for its actions and trust-or-abstain decisions. With an internal belief stated before each action, Belief-Anchored Policy Optimization (BAPO) turns the settled predictions into per-step advantages by anchoring credit on the policy's own forecasts. To turn the trained forecasts into better actions at inference, thereby facilitating effective test-time scaling, ANTE-TTO (test-time optimization) selects among candidate actions without an external scorer. ANTE raises Qwen3-32B from 25.0% to 57.2% pass@1 on SWE-bench Verified, 6.2% above a matched GRPO baseline trained without the belief state; its world model's accuracy climbs from 44% to 69% during RL, and the learned foresight capability transfers across harnesses and out-of-distribution domains. In summary, we aim to build a code world model that knows what has happened, what holds now, and what the next action will achieve.