DeltaTTG: Training Multi-Turn Agents with a Token-to-Go Progress Meter

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents — specifically step-level credit assignment and process supervision for multi-turn agent training under sparse, verifiable outcome rewards.

Technical level: Advanced. The paper assumes familiarity with policy-gradient reinforcement learning, group-relative advantage estimation (GRPO), advantage redistribution, and agentic benchmarks.

Scope: The paper proposes DeltaTTG, a method that trains a reusable "progress meter" to predict how many tokens remain before task completion (Token-to-Go), then uses changes in that prediction to redistribute a trajectory's outcome reward across its individual steps.

What This Paper Is About

Multi-turn agents trained with reinforcement learning from verifiable rewards typically receive only a single success/failure signal at the end of a long sequence of actions, so the training signal says little about which individual steps helped or hurt. Existing fixes either compare actions within the current rollout group — which depends on reliable state matching and diverse sampled outcomes — or score steps directly, which requires costly LLM judges, human annotations, or task-specific heuristics. DeltaTTG instead learns a reusable progress signal from ordinary recorded agent trajectories, using "Token-to-Go" (the number of tokens still to be generated before success) as an observable, low-cost proxy for remaining completion effort.

Key Contributions

1. Token-to-Go as a supervision target. The authors propose TTG — the count of assistant-generated tokens remaining until task completion, extracted for free from recorded trajectories — as an observable proxy for task progress. Labels come from trajectories pooled across models of different families, sizes, and capability levels, with failed continuations assigned a large remaining-work cost.

2. DeltaTTG credit assignment. A progress meter is trained (SFT followed by GRPO) to output a binary completion estimate and a nonnegative remaining-work estimate. During actor training, the frozen meter scores each state, and the change in its estimate between consecutive states determines how the trajectory's outcome-derived advantage is distributed across steps — without adding any new reward or requiring matched peer states.

3. Empirical validation across two actor backbones and three benchmark suites. DeltaTTG is evaluated with Qwen3-4B and Llama-3.1-8B-Instruct on ALFWorld, PaperSearchQA, and BFCLv3, with additional analysis on sample efficiency, out-of-domain generalization, compute cost, and cross-domain meter transfer.

4. Direct validation of the meter itself. The paper assesses whether TTG is a good step-quality signal on AgentProcessBench, whether a fixed meter can guide test-time action selection across actor scales (Qwen3-1.7B, 4B, 8B, 14B), and whether a meter trained in one domain transfers to another.

Main Findings

• Best average performance with both backbones. DeltaTTG reaches an average score of 72.86 with Qwen3-4B versus 70.41 for the strongest baseline (DAPO), and 66.49 with Llama-3.1-8B-Instruct versus 61.69 (GiGPO). Averaged across ALFWorld, PaperSearchQA, and BFCLv3, this corresponds to gains over GRPO of 5.7 and 7.2 points respectively.

• Leading on most individual metrics. DeltaTTG obtains the highest mean (including ties) on 13 of the 16 open-backbone metrics. The largest gains appear on BFCLv3 tool use: +4.8 points on Multi-Turn over the strongest RL baseline with Qwen3-4B, and +7.6 (Non-Live) and +9.0 (Live) points with Llama-3.1-8B-Instruct.

• One consistent weakness. DeltaTTG trails the strongest baseline on the BFCLv3 Hallucination/Irrelevance category with both backbones (76.02 with Qwen3-4B and 38.95 with Llama-3.1-8B-Instruct).

• Better sample efficiency. On ALFWorld Seen with Qwen3-4B, DeltaTTG reaches approximately 80% test success by step 75, whereas GiGPO requires around 125 steps — roughly 40% fewer optimization steps in that run. A similar advantage holds when progress is measured in actor-generated response tokens.

• Compute stays near GRPO. Under a fixed 150-step ALFWorld budget including progress-meter inference, DAPO uses 2.7× GRPO's GPU-hours due to dynamic sampling, while DeltaTTG matches DAPO's success rate at roughly GRPO's cost. The ALFWorld progress meter requires 10.45 GPU-hours of RL training and is reused across actor models.

• Out-of-domain generalization. Qwen3-4B checkpoints trained on PaperSearchQA (evaluated on BrowseComp-Plus) and on AgentWorldModel (evaluated on τ²-bench and Terminal-Bench 2.0, the latter reported as pass@8 over 89 tasks) achieve the highest performance among compared methods on all three out-of-domain benchmarks.

• TTG outperforms alternative supervision targets on step quality. On AgentProcessBench with neutral steps excluded, uniform random guessing has expected accuracy of 50% and direct prompting achieves 50.83% on the held-out test split, compared with 56.91% for Token-to-Go supervision. Against Future Score (terminal returns) and Step-to-Go (remaining step counts), Token-to-Go achieves the highest step-pooled Overall accuracy on both validation (61.83%) and test (56.91%) splits.

• Ablations favor the full method. On ALFWorld, full DeltaTTG achieves 91.67 Seen and 87.81 Unseen average success, higher than every ablated variant. Removing TTG supervision (replacing it with outcome-value, completion-logit, or binary-feasibility signals), reversing sign-aware allocation, skipping meter training, or setting ρ = 0 all reduce average success on both splits.

• Meter-guided test-time selection helps across actor sizes. With a fixed ALFWorld meter guiding Qwen3-1.7B, 4B, 8B, and 14B actors — four candidate actions trial-executed in separate environment branches, with the meter selecting the largest estimated progress — guided rollouts achieve higher success rates and fewer average interaction steps across all scales and both Seen/Unseen splits, exceeding random selection under the same four-candidate budget.

• Cross-domain meter transfer works. A meter trained only on ALFWorld trajectories, used without target-domain adaptation to guide actor optimization on PaperSearchQA, outperforms GRPO on Original, Paraphrased, and Overall evaluation.

Methodology in Plain English

Step 1 — Define an observable progress label. For each recorded trajectory, the authors compute Token-to-Go at each point as the number of assistant-generated tokens still to come before the end of the trajectory. This count includes reasoning, tool calls, and final answers, but excludes environment observations and tokens already in the prefix. Each trajectory also carries a binary success label. The core hypothesis is that, across many trajectories, beneficial actions from the same state leave fewer tokens to completion on average than harmful ones — even though exact token counts vary with model style and interaction harness.

Step 2 — Train a shared progress meter. Trajectories from diverse policies are split at action boundaries into prefixes and suffixes. Each prefix gets a completion label and (for successful trajectories) a TTG target. The meter is first trained by supervised fine-tuning to produce a structured answer — a completion prediction and a remaining-work estimate — then refined with GRPO using a reward that gives 0.2 + 0.8u(v̂, v) for correctly predicted successes, 1 for correctly predicted failures, and −1 otherwise, where u is a relative token-accuracy measure between predicted and true TTG. The meter is then frozen. One meter per domain serves actors of different families and sizes without retraining; each is initialized from Qwen3-4B-Instruct-2507.

Step 3 — Redistribute outcome advantages. During actor training, each trajectory still gets its usual group-relative outcome advantage from GRPO. The frozen meter scores every decision history and converts those estimates into state costs: the predicted remaining work for estimated-success states, and a large constant C for estimated-failure states, with terminal cost 0 for success and C for failure or truncation. The credit signal for an action is the reduction in predicted remaining work between consecutive states. These deltas are converted into nonnegative allocation weights, with a sign-aware rule that gives more positive credit to estimated progress on positive-advantage trajectories and negative credit of greater magnitude to estimated regression on negative-advantage trajectories. The weights are mixed with a uniform share using ρ = 0.2, normalized to unit token mean within each trajectory, and capped at a multiplier of M = 5. The actor is then optimized with the standard clipped surrogate loss (clipping widths 0.2 and 0.28, dual clipping coefficient 3) — only the advantage allocation changes, not the reward. Trajectory advantages set the update direction; step weights set its strength.

Step 4 — Evaluate. The authors compare against ReAct, GRPO, DAPO, GiGPO, and HGPO (plus Reflexion and two closed-model references in the appendices) across ALFWorld (3,553 training tasks; Seen 140 and Unseen 134 test tasks), PaperSearchQA (54,907 training examples; 5,000 test examples over 16 million PubMed abstracts), and BFCLv3 (Non-Live 1,150; Live 1,351; Multi-Turn 800; Hallucination/Irrelevance 1,122 cases), training tool-use policies on AgentWorldModel (9,850 tasks across 985 environments). Experiments build on verl-agent over veRL; the AgentProcessBench analysis uses AMD Instinct MI325X GPUs and all other experiments use NVIDIA H200 GPUs.

Why This Matters

Impact on research. The paper challenges the assumption that step-level credit requires matched peer states within a rollout group, an external judge, human annotations, or task-specific heuristics. It demonstrates that a single frozen model trained on cheap, already-recorded trajectories can supply a reusable progress signal — and that this signal transfers both across actor scales (1.7B to 14B) and across domains (ALFWorld meter applied to PaperSearchQA training). This reframes process supervision as a once-per-domain artifact rather than a per-run cost.

Real-world applications:

• Household and embodied assistants — ALFWorld covers navigation and object manipulation tasks such as picking, cleaning, heating, cooling, examining in light, and placing two objects.
• Scientific literature research assistants — PaperSearchQA requires retrieving evidence from 16 million PubMed abstracts to answer biomedical questions, including paraphrased queries.
• Tool-calling and API-driven agents — BFCLv3 and τ²-bench test function selection, argument generation, multi-turn tool use, and coordination with users in airline, retail, and telecom service scenarios.
• Software and systems agents — Terminal-Bench 2.0 covers command-line tasks in software development, system administration, and data processing with executable outcome checks; BrowseComp-Plus covers hard web information-seeking over a fixed corpus of 100,195 documents.

Industry relevance. Sparse terminal rewards are the default in production agent training, and fixing step-level credit typically means paying for judges, annotators, or extra rollouts. DeltaTTG keeps actor-training GPU-hours close to GRPO, needs no additional reward signal, and amortizes a 10.45 GPU-hour meter training run across multiple actor models — making finer-grained credit assignment comparatively cheap to adopt on existing RLVR pipelines.

Future Directions

1. Broader transfer validation. The authors note that evaluation covers only two actor model families and several domains, and that broader transfer requires further validation and may need adaptation.

2. Sensitivity of the failure-state cost C. The AgentWorldModel training audit found that only 6.3%–10.5% of successful trajectories contain any state the meter judges unsuccessful, so C seldom affects credit allocation on successful trajectories. The authors explicitly state this does not establish robustness to C and plan a more detailed sensitivity analysis.

3. Quantifying training variability. Reported standard deviations reflect repeated evaluations of a single trained checkpoint rather than independent training runs, leaving training variability unquantified.

4. Extending the meter beyond current domains. The meter is currently trained per domain (three meters for text-based interaction, information seeking, and complex engineering tasks), and the demonstrated cross-domain result is one direction of transfer; generalizing to new task families without adaptation remains open.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents, particularly those concerned with credit assignment, process supervision, and reward shaping under sparse verifiable rewards. It is also relevant to practitioners building multi-turn tool-use, search, or embodied agents who need step-level training signal without paying for human step annotations or external LLM judges. Readers should be comfortable with policy-gradient methods, GRPO-style group-relative advantages, and agentic benchmark evaluation to follow the methodology and tables.

Authors’ abstract

Reinforcement learning with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that say little about individual steps. Within-group step comparisons depend on reliable state matching and sufficiently diverse sampled outcomes, while direct step scoring may require costly external evaluators, human annotations, or task-specific heuristics. We introduce DeltaTTG to learn a reusable progress signal through a shared progress meter, using low-cost supervision from diverse policies' recorded trajectories without task-specific progress rules. Its supervision target, Token-to-Go (TTG), the number of tokens remaining to be generated until task completion, serves as an observable proxy for remaining completion effort. During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale. On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves the average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct, respectively, outperforms all compared RL baselines on average, and keeps actor-training GPU-hours close to GRPO. Further experiments show improved sample efficiency and out-of-domain performance, and support the meter's ability to distinguish step quality and guide test-time action selection across actor sizes. The code is available at https://github.com/Xuyan923r/DeltaTTG.