DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

AI summary

Overview

• Research area: Reinforcement learning for large language model agents, specifically reward design and credit assignment for long-horizon tool-using agents in settings without a programmatic verifier.
• Technical level: Advanced. The paper is written for readers comfortable with policy-gradient RL (GRPO), advantage estimation, and agent benchmarks such as AppWorld and τ-bench.
• Scope: The paper introduces DRACO, an outcome-blind training method that builds rubrics dynamically from rollouts and redistributes trajectory-level rubric scores into per-step advantages within GRPO, evaluated on AppWorld and τ-bench.

What This Paper Is About

Reinforcement learning from verifiable rewards (RLVR) assumes a programmatic checker that can confirm task success, but many real-world agent domains (customer support, open-ended research) have no such oracle. The paper targets the "outcome-blind" regime, where no ground-truth success or gold-answer signal is available at any point during training, and rewards must come entirely from process criteria such as rubrics. The goal is to make rubric-based rewards usable for long-horizon agents, where a trajectory spans tens of interdependent tool-calling steps and a single trajectory-level scalar is a poor learning signal.

Key Contributions

1. Formulates outcome-blind rubric-based RL for long-horizon tool-using agents, positioning it against the rubric, step-credit, and process-reward literature, nearly all of which anchors its signal to a ground-truth outcome.
2. Proposes two complementary mechanisms: dynamic per-trajectory rubric generation that adapts criteria to what a given rollout actually does, and distribution of rubric-conditioned advantage to steps within GRPO. Neither requires a verifier.
3. Provides a closed-form credit rule with formal guarantees, satisfying seven properties including total-push conservation, sign preservation, and length independence (Appendix E.9), with no trained attribution module.
4. Reports empirical results on AppWorld and τ-bench, including separate and joint component ablations, an outcome-reward baseline, a self-judge replacing the frontier judge, and analysis of training effects on efficiency and on the reward the policy actually optimizes.

Main Findings

• Gains on AppWorld with Qwen3.6-27B: DRACO improves AppWorld TN TGC/SGC from 69.4/41.1 to 85.3/70.6 (+15.9/+29.5) and AppWorld TC TGC from 49.7 to 61.5. The paper reports +15.9 points over the base model overall.
• Gains with the smaller base model: On Qwen2.5-32B-Instruct, DRACO raises AppWorld TN TGC/SGC from 35.7/17.3 to 62.9/42.3 (+27.2/+25.0), the larger gain of the two base policy models, closing most of the gap to the outcome-aware SALT baseline (66.2/47.9), which uses ground-truth reward.
• Beats a verifier-trained run: DRACO outperforms a GRPO run trained on AppWorld unit tests by +5.3 TGC and +11.3 SGC on AppWorld TN, and +1.6 TGC and +2.4 SGC on AppWorld TC, despite using no verifiers itself.
• Consistency improves more than discovery: On AppWorld TN, TGC p³ increases by 25.2 points while pass@3 rises by only 3.9; the untrained model could often solve tasks at least once, and DRACO makes those successes reliable.
• Zero-shot transfer to τ-bench: Training exclusively on AppWorld, DRACO raises τ-bench Banking success rate from 15.8 to 20.4, without access to any verifier, gold answers, or reference trajectories.
• Both components are needed together: On AppWorld TN, adding step credit to per-trajectory rubrics is worth +3.2 TGC and +5.7 SGC, and the two components together are worth +4.2 TGC and +10.7 SGC over the fixed rubric alone, growing to +8.1 TGC and +14.3 SGC at p³. Either component alone contributes far less (+0.8 and +1.0 TGC). On AppWorld TC, step credit on a fixed rubric costs 3.7 TGC at p³, while on per-trajectory rubrics it adds +1.4 TGC.
• Beats an outcome reward model reference: The outcome-reward setting, trained on AppWorld unit tests with vanilla GRPO, scores 80.0 TGC and 59.3 SGC on AppWorld TN, below DRACO's 85.3 and 70.6.
• A self-judge can replace the frontier judge: Replacing GPT-5.4 with the policy model judging its own rollouts cuts judge cost for 100 training steps from $1607 to $316, a 5.1x saving. On AppWorld TN it scores 81.1/62.7 TGC/SGC against the outcome-aware reference's 80.0/59.3, and on τ-bench it achieves the best SR of any setting (21.1).
• Self-judge agreement: Replaying scoring calls through the policy checkpoint shows 89.4% of 60,689 criterion-level verdicts agree with the frontier judge, against 72.0% for a judge that passes everything. Where they disagree, the self-judge is lenient: it passes 30.4% of criteria the frontier judge failed while failing only 1.3% of the ones it passed.
• Lower evaluation cost and fewer turns: On AppWorld TN, DRACO reaches 85.3 TGC for $8.27 against 69.4 at $10.77 untrained; on AppWorld TC, 61.5 for $38.03 against 49.7 at $43.56. Agent turns drop from 18.7 to 14.7 on AppWorld TN and from 22.9 to 20.7 on AppWorld TC with Qwen3.6-27B. On τ-bench, turns rise slightly for every setting.
• Dynamic rubrics keep discriminating: Against static rubrics, the reward saturates in the mid-90s within about 25 steps (static settings average 94.8% and 95.8%), while dynamic settings sit lower and keep moving (66.9% and 74.2%). Re-scoring DRACO's rollouts against the fixed static rubrics they never trained on recovers 91.3%, so per-trajectory rubrics subsume the static criteria.
• Rollout termination cleans up: Dynamic settings converge so that the agent almost always submits an answer, with length, turn-budget, server-failure, and no-action endings fading to negligible. DRACO w/o Cred. spends the first 30 steps oscillating, at times dropping below 30% submitted, whereas DRACO settles within roughly the first 40 steps.
• Difficulty breakdown: On AppWorld TN, step credit on per-trajectory rubrics improves TGC at every difficulty level and is largest on medium tasks (+7.2 TGC, +14.6 SGC), and lifts easy scenario completion by +8.2 SGC.
• Random rewards are far weaker: Training on random rewards reaches 74.0 TGC and 50.0 SGC on AppWorld TN, well below DRACO's 85.3 and 70.6.

Methodology in Plain English

The setup is a policy model interacting with a stateful environment over an episode of tens of steps, producing interleaved reasoning and tool calls. Two pieces make up the method.

Dynamic per-trajectory rubrics. Rather than authoring one fixed rubric for an entire task distribution, a judge first proposes criteria from the task instruction alone, then extends them once per sampled rollout, adding sub-goals the rollout reveals and ways the policy actually fails. Proposals from all trajectories in a group are merged and deduplicated into a single criterion set. Generation calls are instructed to keep criteria mutually exclusive and collectively exhaustive, because the reward is a rate and overlapping criteria would double-count one mistake. A "discriminative dropout" step then keeps a criterion only if some group member failed it. A frozen external judge scores each trajectory against every surviving criterion with pass, fail, or not applicable, plus a justification and the steps responsible. The trajectory reward is the pass-minus-fail count divided by the applicable verdict count, so trajectories with different numbers of applicable criteria remain comparable, and the reward contains no task-success or gold-answer term.

Rubric-conditioned step credit. Placing that reward into GRPO still gives every token in the trajectory the same advantage. DRACO reallocates the trajectory advantage across steps according to which steps the judge's verdicts cite, without changing the trajectory's total push. Each step's quality is the pass fraction of the criteria citing it; steps cited by no criterion inherit the mean quality over cited steps. The step weight depends on the sign of the trajectory advantage: a winner weights steps by their quality, a loser by one minus their quality, so credit is never inverted relative to the judge's verdict. Each token in a step receives the same step advantage, scaled so that a step's total contribution depends on its quality weight rather than its length. Summing over steps conserves the trajectory's total push exactly — the same total baseline GRPO applies to those tokens. When no step is cited, the update falls back to vanilla GRPO.

Training loop. Each step samples a group of trajectories per task, builds the merged rubric, scores every trajectory with the frozen judge, drops criteria no member failed, computes the trajectory reward over survivors, standardizes within the group into advantages, computes step qualities, weights, and step advantages from the retained criteria's citations, and updates the policy.

Experimental setup. Four outcome-blind settings start from the same base policy Qwen3.6-27B and are named for what they remove: DRACO w/o Dyn. & Cred. (static rubric, no step credit), DRACO w/o Dyn. (static rubric with step credit), DRACO w/o Cred. (dynamic rubrics without step credit), and DRACO. Training uses LoRA adapters with GRPO, batch size 16, group size 6, on 8 H100 GPUs, with GPT-5.4 at temperature 0.1 as the single judge for all reward operations. Training data is the AppWorld training split (90 tasks). Qwen3.6-27B trains for 100 steps (20 epochs) and Qwen2.5-32B-Instruct for 75 (15 epochs), with results averaged over the final three checkpoints and three evaluation runs per checkpoint. Evaluation covers AppWorld TN (168 tasks), AppWorld TC (417 tasks), and τ-bench Banking (97 tasks), with success measured by Task Goal Completion, Scenario Goal Completion, and state-matching Success Rate, plus pass@k and pass^k over three runs.

Why This Matters

Impact on research. The paper defines and attacks a setting that most reward-learning work assumes away: training a long-horizon agent with no verifier and no gold answer anywhere in the loop. It shows that a process-criteria signal, redistributed at step granularity, can match or exceed a verifier-trained run of the same budget, and it offers a closed-form, learned-module-free credit rule as an alternative to trained progress estimators, recurrent attribution, or per-position judge calls. It also demonstrates that a policy model grading its own rollouts can substitute for a frontier judge at a 5.1x cost reduction, which changes what is economically feasible in this line of work.

Real-world applications:

• Customer-support agents, where success is judged by conversation quality and issue resolution rather than a unit test.
• Open-ended research or deep-research agents, whose multi-stage workflows have no programmatic success check.
• Enterprise task automation (booking, refunds, record updates) where behavior can be graded against written policies and compliance criteria.
• Any agent deployment where the target of optimization must be explicit and auditable, since the reward is a set of natural-language criteria rather than a learned scalar.

Industry relevance. The method needs no verifier to construct, uses LoRA adapters and a single judge model, and lowers both evaluation cost and agent turns relative to the untrained policy — DRACO reaches 85.3 TGC for $8.27 where the untrained model scores 69.4 at $10.77 on AppWorld TN. The self-judge result, cutting judge cost from $1607 to $316 for 100 training steps, directly addresses the largest cost in the pipeline.

Future Directions

• Validate the criteria themselves. The paper states that on unverifiable tasks there is no independent signal to check whether the rubric describes the task faithfully, and that doing so would require human raters; a chance-corrected measure of judge agreement with human annotators is also not reported.
• Verify that credit lands on the right steps. The authors note that end-task performance does not establish correct attribution: crediting the wrong steps can still improve a policy, and crediting the right steps can fail to.
• Characterize training-time variance from discriminative dropout. Because a criterion survives only if some group member failed it, the same prompt can be scored against different criteria at different points in training; the resulting variance is not measured, and repeated runs are inference-time repeats of fixed checkpoints.
• Understand required judge quality. Since the judge defines the objective rather than estimating it, the paper calls understanding the judge quality needed for the method a natural direction, especially given that a judge can be internally consistent yet systematically wrong.

Target Audience

Researchers and engineers working on reinforcement learning for LLM agents, post-training and reward design, and agentic evaluation. It is most useful to readers already familiar with GRPO, advantage estimation, and tool-use benchmarks such as AppWorld and τ-bench, and to practitioners deciding how to train long-horizon agents in domains where no programmatic success checker exists.

Authors’ abstract

Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.