Self-Credit: Credit Assignment for Long-Horizon Agentic RL via Intra-group Rescaling

AI summary

Overview

Research area: Reinforcement learning for large language model (LLM) agents, specifically credit assignment under sparse outcome rewards in long-horizon, tool-using tasks.

Technical level: Advanced — the paper assumes familiarity with policy-gradient RL, GRPO-style group-relative advantages, and multi-turn agent rollouts.

Scope: The paper introduces Self-Credit, a method that turns a single sparse outcome reward into turn-level training signal for LLM agents by having the policy cite its own critical turns and then rescaling that credit using statistics from other rollouts of the same prompt.

What This Paper Is About

LLM agents that solve long-horizon tasks (searching, browsing, opening pages over dozens of turns) are typically trained with reinforcement learning from verifiable rewards, where a single 0/1 outcome reward arrives at the end. Standard group-based RL such as GRPO broadcasts that same advantage to every token in the trajectory, so the model gets no signal about which of its many actions actually mattered. The goal of this work is to produce dense, turn-level supervision from that same sparse reward without adding external judges, learned critics, or extra human annotation.

Key Contributions

1. Self-Credit, an end-to-end agentic RL algorithm that reuses the policy's own compaction summaries for turn-level credit assignment: when the context is compacted, the policy writes a summary that cites the indices of the few earlier turns that contained decisive evidence, and those citations become the credit signal.

2. Intra-group rescaling, a refinement that compares cited actions across all successful and failed rollouts for the same prompt, matching turns by the evidence they retrieve rather than by tool-call text. Group-level agreement and disagreement is used to strip unwarranted credit, emphasize credit on turns unique to successes, and strengthen negative credit on turns cited only by failures.

3. Advantage reweighting with an annealing schedule: rescaled per-turn weights multiply the trajectory-level advantage so that tokens in cited turns receive a stronger or weaker update than the rest of the trajectory, with the repeated-failure penalty weight annealed over training to trade off stabilization early and exploration later.

4. Empirical demonstration across model scales and benchmarks that Self-Credit beats zero-shot, GRPO, GiGPO, SUPO, and ECHO baselines on BrowseComp-Plus, GAIA, and xBench-DeepSearch, plus ablations isolating each component and analyses of citation quality and rescaling precision.

Main Findings

• BrowseComp-Plus gains over base models: Self-Credit improves over the base agents by 37.0 points on Qwen3-8B (from 3.6 to 40.6 accuracy) and 36.2 points on Qwen3-32B (from 10.8 to 47.0), and beats the strongest agentic RL baseline by 8.5 points (8B, against ECHO's 32.1) and 5.6 points (32B, against ECHO's 41.4). Results are over 3 seeds.

• Deep research gains on a different model family: Training Gemma4-E4B, Self-Credit reaches 39.2 on GAIA versus 34.3 for the strongest baseline (GRPO), a 4.9-point improvement, and 16.7 on xBench-DeepSearch versus 12.0 for GiGPO, a 4.7-point improvement.

• Better turns-per-accuracy tradeoff: On Qwen3-32B BrowseComp-Plus, Self-Credit reaches higher accuracy in 39.4 average turns versus ECHO's 45.8 turns at 41.4 accuracy; on Qwen3-8B, Self-Credit uses 37.9 turns versus ECHO's 62.6. GRPO and GiGPO produce much shorter trajectories (13.2 and 10.9 turns at 32B) with lower training reward, which the authors read as evidence that neither minimizing nor maximizing interaction length is sufficient.

• The repeated-failure penalty is the single most important component: Removing it drops BrowseComp-Plus accuracy from 47.0% to 34.9% on Qwen3-32B and from 40.6% to 26.5% on Qwen3-8B. Without it, the Qwen3-8B model learns to search excessively and never commits to an answer.

• Self-generated citations beat arbitrary turns: Applying the same rescaling and reweighting to all turns instead of model-cited turns drops accuracy to 42.2 (32B) and 27.7 (8B); using random citations drops it to 38.6 and 32.5. Removing intra-group rescaling entirely gives 41.0 and 28.9; removing mixed-outcome rescaling gives 41.0 and 31.3.

• Improved tool-use quality: On BrowseComp-Plus with Qwen3-32B, Self-Credit achieves Gold Searched 66.3%, Gold Opened 49.4%, Answer Recall 53.8%, and Evidence Recall 47.0%, versus ECHO's 62.7 / 30.1 / 51.6 / 48.3 and SUPO's 53.0 / 25.3 / 43.2 / 39.6. The Gold Opened gap is the largest, roughly 19 points over ECHO.

• Citations are higher quality than ECHO's turn selections: Against a Claude-4.6-Sonnet judge, Self-Credit reaches Judge F1 of 71.7 (Qwen3-32B) and 69.3 (Qwen3-8B), versus ECHO's 60.6 and 62.4, and it is notably better at identifying critical search turns (Searched F1 59.7 and 58.1 versus ECHO's 39.7 and 37.4).

• Rescaling fires selectively and accurately: Measured over an early stage (0–60 steps) and late stage (60–115 steps) of BrowseComp-Plus training, intra-group rescaling is applied to only a subset of cited turns, and its firing precision — whether it credits turns tied to gold documents and downweights or penalizes the rest — is higher than a random selection baseline.

• Generalization check: The authors report in an appendix that Self-Credit-trained models generalize better to out-of-distribution shorter-horizon search QA datasets, though the specific numbers are not in the main text provided.

Methodology in Plain English

The method piggybacks on context compaction, a technique already used to keep long agent histories inside the model's context window. When the running context hits a threshold (C tokens), the policy model is asked to summarize the history and, as the last line of the summary, list the turn numbers whose findings were decisive — the prompt asks for "usually 2 to 5 turns, NEVER every turn." The old history is discarded and the model continues from the query plus summary. That citation list is the raw credit signal, and malformed citations are penalized by zeroing the rollout's reward.

Raw self-citations can be gamed — a model could simply cite everything. The fix uses information already present in group-based RL: for each prompt, G rollouts are sampled, and some actions repeat. Turns are matched across rollouts by what they retrieved rather than by the wording of the tool call — two search turns match if the Jaccard similarity of their retrieved document-ID sets is at least 0.6, and two page-opens match only if they opened the same document. For each cited turn the method counts how many successful rollouts and how many failed rollouts contain a matching turn, then assigns a weight in three cases: cited by at least τ+ = 2 successes and no failures gets a boost above the base weight β; cited by at least τ− = 2 failures and no successes gets a negative weight w_f ≥ 1; cited by both successes and failures loses the boost. Everything else keeps the base weight, and uncited turns are unchanged.

Those weights multiply the usual trajectory-level GRPO advantage, so every token in a turn shares one reweighted advantage. Summary turns and final-answer turns are exempt from matching, and in successful rollouts they are treated as automatically cited so summarization and answer commitment still get credit. Because aggressive failure penalties early could over-stabilize search behavior, the failure weight w_f is annealed — strong early, weaker later.

Training used Qwen3-8B, Qwen3-32B, and Gemma4-E4B in thinking mode, with asynchronous rollouts in VeRL and Megatron. BrowseComp-Plus used 747 training examples and 83 test examples with a static corpus; deep research tasks used 1K QA pairs from OpenResearcher. Evaluation used LLM-as-a-judge accuracy with a separate Qwen3-32B judge (rubric from Gupta et al. for deep research), temperature 0.7, and an open web search API at test time for the deep research benchmarks.

Why This Matters

Sparse-reward RL is the default way LLM agents are trained, and it scales poorly as trajectories lengthen: one bit of feedback has to supervise more and more turns. Self-Credit shows that useful dense signal can be extracted from artifacts the model already produces and from statistics the training algorithm already computes — no reward model, no critic network, no process annotations. That makes it cheap to adopt inside existing group-based RL pipelines.

Real-world applications:

• Deep research and enterprise search assistants that must chain many queries and page reads, where the paper reports gains on GAIA and xBench-DeepSearch.
• Customer-support and workflow agents whose success is a single resolved/not-resolved signal but whose trajectories contain many tool calls.
• Coding and browser-use agents, where the matching recipe (match by what the action retrieved or returned, not by its text) transfers to any environment with identifiable observations.
• Long-running autonomous pipelines where context compaction is already mandatory, making citation nearly free to add.

Industry relevance: the technique requires no additional inference-time model and no extra supervision, it reduces turns-per-answer (a direct cost lever for agentic products), and it addresses reward hacking of self-generated feedback — a known failure mode when models are trained on their own outputs.

Future Directions

• Removing the independence assumption. The method assumes different turns in a rollout contribute independently to the outcome; relaxing this would let the rescaler reason about action sequences rather than isolated turns.
• Generalizing turn matching. The authors present matching as a task-specific function of observations and note it "can be applied unchanged to other agentic tasks once such a function" exists — building and validating those matchers for code, GUI, and robotics environments is open work.
• Extending beyond outcomes to partial credit. All supervision still derives from a binary reward; combining Self-Credit with recall-style or sub-goal signals could push further on evidence-heavy tasks, where Evidence Recall (47.0 at 32B) lags Answer Recall (53.8).
• Understanding and tuning the curriculum. The annealing of the failure penalty was chosen to balance stabilization and exploration; a principled schedule, and a principled choice of β, w_s, w_f, and the thresholds τ+ and τ− (fixed at 2 here), is not derived.

Target Audience

RL and post-training researchers working on LLM agents; engineers building tool-using or deep-research agents who already run GRPO-style training and want denser credit without a reward model; and graduate students studying credit assignment, reward sparsity, or context management in multi-turn RL. Readers without background in policy-gradient methods and group-relative advantages will need to consult the preliminaries section and the GRPO formulation in Appendix A first.

Authors’ abstract

LLM agents are commonly trained with RLVR, but the sparse nature of typical RLVR rewards makes credit assignment challenging, especially for long-horizon tasks. We propose Self-Credit, an end-to-end agentic RL method that creates dense supervision signals from sparse outcome rewards using self-generated credit assignment via intra-group rescaling. Self-Credit builds on group-based RL with three components. First, during its rollouts, the policy model generates summaries with citations to earlier turns containing critical evidence, which serve as self-generated credit. Second, we refine this model-generated credit with intra-group rescaling, which compares cited actions across successful and failed rollouts for the same prompt. This approach uses group-level agreement and disagreement to remove undeserved credit and further emphasize credit assigned to critical actions. Third, advantage reweighting uses the rescaled credit weights to update the advantages of the cited turns. On BrowseComp-Plus, Self-Credit improves over the base models by 37.0 and 36.2 points on Qwen3 8B and 32B respectively, and outperforms the strongest agentic RL baselines by 8.5 and 5.6 points, respectively. Self-Credit also improves over the strongest baselines by 4.9 points on GAIA and 4.7 points on xBench-DeepSearch. Analyses show that self-generated credit provides useful signals for selecting critical actions, and that intra-group rescaling, especially on failed rollouts, is important across model scales.