Which Agent's Skill Should Change? Group-Relative Credit Assignment for Evolving Multi-Agent Teams

AI summary

Overview

Research area: Natural Language Processing — specifically multi-agent LLM systems, automated credit assignment, and self-improving agents that evolve natural-language "skills" without weight updates.

Technical level: Advanced. The paper builds on GRPO-style group-relative advantage estimation and multi-agent reinforcement learning concepts, and assumes familiarity with LLM agent pipelines, prompting-based adaptation, and benchmark evaluation (HotpotQA, LoCoMo).

Scope (one sentence): The paper introduces GRSE, a training-free method that compares multiple executions of the same question to decide which agent's skill deserves credit or blame, then edits that skill — evaluated against MASkills on HotpotQA and LoCoMo with two open-weight actor models.

What This Paper Is About

Teams of language-model agents — one retrieving evidence, another answering — are judged by a single team-level score, so improving the team requires figuring out which agent's behavior actually caused a success or a failure. Existing skill-learning methods like MASkills credit skills merely for being present in a successful trajectory, so a harmful skill that happens to appear in good runs still earns positive credit. The paper's goal is to assign credit by comparison rather than by presence: run the same question several times under the same skill libraries, see which behaviors differ between better and worse runs, and revise only the responsible agent's skill.

Key Contributions

1. A formulation of skill evolution as behavioral credit assignment. With model parameters and agent roles frozen and natural-language skill libraries as the only editable units, the paper frames team improvement as attributing a shared outcome to a specific agent and a specific skill — while preserving the handoffs needed to interpret each agent's behavior.

2. Group-relative credit with structured, checkable diagnosis. Inspired by GRPO, GRSE samples multiple complete team rollouts per question and computes within-group relative advantages. A critic then attributes differences to agent–skill pairs, grounding each attribution in one of eight predefined failure types, cited failed and successful execution steps, and an edit action (refinement, induction, or no edit). Interface checks reject attributions that reference non-existent steps, mismatched agents, or rollout rewards that contradict the cited evidence.

3. Cross-question aggregation that does not double-count. Attributions are aggregated into edit directions for existing skills and into induction evidence for new ones; the support weight of an induction pattern counts distinct questions, so repeated rollouts of one question are not treated as independent evidence.

4. Validation-gated skill updates. Candidate libraries are produced with the editing operators and library limit of MASkills, but each candidate is kept only if it does not lower a held-out validation score beyond a tolerance δ, evaluated per agent against the pre-update team.

Main Findings

• HotpotQA gains over MASkills with both actors. GRSE scores 83.7 vs. 80.6 F1 with Qwen3.8-27B (+3.1) and 82.8 vs. 82.3 with Gemma-4-31B-it (+0.5). With Qwen3.8, GRSE comes within 0.4 F1 of the strongest single-model baseline, Self-Refine (84.1).

• Largest LoCoMo gains on multi-hop questions. GRSE beats MASkills by +6.7 F1 and +5.3 BLEU with Qwen3.8, and by +3.2 F1 and +2.8 BLEU with Gemma4 on multi-hop. It also achieves the highest temporal-question scores with both actors (73.4 F1 with Qwen3.8, 67.4 F1 with Gemma4).

• One category favors the prior method. Open-domain, the smallest LoCoMo category (52 questions), favors MASkills with both actors; GRSE improves over MASkills on the other three categories. (The conclusion section says "except multi-hop," which conflicts with the reported tables; the tables and abstract indicate open-domain.)

• Skills can hurt: GRSE breaks fewer correct answers. On HotpotQA with Qwen3.8, MASkills loses 1.8 F1 to questions moving from correct to wrong, versus 0.6 for GRSE; on LoCoMo with Gemma4 it loses 6.8 versus 3.1. With Gemma4 on LoCoMo, MASkills ends below the empty library on temporal questions (−4.3 F1), while both GRSE variants improve over the empty library in every category (+1.8 to +11.2 F1).

• Gains come from grounded retrieval, not from answering without evidence. With Gemma4 on LoCoMo, MASkills leaves the share of questions whose retrieved evidence cites no dialogue turn at 22%, and 9 of its 34 declined-to-correct questions lack evidence. GRSE lowers that share to 14%, with only 5 of 61 lacking evidence.

• Learned skills differ in kind by method. On LoCoMo, retrieval procedures are the largest skill type for every method and actor. With Gemma4, GRSE learns more abstention rules than MASkills (18% vs. 5%). On HotpotQA with Qwen3.8, GRSE mostly writes answer-format rules (57%) while MASkills mostly writes reasoning procedures (50%), which the authors link to MASkills' higher rate of broken answers.

• Robustness to corrupted libraries. Starting from a library with planted harmful/irrelevant skills on LoCoMo single-hop (453 questions, Qwen3.8), GRSE recovers the score of the helpful skills alone (66.9 vs. 66.7 F1, +6.3 over the perturbed start) and halves abstention (9.7% to 4.6%), whereas MASkills ends slightly below its starting point (58.4 F1, −2.2). GRSE beats MASkills by +8.5 F1 in this setting.

• Correct credit signs. After the first update, GRSE assigns the correct sign to all five helpful/harmful planted skills and separates the two kinds by 0.31, whereas MASkills gets two of the five wrong and separates them by only 0.03. MASkills raises the utility of the harmful verbatim-evidence-guard skill over the next two iterations; GRSE prunes it in iteration 3.

• Learned skills transfer across multi-hop tasks. Swapping final libraries between HotpotQA and LoCoMo multi-hop (Qwen3.8) never hurts relative to the empty library, and GRSE's in-domain libraries still beat its transferred ones (+1.9 and +1.2 F1), indicating no overfitting to the source task. MASkills' in-domain libraries instead fall below its transferred ones (−3.5 and −2.1).

• Ablations confirm both components matter. On a shared library across the four LoCoMo categories (Qwen3.8, 842 test questions), GRSE reaches the best category average (54.9 F1, 42.2 BLEU), +2.9 F1 over the empty library; removing structured diagnosis or using G=1 lowers average F1 by 1.0 and 0.8 respectively.

• Equal-reward groups still carry information. A DAPO-style filter that discards the 7–13 of 32 groups per iteration with identical rewards lowers the category average by 0.5 F1 and 0.6 BLEU.

• Dataset labels bias the shared library. Adding labels tops the pooled "All" score only via single-hop questions (54% of the test set, +4.7 F1 and BLEU), trails GRSE on every other category (most on open-domain, −6.7 F1), and has the lowest category average (53.0 F1).

• Cost is confined to training. G=4 raises actor-token cost by 1.6–2.9× and wall-clock time by 1.3–1.8× over MASkills; test-time inference is unchanged.

Methodology in Plain English

The team starts with two agents (a retriever and an answerer) whose model weights never change. Each agent's behavior is defined entirely by a library of plain-English "skills" — reusable procedures the agent reads and follows. Initially these libraries are empty.

Each training iteration has four moves:

1. Ask the same question several times. For each question in the batch, the team is run four times from a fresh context, keeping the skill libraries identical. Each run gets a reward between 0 and 1, and rewards are normalized within the group (subtracting the group mean, dividing by the group standard deviation) to produce a relative advantage, exactly as in GRPO. Groups where all four runs score the same get an advantage of zero but are still shown to the critic.

2. Compare the runs agent by agent. A single fixed critic model (Qwen3.8-27B for all methods) reads all four complete trajectories side by side — actions, messages, tool calls, final answers — and asks which agent's behavior differed between the high- and low-advantage runs. This is the key move: the agent whose answer is wrong is often not the agent whose behavior needs to change. A reasoner that says "not mentioned" is behaving correctly if the retriever never surfaced the evidence.

3. Force the attribution to be checkable. Each attribution must name a failure type from eight options (skill not followed, skill defect, missing capability, handoff error, evidence missing, answer format, tool failure, uncertain attribution), cite specific failed and successful steps from specific rollouts, and propose an action. Automated checks reject attributions whose cited steps don't exist, belong to the wrong agent, cite a full-reward run as a failure, or propose a refinement of a skill that isn't in the cited rollout. Tool failures and uncertain attributions may only yield "no edit."

4. Aggregate across questions, then edit and validate. Credit for an existing skill is summarized into an edit direction (reusing MASkills' aggregation operator), while evidence for new skills is grouped into patterns whose support counts distinct questions only. The MASkills editing operators then refine, prune, consolidate, or induce skills. Every candidate library is scored on a validation set and accepted only if it does not drop below the current team's score minus a tolerance, tested one agent at a time.

Evaluation uses HotpotQA's distractor-hard subset (500 test questions, answer-token F1) and LoCoMo's four categories (multi-hop 156, temporal 179, open-domain 52, single-hop 453 questions; category F1 and smoothed BLEU). All skill methods start from empty libraries and run five iterations; final libraries are evaluated with temperature-zero decoding, averaged over three training runs.

Why This Matters

Research impact. The paper isolates a specific flaw in trajectory-level credit: presence is not responsibility. A skill that appears in every rollout collects credit for successes it did not cause, and the robustness experiment shows this concretely — MASkills keeps increasing the utility of a planted harmful skill across iterations. Group-relative comparison, already standard for optimizing weights, is repurposed here as a diagnostic instrument for text-level behavior, and the structured-diagnosis-plus-checks layer makes the resulting attributions auditable rather than free-form narration. The finding that labels bias a shared library toward the majority category is also a useful caution for anyone building multi-task skill banks.

Real-world applications.

• Enterprise and customer-support copilots built from a retriever plus a responder, where the failure is usually upstream in search rather than in the phrasing of the answer.
• Clinical and biomedical question answering (relevant to the Mayo Clinic authors), where "declining to answer" versus "answering without cited evidence" is a safety distinction, not just an accuracy one — GRSE's abstention-rule learning and evidence-citation metrics target exactly this.
• Long-conversation memory assistants over meeting or chat logs, the LoCoMo setting, where the right dialogue turn must be found before any reasoning can help.
• Multi-agent coding or research assistants where a planner, a tool-calling agent, and a writer hand off work and one wrong output can originate several steps earlier.

Industry relevance. The method requires no fine-tuning, no gradients, and no parameter access — only inference calls and a fixed critic model — so it can be layered onto deployed agent products whose skills are prompt artifacts. The validation gate means a bad update is discarded rather than shipped. The cost profile is confined to offline training (1.6–2.9× actor tokens, 1.3–1.8× wall-clock over MASkills), which matters for teams deciding whether the accuracy gain justifies the extra rollouts.

Future Directions

• Scale beyond two agents and two benchmarks. The authors note GRSE is defined for any number of agents but was only tested on two-agent sequential teams, two question-answering benchmarks, two open-weight actors, and one critic model. Whether group-relative attribution stays reliable as teams grow — and as handoffs become less linear — is open.

• Reduce the rollout cost. Sampling four trajectories per question is the main efficiency price. The DAPO-filter ablation suggests equal-reward groups are not useless, so smarter sampling or adaptive group sizes could cut cost without losing signal.

• Diagnose the open-domain gap. Open-domain is the one LoCoMo category where MASkills wins with both actors, and it is also the smallest (52 questions). Whether this reflects a genuine weakness of behavioral credit, or a data-size artifact, is unresolved.

• Move beyond question-level aggregation. Support is counted per distinct question, and the label-conditioned variant shows the aggregation is sensitive to category imbalance. Alternatives to per-question aggregation across heterogeneous task mixes, and use of the redundancy/conflict links the critic already produces, remain unexplored.

Target Audience

Researchers and advanced practitioners working on LLM agent systems, self-improving agents, and multi-agent credit assignment; engineers building retrieval-plus-reasoning pipelines who want agents to improve from their own execution logs without fine-tuning; and readers interested in how RL ideas such as group-relative advantages and counterfactual baselines translate into text-level, editable behavior. Readers without background in reinforcement learning or agent evaluation will find the method sections dense, though the core intuition — compare runs of the same task to find who is responsible — is accessible.

Authors’ abstract

Cooperative language-model agents can improve without weight updates by revising reusable skills, but only if a shared team reward is traced to the responsible agent and skill. Existing methods credit the skills present in each trajectory, yet presence is not responsibility: a harmful skill active in successful runs still earns positive credit. We present \textbf{GRSE}, \textbf{G}roup-\textbf{R}elative \textbf{S}kill \textbf{E}volution, which assigns behavioral credit by comparison: inspired by GRPO, it executes each question several times under the same skill libraries and contrasts the executions to credit agent--skill pairs, with structured diagnosis of failures. On HotpotQA and LoCoMo with two open-weight actors, GRSE outperforms MASkills, the closest prior method, on HotpotQA and every LoCoMo category except open-domain, with the largest gains on multi-hop questions ($+6.7$ and $+3.2$ F1). Its gains rest on better-grounded retrieval rather than answers given without evidence, and its learned skills transfer between the multi-hop tasks. GRSE is robust to corrupted libraries: starting from planted harmful skills, it credits every planted skill with the correct sign and recovers the score of a clean library ($+6.3$ points), whereas MASkills misattributes two and falls below its starting point.