MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

AI summary

Overview

Research area: Natural Language Processing — reinforcement learning and post-training for LLM-based multi-agent systems, specifically retrieval-augmented open-domain question answering.

Technical level: Advanced. The paper assumes familiarity with GRPO/PPO-style policy optimization, KL-based distillation, and agent tool-use loops.

Scope (one sentence): The paper introduces MultiAgent-OPD, a two-stage recipe that jointly trains a shared-parameter Leader–Searcher–Answerer team end to end by giving Searchers token-level supervision from a frozen, task-trained Teacher while training the Leader and Answerer with outcome rewards, and evaluates it on seven QA benchmarks against cascaded, hierarchical, and RL-based search baselines on Qwen3-4B-Instruct-2507.

What This Paper Is About

In a hierarchical multi-agent system, a Leader splits a question into subtasks, Searchers gather and summarize evidence, and an Answerer writes the final response. Training all of these roles together is hard because only the final answer is scored, so a Searcher that did good work gets no credit if the Leader ignored its report or the Answerer misused it. The paper's goal is to give each role a learning signal that fits its actual job: outcome rewards for the Leader's coordination and the Answerer's answer, and dense, token-level guidance for the Searchers from a separate policy trained in a simpler Searcher–Answerer workflow.

Key Contributions

1. A framework (MultiAgent-OPD) for end-to-end hierarchical multi-agent training that combines local Searcher supervision with outcome-based optimization of the Leader and the Answerer, with all three roles updating one shared student model.
2. A formulation of Searcher process supervision as on-policy distillation from a task-trained Teacher. The Teacher is first trained in a simpler Searcher–Answerer workflow using answer correctness, then frozen and used to score Searcher tokens on the student's own evolving subtask trajectories.
3. Empirical results on seven QA datasets reporting 38.68% macro avg@4 and 48.73% macro pass@4, higher than the included cascaded, hierarchical, Prompted Critic, and RL-based search baselines.
4. Diagnostic comparisons isolating the Searcher learning signal, the value of task-specific Teacher training, and the effect of direct Searcher supervision — including a documented collapse of search behavior when Searchers are trained purely from final-answer rewards.

Main Findings

• Top score on the seven-dataset macro average: MultiAgent-OPD reaches 38.68% macro avg@4, exceeding Prompted Critic (34.62%) by 4.06 percentage points (95% CI [+2.98, +5.12]) and C-MAS + RL (36.07%), the trained cascaded system, by 2.61 percentage points (95% CI [+1.49, +3.73]).
• Beats every included RL-based search agent: the strongest of the five is Tree-GRPO at 37.37%, which MultiAgent-OPD exceeds by 1.31 percentage points. The other four report Search-R1 33.39%, DeSA 36.96%, MemSearcher 36.81%, and Dr. MAS 35.54%.
• Higher coverage across attempts: 48.73% macro pass@4 versus 45.97% for C-MAS + RL and 45.96% for Prompted Critic; the best included RL agent is MemSearcher at 47.64%.
• More consistent answers: 29.27% of questions are answered correctly on all four attempts, versus 27.05% for C-MAS + RL and 23.24% for Prompted Critic.
• Hierarchy beats cascade even untrained: the base hierarchical system (H-MAS base) reaches 28.60% macro avg@4 against 25.09% for C-MAS base, a 3.51 percentage point gap, with macro pass@4 rising from 34.02% to 39.32%.
• Prompted judging is weaker than trained local supervision: C-MAS + RL (Stage-I policy in the cascaded workflow) scores 36.07% versus 34.62% for Prompted Critic, an M-GRPO-like combination of team rewards and a prompted judge of Searcher reports; their pass@4 values are nearly tied (45.97% vs 45.96%).
• Outcome-only Searcher training collapses retrieval: replacing Searcher OPD with group-relative advantages from final-answer rewards drops macro avg@4 to 17.78%, which is 10.82 points below H-MAS base (28.60%). In that variant, none of its 20,000 test workflows records a search action, even though 99.51% still produce subtasks and reports and 99.03% produce a final answer.
• The Teacher must be task-trained: swapping in the untrained base model as Teacher, while keeping the distillation objective, drops macro avg@4 from 38.68% to 29.50% and pass@4 from 48.73% to 40.86%, with the full method higher on every dataset in both metrics.
• Removing direct Searcher losses hurts: without them, macro avg@4 is 29.82% and pass@4 is 39.86%, even though the Leader and Answerer keep their outcome rewards and Searchers still change through shared parameters.
• Accuracy costs more inference work: MultiAgent-OPD uses 5.39 search actions and 9,172 input tokens per workflow, versus 4.45 and 6,786 for Prompted Critic and 2.86 and 3,886 for C-MAS + RL. Relative to Prompted Critic, searches rise by 21.4% and input tokens by 35.2%, while output tokens fall from 885 to 697. The authors note C-MAS + RL reaches 36.07% with the fewest searches, so workload alone does not explain the ranking.
• Gains are not an artifact of answer formatting: MultiAgent-OPD has a 0.68% macro format-failure rate and 0 rescued answers under flexible extraction; even if every Prompted Critic format failure were repaired, the remaining gap would be +0.33 pp (95% CI [−0.80, +1.46]), and against C-MAS + RL it would remain +2.51 pp (95% CI [+1.39, +3.65]).
• Largest per-dataset edge over C-MAS + RL is on Bamboogle: +7.40 percentage points; overall MultiAgent-OPD exceeds Prompted Critic in avg@4 on all seven datasets and C-MAS + RL on five.

Methodology in Plain English

The team uses one shared model, Qwen3-4B-Instruct-2507, and a role prompt to decide whether a given call acts as Leader, Searcher, or Answerer. Training happens in two stages.

Stage I — build the Teacher. The model is trained in a simpler two-role workflow: a Searcher receives the question, issues queries to a fixed retriever, reads the returned passages, and writes a report; an Answerer then answers from the report alone. Rewards are mostly answer correctness (0.9 × team exact-match outcome) plus a small 0.1 × behavior bonus (0.5 for summarizing after one search step, 1 after two or more, 0 otherwise), with a total reward of −1 for malformed actions. This trained policy is then frozen as the Teacher.

Stage II — train the team. The hierarchical student starts again from the base model. For each question it samples a group of full team trajectories: the Leader breaks the question into subtasks, spawns waves of parallel Searchers, collects their summaries, and hands evidence to the Answerer, which produces five answer candidates per request. Each role then gets a different advantage:

• Leader: GRPO advantage from a team-outcome reward, 0.9 × exact-match outcome + 0.1 × a bounded search-activity bonus of min(tool steps with a search, 6)/6.
• Answerer: GRPO advantage from each candidate's own exact-match score, normalized among candidates within the same workflow.
• Searchers: no outcome reward at all. Each generated token is scored by the frozen Teacher evaluated on exactly the same role prompt, subtask, retrieved passages, and generated prefix as the student. The teacher–student log-probability ratio becomes the token's advantage, scaled by β = 0.01. Tokens from the retriever and padded positions are masked out.

Everything is optimized with a PPO clipped objective and an entropy bonus, with a KL penalty of 10^-3 toward the Teacher applied to Leader and Answerer rewards (Searchers get no extra KL term because the distillation term already plays that role). Because the Teacher sees the student's own prefixes and the Leader's current subtask decomposition, the supervision follows the team as its policies change. At test time the Teacher is gone and the single student plays all three roles.

Evaluation uses training on NQ and HotpotQA and a fixed test set of 5,000 questions drawn from NQ (812), TriviaQA (812), PopQA (812), HotpotQA (813), 2WikiMultiHopQA (813), MuSiQue (813), and Bamboogle (125), sampled with seed 20260728 after removing 14 entries that matched normalized training questions. Each system runs four times per question with one Answerer candidate per request, giving 20,000 records per system and 100,000 across the five in-house systems. Scoring is normalized exact match; the headline number is macro avg@4, weighting the seven datasets equally, with uncertainty from 4,000 question-bootstrap draws (seed 20260912).

Why This Matters

Impact on research. The paper makes a concrete argument about where credit should come from in multi-agent RL: not from an LLM judge reading a subagent's report, and not from the final answer alone, but from a policy that already knows how to do the subagent's job, queried at exactly the states the student reaches. The diagnostic showing that outcome-only Searcher training eliminates search entirely is a useful negative result — it shows that sparse final-answer rewards can quietly optimize away the very tool use the system depends on. The result is also a template for reusing a policy trained in a cheap, simple workflow to supervise the same role inside a more complex one.

Real-world applications:

• Retrieval-augmented enterprise assistants where a planner delegates sub-queries to parallel searchers over internal document stores and must decide which evidence reaches the final answer.
• Deep-research and report-generation products that fan out searches across subtasks and synthesize a cited answer, where per-searcher quality is otherwise unmeasured.
• Customer-support and ticketing agents with a triage/leader layer that routes to specialist lookups and needs to learn from resolution outcomes.
• Tool-orchestration platforms where one controller calls search, database, or API subagents and where training a controller jointly with its workers is currently avoided in favor of freezing the workers.

Industry relevance. The method targets a practical pain point: teams already deploy Leader–Subagent harnesses, but as the paper notes, systems like PARL optimize the Leader with frozen subagents, leaving subagent adaptation outside joint training. MultiAgent-OPD keeps a single shared model for all roles, so serving requires no extra model at inference — a meaningful operational property for production deployment, at the cost of roughly 35% more input tokens than the Prompted Critic baseline.

Future Directions

• Matched-budget comparison. The authors explicitly note they do not compare systems under a matched retrieval or token budget, and that systems run at their own workflow budgets (5 environment steps for cascaded, 12 for hierarchical). A compute- or search-matched comparison would clarify how much of the 2.61-point gain over C-MAS + RL is signal versus extra search.
• Stability across training runs. The reported intervals come from question-level bootstrapping of fixed checkpoints, not from independently trained seeds, and the training-dynamics figure is described as one smoothed historical run per method. Variance across seeds remains unmeasured.
• Better local supervision than log-probability ratios. The authors are careful that the teacher–student log-ratio is a preference weight, not a causal estimate of a token's contribution, and that the Teacher can be wrong, is trained on whole questions rather than delegated subtasks, and can reduce exploration through reverse KL. Alternatives — verifiers, step-level rewards, or hindsight relabeling as in cited self-distillation work — are natural follow-ups.
• Generalizing beyond open-domain QA and English. The evaluation is limited to a fixed English open-domain QA set with exact-match scoring, which cannot detect semantic overlap or pretraining contamination. Extending to other task families, other hierarchies, and evaluation of whether retrieved evidence actually supports the answer would test how far the recipe travels.

Target Audience

Researchers and engineers working on multi-agent LLM systems, RL post-training, or agentic retrieval who already understand policy-gradient methods and want a concrete design for assigning credit across planner, retriever, and answerer roles. It is most useful to practitioners deciding between a heavy prompted-judge setup and a trained local supervisor, and to those building production agent harnesses where a single shared model must serve every role. Readers new to RL fine-tuning will find the empirical diagnostics readable but will need background to follow the advantage derivations.

Authors’ abstract

In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly…