Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

AI summary

Overview

Research area: Multi-agent large language model (LLM) systems, reinforcement learning post-training, and multi-agent collaboration.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, group-relative advantages), Bayesian posterior inference, and multi-agent evaluation benchmarks.

Scope: A post-training method called CoSIR that rewards individual messages in multi-agent LLM discussions using a computable Bayesian "team surprisal" signal derived from a simplified rating-table environment, evaluated for transfer to seven unseen collaboration tasks across three model backbones.

What This Paper Is About

LLM agents working in teams often fail to collaborate well: they withhold useful findings, proceed without asking for clarification, ignore teammates' input, or abandon correct answers under majority pressure. Training these behaviors is hard because a final team outcome gives little feedback about the value of any individual message. The paper's goal is to make the value of sharing or requesting a piece of information directly computable, so that collaboration behaviors can be trained in a simple synthetic environment and then transfer to more complex, unseen natural-language tasks.

Key Contributions

1. A scalable collaboration training environment. The authors construct a rating-table task in which agents each observe only part of a table of ratings and must jointly select the option with the highest total. Because every shared or requested rating maps to a specific table entry, the environment can identify, verify, and track the information exchanged.

2. Verifiable information rewards. They introduce a Bayesian evaluator that scores shared ratings by the reduction they produce in "team surprisal" (the sum of negative log probabilities each agent assigns to the correct option), and scores requests by the potential reduction if the requested rating were truthfully provided. Rewards are computed numerically from the table and information records rather than assigned by a learned reward judge.

3. Rewards for using received information. Complementary supervision on private answers scores the option an agent selects by the probability that it is best given the agent's available information, and scores reported totals by whether they correctly combine the agent's known ratings.

4. Demonstrated cross-task generalization. Post-training Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it via self-play yields improvements on all seven unseen collaboration tasks for every model, with the highest aggregate score among compared methods and general capabilities largely preserved.

Main Findings

• CoSIR achieves the highest SUM across all three backbones. SUM is the equal-weighted sum of the seven task scores. Qwen3-8B Base scored 352.60 and CoSIR scored 384.44 (+31.85); Qwen3-4B Base scored 333.17 and CoSIR scored 357.87 (+24.70); Gemma-3-4B-it Base scored 208.02 and CoSIR scored 235.94 (+27.92).

• Every trained model beats its own starting checkpoint on all seven tasks. The paper reports that each trained model achieves a higher mean score than Base on all seven tasks, indicating the gains are not driven by a single benchmark.

• CoSIR exceeds the strongest alternative by 19.79, 22.95, and 24.86 points for Qwen3-8B, Qwen3-4B, and Gemma-3-4B-it respectively. Inference-time methods ToM and TalkHier lead on some individual tasks (ToM on SILO-BENCH with Qwen3-8B; TalkHier on BenchForm with all three models), but both reduce SUM relative to Base for Qwen3-4B and Gemma-3-4B-it (ToM -13.95 and -19.55; TalkHier -11.42 and -4.50).

• Fine-grained rewards beat final-outcome feedback. Outcome-RL uses the same self-play environment and initialization but rewards only the final team outcome. CoSIR exceeds it by 20.72, 22.95, and 24.86 points for Qwen3-8B, Qwen3-4B, and Gemma-3-4B-it respectively, scoring higher on all seven tasks with Qwen3-8B and six tasks with each other model, tying on MathChat with Qwen3-4B, and losing on HiddenBench with Gemma-3-4B-it.

• Free-form transfer improves on SplitQA. SplitQA, introduced here, requires agents to combine facts from two separate HotpotQA supporting paragraphs to produce a free-form answer. CoSIR improves over Base by 2.50, 0.36, and 8.21 points for Qwen3-8B, Qwen3-4B, and Gemma-3-4B-it; the paper notes the Qwen3-4B improvement is small.

• Agents disclose more private evidence after training. On HiddenBench, private facts disclosed rose 68.5% to 74.3% (Qwen3-8B), 24.5% to 33.9% (Qwen3-4B), and 9.9% to 15.3% (Gemma-3-4B-it). Messages responding to teammates rose 59.4% to 65.8%, 64.0% to 73.4%, and 93.1% to 95.4%.

• Agents share more usable local data and fewer result-only messages. On SILO-BENCH, agents listing all their own data rose 55.3% to 65.4%, 74.8% to 78.4%, and 76.8% to 85.9%, while result-only messages fell 16.2% to 12.8%, 10.2% to 8.1%, and 10.6% to 9.3%.

• Supporting facts and correction attempts increase. On SplitQA, holders stating supporting facts rose 52.9% to 58.2%, 56.8% to 62.9%, and 33.6% to 35.7%. On MathChat, messages with correction language rose 2.5% to 11.1%, 4.9% to 6.2%, and 40.1% to 44.4%.

• Information is shared earlier. For Qwen3-8B, first-message disclosure rose from 61.1% to 69.0% on HiddenBench and from 41.1% to 51.1% on SplitQA.

• MAST failure analysis shows fewer coordination and verification failures. Pooled across the three models on HiddenBench, SplitQA, and MathChat, coordination failures fell from 67.8% to 62.0% and verification/termination failures from 58.8% to 53.5%. Ignoring teammates' input fell from 26.3% to 22.8%, proceeding without clarification from 36.4% to 32.0%, and no or incomplete verification from 51.3% to 45.9%. Specification and system-design failures were essentially unchanged (88.2% versus 88.3%), and repeated steps and information withholding did not improve (85.1% versus 85.6%; 18.5% versus 18.7%).

• General capabilities are largely preserved. Across GSM8K, MMLU, and IFEval, changes from Base range from -0.6 to +0.8 percentage points.

Methodology in Plain English

The researchers built a deliberately simple training game. Each instance has four options, five attributes, two to four agents, and a hidden table of ratings in {-1, 0, +1}; the team must pick the option with the highest total rating. Every agent sees only part of the table, so no single agent has enough information alone. Tables with tied highest totals are discarded.

Agents then discuss in natural language over a fixed number of rounds, making rating statements and requests. A shared parser (rules plus the frozen initial Qwen3-8B model) extracts ratings, totals, and requests from messages.

To score messages, the authors use a Bayesian evaluator with a uniform prior over all complete rating tables that have a unique best option. Given what an agent knows, the evaluator computes the posterior probability that each option is best. "Team surprisal" is the sum across agents of the negative log (base 2) probability assigned to the correct option — so it measures how unlikely the truth looks from the team's current knowledge. Sharing a rating is rewarded by how much it reduces team surprisal for the other agents; a request is rewarded by the reduction that would occur if a teammate truthfully supplied the requested rating, computed immediately without waiting for a reply.

Two safeguards matter. A statement only earns positive reward if it is both correct and supported by the speaker's own briefing or a received report — so lucky guesses and faithfully repeating false reports earn nothing, and fabricated statements get an extra penalty. Statements that are supported but temporarily raise surprisal are not penalized negatively; each statement carries a small communication cost.

Separately, private answers are scored: the choice score is the evaluator's probability that the selected option is best given the agent's own information, and the calculation score is the fraction of options whose reported totals match the reference totals. Unknown ratings count as zero.

Training uses self-play, where one model plays all agents. Multiple candidate responses are sampled per turn, scored, one is randomly selected to continue the discussion, and all contribute to training. Rewards are assigned to the specific tokens that produced statements, requests, or answers, and optimized with a variant of PPO. Early in training, when useful questions are rare, the authors inject a few demonstrations of the best available request, trained with a clipped imitation update but never selected to continue the discussion. A reference-model penalty and training on the initial checkpoint's responses to general prompts help retain general ability.

Why This Matters

Impact on research. The paper reframes multi-agent collaboration training as a credit-assignment problem, showing that a synthetic environment with verifiable facts can produce behavior rewards that transfer to open-ended natural-language tasks. It offers an alternative to outcome-only multi-agent RL and to inference-time scaffolding such as Theory-of-Mind prompting and hierarchical workflows, and it introduces SplitQA as a free-form test of combining distributed evidence.

Real-world applications:

• Enterprise decision support, where different teams or systems hold fragments of a rating, risk, or compliance picture and must reach a joint recommendation.
• Multi-agent question answering and research assistants, where one agent holds a supporting document and another must know to ask for it.
• Distributed data analysis and computation, where each agent sees only part of a dataset and the team must aggregate correctly rather than trust a pre-computed summary.
• Collaborative troubleshooting or customer-support pipelines, where identifying and correcting a teammate's error matters as much as producing an answer.

Industry relevance. The method uses open-weight models (Qwen3-4B, Qwen3-8B, Gemma-3-4B-it) and automatic numeric rewards, so no human labeling or learned reward judge is required for the training signal. However, the reward depends on a toy environment with known ground-truth facts and a computable evaluator, so direct industrial deployment would require a task whose information value can be computed.

Future Directions

• Extending the reward to open-ended tasks. The authors state explicitly that extending the scoring approach would require a way to assess information value when the complete state is unavailable.
• Reducing remaining failure modes. Repeated steps and information withholding did not improve, and several MAST failure categories remain frequent, leaving room for training signals that target procedure and specification failures rather than information exchange alone.
• Broadening the model and task suite. The evaluation covers three backbones and seven collaboration tasks; larger models, longer discussions, and additional domains are untested.
• Closing the gaps against alternatives. Inference-time workflows and Outcome-RL remain stronger on some model-and-task combinations, such as TalkHier on BenchForm and Outcome-RL on HiddenBench with Gemma-3-4B-it, which the paper leaves unresolved.

Target Audience

Researchers and practitioners working on LLM agents, multi-agent systems, and reinforcement learning post-training; engineers building collaborative agent pipelines who want behavior-level rather than outcome-level training signals; and readers interested in whether simple, verifiable synthetic environments can produce collaboration skills that generalize to harder, open-ended tasks.

Authors’ abstract

Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.