Moral Hazard in Multi-Agent Language Models

AI summary

Overview

Research area: Multi-agent systems and AI safety — specifically, incentive alignment in teams of language-model agents, drawing on contract theory (Holmström's moral hazard in teams) and reinforcement/optimization methods.

Technical level: Advanced. The paper combines formal game theory (Nash equilibria, marginal-incentive conditions), multi-agent textual environments, prompt optimization, and weight-level fine-tuning, with a heavy appendix apparatus.

One-sentence scope: The paper builds a controlled textual game that isolates costly, other-benefiting information acquisition in teams of language agents, measures which of fourteen open-weight and four frontier models follow the predicted private-incentive boundary, and introduces a mechanism-aligned prompt-optimization algorithm (CREDIT) that rewards causal contribution rather than query frequency.

What This Paper Is About

In team settings, an agent may be asked to do something that costs it tokens, latency, or reward while mainly benefiting someone else. The paper asks whether language-model agents will pay that cost — for example, spending a query to discover a hidden safety fact that helps a partner's downstream decision — when they alone bear the cost and receive little individual credit.

To study this, the author instantiates Holmström's classic moral hazard-in-teams structure as a finite textual game, the Dialogue Moral Hazard Game, in which an agent chooses between a local payoff and paying a query cost to reveal a hidden unsafe option in another agent's case. The goal is to separate the stages of the behavior — acquiring information, communicating it, using it downstream, and achieving team success — rather than treating aggregate success as the only measure.

Key Contributions

1. A controlled paradigm. The Dialogue Moral Hazard Game separates costly information acquisition, public communication, downstream use, and team outcome, and supports matched incentive and information-structure interventions across fourteen open-weight checkpoints and four frontier models.

2. Two information regimes. Under a fixed query mechanism, API-served policies can track the Holmström private-incentive boundary — GPT-5.6 Sol, Claude Opus 4.8, and Nemotron-3 Ultra within mean absolute errors of 0.013, 0.030, and 0.024 — while diagnostic adaptation exposes stage-specific preservation and repair, and outcome-directed GEPA can replace direct revelation with an observable predictor.

3. Mechanism-aligned optimization. Motivated by cue competition and blocking, CREDIT (Counterfactual Replay for Evidence-Driven Information Transfer) uses matched hidden-state twins and counterfactual net-VOI credit to score query trajectories by their marginal contribution rather than by aggregate success or query frequency.

4. Construct-validity instrumentation. The paper introduces rank–label interventions (primary, balanced, reversed) that share visible templates and sampling streams, letting the author freeze optimized prompts and test which information channel each learned policy depends on.

Main Findings

• The game works as an incentive probe. The game's reward is R_game = w_L·(1/n)ΣL_i + w_F·(1/n)ΣF_i + w_T·T − k·(1/n)ΣQ_i with primary values (w_L, w_F, w_T, k) = (0.35, 0.15, 0.50, 0.10). The coordinated-profile crossover is k^profile = 0.30, distinct from the unilateral all-query Nash cutoff k^NE = 0.15.

• Three frontier/API models track the derived boundary. In a private-share incentive-isolation experiment, the predicted threshold is α*(k) = (w_L + k)/(n·w_T) = 0.35 + k. GPT-5.6 Sol, Claude Opus 4.8, and Nemotron-3 Ultra follow it across nine query costs with mean absolute errors of 0.013, 0.030, and 0.024.

• Boundary tracking is sharp for Sol and Nemotron. Sol's pooled query rate moves from 0.0% immediately below indifference to 68.9% at the boundary and 100.0% immediately above. Opus shows rates of 14.8%, 100.0%, and 100.0%. Nemotron moves from 0/405 immediately below to 11/405 at indifference and 404/405 immediately above.

• Muse and Fable have no estimable boundary. Both remain above 50% querying throughout the sampled grid. Muse responds directionally, with a high-share-minus-low-share contrast of +23.1 points (95% bootstrap CI [18.8, 27.7]). Fable shows a +4.7-point share contrast ([2.7, 6.9]) and a −1.8-point cost contrast whose interval crosses zero ([−4.4, 0.4]).

• Autonomous cost sweeps differentiate models. Sol's query rate falls from 98.9 ± 1.9% at k=0.15 to 4.4 ± 1.9% at k=0.40 while local-reward preservation rises from 1.1 ± 1.9% to 94.4 ± 1.9%. Opus declines from 100.0 ± 0.0% at k=0.00 to 45.6 ± 9.6% at k=0.30 and 13.3 ± 3.3% at k=0.40, with local reward rising from 0.0 ± 0.0% to 86.7 ± 3.3% and format validity at 100%. Muse falls from 76.7 ± 5.8% to 3.3 ± 3.3% between k=0.00 and k=0.40, with team success falling from 62.2 ± 3.8% to 0.0 ± 0.0% and unsafe choice rising from 14.4 ± 1.9% to 36.7 ± 3.3%. Fable switches early: query falls from 55.6 ± 1.9% at k=0.00 to 5.6 ± 1.9% at k=0.15 and 0.0 ± 0.0% at k=0.40, while team success remains 0.0 ± 0.0% at every cost.

• Comprehension is not the bottleneck. Private-payoff comprehension is 100.0% for Sol, Muse, Fable, and Opus, and 99.7% for Nemotron. Yet Fable and Opus query in 97.9% and 46.3% of privately dominated decisions, respectively — consistent with safety-prioritizing policies rather than arithmetic mistakes.

• Base models fail at separable stages. Gemma-2-2B IT and OLMo-7B query frequently but realize only 9.2% and 11.2% information transfer. OLMo3.1-32B reaches 27.3 ± 6.9% team success through query-mediated transfer, whereas Qwen3.6-27B reaches 0.9 ± 0.4% despite comparable scale. Mistral NeMo 12B fails earlier, at 7.4 ± 0.9% format validity.

• Frontier base performance spans the full range. GPT-5.6 Sol records 100.0 ± 0.0 on team success, query rate, information transfer, and validity; Claude Opus 4.8 records 77.8 ± 15.4 team success and 100.0 ± 0.0 validity; Muse Spark 1.1 records 22.2 ± 3.8 team success and 51.1 ± 5.1 query. The matched Random control anchors chase at 11.7 ± 11.3 team success.

• Weight-level interventions produce stage-specific changes. For SmolLM3-3B, SFT lifts team success from 0.9 ± 0.8% to 64.7 ± 7.5% and information transfer from 3.7 ± 0.5% to 68.3 ± 11.0%; SFT+RLOO reaches the same 64.7% team-success mean while shrinking the across-seed standard deviation from 7.5 to 1.4 points, whereas standalone RLOO stays near base.

• Outcome-directed optimization can bypass the mechanism. GEPA raises Muse team success from 22.2% to 100.0% while reducing query use from 51.1% to 0.3%. Frozen-prompt interventions show this depends on a learned rank–label mapping rather than direct revelation: changing the mapping reduces team success from 100.0% to 12.5% and then 0.0%.

• CREDIT preserves query-mediated behavior. Across three optimization seeds each for Qwen3-4B IT, OLMo-7B, Gemma 4B, Muse Spark 1.1, and Claude Opus 4.8, CREDIT yields query-mediated policies that persist across all three mappings. Muse and Opus complete the pathway at ceiling, while downstream information use remains a bottleneck for the open-weight models.

• Scale does not determine success. Appendix K reports a descriptive parameter-scale analysis of fourteen open-weight checkpoints spanning 0.6B–55B active parameters under base and GEPA policies; team success is not monotonically ordered by size. SFT, RLOO, and SFT+RLOO cover thirteen, eleven, and twelve checkpoints respectively.

• Protocol execution is separable from cooperation. For OpenThinker3-7B and Qwen3.5-9B, a 32,768-token control sharply raises format validity without yielding any team success.

Methodology in Plain English

The author turns a classic economics problem into a playable text game for language models. Agents sit in a ring: each owns a "case" with several options, one of which is secretly unsafe. Only another agent can discover which option is unsafe, and only by spending a query cost. Play has four stages — observation, work, note, and final decision. At the work stage, the agent either takes the local payoff or pays to see the hidden fact. At the note stage, a querying agent may post a warning to a public board. At the final stage, everyone picks an action, and the team wins only if every agent avoids its unsafe option.

The reward is deliberately structured so querying is costly and the team-success component is all-or-nothing, giving no marginal credit to a single query. This creates the moral hazard wedge: querying can be socially worthwhile yet privately unattractive.

To test whether models respond to incentives rather than merely guessing, the author runs three lines of experiments. First, cost sweeps vary the query cost k and watch how query rate, local reward, unsafe choice, and team success move — comparing observed behavior against the mathematically derived crossover points. Second, a private-share experiment holds partner behavior fixed and varies the focal agent's share α of team output, isolating a predicted indifference boundary that can be checked per model. Third, rank–label interventions change the relationship between a case's visible utility rank and which option is hidden-unsafe (primary, balanced, reversed), which exposes whether a policy is really querying or instead exploiting a statistical shortcut.

For learning, the author applies supervised fine-tuning on scripted trajectories, RLOO policy gradients, both combined, and GEPA (natural-language prompt optimization against the game reward, via DSPy), with LoRA used for weight-level changes. CREDIT is then introduced as a distinct algorithm: it constructs matched "twins" of each episode that share identical visible inputs but differ in the hidden unsafe option, replays both query and no-query trajectories for each twin, and awards credit only when querying appears in every twin and beats the best local alternative in every hidden state. A four-stage curriculum (query discovery, communication, information use, joint optimization) sequences prompt learning so that credit can be assigned stage by stage before supports are removed.

Why This Matters

Impact on research. The paper argues that aggregate team success is a misleading optimization target: identical scores can arise from direct information revelation or from a learned predictive shortcut. The rank–label interventions and net-VOI credit assignment give the multi-agent evaluation community a template for mechanism-level rather than outcome-level assessment, and connect language-agent behavior to a well-developed economic theory of incentives.

Real-world applications:

• Auditing multi-agent LLM deployments where one agent is expected to verify claims, inspect plans, or flag risks that mainly protect a different agent or the whole system.
• Designing cost and credit-sharing schemes for tool-calling agents, so that verification steps are not systematically under-incentivized by all-or-nothing outcome rewards.
• Debugging pipelines that fail silently — the paper shows failures can localize to acquisition, transmission, or downstream use, each requiring different fixes.
• Evaluating whether a fine-tuned or prompt-optimized agent still performs a safety-relevant step, rather than merely producing the right final answer through a correlated cue.

Industry relevance. Any

Authors’ abstract

Cooperation can fail when socially valuable effort is costly, weakly observable, and mainly benefits others. Drawing on Holmström's team moral-hazard model, we introduce the Dialogue Moral Hazard Game, a controlled textual game that operationalizes this hidden-action structure for language agents. In each episode, an agent can preserve an immediate local reward or pay a query cost to reveal a hidden safety fact that primarily helps another agent's downstream decision. We evaluate seven open-weight language models and one frontier API model, decomposing behavior into query use, realized information transfer, local-reward preservation, unsafe choice, format validity, and team success. Base open-weight models commonly preserve local reward without team success or query without communicating information that changes the final decision. GPT-5.6 Sol reaches ceiling behavior in the primary setting, and autonomous sweeps respond strongly to query cost and team reward. In a 3,015-decision incentive-isolation experiment with scripted partners, its empirical query threshold tracks the Holmström-derived private-share boundary across nine query costs with mean absolute error 0.013. We then use supervised fine-tuning, RLOO, sequential SFT+RLOO, and GEPA prompt optimization as diagnostic update mechanisms where coverage permits. Their effects are heterogeneous: OLMo-7B shows the clearest mechanism-consistent weight-level improvement, whereas GEPA sometimes improves team success while reducing or eliminating costly queries. Thus, optimization can shift aggregate reward without recovering the intended cooperative mechanism, motivating evaluations that report mechanism-level behavior rather than team success alone.