DIANOIA: Diagnostic Decomposition and Joint Optimization for Multi-Agent Reasoning

AI summary

Overview

Research area: Multi-agent large language model (LLM) reasoning systems and inference-time compute allocation.

Technical level: Advanced. The paper combines a large empirical evaluation with an information-theoretic formalization (Bonferroni coverage bounds, Fano-type selection-error bounds, a potential-game convergence claim, and a subadditivity analysis).

Scope (one sentence): The paper proposes a three-channel diagnostic decomposition of where multi-agent reasoning gains come from, embeds it in a four-rule diagnostic protocol, and instantiates the prescriptions as a four-phase system called DIANOIA, evaluated on GSM8K, AIME-2025, MBPP, BFCL-SP, and VitaBench across Qwen, DeepSeek, Gemini, and Claude models.

What This Paper Is About

Multi-agent LLM systems often beat single-agent baselines, but practitioners cannot predict which design will work for a new task or explain why one fails. The authors argue this gap exists because the field lacks a diagnostic framework with measurable primitives and testable predictions. Their goal is twofold: provide a decomposition of multi-agent gain into three individually measurable channels, and build a single system (DIANOIA) that simultaneously acts on all three, validated with budget-matched comparisons and a prospective held-out test.

Key Contributions

1. A prescriptive diagnostic decomposition. Multi-agent reasoning gain is decomposed into coverage (exploration gain), information fidelity (information gain), and synthesis (aggregation gain), each with a measurable estimator. The decomposition places prior methods on a three-axis design map and yields a four-rule diagnostic protocol (R1–R4) that forms qualitative bottleneck hypotheses for tasks with measurable quality signals.

2. DIANOIA, a four-phase system. The phases Propose, Execute, Review, and Synthesize mirror the three channels. The authors state that each individual ingredient appears in prior work; their contribution is combining them under a single diagnostic framework and evaluating their interaction. Phase-level and role-level ablations, plus a Best-of-3 control, show that no tested proper subset recovers the full gain.

3. Budget-matched and cross-model evaluation. Comparisons on GSM8K, MBPP, and BFCL-SP, a small-N AIME-2025 stress test, a prospective held-out VitaBench evaluation, and full-scale MBPP replications across Qwen, DeepSeek, and Gemini. A Pareto-frontier sweep on MBPP (39 configurations) shows DIANOIA matches the strongest baseline with roughly 5x fewer tokens.

4. Formal scaffolding with explicitly stated scope. An exact definitional identity (Eq. 2), a conditional subadditive upper bound (Eq. 3), and idealized bounds (Propositions 3.2–3.4, Theorem 3.5). The authors explicitly separate which claims are tautological accounting, which hold only under stated verifier and independence conditions, and which are idealizations that "describe best-case trends, not calibrated scaling laws."

Main Findings

• Headline benchmark results: DIANOIA achieves 91.1% on GSM8K, 93.3% on the complete 30-problem AIME-2025 set, 84.6% on MBPP, and 92.3% on BFCL-SP (all with reported 95% confidence intervals). It improves over the strongest multi-agent baseline by +1.3pp (GSM8K), +6.6pp (MBPP), and +3.5pp (BFCL-SP).

• MBPP Pareto efficiency: Self-Consistency, Two Heads, and ReConcile plateau below 78% at every token budget tested. MoA reaches 84.2% only by stacking three layers at 7.73M tokens, while DIANOIA reaches 84.6% at 1.54M tokens — approximately a 5x token-budget advantage at matched accuracy — and continues to 88.8% at 7.36M tokens.

• Aggregation gain is isolated empirically: The Best-of-3 control (same verifier, no multi-agent structure) reaches 78.4% on MBPP, 6.2pp below DIANOIA. AgentCoder reaches 81.2% on MBPP, 3.4pp below DIANOIA on Qwen3-30B-A3B.

• Joint optimization beats partial optimization (MBPP): Baseline (K=1,R=0,S=0) 76.0%; Explore-only 81.2%; Info-only 79.8%; Aggr-only 76.8%; two-dimension 83.6%; DIANOIA-full 84.6%.

• AIME-2025 is a stress test, not a superiority claim: An exploratory paired McNemar exact test against ReConcile yields p=0.039 from only nine discordant pairs; the paper states the wide confidence interval precludes a precise effect-size or model-superiority claim. Self-Consistency degrades on AIME-2025 (−13.3pp), matching the predicted voting-amplification regime.

• Measured channel diagnostics: pairwise proposer correlation ρ̄ = −0.15; reviewer error 5.8% on MBPP (execution) vs. 17.5% on GSM8K (pseudo-verifier); GSM8K pseudo-verifier precision 94.0% / recall 86.6%; synthesis breakdown 16.4% / 78.1% / 5.5% across selection / selection-with-repair / rescue.

• Subadditivity check: the synergy coefficient γ = Δ_joint / Σ Δ_i is 0.88 on MBPP and 0.852 on BFCL-SP, consistent with the predicted subadditive regime. The authors stress γ is computed after the runs, is not held out, and is not used to define or validate R1–R4.

• Idealized theory overshoots measured scaling: increasing synthesis iterations from S=1 to 2 to 3 changes MBPP accuracy from 83.6% to 84.0% to 84.6%; increasing reviewers from R=1 to R=3 changes accuracy from 84.6% to 85.8% at approximately 2.6x the token cost.

• Role diversity as a principle, not a specific prompt: any structured role design beats same-prompt plus temperature diversity (83.2%), but within the structured family confidence intervals overlap (three-role triplet 84.6%, two-role subsets 81.4–84.4%, alternative triplet 83.4%).

• Cross-model replication: on Qwen3.6-35B-A3B, DIANOIA leads on every cell at point-estimate level (MBPP 88.8%, BFCL-SP 93.0%, GSM8K 91.0%). Full-scale MBPP (N=500) results: DeepSeek-V4-Flash 88.8% single / 97.0% Best-of-3 / 98.2% DIANOIA; Gemini-3.5-Flash 93.0% single / 93.8% Best-of-3 / 96.8% DIANOIA.

• Best-of-N behavior is predicted, not just measured: on execution-grounded tasks Best-of-3's gap over the single model widens with a stronger base (+2.4 to +4.8pp MBPP), but on pseudo-verified GSM8K the gap inverts to −0.4pp, consistent with an information-ceiling bottleneck.

• Prospective held-out diagnostic check: on VitaBench (N=400), the bottleneck hypothesis was recorded before running DIANOIA; pass rate improved from 29.5% to 45.0% (+15.5pp). VitaBench uses Qwen3.5-397B-A17B rather than the headline base model, which the authors flag as an explicit scope caveat.

• Cross-host realization: the same K=3, S=3, three-role prescriptions also realize as a Claude Code skill with Claude Opus 4.7, yielding +6.7pp on BigCodeBench-Hard (33.3% to 40.0%, N=30) and Hard MBPP (80.0% to 86.7%, N=15); the small N precludes tight confidence intervals.

• Robustness to corrupted evidence: a controlled GSM8K stress test keeps accuracy within 91–93% through 25% injected evidence corruption, falling to 88% at 30%.

• Diagnostic protocol predictions held within-study: P1 (largest gains on execution-grounded MBPP/BFCL-SP, +6.6/+3.5pp), P2 (smallest gain on high-p GSM8K, +1.3pp), P3 (Self-Consistency degrades on AIME-2025, −13.3pp), P4 (DIANOIA exceeds Best-of-3 by +6.2pp on MBPP). The authors characterize these as within-study consistency checks, not calibrated theorem derivations.

Methodology in Plain English

The authors start from a simple accounting idea: the improvement a multi-agent system gets over a single model can be split into three conceptually distinct sources. Coverage is the benefit of seeing more — running several differently-flavored proposers so at least one finds a correct path. Information is the benefit of seeing clearly — having a reliable signal, such as actually executing the code, that distinguishes right answers from wrong ones. Synthesis is the benefit of deciding wisely — combining proposals and feedback into a final answer instead of just picking one.

They formalize this as an exact identity (final quality equals coverage times effective efficiency) plus a definitional attribution of each gain term, and then a conditional subadditive upper bound: the total gain is generally less than the sum of the three parts because the parts interact multiplicatively. They are explicit that the attribution is not unique and that they chose it because each term maps to a design lever and an empirical estimator.

They then build the system to match: three role-diverse proposers (Minimalist, Skeptic, Explorer) generating candidates in parallel; an execution phase using the highest-fidelity feedback available (sandboxed execution for code, LLM-based pseudo-verification for math); a review phase where reviewers convert raw execution results into diagnoses (why it failed, which parts passed, how to fix); and a synthesis phase with iterative refinement and closed-loop validation, including trajectory grafting and re-execution.

To test whether this matters, they sweep each baseline along its own native scaling knobs to build accuracy-vs-token Pareto frontiers, run ablation tables that switch individual channels on and off, and hold out VitaBench — recording the predicted bottleneck channel before running the system. The default configuration is K=3 proposers, R=1 reviewer, S=3 synthesis iterations, with all multi-agent methods using Qwen3-30B-A3B-Instruct-2507 in thinking-disabled zero-shot mode.

Why This Matters

Impact on research. The paper shifts the framing of multi-agent design from "which architecture wins" toward "which channel is the bottleneck, and how should tokens be allocated to it." It provides measurable estimators (ρ̄, η*(s), the aggregation gap, and γ as a post-run check) so that designers can test components before committing, and it gives a common axis for comparing systems that instantiate the prescriptions differently. The paper is also unusually explicit about the boundaries between exact identities, conditional bounds, and idealizations, which is a useful methodological template.

Real-world applications:

• Code generation and automated repair. MBPP and the Claude Code skill results point to deployment in coding assistants where execution serves as a deterministic verifier and reviewer diagnostics can drive repair rather than mere selection.
• Function calling and API/tool orchestration. BFCL-SP uses schema validation as a deterministic verifier; DIANOIA's 92.3% suggests value in agentic pipelines where tool-call correctness is machine-checkable.
• Mathematical problem solving and tutoring. GSM8K and AIME-2025 show gains persist under weaker pseudo-verifiers, though bounded by verifier fidelity (94.0% precision / 86.6% recall).
• Long-horizon interactive agents. The VitaBench result (29.5% to 45.0% pass rate) is early evidence for open-world, multi-step interactive tasks, where the paper predicts information fidelity is the primary bounded channel.

Industry relevance. The central practical message is cost: matching the strongest baseline's MBPP accuracy at roughly 5x fewer tokens, and dominating the measured accuracy–token Pareto frontier, converts a research question into a serving-cost question. The framework's explicit statement that wall-clock latency is serving-dependent, and that the aggregation margin contracts as systems approach the execution ceiling, is directly relevant to teams choosing where to spend inference budget.

Future Directions

• Extending beyond tasks with reliable quality signals. The paper states that for open-ended generation, dialogue, or culturally grounded reasoning, reliable quality signals are often unavailable, so the practical value of the information channel is substantially reduced and the framework becomes correspondingly less prescriptive. (The paper's limitations section is truncated in the provided content beyond this point.)

• Closing the gap between idealized bounds and measured scaling. The exponential-tightening predictions under independent reviewers and synthesis iterations did not match the observed magnitudes (83.6% to 84.6% across S=1 to 3). Understanding residual reviewer correlation (ρ_R > 0), which the paper models as flooring error at ϵ_∞ := ϵ_0 ρ_R + ϵ_0^{K−1}(1 − ρ_R), is an open question.

• Strictly matched-base prospective evaluation. VitaBench used Qwen3.5-397B-A17B rather than the headline Qwen3-30B-A3B because the latter produces a near-floor operating point. A matched-base prospective test on a hard open-world benchmark would strengthen the diagnostic protocol's external validity beyond the current "empirical boundary evidence."

• Determining how much of the protocol is model-family-invariant. Replications span Qwen, DeepSeek, Gemini, and Claude, but the direct DIANOIA-vs-AgentCoder margin contracts from 3.4pp on Qwen3-30B-A3B to 0.4pp on Qwen3.6-35B-A3B (within CI overlap). Whether the channel prescriptions remain useful as base models saturate verifiable-task ceilings is unresolved.

Target Audience

Researchers and advanced practitioners in multi-agent LLM systems, inference-time scaling, and test-time compute allocation. It will be most valuable to (a) readers who need a vocabulary and measurable primitives for diagnosing why a multi-agent pipeline underperforms, (b) engineers building code-generation, tool-calling, or agentic assistant products who need to reason about token budgets versus accuracy, and (c) theoretically inclined readers interested in how ensemble-learning and verifier-fidelity intuitions transfer to the test-time multi-agent setting. Readers looking for a purely empirical benchmark report will find the paper's heavy formal scaffolding and explicit scope caveats demanding; readers without familiarity with information-theoretic bounds (Fano-type, Bonferroni) will need to treat Sections 3 and the appendices as reference material rather than prerequisites.

Authors’ abstract

Multi-agent LLM systems consistently outperform single-agent baselines, yet practitioners still cannot predict which design works for a new task or diagnose why one fails. We argue this gap persists largely because the field lacks a diagnostic framework with measurable primitives and testable predictions. We introduce \textbf{DIANOIA}, a three-channel decomposition of multi-agent reasoning gain into coverage, fidelity, and synthesis, each of which is empirically measurable. From this decomposition, we derive a diagnostic protocol that identifies the bottleneck channels for any given task. We instantiate the protocol as a multi-agent system whose three components mirror the channels: role-diverse proposers for coverage, execution-grounded verification for fidelity, and iterative synthesis. On GSM8K, AIME-2025, MBPP, and BFCL-SP, our method outperforms strong multi-agent baselines under matched token budgets, dominating the Pareto frontier on MBPP at $\sim$$5{\times}$ token savings and reaching $+4.6$pp at matched cost. On every benchmark, the protocol picks the right bottleneck channels; the system we built around it leads across models. We release code, adapters, diagnostic metrics, and a Claude Code skill at https://anonymous.4open.science/r/DIANOIA4MAS. DIANOIA reframes multi-agent design as channel-aware resource allocation: diagnose which channel is the bottleneck for your task, then invest tokens accordingly.