Measuring Collapse and Correction in Homogeneous-Panel LLM Debate

AI summary

Overview

Research area: Natural Language Processing / multi-agent large language model evaluation, specifically the measurement of multi-agent debate on multiple-choice question (MCQ) benchmarks.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting, benchmark accuracy metrics, and basic rank-correlation statistics (Spearman rho, AUC, confidence intervals), but its central argument is conceptual rather than architectural.

One-sentence scope: The paper proposes an auditable "transition ledger" for homogeneous three-agent LLM debate that separates collapses (correct majority becomes wrong) from corrections (wrong majority becomes correct), and uses held-out replay to show that a freeze policy which prevents collapses can still be net-harmful because of the corrections it discards.

What This Paper Is About

Multi-agent LLM debate is usually judged by whether final answers improve, but final accuracy hides two opposite events: a discussion can rescue an initially wrong majority (a correction) or destroy an initially correct one (a collapse). The authors build a measurement protocol that records every run as a transition over preserved, collapse, correction, and unrepaired outcomes, then scores proposed interventions by both the collapses they prevent and the corrections they lose. Their goal is not a new debate system but an evaluation target that other labs can reuse to compare model–scaffold rows under the same denominators and a signed utility ledger.

Key Contributions

1. A reusable audit protocol for multi-agent LLM debate that decomposes final accuracy into preserved, collapse, correction, and unrepaired transitions, with collapse onset and weighted signed utility.
2. A signed-replay ledger that scores interventions as (prevented, lost, net) under declared weights, together with parser audits, cost cards, release flags, and zero-API rebuild scripts.
3. An MMLU-Pro case study showing that a compact 8-probe compliance screen is a triage signal with limited marginal value over capability pressure, and that freeze-style policies can prevent collapses while losing more corrections (29 prevented vs. 108 lost under equal weights), for a net of -79.
4. Round-level trace localization showing that most tracked collapses originate in the first debate round, and that Round 1 features improve out-of-fold collapse prediction from AUC 0.669 to 0.768 (and to 0.846 after adding Rounds 2 and 3).

Main Findings

• Movement is not improvement: Sonnet 4.5 and Llama-3.1-8B have nearly identical debate flip rates (0.705 vs. 0.726), yet Llama's conditional collapse rate is about four times higher (8.46% vs. 2.15%). The authors state the missing variable is direction, not motion.
• Collapse is common in this setting: Across 6,925 MMLU-Pro debates, the protocol identifies 253 collapses. For the most susceptible models, the paper reports that more than one in ten initially correct majorities are lost.
• The pre-debate screen works as triage, not as a predictor: The 8-probe total flip rate alpha_tot shows a high unadjusted family-level association with conditional-collapse risk (G=7, Spearman rho = +0.8929, exact two-sided p = 0.0123). But initial-majority accuracy is a close free comparator (rho = +0.821 at the family level), leaving a marginal rank gain of about delta rho = 0.07, and the family-level partial after initial-majority accuracy (rho = +0.767, p = 0.0877) is described as no longer confirmatory. The authors explicitly do not treat the screen as calibrated, capability-adjusted, or a per-question oracle.
• The screen is most useful when it disagrees with capability: Capability pressure ranks Qwen3-4B above Llama-3.1-8B in risk because their initial-majority accuracies are 45.3% and 50.7%; the probe reverses this order (alpha_tot = 0.774 vs. 0.607), matching the observed conditional-collapse ordering (8.46% vs. 5.11%). A post-hoc audit of all 91 model-row pairs finds the screen reorders 26 pairs relative to initial accuracy, and observed conditional collapse follows the screen in 13 cases and initial accuracy in 12 (one tie).
• Collapse is usually an early event: Of the 253 collapses, onset is Round 1 for 149 (58.9%), Round 2 for 47 (18.6%), and Round 3 for 57 (22.5%). The paper notes Round 3 exceeding Round 2 is not a contradiction, since onset is the first correct-to-wrong majority switch.
• Round 1 traces add predictive power: Pooled out-of-fold AUC rises from 0.669 to 0.768 after adding Round 1 trajectory features (delta = +0.099, 95% bootstrap CI [+0.080, +0.118]), and to 0.846 after adding Round 2/Round 3 features (delta = +0.077, 95% CI [+0.064, +0.092]). A four-round trace-schema replication agrees, with Round 1 the largest bucket at 40/75 collapses.
• Freeze policies are net-negative at equal weights: A leave-one-model-out probe-gated freeze over 6,525 debates prevents 29 of 251 collapses and loses 108 of 804 corrections, net -79, with a break-even ratio of 3.72. A Round 1 majority-change rule over 1,255 traces prevents 56 and loses 69, net -13 (break-even 1.23). A learned Round 1 stump prevents 2 and loses 31, net -29 (break-even 15.50). The paper notes these rows are not a matched comparison because the cohorts differ.
• Weighting changes the winner: At a collapse weight of r = 2, the probe-gated freeze is still net-negative (-0.77 percentage points) while the Round 1 majority-change rule turns positive (+3.43 percentage points).
• Probe channels are behaviorally separable but uneven: Anti-argument and anti-social system prompts elicit different flip distributions in all 16/16 tested models, with grand mean Cohen's d = 0.53 and 12/16 paired tests surviving Holm–Bonferroni correction. Qwen3-4B (d = 0.092) and Qwen3-8B (d = 0.057) are near-zero separability cases, and the authors treat separability as a diagnostic check rather than a uniform mechanism claim.
• Per-question diagnostics favor disagreement: Once initial answers exist, initial disagreement is a stronger runtime diagnostic than probe-derived quantities; adding the S/A ratio changes AUC by at most 0.004. Mean probe FR ranges 0.455–0.735, S/A 0.394–0.576, and initial disagreement 0.550–0.858 across the reported model columns.
• Transfer checks are about accounting, not screen transfer: On GPQA (N = 198), Mistral Small 4 is nearly accuracy-neutral because collapses (20) and corrections (21) almost cancel, while DeepSeek V4 Flash gains despite nonzero collapse because corrections (47) dominate collapses (10). Excluding four Mistral questions lacking a valid initial majority changes its signed net from +1 to -1.
• Private revision is weaker than peer debate in the follow-up: Across 12 repeated MCQ settings, private revision is 1.44 to 4.81 points less accurate than peer debate when ties count as wrong; crediting ties at their expected value narrows the difference to between -1.62 and +0.38 points. In eleven reasoning-mode rows, collapses and corrections both persist and private revision has a negative point difference in all eleven (intervals below zero in seven). The authors state this bounds, rather than settles, the comparison with compute-matched solo reasoning.
• Probe reliability is measurable: The high-FR item selection is outcome-blind; split-half reliability is 0.873, test–retest reliability is 0.760, and inter-agent ICC is 0.630–0.633. A 195-question shared-pool sensitivity keeps S/A positive in 13/15 models, and full 1,000-question estimates remain inside the per-model bootstrap intervals.

Methodology in Plain English

The authors study a deliberately narrow setting: homogeneous, closed-book debate among three copies of the same model answering multiple-choice questions. Three agents answer independently (Round 0), then discuss and update over Rounds 1–3 while seeing all current answers, and the final majority becomes the group response. A collapse is a correct Round 0 majority that becomes a wrong Round 3 majority; a correction is the reverse; onset is the first round where a correct running majority becomes wrong. All labels are rule-based with no LLM judge or human coder — the parser prefers an explicit "Final Answer: X" pattern, then answer/choose variants, then standalone option letters, then a last-valid-letter fallback, and majority votes filter unparsed answers and break ties alphabetically.

Before running the debate sweep, each item is re-asked under an 8-probe battery built from a 4x2 design: four argument strengths (weak, moderate, strong, very strong counterarguments) crossed with two social-channel settings (solo counterargument vs. the same cue attributed to two other agents). The total flip rate alpha_tot becomes the pre-debate screening measurement, with separate adversarial (alpha_adv) and corrective (alpha_cor) channels conditioned on initial correctness.

For interventions, the authors use signed replay. A fired freeze returns the initial majority instead of the saved final majority, so it prevents a collapse when the initial majority was correct but loses a correction when debate would have recovered. Utility is stated as prevented collapses weighted against lost corrections, and the break-even ratio is the smallest collapse-to-correction weight ratio at which net utility is non-negative. The probe-gated freeze is tuned on held-in models and then evaluated frozen on a held-out model (leave-one-model-out). The analysis aggregates correlated model variants into seven family means (Anthropic, DeepSeek, Google, Meta, OpenAI, Phi, Qwen) for the primary exact Spearman test, keeping the 14-model-row view as sensitivity; the pre-registration had targeted 18 model rows.

Why This Matters

Impact on research. The paper argues that standard final-accuracy evaluations conflate two opposing mechanisms and therefore cannot distinguish a scaffold that helps from one that merely moves answers around. Its reusable object is not a tuned controller but a row schema, rule-based coder, parser/stability audits, cost accounting, release flags, and zero-API rebuilds at https://lixin.ai/DebateLedger, so a new model–scaffold row can recover the question pool, parser failures, initial-majority denominators, transition counts, and each gate's prevented/lost/net cells under the same definitions. The negative intervention rows are presented as calibration baselines for the ledger rather than as competing controllers.

Real-world applications (as implications of the proposed evaluation surface, which the paper does not itself deploy):

• Auditing debate or multi-agent scaffolds before deployment by checking whether accuracy gains come from genuine recoveries or from cascades that happen to net out.
• Allocating expensive trace logging and human review to model–scaffold pairs flagged as high-risk by a cheap pre-debate screen.
• Deciding when to intervene during a conversation (freeze, abstain, or defer) using round-level traces rather than final answers. The paper's own replay results caution that early disagreement signals both harmful cascades and useful recoveries.
• Comparing judge–debater, abstention, or deferral systems on shared transition cells instead of incompatible headline accuracies when the user's stated utility weights change.

Industry relevance. The results are a caution against optimizing for collapse prevention alone: under equal weights the probe-gated freeze lost over three times as many corrections as collapses it prevented (108 lost vs. 29 prevented). The round-level finding that many collapses lock in during the first debate round, combined with the finding that the same early disagreement can precede useful recovery, suggests that runtime controllers need correction-preserving uncertainty estimates, prespecified weights, and held-out signed replay before being trusted. The paper also reports a concrete artifact boundary: aggregate selection and accounting outputs are open, while tuned convince-wrong prompt text, transcript-derived feature matrices, and strict matched gate matrices remain gated, and provider reruns may drift.

Future Directions

• Building controllers that act on round-level risk without erasing recoveries, since the round-level diagnostics localized risk (AUC 0.669 to 0.768 after Round 1 features) but did not translate into a net-positive freeze policy.
• Establishing whether the transition accounting transfers beyond the primary benchmark. The GPQA, ARC, TruthfulQA, and OpenRouter/Gemini rows are described as post-hoc stress checks, and mixed-model panels and GSM8K rows added after review are feasibility checks rather than transfer evidence.
• Testing whether debate outperforms a compute-matched solo reasoning baseline. The follow-up finds collapses and corrections in every reasoning-mode row and private revision less accurate than peer debate when ties count as wrong, but the authors state it does not establish this comparison.
• Addressing the small, uneven realized cohort (14 model rows across 7 families instead of the pre-registered 18) with held-out-family diagnostics, and extending the protocol to settings the paper explicitly does not cover: heterogeneous judge–debater systems, tool use, retrieval, free-form generation, rubric- or judge-scored answers, and agent workflows that gather evidence over longer contexts.

Target Audience

Researchers and engineers who evaluate or deploy multi-agent LLM debate, ensemble voting, or aggregation scaffolds; benchmark and evaluation-methodology researchers interested in metrics that separate helpful from harmful answer revision; and practitioners designing intervention, abstention, or early-stopping policies who need to justify a controller against the corrections it discards. Readers looking for a new debate architecture, a deployable controller, or benchmark-transfer evidence will not find those claims here, as the authors explicitly disclaim them.

Authors’ abstract

Multi-agent large language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On 6,925 MMLU-Pro debates, the protocol identifies 253 collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents 29 collapses but loses 108 corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate 8-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high (G=7, Spearman rho=0.893, exact two-sided p=0.0123), but initial-majority accuracy is a close comparator (rho=0.821; family partial rho=0.767, p=0.0877), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model-scaffold rows can be compared under the same denominators and signed utility ledger.