MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination

AI summary

Overview

Research area: Confidence calibration and coordination among heterogeneous foundation models (multi-agent / multi-model systems), with an online-learning formulation.

Technical level: Advanced. The paper is empirically heavy, includes a formal estimator theorem, and assumes familiarity with calibration metrics (ECE), exponential moving averages, and multi-model evaluation protocols.

Scope: One sentence: the paper introduces MARGIN, a runtime calibration method that learns per-model, per-confidence-band corrections to self-reported confidence from observed correctness outcomes, and evaluates it against five online calibration baselines on code generation, question answering, and mathematics tasks.

What This Paper Is About

When a coordinator has to pick or weight answers produced by several different foundation models, it relies on each model's self-reported confidence. The problem is that "90% confident" can mean something entirely different for one model than for another, and the meaning can also drift when the workload changes, even though the models themselves are unchanged. MARGIN's goal is to learn model-specific corrections to those confidence scores at runtime, using only observed answer outcomes, so that a coordinator's answer-selection weights become more trustworthy as the workload evolves.

Key Contributions

1. A confidence-band runtime calibration mechanism. MARGIN maintains exponentially weighted estimates of accuracy and stated confidence for each (model, confidence-band) pair, corrects reported confidence by their ratio, blends sparse bands toward a model-level estimate, and uses the corrected scores to weight answers in a collective vote. No model retraining, no model internals, no held-out calibration set.

2. Comparative evidence on adaptation after task-distribution change. Five online calibration baselines (sliding-window histogram, decayed histogram, windowed accuracy in replace and multiply modes, and online Platt scaling) receive identical feedback and retain their own learned state across each transition. MARGIN achieves lower post-shift expected calibration error (ECE) than all five in two code-generation transitions and than four of five in a question-answering transition; the remaining question-answering comparison (online Platt) is inconclusive.

3. An empirical diagnosis of confidence inversion on BigCodeBench, including a model-mean confidence-versus-accuracy association, pairwise resolution of correct vs. incorrect response pairs, answer selection, and a selected-model-feedback diagnostic that tests dependence on observing outcomes for the whole responding pool.

4. A stationary estimator analysis (Theorem 1) showing that an asymmetric exponentially weighted moving average — using different learning rates after successes and failures — is asymptotically biased under stationary i.i.d. Bernoulli outcomes, and is unbiased only when the two rates are equal.

Main Findings

• Confidence can be inverted across a heterogeneous pool. On BigCodeBench, across the 18-model pool, mean stated confidence is negatively correlated with observed pass rate (r = −0.692, 95% interval [−0.759, −0.514]). Raw pairwise resolution is 42.37% with interval [38.99, 45.58], i.e. below the 50% random-choice reference. The paper states that among correct/incorrect response pairs, choosing the more confident responder performs below chance.

• MARGIN adapts better than the online baselines after coding workload changes. On HumanEval → BigCodeBench, MARGIN reaches ECE 12.99 versus 15.32 for online Platt (the lowest-ECE alternative on that row), a 2.33 percentage-point reduction. On MBPP → BigCodeBench, MARGIN reaches 11.89 versus 13.42 for online Platt, a 1.53 percentage-point reduction. Differences from the windowed estimators are larger, and every displayed coding-comparison interval lies below zero.

• The advantage extends to a question-answering benchmark change. On MMLU (STEM) → MMLU-Pro (business), MARGIN reaches ECE 2.85, compared with 6.87–7.37 for the two histogram and two windowed-accuracy baselines, all four paired comparisons favouring MARGIN. Online Platt reaches 3.17, and that comparison is inconclusive.

• Improvements over raw confidence are large but not by themselves discriminating. Against uncalibrated confidence, MARGIN lowers ECE in all six transitions: HE → BCB (66.79 to 12.99), MBPP → BCB (66.79 to 11.89), MMLU-shift (18.70 to 2.85), GSM8K → MATH (26.16 to 3.45), HE → HE+ (8.57 to 3.57), and MBPP → MBPP+ (17.96 to 3.07). The paper notes this comparison alone does not distinguish MARGIN from other online methods.

• Some comparisons are inconclusive. On the GSM8K → MATH transition, the pointwise interval favours MARGIN against online Platt, but the other four comparisons are inconclusive. All five comparisons on each of the two same-task hardening checks (HE → HE+ and MBPP → MBPP+) are inconclusive.

• Calibration improves coordination in cold-start code-generation experiments. Using the 18-model pool with no prior history: on HumanEval, pairwise resolution rises from 46.53% to 81.66% (+35.13); on MBPP from 49.94% to 81.25% (+31.31); on BigCodeBench from 42.37% to 63.36% (+20.99), each with positive paired difference intervals. ECE falls from 12.80 to 2.02, 17.51 to 1.90, and 72.72 to 13.04 respectively.

• Answer-selection accuracy improves on two of three benchmarks. MBPP selection rises from 87.94% to 92.26% (+4.32, interval [+1.87, +6.96]) and BigCodeBench from 13.89% to 27.87% (+13.98, interval [+7.90, +19.97]). The abstract describes these as increases of 4.3 and 14.0 percentage points on two of three benchmarks. The HumanEval selection difference is inconclusive (+0.82).

• Tie handling materially affects the raw baseline. Raw confidence leaves the leading answer tied on 62.2%, 59.9%, and 78.5% of HumanEval, MBPP, and BigCodeBench tasks. With ties resolved uniformly in expectation, the selection differences become 8.79, 10.60, and 12.75 percentage points, each with a positive paired interval.

• Asymmetric success/failure updates bias the estimate. Theorem 1 gives the steady-state expectation α_up·θ / (α_up·θ + α_down·(1−θ)) for a two-rate EWMA, which equals θ if and only if α_up = α_down; otherwise the asymptotic bias is θ(1−θ)|α_up − α_down| / (α_up·θ + α_down·(1−θ)). This motivates MARGIN's symmetric update rate.

Methodology in Plain English

The coordinator sees a task, a set of answers from a pool of models, and each model's stated confidence. It splits the confidence range [0, 1] into K = 3 equal-width bands and, for each model and band, keeps two running averages: how often that model was actually right in that band, and what confidence it typically claimed in that band. Dividing the first by the second gives a correction factor — if a model claims 0.90 confidence but is right only 0.60 of the time in that band, the factor is about 0.67, and a new 0.90 claim becomes roughly 0.60.

The running averages are exponentially weighted moving averages with learning rate α = 0.04, so older outcomes fade with roughly a 1/α observation memory. Because a band may have few observations, the band-level factor is shrunk toward a model-level factor using a credibility weight with k_s = 100; an empty band borrows only from the same model's other bands, never another model's history. Corrected confidences are clipped to [0, 1] and used as vote weights: each distinct answer accumulates the corrected confidence of the models that produced it, and the highest-scoring answer group is selected. For code, answers are grouped by an exact-code signature (line endings normalised, leading/trailing newlines stripped, indentation and internal whitespace preserved) rather than reference-string matching.

Prediction and feedback are separated in an event-driven procedure: candidates are scored with the state available at that moment, the selected answer is returned, and the confidence and band are stored; only when the correctness label arrives are the stored records updated. The current task's outcome never influences its own scoring. Unknown or quarantined labels are discarded rather than recoded as failures.

Evaluation design: shift experiments give every online method the same source observations and let each carry its own state into the target workload without reset; cold-start experiments initialise each method directly on the target benchmark. Baselines run with fixed configurations (sliding-window histogram with W = 200 observations and ten equal-width bins; decayed histogram with decay 0.98; windowed accuracy over the last 200 observations; online Platt with learning rate 0.1 and L2 coefficient 10⁻⁴). ECE uses ten equal-width bins, distinct from MARGIN's three state bands, and is pooled across responders and reported on a 0–100 scale. Metrics aggregate 100 recorded shuffle histories, and paired 95% intervals use 10,000 problem-cluster bootstrap draws.

The workload set: HumanEval (164 tasks), MBPP (257 tasks), and the hard subset of BigCodeBench (144 of 148 tasks, after four were quarantined). Transitions are HE → BCB (164 → 144), MBPP → BCB (257 → 144), MMLU STEM → MMLU-Pro business (189 → 200), GSM8K → MATH (200 → 196), HE → HE+ (164 → 163), and MBPP → MBPP+ (257 → 224). The canonical pool has 18 models, split between nine cloud-served models from the Qwen, DeepSeek, GPT-OSS, MiniMax, and GLM families and nine locally served Qwen, LLaMA, Phi, DeepSeek Coder, and Mistral variants; Gemma-4-31B-it was collected but excluded because mixing model generations would conflate capability with era. Shift comparisons use a fixed nine-cloud subset.

Why This Matters

The work reframes miscalibration as a coordination problem rather than an individual-model diagnostic: if a coordinator uses raw confidence to weight answers, an inverted confidence signal can make the ensemble worse than random choice among tied candidates, as the BigCodeBench diagnosis shows. It also argues that corrections must be revisable, since a correction learned on one task distribution may not transfer when the questions change while the models stay fixed.

Real-world applications

• Model routing and ensemble selection inside production assistants, where several vendor models answer the same prompt and one answer must be chosen.
• Code assistants that can run tests: executable outcomes supply the correctness feedback MARGIN needs, and the paper's code benchmarks are evaluated this way.
• Question-answering systems whose query mix drifts (for example from technical to business topics), where static post-hoc calibration fitted earlier may degrade.
• Enterprise multi-model orchestration platforms that want better calibrated confidence without retraining or acquiring a separate calibration dataset for every model.

Industry relevance: The author is affiliated with Ericsson, Athlone, Ireland, and the method is designed for a concrete operational constraint: a coordinator that sees only answers, stated confidence, and later correctness feedback, with no access to model weights, logits, or training data. That makes it deployable as a coordination layer over third-party model APIs, where changing workloads and heterogeneous responders are the norm.

Future Directions

• Reduce feedback dependence. The primary experiments assume correctness feedback for every usable response; the selected-model-feedback diagnostic in Section 7.5 tests the case where only the highest-calibrated-confidence responder receives an update. How well MARGIN performs under partial or noisy feedback is a natural extension, and the truncated paper content does not report those results here.

• Extend beyond binary correctness and single-turn answering. The paper states that multiclass probability calibration would require an explicitly defined class-specific construction and is not evaluated, and that interactive deliberation, planning, strategic adaptation, and trajectory-dependent confidence lie outside the evaluated setting.

• Address inconclusive comparisons. The GSM8K → MATH comparisons other than online Platt, the same-task hardening checks, and the MMLU-shift comparison against online Platt are all inconclusive, so when and why the advantage fails to materialise remains open.

• Broaden the evidence base. LiveCodeBench, CodeContests, TriviaQA, and the original dynamic-pool experiments were excluded because their historical evaluator or reference evidence was insufficiently authenticated; the paper also notes its scoring authentication does not extend to the complete original collection environment. Recovering that evidence is a clear next step.

Target Audience

Researchers and practitioners working on LLM calibration, uncertainty quantification, and multi-model coordination will benefit most, along with engineers building model-routing or ensemble-selection systems that already capture correctness outcomes. Readers need comfort with calibration metrics and online estimation; the paper is not an introductory treatment, and the shift, sensitivity, selected-model-feedback, band-granularity, and appendix results are beyond the portion of the content reviewed here.

Authors’ abstract

When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads. This paper presents MARGIN (Multi-Agent Runtime Grading via Incremental Normalisation), a runtime calibration method that learns model-specific confidence corrections from observed answer outcomes without retraining the models or requiring a held-out calibration set. MARGIN tracks recent accuracy and stated confidence within confidence bands, uses their ratio to correct reported confidence, and blends sparse-band corrections toward a model-level estimate. The corrected scores weight candidate answers in a collective decision. Evaluation covers code generation, question answering, and mathematics, using an 18-model pool and a nine-model subset for distribution-shift experiments. On BigCodeBench, model-mean confidence is negatively related to accuracy; among correct/incorrect response pairs, choosing the more confident responder performs below chance. Against five online calibration baselines receiving identical feedback and retaining their learned state across each transition, MARGIN achieves lower post-shift expected calibration error than all five in two code-generation transitions and than four in a question-answering transition; the remaining question-answering comparison is inconclusive. In separate code-generation coordination experiments, calibration improves the ranking of correct responses and increases answer-selection accuracy by 4.3 and 14.0 percentage points on two of three benchmarks relative to uncalibrated confidence weighting. These results support model-specific runtime calibration for coordination under changing workloads when correctness feedback is available for the participating responders.