UMAS: System-Level Uncertainty Quantification for Multi-Agent LLM Systems

AI summary

Overview

Research area: Multi-agent systems (MAS) built on large language models, with a focus on uncertainty quantification (UQ) and reliability estimation.

Technical level: Intermediate. The paper assumes familiarity with token log-probabilities, AUROC-style evaluation, and collaborative agent protocols such as Debate and DyLAN, but the core idea is explained with relatively accessible math.

Scope: The paper proposes UMAS, a parameter-free, cost-neutral framework that assigns uncertainty to every agent generation in a multi-agent interaction graph and aggregates these into a single system-level uncertainty score, evaluated across two MAS paradigms, two main LLM backbones, and five benchmarks.

What This Paper Is About

Multi-agent systems often answer questions more accurately than a single LLM, but they still hallucinate and produce wrong outputs. The paper shows that standard uncertainty methods — which look only at the final answer — become less reliable (lower AUROC) when applied to a MAS, even though task accuracy goes up, because they ignore the multi-round, multi-agent interaction that produced that answer. UMAS addresses this by tracking confidence at every agent node, discounting revisions that lean on unreliable peers, and averaging confidence over the whole interaction graph to produce one system-level uncertainty score without extra sampling or model calls.

Key Contributions

1. Identifies a failure mode in MAS uncertainty quantification. The authors document that applying standard single-agent UQ to the final MAS output can significantly degrade uncertainty reliability, and attribute this to a structural mismatch: MAS uncertainty depends on multi-round interaction and on the credibility of inter-agent influence.

2. Proposes UMAS, a parameter-free, cost-neutral UQ framework. UMAS assigns a node-level confidence to each agent response at each round, adjusts it using a credibility signal derived from peer confidences, and aggregates node-level uncertainty into a system-level estimate using only logits already produced during a normal MAS run.

3. Introduces an ablation variant, UMAS_FIX, which uses the same trajectory-level aggregation but skips credibility-aware updating. Comparing it to single-output baselines and to full UMAS cleanly separates the gain from aggregation alone versus the gain from credibility weighting.

4. Demonstrates downstream utility. Beyond AUROC, the framework supports selective prediction (choosing among multiple trajectories) and hallucination detection, and is evaluated across two MAS paradigms, two main backbones plus a third in the appendix, and five benchmarks.

Main Findings

• Standard UQ degrades on MAS output: Despite MAS achieving higher task accuracy than a single LLM, applying standard UQ to the terminal MAS response yields lower AUROC, motivating the paper's structure-aware approach.

• Large AUROC gains: Across the ten dataset cells per backbone, the best baseline averages 63.00 (Llama 4 Scout) and 64.86 (GPT-4o mini), while UMAS reaches 74.93 and 74.02 respectively; UMAS_FIX reaches 70.32 and 71.49. The abstract reports an average gain of 10.54 AUROC points over state-of-the-art baselines.

• Aggregation is the primary driver, credibility weighting adds more: Trajectory-level aggregation alone (UMAS_FIX over the best baseline) contributes +7.32 / +6.63 AUROC across the two backbones, and credibility-aware updating (UMAS over UMAS_FIX) adds a further +4.61 / +2.53. The credibility contribution is largest when peer confidences vary widely (e.g., GSM-Hard, MedMCQA) and shrinks when peers are uniformly confident.

• Selective prediction improves system accuracy: With K = 5 independent trajectories at temperature 0.5 using GPT-4o mini, absolute accuracy gains under UMAS range from 5.5 points (DyLAN-MMLUP) up to 12.6 points (Debate-AQUA). UMAS or UMAS_FIX gives the highest selection accuracy in every evaluated cell and never falls below the strongest UQ baseline.

• Better hallucination detection: Using a 20% validation set (N = 40) to pick a threshold, specificity at varying F1 constraints (δ from 0.1 to 0.9) shows UMAS_FIX and UMAS consistently outperforming all baselines, with UMAS closer to the upper-right corner in most settings.

• Robust to scale: Both variants beat all baselines for agent counts and rounds N, M in {2, 3, 4}, and across seven Llama variants from 1B to 400B total parameters (MMLU used for models below 8B since MMLU-Pro exceeds their capability).

• Metric-agnostic: Substituting PE or PPL for MSP as the node-level confidence produces gains over the raw metric in 11 of 12 configurations; the sole exception is PPL on MBPP, essentially on par (59.87 vs. 60.03).

• Cheap uncertainty computation: On Llama 4 Scout under Debate on MMLU-Pro, UMAS adds 0.143 ms of uncertainty time on top of 84.17 s inference latency with no GPU needed and 80.90 AUROC, versus, for example, SE at 1813.92 ms and 47.81 AUROC, and Deg at 3726.16 ms and 54.89 AUROC.

Methodology in Plain English

The system is modeled as a directed graph: each agent at each round is a node, and an edge exists when one agent's output feeds into another's revision.

Node confidence. Each generation already carries a log-probability (maximum sequence probability, MSP). The paper uses the log of that probability as the node's raw confidence — it costs nothing extra because the logits come free with generation. A larger (less negative) value means higher confidence.

Peer credibility. When agent i revises its answer using agent j's output, the framework compares j's dynamic confidence to i's previous dynamic confidence as a relative ratio. Because confidences are negative, a more confident peer produces a negative score and a less confident peer a positive one. A sigmoid maps these to (0,1), which keeps extreme ratios from dominating. The node's trust weight is then one minus the average of those mapped scores, so credible peers yield a larger weight.

Confidence updating. The dynamic confidence is a credibility-weighted moving average: with low peer credibility the weight goes to zero and the previous estimate is retained, preventing spurious overconfidence; with high peer credibility the weight goes to one and confidence tracks the current response.

System-level score. The final uncertainty is the negative average dynamic confidence over all nodes, chosen because it is length-normalized and stable across runs with different numbers of agents and rounds.

The ablation UMAS_FIX keeps this averaging but uses raw node confidence, skipping the updating step. Notably, all multi-sampling baselines were standardized to 5 samples, and consistency-based methods were applied to a condensed representation (marked with an asterisk) to keep them tractable on long MAS trajectories.

Why This Matters

Impact on research. The paper reframes UQ for LLM systems from a property of a single output to a property of an interaction trajectory, and shows the two design choices (aggregation and credibility weighting) can be measured separately. It also warns that the common practice of scoring only the final MAS answer can be actively misleading.

Real-world applications:

• Medical question answering (MedMCQA is one of the benchmarks), where flagging unreliable outputs before they reach a clinician matters.
• Code generation (MBPP), where a low-confidence response can be routed to review rather than shipped.
• General-knowledge and math assistants (MMLU-Pro, AQUA-RAT, GSM-Hard) that need to abstain or escalate when unsure.
• Cost-sensitive deployments that want reliability signals without paying for repeated sampling.

Industry relevance. Because UMAS needs no extra generations, no auxiliary models, and no GPU for the uncertainty step (0.143 ms), it can be layered onto existing multi-agent pipelines as a monitoring signal. The cost comparison is stark: the same AUROC gains are unavailable to multi-sampling methods whose uncertainty step costs hundreds to thousands of milliseconds.

Future Directions

• Black-box deployment. UMAS requires token-level logits, so it does not apply to APIs that expose only completion text. Extracting comparable confidence signals from text-only interfaces is an open problem.

• Adversarial and role-asymmetric MAS. The credibility rule assumes cooperative peers. In proposer–critic or other asymmetric protocols, a critic's confidence cannot be read without the semantic content and role of its critique, and agents that strategically misreport confidence are unexplored.

• Shared misconceptions. When all agents are confidently wrong in the same way, no peer disagrees and credibility signals cannot detect the failure. A mechanism that compares against external evidence or a genuinely independent source would be needed.

• Inheriting better base metrics. Since UMAS is metric-agnostic, it could absorb future single-pass UQ advances at no architectural cost — the paper demonstrates this only for MSP, PE, and PPL.

Target Audience

Researchers and engineers working on multi-agent LLM architectures, LLM reliability and hallucination detection, or uncertainty quantification. It is most useful to readers who already understand why log-probabilities serve as a confidence proxy and who want a cheap, deployable reliability signal for agentic pipelines — and to practitioners deciding whether the extra cost of sampling-based UQ is justified in a multi-agent deployment.

Authors’ abstract

Multi-Agent Systems (MAS) are a promising paradigm for improving the reasoning capabilities of large language models, yet they remain prone to hallucinations and erroneous outputs. Uncertainty Quantification (UQ) is crucial for assessing system reliability, but existing methods are largely designed for single-agent settings; how to exploit MAS structure for UQ while keeping inference cost-neutral remains underexplored. We propose UMAS, a parameter-free, cost-neutral, system-level UQ framework that estimates the credibility of inter-agent influence from each generation's intrinsic confidence. UMAS uses these credibility signals to modulate how uncertainty is updated at each node, and then aggregates node-level uncertainty into a system-level estimate. Extensive evaluations on Debate and DyLAN across diverse benchmarks show that UMAS consistently outperforms state-of-the-art baselines by an average of 10.54 AUROC points. Beyond uncertainty estimation, UMAS enables hallucination detection and uncertainty-aware answer selection, improving MAS accuracy by up to 12.6 points on specific tasks while enhancing reliability. Code is available at https://github.com/Dby52-04/UMAS.