AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems

AI summary

Overview

Research area: AI safety and privacy evaluation for multi-agent Large Language Model (LLM) systems, spanning AI security benchmarks, multi-agent architectures, and regulatory data-minimization compliance.

Technical level: Advanced. The paper combines a formal privacy leakage definition grounded in contextual integrity theory, a seven-channel instrumentation taxonomy, a three-tier detection pipeline with threshold calibration, and a 32-class attack taxonomy.

Scope in one sentence: A 1,000-scenario benchmark that measures whether sensitive data leaks through the internal coordination channels of multi-agent LLM systems, not just through the final text shown to users.

Accepted for publication in IEEE Access, 2026 (Manuscript ID: Access-2026-23002); DOI 10.1109/ACCESS.2026.3704541. The work was conducted at the Department of Computer and Software Engineering, Polytechnique Montréal.

What This Paper Is About

Multi-agent LLM systems delegate subtasks to specialized agents, and in doing so pass task context through inter-agent messages, shared memory, and tool arguments. Existing privacy benchmarks only inspect what agents say to users, so they cannot see whether sensitive data leaked along these internal pathways. The authors built AgentLeak to instrument and measure those hidden channels, and to quantify how much privacy risk standard output-only auditing misses.

A motivating example from the paper: in an audited multi-agent healthcare workflow, a scheduling agent returned a clean appointment confirmation that passed any output audit, while its delegation message to a verification agent carried the patient's complete medical record.

Key Contributions

1. AgentLeak benchmark. A 1,000-scenario benchmark across healthcare, finance, legal, and corporate domains, with seven-channel instrumentation and large-scale analysis of final outputs (C1), inter-agent messages (C2), and shared memory (C5). Each scenario includes a sensitive data vault, an allowed disclosure set, and ground-truth labels.
2. Leakage channel taxonomy. Seven pathways through which sensitive data can escape in agentic systems, explicitly separating external channels (C1, C3, C4, C6, C7) that cross system boundaries from internal channels (C2 inter-agent messages, C5 shared memory) that prior benchmarks typically do not cover.
3. Coordinator-worker topology evaluation. Privacy leakage evaluated in coordinator-worker configurations across five LLMs and four domains, establishing a baseline for future multi-topology studies.
4. Three-tier detection pipeline. Canary matching, structured field extraction, and LLM-as-Judge detection applied at scale to C1, C2, and C5, and exposed through an SDK for all instrumented channels.
5. Privacy-utility tradeoff quantification. A stated Pareto analysis showing that current defenses cannot simultaneously maintain task completion and preserve privacy on internal channels. (The paper content available here does not include the numeric Pareto results.)

Main Findings

• Multi-agent architecture lowers final-output leakage. Final-output leakage (C1) is 27.2% in multi-agent configurations versus 43.2% in single-agent mode.
• But internal channels raise total exposure. Aggregated across C1, C2, and C5, total system exposure reaches 68.9% — a 1.6x increase over the single-agent baseline. A trace counts as leaked if any of C1, C2, or C5 leaked.
• Inter-agent messages are the dominant leak path. Channel C2 leaks at 68.8% compared with 27.2% for final outputs (C1), which means output-only audits miss 41.7% of violations.
• Internal channels leak more than external ones. Mean internal-channel leak rate is 57.8% versus 27.2% for external channels, a 2.1x difference. C2 reaches 68.8%, exceeding external output by 2.5x; C5 (shared memory) sits at 46.7%.
• The pattern is consistent. Across all five models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Mistral Large, Llama 3.3 70B) and all four domains, the relationship C2 >= C1 holds.
• Detection pipeline performance. On a 1,000-segment test set, the full three-tier pipeline achieves precision 0.952, recall 0.934, F1 0.943, with FPR 4.8%. Tier 3 alone (LLM-as-Judge) reaches precision 0.941, recall 0.926, F1 0.933, FPR 6.2%. Regex-only (Tier 1) has perfect precision (1.000) but recall of only 0.080 (F1 0.148); adding NER/pattern matching (Tiers 1+2) raises recall to 0.180 (F1 0.305).
• Threshold calibration. The semantic similarity threshold τ = 0.72 was chosen as the highest value keeping FPR below 5% while maintaining near-maximal F1 (0.938) on a held-out 200-segment calibration set. At that threshold, FPR is 4.8% and FNR is 7.4%.
• Judge robustness. Across three judges on 240 labeled trace segments (120 positives, 120 negatives), Qwen-2.5-7B achieved FPR/FNR of 4.8%/7.4%, Llama-3.1-8B 5.6%/8.1%, and GPT-4o-mini 4.1%/6.9%. Pairwise Cohen's kappa ranged from 0.82 to 0.87.
• Reported rates are lower bounds. Qwen-2.5-7B was selected as the default judge for cost reasons (7B parameters), which introduces a conservative bias since GPT-4o-mini achieves lower FNR (6.9%).
• Tier attribution. Leaks are attributed to the lowest-numbered tier that detects them, so the reported 8%/10%/82% tier distribution reflects each tier's marginal contribution after higher-priority tiers claimed their detections.
• Scenario strictness. Across all 1,000 scenarios, the median vault contains 29 fields (range 21–39) while the allowed set comprises 3 fields (range 2–5), a median vault-to-allowed ratio of 9.7:1.
• Annotation reliability. Inter-rater reliability (Cohen's kappa) was 0.82 for Healthcare, 0.79 for Finance, 0.81 for Legal, and 0.75 for Corporate — all above 0.6.
• Not reported in the available content: per-family Attack Success Rate values for families F1 through F4, Task Success Rate results, and the numeric privacy-utility Pareto analysis. F5 and F6 attacks are included in aggregate metrics but not tracked separately at the trace level due to instrumentation constraints.

Methodology in Plain English

The authors built an evaluation harness that plugs into existing multi-agent frameworks and records every place data can travel. A normalized JSON Lines (JSONL) trace store captures events across seven channels: final outputs (C1), inter-agent messages (C2), tool inputs to external APIs (C3), tool outputs (C4), shared memory (C5), logs and telemetry (C6), and persistent artifacts (C7).

Each of the 1,000 scenarios is a structured record containing a natural-language task, a private vault of sensitive fields (each labeled PHI, PII, or PCI with a sensitivity weight), an allowed set of fields genuinely needed for the task, tool specifications, agent topology, attack level, and ground-truth leakage labels. Leakage is defined against the allowed set: a leak occurs when a vault field outside the allowed set appears in a channel's content above a similarity threshold. This operationalizes data minimization as a testable criterion, consistent with GDPR Article 25, HIPAA's minimum necessary standard, and Quebec's Law 25.

Data combines synthetic generation (the Faker library, with SSNs in valid three-two-four format, Luhn-valid credit cards, hospital-convention medical record numbers, real area codes) and real data for validation: ai4privacy/pii-masking-200k (3,624 PII samples) and AGBonnet/augmented-clinical-notes (2,202 clinical notes). Sensitive values come in three tiers — explicit canaries, format-valid fakes, and natural-language facts requiring semantic understanding.

Allowed sets were validated by two independent domain experts per vertical, rating each vault field on a four-point scale; fields rated 3 or higher by both annotators entered the allowed set.

Detection works in three tiers: regex canary matching, structured field audit (SSN, Luhn-valid credit cards, ICD-10 codes), and LLM-as-Judge semantic analysis. The judge threshold was tuned by grid search over τ in [0.60, 0.85] using 200 manually labeled trace segments held out entirely from the 1,000-scenario evaluation. Adversaries are modeled at three levels: A0 (benign, misconfiguration only), A1 (weak, manipulating external sources such as poisoned APIs or compromised documents), and A2 (strong, direct access via compromised tools, malicious dependencies, or malicious insiders).

Why This Matters

Impact on research. The paper extends the "leaky thoughts" observation from single-model internal reasoning (Green et al., EMNLP 2025) to the architectural level: multiple agents coordinating with each other. It provides the first benchmark the authors identify as covering multi-agent topology, internal channels, execution-trace-based detection, and reproducible ground-truth vaults simultaneously — AgentDojo, AgentDAM, PrivacyLens, TOP-Bench, and ASB each miss at least one of these criteria.

Real-world applications:

• Healthcare: coordinating scheduling, insurance verification, and clinical agents without propagating full medical records through delegation messages, in line with HIPAA minimum necessary requirements.
• Finance: know-your-customer and anti-money-laundering workflows where account identifiers and creditworthiness data move among compliance teams and service providers under GLBA and PCI-DSS.
• Legal: contract analysis and discovery where unauthorized disclosure can cause complete privilege waiver.
• Corporate: incident response, HR investigations, and merger due diligence involving employee records, compensation data, and proprietary strategic information.

Industry relevance. Existing defenses — Lakera Guard, Meta's PromptGuard, NVIDIA NeMo Guardrails, LlamaGuard, and privacy-focused prompts — operate at system boundaries. The paper reports that none intercept inter-agent messages, memory writes, or coordination logs, and that the frameworks surveyed (LangChain, CrewAI, AutoGPT, AutoGen, MetaGPT) lack default sanitization for inter-agent messages or memory access restrictions. For organizations deploying coordinator-worker systems in regulated domains, the reported result is that output-only auditing misses the majority of privacy violations.

Future Directions

1. Extend per-family attack attribution. F5 (Reasoning and Chain-of-Thought) and F6 (Evasion and Obfuscation) attacks are currently only counted in aggregate; the authors state future versions will extend family-level Attack Success Rate reporting to all six families.
2. Enable per-field leakage attribution. The current trace pipeline records binary channel-level leakage, which is why Weighted Leakage Score analyses were deferred in favor of ELR, CLR, and ASR. Per-field attribution would let the severity-weighted WLS be reported at scale.
3. Re-tune detection for other languages. The τ = 0.72 calibration applies to English scenarios only, and the authors flag that multilingual deployments may require re-tuning.
4. Broaden topology coverage. The evaluation covers coordinator-worker configurations only; AgentLeak is positioned as a baseline for future multi-topology studies.

Target Audience

AI safety and privacy researchers studying agentic systems; security engineers and architects deploying multi-agent LLM pipelines in regulated industries; compliance and privacy officers at healthcare, financial, legal, and enterprise organizations who need to reason about data minimization in automated workflows; and benchmark designers who need a framework-agnostic evaluation harness with reproducible, hash-verified scenarios.

Authors’ abstract

Multi-agent Large Language Model (LLM) systems create privacy risks that current output-only benchmarks cannot measure. When agents coordinate on tasks, sensitive data may pass through inter-agent messages, shared memory, and tool arguments, all pathways that final-output audits typically do not inspect. We introduce AgentLeak, a benchmark for evaluating internal-channel privacy leakage in multi-agent LLM systems. AgentLeak instruments seven privacy-relevant communication pathways and provides a large-scale empirical evaluation focused on final outputs, inter-agent messages, and shared memory. Across 1,000 scenarios spanning healthcare, finance, legal, and corporate domains, five production LLMs (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Mistral Large, and Llama 3.3 70B), and 4,979 validated execution traces, we find that multi-agent configurations reduce final-output leakage (C1: 27.2% vs 43.2% in single-agent mode) compared with single-agent baselines but introduce internal channels that raise total system exposure to 68.9% (aggregated across C1, C2, C5). Inter-agent messages (C2) leak at 68.8%, compared with 27.2% for final outputs (C1), meaning that output-only audits miss 41.7% of violations. Across all five models and four domains, the pattern C2 $\geq$ C1 holds consistently. These results suggest, within the evaluated coordinator-worker setting, that privacy risk in multi-agent systems is strongly shaped by architectural coordination channels rather than final-output behavior alone: it arises from internal channels that remain invisible to standard output-level defenses.