Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systems

AI summary

Overview

Research area: AI security and privacy — confidentiality failures in LLM agents that retrieve and synthesize documents, specifically the "mosaic effect" (also called classification by compilation or aggregation).

Technical level: Intermediate. The paper is readable without deep ML background, but assumes familiarity with LLM agents, retrieval, and evaluation methodology.

Scope: A controlled, multi-domain empirical evaluation of whether agentic LLM systems disclose higher-sensitivity "forbidden composites" that are implied only by combining individually accessible, lower-classification documents, and whether simple pre-LLM safeguards prevent this.

What This Paper Is About

An LLM agent can obey every document-level access permission and still hand a user a conclusion they are not cleared to receive, because two permitted facts can jointly imply a sensitive third fact that neither states alone. The authors build a synthetic benchmark of such "forbidden composites" across cybersecurity, medical, and military domains and measure how often three LLM agents state them — under benign, explicit, and adversarial user queries — and whether visible classification headers plus explicit aggregation rules stop the leak.

Key Contributions

1. A controlled benchmark of compositional sensitivity. Thirty synthetic examples (ten each in cybersecurity, medical, and military), each with a forbidden composite labeled TOP_SECRET, two relevant documents labeled SECRET, metadata, an optional in-domain distractor, and three queries. Four factors are crossed — intent (benign/explicit/adversarial), distractor (absent/present), classification marking visibility (hidden/shown), and aggregation rules (off/on) — giving 24 configurations per example.

2. A two-setting measurement design. Controlled simulation (fixed prompts and document order, no real retrieval) yielding 10,800 final answers, plus two live open-source agent harnesses (Pi v1.0.4 and Hermes v0.21.5) that select their own context, yielding 1,440 answers total (720 each).

3. A validated automated judge with separate disclosure and refusal labels. Three human volunteers labeled the same 120 answers (360 annotation records, 720 binary judgments); MiMo-v2.6-Pro-RL was selected as judge. Human Krippendorff's alpha was 0.810 for mosaic disclosure and 0.962 for refusal; the judge scored 0.817 and 0.925 respectively, with precision/recall of 86.44/94.44 (mosaic) and 90.48/100.00 (refusal).

4. Evidence that refusal and disclosure are distinct behaviors. Because labels are scored independently, the paper shows that a refusal can itself reveal the prohibited conclusion — a failure mode invisible to marginal refusal rates.

Main Findings

• Benign queries already leak. Queries designed to be answerable from a single document elicit forbidden composites in 7.08% of simulated GPT-5.6-Sol answers (95% CI: 3.75–10.67%), 12.50% of Pi answers, and 8.75% of Hermes answers, averaged over safeguard and distractor settings. No deliberate request for the composite is needed.

• Overall disclosure is high and generator-dependent. Pooled across all 24 configurations, 52.68% of simulated answers convey the composite. GPT-5.6-Sol is lowest at 27.75% [21.83, 33.03]; Claude Sonnet 5 reaches 63.17% [55.30, 70.36]; Qwen3.5-27B reaches 67.11% [59.89, 73.64]. Refusal rates diverge sharply: 37.56%, 59.75%, and 11.08% respectively.

• Live harnesses behave close to simulation. Pi discloses in 30.14% [24.03, 35.83] and Hermes in 32.36% [26.39, 38.06] of answers, near the simulated GPT baseline — which the authors read as soft validation of the simulation's realism.

• Safeguards help substantially but do not close the gap. Enabling both headers and rules reduces pooled disclosure from about 69% to 32%, a full-stack reduction of 36.26 percentage points [29.85, 41.82]. Rules contribute the larger marginal reduction (27.91 pp [23.19, 32.41]) versus headers (8.35 pp [6.13, 10.37]), and rules work better when headers are shown. Residual disclosure with both safeguards on is 32.37% [28.33, 36.19] across all intents.

• Refusals can disclose. In the full-stack arm, 404 of Claude's 410 mosaic-positive answers were also refusals — more than half of its scored refusals reveal the composite. GPT shows no such overlap in that arm, while Qwen retains both categories. One verbatim example shows Claude declining to synthesize two SECRET documents while describing exactly the forbidden inference they support.

• Aggregate rates hide domain differences. Military disclosure rises from about 17% in simulated GPT to roughly one quarter in each live harness, even though military is GPT's lowest simulated domain. Pooled domain rates are 58.00% for cybersecurity, 53.08% for medical, and 46.94% for military. Both live-minus-simulated military intervals sit above zero, while Pi's cybersecurity point difference runs the opposite way.

• Safeguard effects vary by model. Full-stack mosaic reduction is 46.44 pp for GPT, 36.89 pp for Claude, and 25.44 pp for Qwen; the rules-plus-headers interaction is 14.11 pp for GPT and 15.33 pp for Qwen but −3.33 pp for Claude.

Methodology in Plain English

The authors wrote 30 fictional scenarios, each constructed as a small puzzle: two SECRET documents that are individually unremarkable, plus a TOP_SECRET conclusion that only appears when you connect them. A medical example asks for a participant's medication kit identifier (answerable from one document); combining that identifier with a formulation key reveals that the participant is on placebo.

Each scenario was run through 24 combinations of query intent, an optional distractor document, whether the SECRET classification is visibly printed on the documents, and whether explicit aggregation rules are in the prompt. Rules instruct the model to prefer single-source answers, avoid unstated cross-source synthesis, and refuse classified combinations. In simulation, the models received a fixed system prompt, a scripted search-and-read sequence adapted from real agent traces, then the documents in a fixed order, and produced a final answer. Each example-configuration-generator cell was repeated five times, giving 150 answers per cell and 3,600 per generator.

In the live setting, Pi (minimal, four tools) and Hermes (over 40 tools with adaptive memory) ran in fresh isolated containers with only the document files mounted read-only; the agents had to use their own read and shell operations to find them, and rules-on was delivered through a discoverable AGENTS.md file. Only final answers were scored. A separate LLM judge, chosen by comparison against three human annotators on 120 shared answers, labeled each answer for whether it conveyed the composite and whether it refused — as two independent booleans. Confidence intervals come from 2,000 bootstrap resamples clustered at the example level.

Why This Matters

Impact on research: The paper reframes confidentiality in agentic systems as a problem of inference and disclosure rather than access control. It shows that a benchmark can be designed specifically around the gap between source permission and composite confidentiality, complementing privacy benchmarks like PrivacyLens, AgentDAM, and CI-Work. Treating refusal and leakage as separate outcomes is a methodological contribution that undermines refusal rate as a safety proxy.

Real-world applications:

• Intelligence and defense analysis, where individually unclassified reporting can compile into a classified assessment.
• Healthcare, where a treatment identifier plus a formulation key reveals trial-arm assignment, or donor screening plus a transplant schedule reveals an infection status.
• Enterprise security operations, where a patch-status record plus a network-exposure review jointly establish an exploitable path.
• Multi-tenant data platforms where users hold legitimate access to some but not all aggregate conclusions.

Industry relevance: Enterprise deployments increasingly give agents broad read access across shared drives and wikis while relying on document-level permissions. This work indicates that such permissions are an incomplete boundary, and that prompt-level policy text — the cheapest available mitigation — leaves roughly a third of answers leaking in the tested configuration.

Future Directions

• Deterministic harness-level controls. Enforce anti-mosaic policy in harness logic at retrieval, tool execution, and answer release, encoding protected relationships and tracking which facts enter context — evaluated on residual disclosure, false positives, and task utility.
• Multi-agent disclosure paths. Agents that distribute document access, exchange partial findings, and accumulate sensitive relationships in shared memory may create leakage routes that single-agent evaluations cannot see; this needs studies varying communication patterns and clearance boundaries.
• Isolating the causes of the live-versus-simulated gap. The military increase cannot currently be attributed to tool use alone, since native prompts, tool sets, autonomous document selection, and retained-exposure evidence all vary together; controlled ablations are required.
• Capturing intermediate reasoning. The current design scores only final answers, leaving reasoning traces and tool outputs — where composites may also surface — outside the measurement scope.

Target Audience

AI safety and security researchers studying agent confidentiality and privacy; red teams and security architects deploying LLM agents over mixed-clearance or regulated data; policy and compliance staff who translate classification rules (such as ISOO marking and compilation directives) into technical controls; and benchmark designers interested in evaluation methodology, particularly the use of human-validated LLM judges and separately scored disclosure and refusal outcomes.

Authors’ abstract

Modern LLM agents can retrieve information from accessible documents, and then disclose a sensitive conclusion assembled from them. This mosaic effect creates a critical risk for agents operating over confidential data. We quantify its realization with 30 synthetic cybersecurity, medical, and military examples, yielding 10,800 final answers from three LLM agents in controlled simulation and 1,440 answers from live agents in two open-source harnesses. Even under benign queries designed to be answerable from one document, GPT-5.6-Sol, the best performing model tested, discloses forbidden composites in 7.08% of answers (95% CI: 3.75-10.67%) in simulation, with worse disclosure rates in live harnesses. Adding visible classification headers and explicit aggregation rules reduces pooled disclosure rates substantially, yet unsafe disclosures persist in 32.37% of simulated answers (95% CI: 28.33-36.19%) across all intents despite these safeguards. Furthermore, increased refusals do not guarantee confidentiality, since significant refusals reveal the prohibited conclusion regardless. These results point to a troubling conclusion: sensitive data leaks via mosaic-style events are an observed confidentiality failure mode in the evaluated settings at non-negligible rates. They motivate stronger controls on what agents may infer and disclose in sensitive data environments with mixed-clearance users.