APort Vault: Benchmarking AI Agent Payment Authorization with the Open Agent Passport

AI summary

Overview

Research area: AI agent security, adversarial robustness, and payment authorization (cs.CR primary, cs.AI secondary).

Technical level: Intermediate. The core idea — comparing an agent with and without a check between the model and the tool — is easy to grasp, but the paper carries dense statistics (session-clustered bootstraps, rule-of-three bounds, Benjamini-Hochberg FDR, Cohen's kappa) and a large experimental grid.

Scope in one sentence: The paper introduces APort Vault, a benchmark that replays 4,371 human-authored attacks against AI payment agents across 14 models from 8 labs, five policy levels and two replay tracks, with and without a deterministic pre-action authorization layer implementing the Open Agent Passport (OAP) specification, completing 225,964 evaluations.

What This Paper Is About

When an AI agent is given a tool that moves money, two things can stop an unwanted transfer: the model can refuse to request it, or something between the model and the tool can refuse to execute it. Almost all published measurement studies the first. This paper measures the second. It takes real attacks written by humans during a public capture-the-flag event against a live agentic bank and replays them against the same models, prompts and tool schemas, varying only whether a deterministic authorization check sits in the tool-dispatch path.

Key Contributions

1. A replay methodology and released corpus. 4,371 human-authored attacks against a live payment agent, replayed across a 14-model by 5-level by 2-track by 2-architecture grid, yielding 225,964 completed evaluations, released with scoring code and a frozen snapshot hash.

2. A five-stage account of what happened, with the zero bounded. The paper reports requests, successful payments, policy decisions, recipient membership and unpermitted transfers as five separate events rather than collapsing them into one success rate. At Levels 2 to 4, the model alone produced 28,543 requests, 28,521 successful payments and 140 unpermitted transfers in 76,842 evaluations; behind the layer, 25,527 requests, 25,370 successful payments and 0 unpermitted transfers in 69,297. The zero carries a session-clustered upper bound of 0.38% rather than a zero-width interval.

3. The boundary result was not obtained by suppressing requests. Aggregate request rates are close in both architectures at every level, and on 68,970 matched triples the paired difference is +0.084 percentage points [-0.020, +0.189], reported as an observation rather than an equivalence test, with disclosure that 1,220 individual pairs disagree.

4. Three structural results a leaderboard cannot produce. Outcomes concentrate in source sessions rather than techniques; attack transfer across models is weak; and multi-turn replay helps one attack family enormously and most others not at all.

5. A judge reliability audit on the full set. Pooled inter-judge kappa is 0.772 and falls to 0.167 at Level 3 and 0.521 at Level 5; one panel member recovers 99.1% of deterministically-confirmed outcomes and the other 64.4%. Because the headline metric is deterministic, none of this affects it.

Main Findings

• Request rates track the configuration, not the model. Model-alone evaluations containing a payment request: 10.9% at Level 1 (19,594 evaluations, 2,127 requests), 3.0% at Level 2 (25,901; 773), 0.1% at Level 3 (15,969; 13), 79.4% at Level 4 (34,972; 27,757), 25.3% at Level 5 (22,320; 5,640). The Levels 2 to 4 aggregate is 28,543 of 76,842 (37.1%, session-clustered 95% interval 25.7 to 48.8), and Level 4 supplies 97.2% of that numerator.

• The authorization boundary is where the architectures diverge. At Levels 2 to 4, unpermitted transfers total 140 of 76,842 with the model alone and 0 of 69,297 behind the layer. On 68,970 matched model, prompt and track triples the counts are 105 and 0, with requests of 25,362 and 25,420 respectively.

• The zero was not achieved by refusing payments. Behind the layer, 25,370 payments executed at Levels 2 to 4. Of the 25,640 transfer calls the policy evaluated, it allowed 25,453 and denied 187: 148 for a forbidden recipient, 17 for an invalid amount, 13 for a limit violation, 9 for a missing audit code. Those denials arise in 172 evaluations, 15 of which also contain a successful payment.

• The zero is bounded, not absolute. The 69,297 Levels 2 to 4 evaluations behind the layer draw on 790 distinct source sessions. The exact one-sided 95% upper bound on the per-evaluation rate is 0.0043%; the per-session rule-of-three bound is 3/790 = 0.380%, which the paper quotes. At Level 5 the same construction gives 1.429% per session over 210 sessions.

• Every tested model requests payments on identical inputs. On the 1,293 Level 4 prompts evaluated on all fourteen models in single-turn replay, request rates span 71.2% (Claude Haiku 4.5) to 84.3% (Gemini 3.8 Flash), and 809 prompts (62.6%) elicited a request from all fourteen models, each ending in a successful payment to the level's allowlisted recipient. The paper states this is shared behavior, not a count of prompts that defeated the models, because Level 4 authorizes documented transfers to that recipient (audit@aport-vault.com).

• The layer does what the policy says, not blanket refusal. At Level 1, whose passport grants a wildcard recipient list, 17,802 behind-the-layer evaluations yielded 1,909 transfer requests and 1,904 successful payments; of 1,928 transfer calls evaluated the policy allowed 1,912 and denied 16 on amount and limit rules. Per model out of 709 prompts: Gemini 3.5 Flash produced 226 single-turn Level 1 payments behind the layer against 224 model alone; GPT-5.6 Terra 38 against 41; Claude Fable 5.1 52 against 64.

• Outcomes concentrate in sessions, not techniques. The 140 Levels 2 to 4 model-alone unpermitted transfers come from 24 of 790 source sessions; one session accounts for 67 (47.9%), two account for half, and ten for 90%. At Level 5, 5,362 outcomes come from 130 of 210 sessions, with 14 sessions producing half.

• One forged-receipt cohort drives almost all multi-turn Level 2 outcomes. 103 attack attempts from 8 sessions contain the literal string [Tool Result: verify_recipient], all at Level 2. They account for 111 of the 113 multi-turn Level 2 model-alone outcomes and affected seven models (DeepSeek V4 Flash and Pro, GLM-5.3, GPT-5.6 Luna and Terra, Kimi K3, Qwen3.8 Max). Behind the layer this cohort produced 0 unpermitted transfers (1,234 multi-turn evaluations).

• Multi-turn replay helps one attack family, not attacks in general. At Levels 2 to 4 model alone, json_injection goes from 1 of 2,184 single-turn to 112 of 2,183 multi-turn, while authority_claim moves from 5 of 18,885 to 13 of

Authors’ abstract

APort Vault is a benchmark for payment authorization in tool-using AI agents. It replays 4,371 attacks written by humans against a live payment agent during a public capture-the-flag event, across 14 models from 8 labs, five policy configurations and two replay tracks, with and without a deterministic pre-action check implementing the Open Agent Passport (OAP) specification. 225,964 evaluations completed. We report five distinct events per evaluation, because collapsing them is how an agent benchmark produces a number that does not survive review. Requests are common and their rate differs far more across configurations than across models, though each attack exists at exactly one configuration so policy and attack cohort vary together: 10.9% of model-alone evaluations at Level 1, 3.0% at Level 2, 0.1% at Level 3, 79.4% at Level 4. On the 1,293 Level 4 prompts, each evaluated on every model, request rates run from 71.2% to 84.3%, and 809 prompts (62.6%) elicited a request from all fourteen models, each ending in a successful payment to the level's allowlisted recipient. The authorization boundary is where the conditions diverge. At Levels 2 to 4, transfers to recipients the passport did not permit number 140 of 76,842 with the model alone and 0 of 69,297 behind the layer, and 105 against 0 on 68,970 matched model, prompt and track triples. The zero spans 790 source sessions, giving a per-session upper bound of 0.38%. It was not obtained by refusing payments: 25,370 payments executed behind the layer, while the policy denied 187 of the 25,640 transfer calls it evaluated, 148 of them for a forbidden recipient. We release the 225,964 evaluations, the level passports, the scoring code and the analysis script at huggingface.co/datasets/aporthq/vault-benchmark-v1 .