ABF-Bench: A Verifiable Benchmark for Agentic Blockchain Forensics

AI summary

Overview

Research area: LLM agents, agentic benchmark design, and blockchain forensics / anti-money-laundering (cs.CR).

Technical level: Advanced. The paper assumes familiarity with blockchain ledger models (account-based Ethereum and UTXO Bitcoin), DeFi and bridge mechanics, and the design of agent benchmarks with verifiable rewards.

Scope (1 sentence): The paper introduces ABF-Bench, an interactive benchmark that turns evidence-backed cross-chain fund tracing into a verifiable agent task with a metered ledger interface, a deterministic verifier, and a hidden reference graph, and uses it to compare a purpose-built ABF agent harness against general-purpose agents (Codex, Claude Code) and fixed-order tracers.

What This Paper Is About

Blockchain launderers constantly change tactics — splitting funds, swapping through DeFi, bridging across chains, and depositing into mixers or custodial services — so fixed tracing procedures fall behind, yet the public ledger preserves every recorded movement in checkable form. The paper asks whether LLM agents can carry out this investigation adaptively while every claimed fund movement remains verifiable against the ledger records the agent actually retrieved. To answer this, the authors build a benchmark from a real laundering incident in which tasks are well-posed, each reference edge has a retrievable witness, and agents must find the evidence themselves under a query budget.

Key Contributions

1. A task formulation. The authors formulate evidence-backed blockchain fund tracing as an interactive, verifiable agent task in which a solver queries a metered environment and submits operations, each citing the ledger records that support it — opening blockchain forensics to LLM agents as a verifiable domain.

2. The ABF-Bench benchmark and a reusable case-to-task pipeline. A generator turns a real laundering incident into well-posed cross-chain tasks: conservative rules determine each reference graph, tracing halts wherever the records leave a continuation unresolved, and every reference edge has a verified witness retrievable within the task's query budget. The pipeline uses within-ledger and across-ledger iterations plus human review.

3. The ABF agent harness. A harness built around an agent-maintained working memory (findings, leads, notes), memory tools, a submission preview, context compaction, and five short heuristics, evaluated alongside general-purpose agents and fixed-order tracers.

4. A diagnostic evaluation. Experiments test three stated hypotheses — H1 efficiency, H2 adaptability, H3 reliability — using evidence recall against queries spent, claimed operations and custody recognition, and fidelity plus unreported-transfer rates.

Main Findings

• Harness beats general-purpose agents on recall. The ABF agent harness reaches 0.75 edge recall versus 0.47 for Codex with the same model (gpt-5.5) and 0.10 for Claude Code (claude-opus-4-8), with fidelity 0.98, edge precision 0.92, 43k output tokens, $9.4 cost, and 47% of budget used.

• Claude Code's low score comes from unretrieved citations. Its 0.10 is set by the retrieval condition; scored on verdicts alone it reaches 0.54, and the gap is traced to citing records it left unretrieved.

• A breadth-first tracer still wins on final recall. Tracer bfs reaches 0.88, dfs 0.77, and greedy 0.54 recall. The harness beats bfs on only 7 of 31 tasks, and excluding tracer runs that ended without a submission raises bfs only to 0.91.

• Agents stop early. Every agent's recall comes at under half the budget spent (harness 47%, Codex 33%, Claude Code 16%), while the tracers spend nearly all of it (0.92, 0.87, 0.88 budget used). The best-in-hindsight tracer overtakes the harness after 54% of the budget, and the harness leaves 53% of its budget unused on average.

• H1 (efficiency) holds early. At a quarter of the budget the harness reaches evidence recall of 39% against 35% for the best-in-hindsight tracer and still leads at half the budget (70% versus 64% for bfs); both general-purpose agents trail. The harness leads at its own stopping point on 26 of 31 tasks. At full budget, evidence recall is 75% (harness) versus 91% (bfs).

• H2 (adaptability) holds for custody. The harness and agents identify undisclosed custody far more often: the abstract reports 89% versus 0% for the fixed-order tracers, and the body reports that each agent stops at three quarters or more of the inspected custody points, while tracer stops all came with budget spent. bfs instead follows service sweeps, emitting 367 valid edges beyond undisclosed custody and reaching more reference edges into these points (87% versus 63% for the harness).

• Operation recognition is comparable, crossings are not. The harness claims 83% of retrieved operations against 85% for bfs, but Codex claims only 44% of retrieved crossings, failing to carry a deposit memo to the Bitcoin payout.

• H3 (reliability) mostly holds. The harness and Codex reach the fidelity of bfs (0.98), while Claude Code falls to 0.95. Every agent omits at most 2.5% of transfers whose reference evidence it retrieved (harness 2.5%, Codex 1.0%, Claude 1.8%, bfs 0.7%, dfs 1.1%, greedy 0.0%). Every solver has at least 94% of its operations accepted by the verifier.

• Recall degrades with task size and branching. From the smallest to the largest third of tasks by reference edges, the harness drops from 0.82 to 0.69, and both general-purpose agents lose a larger share. Harness recall correlates more strongly with the largest fan-out of a single holding (Spearman ρ = −0.66) than with the number of reference edges (−0.41).

• Run-to-run noise is substantial. Within-task standard deviation of recall is 0.18 for the harness and 0.17 for Codex, while mean recall across rounds agrees within 0.01.

• Possible case-knowledge leakage. The served text omits both Bybit and Lazarus, yet Claude Code names the incident in 21 of 31 surveyed transcripts, versus 2 of 33 for Codex and 0 of 33 for the harness.

Methodology in Plain English

The authors start from a real case — the Bybit theft of February 2025, followed for 90 days across Ethereum and Bitcoin, with seeds taken from the FBI notice and Bybit's published incident addresses. A generator walks the ledger from those seeds: an inner loop per ledger expands the frontier and replays records to decide which holdings contain traced value, and an outer loop links evidenced deposits to payouts across chains, repeating until the two rounds agree. Human reviewers audit the graph and revise the scope and rules, and a task is published only if the verifier accepts a witness for every edge it retains. Of 58 tasks from the authoring pass, 33 pass the release filter; 31 are evaluated (one excluded after repeated environment failures for one arm, one after its amended graph outgrew its published budget).

Each task ships with seeds, block windows, a hop cap, disclosed stops, available actions, submission rules, and a query budget computed from the reference plan alone as B(t) = ⌈2(η(t) + δ(t))⌉. Solvers interact through a shared metered interface where each well-formed query costs 1 and each page of an activity listing counts as one query; malformed calls, submission, and internal computation are free. A solver submits a report grouping claims into operations, each citing ledger locators. A deterministic verifier returns valid, invalid, or indeterminate for each operation purely from the cited records — it never consults the reference graph — and a separate grader then scores recall, fidelity, and edge precision against the hidden reference graph. Crucially, a reference edge counts as recovered only if the report contains it in a valid operation and the run itself retrieved every record it cites. The three solver families — fixed-order tracers (bfs, dfs, greedy) implemented independently from the public specification, the Codex and Claude Code command-line harnesses restricted to benchmark tools, and the ABF agent harness — all receive the same public input and the same budgets.

Why This Matters

Impact on research. The paper adds forensics to the small set of domains where an agent's answer can be checked mechanically without trusting its reasoning, alongside test-verified code fixes, environment state, and captured flags. Because the adversary generates the difficulty for free and the ledger keeps every claim checkable, the verdicts that score agents could also serve as training signals, and the paper documents concrete failure modes (early stopping, citing unretrieved records, losing recall on high-fan-out tasks) that agent designers can target.

Real-world applications:

• Exchange and custodian compliance teams tracing stolen funds after a breach, where each traced hop must be backed by evidence another investigator can re-check.
• Incident response during an active hack, where an agent must decide quickly which outflows to inspect under a limited query budget.
• Law-enforcement and blockchain-analytics casework that needs auditable, citation-backed fund-flow reports rather than opaque risk scores.
• Detection of address-poisoning and look-alike-address scams, which the released tasks include as visible activity.

Industry relevance. The results compare model API cost directly ($9.4 for the harness, $2.7 for Codex, $6.5 for Claude Code on these tasks), and the finding that a fixed breadth-first tracer achieves 0.88 recall at 92% budget use while agents stop at under half the budget frames a practical trade-off between automation cost and completeness. The gap between the harness (0.75) and Codex (0.47) on the same model shows that harness design, not just the underlying model, drives forensic performance.

Future Directions

• Closing the early-stopping gap. Every agent leaves budget unused and branches unexamined — a quarter of reference edges for the harness and more than half for Codex — so better stopping and lead-exhaustion policies are the clearest headroom, since breadth-first tracing shows at least 0.88 is attainable within the same budgets.
• Handling branching rather than size. Recall tracks the largest fan-out of a single holding (ρ = −0.66) more than the number of edges (−0.41), pointing to exploration strategies that manage wide fan-outs rather than merely longer traces.
• Contamination and case knowledge. Claude Code named the incident in 21 of 31 transcripts despite the names being withheld; the authors note a mention only shows the knowledge is available, leaving open how much pretrained case knowledge shapes custody decisions and how to control for it.
• Extending coverage. The construction interface separates case-specific seeds from the tracing procedure, so new cases on supported ledgers need only their own seeds and declaration, while a new ledger or operation type requires new evidence rules and implementation support; the paper also notes that whether the three hypotheses hold for agents in general remains open.

Target Audience

Researchers building and evaluating LLM agents on verifiable, tool-using tasks; blockchain-forensics and anti-money-laundering practitioners interested in auditable, evidence-cited tracing; benchmark designers concerned with task validity, outcome validity, and reproducibility; and compliance or incident-response engineers at exchanges and analytics vendors assessing where agents currently fall short of rule-based tracers.

Authors’ abstract

Blockchain forensics is adversarial and open-ended: money launderers split, swap, and bridge funds, cut their trail at mixers or custodial services, and change tactics from one incident to the next. Investigators must keep adapting their tracing in response. Yet the immutability of public ledgers preserves every recorded movement in public view, making much of the investigation independently verifiable. We introduce ABF-Bench, an interactive benchmark that makes evidence-backed fund tracing a verifiable agent task, in which every claimed movement must cite ledger records the agent retrieved. A case-to-task pipeline builds well-posed tasks across ledgers: conservative rules halt tracing wherever the records leave a continuation unresolved, and each task's reference graph is the answer those rules determine. Built from real laundering activity, the tasks combine multi-hop transfers, DeFi exchanges, cross-chain round trips, custodial and mixer deposits, and address poisoning. We also propose the ABF agent harness, built around a working memory, and evaluate it alongside general-purpose agents and fixed-order tracers. The harness reaches 75% edge recall, against 47% for Codex with the same model and 10% for Claude Code, which loses recall to overconfidence: it cites records it left unretrieved. Compared with fixed-order tracers, the harness acquires evidence faster than the best of them early in the budget and recognizes undisclosed custodial services that they walk past (89% vs. 0%), with 98% fidelity. However, agents tend to stop early and lose recall on larger tasks, which holds their final recall below the best tracer. Beyond exposing these gaps, ABF-Bench opens blockchain forensics to LLM agents as a verifiable domain, where the verdicts that score agents also offer potential training signals.