AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis

AI summary

Overview

Research area: Natural Language Processing, specifically LLM evaluation methodology applied to anti-money-laundering (AML) transaction classification on synthetic financial graphs.

Technical level: Advanced. The paper combines survey-sampling theory (Horvitz-Thompson estimation, Neyman allocation), gradient-boosted-tree feature pipelines, and LLM prompting, and it reports confidence-interval coverage and design-based estimators alongside model scores.

Scope (one sentence): The paper introduces AMLworld-Compact, a downsampled evaluation set built with an "HT-Coreset" protocol that retains every illicit test transaction and samples benign transactions by supervised difficulty, reducing LLM evaluation targets by 271x on AMLworld's HI-Small split and 611x on LI-Small while providing weighted estimates of full-test performance.

What This Paper Is About

Evaluating an LLM on an AML graph means sending one graph prompt per transaction, and AMLworld's two smallest test splits each contain over a million transactions, with illicit rates of 0.12% on HI-Small and 0.05% on LI-Small. Prior work evaluated LLMs on small balanced subsets, but a balanced subset changes unweighted precision and F1 because it artificially enriches the rare class. The paper's goal is to build a much smaller evaluation set that keeps every rare illicit case available for diagnosis, corrects for the altered class proportion with importance weights, and still estimates what performance would look like at full-test prevalence.

Key Contributions

1. An evaluation set with a full-test reference. The released sets contain 3,753 HI-Small and 2,268 LI-Small targets, reducing target counts by 271x and 611x. Each target carries inclusion weights, GFP features, and a typed-subgraph prompt, so later models can reuse the same cases and estimate metrics for the same population. The protocol was compared across 12 budgets and 7 alternative samplers, holding out the evaluated booster family over 500 draws per configuration.

2. LLM evaluation and error analysis. Seven open-weight LLMs were run with the original prompts and scored both with HT weights (full-test estimates) and unweighted (subset diagnostics). The paper analyzes where ML and LLM errors overlap and audits written traces.

3. An outcome-stratified trace audit. A 1,000-record audit of HI-Small few-shot outputs, plus a supplementary routing diagnostic (Doubt Triage) showing that the label-derived route, not the LLM, accounts for most of an apparent hybrid gain.

4. Open release. Evaluation sets, construction code, and audit tools are released (repository names given as typhoon-ai/AMLworldCompactEval and github.com/nat-nischw/AMLworld-Compact); code and ML parameters are MIT-licensed and the evaluation dataset uses the Community Data License Agreement, Sharing, Version 1.0.

Main Findings

• Large target reduction with exact recall information. HT-Coreset retains all 1,251 HI-Small and 756 LI-Small illicit edges and all 8 typologies. Retained illicit edges receive weight one, so recall is exact for fixed predictions; precision and F1 are ratio estimates with possible finite-sample bias.

• Smaller point error than uniform sampling under a held-out predictor family. At the released budgets, F1 RMSE was 0.00 to 2.58 percentage points for HT-Coreset versus 4.44 to 7.41 percentage points for uniform benign sampling with HT weights (Table 2, 500 draws per configuration, both LGB-to-XGB and XGB-to-LGB directions on both splits). HT-Coreset had the lowest F1 RMSE in all 12 dataset-direction-budget comparisons.

• Intervals remain broad even when point estimates improve. Mean 95% confidence-interval widths at the released budgets were 27.3 to 55.0 percentage points, and coverage of full-test F1 was 99.6% to 100% across configurations. The authors describe these intervals as conservative.

• LLM weighted precision sits near the predict-all baseline. On HI-Small, HT-weighted F1 ranged from 0.230% to 0.253% against a predict-all-illicit reference of 0.246%; precision lift was 0.94 to 1.03. Unweighted precision ranged from 31.6% to 34.1%, close to the subset's 33.3% illicit proportion.

• High recall without selectivity. Zero-shot recall reached 97.8% to 100% for six models and 82.9% for Qwen3.5-27B on HI-Small, yet unweighted F1 was 40.7% to 50.6%, below the ML reference's 68.1%.

• Larger models do not consistently score higher. Excluding the context-limited Qwen3.5-27B deployment, HT-weighted F1 varied by 0.017 percentage points across a roughly 5.7x active-parameter range. The 3B-active Qwen3.5-35B-A3B reached 16.5% typology macro-F1, within 1.9 percentage points of the 17B-active Qwen3.5-397B at 18.4%.

• Demonstrations help typology more than detection. Few-shot typology macro-F1 exceeded zero-shot for all seven models on HI-Small and six on LI-Small, with a mean HI-Small gain of 5.3 percentage points. In contrast, mean unweighted F1 fell by 1.6 percentage points (5/7 models decreased) while HT-weighted F1 rose by 0.002 percentage points (5/7 increased).

• Lexical markers do not establish correctness. In the 1,000-record HI-Small few-shot audit (from the run labeled 42, including 148 empty traces), revised lexical checks passed for 84.5% of correct binary verdicts (397/470) and 56.2% of incorrect verdicts (298/530). On the first 8,000 characters, matching the API judges' window, the rates were 82.6% and 55.7%.

• A hybrid routing gain traced to the route, not the LLM. Doubt Triage, which consults an LLM only on ML-negative edges in the weight-one stratum, reached 87.3% zero-shot detection F1 on HI-Small (versus 68.1% for ML) and 84.9% on LI-Small (versus 28.6%). An always-illicit answer on the same route, with no LLM calls, also reached 87.3% and 84.9%.

• False positives dominate LLM errors. On HI-Small few-shot cases, false positives accounted for 38.7% to 63.6% of cases across models, while false negatives accounted for at most 4.7%. Qwen3.5-27B returned no response on 38.7% of cases, mostly due to its deployed context budget.

• A target-specification issue in the original prompts. The evaluated prompts retain a literal <ID> placeholder and do not explicitly identify the target transaction. The target was absent from 324 of 3,753 HI-Small and 297 of 2,268 LI-Small contexts. Among target-present benign cases (2,306 HI-Small, 1,396 LI-Small), five-run mean flag rates across the seven few-shot models ranged from 61.4% to 97.0% and 77.1% to 98.5% respectively.

Methodology in Plain English

The starting point is AMLworld's smallest test partitions (1,015,669 edges for HI-Small and 1,384,810 for LI-Small), partitioned 60/20/20 in file order. Instead of building a balanced subset, the authors keep every illicit edge and spend the entire sampling budget on benign edges, using a ratio of one illicit edge to two benign edges. To decide which benign edges matter, they train LightGBM, XGBoost, and GCPAL on GFP features and average their scores across three scorers and five seeds. Benign edges with high mean scores ("hard negatives," at least 0.40 on HI-Small or 0.24 on LI-Small, half the ensemble's operating thresholds) are likely false positives, so up to 30% of the benign budget is reserved for them; the rest is sampled from three difficulty terciles. Every benign edge gets a positive chance of selection, which allows Horvitz-Thompson weights (the inverse of each edge's inclusion probability) to convert confusion counts on the subset into estimates of full-test counts. The same targets and predictions can then be reported two ways: weighted for full-test performance and unweighted for diagnostics on the enriched subset.

For LLM evaluation, each selected target is serialized as text from its source account via breadth-first search to depth k=2, keeping at most 50 unvisited neighbor accounts per expanded account per hop, using the typed-graph template from Pirmorad (2025). Seven open-weight thinking models spanning 3 to 27 billion active parameters across three families were evaluated with 5 stochastic runs each, in few-shot (12 demonstrations: one per typology plus 4 benign) and zero-shot variants. To test whether the sampling design transfers, the construction family and the evaluated family were swapped: strata built from LightGBM scores were used to evaluate XGBoost, and vice versa, with all cutoffs and budgets fixed before 500 draws per configuration.

Why This Matters

The paper addresses an evaluation cost problem that applies to any rare-event classification task: one prompt per rare case is expensive at full-test scale, but naive balanced sampling distorts the very metrics you want to report. AMLworld-Compact offers one construction template for resolving that tension, and it also documents failure modes in the evaluation pipeline itself (targets missing from prompts, lexical checks that pass for wrong answers) that are easy to overlook.

Real-world applications:

• AML and financial-crime compliance monitoring, where suspicious-transaction models operate at illicit rates near 0.1% and precision-recall tradeoffs decide analyst workload.
• Benchmark construction for LLM evaluation in domains with extreme class imbalance, such as fraud, intrusion detection, or rare-disease screening.
• Model and prompt selection for banks deciding whether a vendor's LLM can assist with alert triage, using weighted estimates at full-test prevalence rather than enriched-subset scores.
• Trace auditing and process supervision for regulated AI, where the paper's finding that lexical rubric passes occur in over half of incorrect verdicts is directly relevant to how audit tools should be validated.

Industry relevance: Financial institutions and AML technology vendors face both the compute cost of large-scale model evaluation and regulatory pressure to justify detection claims. The paper reports over-prediction by LLMs and explicitly states that the results do not support LLM-only deployment, which is a realistic framing for compliance teams considering LLMs as an assistive layer on top of existing ML systems. The released weights, case IDs, and code also let third parties reproduce comparisons on a fixed population.

Future Directions

• Paired comparison of original and target-marked inputs. A target-marked version of the prompts is released but has not been evaluated with an LLM. The paper proposes a paired comparison holding models, demonstrations, and scoring fixed, to isolate the combined effect of the input changes (target marking, context-edge membership, endpoint names, target fields, time summaries, and instructions).

• Transfer of the sampling design to unrelated predictors. The held-out-family study tests only two related GFP boosters. Whether HT-Coreset's RMSE advantage holds for predictors that flag benign edges outside the constructor's hard stratum remains open.

• Checking sampling uncertainty against full-test LLM predictions and independent coreset draws. The current uncertainty analysis conditions on the frozen design and the five fixed prediction vectors, and the paper notes this has not been validated against full-test LLM predictions.

• Stability of trace-level findings across runs. The error decomposition and lexical audit use the available run, and the paper states that stability across runs is untested. It also notes that the 28-record human check has low agreement on the Match rule (kappa = 0.10) and cannot independently validate graph grounding.

Target Audience

This paper is most useful to researchers and engineers who build evaluation benchmarks for class-imbalanced tasks, particularly those who need defensible estimates of full-population performance from small, weighted subsets. It is also relevant to applied NLP practitioners evaluating LLMs on structured or graph-derived inputs, to AML and fraud analytics teams weighing LLM assistance against existing ML pipelines, and to anyone designing rubric-based audits of model reasoning traces, since the paper shows that such rubrics pass for a majority of correct answers but also for over half of incorrect ones.

Authors’ abstract

Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval and https://github.com/nat-nischw/AMLworld-Compact.