RAT-Bench: A Comprehensive Benchmark for Text Anonymization

AI summary

Overview

• Research area: Natural Language Processing, with a focus on privacy-preserving text processing, named entity recognition (NER), and Large Language Model (LLM) based anonymization.
• Technical level: Intermediate.
• Scope: The paper introduces RAT-Bench, an extendable, multilingual benchmark that evaluates text anonymization tools by the residual re-identification risk that an LLM-based attacker can still exploit, rather than by recall of redacted identifiers.

What This Paper Is About

Text that contains personal information is routinely fed into LLMs for training, fine-tuning, or querying, and is usually scrubbed of identifying information first using tools such as Microsoft's Presidio or Anthropic's PII purifier. These tools have traditionally been judged on whether they remove specific identifiers like names, which leaves unclear whether the remaining text can still be used to re-identify a person. RAT-Bench addresses this gap: it generates synthetic text grounded in real U.S. demographic statistics and measures how much re-identification risk survives after each tool has done its work.

Key Contributions

1. A risk-based benchmark grounded in real demographics. RAT-Bench samples indirect identifiers from U.S. Census Public Use Microdata Sample (PUMS) data—referred to as the 1% PUMS in the introduction and the 5% PUMS in the experimental setup—using 9 attributes also considered by Rocher et al. (2019): state of residence, gender, date of birth, race, marital status, highest level of education obtained, employment status, occupation, and citizenship status. This allows residual risk to be computed over the entire U.S. population instead of being reported as recall.
2. A three-level difficulty design across three scenarios and three languages. Attributes are stated explicitly in standard form (easy), explicitly in non-standard or obfuscated form (hard), or only implied through context (implicit). Text is generated in medical appointment transcripts, AI chatbot interactions, and meeting transcripts, with a main English set of 100 records plus 50 benchmark entries each in Spanish and Simplified Chinese.
3. An end-to-end evaluation of 12 anonymization tools plus 4 variants of an iterative anonymizer. Tools span NER-based systems (Azure Language Studio, Presidio, Scrubadub, GliNER, UniNER, TextWash), a perturbation method (TEM with epsilon = 11), and LLM-based prompting approaches (the Anthropic PII purifier and the Rescriber prompt applied to GPT-4.1, Gemini 2.5 Flash, and Llama 3.1–8B). The iterative anonymizer from Staab et al. (2025) is tested with four different attribute lists.
4. An open, extendable release. The dataset, code, and a leaderboard are openly accessible, and the authors invite community expansion, in particular to other geographies.

Main Findings

• Anonymization helps, but no tool removes the risk. With no anonymization, re-identification success is 100% for easy, 98% for hard, and 99% averaged across explicitly mentioned identifiers, and 32% for implicit ones. Averaged across tools, those figures fall to 44% (easy), 69% (hard), 57% (average), and 28% (implicit)—still far from negligible.
• Azure reduces risk the most, but by over-redacting. Azure achieves 22% (easy), 33% (hard), 28% (average), and 21% (implicit), but it often removes non-sensitive entities, producing a low BLEU score relative to the original text.
• LLM-based anonymizers offer the better privacy-utility trade-off, at higher computational cost. GPT-4.1 with the Anthropic PII purifier prompt reaches 26% (easy), 58% (hard), 42% (average), and 19% (implicit), the lowest average and lowest implicit risk among the one-shot tools, while preserving more of the surrounding text. Gemini with the same prompt scores 48%, 64%, 56%, and 33%; Llama 3.1–8B with the same prompt scores 37%, 72%, 55%, and 21%; the Rescriber prompt yields 45%, 76%, 61%, and 30% for GPT-4.1 and 36%, 63%, 50%, and 25% for Llama.
• Other NER tools vary widely. Presidio: 41%, 77%, 59%, 30%. Scrubadub: 37%, 89%, 63%, 29%. GliNER: 67%, 71%, 69%, 28%. UniNER: 41%, 81%, 61%, 37%. TextWash: 39%, 55%, 47%, 32%.
• The perturbation-based method performs worst. TEM reaches 90% (easy), 85% (hard), 88% (average), and 31% (implicit).
• Non-standard identifier formats are a major failure mode. Risk rises from 44% when identifiers are written cleanly to 69% when they appear in non-standard or obfuscated form.
• The source of risk shifts with difficulty. For easy identifiers, residual risk generally comes from indirect identifiers because direct identifiers are redacted in most cases. In the hard setting, direct identifiers become the majority source of re-identification for NER-based methods such as Scrubadub and Presidio, while LLM-based methods remain more robust.
• Implicitly conveyed attributes defeat current tools. Anonymizers only marginally reduce risk when identifiers are implied rather than stated, and none perform well.
• Iterative anonymization is highly effective when the attribute list is right—and fragile when it is not. With the exact list of 6 direct and 9 indirect identifiers (Ideal), re-identification falls to 2% (easy), 18% (hard), and 18% (implicit), outperforming every tool in the main table. Extending the list with 10 additional PUMS attribute names (Ideal-extended) gives 8%, 18%, and 10%. Weakening to a random subset of 5 Ideal attributes (Generalization) gives 44%, 64%, and 22%, and the out-of-the-box Presidio attribute list gives 56%, 84%, and 26%—comparable to some one-shot LLM anonymizers but at significantly higher computational cost.
• Language transfer splits by method type. On easy, explicitly mentioned identifiers, Azure scores 16% (English), 14% (Spanish), and 14% (Chinese); GPT-4.1 with the Anthropic prompt scores 24%, 26%, and 24%; Presidio scores 48%, 88%, and 86%. NER-based Presidio almost doubles its risk outside English, while LLM-based anonymizers transfer effectively.
• Risk compounds with more identifiers. With a single easy direct identifier and no indirect identifiers, the best tool (GPT-4.1) misses only 1% of identifiers while the worst (TEM) misses 33%. With 5 direct identifiers, 94% of profiles are re-identified under TEM. With 3, 5, and 9 easy indirect identifiers and no direct identifiers, re-identification rates rise for all methods, reaching 60% for Presidio.
• The risk threshold choice is not decisive. The authors report that the exact value of the threshold has minor impact on the overall success rate of different anonymizers or on their ranking.
• Recall is an inadequate metric. A tool can achieve high recall by removing low-risk attributes while leaving substantial risk from high-entropy attributes such as dates of birth, rare values, or identifying combinations.

Methodology in Plain English

The authors build each benchmark entry from a real demographic record rather than inventing identifiers from scratch.

1. Sample indirect identifiers. For each entry they draw 5 indirect identifiers from the 9 PUMS attributes, ensuring that different identifier combinations are equally represented.
2. Pick a high-risk record. They sample a record from the PUMS dataset and keep it only if its baseline correctness computed with those sampled indirect identifiers exceeds 0.9—that is, if the combination is genuinely identifying.
3. Attach realistic direct identifiers. They generate 6 direct identifiers consistent with the record: name, social security number, credit card number, phone number, address, and email address. Names are drawn from empirical distributions conditioned on gender and age; addresses match the right geography.
4. Select target attributes and generate text. One direct identifier and the sampled indirect identifiers become the target set. A randomly chosen scenario (medical appointment transcript, chatbot interaction, or meeting transcript) and difficulty level determine how GPT-4.1, used as the text generator, writes the text so that the target attributes are conveyed. Texts with explicit identifiers run 750–1000 words; texts with implicit identifiers run 1500–2000 words.
5. Anonymize, then attack. Each text passes through an anonymization tool. Then GPT-4.1, prompted with the attacker template from Staab et al. (2024), guesses the target attributes from the anonymized text. The authors note they also tried later GPT generations but that GPT-5 often refuses to guess an individual's attributes.
6. Score the risk. If the attacker recovers at least one direct identifier, the record is considered re-identified and the risk is set to 1. Otherwise, risk is computed over the correctly guessed indirect identifiers following Rocher et al. (2019)—the probability of being correctly identified in the U.S. population. A record counts as successfully re-identified if that risk exceeds 0.2, corresponding to k = 5 anonymity, in line with recommendations for anonymized data releases. The headline metric is the re-identification success rate: the fraction of records in a benchmark subset whose risk exceeds the threshold.
7. Measure utility. BLEU between the anonymized and original text captures how much surrounding content survives. The authors measure this only on texts containing explicit identifiers, because many tools made minimal redactions on implicit ones.

The authors note that 0.2 is a comparatively lenient threshold, since some guidance places acceptable re-identification risk at 5% or lower.

Why This Matters

Impact on research. Prior text anonymization benchmarks either rely on limited sets of manually annotated attributes that are costly to extend, or use synthetic identifiers that do not reflect real demographic distributions—which means they cannot translate leftover identifiers into a meaningful re-identification risk and instead default to recall. By grounding the benchmark in population statistics and measuring what a strong attacker can infer, RAT-Bench changes what counts as success and exposes failure modes, such as non-standard and implicit identifiers, that recall-based scoring hides.

Real-world applications:

• Healthcare data sharing. Clinical and medical-text de-identification is a long-standing target, and the paper's medical appointment transcript scenario reflects the settings where specialized scrubbers have been developed.
• Chatbot and assistant data. AI companies use user-chatbot interactions to improve models, one of the three scenarios in the benchmark, and this data must be scrubbed before use.
• Meeting and workplace transcripts. Meeting transcripts form the third scenario, a common source of shared internal text in organizational settings.
• Multilingual and cross-jurisdiction data flows. Privacy regimes in the EU, California, Brazil, and Singapore treat anonymous or de-identified data as out of scope of privacy law, so organizations moving data internationally need tools that hold up outside English.

Industry relevance. The evaluated tools include widely deployed commercial and open-source systems—Microsoft's Presidio and Azure Language Studio, Scrubadub, GliNER, UniNER, TextWash, TEM, and Anthropic's PII purifier prompt—so the reported trade-offs map directly onto choices practitioners make today. The paper characterizes a privacy-utility-cost triangle: NER-based tools are lightweight but may over-redact or miss non-standard identifiers, while LLM-based tools are more precise across languages but computationally heavier. The legal framing also matters for compliance teams, since the benchmark's risk threshold follows the k = 5 convention and the paper notes that UK guidance for the motivated intruder test specifically requires considering LLM-driven re-identification risk.

Future Directions

• Generalization to unseen attributes. The iterative anonymizer's performance collapses when its attribute list is noisy or generic, so a key open question is how to specify or discover attribute sets for new settings without degrading to the Generalization and out-of-the-box configurations.
• Handling non-standard and implicit identifiers. Risk rises from 44% to 69% when standard formatting is dropped, and tools barely reduce risk when identifiers are only implied—both areas need new techniques rather than better versions of current ones.
• Making iterative anonymization practical. Ideal iterative anonymization is highly effective but carries a significantly higher computational cost than one-shot methods, so speedup is an explicit recommendation for developers.
• Geographic and linguistic expansion. The authors release RAT-Bench as extendable and encourage community efforts to expand it, in particular to other geographies; the current non-English coverage is Spanish and Simplified Chinese, each with 50 entries and easy explicit identifiers only.

Target Audience

This paper is most useful to NLP and privacy researchers working on de-identification and re-identification risk, engineers and product teams selecting or building anonymization pipelines for sensitive text, and privacy lawyers or compliance specialists who need to justify anonymization claims under regimes such as the GDPR or CCPA. It is also relevant to developers of the evaluated tool classes who want a concrete list of failure modes to design against.

Authors’ abstract

Data containing personal information is increasingly used to train, fine-tune, or query Large Language Models (LLMs). Text is typically scrubbed of identifying information prior to use, often with tools such as Microsoft's Presidio or Anthropic's PII purifier. These tools have traditionally been evaluated on their ability to remove specific identifiers (e.g., names), yet their effectiveness at preventing re-identification remains unclear. We introduce RAT-Bench, a comprehensive benchmark for text anonymization tools based on re-identification risk. Using U.S. demographic statistics, we generate synthetic text containing various direct and indirect identifiers across domains, languages, and difficulty levels. We evaluate a range of NER- and LLM-based text anonymization tools and, based on the attributes an LLM-based attacker is able to correctly infer from the anonymized text, we report the risk of re-identification in the U.S. population, while properly accounting for the disparate impact of identifiers. We find that, while capabilities vary widely, even the best tools are far from perfect in particular when direct identifiers are not written in standard ways and when indirect identifiers enable re-identification. Overall we find LLM-based anonymizers, including new iterative anonymizers, to provide a better privacy-utility trade-off albeit at a higher computational cost. Importantly, we also find them to work well across languages. We conclude with recommendations for future anonymization tools and will release the benchmark and encourage community efforts to expand it, in particular to other geographies.