EdgeDeID: A 0.6B Clinical De-identifier Trained Without Real Patient Data

AI summary

Overview

Research area: Clinical natural language processing — specifically protected health information (PHI) de-identification of clinical notes using small, locally runnable language models trained entirely on synthetic data.

Technical level: Intermediate. The paper is readable with a working understanding of supervised fine-tuning, LoRA adapters, and reinforcement learning from rewards; the clinical and regulatory framing (HIPAA Safe Harbor) is explained from scratch.

Scope: The paper builds and evaluates EdgeDeID, a 0.6B-parameter generative de-identifier that emits a typed XML schema of identifiers, trains it without any real patient text, and measures its in-domain performance, its transfer to real MIMIC ICU nursing notes, and the value of three reliability add-ons (GRPO, a rule-based QC layer, and a learned verifier).

What This Paper Is About

De-identifying clinical notes by sending them to a hosted language model defeats the purpose, because the unredacted record is disclosed to a third party. The obvious alternative — a model small enough to run on hospital hardware — is weaker and makes more mistakes, and a missed identifier is an irreversible disclosure. The authors ask how far a small model can get when it is trained only on synthetic notes written by other language models, with no real patient data at any stage, and they analyze where that approach breaks down.

Key Contributions

1. EdgeDeID, a 0.6B generative de-identifier trained on notes from three LLM teachers (Claude 3.5 Haiku, DeepSeek-V3, Hermes 3 405B) that are re-annotated into a single convention by a first-pass labeler (Qwen3-1.7B-Base). It reaches 0.994 micro-F1 and a 0.99 zero-leak rate on a 100-note, manually verified synthetic benchmark, against 0.857 F1 and 0.12 zero-leak for the best of eight off-the-shelf systems. The same backbone without fine-tuning scores 0.199. Quantized to 4 bits, it runs on a CPU in 615 MB.

2. A verifier adapter that judges whether a candidate annotation from any system is complete, reaching 0.94 AUROC on the outputs of nine other systems in-domain. Paired tests show that, in-domain, neither GRPO with a recall-weighted reward nor a rule-based detector layer improves a supervised model already at 0.999 recall, and the paper explains why a self-assessment tag trained by reward alone is never emitted.

3. A transfer analysis on MIMIC ICU nursing notes that scores Safe Harbor identifiers separately, showing F1 falls to 0.478 — below an encoder trained on real i2b2 notes (0.675) — and tracing the failures to register (telegraphic style) and low PHI density rather than clinical content. Three hand-written in-context examples recover 0.08 F1 without any annotated target data.

4. An efficiency characterization with single-note latency and memory numbers across all baselines, plus a schema-aware tokenizer extension (69 added tokens) that shortens annotations by 37.5% (536 to 335 tokens on average).

Main Findings

• In-domain performance is near-ceiling, but the benchmark is in-domain. EdgeDeID (SFT) reaches 0.990 precision, 0.999 recall, 0.994 F1, 0.997 F2, 0.986 macro-F1, and misses no PHI in 99 of 100 notes, with a 1.00 parse rate. The best external system, the 1.5B-parameter OpenAI Privacy Filter, reaches 0.857 F1 and 0.12 zero-leak, and no external system exceeds 0.76 recall.

• General-purpose PII filters lack the categories. OPF's eight categories cover no age, place, or device, so any note containing one is a leak; the strongest commercial filter leaves at least one identifier in 88% of benchmark notes, mostly in categories its schema has no label for.

• Training accounts for both accuracy and format. Zero-shot Qwen3-0.6B-Base scores 0.199 F1 and produces parseable output for only 27% of notes.

• GRPO changed nothing in-domain. Because the supervised model already had 0.999 recall and 0.990 precision, nearly every sampled completion received the same reward, and all paired differences from SFT on the benchmark were zero.

• The self-assessment tag was never emitted by either the SFT or the GRPO model, even when the GRPO model was prompted with its own training instructions that make the tag mandatory. Since the tag was absent from supervised targets, almost no sampled completion contained it, the calibration term was zero for every completion in a group, and it contributed nothing to the group-standardized advantage.

• The rule-based QC layer gave no measurable in-domain benefit. Every output already parsed, no note met the criteria for escalation to the NER pass, and pattern detection added one entity that was not PHI; precision changed by −0.001.

• The verifier works in-domain but not out-of-domain. Trained only on synthetic candidates, the SFT adapter reaches 0.987 AUROC on the pooled outputs of eleven systems on the benchmark, and 0.938 on the nine other systems alone. On OPF's outputs alone it reaches 0.789, below the base model without an adapter (0.819). At a threshold of P(incomplete) > 1/3 it flags 75% of outputs containing 93% of the incomplete ones. A GRPO stage on the verdict left AUROC unchanged and worsened calibration (ECE 0.139 to 0.213). On MIMIC it reaches 0.661, versus 0.658 without an adapter.

• Out-of-domain F1 drops sharply. On 2,434 MIMIC ICU nursing notes, SFT reaches 0.478 F1, versus 0.675 for RoBERTa-i2b2. No system is robust across both registers; each performs best on text resembling its own training data. GRPO reaches 0.523, trading recall (−0.052) for precision (+0.112).

• Most MIMIC misses are whole-note failures. Of 713 patient and relative names and dates, 200 are missed; for 155 of these the note's output was empty (103) or unparseable (52). In notes with a well-formed annotation the miss rate is 8.1%. Training notes carry 10.8 PHI mentions per 1,000 characters versus 0.92 in MIMIC, 30% of MIMIC notes are mostly upper case, and 81% of their dates are a bare month and day.

• Controlled edits confirm the register account. With 25 examples each, prepending a "Patient: name" header recovers 76% of missed names, keeping only the target sentence recovers 68%, and title-casing the name recovers 20%.

• False positives are register errors. The model tags "Aline" (arterial line), "Swan" (Swan-Ganz catheter), and "Foley" as names, and reads shift boundaries ("7a-7p"), military times ("2030"), and numeric settings ("10/5") as dates. Ignoring entity type and dropping predicted ages under 90 raises precision only from 0.407 to 0.454.

• Three in-context examples recover part of the gap. F1 rises by 0.083 for SFT (0.478 to 0.562) and 0.084 for GRPO (0.523 to 0.607), both significant; with the QC layer GRPO reaches 0.609, second only to RoBERTa-i2b2. Safe Harbor recall rises for GRPO (0.598 to 0.663) but falls for SFT (0.666 to 0.622).

• The QC layer trades precision for Safe Harbor recall. On SFT it raises Safe Harbor recall from 0.666 to 0.729 and reduces notes with a missed Safe Harbor span from 160 to 148, but precision falls from 0.407 to 0.280: the layer adds 1,376 entities, of which 141 match a gold span. Nearly all come from XML repair, which recovers tags for the 2.6% of unparseable outputs; pattern detection adds 33 entities and the NER fallback runs on one note.

• Efficiency favors encoders despite the small footprint. EdgeDeID is one to two orders of magnitude slower than encoder taggers because it writes output token by token. Served with vLLM on one GPU it takes 0.5 s per note (511 ms) and handles 31 notes per second under 100 concurrent requests; 4-bit llama.cpp on eight CPU threads takes 3.9 s per note and 615 MB, the smallest footprint in the table, versus 12.4 s for OPF on the same CPU. Quantization barely changes accuracy: 0.994 F1 and 0.98 zero-leak versus 0.99 at full precision.

Methodology in Plain English

The authors define the task as detection, not masking: the model must write out a structured XML object listing every identifier it finds in a note, sorted into ten types (name, age, date, place, address, phone, email, id, url, device) chosen to cover the direct identifiers of HIPAA Safe Harbor. A name carries its surface variations, a date its original and normalized forms, and an id a subtype.

Building data without patients. Three LLM teachers with different writing styles write synthetic notes: each receives a target length and a list of identifiers to include, and returns the note with its own annotation. Because the teachers label inconsistently and have blind spots (Hermes 3 tends to miss dates written as DD-MM-YY), their labels are not used directly. Instead a small model (Qwen3-1.7B-Base) is fine-tuned on a small set of teacher-written notes whose labels were cleaned with rules and heuristics, and then used as a first-pass labeler to re-annotate the whole corpus — replacing three conventions with one. 37 augmentations (date formats, name spellings, phone separators) change how an entity is written without changing what it refers to. The result is 25,972 training notes and a held-out synthetic split of 404.

Training. Qwen3-0.6B-Base is fine-tuned on a template with three blocks — instructions, worked examples, and the note — with examples present in 12% of training prompts so the model can also accept few-shot examples at inference. The loss is applied only to the target XML. Because the base tokenizer fragments schema tags into many sub-word pieces, the authors add 69 tokens for the schema's container, content, and attribute tags, initializing new embeddings to the mean of existing ones; this shortens a sample by 15.3% and the generated annotation by 37.5%.

Reinforcement learning. GRPO samples eight completions per prompt and standardizes each reward against the others; a completion that does not parse gets zero. Otherwise the reward is a weighted sum of presence (0.30, fraction of tagged entities appearing verbatim in the note), completeness (0.40, entity-level F2), calibration (0.15, whether a claimed completeness tag is accurate), and length (0.15, full credit to 1,024 tokens), with a KL penalty of 0.04 to the SFT policy.

Three ways to catch a miss. A self-assessment tag written by the extractor itself in the same pass; a deterministic QC layer that repairs malformed XML, adds regex matches for six fixed-format types, and falls back to spaCy NER only on notes that look unreliable; and a learned verifier, a LoRA adapter on the SFT model that reads a note plus a candidate annotation from any system and returns only a verdict, so one forward pass gives P(incomplete). Verifier training candidates are gold annotations corrupted by dropping entities, truncating spans, deleting categories, or adding spurious entities, plus the outputs of four baseline systems.

Evaluation. The in-domain benchmark is 100 synthetic notes generated separately from training, with annotations manually verified so every gold value appears verbatim. Out-of-domain evaluation uses the PhysioNet de-identification corpus: 2,434 ICU nursing notes with 1,778 human-annotated PHI spans, of which the authors count 823 as Safe Harbor identifiers of a patient or relative and 593 as clinician names. Matching is typed and case-insensitive; a gold entity counts as found only when a prediction of the same type fully contains it, while a prediction counts as correct if it overlaps a gold entity at all — a recall-first, per-note protocol.

Why This Matters

The paper's central practical claim is that a model small enough to run inside a hospital, trained without ever touching a real record, can outperform much larger general-purpose privacy filters on the categories clinical de-identification actually requires — and that the remaining gap is a data-coverage problem (writing style and identifier density) rather than a capability ceiling.

Real-world applications:

• Local redaction for institutions that cannot use APIs. Hospitals barred from sending records to third parties, or unwilling to negotiate business associate agreements, can run a 0.6B model on existing CPU hardware in 615 MB.
• Pre-screening before human review. A de-identifier that flags notes for review, paired with the verifier that ranks candidate annotations by P(incomplete), can target scarce expert attention at the outputs most likely to leak.
• High-recall review mode. The QC layer's behavior on MIMIC — raising Safe Harbor recall from 0.666 to 0.729 at a cost of 0.13 precision, mostly by repairing malformed outputs — suits sites that review output rather than release it automatically.
• Building de-identification for new note types or languages. The recipe (multi-teacher generation, consistent relabeling, typed output schema) transfers by changing the generator prompts and the schema, without needing annotated patient text first.

Industry relevance: The result is a direct challenge to the assumption that commercial PII filters suffice for clinical text — the strongest evaluated commercial filter left an identifier in 88% of notes, largely in categories its schema has no label for. It also quantifies the cost of the small-model route honestly: latency one to two orders of magnitude above encoder taggers, and a real out-of-domain failure mode that no amount of in-domain tuning fixed.

Future Directions

• Broaden synthetic coverage across registers and PHI densities. The authors identify this as the main target for future work: the MIMIC failures track telegraphic style and low identifier density, so training notes should include sparse and PHI-free notes, not just dense explicit ones. The controlled edits (header prepending recovers 76% of missed names) suggest what such notes should look like.
• Decide whether a generative model is the right use of synthetic data. The authors did not train an encoder tagger on their synthetic corpus, so it remains open whether a span-labeling encoder would be faster and equally accurate on the same data.
• Make reinforcement learning meaningful at this scale. GRPO only moved the model where there was room to improve (MIMIC), shifting it toward precision. Whether a different reward design, or a larger backbone, would change what the model can detect rather than how it answers, is unresolved.
• Evaluate on standard benchmarks and more institutions. The i2b2/UTHealth corpora were not accessible to the authors, so MIMIC ICU notes are their only real-text evaluation — one note type from one institution, exercising seven of their ten categories. Access to i2b2 and additional note types would test the transfer claim properly.

Target Audience

Clinical NLP and clinical informatics researchers working on de-identification and privacy-preserving text processing; ML engineers at hospitals or health systems who need an on-premises redaction component and are weighing small fine-tuned models against commercial filters; and researchers interested in synthetic-data training and in negative results for reinforcement learning and rule-based post-processing at small model scale. Readers focused purely on state-of-the-art de-identification scores will find the paper more useful for its error analysis and its out-of-domain failure case than for its headline number, which the authors themselves describe as an in-domain upper bound on a 100-note synthetic benchmark.

Authors’ abstract

De-identifying clinical notes with a hosted language model discloses the information the step is meant to protect. We present EdgeDeID, a 0.6B-parameter decoder-only de-identifier that runs locally and is trained without real patient text: three LLM teachers write synthetic notes, a single labeler re-annotates them into one convention, and the model learns to emit a typed XML schema of the identifiers it finds. On a 100-note synthetic in-domain benchmark with manually verified annotations it reaches 0.994 micro-F1 and leaves no missed identifier in 99% of notes, against 0.857 and 12% for the strongest of eight off-the-shelf systems. Quantized, it runs on CPU in 615 MB. Of three additions to supervised training, only a verifier adapter helps: it separates complete from incomplete outputs of other systems at 0.94 AUROC, whereas reinforcement learning with a recall-weighted reward and a rule-based post-processing layer leave in-domain results unchanged. On MIMIC ICU nursing notes F1 falls to 0.48. The errors follow two properties our synthetic notes lack, telegraphic style and low identifier density, and three hand-written in-context examples recover 0.08 F1 without annotated target data.