From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning

AI summary

Overview

Research area: Machine unlearning for large language models (LLMs), specifically the automated construction of "forget sets" — the data used both to remove targeted knowledge from a model and to evaluate whether that removal worked.

Technical level: Intermediate. The paper assumes familiarity with fine-tuning objectives (gradient ascent, KL regularization, preference optimization), embedding-space similarity, and unlearning evaluation protocols, but its central idea — where the forget data comes from — is explained in plain terms.

Scope: The paper formalizes two unlearning granularities (domain-level and instance-level) and proposes BiForget, a target-model-guided framework that synthesizes forget sets for both, evaluated on Harry Potter, WMDP (biosecurity and cybersecurity), and TOFU.

What This Paper Is About

Unlearning benchmarks often use forget sets that do not match what a model actually memorized, so claimed forgetting can be overstated or understated. The authors argue that the forgetting scope should cover not just verbatim training text but also semantically equivalent paraphrases, and that this scope differs depending on whether a user wants to erase a broad concept (a "domain") or a specific fact (an "instance"). Their goal is an automated framework that generates forget data aligned with the target model's own internal knowledge, without relying on a separate, stronger external generator.

Key Contributions

1. Formalization of two forgetting granularities. The paper is described as the first to explicitly formalize domain-level forgetting (a coherent semantic domain, defined via a domain indicator function over the corpus) and instance-level forgetting (specific statements such as "Ron is 16 years old"), each with a distinction between a "real" forget set drawn from the pre-training corpus and an "ideal" forget set that additionally includes semantically equivalent variants.
2. The BiForget synthesis framework. An automated, near-zero-human-effort pipeline that uses the target model itself rather than an external generator, combining (i) seed-guided synthesis, (ii) adversarial probing via jailbreaking and membership inference, and (iii) information rephrasing for instance-level variants, with a SimCSE-based convergence criterion for early stopping.
3. A unified quality evaluation suite. Three metrics — relevance (domain-centroid distance measured on t-SNE projections of top-K nearest embeddings, using 1,000 sampled instances per domain), diversity (the remote-clique metric rather than Self-BLEU), and efficiency (number of 128-token chunks).
4. Cross-benchmark validation. Evaluations on Harry Potter, WMDP biosecurity and cybersecurity, and TOFU, with results reported as averages over five independent runs with five random seeds. Code is released at https://github.com/XiaoyuXU1/Biforget.

Main Findings

• Harry Potter quality gains. BiForget_HP achieves the lowest centroid distance (14.94) and highest remote-clique (0.5824) while using 4,122 chunks, versus 36.44 / 0.5277 / 8,401 for HP book and 48.11 / 0.5324 / 20,806 for Textbook_HP. The abstract summarizes this as roughly 20 improvement in relevance, roughly 0.05 in diversity, and halving the data size.
• Biosecurity quality gains. BiForget_bio reaches the best relevance (19.86) and diversity (0.3631) with 9,196 chunks, compared with Official_bio (44.40, 0.1365, 24,453), Textbook_bio (29.71, 0.1534, 20,505), Keyword_bio (44.07, 0.1813, 20,000), and Filter_bio (37.00, 0.3366, 26,105).
• Cybersecurity is a mixed case. BiForget_cyber attains the highest diversity (0.3240) but a larger centroid distance (49.37) than Official_cyber (9.00), which uses 1,000 chunks; Textbook_cyber, Keyword_cyber, and Filter_cyber score 63.43, 84.30, and 57.07 respectively, with Filter_cyber using 92,737 chunks. The authors attribute the gap to weaker target-model domain knowledge in cybersecurity.
• Forgetting efficacy on Harry Potter. Across GA, GA_KL, NPO, NPO_KL, and OBLIVIATE, BiForget yields comparable or better forgetting than HP book and Textbook. Under GA_KL, BiForget reaches utility 20.71 versus 11.98 (HP book) and 9.23 (Textbook); under NPO_KL, 20.77 versus 14.49 and 9.33; under OBLIVIATE, 15.58 versus 9.05 and 5.58. The retrain reference reports VerbMem 14.30, KnowMem 28.90, PrivLeak 0.00, and Utility 74.5.
• WMDP forgetting and utility. With RMU, BiForget lowers WMDP-bio to 26.54 (a 62.7% reduction versus the original model's 71.09) while MMLU is 62.70 (a 1.7% drop). With ELM, WMDP-bio is 29.32 (58.8% reduction) but MMLU drops to 57.27 (10.2% reduction). With OBLIVIATE, WMDP-bio is 24.43 (65.6% reduction) and MMLU 61.02 (4.3% reduction).
• TOFU diversity and instance forgetting. BiForget raises diversity on forget01 from 0.4354 to 0.5471 (+0.1117, +25.66%), on forget05 from 0.5880 to 0.6416 (+0.0536, +9.12%), and on forget10 from 0.5947 to 0.6344 (+0.0397, +6.67%). On forget01 with OBLIVIATE, the combination of F.Q. 0.92 and M.U. 0.65 is described as the optimal trade-off (F.Q. improves from 0.08, M.U. unchanged from 0.65).
• Adversarial probing matters. Ablation on Harry Potter with GA shows PrivLeak of −15.08 for full BiForget, versus −22.66 without jailbreaking (Δ = 7.58), −21.67 without membership inference (Δ = 6.59), and −24.46 without both (Δ = 9.38), where values inside [−5%, 5%] indicate stronger robustness.
• Convergence behavior. Semantic similarity converges as sample size grows for both granularities; instance-level TOFU synthesis often converges within a single round (max_rounds = 1) because rephrasing induces small semantic shifts.
• Parameter settings. Remote-clique stabilizes around (points_per_round, N, M) = (6, 1000, 1000); points_per_round was varied from 2 to 10 and M and N between 200 and 2,000. On a single H100, BiForget takes approximately 18,000 seconds to synthesize the Harry Potter dataset.

Methodology in Plain English

The core idea is to let the model being unlearned write its own exam. Rather than asking a stronger external model to generate forget data — which can mismatch the target model's actual knowledge boundaries — BiForget prompts the target model itself.

For domain-level synthesis, the target model first enumerates domain-relevant seeds (concepts or characters) from prompts. Stage I instantiates QA-style and information-synthesis templates with those seeds, keeping samples classified as in-domain, with decoding temperature varied (T ∈ {0.6, 0.8, 1.0, 1.2}) to promote diversity. Stage II probes for knowledge that ordinary prompting misses: jailbreaking templates elicit safety-sensitive responses, and a membership-inference probe keeps generated QA pairs whose Min-k% token probability exceeds a threshold (k = 20, τ = 0.3). Growth is stopped when embedding-space diversity stops improving, using SimCSE similarity and a threshold ε = 0.001.

For instance-level synthesis, each target statement is used as a seed and the target model is asked to produce semantically equivalent variants differing in perspective, structure, or style, so the model cannot simply suppress a fixed template pattern. A larger diversity batch delays the convergence check so at least one complete round runs before early stopping.

Quality is then judged without assuming any "ideal" forget set: relevance by centroid distance using t-SNE, diversity by remote-clique, and efficiency by chunk count. Downstream unlearning is tested with several algorithms on three settings — Harry Potter (target model muse-bench/MUSE-Books_target), WMDP biosecurity/cybersecurity (Llama-3-8B-Instruct), and TOFU (Llama-3.1-8B-Instruct) — and robustness is further checked against Enhanced GCG adversarial prompts.

Why This Matters

Impact on research. The paper argues that unreliable forget sets are a bottleneck for the entire unlearning literature, since weak benchmark data produces conclusions that either overstate or understate real forgetting. By reframing the target as an "ideal" forget set that includes paraphrases and semantic equivalents, it pushes evaluation away from surface-form removal and toward information-level removal, and it offers a reproducible, automated alternative to expert-curated benchmarks such as WMDP, which required manual collection and filtering.

Real-world applications (as framed by the paper):

• Regulatory compliance, such as responding to the EU's "Right to be Forgotten" by demonstrably removing specific content.
• Privacy protection for personally identifiable or sensitive records, with membership-inference robustness as the success signal.
• Copyrighted content removal, illustrated by the Harry Potter universe example.
• Safety-critical domain restriction, illustrated by biosecurity and cybersecurity knowledge that could be misused.

Industry relevance. Building forget sets by hand is resource-intensive and hard to scale, and using a stronger external generator introduces mismatch and cost. A framework that runs on the target model, reuses prompt templates generated once offline, produces smaller datasets, and reports practical synthesis time (roughly 18,000 seconds on one H100) is directly relevant to teams that must operationalize data-deletion requests and safety evaluations under compute and budget constraints.

Future Directions

• Extending to continual and multi-domain unlearning, where forget requests interact dynamically over time; the current study covers only single-request unlearning.
• Improving synthesis for weak-knowledge domains. Target-model-guided synthesis degrades when the model has limited domain expertise, as the paper reports for cybersecurity; stronger few-shot domain conditioning is suggested as a mitigation, at a cost to scalability.
• Establishing stronger gold-standard references for safety-critical domains, such as retrained models or better aligned domain-specific proxies, to more reliably judge synthesized forget sets.
• Scaling paraphrase coverage. The authors note intent to improve synthesis so it better captures semantically equivalent variants at scale, and to extend BiForget to larger-scale settings.

Target Audience

Researchers and practitioners in LLM safety, privacy, and machine unlearning who need to build or audit forget sets; benchmark designers frustrated by template overfitting in datasets like TOFU; and engineering teams implementing data-deletion or content-restriction requests who want an automated, compute-conscious synthesis pipeline. Readers without prior exposure to unlearning objectives will need to consult the cited fine-tuning methods (GA, NPO, RMU, ELM, OBLIVIATE) for full context.

Authors’ abstract

Although machine unlearning is essential for removing private, harmful, or copyrighted content from LLMs, current benchmarks often fail to faithfully represent the true ``forgetting scope'' learned by the model. We formalize two distinct unlearning granularities, domain-level and instance-level, and propose \BiForget, an automated framework for synthesizing high-quality forget sets. Unlike prior work relying on \emph{external} generators, \BiForget exploits the target model per se to elicit data that matches its internal knowledge distribution through seed-guided and adversarial prompting. Our experiments across diverse benchmarks show that it achieves a superior balance of relevance, diversity, and efficiency. Quantitatively, in the Harry Potter domain, it improves relevance by ${\sim}20$ and diversity by ${\sim}$0.05 while \emph{halving} the total data size compared to SOTAs. Ultimately, it facilitates more robust forgetting and better utility preservation, providing a more rigorous foundation for evaluating LLM unlearning.