Rethinking Benign Relearning: Syntax as the Hidden Driver of Unlearning Failures

AI summary

Overview

Research area: Machine unlearning for large language models, specifically the robustness failure mode known as benign relearning.

Technical level: Intermediate. The paper is readable without deep unlearning background, but it assumes familiarity with fine-tuning, gradient ascent, and preference optimization.

Scope: The paper argues that syntactic (surface-structure) similarity, rather than topical relevance, is the main driver of why forgotten content reappears after benign fine-tuning, and proposes a paraphrase-based fix called syntactic diversification.

What This Paper Is About

Machine unlearning tries to make a model forget a designated "forget set" while keeping its general abilities. But unlearned models can "relearn" forgotten content after being fine-tuned on harmless data that has no explicit overlap with the target content, a phenomenon called benign relearning. This paper challenges the common explanation that topical relevance causes this recovery, and instead shows that shared sentence structure (syntax) is the real driver.

Key Contributions

1. Reassessing topical relevance in BLUR. The authors show that BLUR's conclusion that higher topical relevance yields stronger recovery is confounded by two design choices: the relearn tiers differ in dataset size (so they receive different numbers of gradient updates at a fixed epoch count), and recovery does not increase monotonically with training. Under a standardized step budget with evaluation at every step and maximum-score reporting, the advantage of topically relevant data largely disappears.

2. Identifying syntactic similarity as the primary driver. Using controlled relearn sets on TOFU, the authors show that a syntactically similar set (same surface structure, different entities) consistently produces more recovery than a topically relevant set (same entities, different question format) across GA, NPO, and SCRUB.

3. Mechanistic explanation via representation, gradient, and loss-ratio analysis. Syntactically similar relearn sets sit closer to the target set in hidden-state representations and induce gradient directions more aligned with target fine-tuning. A loss-ratio analysis shows unlearning disproportionately suppresses repeated answer templates rather than the actual keywords.

4. Syntactic diversification as a remedy. The authors paraphrase forget queries into heterogeneous structures with GPT-4o before unlearning, which suppresses relearning, accelerates forgetting, and improves retained model utility on Real Authors, World Facts, and the Retain set.

Main Findings

• Topical relevance is not the dominant factor. In the WHP benchmark, the low topical relevance tier (filled with Lorem Ipsum text) achieves recovery similar to both the high and mid tiers. In many cases the mid tier achieves recovery nearly comparable to the high tier.

• Syntactic similarity ordering conflicts with topical ordering. Reported syntactic similarity scores are: WMDP — D_hi 0.2244, D_mid 0.2059, D_low 0.1771; WHP — D_hi 0.1894, D_mid 0.1767, D_low 0.1818; RWKU — D_hi 0.2250, D_mid 0.2215, D_low 0.1883. In WHP, D_low's syntactic similarity is comparable to D_hi and D_mid, matching its comparable relearning effectiveness.

• Controlled TOFU comparison. The syntactically similar relearn set has syntactic similarity 0.4513 to the target set, versus 0.2349 for the topically relevant set. Across GA, NPO, and SCRUB, the syntactically similar set consistently achieves higher relearn success. Under GA at unlearning step 50, the topically relevant set shows no recovery even after many relearning steps, while the syntactically similar set restores forgotten keywords with only a small number of updates.

• SCRUB is the most relearning-vulnerable method tested. SCRUB suppresses target keywords much earlier than GA and NPO, but the syntactically similar set is able to fully restore the forgotten content.

• Representations and gradients align with syntax. Across GA, NPO, and SCRUB, the syntactically similar set shows substantially higher representation similarity (cosine similarity of average last-token hidden states) and gradient similarity (cosine similarity of average loss gradients) to the target set than the topically relevant set, and this alignment correlates with higher relearn success rates.

• Unlearning suppresses templates, not keywords. The loss ratio (average NLL on template tokens divided by average NLL on keyword tokens) steadily increases during unlearning, because target queries follow rigid surface forms and answers repeat similar templates. These structures are what get suppressed, leaving keywords under-suppressed.

• Diversification works. With the diversified forget set, the average syntactic similarity between the syntactically similar relearn set and the forget set drops from 0.4513 to 0.2241. Models unlearned with the diversified set show no reemergence of target keywords even after 50 unlearning steps across relearning, whereas models unlearned with the original forget set recover rapidly. The loss ratio converges to 1 under diversification.

• Utility improves. On Real Authors, average metric rises from 0.4014 (original forget set) to 0.4852 (diversified); on World Facts from 0.6056 to 0.6104; on the Retain set from 0.1607 to 0.3128.

• Safety training is weaker than unlearning. Methods such as DPO, which suppress outputs with refusals rather than removing knowledge, prove far more vulnerable under syntactic relearning than unlearning methods (reported in Appendix E).

• LoRA-based relearning is a concern. Syntactic relearning vulnerabilities persist under LoRA-based unlearning, and LoRA-based relearning achieves faster and more effective recovery than full-parameter relearning despite far lower resource cost.

Methodology in Plain English

The authors start with an existing benchmark, BLUR, which sorts benign fine-tuning data into three tiers of topical relevance and concludes that more relevant data causes more recovery. They re-run those experiments with gradient ascent, negative preference optimization, and their KL-regularized variants, but fix two flaws: they give every tier the same number of training steps, and they evaluate after every step rather than only at the end of an epoch, reporting the maximum. They measure recovery with ROUGE-L against the base model's answers.

Then they isolate the two candidate drivers. On TOFU, they build two relearn sets that pull in opposite directions: one shares entities with the target set but uses different question formats, the other shares the exact question format but asks about entirely different authors. They quantify surface overlap with a normalized Levenshtein distance, scoring similarity between sentence pairs and averaging across the datasets.

To explain the pattern, they compute cosine similarity between the unlearned model's hidden representations for the target set and each relearn set, and cosine similarity between the loss gradients each set induces. They also split answers into template tokens (generic repeated phrasing) and keyword tokens (the specific name to be forgotten), and track the ratio of the two negative log likelihoods as unlearning proceeds.

Finally, they test the fix: prompt GPT-4o to generate multiple paraphrases of each target query, filter them for quality, and unlearn on this diversified set instead.

Why This Matters

• Research impact: The paper reopens a conclusion treated as settled in the unlearning literature (that topicality drives benign relearning) and offers a mechanistic account — template suppression plus keyword under-suppression — that other robustness work can build on.

• Real-world applications:

  • Copyright takedown pipelines: a model that "forgot" a book passage can reproduce it after ordinary fine-tuning on unrelated text with similar structure.
  • Privacy compliance: personal data removed under regulations may resurface through routine downstream fine-tuning.
  • Fine-tuning-as-a-service platforms: providers must decide whether to reject fine-tuning requests that are topically benign but structurally similar to previously removed content.
  • Safety-critical domain unlearning, such as removing hazardous biological or chemical knowledge in the WMDP setting.

• Industry relevance: The authors note that filtering requests by topical overlap is feasible, but requests carrying syntactically similar yet ostensibly benign data are much harder to detect. Rejecting them degrades user experience; accepting them opens a route for forgotten knowledge to return. The finding that LoRA-based reuse amplifies the vulnerability matters because PEFT is the standard efficient adaptation path in deployment.

Future Directions

• Extending the analysis beyond normalized Levenshtein distance to other structural metrics the paper discusses: template-mining similarity and parse-tree similarity (Appendix I).
• Testing syntactic diversification on more model families and training setups beyond the reported extensions, including the Phi model results in Appendix B.3.
• Evaluating in more realistic unlearning scenarios, which the authors place in Appendix C, rather than the controlled TOFU setting.
• Exploring broader structural factors in data and model design beyond syntax, as the conclusion calls for, to build more resilient unlearning.

Target Audience

Researchers and engineers working on machine unlearning, LLM safety, and fine-tuning infrastructure. Also useful for privacy and compliance teams at organizations that serve models and must reason about whether a removal is durable. Readers seeking a first introduction to unlearning should expect to consult the cited background work on GA, NPO, and SCRUB, which this paper assumes rather than derives.

Authors’ abstract

Machine unlearning aims to remove specific content from trained models while preserving overall performance. However, the phenomenon of benign relearning, in which forgotten information reemerges even from benign fine-tuning data, reveals that existing unlearning methods remain fundamentally fragile. A common explanation attributes this effect to topical relevance, but we find this account insufficient. Through systematic analysis, we demonstrate that syntactic similarity, rather than topicality, is the primary driver: across benchmarks, syntactically similar data consistently trigger recovery even without topical overlap, due to their alignment in representations and gradients with the forgotten content. Motivated by this insight, we introduce syntactic diversification, which paraphrases the original forget queries into heterogeneous structures prior to unlearning. This approach effectively suppresses benign relearning, accelerates forgetting, and substantially alleviates the trade-off between unlearning efficacy and model utility.