DynEdit: Dynamic Entropy-Guided Sequential Editing for Large Language Models

AI summary

Overview

Research area: Natural Language Processing — knowledge editing for large language models (parameter-modifying / locate-and-edit methods), specifically long-sequence ("long-form") editing.

Technical level: Intermediate. Readers should know roughly how transformer hidden states, next-token probabilities, cross-entropy loss, and BLEU/Rouge-L/BERTScore work, but the paper's central idea (entropy as a difficulty signal) is intuitive.

Scope (1 sentence): The paper proposes DynEdit, a framework that uses token-level predictive entropy to decide where to place editing windows and how strongly to weight individual tokens when rewriting long-form knowledge into an LLM's parameters, and reports benchmark comparisons against ROME, MEMIT, AlphaEdit, UnKE, and AnyEdit.

What This Paper Is About

Updating a single fact ("X is now Y") in an LLM's weights is well studied, but updating a long answer — a derivation, a code snippet, a poem — with the same technique is hard: older methods inject a correction at essentially one position, which cannot supervise a whole multi-token target. Recent sliding-window editors fix this by chopping the target into equal chunks, but they treat every region as equally difficult. DynEdit argues that a target sequence is highly non-uniform — some tokens are easy and predictable, others are uncertain and carry the actual facts — and uses the pre-edit model's own token entropy to spend editing effort where it matters.

Key Contributions

1. An empirical diagnosis. The authors show with preliminary analysis that token entropy under the pre-edit model correlates positively with editing difficulty, and that high-entropy tokens are predominantly content words (entities, nouns, verbs, adjectives, adverbs) while low-entropy tokens are mostly function words (conjunctions, prepositions, pronouns, auxiliaries). They use this to name two failure modes of prior work: difficulty-agnostic editing allocation and token-level optimization imbalance under fixed chunking.
2. Entropy-guided non-uniform sliding. Instead of a fixed stride, DynEdit computes the average entropy of the current window, compares it to the sequence mean, and shortens the stride where the local region is harder (a monotone-decreasing scheduling function Φ). Fixed-window editing is explicitly framed as the special case where Φ is constant.
3. Entropy-guided token-level optimization. Within each window, tokens get weights ω_t = ω₀ + κ·ĥ_t from min–max normalized entropy, so information-dense tokens are optimized more strongly than function words; the loss is computed over the remaining suffix with weak suffix weights to preserve future continuation. A proposition on "gradient dilution" argues that uniform aggregation lets dispersed background gradients dominate the critical update direction.
4. Broad evaluation. Experiments on UnKEBench, CounterFact, MQuAKE, and EditEverything, with Memit- and UnKE-based backbones, on Llama-3-8B and Qwen-2.5-7B (Qwen-3-8B and AlphaEdit variants in the appendix), plus ablations, general-ability checks on MMLU and IFEval, and length-bucket analysis.

Main Findings

• Long-sequence editing improves consistently: Under the MEMIT backbone, DynEdit beats AnyEdit on nearly all datasets for both base models, with up to +12.5 BLEU on UnKE-Original; under the UnKE backbone, DynEdit* improves over AnyEdit* by up to +3.5 BLEU. For example, on Qwen-2.5-7B / UnKE-Original, BLEU rises from 86.86 (AnyEdit) to 95.96 (DynEdit), and on Llama-3-8B / UnKE-Original from 93.20 (UnKE) to 99.19 (DynEdit*).
• Paraphrase robustness improves most: The authors report gains of up to +14.0 BLEU on UnKE paraphrase and +7.5 BLEU on CounterFact paraphrase over AnyEdit. In Table 1, Llama-3-8B paraphrase BLEU goes from 72.96 (AnyEdit*) to 79.94 (DynEdit*), and Qwen-2.5-7B from 47.17 (AnyEdit*) to 66.13 (DynEdit*). The contribution summary separately cites +18.96 BLEU over AnyEdit* on UnKE-Paraphrase.
• Compositional / multi-hop gains: On MQuAKE, DynEdit reaches 99.98 / 99.99 / 99.96 (BLEU / BERT / Rouge-L) on Llama-3-8B and 99.83 / 99.90 / 99.63 on Qwen-2.5-7B under the UnKE backbone; the paper states up to +8.3 BLEU on difficult MQuAKE and paraphrased long-form settings.
• Diverse formats: On EditEverything, DynEdit beats AnyEdit on all five domains for both models. Llama-3-8B Rouge-L: math 99.31, code 98.93, poetry 97.50, news 97.81, chemistry 99.83; Qwen2.5-7B Rouge-L: 99.73 / 99.11 / 98.13 / 98.98 / 99.58. BERTScore also improves (e.g., Llama-3-8B math 70.73 → 97.76). The paper highlights up to +27.03 Rouge-L on math and larger gains on structurally complex, semantically dense domains.
• Length robustness: Splitting UnKE targets into four quantiles Q1–Q4, both methods degrade with length, but DynEdit's curve is flatter and position-wise error analysis shows AnyEdit accumulating error toward later tokens while DynEdit stays lower and more stable; the paper cites a +23.88 BLEU gain on the longest bucket.
• Ablation: Removing either dynamic sliding or entropy weighting hurts across datasets and models; removing sliding causes the larger drop, because without adaptive placement the token weighting has less to work with. Ablated variants still beat AnyEdit in most settings.
• General ability is largely preserved: On Llama-3-8B, MMLU is 67.16 (pre-edited), 65.81 (AnyEdit), 66.03 (DynEdit), 65.72 (AnyEdit*), 65.78 (DynEdit*); IFEval (L) is 75.60 / 64.21 / 71.56 / 75.91 / 75.99. On Qwen-2.5-7B, DynEdit* reaches the best MMLU (73.53) and IFEval scores (70.93 strict, 74.55 loose).
• Two motivating findings: Finding 1 — token entropy is positively correlated with editing difficulty, making it a usable proxy for hard regions. Finding 2 — high-entropy tokens are mostly semantically informative content words.

Methodology in Plain English

Start with the standard "locate-and-edit" recipe: to make a model say Y instead of its old answer for prompt X, you add a small learnable vector δ to the hidden state at a chosen layer and position, and fit δ by minimizing the negative log-likelihood of the target tokens. For long targets, editors like AnyEdit slide a window across Y and fit a new δ at the end of each window, replaying earlier edits.

DynEdit changes two things, both driven by one cheap pre-computation: run the unedited model over the prompt plus gold answer, and record the Shannon entropy of its next-token distribution at every target position. This is done once, before any editing.

• Where to edit: For each step, look at the current window, average its entropy, and subtract the target sequence's mean entropy. If the window is harder than average, take a smaller step forward (more overlap, more windows spent there); if it is easier, take a bigger step. Because the step-size function is monotone decreasing in relative difficulty, hard regions automatically get denser coverage and easy regions get skipped through quickly.
• How hard to push each token: Normalize the entropies to [0,1] and give each token a coefficient ω₀ + κ·ĥ_t, so high-entropy tokens contribute more to the loss. Tokens inside the current window get these full weights; tokens beyond it get a weak suffix weight so the edit does not wreck the rest of the answer. A norm penalty on δ (inherited from the backbone editor) keeps the perturbation small.
• Backbone-agnostic: DynEdit only changes how local δ's are scheduled and weighted; the resulting hidden targets are still projected into weights by MEMIT or UnKE. Hence "DynEdit" (MEMIT-based) and "DynEdit*" (UnKE-based) in the tables.

The theoretical justification is a short proposition: if background-token gradients are highly dispersed (low pairwise cosine similarity), uniform averaging lets their orthogonal noise dominate the aggregate and misaligns the update from the critical-token direction; up-weighting critical tokens raises the alignment lower bound.

Why This Matters

Impact on research. The paper reframes long-form editing as a resource allocation problem rather than a purely geometric one, and offers a signal (next-token entropy) that is free, model-internal, and computed once. Because it plugs into existing editors without retraining them, it is a cheap add-on that other editing methods can adopt, and it gives the field a concrete diagnostic (difficulty-agnostic chunking) to test against.

Real-world applications:

• Enterprise knowledge maintenance: correcting outdated multi-sentence policy, product, or compliance answers inside a deployed model without a retraining cycle.
• Factual repair of long technical content: fixing derivations, code snippets, or chemistry procedures where a handful of critical tokens (numbers, identifiers, formulas) determine correctness.
• Multilingual / paraphrased query handling: edits that survive rewording, which matters for customer-facing assistants where users rarely repeat the training phrasing.
• Continual model updates: sequential editing pipelines that need each new edit to land without eroding general capability on MMLU- or IFEval-style tasks.

Industry relevance. The method is a drop-in modification to an existing editing stack, needs only one forward pass to compute entropies, and preserves general benchmarks — the three properties practitioners usually demand before touching a production model. Applications such as long-form QA, code assistants, and domain copilots where answers span many tokens are the natural fit.

Future Directions

• Beyond Shannon entropy as the difficulty signal: the paper uses a single uncertainty measure; other signals (gradient magnitude, attention mass, entity tags) might correlate better or be cheaper, and the relative merits are untested here.
• Tuning and stability of the scheduler: the stride controller Φ and the coefficients ω₀, κ, and the suffix weight ρ are introduced with monotonicity constraints; the appendix advertises a parameter analysis (C.6), but the interaction between window size w, overlap, and the entropy threshold τ is an open design question.
• Efficiency at scale: entropy estimation is argued to be lightweight, and an efficiency analysis is promised in Appendix C.4, but the truncated content does not report wall-clock or memory costs, so the practical overhead versus AnyEdit is not quantified here.
• Stress tests beyond the current suite: batch editing and case studies are relegated to the appendix (C.5, C.7); broader lifelong-editing sequences, adversarial paraphrase sets, and other backbones (only Qwen-3-8B and AlphaEdit appear, in the appendix) remain to be explored.

Target Audience

Graduate students and applied researchers working on model editing, knowledge updating, and continual learning for LLMs; engineers maintaining deployed models who need targeted factual repairs without full retraining; and anyone already using MEMIT-family or sliding-window editors who wants a low-cost upgrade. Readers new to the area will need background in locate-and-edit methods (ROME/MEMIT) and sequential window editing (UnKE/AnyEdit) to follow Sections 3–4.

Authors’ abstract

Knowledge editing aims to update outdated or incorrect knowledge in large language models without full retraining. Existing parameter-editing methods remain limited for long-sequence knowledge: single-point editors provide insufficient coverage for extended targets, while fixed chunk-based editors distribute editing effort uniformly across rigid predefined windows, failing to prioritize difficult, information-dense tokens. In this work, we show that token entropy is closely associated with both editing difficulty and semantic importance: high-entropy regions are harder to edit and often contain key factual content. Motivated by this observation, we propose DynEdit, a dynamic framework for long-sequence knowledge editing. DynEdit uses token-level entropy to guide non-uniform sliding, increasing the coverage of difficult high-entropy regions, and further applies entropy-guided token-level optimization to strengthen updates on information-dense tokens within each editing window. Experiments on three benchmarks and two backbone models show that DynEdit consistently outperforms competitive baselines across diverse long-form editing settings, achieving gains of up to +14.0 BLEU points on challenging paraphrase queries and near-saturated semantic scores on several diverse knowledge domains.