Multiplicative Orthogonal Sequential Editing for Language Models

AI summary

Overview

Research area: Knowledge editing (model editing) for large language models, specifically the sequential and batch-sequential editing setting.

Technical level: Advanced. The paper combines empirical model-editing experiments with matrix analysis (Frobenius norm, spectral norm, condition number, Moore-Penrose pseudoinverse, SVD, orthogonal Procrustes).

Scope: This paper diagnoses why repeated additive weight updates destabilize the edited parameter matrix during sequential editing, and proposes a multiplicative orthogonal editing paradigm (MOSE) that preserves numerical stability and is evaluated against six baseline methods on three LLMs.

What This Paper Is About

Most knowledge editing methods modify a model's weights by adding an update matrix to the original parameter matrix. Across many sequential edits, these additions accumulate and push the edited matrix away from its original structure, increasing measures such as the Frobenius norm and condition number and thereby degrading both editing performance and the model's general abilities. This paper asks whether an editing update can be designed so that it does not damage those numerical stability indicators at all.

Key Contributions

1. Diagnosis of the degradation mechanism. The paper analyzes the key factor behind losses in editing performance and general abilities after sequential editing, focusing on deviation of the parameter matrix and its numerical stability. It offers both a statistical comparison against existing additive methods and a mathematical analysis showing that left-multiplying a matrix by an orthogonal matrix does not change its Frobenius norm or condition number.

2. A multiplicative editing paradigm (MOSE). Instead of computing an additive update matrix, MOSE derives a multiplicative update: an orthogonal matrix is optimized and left-multiplied onto the original parameter matrix. The optimization minimizes output error on both the knowledge to be edited and the knowledge to be preserved, and reduces to a constrained least-squares problem with a closed-form solution via SVD.

3. A layer selection algorithm. MOSE identifies the layer with the strongest activation for the target knowledge and then selects the editing layer by a normalized error criterion, and it edits the selected layer together with its two adjacent layers.

4. Empirical validation across models and settings. MOSE is evaluated against six editing methods on three LLMs, with results reported for single-sequential (4000 edits) and batch-sequential (batch size 10, 500 edits) scenarios.

Main Findings

• Overall improvement figures. MOSE achieves a 12.08% improvement in sequential editing performance compared to current methods, while retaining 95.73% of general abilities across downstream tasks.

• Single-sequential editing (4000 sequential edits, Table 1). On CounterFact with LLaMA3-8B, MOSE reaches 0.9887 reliability, 0.9863 generalization and 0.8972 locality, compared with AlphaEdit at 0.9018 / 0.8260 / 0.7831, PRUNE at 0.7886 / 0.7140 / 0.5763, EMMET at 0.6398 / 0.4937 / 0.3253, and RECT at 0.5688 / 0.3288 / 0.2517. ROME and MEMIT both score 0.0000 on all three metrics.

• On ConceptEdit-Inter, LLaMA3-8B. MOSE reports 0.7859 reliability, 0.7275 generalization, 0.6856 locality; AlphaEdit reports 0.7012 / 0.6007 / 0.5212, PRUNE 0.6295 / 0.4414 / 0.3193, EMMET 0.3877 / 0.2258 / 0.1724, RECT 0.3531 / 0.2084 / 0.1502, and ROME and MEMIT again 0.0000.

• On Qwen2.5-7B, CounterFact. MOSE reports 0.9981 reliability, 0.9902 generalization, 0.9098 locality; AlphaEdit 0.9519 / 0.9241 / 0.8418; PRUNE 0.8115 / 0.7860 / 0.6823; EMMET 0.6702 / 0.5589 / 0.4771; RECT 0.6203 / 0.4745 / 0.3582; ROME and MEMIT 0.0000.

• Batch-sequential editing (batch size 10, 500 steps, Table 2). MOSE on LLaMA3-8B with CounterFact reports 0.9422 / 0.9361 / 0.8819 versus AlphaEdit 0.8222 / 0.7835 / 0.7091 and PRUNE 0.7738 / 0.6899 / 0.5190. On Qwen2.5-7B with CounterFact, MOSE reports 0.9775 / 0.9514 / 0.9031 versus AlphaEdit 0.9427 / 0.8911 / 0.8094.

• Numerical stability is preserved by construction. The paper proves that for an orthogonal matrix R and any matrix W, the Frobenius norm satisfies the relation between W' = RW and W, and that the condition number κ2(W') = κ2(W) even when W is not invertible, because the singular values of RW equal those of W and (RW)† = W†R⊤.

• Additive methods destabilize; orthogonal left-multiplication does not. The statistical comparison (LLaMA3-8B, CounterFact) shows ROME and MEMIT cause substantial increases in Frobenius norm and condition number, while RECT and AlphaEdit limit deviation better but still show noticeable degradation as the number of edits grows. Sequentially left-multiplying by random orthogonal matrices avoids significant growth in norms and condition numbers.

• General abilities under sequential editing. Using CounterFact on LLaMA3-8B, the paper reports that with traditional methods general task performance fluctuates significantly and tends to decline as the number of edits increases, whereas MOSE better preserves downstream task performance.

• Representation geometry (t-SNE, LLaMA3-8B). With 2000 edits, the paper caches per-edit increments ΔW_i and compares representations under a single update versus the accumulated update ΔW_total. RECT produces clearly separated distributions between the two regimes, while MOSE produces two distributions that nearly overlap with closely matched marginal statistics.

• Layer selection ablation (LLaMA3-8B, CounterFact, batch-sequential). Editing a predefined layer (matching the ROME baseline layer) yields the weakest performance; editing a single layer chosen by the algorithm improves results; editing the selected layer plus its two neighboring layers (full MOSE) performs best.

• Runtime cost (CounterFact, single A800-80G GPU, Table 3). Average editing time per sample is 12.09 s for ROME, 16.16 s for MEMIT, 12.88 s for RECT, 18.53 s for AlphaEdit, and 18.60 s for MOSE. The paper attributes MOSE's slightly higher cost to SVD computation and describes the trade-off as reasonable.

Methodology in Plain English

The starting point is the observation that repeated additive edits behave like adding noise to a weight matrix that never gets removed. The authors instead treat editing as a rotation of the weight matrix: an orthogonal matrix preserves lengths and angles, so rotating the matrix cannot inflate its norm or condition number.

Concretely, MOSE keeps the same editing objective used by additive batch methods — minimize a weighted sum of (a) the error on the knowledge that should be preserved and (b) the error on the new key-value edits, with a regularization parameter λ balancing the two. The difference is the variable being optimized: a matrix R constrained to satisfy R⊤R = RR⊤ = I, applied by left-multiplication. This converts the problem into a standard orthogonal Procrustes problem. Stacking the preservation term and the edit term into matrices A and B, the optimum comes from the SVD of M = BA⊤ = UΣV⊤, giving the closed-form solution R = UV⊤. The key vectors, edit keys and target values (K0, KE, VE) are obtained using procedures from prior work (ROME and MEMIT).

For layer choice, MOSE first measures which feed-forward layer responds most strongly to the input, then picks the layer minimizing a normalized error term that divides by the spectral norm of that layer's weight matrix and its activation response — so layers are compared on a fair scale. The chosen layer and its two immediate neighbors are all edited.

Why This Matters

The paper targets a practical failure mode of continual knowledge updating: methods that look effective for a handful of edits collapse over thousands of edits. By tying numerical stability of the weight matrix to editing and general-ability retention, it offers a different design axis than previous work (RECT, EAC, PRUNE, AlphaEdit), which stays within the additive framework.

Real-world applications:

• Keeping deployed assistants factually current — updating facts in a model without retraining when the underlying knowledge changes.
• Multi-tenant or frequently revised knowledge bases — where many edits must be applied in sequence and earlier edits must survive later ones.
• Domain adaptation in regulated settings — where the model must absorb new domain facts while retaining general language and reasoning behavior.
• Long-lived model maintenance — where the alternative is expensive periodic retraining after accumulating stale knowledge.

Industry relevance: the 12.08% editing-performance gain and 95.73% general-ability retention are framed as the headline practical numbers, and the reported runtime (18.60 s per sample for MOSE) is a direct measure of deployment cost. Because the method modifies parameters rather than adding external modules, it fits existing locate-then-edit pipelines, though it requires an SVD step per edit.

Future Directions

• Reducing the SVD overhead. The paper notes MOSE is slightly slower than additive baselines because of the singular value decomposition; a cheaper or approximate route to the orthogonal matrix is a natural extension.
• Reporting across all three models. Three LLMs are named (LLaMA3-8B, LLaMA2-13B, Qwen2.5-7B), but the main-text tables show LLaMA3-8B and Qwen2.5-7B; the paper directs readers to the Appendix for more comprehensive results. Broader reporting would clarify whether the effect holds uniformly across model sizes.
• Extending the paradigm beyond the tested edit types and layers. The experiments cover factual datasets (ZsRE, CounterFact) and a conceptual dataset (ConceptEdit), with edits applied to feed-forward layers; whether multiplicative orthogonal updates suit other components or editing types is left open.
• Longer-horizon and larger-batch scaling. The reported settings are 4000 single edits and 500 batch edits at batch size 10; how the numerical-stability guarantee translates to more extreme sequential and batch regimes remains an open question.

Target Audience

Researchers working on knowledge editing, model maintenance (researchers building on ROME, MEMIT, RECT, EMMET, PRUNE, AlphaEdit), interpretability and model-stability researchers who care about weight-matrix conditioning, and engineers who need to update deployed LLMs repeatedly without retraining and want to understand the stability-versus-cost trade-off. Readers should be comfortable with linear algebra at the level of norms, singular values and SVD.

Authors’ abstract

Knowledge editing aims to efficiently modify the internal knowledge of large language models (LLMs) without compromising their other capabilities. The prevailing editing paradigm, which appends an update matrix to the original parameter matrix, has been shown by some studies to damage key numerical stability indicators (such as condition number and norm), thereby reducing editing performance and general abilities, especially in sequential editing scenario. Although subsequent methods have made some improvements, they remain within the additive framework and have not fundamentally addressed this limitation. To solve this problem, we analyze it from both statistical and mathematical perspectives and conclude that multiplying the original matrix by an orthogonal matrix does not change the numerical stability of the matrix. Inspired by this, different from the previous additive editing paradigm, a multiplicative editing paradigm termed Multiplicative Orthogonal Sequential Editing (MOSE) is proposed. Specifically, we first derive the matrix update in the multiplicative form, the new knowledge is then incorporated into an orthogonal matrix, which is multiplied by the original parameter matrix. In this way, the numerical stability of the edited matrix is unchanged, thereby maintaining editing performance and general abilities. We compared MOSE with several current knowledge editing methods, systematically evaluating their impact on both editing performance and the general abilities across three different LLMs. Experimental results show that MOSE effectively limits deviations in the edited parameter matrix and maintains its numerical stability. Compared to current methods, MOSE achieves a 12.08% improvement in sequential editing performance, while retaining 95.73% of general abilities across downstream tasks. The code is available at https://github.com/famoustourist/MOSE.