Omega-S: A Functional Resilience Index for LLM Fine-Tuning

AI summary

Overview

Research area: Continual learning / catastrophic forgetting in large language model fine-tuning, specifically a regulariser derived from an ecological network index and applied to LoRA adapters.

Technical level: Advanced. The paper works with spectral graph quantities (third spectral moment, Fiedler value, Hutchinson trace estimation), elasticity measurements, exact sign tests and Wilcoxon signed-rank tests, and a controlled design-space enumeration. A reader needs comfort with matrix algebra and basic statistics, though the motivation and headline result are accessible.

Scope in one sentence: The paper introduces Omega-S, a data-free regulariser built from a weight matrix's connectivity graph, shows it preserves more prior-task capability than no regularisation, weight decay or EWC on Llama-3-8B, and then measures that the mechanism does not do what its topological framing implies.

A note on the source: the paper content supplied here is truncated. Several appendices the main text refers to (Appendix A, B.11, B.12, B.13, E, Table 8, Table 11) and the tail of the ablation mean row are not included, so their contents are not reported in this summary.

What This Paper Is About

Fine-tuning an LLM on new data erases some of what it previously learned, a problem called catastrophic forgetting. The standard remedy, Elastic Weight Consolidation (EWC), requires keeping a copy of the old weights and passing previous-task data back through the model to estimate a Fisher matrix, which is often impossible in production because the old data is unavailable, private or expired. The paper asks whether a penalty computed from the current weight matrix alone, with no previous-task data and no stored weights, can preserve prior capability just as well, and it traces where that penalty's actual effect comes from.

Key Contributions

1. A regulariser taken from an ecological index. Omega-S is defined as a penalty on the connectivity graph WWᵀ of a weight matrix, built from Tr(A³) and estimated by Hutchinson's method, giving cost O(N²) rather than O(N³). It requires no previous-task data, no Fisher matrix and no stored copy of the old weights, and adds under 4% to the cost of a training step. The paper defines two forms of the penalty explicitly, a raw trace penalty and a log-ratio composite, and states which experiments used which.

2. A mapped design space rather than an assumed one. Three choices are enumerated: how the graph is built from the weights (six alternatives screened statically), which form of the objective is used, and how the modularity factor is oriented. The variant that looks most promising a priori, a contrast-preserving construction that revives the clustering term, is measured and makes retention worse on all ten seeds.

3. A retention result on Llama-3-8B with LoRA. After a sequential fine-tune from code to prose, the selected variant keeps more code capability than no regularisation on 9 of 10 seeds, beats tuned weight decay on 10 of 10 seeds and tuned EWC on 8 of 10 on the same absolute measure, with every arm re-measured in one session.

4. A mechanism measurement that contradicts the method's name. Although the objective is topological by construction, three of its four factors are numerically inert; the effect is carried by the degree-variance term. The paper reports this as a design shortcoming rather than a detail, and quantifies how much run-to-run variation the setting carries (standard deviation 0.104 in retention ratio under an identical configuration).

Main Findings

• Retention on Llama-3-8B (ten seeds). With LoRA adapters (rank 8, alpha 16) inserted into q_proj and v_proj, fine-tuned from code (CodeAlpaca-20k, 5,000 samples) to prose (Wikitext-2, 5,000 samples) and measured by HumanEval pass@1 (164 problems), Omega-S leaves 0.238 absolute pass@1 after the prose task against 0.173 with no regulariser, a 37.7% relative increase. As a retention ratio the same comparison gives 62.9% to 84.1%. Exact sign test, one-sided p = 0.011; Wilcoxon p = 0.006.

• Comparison against baselines. On the same absolute measure, Omega-S beats tuned weight decay on 10 of 10 seeds (Table 1: p = 0.001 signs / 0.002 Wilcoxon) and tuned EWC (lambda = 10³, swept) on 8 of 10 (p = 0.055 / 0.014). Mean retention against tuned weight decay rises from 61.9% to 84.1%. The paper states Omega-S achieves this while asking for strictly less than EWC, which needs previous-task data; it does not claim lower per-step compute, since the Hutchinson estimate's cost was not benchmarked against EWC's.

• Not equivalent to row-norm balancing. A direct control penalising the variance of row norms, at Omega-S's calibration target, over the same ten seeds in the same session, reaches 17.5% retention against 84.1% and loses on all ten seeds (p = 0.002). Degree variance is correlated with row-norm variance at roughly r ≈ 0.60, leaving roughly 40% of degree variance unexplained by row norms.

• Three of four objective factors are inert. Elasticity with respect to the weights is at or below 10⁻⁴ for the clustering, density and modularity terms, against 9 × 10⁻³ for the degree-variance term, both on base weights and after a full fine-tune. During a run the degree variance falls 5.87% in median while the other three stay under 0.1%. On seed 456, clustering moves from 0.999708 to 0.999702, while degree variance drops from 201.6 to 182.8.

• Why the clustering channel is dead. The pseudo-adjacency is A = σ(|WWᵀ|), and the logistic map compresses all entries into [0.5, 1). The graph is close to an all-ones matrix, so normalised clustering is pinned near its ceiling (C ≈ 0.9997) regardless of the underlying weight structure. The paper names this a design shortcoming: the log map that fixed the raw penalty's scale sensitivity is the map that disabled the topological channel.

• The contrast-preserving reformulation makes things worse. The cosine-affinity construction lifts the triadic excess C/D from 1.0000 (equal to its permutation null) to a median of 1.36, and the cosine composite loses on all ten seeds, 0.547 against 0.841, Wilcoxon p = 0.002. The retention ordering is monotone in how much the clustering channel moves, the opposite of what a topological reading would predict.

• Degree variance means two different things by module shape. The reference implementation applies σ(|WWᵀ|) when W is non-square and σ(|W|) elementwise when W is square. In Llama-3-8B, q_proj is square and v_proj is not, so the reported runs used one module of each kind and two constructions at once. On q_proj, the degree-variance term differs by a factor of roughly 400 between the two branches. Row norms explain 96% of degree variance in layers 8 to 31 of q_proj (magnitude), but only 29% on average across the same layers of v_proj, leaving 71% as alignment structure.

• Which channel carries the effect (ablation). Restricting the penalty to one module type at a time, the alignment channel (v_proj) leads the magnitude channel (q_proj) on 8 of 10 seeds, with a mean of 0.750 for the v-only arm (the q-only mean is cut off in the supplied text). On seed 1011 the v-only arm reaches 1.000 against 0.708 for the full objective.

• Modularity orientation is a wash at ten seeds. The framework defines M as a modularity measure and puts its inverse in the objective; the reference implementation estimated algebraic connectivity, the inverse quantity. As implemented (M ≈ λ₂) gives 76.6% retention, as defined (M ≈ 1/λ₂) gives 84.1%, a paired difference of +0.075 that is smaller than the run-to-run standard deviation of 0.104. Inverting the term changes the scale of the objective from -1.06 to -10.73 on a representative matrix, and the paper attributes the difference to the operating point the penalty gradient calibration lands on rather than to a topological effect. The control: the same inversion under the cosine construction changes nothing (4 of 10 seeds, mean difference +0.010, p = 0.83).

• Run-to-run variation is large. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. The paper reports this because it has not found it quantified for low-rank fine-tuning of language models, and because it bounds the reading of any seed-paired comparison, including its own.

• The margin has the shape of a ceiling. The per-seed advantage over the unregularised arm ranges from -4.4 to +54.3 percentage points, and the gain shrinks as that model's own retention rises (rank correlation ρ = -0.73). The single seed where Omega-S loses to the unregularised

Authors’ abstract

Fine-tuning a large language model on new data degrades what it previously learned. We present Omega-S, a drop-in penalty computed from the weight matrix alone: it needs no previous-task data, no Fisher matrix and no stored copy of the old weights. It is three lines in an existing training loop and adds under 4% to the cost of a step. Retention. On Llama-3-8B with LoRA, fine-tuned from code to prose and measured by HumanEval over ten seeds, Omega-S retains more of the original capability than no regularisation on 9 of 10 seeds (0.173 -> 0.238 absolute pass@1; sign test one-sided p=0.011, Wilcoxon p=0.006), as a retention ratio, 62.9% -> 84.1%. It also beats tuned weight decay on 10 of 10 seeds (p=0.002) and tuned EWC on 8 of 10 (p=0.014), every arm re-measured in the same session. Mechanism, measured rather than asserted. Omega-S is topological by construction, its objective built from Tr(A^3), but we measured which of its four factors actually moves and three do not: their elasticity with respect to the weights is at or below 1e-4, against 9e-3 for the degree-variance term. As implemented, the composite reduces to a penalty on the variance of node degrees, which means row magnitude in square modules and directional alignment in non-square ones. We report this because a method whose name promises one thing and whose gradient does another should say so. We also enumerate the open design choices, including a contrast-preserving construction that does what it was designed to do and makes retention worse on all ten seeds. Repeating an identical configuration, same seed and same hardware, gives a standard deviation of 0.104 in retention ratio. We have not found this quantified for low-rank fine-tuning of language models, and it bounds every seed-paired comparison in this literature, ours included. Code, per-seed results and the full record of negative results are available.