Per-Axis Weight Deltas for Frequent Model Updates

AI summary

Overview

Research area: Efficient deployment and serving of many fine-tuned large language model variants, specifically 1-bit delta compression of fine-tuned weights relative to a shared base model.

Technical level: Intermediate. The paper assumes familiarity with fine-tuning, quantized checkpoint formats (FP16/BF16, 1-bit masks), and standard zero-shot evaluation of LLMs, but the core idea is simple enough to follow without deep background.

Scope: The paper proposes and evaluates a 1-bit weight-delta representation that adds per-row or per-column FP16 scale vectors learned from a small calibration set, then measures zero-shot accuracy, on-disk artifact size, and cold-start load time against an uncompressed fine-tuned model and a single-scalar 1-bit delta baseline.

What This Paper Is About

Serving many task-specialized fine-tuned models is expensive because each variant is usually stored and loaded as a complete checkpoint, even though fine-tuned weights differ from their base model only by relatively small residuals. The authors ask whether that residual can be stored as a 1-bit sign mask plus a lightweight per-axis scaling vector, so that many variants can share one base model while still reproducing the fine-tuned model's behavior. The goal is to keep storage and cold-start costs low without losing the accuracy of the full fine-tuned checkpoint.

Key Contributions

1. A per-axis 1-bit delta representation. Instead of a single learned scalar per weight matrix, the method stores the sign mask B = sign(W_f - W_b) together with a learned FP16 scale vector that is either a row vector (1 × d_out) or a column vector (d_in × 1), broadcast across the other axis.

2. Dynamic per-layer axis selection. For each target layer the authors instantiate both the row and column variants, train each, and keep whichever achieves the better end-to-end validation loss, producing a compressed student stacked on top of the shared base model.

3. An output-matching calibration procedure rather than weight reconstruction. The scale vectors are fitted to match activations and logits of the fine-tuned teacher using cached (X, Y) pairs from 50 C4 samples per layer, followed by joint end-to-end training of all selected vectors on 150 C4 examples, with the sign masks and base weights frozen.

4. A systems-side delta loader. Packed deltas are transferred in a single operation per module, which the authors report reduces cold-start latency and storage relative to loading a full FP16 checkpoint, while avoiding dense reconstruction at inference.

Main Findings

• Llama pair (Llama-3.1-8B base / Llama-3.1-8B-Instruct target): The Vector (row/col) method reaches an average zero-shot accuracy of 70.23 across ARC-Challenge, ARC-Easy, HellaSwag, PIQA, and Winogrande, versus 69.95 for BitDelta (scalar) and 69.26 for the uncompressed Baseline. Per-benchmark Vector scores are 53.58 (ARC-C), 82.99 (ARC-E), 59.78 (HellaSwag), 80.63 (PIQA), 74.19 (Winogrande).

• Size of the gain for the Llama pair: Vector (row/col) improves the average over the Baseline by 0.97 points and over BitDelta (scalar) by 0.28 points.

• Qwen3 pair (Qwen3-14B-Base / Qwen3-14B): Vector (row/col) averages 71.71 versus 71.05 for BitDelta (scalar) and 71.34 for the Baseline. Scores are 58.70 (ARC-C), 84.34 (ARC-E), 62.07 (HellaSwag), 80.52 (PIQA), 72.93 (Winogrande). Vector improves over the Baseline by 0.37 points, while BitDelta performs 0.29 points below the Baseline.

• Phi-4 pair (Phi-4 / Phi-4-Reasoning): Vector (row/col) averages 71.00, below BitDelta (scalar) at 71.06 but above the Baseline at 70.74. Scores are 55.63 (ARC-C), 82.95 (ARC-E), 59.35 (HellaSwag), 80.85 (PIQA), 76.24 (Winogrande). This is the one pair where the per-axis method does not beat the scalar baseline.

• Where gains concentrate: The authors report gains are consistent on ARC-Challenge, ARC-Easy, and Winogrande; HellaSwag is on par; PIQA shows a small drop versus BitDelta (scalar).

• Storage footprint: The delta artifact for the 8B setting is approximately 2.97 GB on disk, about 5.24× smaller than a full FP16 checkpoint. Table 2 reports 2980 MB for Vector (row/col) and 2974 MB for BitDelta (scalar) on the Llama pair; 4774 MB and 4775 MB (≈6.19× and ≈6.18× smaller) on the Qwen3 pair; 3768 MB and 3760 MB (≈7.78× and ≈7.80× smaller) on the Phi-4 pair.

• Cold-start load time: Under identical allocator/seeds and cold-start conditions on Llama-3.1-8B, the average load time over 10 runs to apply the vector delta on top of the base is 0.80 s, versus 2.08 s to load the entire fine-tuned FP16 checkpoint.

• Axis preference by sub-type: Attention projections (q, v_proj, o_proj) and MLP down_proj tend to select the row axis, while gate_proj and up_proj show a stronger column preference, with k_proj more mixed — the authors attribute this to differing input/output aspect ratios of the corresponding weight matrices.

Methodology in Plain English

The starting observation is that a fine-tuned model's weights are close to its base model's weights, so only the difference needs to be stored. That difference is reduced to its sign — one bit per weight entry, +1 or -1 — which captures direction but not magnitude. To recover magnitude cheaply, the authors attach one small vector of FP16 numbers per layer: either one number per output row or one number per input column, which is multiplied elementwise against the sign mask and added back onto the base weights. Choosing row versus column is done automatically per layer by testing both.

The scale vectors are not computed analytically; they are learned. The authors run the fine-tuned model as a teacher and a partially compressed student, use forward hooks to cache each layer's input (from the student) and output (from the teacher) as BF16 tensors, and fit the scale vectors with AdamW for five epochs under an MSE loss on layer output. A validation shard picks the better axis per layer, and the whole set of selected vectors is then fine-tuned jointly so the stacked student matches the teacher's logits end to end.

Implementation details: Llama-3.1-8B was used with Llama-3.1-8B-Instruct as teacher, spread over two RTX 4090 GPUs; masks stay packed at 1 bit along the input axis, scale vectors are FP16, and base weights are kept as (in, out) BF16 with non-blocking transfers and a single .to(device) per module.

One reporting inconsistency should be noted: the experimental setup text states a learning rate of 1×10⁻⁴ for the vector scales, while the Table 1 caption states 1e-5.

Why This Matters

Impact on research. The paper sits in the continual and compatible foundation model update space, and its claim is deliberately modest but useful: adding per-axis structure to an existing 1-bit delta scheme costs almost nothing in storage and yields measurable accuracy improvements on two of three model pairs tested. It also provides descriptive evidence about which weight axes carry the anisotropic part of a task-induced delta, which is a concrete empirical hook for follow-up work on delta parametrization.

Real-world applications.

• Inference providers that host many domain- or task-specialized variants of a single base model and need to hot-swap them per request.
• Continual adaptation pipelines where new model variants are introduced frequently and per-variant storage and load time dominate cost.
• Multi-tenant deployments where each tenant has a private fine-tune that must be stored alongside a shared base.
• Memory-constrained serving environments, where the delta path's 0.80 s load time versus 2.08 s for a full FP16 checkpoint matters for cold-start behavior.

Industry relevance. The method is described as drop-in, requires minimal calibration data, and avoids dense reconstruction at inference by adding residual terms once for a selected model — properties aligned with deployment constraints rather than training-time research interests.

Future Directions

• Blockwise per-group scaling, which the authors list explicitly as future work, presumably to capture finer-grained variation than a single row or column vector.
• Learning the sign structure itself. The authors state the sign mask is fixed and never learned, and suggest that at aggressive bit budgets learning B may improve downstream performance.
• INT4/FP8 co-design, also named in the conclusion, to combine the delta representation with other low-precision weight formats.
• Broader multi-tenant evaluations. The current evaluation covers three model pairs on five zero-shot benchmarks with a 150/50 C4 calibration budget; the authors call for wider multi-tenant testing.
• Open questions raised by the limitations: whether near-isotropic layer deltas (where a single global scale would suffice) can be detected and skipped, how sensitive the method is to distribution shift between calibration and deployment data, and whether an on-the-fly variant that applies residuals per forward pass is viable without fused GEMM kernels.

Target Audience

Researchers and engineers working on efficient fine-tuned model serving, delta or adapter compression, and continual foundation model updates will find the most value here. It is also readable by practitioners who deploy multiple fine-tuned variants of one base model and care about checkpoint size and cold-start load time, and by students interested in a compact, reproducible example of activation-matching calibration for weight compression. Readers looking for large accuracy gains, throughput measurements during generation, or results beyond zero-shot multiple-choice benchmarks will not find them reported in this paper.

Authors’ abstract

Serving many task-specialized LLM variants is often limited by the large size of fine-tuned checkpoints and the resulting cold-start latency. Since fine-tuned weights differ from their base model by relatively small structured residuals, a natural approach is to represent them as compressed deltas. We propose a simple 1-bit delta scheme that stores only the sign of the weight difference together with lightweight per-axis (row/column) FP16 scaling factors, learned from a small calibration set. This design preserves the compactness of 1-bit deltas while more accurately capturing variation across weight dimensions, leading to improved reconstruction quality over scalar alternatives. From a systems perspective, a streamlined loader that transfers packed deltas in a single operation per module reduces cold-start latency and storage overhead, with artifacts several times smaller than a full FP16 checkpoint. The method is drop-in, requires minimal calibration data, and maintains inference efficiency by avoiding dense reconstruction. Our experimental setup and source code are available at https://github.com/kuiumdjiev/Per-Axis-Weight-Deltas-for-Frequent-Model-Updates.