Low-Rank Ternary Adaptation for Fine-Tuning Transformers

AI summary

Overview

Research area: Parameter-efficient fine-tuning (PEFT) of quantized Transformer models, specifically ternary (1.58-bit) language models and vision transformers.

Technical level: Intermediate. The paper is readable without deep quantization expertise, but its core idea relies on the Kronecker product, low-rank decomposition, and straight-through estimators.

Scope: The paper introduces a fine-tuning adapter whose updates are discrete (keep, zero, or flip the sign of ternary weights), factorized into two small ternary matrices, and mergeable into a ternary model without dequantization or requantization.

What This Paper Is About

Ternary models restrict all weights to {-1, 0, 1}, which corresponds to about log2(3) ≈ 1.58 bits per weight and large memory savings, but existing low-bit LoRA methods cannot directly fine-tune them. Prior approaches either dequantize the low-bit base weights to higher precision before merging an adapter (QLoRA) or update only quantization parameters (QA-LoRA), and neither produces a merged model that stays ternary. This paper proposes a multiplicative ternary adapter that operates natively on ternary weights, remains ternary after merging, and requires no dequantization step.

Key Contributions

1. A ternary multiplicative adaptation that models discrete updates of ternary weights — sign flips or zeroing — directly in the ternary domain for both ternary LLMs and vision transformers.
2. A low-rank Kronecker decomposition of the ternary update into two small ternary matrices, achieving high expressivity while remaining parameter-efficient.
3. An adapter that can be fully merged into the base ternary model, enabling deployment with zero inference-time overhead and no dequantization or requantization.
4. An evaluation across six models spanning language and vision: ternarized Llama-3.2 1B and 3B (PTQ from SpinQuant), three pre-trained ternary LLMs (Falcon-E-1B-Instruct, Falcon-E-3B-Instruct, BitNet b1.58 2B4T), and a ternary ViT-B/16 fine-tuned on ImageNet-100.

Main Findings

• Llama-3.2-1B gains: Against the ternary SpinQuant baseline (1.58-bit), the adapter improves six of nine benchmarks, is equal or at most 0.8 percentage points worse on the other three, raises the average from 33.2 to 35.3, and lowers WikiText-2 perplexity from 86.5 to 44.6. It surpasses the 2-bit SpinQuant baseline on seven of nine benchmarks and nearly matches its 43.3 perplexity while staying ternary.
• Llama-3.2-3B gains: The adapter improves over the ternary SpinQuant baseline on all nine benchmarks, raising average accuracy from 31.9 to 38.3 and reducing perplexity from 45.6 to 22.3. It also outperforms the 2-bit SpinQuant baseline on every benchmark, lowering perplexity from 41.7 to 22.3.
• Outperforms requantized QLoRA: On Llama-3.2-3B fine-tuned on Alpaca, QLoRA merged and requantized to 1.58-bit reaches 37.5 average accuracy and 22.9 perplexity, whereas the proposed method reaches 38.3 and 22.3.
• Improves pre-trained ternary models on GSM8K: Exact-match accuracy rises from 60.1 to 63.0 (+2.9) for BitNet 2B 4T at 4 fewshots, from 52.0 to 55.1 (+3.1) for Falcon Edge 1B at 5 fewshots, and from 65.4 to 66.4 (+1.0) for Falcon Edge 3B at 5 fewshots.
• Beats requantized QLoRA on vision: On a ternary ViT-B/16 fine-tuned on ImageNet-100, the merged ternary model reaches 83.0 Top-1 accuracy versus 78.9 for QLoRA requantized to 1.58-bit, while full ternary fine-tuning reaches 85.7 and QLoRA with 16-bit adapters reaches 85.3.
• Zero-pruned weights cannot be reactivated: Weight-transition analysis over N = 973,078,528 ternary weights (self-attention and feed-forward projection matrices across 16 Transformer layers of Llama-3.2-1B) shows no element ever transitions out of the value 0. All accuracy gains come from reassigning the signs of already non-zero weights, a structural property of the multiplicative update: if a base weight is 0, its updated value is 0 for any adaptation entry.
• Small parameter footprint: The balanced Kronecker factorization for Llama-3.2-1B yields about 0.06% trainable parameters, while still covering every weight in each layer.
• Memory scaling advantage: With FP32 proxies, the adapter uses 8d bytes for parameters, 8d for gradients, and 16d for the optimizer per square layer, versus 8dr, 8dr, and 16dr for LoRA of rank r. Matching the expressivity of the proposed update would require LoRA rank r = d, i.e. quadratic O(d²) adaptation memory, while the proposed method uses O(d).
• FLOPs: The training forward pass costs 2d² operations for a square layer, compared with d²(2r+1) for LoRA. After merging, both methods incur zero extra inference-time FLOPs.

Methodology in Plain English

Rather than adding a continuous, floating-point correction to each weight (as LoRA does), the method multiplies each ternary base weight by a value drawn from {+1, 0, -1}. Multiplying by +1 keeps the weight, by 0 zeroes it out, and by -1 flips its sign. Because all three factors are themselves ternary, the result always stays in {-1, 0, 1}, so the adapted weights remain ternary throughout training and after merging.

The full set of these multiplicative factors would be as large as the weight matrix itself. To avoid that, the authors build it as a Kronecker product of two much smaller ternary matrices, A and B. The Kronecker product expands small factors into a large mask, so the number of trainable values is p·q + r·s, and for square layers with a balanced choice of factors this is just 2d. The Kronecker product also gives the update a high effective rank, since rank(A ⊗ B) = rank(A) · rank(B).

Training a discrete matrix directly is hard, so the method keeps real-valued proxy matrices and projects them to ternary values ({-1, 0, 1}) in the forward pass, using thresholds computed from the absolute mean of each matrix and passing gradients through with a straight-through estimator. The authors also test three initializations that make the adapted weights initially identical to the pretrained ternary weights: All-ones, Balanced (equal numbers of +1 and -1 with sign compensation on the base weights), and Normalized (entries sampled from U(0.6, 1.4) with random signs, normalized to mean absolute value 1). Balanced is used for the PTQ LLM and ViT backbones, Normalized for the GSM8K experiments, and All-ones only in the weight-transformation analysis.

The adapter is applied to all self-attention and feed-forward blocks, excluding the task-specific output head. The PTQ backbones are fine-tuned for one epoch with AdamW at learning rate 1.5 × 10⁻³, linear decay, warmup ratio 0.03, and on-device batch size 16 on a single NVIDIA A40 GPU (48GB). Pre-trained ternary backbones use a learning rate of 1.0 × 10⁻⁴. After fine-tuning, the proxy matrices are discarded and only the merged ternary weights are deployed.

Why This Matters

Impact on research. The paper reframes fine-tuning of ternary models as a discrete, multiplicative problem rather than a continuous additive one, and shows that a merged model can stay ternary. It also provides a concrete comparison table of properties (ternary base weights, dequant-free merge, ternary merged model, update type) that positions QLoRA, QA-LoRA, and this method against each other, and it documents a structural limitation — weights pruned to zero can never be reactivated — that future low-bit adaptation work will have to reckon with.

Real-world applications (plausible implications of 1.58-bit merged models):

• Deploying fine-tuned language models on lower-end hardware, where the paper notes ternary quantization is suitable for throughput and memory efficiency.
• Task specialization of already-quantized models, such as adapting pre-trained ternary backbones to math reasoning tasks as demonstrated on GSM8K.
• Domain adaptation of ternary vision encoders, as shown by fine-tuning a ternary ViT-B/16 on ImageNet-100.
• Memory-constrained serving, given the paper's illustrative comparison of a 16-bit Llama 8B model at 16 GB dropping to a theoretical 1.6 GB at 1.58 bits per weight.

Industry relevance. Because the adapter is merged and then discarded, the deployed model is identical to the original ternary layer in shape, precision, and activations — no runtime overhead and no separate full-precision adapter to carry, unlike QLoRA. The paper is explicit that accuracy gains are achieved without additional parameters or latency at inference time. The 1.58-bit base weights are stored in UINT8 packed format with BFloat16 scales, which fits realistic deployment pipelines.

Future Directions

• Recovering pruned connections. The multiplicative update cannot reactivate weights that quantization set to zero. Since all observed accuracy gains come from sign redistribution among non-zero weights, a natural open question is whether a mechanism for reintroducing sparsity-pruned connections could yield further improvements.
• Extending beyond the tested settings. The paper reports results on two ternarized Llama models, three pre-trained ternary LLMs, and one ternary ViT-B/16. Whether the approach generalizes to other architectures, quantization schemes, and scales is not established in the reported content.
• Choosing factorizations and initializations. The paper fixes one balanced factorization scheme for square and non-square layers and selects different initializations for different experiment tracks. Whether the factorization and initialization can be selected adaptively, or jointly optimized, is not resolved in the reported content.
• Scaling with model size. The largest PTQ backbone evaluated is Llama-3.2-3B; the conclusions are stated as applying to "ternary transformer backbones" generally, but the paper does not report behavior at substantially larger scale.

Target Audience

Researchers and engineers working on parameter-efficient fine-tuning, low-bit and ternary quantization, and efficient deployment of language and vision transformers. It is also relevant to practitioners who need to specialize an already-quantized checkpoint without giving up the memory and compute benefits of the quantized format, and to readers interested in discrete or multiplicative alternatives to standard LoRA-style additive updates.

Authors’ abstract

Ternary transformers offer extreme memory and compute efficiency, but existing low-bit LoRA-based methods cannot directly fine-tune ternary weights. Current approaches either require dequantization, restoring low-bit base weights to higher precision to merge with adaptation weight, or update only quantization parameters, preventing a merged model that remains ternary. We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights such as sign flips or zeroing through a low-rank Kronecker factorization into two small ternary matrices applied element-wise to ternary weights. This design is parameter-efficient and expressive, preserves the ternary domain, and supports direct merging without dequantization. Experiments on six models across language and vision, including ternarized LLaMA-3 1B and 3B and a ternary ViT-B/16, demonstrate that our method recovers much of the performance lost to quantization and outperforms strong low-bit and ternary baselines. Code is available at https://github.com/alexmanoo/ternary_adaptation.