FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic

AI summary

Overview

Research area: Efficient machine learning systems — low-precision (FP8) training and parameter-efficient fine-tuning of large language models (LLMs).

Technical level: Advanced. The paper assumes familiarity with LoRA, floating-point formats, quantization-scaled matrix multiplication, and GPU kernels.

Scope: The paper analyzes why FP8 quantization fails to accelerate LoRA fine-tuning, then proposes FALQON, a framework that merges LoRA adapters into an FP8-quantized backbone to eliminate quantization overhead and deliver up to roughly 3x faster fine-tuning.

What This Paper Is About

LoRA fine-tunes LLMs by training small low-rank matrices, and FP8 arithmetic is theoretically twice as fast as FP16 on modern GPUs. The problem is that FP8's quantization overhead (a max-reduction to compute a scale, then element-wise scaling) scales as O(n^2) while matmul scales as O(n^3), so overhead dominates for the small matrices LoRA uses. The paper's goal is to keep FP8's speed and LoRA's memory savings while removing the redundant quantization steps introduced by LoRA's separate computational path.

Key Contributions

1. Overhead analysis: The authors measure FP8 quantization overhead for LoRA and show that existing FP8 methods target large-dimensional matrix multiplication, so applying them directly to small LoRA adapters yields substantial overhead and limited speedup.
2. Melded LoRA: A framework that merges LoRA adapters directly into an FP8-quantized backbone during fine-tuning, treating the initial quantization error as an implicit LoRA initialization and removing the separate adapter path.
3. Reformulated forward and backward paths: The gradient of the B matrix is rewritten using (Ax)^T, and A is concatenated onto the quantized backbone (W' = [W̃; Ã] ∈ ℝ^(m+r)×n) so that the backbone output and the intermediate activation needed for gradients are produced in a single forward pass without extra quantization.
4. Row-wise proxy update with top-k selection: A proxy buffer stores only changes to B; only the top-k rows with the largest update magnitudes are selectively written into the quantized backbone, avoiding updates that would vanish under low-bit representation.

Main Findings

• FP8 is slower than FP16 for LoRA: In preliminary breakdown analysis using linear layers from LLaMA-7B (self-attention, up-projection, down-projection) at rank r=64 on an RTX 4090, FP8 reduced computation time but quantization overhead dominated, producing approximately three to four times higher latency. FP8 showed consistently lower throughput than FP16 across ranks 16–128 (roughly half), and remained degraded even at ranks 256 and 512. By comparison, TorchAO reports an empirical 1.38x speedup for LLaMA-7B pretraining.
• Where FP8 pays off: FP8 arithmetic only becomes beneficial at larger matrix sizes, empirically around n ≥ 4096, because the O(n^3) matmul cost outweighs the O(n^2) quantization overhead; the paper states the crossover empirically occurs when each matrix dimension exceeds approximately 4K elements.
• Fine-tuning speed: FALQON reached 1.80 s/step for LLaMA-7B on Alpaca versus 5.45 (QLoRA), 9.44 (QA-LoRA), and 8.27 (IR-QLoRA), a 3.02x speedup. On 13B it reached 3.26 s/step versus 9.37, 18.02, and 14.46 (2.87x). On OASST1: 1.79 s/step for 7B (3.04x) and 3.24 s/step for 13B (2.89x).
• Accuracy held roughly level: On Alpaca MMLU average, FALQON scored 0.3491 (7B) and 0.4644 (13B) versus QLoRA 0.3272/0.4443, QA-LoRA 0.3548/0.4729, and IR-QLoRA 0.3388/0.4349. On OASST1, FALQON scored 0.3481 (7B) and 0.4645 (13B) versus 0.3564/0.4605, 0.3609/0.4769, and 0.3605/0.4620.
• Fewer trainable parameters: FALQON used 80M trainable parameters for 7B and 125M for 13B, versus 160M/250M for QLoRA and 89M/140M for QA-LoRA and IR-QLoRA.
• Against FP quantization baselines: In Table 2, FALQON ran at 1.79 s/step with Alpaca MMLU 0.3491 and OASST1 MMLU 0.3481, versus FP16 LoRA at 2.87 s/step (0.3456, 0.3656), TorchAO at 2.18 s/step (0.3393, 0.3452), Fishman et al. at 2.29 s/step (0.3537, 0.3401), and FP6-LLM (E2M3 and E3M2) at 2.72 s/step (0.2295/0.2509 and 0.2308/0.2330).
• Commonsense reasoning: FALQON averaged 0.6320 across ARC-C, ARC-E, BoolQ, HellaSwag, OBQA, PIQA, and Winogrande, versus LoRA FP16 0.6464, QLoRA NF4 0.6438, QA-LoRA INT4 0.6329, IR-QLoRA NF4 0.6463, TorchAO 0.6455, Fishman et al. 0.6447, and FP6-LLM 0.3273 (E2M3) and 0.3188 (E3M2). Gaps on ARC-E and PIQA were within 1–2%, while OBQA showed a more pronounced gap.
• Top-k selection is nearly free: Top-k overhead was +9.97 ms for 7B and +13.85 ms for 13B (under 0.6% of total step time), while total step time fell by 45.89 ms and 97.12 ms (from 1814.98 to 1769.09 ms, and from 3307.48 to 3210.36 ms).
• Cost at scale: On MovieLens-1M with 6,040 users, training LLaMA-7B on 8 GPUs took 35.7 days on RTX 4090 (FALQON) versus 89.3 (QLoRA) and 153.7 (QA-LoRA), 37.7 days on L40S versus 98.3 and 164.0, and 13.3 days on H100 versus 31.1 and 25.1. Cost reductions of 4,030 and 8,357 USD on RTX 4090, 25,057 and 48,533 on L40S, and 27,703 and 19,695 on H100.
• Robustness to hyperparameters: Across learning rates from 2e-1 to 2e-5 and top-k rows of 1, 5, 10, 20, 30, and 50, the best reported MMLU was 0.3491 (lr 2e-3, k=10). Across batch sizes 2–16 and ranks 16–128, metrics ranged narrowly between 0.3418 and 0.3494. Top-k selection overhead was measured at less than 0.6% of total step time.

Methodology in Plain English

The authors start by measuring where FP8 LoRA training actually spends its time, showing that quantizing small adapter matrices costs more than the faster math saves. Their fix is to stop treating LoRA as a separate add-on. When a backbone weight is quantized to FP8, a small error appears between the original weight and the dequantized version. FALQON captures that error with a rank-r SVD and treats it as the initial LoRA adapter, so the quantized backbone already contains the adapter. It then appends the A matrix onto the bottom of the quantized weight matrix, so one large matmul produces both the layer output and the intermediate value needed to compute gradients, removing extra quantizations. Only B remains trainable, and its changes are held in a proxy buffer rather than stored as full LoRA weights; each step, only the top-k rows with the largest update magnitudes are added into the quantized backbone, since small changes would disappear in low-bit representation anyway. They test on LLaMA-7B and 13B fine-tuned on Alpaca and OASST1, evaluated on MMLU and six commonsense benchmarks, using a single 24GB RTX 4090, a Paged AdamW optimizer, batch size 16, learning rate 2e-5, and 1,875 training steps.

Why This Matters

The paper shifts the focus of quantized LoRA research from memory savings to training speed, showing that previous quantized LoRA methods can actually be slower than FP16 LoRA. It also makes an end-to-end FP8 workflow practical, removing the need for post-training quantization and thereby simplifying deployment. Notably, TorchAO and Fishman et al. could not fine-tune the 13B model due to out-of-memory errors because they retain high-precision tensors, while FALQON fit it.

Real-world applications:

• Personalization: Training many independent adapters per user, as in recommendation settings such as RecLoRA and PLoRA.
• Multi-task learning: Building and maintaining task-specific adapters, as in Mixture of LoRA Experts.
• Domain adaptation and multilingual summarization: Producing numerous domain- or language-specific adapters where per-adapter training cost matters.
• Cost-sensitive cloud fine-tuning: Batch-training thousands of adapters on rented GPUs, where the reported per-device cost reductions directly affect budgets.

Industry relevance: The scalability analysis prices training on RTX 4090, L40S, and H100 instances, showing that up to 3x throughput translates into thousands of dollars saved per training run. Because modern GPUs and NPUs already support FP8 natively, the approach targets existing hardware rather than requiring new accelerators.

Future Directions

• The paper explicitly calls for further sensitivity studies on the LLaMA-13B model (Appendix H) and for detailed results at larger LoRA ranks, so the behavior outside the tested ranks 16–128 and the 7B scaling remains incomplete in the main text.
• Extending the analysis beyond the single 24GB RTX 4090 setup is anticipated; the paper references comparisons on other GPUs in Appendix G, but the broader hardware landscape is not fully covered here.
• The per-tensor scaling scheme used here is inherited from standard FP8 practice; whether finer-grained scaling or unit-scaling approaches (cited as related work) combine with melded LoRA is an open question.
• The gradient computation deliberately trains only B and freezes A. The authors cite prior work suggesting this is empirically sufficient, but revisiting whether A should also adapt, and how top-k row selection interacts with accuracy on harder tasks (the OBQA gap being the clearest signal), remains unresolved.

Target Audience

Readers who will benefit most are systems and efficiency researchers working on low-precision training, quantization, and parameter-efficient fine-tuning, along with ML engineers who need to fine-tune many LLM adapters under tight GPU memory and budget constraints. It is most useful to those already comfortable with LoRA internals, FP8 formats such as E4M3 and E5M2, and GPU kernel-level performance analysis.

Authors’ abstract

Low-bit floating-point (FP) formats, such as FP8, provide significant acceleration and memory savings in model training thanks to native hardware support on modern GPUs and NPUs. However, we analyze that FP8 quantization offers speedup primarily for large-dimensional matrix multiplications, while inherent quantization overheads diminish speedup when applied to low-rank adaptation (LoRA), which uses small-dimensional matrices for efficient fine-tuning of large language models (LLMs). To address this limitation, we propose FALQON, a novel framework that eliminates the quantization overhead from separate LoRA computational paths by directly merging LoRA adapters into an FP8-quantized backbone during fine-tuning. Furthermore, we reformulate the forward and backward computations for merged adapters to significantly reduce quantization overhead, and introduce a row-wise proxy update mechanism that efficiently integrates substantial updates into the quantized backbone. Experimental evaluations demonstrate that FALQON achieves approximately a 3$\times$ training speedup over existing quantized LoRA methods with a similar level of accuracy, providing a practical solution for efficient large-scale model fine-tuning. Moreover, FALQON's end-to-end FP8 workflow removes the need for post-training quantization, facilitating efficient deployment. Code is available at https://github.com/iamkanghyunchoi/falqon.