Technical Deep Dives

Fine-Tuning Foundation Models: LoRA, QLoRA, and Beyond

Master the techniques for adapting large pre-trained models to your specific use case. Covers full fine-tuning, LoRA, QLoRA, adapter methods, and PEFT strategies.

Gabriele Masetti · 2026-08-11

Why fine-tuning needs a cheaper path

Full fine-tuning updates every weight in a pretrained model. For a 7-billion-parameter model trained with Adam in mixed precision, that means roughly 14 GB for BF16 weights, another 14 GB for BF16 gradients, and about 56 GB for the two FP32 Adam moment buffers — on the order of 84–112 GB of GPU memory before activations even enter the picture, depending on how master weights are stored. Scale that to a 65B model and full fine-tuning stops being something you do on a single GPU at all; it becomes a multi-node job with sharded optimizer states.

Parameter-efficient fine-tuning (PEFT) exists because most of that machinery is overkill for adapting a model to a new task. The pretrained weights already encode most of what's needed; the adaptation is usually a low-dimensional correction on top. Low-Rank Adaptation (LoRA), introduced by Edward Hu and coauthors at Microsoft in 2021 (arXiv:2106.09685), formalized that intuition and became the default starting point for efficient fine-tuning.

QLoRA, from Tim Dettmers and coauthors at the University of Washington (arXiv:2305.14314, NeurIPS 2023), then combined LoRA with aggressive quantization to push the same idea onto commodity hardware. Together they reframed fine-tuning from an infrastructure problem into something a single workstation GPU can handle for models up into the tens of billions of parameters.

The LoRA math

LoRA's premise: for a pretrained weight matrix W₀ (shape d × k) inside a linear layer, don't touch W₀ at all. Instead, freeze it and add a parallel low-rank update:

h = W₀x + ΔWx = W₀x + BAx

Here A is a r × k matrix initialized with random Gaussian values, B is a d × r matrix initialized to zero (so ΔW = 0 at the start of training and the model behaves identically to the base model), and r is the rank — typically far smaller than d or k. Only A and B are trained; W₀ stays frozen throughout. The paper also introduces a scaling factor α, so the effective update becomes (α/r)·BAx, which lets you adjust the update's magnitude independently of r during hyperparameter search.

The parameter savings compound quickly. If W₀ is 4096 × 4096 (roughly the attention projection size in a 7B-class model), a full update is about 16.8 million parameters. At r=8, BA is only (4096×8 + 8×4096) ≈ 65,000 parameters — a reduction of roughly 250x for that single matrix. The original paper reports that applying this to GPT-3 175B reduces trainable parameters by 10,000x and GPU memory requirements by 3x relative to full fine-tuning with Adam, while matching or exceeding full fine-tuning quality on RoBERTa, DeBERTa, GPT-2, and GPT-3 benchmarks.

Two properties matter operationally. First, because gradients and optimizer state only need to be tracked for A and B, memory for the trainable-parameter side of the ledger drops by orders of magnitude — the frozen base model's weights still need to reside in memory (or be loaded in quantized form, which is where QLoRA comes in), but there's no gradient or Adam-moment storage for them.

Second, because ΔW = BA has the same shape as W₀, it can be merged back into the base weights after training (W₀ + BA), producing a model with zero additional inference latency and no architectural change. That merging property is why LoRA adapters compose well with serving infrastructure: you can swap adapters at request time without touching the base model, which is the mechanism behind multi-tenant adapter-serving systems like S-LoRA.

Typical hyperparameter choices, per the PEFT library's guidance and community practice (Sebastian Raschka's widely cited experiments among them): r=8 as a reasonable default for straightforward instruction-tuning or stylistic adaptation, r=16–32 for more general-purpose adaptation, and r=64–256 when the target task diverges substantially from the pretraining distribution or the dataset is large and diverse.

Alpha used to be set to twice the rank (α=2r) as a starting point. The reference guidance has since changed. In "LoRA Without Regret" (Thinking Machines Lab, 29 September 2025), John Schulman and colleagues hold α fixed at 32 and tune the learning rate instead, because LoRA's 1/r scaling makes the optimal learning rate roughly rank-independent. In their runs the best LoRA learning rate sits about ten times above the best full-fine-tuning rate on the same task.

Which weight matrices to target matters more than either number. Applying LoRA only to attention query/value projections is cheaper, but the same study found attention-only placement underperforms even at a rank raised to match the parameter count; extending the adapter to all linear layers, and to the MLP up/down projections in particular, is what closes the gap to full fine-tuning.

QLoRA: fitting a 65B model on one GPU

LoRA reduces the trainable parameter count, but the frozen base model still has to sit in memory in some numeric format — normally 16-bit, which for a 65B model is around 130 GB, still out of reach for a single GPU. QLoRA's contribution is to keep the base model quantized to 4 bits while fine-tuning LoRA adapters on top of it, backpropagating through the frozen quantized weights into full-precision (or 16-bit) adapter matrices.

Three engineering pieces make this work without degrading quality:

• 4-bit NormalFloat (NF4). A quantization data type designed around the empirical fact that pretrained neural network weights are approximately normally distributed. Rather than using uniform bins (standard int4), NF4 uses quantile-based bins calibrated to a zero-mean normal distribution, so each bin captures an equal proportion of the weight distribution's probability mass. That is information-theoretically closer to optimal for this specific weight distribution than a generic 4-bit float or integer format.
• Double quantization. The quantization process itself introduces per-block scaling constants (typically one 32-bit float per 64-weight block). QLoRA quantizes those constants too, using an 8-bit representation with a larger block size, cutting roughly 0.37 bits per parameter of overhead on average — meaningful at the scale of tens of billions of parameters.
• Paged optimizers. Built on NVIDIA's unified memory, this feature automatically pages optimizer state between GPU and CPU memory when a GPU would otherwise run out of memory during gradient checkpointing, avoiding hard OOM crashes during training spikes rather than requiring the whole optimizer state to fit statically in VRAM.

The headline result: QLoRA finetunes a 65B parameter model on a single 48 GB GPU while matching full 16-bit LoRA fine-tuning performance — the double-quantized NF4 setup fully recovers the MMLU performance of 16-bit LoRA in the paper's experiments. The paper's Guanaco model family, trained with QLoRA, reached 99.3% of ChatGPT's performance level on the Vicuna benchmark using roughly 24 hours of fine-tuning on a single GPU; a smaller Guanaco variant hit 97.8% in under 12 hours on consumer hardware.

At the small end, the 7B Guanaco model fits in about 5 GB of memory. These are the numbers that made "fine-tune a capable model on a single 3090 or 4090" a realistic weekend project rather than a research-lab exercise.

It's worth being precise about what QLoRA buys you and what it costs: quantizing the base model to 4 bits saves memory but adds a dequantize-on-the-fly step during the forward/backward pass, so QLoRA training is typically slower per step than full-precision LoRA on hardware that has enough memory for the latter. The tradeoff is squarely about fitting large models into small memory budgets, not about raw training throughput.

DoRA and the rest of the adapter zoo

LoRA's constraint — that every update must be expressible as a low-rank matrix — leaves a measurable gap to full fine-tuning on some tasks. DoRA (Weight-Decomposed Low-Rank Adaptation), from Shih-Yang Liu, Min-Hung Chen, and coauthors at NVIDIA (arXiv:2402.09353, an ICML 2024 oral), targets that gap directly. DoRA decomposes each pretrained weight matrix into a magnitude component and a directional component, then fine-tunes the directional component using a standard LoRA update while training the magnitude component separately.

The motivation is that this decomposition better mirrors how full fine-tuning actually updates weights (in an analysis the authors run comparing learning patterns of LoRA versus full fine-tuning), and empirically DoRA consistently outperforms LoRA across LLaMA, LLaVA, and VL-BART benchmarks — commonsense reasoning, visual instruction tuning, and image/video-text tasks — with no added inference cost once merged, since the magnitude/direction decomposition can also be folded back into a dense weight matrix after training.

DoRA isn't the only variant worth knowing. Hugging Face's PEFT library, the de facto reference implementation for this whole family, ships several: LoHa (Low-Rank Hadamard Product) and LoKr (Low-Rank Kronecker Product) borrow decomposition tricks from the diffusion-model adapter literature to squeeze parameter counts further; AdaLoRA allocates rank adaptively across layers based on an importance score rather than using a fixed r everywhere, spending more capacity where the model needs it; X-LoRA composes multiple trained LoRA experts with a routing mechanism, similar in spirit to mixture-of-experts.

All of these interoperate with the same underlying idea — freeze the base, train something small alongside it — but trade off differently on parameter count, expressiveness, and training stability. For most practical work, plain LoRA or QLoRA remains the default; DoRA is worth reaching for when a LoRA run underperforms full fine-tuning by more than you'd like on a task with reasonably diverse data.

Catastrophic forgetting, and why PEFT is partly a mitigation

Fine-tuning on a narrow dataset can degrade a model's broader instruction-following or general knowledge — the classic catastrophic forgetting problem, most visible in full fine-tuning where every weight is free to move and gradient descent has no reason to preserve capabilities the current dataset doesn't reinforce.

A 2024 analysis ("LoRA Learns Less and Forgets Less," arXiv:2405.09673) makes the mechanism explicit: LoRA underperforms full fine-tuning on tasks that require learning substantially new information (like specialized code or new factual domains), but that same constraint — the update lives in a low-rank subspace close to the original weights — also means LoRA forgets less of what the base model already knew.

That framing has since been narrowed rather than overturned. The 2025 Thinking Machines study reports LoRA matching full fine-tuning when three conditions hold together: the adapter covers all linear layers rather than attention alone, the rank is large enough for the dataset (they use r=256 for supervised data at post-training scale, and r=1–32 for reinforcement learning, where a policy gradient carries roughly one bit of information per episode), and the learning rate is raised accordingly. Configured that way, LoRA needs slightly more than two-thirds of the FLOPs full fine-tuning spends per pass.

Two limits survive. A dataset that exceeds the adapter's capacity still pushes LoRA below full fine-tuning, which is the honest version of the 2024 result. And LoRA tolerates large batches worse than full fine-tuning does — Hugging Face's TRL reproduction recommends an effective batch size under 32 — a penalty that raising the rank does not remove.

The tradeoff is legible rather than free: LoRA's parameter efficiency and its resistance to forgetting come from the same source, restricted expressiveness.

Practical mitigations that matter regardless of which adapter method you use: keep epoch counts low (one to three passes over the fine-tuning set is often enough; more risks overfitting to the narrow distribution), mix a slice of general instruction-following data in with domain-specific data so the model keeps rehearsing broader behavior, and evaluate on held-out general benchmarks (not just the target task) before shipping a fine-tuned checkpoint.

None of this is unique to LoRA, but LoRA and QLoRA make the failure mode less catastrophic by construction, since the frozen base weights are always a call away — deleting the adapter or dialing down its merge weight recovers the original model's behavior exactly.

When fine-tuning is the wrong tool

Fine-tuning — even cheap fine-tuning — is not the default answer to "the model doesn't know X" or "the model doesn't behave like Y." Two adjacent tools solve different problems:

• Retrieval-augmented generation (RAG) injects fresh or proprietary information into context at inference time. It's the right choice when the underlying knowledge changes frequently (pricing tables, documentation, inventory), when you need traceable sources for an answer, or when you don't have enough labeled examples to fine-tune well. RAG doesn't touch model weights, so there's no forgetting risk and no retraining cost when the knowledge base updates.
• Prompt engineering / few-shot prompting is cheaper still and often sufficient for format control, tone, or narrow behavioral nudges, especially with capable instruction-tuned base models.

Fine-tuning (via LoRA/QLoRA or otherwise) earns its cost when the target behavior is about how the model responds rather than what it knows — consistent output formatting, domain-specific style or terminology, task specialization (classification, structured extraction) where you need low latency and don't want a long system prompt on every call, or when you're distilling a narrower, cheaper model to match a larger one's behavior on a fixed task.

A reasonable operating heuristic: start with prompting, add RAG when the model lacks retrievable facts it needs, and reach for fine-tuning only once you have enough representative examples (typically hundreds to low thousands for LoRA-scale adaptation) and a behavior that retrieval and prompting genuinely can't fix. It's common in practice for teams to combine all three — a RAG pipeline feeding a LoRA-adapted model that's been prompted with a concise system message — rather than treating them as mutually exclusive.

Tooling in practice

The reference implementation for everything above is Hugging Face's PEFT library, which wraps LoRA, QLoRA (via integration with the bitsandbytes quantization library), DoRA, AdaLoRA, LoHa, LoKr, and X-LoRA behind a consistent API, and produces adapter checkpoints that are just a few hundred megabytes regardless of base model size. bitsandbytes supplies the actual NF4 quantization and 8-bit optimizer kernels QLoRA depends on.

Above that layer sit training frameworks aimed at making the full pipeline (data loading, distributed training, checkpointing) easier to operate:

• Axolotl is YAML-configuration-driven, exposes fine-grained control over nearly every training parameter, and integrates with DeepSpeed and standard distributed data-parallel setups — it's the choice when you need reproducible, version-controlled training configs across a team.
• Unsloth focuses on single-GPU and consumer-hardware efficiency, using custom Triton kernels to accelerate LoRA/QLoRA training and cut memory use substantially versus a stock Hugging Face transformers training loop.
• torchtune, PyTorch's own post-training library, is no longer one of them. Its maintainers announced on 15 July 2025 that active development was stopping, promising critical bug fixes and security patches through the end of 2025 and no new features; the repository now opens with the line "Torchtune is no longer actively maintained: torchtune development wound down in 2025". Existing pipelines still run. New ones should start elsewhere.

Checkpoints are broadly interoperable across the live tools because they all serialize adapters in the Hugging Face transformers/PEFT format underneath — an adapter trained in one framework typically loads and merges cleanly in another, which matters if a team prototypes in one tool and standardizes production training in a different one.

Putting it together

The practical decision tree for adapting a foundation model looks like this: reach for prompting first, layer in RAG when the task needs facts the model doesn't have, and treat fine-tuning as the tool for changing behavior rather than knowledge. Within fine-tuning, LoRA is the default for anything that fits on your available GPUs at 16-bit; QLoRA is the fallback when the base model doesn't fit, at the cost of some training speed; DoRA is worth testing when a LoRA run leaves quality on the table and the extra directional-decomposition overhead is affordable.

The knobs now have an order. Target modules first: cover every linear layer before tuning anything else. Learning rate second, roughly an order of magnitude above what the same run would use for full fine-tuning. Rank third, sized to how much the dataset actually has to teach rather than to a default. Then validate on general-capability benchmarks alongside the target task, to catch forgetting before it ships.