Parameter-Efficient Adaptation: LoRA, QLoRA, and Adapters

Parameter-efficient methods keep most base weights frozen and train a smaller set of parameters or modules. This can reduce memory, storage, and iteration cost.

The resulting adapter can still alter safety, style, domain behavior, and tool use across many prompts. That is a measurement, not a caution invented for this lesson. In 2023 a quantized LoRA run stripped the safety training out of released chat models. Lermen and two colleagues state it plainly in their abstract: “With a budget of less than $200 and using only one GPU, we successfully undo the safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B and on the Mixtral instruct model.” For the 70B model they report refusal rates of about 1% on two refusal benchmarks. Performance on two general benchmarks was retained. The model kept its capabilities and lost its refusals.

The training run that did this was the cheapest kind in this lesson. Data and release controls therefore remain as important as in full fine-tuning.

The base matrix remains frozen while a trainable update is applied during the forward pass. How much capacity that update needs was measured in the LoRA paper itself, in Table 6 on GPT-3 175B. Adapting both the query and value projections, WikiSQL validation accuracy was 73.4 at r = 1 and 73.5 at r = 64. MultiNLI was 91.3 at r = 1 and 91.4 at r = 64. A rank of one matched a rank of sixty-four. Adapting the query projection alone did not behave the same way: 68.8 at r = 1 on WikiSQL, rising to 70.5 at r = 4.

The authors draw the conclusion themselves, in the section asking what the optimal rank is: “We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient.” Read the two halves of Table 6 together and the ordering is clear. Which modules you adapt carried the result. The rank was the cheaper of the two decisions.

The best option depends on hardware, variant count, serving, and governance. Each row here has a published measurement behind it rather than a reputation.

The adapter row is the oldest. In 2019 adapter modules were tested by transferring BERT to 26 text classification tasks. Houlsby and seven co-authors give the trade in one sentence: “On GLUE, we attain within 0.8% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.” One caution for anyone citing it: the arXiv abstract of the same paper states 0.4% where the proceedings state 0.8%. The discrepancy is visible on the two pages.

Prompt tuning sits further along the same axis. A 2021 paper on prompt tuning put the number in a figure caption: “With a T5 “XXL” model, each copy of the tuned model requires 11 billion parameters. By contrast, our tuned prompts would only require 20,480 parameters per task—a reduction of over five orders of magnitude—assuming a prompt length of 5 tokens.” Its Table 4 gives 20,480 trainable against 11,135,352,832 total, or 0.00018%. Figure 4 keeps task-specific parameters under 0.01% for models over a billion parameters, against 0.1-1% for prefix tuning at inference. That is what "very small trainable state" is worth as a figure. And in the same paper, it is at that scale that the method becomes competitive at all.

Low storage cost can let variants multiply unchecked, and the ceiling is far above most teams' inventories. A serving system called S-LoRA measured it in 2024. Sheng and eleven co-authors write in their comparison with other systems: “Remarkably, S-LoRA can serve 2,000 adapters simultaneously, maintaining minimal overhead for the added LoRA computation. In contrast, vLLM-packed needs to maintain multiple weight copies and can only serve fewer than 5 adapters due to the GPU memory constraint.” On a single A100 (80GB) they improved throughput by up to 30× over HuggingFace PEFT and up to 4× over vLLM with naive LoRA support.

Two thousand adapters on one card is an inventory, not a deployment. Every failure below is a property of inventories rather than of training.

Treat base, tokenizer, adapter, prompt, and routing as one compatibility set.

Step 1 is where rank and target modules are frozen into the record, and Table 6 of the LoRA paper shows why the second matters more than the first. On GPT-3 175B, WikiSQL validation accuracy was 73.4 at r = 1 and 73.5 at r = 64 when the query and value projections were both adapted — a tenth of a point across a sixty-fourfold difference in rank. Adapting the query projection alone started at 68.8 and needed r = 4 to reach 70.5. Rank is cheap to record. The module list is the field that explains a result later.

LoRA, QLoRA, and adapters let a small team experiment and specialize under smaller hardware budgets. They also create more variants and compatibility surfaces to govern. And they are not free in quality. A 2024 head-to-head on programming and mathematics ran instruction finetuning on about 100K prompt-response pairs and continued pretraining on 20B tokens. Biderman and eleven co-authors report the outcome in their abstract: “Our results show that, in the standard low-rank settings, LoRA substantially underperforms full finetuning. Nevertheless, LoRA better maintains the base model's performance on tasks outside the target domain.” They also found full finetuning learning perturbations of rank 10-100× greater than typical LoRA configurations. The constraint that costs in-domain accuracy is the same constraint that limits the forgetting.

The next lesson studies synthetic data. Cheap example generation can accelerate adaptation. But model-generated supervision can reproduce errors and narrow the training distribution.