Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving

AI summary

Overview

Research area: Machine learning systems — specifically parameter-efficient fine-tuning (PEFT) of large language models and the serving infrastructure that runs them.

Technical level: Advanced. The work sits at the intersection of LLM training systems and inference serving, involving kernel-level design, adapter virtualization, and service-level objective (SLO) management.

Scope: The paper describes Loquetier, a framework that runs LoRA fine-tuning and LoRA-based inference inside one shared runtime instead of two separate stacks.

What This Paper Is About

LoRA is a popular, cheap way to adapt a large language model to a new task by training a small set of extra parameters rather than the whole model. In practice, fine-tuning and serving these adapted models are usually done by different systems, and prior work has not fully closed the gap between the two for LoRA-based models. Loquetier's goal is to unify both stages in a single runtime, so that many adapters can live on one shared base model and both the training and inference paths can be executed efficiently together.

Key Contributions

1. A Virtualized Module that isolates PEFT-based modifications, allowing multiple LoRA adapters to coexist on a single shared base model.
2. An optimized computation flow with a custom kernel design that merges the fine-tuning and inference paths during forward propagation.
3. Efficient batching and reduced kernel invocation overhead as a direct result of that merged forward path.
4. A public implementation, released at the project's GitHub repository.

Main Findings

• Unified runtime works: Loquetier integrates LoRA fine-tuning and serving within a single runtime, which the abstract presents as the gap left open by prior integration work.

• Consistent wins across three task settings: The authors report that across three task settings, Loquetier outperforms existing baselines on both performance and flexibility. The abstract does not name the settings or baselines.

• Throughput on inference-only tasks: Up to 3.0× the throughput of the state-of-the-art co-serving system. Absolute throughput figures, hardware, and model sizes are not given in the abstract.

• SLO attainment on unified fine-tuning and inference: 46.4× higher SLO attainment than PEFT. The abstract does not define how SLO attainment is measured here, nor report the underlying latency or deadline targets.

• Details beyond these headline numbers are not in the abstract — no ablation results, no per-task breakdowns, and no comparison table are described.

Methodology in Plain English

Rather than building two separate systems — one for training adapters and one for serving them — the researchers put both into a single runtime. Two ideas do the work. First, adapters are "virtualized": each LoRA adapter's changes are isolated so many of them can share one underlying base model without interfering with one another. Second, instead of writing separate code paths for training and for inference, the framework merges them into one forward pass, implemented with a custom kernel. Because both stages now look the same at the computation level, requests can be batched together more effectively, and the system makes fewer kernel calls overall — which is where much of the efficiency gain is claimed to come from. The team then evaluated this design against existing baselines in three task settings.

Why This Matters

Unifying fine-tuning and serving matters because the standard two-stack setup forces operators to move adapter weights between systems, duplicate the base model in memory, and manage two sets of batching and scheduling logic. Loquetier's claim is that a single runtime with shared base models and a merged computation path removes that duplication and pays off in throughput and in meeting latency targets.

Real-world applications implied by the design:

• Multi-tenant LLM platforms where many customers each need their own adapter over a shared base model.
• Continuous adaptation pipelines, where a model is fine-tuned on fresh data and immediately served without a separate deployment step.
• Latency-sensitive production services that must meet SLOs while mixing training and inference traffic.
• Cost-constrained deployments that want the memory savings of running many adapters on one base model rather than one model copy per task.

Industry relevance: Systems that serve many task-specific model variants are common in commercial LLM offerings, and the reported inference throughput and SLO-attainment gains target exactly the metrics those operators are judged on. A public implementation lowers the barrier to reproducing and adopting the approach.

Future Directions

• Generalizing beyond LoRA: The abstract scopes the framework to PEFT-based modifications; whether the virtualization approach extends to other adapter methods or to full fine-tuning is an open question.
• Scaling behavior: The abstract reports results across three task settings but gives no indication of how the design behaves with far larger numbers of concurrent adapters, longer contexts, or bigger base models.
• The training–inference tradeoff: Merging the two paths helps batching, but how the merged path affects fine-tuning convergence or quality versus a dedicated training stack is not addressed in the abstract.
• SLO policy and scheduling: The abstract reports an SLO attainment improvement but does not describe the scheduling policy behind it, leaving room for work on how requests are prioritized when training and inference contend for the same runtime.

Target Audience

ML systems researchers and engineers working on LLM serving infrastructure, multi-adapter or multi-tenant model deployment, and PEFT. It is also relevant to practitioners deciding whether to consolidate separate fine-tuning and inference stacks, and to readers interested in kernel-level fusion of training and inference computation. Beginners will likely find the systems and kernel terminology difficult without background in LLM serving.

Authors’ abstract

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning (PEFT) technique for adapting large language models (LLMs) to downstream tasks. While prior work has explored strategies for integrating LLM training and serving, there still remains a gap in unifying fine-tuning and inference for LoRA-based models. We present Loquetier, a virtualized multi-LoRA framework that seamlessly integrates LoRA fine-tuning and serving within a single runtime. Loquetier introduces two key components: (1) a Virtualized Module that isolates PEFT-based modifications and supports multiple adapters on a shared base model, and (2) an optimized computation flow with a kernel design that merges fine-tuning and inference paths in forward propagation, enabling efficient batching and minimizing kernel invocation overhead. Extensive experiments across three task settings show that Loquetier consistently outperforms existing baselines in both performance and flexibility, achieving up to $3.0\times$ the throughput of the state-of-the-art co-serving system on inference-only tasks and $46.4\times$ higher SLO attainment than PEFT on unified fine-tuning and inference tasks. The implementation of Loquetier is publicly available at https://github.com/NJUDeepEngine/Loquetier.