PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

AI summary

Overview

Research area: Efficient inference for large language model (LLM) agent systems, specifically KV cache sharing across multiple LoRA adapters.

Technical level: Intermediate. The paper assumes familiarity with transformer inference, KV caching, prefill/decode phases, and LoRA adapters, but its core ideas (avoid recomputing what another agent already computed, and rebuild a shared cache from adapter-free states) are explained in accessible terms.

Scope: The paper proposes PReCache, a training-free framework with two designs (PreLRShared and ReBaseShared) plus two inference schedules (lazy prefill and double batching), and evaluates them on two 8B backbones, two agent benchmarks, and both single-stream and concurrent serving setups.

What This Paper Is About

Multi-LoRA agent systems share one pretrained backbone model and give each role (such as planning, action, or reflection) its own small LoRA adapter. Because each agent's adapter changes the hidden states, every agent must reprocess the entire accumulated shared trajectory and build its own KV cache, even though previous agents already processed most of that text. This creates large memory and computation redundancy in long-horizon tasks. Existing fixes either require extra training or special architectures, or they leave most of the repeated prefill computation in place. PReCache aims to remove the repeated prefill and keep agent-specific cache behavior accurate, without retraining and without restricting which LoRA adapters can be used.

Key Contributions

1. PreLRShared — a scheme that precomputes a compact agent-specific low-rank (LR) cache for every agent at the moment each segment of shared context is first processed. This replaces later full-dimensional backbone processing over the accumulated context with lightweight rank-r down-projections, eliminating the repeated prefill that LRAgent's BaseShared retains.

2. ReBaseShared — a scheme that reconstructs the shared base cache from adapter-free ("neutral") hidden states rather than from the previous agent's adapter-conditioned hidden states, reducing the current agent's dependence on the previous agent's adapted representation, while keeping LR cache precomputation.

3. Two inference schedules — lazy prefill (LP) for single-stream edge inference, which performs adapter-free reconstruction as one contiguous prefill after the current agent's turn, and double batching (DB) for concurrent serving, which runs the adapted and adapter-free paths together in the continuous serving batch during prefill and decoding. The paper states the two schedules produce logically equivalent cache states and differ only in timing.

4. Theoretical and empirical analysis — a derivation of the condition under which the neutral base cache has lower relative error than the previous agent's base cache, plus layer-wise measurements of that geometry on both backbones.

Main Findings

• Accuracy (headline): ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only 1.1% points relative to NonShared (inference without cache sharing). The paper reports that ReBaseShared achieves accuracy comparable to BaseShared while eliminating most of BaseShared's repeated prefill.

• Accuracy (per setting, from Table 2): On LLaMA-3.1-8B, NonShared averages 37.52 on HotpotQA and 69.19 on ScienceQA; ReBaseShared reaches 36.95 (−0.57) and 68.77 (−0.42); BaseShared 36.97 (−0.55) and 68.64 (−0.56); PreLRShared 36.32 (−1.20) and 68.22 (−0.97); FullShared 34.03 (−3.48) and 65.62 (−3.57). On Ministral-8B, NonShared averages 36.72 and 69.07; ReBaseShared 36.08 (−0.62) and 66.47 (−2.60); BaseShared 35.43 (−1.28) and 66.74 (−2.33); PreLRShared 34.77 (−1.95) and 65.78 (−3.29); FullShared 33.37 (−3.35) and 63.39 (−5.68).

• PreLRShared degradation range: PreLRShared shows average degradation ranging from 1.0% to 3.3% points, and the paper reports it remains comparable to or more accurate than the selective recomputation baselines even without neutral reconstruction.

• Single-stream efficiency: PreLRShared reduces TTFT by up to 3.1× and improves per-request throughput by up to 2.3× over NonShared. It stays within 17% of FullShared's TTFT at 66.4k tokens. ReBaseShared with LP reduces TTFT by up to 3.0× and improves per-request throughput by up to 1.6× over NonShared, outperforming all selective recomputation baselines at long trajectories.

• Concurrent serving: Under unsaturated load, PreLRShared and ReBaseShared provide 1.6× and 1.3× higher per-request throughput than NonShared, respectively. The p90 and p99 results show the same saturation trend with larger gaps as queueing accumulates.

• LP vs. DB: In single-stream inference, LP yields lower TTFT and higher throughput than DB. Under concurrent serving, DB sustains lower TTFT and higher throughput than LP as the request rate increases, because it avoids a separate post-turn reconstruction prefill that can delay queued requests.

• Memory: PreLRShared and ReBaseShared retain BaseShared's memory efficiency. At 66.4k tokens they remain within 2% of FullShared while using 17–23% less peak memory than the selective recomputation baselines. DroidSpeak incurs the largest overhead because it additionally retains hidden states before the first recomputed layer; CacheBlend uses more memory than RelayCaching because it recomputes selected tokens across all layers.

• Theoretical condition: Using Euclidean norms at a fixed layer and token position, the neutral base cache has no greater relative error when ρ ≥ 2cosθ, equivalently Γ_err ≥ 1, where ρ is the ratio of the previous agent's offset norm to the current agent's offset norm and θ is the angle between those offsets. Measured layer-wise on LLaMA-3.1-8B-Instruct and Ministral-8B-Instruct across the plan-to-action, action-to-reflect, and reflect-to-plan transitions in HotpotQA, the averaged Γ_err stays above one at every measured layer for both hidden states and base caches.

• Why prefix-based correction does not apply: The paper notes that under its matched-prefix setting, correction variables of prefix-based methods remain unchanged and provide no adapter-specific correction, so their application reduces to FullShared for LoRA-adapter-induced differences.

Methodology in Plain English

The problem in concrete terms. In a three-role agent loop (plan, action, reflect), each turn appends text to one shared trajectory: the user request, retrieved documents, tool observations, and prior agents' outputs. Each new agent reads that whole growing trajectory. KV caching normally avoids recomputing tokens seen earlier by the same model with the same weights, but here the weights differ each turn because each role has its own LoRA adapter. So the same tokens get prefilled over and over, and each agent stores its own copy of the cache.

How prior work splits the difference. LRAgent decomposes the adapted projection Y_i = X_i,adpt(W_0 + A_iB_i) into a base contribution X_i,adpt W_0 and an adapter contribution (X_i,adpt A_i)B_i. Its BaseShared method shares the base cache across agents and keeps only a narrow rank-r LR cache per agent. That cuts memory, but the current agent still has to run full backbone computation over the accumulated trajectory just to produce its own LR cache. Selective recomputation methods instead rebuild critical layers or tokens with the current agent, which trades accuracy against how much gets recomputed.

PReCache's two moves. First, when an agent is already processing a new chunk of context, PReCache applies every agent's down-projection A_k to the hidden states right as they are produced, storing the rank-r result for each of the N agents. The expensive part (running attention and MLP blocks over the text) is done once; only cheap narrow matrix multiplications are multiplied across agents. Later agents simply pick up their own precomputed LR cache.

Second, since even the "base" cache inherits the previous agent's adapter effects through the hidden states, PReCache reruns the shared context with adapters disabled, producing adapter-free hidden states X_neut, and builds the neutral base cache X_neut W_0. This neutral cache is a better stand-in for what the next agent would compute. The paper verifies this with the relative-error analysis and with layer-wise measurements on Ministral-8B over a HotpotQA trajectory.

Scheduling the extra pass. The adapter-free reconstruction is real work, so the paper gives two ways to hide it. Lazy prefill (LP) runs it as one whole-context prefill after the current agent finishes decoding — good for a single stream on one device. Double batching (DB) folds the adapter-free path into the same continuous serving batch as the adapted path, processing the same tokens at the same positions with separate KV cache views — good when many requests share the server.

How they tested. Experiments use the AutoAct multi-hop agent framework with three role-specific agents (plan, action, reflect), web search via the Serper API and Wikipedia lookup. Backbones are LLaMA-3.1-8B-Instruct and Ministral-8B-Instruct; benchmarks are HotpotQA and ScienceQA, each with three difficulty levels. A separate LoRA adapter is trained per role with rank r = 8 on query and value projections. Baselines are NonShared, FullShared, DroidSpeak, CacheBlend, RelayCaching, and BaseShared, with selective recomputation methods set to a 30% recomputation ratio. Efficiency is measured on replayed traces with identical agent schedules and token counts, spanning 1.9k to 66.4k tokens: single-stream runs use one NVIDIA A6000 48GB GPU with ReBaseShared LP, and concurrent serving uses vLLM on one NVIDIA A100 80GB GPU with chunked prefill, paged KV cache, and prefix caching, sweeping 0.5 to 16 QPS on a fixed 17.3k-token trajectory with ReBaseShared DB.

Why This Matters

Impact on research. The paper reframes a specific open problem in multi-agent LLM inference: how to get both memory savings and prefill savings without retraining adapters or constraining their architecture. The comparison table in the paper lays out the trade-off space explicitly — selective recomputation gives repeated-prefill reduction but not accuracy preservation, BaseShared gives accuracy preservation but not prefill reduction, and BaseLRShared gives both but requires adapters trained with a shared LoRA down-projection. PReCache claims all three, and the neutral-reconstruction error condition (ρ ≥ 2cosθ) gives a checkable criterion rather than a heuristic.

Real-world applications:

• Multi-role agent assistants (planning/action/reflection loops, as in AutoAct) that carry long tool-observation histories.
• Multi-hop retrieval-augmented question answering over long retrieved documents, where prefixes are shared across turns.
• Edge or single-device deployments with constrained GPU memory, where per-agent KV cache replication is the binding constraint — the motivating setting for the LP schedule.
• Server-side agent serving platforms that batch many concurrent agent requests and need tail latency (p90/p99) under control — the motivating setting for the DB schedule.

Industry relevance. The claimed benefits map directly to production serving metrics: time-to-first-token, per-request throughput, and peak GPU memory. Since PReCache is training-free and works with existing LoRA adapters, it can in principle be dropped into an existing multi-LoRA deployment without retraining the adapters. The memory result is potentially decisive at scale — remaining within 2% of FullShared at 66.4k tokens while using 17–23% less peak memory than selective recomputation baselines.

Future Directions

• Adapters trained with a shared down-projection. BaseLRShared was excluded from the experiments because it requires training adapters with a shared LoRA down-projection, so it does not support existing adapters with different down-projection weights. How PReCache compares against BaseLRShared, and whether the two can be combined, is left open by the reported setup.

• Scaling beyond the evaluated configurations. The paper evaluates two 8B backbones, two benchmarks, and three agents. It reports agent scaling only via Appendix C.5 (memory efficiency as the number of agents increases) and rank ablations in Appendix D.1 (accuracy saturating from r = 8) and Appendix D.2 (LoRA on all attention projections). Behavior at larger model sizes, larger agent counts, and other domains is not established in the provided content.

• Reconstruction cost of the adapter-free path. ReBaseShared LP and DB both require a pass with adapters disabled. The paper does not report a cost-reduction mechanism for this pass; it only reschedules it. Reducing or approximating the neutral reconstruction while preserving the error condition derived in Appendix A.1 is a natural next step.

• Non-QV LoRA placements and other cache-sharing stacks. The default configuration applies LoRA to query and value projections only, so the key projection has no LR component and is fully shared. Appendix D.2 reportedly tests all attention projections, but the main text does not report per-setting efficiency or accuracy numbers for that configuration.

Target Audience

This paper is most useful to systems and inference engineers working on LLM agent serving, multi-LoRA deployments, and KV cache optimization, and to researchers studying efficient inference, cache reuse, and parameter-efficient multi-adapter serving. Practitioners running vLLM-style serving stacks for agent workloads will find the scheduling discussion (LP vs. DB) and the memory measurements directly actionable. Readers will need working knowledge of transformer inference phases, KV caching, and LoRA to follow the decomposition and error analysis, though the high-level designs are understandable without the math.

Note: the provided paper content ends mid-sentence ("Acr") and references Appendices A.1, A.2, B.1, B.2, C.1–C.5, D.1, and D.2 whose contents are not included, so the specific numbers in those appendices are not reported here.

Authors’ abstract

Multi-LoRA agent systems enable efficient role specialization by sharing a common backbone model. However, each agent repeatedly processes the growing shared trajectory and constructs its own KV cache, introducing substantial memory and computation redundancy in long-horizon tasks. Existing KV cache sharing methods reduce this repeated prefill, but they either require additional training or architectural constraints or retain substantial model computation. Moreover, direct cache reuse causes the current agent to rely on cache states generated by the previous agent's adapter, weakening the role-specific behavior encoded by its own LoRA. We present PReCache, a training-free KV cache sharing framework with two designs, namely PreLRShared and ReBaseShared, that share the base cache computed using the pretrained weights and precompute a compact agent-specific low-rank (LR) cache. To remove repeated prefill, PreLRShared precomputes each agent's LR cache when the shared context is first processed, allowing the current agent to use its own LR cache without reprocessing context processed by previous agents. To improve sharing accuracy, ReBaseShared reconstructs the shared base cache from adapter-free hidden states, reducing the remaining error caused by the previous agent's adapted representation. To minimize its reconstruction cost, we propose two inference schemes tailored to single-stream inference and concurrent serving, performing the same reconstruction after each agent's turn or alongside its execution, respectively. Across multiple models and agent benchmarks, PreLRShared achieves up to a 3.1x TTFT speedup and a 2.3x improvement in per-request throughput over inference without KV cache sharing. ReBaseShared best preserves accuracy overall among the evaluated cache-sharing methods, with an average drop of only 1.1 points relative to inference without cache sharing.