K-Merge: Online Continual Merging of Adapters for On-device Large Language Models

AI summary

Overview

Research area: Machine learning, specifically parameter-efficient fine-tuning and model merging for on-device large language models (LLMs).

Technical level: Intermediate. The paper builds on LoRA and model-merging literature, but its core ideas (similarity scoring, running-average merging, storage budgets) are explained with accessible notation.

Scope: The paper defines and addresses "online continual merging" of LoRA adapters under a fixed on-device storage budget of K adapters, and evaluates two proposed variants (K-Merge and K-Merge++) against zero-shot, single-task LoRA, Linear, TIES, DARE, DARE-TIES and OPCM baselines.

What This Paper Is About

On-device LLMs are commonly extended with many small LoRA adapters, one per downstream task, but mobile storage cannot hold an adapter for every task. In practice these adapters arrive one at a time as users request new problem types or languages, so the device must decide whether to merge each incoming LoRA into an existing one or occupy a free storage slot, without access to the original single-task adapters or any training data. The paper's goal is a lightweight, data-free procedure that absorbs new LoRAs over time while preserving performance on previously supported tasks.

Key Contributions

1. The authors introduce and formalize a new setting, online continual merging of adapters for resource-constrained on-device LLMs under a storage budget of K adapters, where LoRAs arrive sequentially and the total number of tasks γ is unknown to the device.

2. They propose a lightweight, data-free merging strategy that picks the most suitable stored adapter for an incoming LoRA using a cosine-similarity score computed on the flattened low-rank updates ΔW = BA.

3. They introduce history-aware merging, a running-average formulation driven by a per-slot history map that tracks which task indices have been merged into each stored adapter, making the merge order-invariant.

4. They present two variants, K-Merge (merge into the nearest stored adapter unless a slot is free) and K-Merge++ (only merge when similarity exceeds an empirically estimated threshold s), and evaluate them on 40 tasks spanning 5 problem types and 8 languages.

Main Findings

• LoRA fine-tuning gives large gains: Single-task LoRAs roughly double zero-shot scores. On English with Llama-3.2-1B-Instruct the average rises from 17.1 (zero-shot) to 43.6 (LoRA); with Qwen-2.5-1.5B from 24.0 to 45.3.

• Proposed methods beat the strongest baseline at every budget: K-Merge and K-Merge++ outperform Linear at all storage budgets tested, while DARE, DARE-TIES and OPCM are reported as significantly lower than single-task LoRAs and other approaches.

• Score reaches 80-90% of single-task performance with 8 clusters: The authors report that with K = 8 the overall score of their methods reaches about 80-90% of single-task performance.

• K-Merge++ is the most robust to task ordering: In Table 1 on Llama-3.2-1B, worst-case ordering at K = 5 gives 0.80 for K-Merge++ versus 0.75 for K-Merge, 0.69 for Linear and 0.56 for TIES; at K = 7 the values are 0.82, 0.78, 0.73 and 0.67 respectively.

• K-Merge++ overtakes K-Merge over time: K-Merge initially outperforms K-Merge++ at early timesteps, but K-Merge++ is quickly overtaken as it saves storage slots for more diverse LoRAs arriving later; Linear matches the proposed methods at initial timesteps and is then consistently surpassed.

• Cosine similarity is the best similarity choice: With K = 3, K-Merge scores 0.75 using cosine versus 0.66 for L1, L2 and L∞; with K = 5 it scores 0.78 versus 0.69 (L1), 0.67 (L2) and 0.67 (L∞).

• History-aware merging helps: For Linear (equivalent to K-Merge when combined with history-aware merging), score at K = 1 is 0.63 with history-aware merging versus 0.45 with constant weighting, and at K = 3 is 0.75 versus 0.66. TIES, DARE and DARE-TIES obtain worse performance than the proposed solution when history-aware merging is added.

• Merging is cheap: Integrating an incoming LoRA takes between 0.04s and 0.18s when there are 2 to 8 stored multi-tasking LoRAs, and extra memory overhead is negligible because memory is dominated by text generation.

• Storage budget matters: Each Llama-3.2-1B LoRA has 23M parameters and takes 27MB; each Qwen-2.5-1.5B LoRA has 37M parameters and takes 34MB. Storing 5 adapters consumes 135-170MB in total, whereas storing all 40 adapters would take about 1GB.

• Problem-type similarity exceeds language similarity: The appendix analyses report more similarity between LoRAs from the same problem type than between LoRAs from the same language.

Methodology in Plain English

The device is assumed to hold at most K LoRA adapters in a set ℒ. When a new single-task LoRA L^(t) arrives at time step t, the system compares it against every stored adapter using cosine similarity between the flattened low-rank update matrices, averaged over all transformer layers and the key, query, value and output projections. The stored adapter with the highest similarity is the merge candidate.

K-Merge simply stores the incoming LoRA if fewer than K adapters are held, and otherwise merges it into its closest stored adapter. K-Merge++ adds a similarity threshold s: the incoming LoRA is merged only if its similarity to the closest stored adapter is at least s; otherwise it takes a free slot, and once storage is full merging happens regardless of s. The threshold is calibrated on an auxiliary held-out set of LoRAs (unseen problem types Translation to English and Title Generation, and unseen languages Portuguese, Turkish and Serbian), and set to the median of all pairwise similarities, giving s = 0.020 for Llama-3.2-1B and s = 0.028 for Qwen-2.5-1.5B.

Merging itself is a running average. A history map H records, for each stored adapter, the set of task indices that have contributed to it. When a new LoRA is merged into slot c, the merged adapter is (L^(t) + |H[c]| · L_c) / (|H[c]| + 1), and the task index t is added to H[c]. Because each contributing LoRA carries equal weight, the result is order-invariant, and the device never needs to store the individual LoRAs of a cluster. At inference, the task identity (known from a UI selection, keywords in the prompt, or a lightweight classifier, assumed perfect here) is mapped back through the history to the correct stored adapter.

Evaluation covers 5 problem types (Smart Reply, Summarization, Tone Adjustment, Question Answering, Grammar Correction) times 8 languages (English, Spanish, French, German, Italian, Korean, Japanese, Chinese), giving γ = 40 tasks. Metrics are F-0.5 for Grammar Correction, F-1 for Question Answering, weighted ROUGE for Smart Reply, and ROUGE-L for Summarization and Tone Adjustment, normalized against the single-task LoRA score to form an aggregate score S^(t). LoRAs use rank 32 and scaling factor 128, trained with AdamW at a learning rate of 5e-5, dropout 0.05 and mini-batch size 3. K varies from 1 to 8, and results are averaged over 3 random permutations of the LoRA arrival order.

Why This Matters

Impact on research. The paper defines a practical setting that sits between model merging and continual learning: adapters arrive one at a time, no training data or original adapters are available, and storage is capped. It shows that classical merging baselines such as TIES, DARE and OPCM degrade in this online regime, particularly under adversarial task orderings, and that a simple similarity-gated running average is a stronger and cheaper alternative.

Real-world applications:

• Personal assistants on smartphones that gain new problem types (grammar correction, summarization, smart reply) and new languages over time without shipping a new base model.
• Privacy-preserving local processing of personal messages or notes, where user data should not leave the device.
• Offline-capable devices with limited or no connectivity, where adapters are downloaded incrementally and old functionality must keep working.
• Multi-lingual deployments in markets where new language support is requested after the device ships.

Industry relevance. Mobile and edge vendors cannot ship an unbounded library of adapters; the paper quantifies the cost (27MB per Llama-3.2-1B LoRA, 34MB per Qwen-2.5-1.5B LoRA, about 1GB for all 40) and shows that merging latency of 0.04s-0.18s is compatible with on-device use. The method is data-free, so it raises no new data-collection or privacy requirements.

Future Directions

• Extending the approach beyond text LLMs to multimodal tasks, which the authors explicitly leave for future work.
• Testing adapters other than LoRA, since the study is limited to LoRA as the current industry standard.
• Evaluating the method on larger LLMs beyond the on-device sizes studied, which the authors note would require significantly more compute.
• Investigating how adapter merging affects safeguarding mechanisms, since the authors warn that merging can weaken them and that experiments are needed before deployment.
• Exploring routing and task identification more thoroughly, since the current design assumes a perfect task classifier and separates routing from the merging contribution.

Target Audience

Researchers and engineers working on parameter-efficient fine-tuning, model merging, and continual learning for language models. It is also directly relevant to practitioners building mobile or edge LLM products under storage and compute constraints, and to readers interested in how standard merging methods behave when adapters arrive sequentially rather than all at once.

Authors’ abstract

On-device deployment of Large Language Models (LLMs) frequently leverages Low-Rank Adapters (LoRAs) to support diverse downstream tasks under tight resource constraints. To address the limited storage capacity of mobile devices, recent works have explored model merging techniques to fuse multiple LoRAs into a single one. In practice, however, LoRAs are often delivered incrementally, as users request support for new tasks (e.g., novel problem types or languages). This scenario introduces a new challenge: on-device online continual merging, where the objective is to incorporate new LoRAs while preserving the performance on previously supported tasks. In this paper, we propose a data-free and computationally efficient strategy for selecting and merging LoRAs when a new one becomes available, assuming the device can store only a limited number of adapters. Extensive experiments across real-world tasks demonstrate the superiority of our approach compared to alternative strategies while adhering to the storage budget and compute limitations of on-device settings. The project page is available at: https://donaldssh.github.io/K-Merge.