Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation

AI summary

Overview

• Research area: Parameter-efficient fine-tuning (PEFT) of large language models, specifically low-rank adaptation (LoRA) and its variants.
• Technical level: Intermediate. The paper relies on linear algebra (QR/LQ decomposition, unitary matrices, diagonal gating) but the core idea is stated simply: give each input token its own scaling of the LoRA weight.
• Scope: The paper proposes TopLoRA, a modification of LoRA that generates a token-specific diagonal matrix to modulate the low-rank update, and evaluates it on the GLUE benchmark plus mathematical and commonsense reasoning benchmarks across RoBERTa, Gemma-7B, LLaMA-3-8B, and Qwen2.5-14B.

What This Paper Is About

Standard LoRA freezes a pretrained weight matrix and learns two small matrices A and B whose product approximates the weight update. All input tokens pass through the same A and B, so they all receive an identical input-output projection, even though different tokens carry different meanings. The paper argues this shared projection — not only the LoRA rank — is a bottleneck on LoRA's expressiveness, and it introduces TopLoRA, which lets each token have its own input-output projection without increasing the rank.

Key Contributions

1. A projection-based reading of LoRA. Using QR decomposition of B and LQ decomposition of A, the authors rewrite the LoRA update as ΔW = B A = Q_B P Q_A, splitting it into an input space Q_A, an output space Q_B, and an input-output projection P = R_B L_A. They use this to argue that the rank r controls the dimensionality of the input and output spaces, while the shared projection P is a separate limitation.

2. TopLoRA. A method that adds a projector network Γ with parameters Θ, which generates a diagonal matrix Σ_X from each input token X and adjusts the update to ΔW_X = B Γ_Θ(X) A = B Σ_X A. The rank is not increased; only the projection is made token-dependent.

3. A specific parameterization of the diagonal matrix. Σ_X = Diag(Exp(RMSNorm(ΘX))), with Θ initialized using Kaiming initialization, matching the initialization of A. RMSNorm removes sensitivity to the magnitude of the token or Θ, and the exponential maps zero-centered normalized values into strictly positive scaling factors.

4. An explicit decomposition of the output. ΔW_X X = B A X + B(Σ_X − I) A X, where the first term is the standard LoRA output capturing global patterns and the second is an input-dependent adaptation that acts as a learned gating mechanism. When Σ_X = I, TopLoRA reduces to standard LoRA.

Main Findings

• GLUE (RoBERTa). Raising LoRA's rank from 8 to 32 increases average accuracy by 1.05% on RoBERTa-Base and 1.97% on RoBERTa-Large. TopLoRA at rank 8 improves over LoRA at rank 8 by 1.5% (RoBERTa-Base, 84.14 vs 82.55) and 2.01% (RoBERTa-Large, 86.39 vs 84.38), surpassing LoRA at rank 32 (83.60 and 86.35 respectively). TopLoRA at rank 16 reaches 84.48 and 87.55.
• Parameter cost. TopLoRA's parameter count is reported as roughly 1.5 times that of LoRA at the same rank. On RoBERTa-Base at rank 8, TopLoRA uses 0.44M parameters versus 0.29M for LoRA; on RoBERTa-Large at rank 8, 1.18M versus 0.79M.
• Other baselines are more modest on GLUE. DoRA at rank 16 improves about 0.4% over LoRA at rank 16; HydraLoRA at rank 8 improves about 0.5%; MELoRA at rank 16 shows no significant improvement.
• Mathematical reasoning. At rank 8, TopLoRA improves average accuracy over LoRA by 1.67% (Gemma-7B, 73.11 vs 71.44), 3.65% (LLaMA-3-8B, 71.84 vs 68.19), and 1.43% (Qwen2.5-14B, 80.52 vs 79.09). These gains are described as surpassing those achieved by increasing LoRA's rank by a factor of four.
• Commonsense reasoning. At rank 8, TopLoRA outperforms LoRA by 2.02% (Gemma-7B, 87.97 vs 85.95), 1.14% (LLaMA-3-8B, 86.59 vs 85.45), and 0.61% (Qwen2.5-14B, 91.85 vs 91.24). Increasing LoRA's rank fourfold yields only 0.66%, 1.27%, and 0.45%. The paper notes that Qwen2.5-14B already achieves very high accuracy here, making differences between methods less visible.
• Ablation on the projector components. Removing either the exponential function or RMSNorm lowers accuracy on mathematical reasoning at rank 8. On LLaMA-3-8B the average drops from 71.84 to 70.56 without Exp and to 71.77 without RMSNorm; on Gemma-7B from 73.11 to 72.84 and 72.69; on Qwen2.5-14B from 80.52 to 79.98 and 80.23. Omitting RMSNorm causes the larger decline, which the authors attribute to small ΘX values leaving Σ_X close to 1.
• Scalability across rank. On mathematical reasoning with LLaMA-3-8B, TopLoRA is compared across ranks {2, 4, 8, 16, 32, 64, 128} and consistently surpasses LoRA at every tested rank.
• Scalability across tuning granularity. With LoRA and TopLoRA applied by default to query, key, and value weights, the authors also test Q, QV, QKVUD, and QKVOGUD configurations (O = output, G = gate, U = up, D = down projection weights). At rank 8, TopLoRA outperforms LoRA under all these configurations.
• Headline improvement figure. The introduction states a 2-3% accuracy improvement over LoRA at the same rank; the conclusion states 2-4%.

Methodology in Plain English

The authors start with algebra. Any LoRA update B A can be rewritten as Q_B P Q_A by decomposing B and A. Q_A describes which input directions LoRA can look at, Q_B describes which output directions it can write to, and P describes how strongly each input direction maps to each output direction. The rank r sets how many directions Q_A and Q_B contain, which explains why higher rank helps. But P is shared by every token, so two tokens with completely different meanings are pushed through the same scaling pattern.

TopLoRA adds a small network that reads the input token and produces a vector of positive numbers, arranged into a diagonal matrix Σ_X. This diagonal sits between the two triangular factors, changing P into P_X = R_B Σ_X L_A. Concretely, the update for token X becomes B Σ_X A instead of B A. Because the network is shared and only produces a diagonal, the extra parameters are limited — the authors report roughly 1.5 times the LoRA parameters at the same rank. Two design choices matter: RMSNorm removes the influence of the token's magnitude and spreads out the diagonal values across tokens, and the exponential keeps the scaling factors positive and prevents small inputs from collapsing toward zero. The paper explicitly contrasts this with MoELoRA-style mixture-of-experts approaches, arguing that TopLoRA addresses the shared-projection limitation directly and could also be inserted inside each expert.

Evaluation covers NLU on GLUE with RoBERTa-Base and RoBERTa-Large (learning rates 3e-4 and 1e-4, warm-up ratio 0.03, linear decay, applied to query and value weights), and NLG on mathematical reasoning (10k training samples, six sub-tasks) and commonsense reasoning (170k training samples, eight sub-tasks) using Gemma-7B, LLaMA-3-8B, and Qwen2.5-14B (learning rate 1e-4, 100 warm-up steps, linear decay, one epoch, applied to query, key, and value weights). Baselines are LoRA, DoRA, MELoRA, and HydraLoRA, with their trainable parameters matched to LoRA at rank 16 — DoRA at rank 16, HydraLoRA at rank 8 with three B matrices, and MELoRA with Mini-LoRA rank 16 and four Mini-LoRA groups. Training uses AdamW, LoRA dropout 0.05, no weight decay, and each experiment is repeated three times with averages reported.

Why This Matters

The paper offers a different lever for improving LoRA. Instead of spending parameters on higher rank, it spends a smaller amount on making the projection token-dependent, and it reports better accuracy than LoRA at four times the rank on several benchmarks. That reframes the design space for PEFT methods and gives follow-up work a concrete target — the input-output projection — to analyze, modify, or combine with existing approaches such as higher-rank or mixture-of-experts methods.

Real-world applications:

• Task-specific fine-tuning of open LLMs in settings where GPU memory and storage cap how many parameters can be trained, such as adapting Gemma-7B, LLaMA-3-8B, or Qwen2.5-14B to a reasoning task.
• Multi-tenant serving of many adapters, where keeping the rank low reduces the size of each stored adapter while the reported accuracy gains reduce the need to store several higher-rank alternatives.
• Domain adaptation in regulated or specialized fields such as clinical, legal, or financial text, where token-level distinctions in vocabulary carry outsized meaning.
• Reasoning-heavy assistants, given the reported gains on mathematical and commonsense benchmarks, which are the settings where standard LoRA tends to lag full fine-tuning.

Industry relevance: the method targets the standard fine-tuning workflow and is reported to plug into existing LoRA pipelines with only a small parameter increase, which matters for teams that fine-tune and serve many adapters under fixed compute budgets.

Future Directions

• Inference cost. The paper does not report measured inference latency or throughput for TopLoRA. The projector introduces a per-token forward pass and extra multiplication, and whether this is negligible in deployment is not quantified in the provided content.
• The stated limitation. The checklist states that a potential limitation of TopLoRA is discussed in Appendix D, but the appendix text itself is not included in the material provided, so the limitation is not specified here.
• Combining with other LoRA variants. The authors note that TopLoRA could be incorporated into each expert of MoELoRA to further enhance expressiveness, and separately compare against higher-rank methods (HiRA, KronA, MELoRA) in related work — testing those combinations is a natural next step.
• Separating the projector's contribution from added parameters. TopLoRA uses about 1.5 times LoRA's parameters at equal rank. Whether the gains come from the token-wise structure or simply from extra capacity is not isolated in the reported experiments, and the paper does not present a parameter-matched control beyond the existing baselines.

Target Audience

Researchers and engineers working on parameter-efficient fine-tuning of large language models, particularly those who already know LoRA and want a compact, algebraically motivated alternative to simply raising the rank. It is also useful for practitioners running many fine-tuned adapters under memory or storage constraints, and for readers interested in the interpretability of LoRA through the input space / output space / projection decomposition. The reported 2-3% and 2-4% improvement figures are stated in different parts of the paper, so readers citing the headline gain should check both.

Authors’ abstract

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as $BΣ_X A$, where $A$ and $B$ are low-rank matrices (as in standard LoRA), and $Σ_X$ is a diagonal matrix generated from each input token $X$. Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.