On the Convergence Rate of LoRA Gradient Descent

AI summary

Overview

Research area: machine learning theory and optimization, specifically the convergence behavior of the low-rank adaptation (LoRA) algorithm for fine-tuning large models. Technical level: Advanced — the paper is a matrix-calculus convergence proof with supporting empirical work. Scope: a non-asymptotic convergence analysis of the original LoRA gradient descent algorithm under minimal assumptions, plus learning-rate experiments on CIFAR-10 and a 1.1B-parameter language model.

The paper is by Siqiao Mu (Department of Engineering Sciences and Applied Mathematics, Northwestern University) and Diego Klabjan (Department of Industrial Engineering and Management Sciences, Northwestern University), posted as arXiv:2512.18248v3 [cs.LG], under a CC BY 4.0 license. Code is linked at https://github.com/siqiaomu/lora.

What This Paper Is About

LoRA fine-tunes a pretrained weight matrix W0 by training two small adapter matrices A and B, so the final weights are W0 + BA. Although the original loss function is often Lipschitz smooth, the reparametrized loss in terms of A and B is not, which blocks the standard toolkit used to prove convergence rates. Existing LoRA theory either studies asymptotic or infinite-width regimes, or assumes the adapter matrices are uniformly bounded — a condition that artificially restores Lipschitz smoothness. This paper derives the first non-asymptotic convergence rate for the original LoRA gradient descent algorithm without those assumptions.

Key Contributions

1. A first non-asymptotic convergence proof for the original LoRA gradient descent algorithm. The paper shows that the minimum gradient norm converges to zero at rate O(1/log T), where T is the number of iterations, assuming only that the original loss function is Lipschitz smooth (Assumption 3.1) and lower bounded (Assumption 3.2).
2. A new proof technique that stacks the adapter matrices into a single variable V, rewrites the problem in terms of the outer product V V^T, and derives a modified "Lipschitz-like" descent lemma for the reparametrized function. This replaces the boundedness assumptions used in prior work with direct control of the learning rate.
3. A sharpening of the result under boundedness: if there exists a constant C > 0 such that ||V_t|| ≤ C for all t, the rate improves to the classical O(1/T), matching the standard rate of gradient descent on Lipschitz smooth functions and the rates of earlier bounded-adapter analyses.
4. Theory-motivated learning-rate schemes — an adaptive rate based on the parameter norm and the gradient of the original loss, a second adaptive rate that substitutes the loss value for the gradient norm, and a normalized rate based on the gradient norm raised to the power 1/2 — validated empirically with logistic regression and ResNet-18 on CIFAR-10 and with TinyLlama-1.1B-Chat-v1.0 on the Alpaca dataset.

Main Findings

• Main rate: For T steps of LoRA gradient descent, the minimum squared gradient norm satisfies min over t = 0, ..., T−1 of ||∇J(V_t)||² = O(1/log T). The O(·) notation hides dependence on V_0, J(V_0), L*, and L.
• Bounded case recovers O(1/T): If ||V_t|| ≤ C for all iterates, the rate becomes O(1/T), hiding dependence on J(V_0), L*, L, and C. The paper notes this matches the rates of existing works that impose bounded adapter norms and thereby force J to be Lipschitz smooth.
• No explicit rank dependence: The convergence rate does not explicitly depend on the LoRA rank r, except through a factor of O(1/||V_0||²). The paper attributes this to gradient descent being a dimension-free algorithm, since r only controls the dimension of the problem.
• Learning-rate condition for descent: To guarantee descent at each step, the learning rate must satisfy η_t = min{1/(4√2 L (||V_t||² + ||∇L(E_1 V_t V_t^T E_2)||)), 1}. Under this choice, J(V_{t+1}) ≤ J(V_t) − (η_t/4) ||∇J(V_t)||².
• Telescoping bound: Summing the descent inequality gives min ||∇J(V_t)||² ≤ 4(J(V_0) − L*) / Σ η_t over t = 0, ..., T−1. Convergence follows only if this sum diverges.
• Worst-case growth of the parameter norm: Even when ||V_t||² is unbounded, the paper shows the worst case is ||V_t||² = O(t), which makes the sum of learning rates lower-bounded by a harmonic series that is Θ(log T). This slowdown is what produces the O(1/log T) rate.
• Position dependency: Convergence is slowed when iterates move away from the origin and accelerated when they move toward it, because of the ||V_t||² term in the denominator of the learning rate. The paper states this behavior is not typical of gradient algorithms.
• New stationary point at the origin: V = 0 is a stationary point with ∇J(V) = 0 regardless of the structure of the original loss function. LoRA can therefore converge to the origin even if the original global minimum is arbitrarily far away, which the paper cites as an illustration of how LoRA and full-rank fine-tuning may lead to different outcomes.
• Connection to Burer-Monteiro: The V V^T form is exactly the symmetric Burer-Monteiro parametrization (Burer and Monteiro, 2003), and the analysis extends to show that gradient descent with Burer-Monteiro converges to stationary points for general Lipschitz smooth functions.
• CIFAR-10 logistic regression: Training with the adaptive rates η^adapt and η^adapt2, or the normalized rate η^norm, converged faster and more stably than constant learning rates in a similar range of values. A constant rate that is too high destabilizes training; one that is too low slows convergence. η^adapt and η^adapt2 tracked each other closely early in training and diverged as the model converged.
• CIFAR-10 ResNet-18: Both adaptive (η^adapt2) and normalized (η^norm) rates stabilized training compared with a constant rate while maintaining fast convergence, with η^norm outperforming η^adapt2. η^adapt was not tested here due to memory constraints, because standard LoRA implementations avoid computing ||∇L(E_1 V_t V_t^T E_2)||.
• Unbounded parameter norm in practice: For the ResNet-18 model, ||V_t|| stopped growing after a finite number of iterations. For logistic regression, ||V_t|| increased monotonically over 60 epochs, and when training was extended to 1000 epochs, ||V_t|| grew unbounded for all t, suggesting convergence to a stationary point at infinity and a rate meaningfully slower than O(1/T).
• LLM setting (TinyLlama-1.1B-Chat-v1.0 on Alpaca): With A initialized with standard deviation σ = 1e-3, adaptive learning rates maintained faster and more stable convergence than constant rates of similar magnitude. With σ = 1/r, the advantage diminished because ||V_t|| is not only large but also grows extremely slowly relative to its size, making the adaptive rate close to a constant rate. The paper notes convergence may appear to be O(1/T) over a finite number of iterations, although the asymptotic behavior is slower.
• Multiple weight matrices: The analysis extends to networks with multiple weight matrices. For two matrices, the paper shows via Lemma 3.6 that if L is L-Lipschitz smooth, the stacked reformulation is 2L-Lipschitz smooth, so the existing analysis applies directly.
• Choice of extractor matrices: The paper states that the choice of the extractor matrices E_1 and E_2 has almost no impact on the proof, suggesting other subsets of V V^T could serve as alternate parameter-efficient reparametrizations with similar results.

Methodology in Plain English

The core difficulty is that LoRA's reparametrized loss is not Lipschitz smooth, because the gradient of the reparametrized objective contains a multiplicative factor of the parameter matrix V, which can grow without bound. Classical convergence proofs rely on the descent lemma for Lipschitz smooth functions, so they do not apply directly.

The authors work around this in three steps. First, they stack the transposed adapter A^T and the adapter B into a single matrix V, and use fixed "extractor" matrices E_1 and E_2 to pull the product BA out of the top-right block of V V^T. This turns LoRA gradient descent into ordinary gradient descent on a single function J(V) = L(E_1 V V^T E_2), with the bonus that V = 0 is always a stationary point.

Second, because J is not Lipschitz smooth, they prove a modified descent lemma: the change in J between two points is bounded by the usual first-order term plus extra higher-order terms involving ||V_2 − V_1||², ||V_2 − V_1||³, ||V_2 − V_1||⁴, and the norm of the original loss gradient.

Third, they choose a learning rate that is small enough relative to the current parameter norm and gradient norm to make those extra terms negligible, yielding guaranteed descent per step. They then show that even if the parameter norm grows, the sum of learning rates grows like a harmonic series (Θ(log T)), so the telescoped bound on the minimum gradient norm yields O(1/log T). If the parameter norm stays bounded, the sum grows linearly in T and the classical O(1/T) rate returns.

For the experiments, they design learning-rate schedules that follow from the theoretical conditions and test them against constant rates of comparable magnitude.

Why This Matters

This paper addresses a gap between LoRA theory and LoRA practice: nearly all deployed LoRA training updates both adapter matrices simultaneously, but prior convergence proofs either require single-matrix updates, bounded adapter norms, or apply only in asymptotic or infinite-width regimes. By analyzing the original algorithm with minimal assumptions, the work provides a convergence guarantee that applies to the variant people actually use, and offers a mechanistic explanation for why LoRA training dynamics differ from full-rank fine-tuning — notably the stationary point at the origin and the "position dependency" of the convergence speed.

Real-world applications:

• Fine-tuning large language models on task-specific data for chat assistants, code tools, or domain-specific generation, where LoRA is the default parameter-efficient method and low compute budgets matter.
• Adapting vision models to new datasets, as demonstrated with ResNet-18 on CIFAR-10, including settings where batch normalization layers are disabled or altered.
• Memory- and bandwidth-constrained training regimes such as federated LoRA or on-device adaptation, where the behavior of adapter matrix norms directly affects the learning-rate schedule.
• Serving as a theoretical foundation for future LoRA variants, quantized or memory-efficient schemes, since it isolates the geometric effects caused by the low-rank reparametrization itself.

Industry relevance: the paper's practical recommendation — replacing a constant learning rate with a schedule that scales inversely with the parameter norm, the loss value, or the gradient norm — addresses a common tuning problem. With constant rates, practitioners face a narrow window between instability at high values and slow convergence at low values; the proposed schedules were observed to stabilize training and maintain fast convergence in the tested settings, including in a high-dimensional LLM regime where the benefit narrows.

Future Directions

• Extending the analysis from full gradient descent to stochastic gradient descent and to the adaptive optimizers typically used in LoRA training, since the experiments in this paper use mini-batch SGD with batch size 512.
• Characterizing when LoRA iterates converge to the origin stationary point introduced by the reparametrization, and under what initializations or rank choices this behavior can be avoided if it is undesirable.
• Developing learning-rate schedules that retain their benefit in high-dimensional settings such as LLM fine-tuning, where the paper observes that the parameter norm grows very slowly relative to its size and adaptive rates behave close to constant rates. The theoretical asymptotic rate is still slower, but this is not visible over a finite number of iterations.
• Extending the multi-matrix analysis beyond the illustrative two-matrix case to general deep network architectures with many adapter pairs of different sizes, and refining the 2L-Lipschitz smoothness constant in that setting.

Target Audience

Optimization and machine learning theory researchers studying non-convex or bilinear parametrizations; graduate students working on convergence analysis of parameter-efficient fine-tuning; practitioners who tune LoRA learning rates for large language models or vision models and want a principled alternative to constant rates; and engineers evaluating whether LoRA training dynamics can be expected to match full fine-tuning in the regimes they care about.

Authors’ abstract

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the \textit{original LoRA gradient descent} algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(\frac{1}{\log T})$, where $T$ is the number of iterations. We conduct numerical experiments to validate our theoretical findings.