The Numerical Linear Algebra of Large Language Models

AI summary

Authors: Abdelkader Baggag (Qatar Computing Research Institute, HBKU) and Yousef Saad (University of Minnesota) arXiv: 2610.04631v1 [cs.LG], 03 Oct 2026 · CC BY-NC-ND 4.0

Overview

Research area: Numerical Linear Algebra (NLA) and Machine Learning — specifically the matrix and tensor foundations of Transformers and Large Language Models (LLMs).

Technical level: Intermediate. The paper is written for readers with a background in numerical methods, matrix theory, and optimization, but it deliberately builds the machine-learning material up from first principles (starting with a simple multilayer perceptron) so that NLA specialists can follow it.

Scope (one sentence): A survey that (1) explains the core concepts behind LLMs in a way accessible to numerical-methods specialists, and (2) examines the key NLA concepts that LLM techniques rely on, while highlighting recent contributions of NLA to the field.

Note: the supplied text is truncated partway through Section 2.4.1 (Stochastic Gradient Descent). The later sections the article advertises — including the treatment of attention/Transformers and the worked examples of NLA methods in deep learning (Section 4) — are not present in the provided content, so this summary covers only what appears.

What This Paper Is About

The authors argue that Numerical Linear Algebra has repeatedly been reshaped by waves of scientific demand — eigenvalue problems in 1950s–60s aerodynamics (yielding the LR and QR algorithms), then sparse linear systems in computational aerodynamics — and that machine learning is the current wave, with matrix and tensor methods at its core. The problem they address is that the AI field is large, fast-moving, culturally different from NLA, and hard for numerical practitioners to adapt to. Their goal is a bridge document: a matrix-theory-flavored exposition of how LLMs work, plus an account of where NLA ideas already drive the field.

Key Contributions

1. A historical and strategic framing of AI as the latest "wave" that NLA must respond to. The paper situates LLMs in a lineage from Turing's 1950 test through the neural-network work of the late 1950s and 1990s to the post-2010 GPU-driven resurgence, and asks explicitly how the NLA and numerical analysis communities should react to a technology that is "unusually disruptive" in academia (student interest, assessment, and funding priorities).

2. A quantified comparison between AI scaling and Moore's Law. Using four GPT data points, the authors estimate a parameter-doubling time of ≈4.6 months (a factor of 10 every ≈15 months) — far faster than Moore's Law's two-year transistor doubling — and argue this rate is unsustainable because parameter counts are already hitting a limit tied to available data.

3. A from-scratch, matrix-notation derivation of the neural network machinery — multilayer perceptrons, layer-wise affine maps and activation functions, the softmax, and the cross-entropy loss — presented as an approximation problem φ_W(X) ≈ Y, with explicit parameter counting for a digit-classification example.

4. An account of the two numerical-analysis ideas the authors identify as foundational to deep learning: Cybenko's Universal Representation Theorem from approximation theory, and backward differentiation (Griewank), known in machine learning as backpropagation — plus the stochastic optimization framework (SGD for finite-sum problems) used to train such models.

Main Findings

• NLA is central, not auxiliary: The authors state that NLA provides packages used internally (LAPACK) and contributes key ideas and algorithms that reduce computational time, and that matrix/tensor formalism is what makes deep-learning transformations easy to express.
• Backpropagation is efficient because it is a sequence of matrix–matrix products: This is offered as a core reason NLA tools contribute most to improving both LLM training and inference.
• Low-rank approximation and low-precision tensor arithmetic are heavily exploited in LLM training and inference, per the authors' assessment.
• Attention cost is the structural bottleneck: Transformers rely on attention modules that scale quadratically with the number of tokens n, which the authors identify as a primary driver of training cost; mitigation must come from better hardware and from algorithms achieving good accuracy with linear or near-linear scaling.
• LLM parameter growth has outpaced Moore's Law dramatically: 177M (GPT-1, 2018), 1.5B (GPT-2, 2019), 175B (GPT-3, 2020), and an estimated 1.7T (GPT-4, 2023) — a ≈4.6-month doubling, versus Moore's Law's two-year doubling of transistors on a chip (formulated 1965).
• Parameter scaling appears to be saturating: Figure 2 is described as showing parameter counts already hitting a limit and a "data wall," with the note that parameter count is linked to a finite data corpus and may not increase much beyond current levels.
• AGI timelines are framed as an order-of-magnitude argument, not a model: The paper reports Aschenbrenner's 2024 "Situational Awareness" claim that exponential scaling in compute and algorithmic efficiency makes AGI highly plausible by 2027–2028, based on an estimated 6 orders of magnitude of advance from GPT-2 to GPT-4 over 2019–2023 and a further assumed 6 OOMs over 2023–2027. The authors label Figure 3 a "schematic argument about scaling trends, not a precise predictive model."
• Three mutually reinforcing ingredients drive the current wave: improved hardware (GPUs), availability of data, and advances in methods and algorithms — with optimization, statistics, and numerical linear algebra named as the method-side components.
• Openness is a double-edged property of AI: The authors contrast the "bazaar" bottom-up model of AI development (winning, in their framing, over the "Cathedral" top-down model) with the highly protected, competitive nature of chip manufacturing — noting that openness benefits research but complicates containment of misuse.
• Worked parameter count for the MLP example: With 20×20 pixel images (d₀ = 400) and layer widths 400→100, 100→100, 100→10, the layer-wise parameter counts are 40,100; 10,100; and 1,010, for a total of 51,210 trainable parameters — and the count does not depend on the number of training samples n.

Methodology in Plain English

This is a survey/tutorial, not an experimental paper, so there is no dataset, no benchmark, and no empirical evaluation to report. The approach is expository and connective:

• Build from the simplest case upward. The authors begin with a digit-classification multilayer perceptron rather than an LLM, explicitly to "establish notation and principles" in a simpler setting, and because MLPs are key components of Transformers.
• Translate everything into matrix language. Individual feature vectors are recast as rows of a sample matrix X, so that a layer becomes Z_l = σ(Z_{l-1} W_l + b_l) — a matrix–matrix product plus a broadcast bias — which is the form an NLA reader can reason about.
• Walk through the loss. They present the naive Frobenius-norm least-squares objective, explain why it is seldom used in favor of softmax plus cross-entropy, and give the softmax and cross-entropy formulas explicitly, including the one-hot encoding of labels.
• Ground the practice in numerical-analysis theory. Cybenko's 1989 denseness result (for continuous, discriminatory, sigmoidal activation functions) is presented as the justification that the approximation is achievable at all; Griewank's backward differentiation is presented as the reason training is computationally feasible.
• Then introduce stochastic optimization. Classical gradient descent with line search (Cauchy) is contrasted with SGD for "finite sum" objectives of the form φ(w) = (1/n) Σ φᵢ(w), motivating the replacement of the full gradient by a randomly sampled component gradient.
• Frame the whole thing historically and strategically, using scaling plots, a milestone timeline, and comparisons to chip manufacturing to argue about where the field is heading and what it means for the NLA community.

Why This Matters

Impact on research. The paper positions NLA as a first-class contributor to LLM progress rather than a service discipline — through LAPACK, low-rank approximation, and low-precision tensor computation — and argues that the field's culture gap with AI (notation, pace of publication, emphasis) is a real obstacle that numerical researchers must actively close. It also gives the NLA community a compact, matrix-native entry point into LLM literature.

Real-world applications (as grounded in the paper's framing):

• LLM training and inference at scale, where the quadratic cost of attention in the number of tokens is the dominant expense and where low-rank and low-precision tensor methods are the lever.
• Hardware-aware numerical software, since the authors stress GPUs (and, for JAX, custom Tensor Processing Units) as the enabling substrate for training very large models.
• Scientific and engineering computing broadly, given the paper's opening argument that NLA tools developed for aerodynamics-style problems have repeatedly transferred to new domains.
• Education and assessment, where the authors note that students can now use LLMs to solve most homework and exam questions, making conventional testing "rather challenging."

Industry relevance. The paper directly addresses the economics of the field: training a large language model is expensive largely because Transformers must handle many tokens at once with quadratic-cost attention, and it notes the talent-flow effect in which high-paying AI jobs draw students away from other specializations. It also tracks the tooling landscape that industry depends on — TensorFlow (Google Brain, circa 2015), PyTorch (Meta, then Linux Foundation in 2022, and the favored Python package for deep learning), JAX (introduced by Google as a TensorFlow replacement, with better hardware integration), Theano (2007), and Torch-7 (2011) — and ties national-security and containment concerns to the field's unusual openness.

Future Directions

1. How should the NLA / numerical analysis community adapt? The authors pose this as the central question of the article, and suggest a reasonable goal is to keep pursuing intellectual innovation in linear algebra while participating actively in deep-learning research — but they flag that differing culture, notation, and publication pace make this hard.
2. Can the scaling trend continue? The ≈4.6-month parameter doubling clearly cannot be sustained; the paper points to a data wall tied to finite training corpora and asks whether an AI analogue of Moore's Law can even be stipulated.
3. Beating quadratic attention cost. The stated need is for algorithms that deliver good accuracy with linear or near-linear scaling in the number of tokens, complementing hardware gains.
4. Containment and the AGI/ASI horizon. The paper raises containment as a necessity given the field's openness, and reports the AGI-by-2027–2028 argument and the subsequent "Artificial Super Intelligence" discussion as open, high-stakes questions rather than settled predictions.

Target Audience

Numerical linear algebraists, numerical analysts, and scientific-computing researchers who want a matrix-theory entry point into LLMs; graduate students and instructors in numerical methods who need to connect classical material (eigenvalue algorithms, sparse systems, optimization, approximation theory) to modern machine learning; and machine-learning practitioners who want to see their stack described in the vocabulary of matrix and tensor computation. The paper is explicitly not aimed at AI specialists — the authors state they are "by no means experts in AI" and that the goal is not a complete view of the field, but an exposition of LLM ingredients from a matrix-theory perspective.

Authors’ abstract

Numerical Linear Algebra (NLA) has consistently played a vital role in advancing science by providing tools to solve fundamental problems encountered in scientific and engineering applications. Over the decades, it has continually evolved to meet the demands driven by successive waves of scientific discovery. For instance, during the 1950s and 1960s, substantial efforts were devoted to developing methods for solving eigenvalue problems that emerged from the rapidly growing field of aerodynamics. This led to the discovery of the LR and QR algorithms. Later the attention turned to the solution of sparse linear systems that were common in applications like computational aerodynamics. Today we are experiencing yet another wave of major scientific advancement and NLA is once more at the heart of its development. This Machine Learning (ML) wave is proving to be utterly disruptive in science and engineering. Many tools in ML particularly Large Language Models (LLMs) are grounded in matrix and tensor methods. As we are approaching Artificial General Intelligence (AGI), it is clear that matrix methods will be called to play an even more significant role. For the numerical linear practitioner the speed of the current change makes it particularly challenging to adapt. This is a survey article that centers on machine learning techniques, with a particular focus on large language models. It has two main objectives. The first is to clarify the core concepts behind Large Language Models in a manner accessible to specialists in numerical methods. The second is to examine the key Numerical Linear Algebra concepts employed by LLM techniques, while also highlighting several significant recent contributions of NLA to the field.