Technical Deep Dives

Large Language Models: Architecture, Training, and Scaling

How large language models are built, from byte-pair tokenization and scaled dot-product attention to Chinchilla scaling, RLHF, and the trillion-parameter sparse mixtures of experts that open labs shipped in 2026.

Gabriele Masetti · 2026-08-12

The Architecture: Attention Without Recurrence

The modern LLM lineage begins with a single architectural substitution. "Attention Is All You Need" (Vaswani et al., 2017, NeurIPS) replaced the recurrent and convolutional backbones that had dominated sequence modeling — LSTMs, GRUs, ConvS2S — with a mechanism built entirely on attention. The core operation is scaled dot-product attention:

Attention(Q, K, V) = softmax(QKᵀ / √d_k) V

Queries, keys, and values are linear projections of the input embeddings; the softmax over QKᵀ produces a distribution over positions that each token uses to aggregate information from the rest of the sequence. Because every token can attend to every other token in a single operation, the transformer parallelizes across the sequence dimension in a way recurrent networks cannot — RNNs process tokens sequentially, so training time scales with sequence length in a way that resists GPU parallelism.

The original paper reported 28.4 BLEU on WMT 2014 English-to-German translation, a gain of more than 2 BLEU over the best prior result, achieved with substantially less training compute than competing recurrent and convolutional systems.

The paper's other key contribution, multi-head attention, runs several attention operations in parallel over different learned projections of Q, K, and V, then concatenates the results. Each head can specialize — some heads in trained transformers attend to adjacent tokens, others to syntactic dependencies or coreference chains. Because attention has no inherent notion of order, the original architecture added sinusoidal positional encodings to the input embeddings so the model could distinguish "dog bites man" from "man bites dog."

Two architectural lineages descend from this design. Encoder-only models like BERT (Devlin et al., 2018) use bidirectional self-attention and are trained with masked-language-modeling objectives, well suited to classification and representation tasks. Decoder-only models — the GPT lineage from GPT-1 onward, LLaMA, Mistral, and most contemporary chat-oriented LLMs — use causal (masked) self-attention, where each position can only attend to earlier positions, and are trained to predict the next token autoregressively. The decoder-only design won out for general-purpose generative LLMs because a single causal-attention stack supports both pretraining and generation without an architectural break between encoding and decoding.

Tokenization: From Text to Integers

Before any attention computation happens, raw text must become a sequence of integers. Byte Pair Encoding (BPE), originally a data-compression technique from Philip Gage (1994), was adapted for neural machine translation by Sennrich, Haddow, and Birch (2016). Their algorithm starts from individual characters (or bytes) and iteratively merges the most frequent adjacent pair into a new symbol, repeating until a target vocabulary size is reached.

Common words collapse into single tokens; rare or unseen words decompose into smaller, previously-seen subword pieces, which eliminates the fixed-vocabulary out-of-vocabulary problem that plagued earlier word-level models. BPE and its close variants (byte-level BPE, used by GPT-2 and its descendants; SentencePiece, used by many multilingual and open models) now underlie the tokenizers of GPT-2, GPT-3, GPT-4, RoBERTa, LLaMA, and nearly every production LLM released since.

Tokenization is not a cosmetic preprocessing step — it directly shapes model behavior. Vocabulary size trades off against sequence length: a larger vocabulary compresses more text into fewer tokens (cheaper attention, since cost scales with sequence length) but enlarges the embedding and output-projection matrices. Tokenizer boundaries also explain well-documented LLM quirks, such as poor character-level arithmetic or spelling performance, since the model never sees individual characters directly but rather opaque subword chunks.

Pretraining Objectives and the Data Pipeline

Decoder-only LLMs are pretrained with a single, deceptively simple objective: given a sequence of tokens, predict the next one, minimizing cross-entropy loss averaged over a massive corpus. That is the same objective GPT-1 (Radford et al., 2018) used when it demonstrated that unsupervised pretraining followed by supervised fine-tuning outperformed training task-specific architectures from scratch.

GPT-2 (Radford et al., 2019) pushed the same recipe further and showed that sufficiently large next-token predictors perform many downstream tasks in a zero-shot setting, with no task-specific fine-tuning at all — simply by being prompted appropriately. GPT-3 (Brown et al., 2020) scaled this to 175 billion parameters and demonstrated strong few-shot, in-context learning: providing a handful of input-output examples in the prompt, with no gradient updates, was enough to reach competitive performance on translation, question answering, and arithmetic-style tasks.

Encoder-only models use a different objective. BERT is trained with masked language modeling (predicting randomly masked tokens from bidirectional context) plus, in the original paper, a next-sentence-prediction auxiliary task. The bidirectionality makes BERT-style models strong at representation learning for classification and retrieval, but unsuitable for open-ended autoregressive generation, which is one reason the decoder-only, next-token-prediction recipe became the dominant paradigm for general-purpose LLMs.

Scaling Laws: Turning Model Building Into Prediction

Perhaps the most consequential empirical result in this field is that pretraining loss behaves predictably as a function of scale. Kaplan et al. (2020), "Scaling Laws for Neural Language Models," showed that cross-entropy loss follows a power law in model size, dataset size, and compute, holding across more than seven orders of magnitude, with architectural details such as depth-to-width ratio mattering comparatively little within a wide range. A simplified form of the relationship is:

L(N) ≈ (N_c / N)^α

where N is parameter count and α is an empirically fit exponent well below 1, meaning returns diminish but remain predictable rather than erratic. The practical implication of the Kaplan analysis was that, under a fixed compute budget, it is more efficient to train very large models on comparatively modest data and stop well short of convergence than to fully converge a smaller model.

Hoffmann et al. (2022), the Chinchilla paper from DeepMind ("Training Compute-Optimal Large Language Models"), revised this conclusion. Training over 400 models ranging from 70 million to more than 16 billion parameters on token counts from 5 billion to 500 billion, and cross-checking the result with three independent estimation methods, the authors found that compute-optimal training requires scaling model size and training tokens at roughly equal rates — as a practical rule of thumb, on the order of 20 training tokens per parameter.

Under Kaplan-style scaling, contemporaneous models like Gopher (280B parameters) had been substantially undertrained relative to their size. To validate the revised law, DeepMind trained Chinchilla, a 70B-parameter model on 1.4 trillion tokens, using the same compute budget as Gopher; Chinchilla outperformed the four-times-larger Gopher on a broad evaluation suite while also being cheaper to run at inference time, since inference cost scales with parameter count, not training tokens.

That result reoriented the field: LLaMA (Touvron et al., 2023) explicitly built on Chinchilla-style reasoning, training smaller models (7B–65B parameters) on more tokens than earlier scaling laws would have recommended, reporting that LLaMA-13B outperformed the 175B GPT-3 on most benchmarks and that LLaMA-65B was competitive with Chinchilla-70B and PaLM-540B.

From Pretrained Model to Assistant: Instruction Tuning and RLHF

A model trained purely to predict the next token on internet text is not automatically a helpful assistant — it is a very capable text continuer, which is not the same thing as a system that answers questions, follows instructions, or declines unsafe requests. Ouyang et al. (2022), "Training Language Models to Follow Instructions with Human Feedback," introduced the InstructGPT recipe that closed this gap and underlies most deployed chat models today.

The pipeline has three stages: supervised fine-tuning on human-written demonstrations of desired behavior; training a reward model on human rankings of multiple model outputs for the same prompt; and using that reward model to fine-tune the policy with reinforcement learning (proximal policy optimization).

The paper's headline result is striking: human labelers preferred outputs from a 1.3-billion-parameter InstructGPT model over outputs from the 175-billion-parameter base GPT-3, despite the RLHF-tuned model being more than 100 times smaller, and the tuned models showed measurable gains in truthfulness and reductions in toxic generation with only minor regressions on standard NLP benchmarks.

RLHF (and its variants, including reward-model-free approaches such as direct preference optimization) has since become the standard bridge between a raw pretrained model and a deployable assistant, and it is the reason "prompting a base model" and "talking to ChatGPT or Claude" are such different experiences even when the underlying pretrained weights are related.

Sparse Architectures: Mixture of Experts

Dense transformers activate every parameter for every token, which means inference and training cost scale directly with total parameter count. Mixture-of-Experts (MoE) architectures break this coupling by replacing a dense feed-forward block with a bank of expert feed-forward networks and a learned router that activates only a subset of them per token.

The Switch Transformer (Fedus, Zoph, and Shazeer, 2021) simplified prior MoE routing schemes by routing each token to exactly one expert (top-1 routing rather than top-k), which reduced communication overhead and training instability while still scaling dense T5-based models to hundreds of billions and into the trillion-parameter range, reporting up to 7x pretraining speedups over dense baselines at equivalent compute.

Mixtral 8x7B (Mistral AI, 2023–2024) brought sparse MoE to a widely used open-weight model. It uses the same underlying architecture as Mistral 7B, but each transformer layer contains eight expert feed-forward blocks, with a router selecting the top 2 experts per token per layer. The result is a model with roughly 47 billion total parameters of which only about 13 billion are active for any given token — giving Mixtral much of the quality benefit of a larger dense model at closer to the inference cost of a smaller one.

Trained with a 32k-token context window, the base model was reported to match or outperform Llama 2 70B and GPT-3.5 across evaluated benchmarks, with particularly large margins on mathematics, code generation, and multilingual tasks, and the instruction-tuned variant was reported to surpass GPT-3.5 Turbo, Claude 2.1, Gemini Pro, and Llama 2 70B Chat on human-evaluated benchmarks.

The benchmark comparison is a 2023 artefact, but the design outlived it, and at a scale Mixtral only gestured at. DeepSeek-V4-Pro, released on 24 April 2026 under an MIT licence, carries 1.6 trillion total parameters and activates 49 billion per token, with a 1-million-token context and 80.6% on SWE-bench Verified; its smaller sibling V4-Flash runs 284 billion total and 13 billion active. Moonshot's Kimi K3 goes further still: 2.8 trillion total parameters, 104 billion active, 896 experts with 16 routed per token. Mixtral's ratio of total to active parameters was about 3.6 to 1; DeepSeek-V4-Pro's is roughly 33 to 1 and Kimi K3's roughly 27 to 1.

MoE illustrates a broader principle in scaling LLMs: total parameter count and active (compute-bound) parameter count are no longer the same quantity, and architecture choices increasingly target that distinction directly rather than treating "model size" as a single number.

Positional Encoding and Long Context

The original sinusoidal positional encoding is largely absent from current-generation LLMs. Rotary Position Embeddings (RoPE), introduced by Su et al. in the RoFormer paper (2021), instead rotate query and key vectors by an angle proportional to token position before the attention dot product, so that the dot product between any two positions becomes a function of their relative distance rather than their absolute positions.

The relative-position property, combined with the fact that RoPE requires no learned parameters and can be applied directly inside fused attention kernels, made it the de facto standard in LLaMA-family and most subsequent open models, and it underlies techniques for extending context length after pretraining by rescaling the rotation frequencies.

Extending usable context has been a parallel engineering track to the modeling-objective work above. FlashAttention (Dao et al., 2022) is not a new attention formula but an IO-aware exact implementation of the same softmax(QKᵀ/√d_k)V computation: by tiling the computation and fusing operations to avoid materializing the full attention matrix in slow GPU memory, it reduces memory scaling from quadratic to linear in sequence length while computing mathematically identical results, which is a major reason long-context training became computationally tractable at all.

On the architecture side, grouped-query attention (GQA) reduces the number of distinct key/value projections shared across query heads, interpolating between standard multi-head attention and the more aggressive multi-query attention, cutting the size of the key-value cache that must be stored during autoregressive decoding — a direct lever on inference cost and achievable context length.

The combined effect of these techniques is visible in deployed systems, and the ceiling has moved twice. GPT-4 launched with an 8k context and a 32k variant, while GPT-4 Turbo extended this to 128k tokens; Claude models expanded from tens of thousands of tokens to a widely used 200k-token context; and Gemini 1.5 Pro was released in 2024 with a 1-million-token context window that was then a record. By 2026 a million tokens had become a default rather than a headline: DeepSeek's V4 release of 24 April 2026 made a 1M-token window standard across its official services, and Moonshot's open-weight Kimi K3 ships the same window. None of these gains came from a new attention mechanism alone — they are the compound result of positional-encoding changes, IO-aware kernels, and KV-cache-efficient attention variants working together.

Where the Threads Converge

None of these components is sufficient in isolation. A transformer without BPE-style subword tokenization cannot handle open vocabulary text efficiently; a model trained purely on next-token prediction without RLHF is a raw completion engine rather than an assistant; scaling laws without an architecture capable of using additional compute (attention's parallelism, MoE's conditional computation) would simply describe a wall rather than a path forward.

What Kaplan (2020) and Hoffmann (2022) added was not a new component but a predictive theory of how the existing components — parameters, data, and compute — trade off, which turned frontier model development from trial-and-error into resource allocation under a known cost curve. The trajectory from Vaswani's original 65-million-parameter base transformer to GPT-3's 175 billion parameters, to Chinchilla's demonstration that a correctly-provisioned 70B model beats an undertrained 280B one, to Mixtral's 47B-total/13B-active sparse design and on to the trillion-parameter sparse models of 2026, is not a story of one idea winning but of the attention mechanism, tokenization, training objectives, RLHF, and scaling laws being refined largely independently and then recombined — which is also why isolated benchmark comparisons between model "generations" are often less informative than comparing which of these five levers actually changed.