Scaling Laws, Data Budgets, and Compute Allocation

More parameters do not settle the question. The cleanest demonstration is a published scaling prescription that a later experiment reversed.

Cross-entropy loss falls as a power law in model size, dataset size and training compute. That was the finding of Scaling Laws for Neural Language Models in 2020, fitted across trends spanning more than seven orders of magnitude. The prescription that came with it was explicit: “Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.”

Two years later Chinchilla overturned exactly that prescription. A larger model can underperform a smaller model trained on more appropriate data or for more tokens. How well the training goes depends on how model capacity, dataset size, optimization and compute are allocated together.

Note what did not fail in 2020. The power law was real. It was fitted over more than seven orders of magnitude, and it still did not survive a different allocation of the same compute.

That is what a scaling law is: an empirical trend inside a measured regime. It is useful for estimating tradeoffs. It is not a guarantee of product quality, safety, or transfer to a different architecture and data distribution. Kaplan and colleagues named their regime honestly. Readers two years later were the ones who treated it as permanent.

A decision that improves one stage can increase cost elsewhere. The link between training and serving has now been priced rather than merely asserted.

Serving cost belongs inside the law. Beyond Chinchilla-Optimal, from Sardana and colleagues in 2023, modified the Chinchilla laws to include inference and trained 47 models to validate the result: “We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal.” In their analysis quality kept improving out to 10,000 tokens per parameter.

The number that moves the optimum is a demand forecast — roughly 1 billion inference requests — not an architecture choice. A team that expects that volume and optimizes the training budget alone will build the wrong model, correctly.

Each choice changes the evidence needed for success. Two of the three now have a measured limit rather than a caution.

Repeating data pays almost as well as fresh data at first, and then less and less. Scaling Data-Constrained Language Models, from Muennighoff and colleagues in 2023, ran roughly 400 training runs, up to 900 billion training tokens and 9 billion parameters: “We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero.” Four epochs is a number to plan against. Past it, each further pass is worth less than fresh text; the paper finds meaningful gains up to around 16 epochs and almost none by 40.

Inference compute has a measured payoff and a stated precondition. Snell and colleagues allocated test-time compute per prompt according to difficulty and improved efficiency by more than 4x over a best-of-N baseline. That was 2024. The FLOPs-matched comparison is the striking one: “Additionally, in a FLOPs-matched evaluation, we find that on problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model.” Read the condition as carefully as the 14x. The payoff was measured on problems where the small model already had non-trivial success rates. That is precisely where extra deliberation has something to work with.

Compare alternatives under the same decision target. Every anchor above is an experiment someone ran because a plausible argument was not enough: over 400 models for the equal-scaling rule, roughly 400 runs for the repetition threshold, 47 models to price inference into the law. The steps below are the small version of the same discipline.

The five steps can be run on an invented budget, using two pieces of this lesson: the Epoch refit L(N,D) = 1.8172 + 482.01/N^0.3478 + 2085.43/D^0.3658, and the cost form Sardana and colleagues minimise, about 6 × N × D training FLOPs plus 2 × N per generated token. The product, the budget and the demand forecast are invented; every figure is computed from those inputs.

Larger training programs can produce broad capabilities. They also increase data, evaluation, security and serving obligations. And the smaller model wins often enough that it deserves a named case rather than a hedge.

LLaMA is that case. Touvron and colleagues at Meta AI trained 7B–65B models in 2023 on trillions of tokens from publicly available datasets only, and reported: “In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B.” Thirteen billion parameters against 175 billion. The smaller model is also the one you can afford to serve.

Small models can dominate when latency, privacy or specialization matters, and the published record now says they can also dominate on capability at a fraction of the size. The next lesson examines benchmarks and contamination. A scaling claim becomes useful only when the evaluation evidence measures the behavior the product needs — which is why it mattered, above, that 67.5% turned out to be an average over 57 subjects and 14,079 test questions.