How machines learn
Parameters, Hyperparameters, and Model Capacity
Build a clear mental model of learned parameters, chosen hyperparameters, model capacity, and the trade-offs created by flexible function families.
By the end you can
- Distinguish learned parameters from hyperparameters and fixed configuration
- Explain model capacity as the range of patterns a model can represent
- Recognize how capacity interacts with data, noise, and regularization
- Describe why a larger model is neither automatically better nor automatically overfit
Comparison
The settings inside a learning project do not all change the same way
Confusing parameters and hyperparameters makes experiments difficult to interpret.
Learned parameters
Values adjusted by the training algorithm to reduce the objective.
- Derived from training examples
- Can number from a few to billions
- Stored as part of the trained model
- Example: coefficients in a linear predictor
Hyperparameters
Choices that shape the model or training procedure.
- Selected by people or a search process
- Compared using validation evidence
- Not fitted directly as ordinary parameters
- Example: tree depth or regularization strength
Fixed system configuration
Operational choices outside the mathematical model.
- May affect data or deployment behavior
- Needs versioning and testing
- Can invalidate comparisons when changed
- Example: feature window or timeout policy
Capacity is the size of the model's possible answer space
A low-capacity model can express only a restricted family of relationships, while a higher-capacity model can fit more complicated boundaries, interactions, or sequences. Capacity is not just a parameter count. Architecture, parameterization, constraints, and effective regularization shape it as much as size does.
Cut a trained network down to a small fraction of its weights and it can still do the job. In 2018 Jonathan Frankle and Michael Carbin pruned trained networks, then retrained the surviving subnetwork from its original initialization. Their abstract reports: “We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10.” Those subnetworks matched the full network's test accuracy in a comparable number of iterations. A subnetwork under 10–20% of the original size holds the accuracy. The count was never the thing doing the work. Four researchers at Uber AI re-ran the procedure in 2019 and took its three components apart one at a time.
The useful question is whether the model family can represent the stable structure of the task without fitting noise and accidents, and the answer depends on the data and on the evaluation setting.
Shuffle the labels at random and a standard image classifier learns them anyway. That is the outer edge of the question, and in 2017 it was tested by wrecking the data rather than by changing the model. Standard image classifiers, Chiyuan Zhang and four co-authors report, “easily fit a random labeling of the training data”. The phenomenon “is qualitatively unaffected by explicit regularization”, and it “occurs even if we replace the true images by completely unstructured random noise”. Training error went to zero on data that contained nothing to learn. A shortage of capacity, then, is not what keeps these models honest. A model that fits its training set proves a good deal less than it appears to.
Capacity is relative to a task and dataset; it is not a universal label attached to a model.
Analogy
An analogy: choosing a stencil for a drawing
A coastline can be traced with a rigid ruler, a flexible curve, or an unconstrained thread. The ruler misses real bends; the thread can follow every pebble and measurement wobble.
Model capacity behaves similarly: too little flexibility misses structure, while excessive flexibility can follow noise. Training algorithms, regularization, and data scale can make a flexible model behave far more simply than its maximum theoretical capacity.
Key idea
Parameter count is an incomplete capacity measure
Two models with the same number of parameters can express very different functions, because architecture, connectivity, activation, and constraints all change what the parameters are allowed to do. The converse holds as well. A large model may generalize well when trained on abundant data with suitable regularization.
Avoid statements such as "more parameters always mean overfitting." Parameter count is one clue, not a complete diagnosis.
There is a published curve behind that warning. Test error traces the textbook U as capacity grows. It peaks where the model becomes just able to interpolate the training data. Then it descends a second time. Mikhail Belkin and three co-authors reported that curve in PNAS in 2019: “increasing model capacity beyond the point of interpolation results in improved performance”. Preetum Nakkiran and colleagues found the same double descent in modern networks in 2019, as a function of model size, dataset size and training time. A parameter count says nothing about which side of that peak a model is on.
The cleanest demonstration holds compute fixed and lets size vary. Over 400 models, from 70 million to 16 billion parameters, trained on 5 to 500 billion tokens: that is what Jordan Hoffmann and colleagues at DeepMind ran in 2022. Their finding: “We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.” Model size and training tokens should be scaled equally. Chinchilla has 70B parameters and was trained on 1.4 trillion tokens, about 20 tokens per parameter. It spent the same compute budget as the 280B-parameter Gopher and reached 67.5% average accuracy on MMLU, over 7% better than Gopher. It also beat GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B). A quarter of Gopher's parameters, ahead of every one of them: the count was not the axis that decided the outcome.
The regulator drawing a binding capability line made the same judgement. The EU's AI Act, published in 2024, presumes a general-purpose AI model to have “high impact capabilities”, and therefore systemic risk, when the cumulative training compute exceeds 10^25 floating-point operations. That is Article 51(2). Recital 111 gives the reasoning: “According to the state of the art at the time of entry into force of this Regulation, the cumulative amount of computation used for the training of the general-purpose AI model measured in floating point operations is one of the relevant approximations for model capabilities.” The number of parameters of the model is not the headline test at all. It sits in Annex XIII as one criterion among data size, compute, modalities and benchmarks.
Judge effective behavior with learning curves and held-out evidence, not a single size number.
Visual
A ladder of flexibility
Many hyperparameters create nested families: increasing a setting allows the model to express everything the smaller setting could, plus additional patterns.
Simple family
Few degrees of freedom; strong assumptions; low variance but possible systematic bias.
Moderate family
Enough flexibility for common interactions while still constrained.
Rich family
Can model complex relationships and rare combinations; needs stronger evidence and control.
Unconstrained fit
Can reproduce training peculiarities with little reason to expect transfer.
Validation can be overfit through repeated choices
Hyperparameters are not ordinary learned parameters. Repeated experimentation can adapt them to the validation set anyway, and once hundreds of decisions are made after observing the same validation results, that set stops being independent evidence.
Keep a final test set untouched. Record the number and logic of trials, and use fresh evaluation data when a project has undergone extensive tuning.
A reported advance can live in the search rather than in the model. That has been measured directly. In 2018 several popular recurrent architectures were re-run, this time “with large-scale automatic black-box hyperparameter tuning”. Gábor Melis and two co-authors reached the “somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models”, and set “a new state of the art on the Penn Treebank and Wikitext-2 corpora” with the design those newer models had supposedly replaced. Nothing was wrong with the newer architectures. They had been compared against baselines nobody had tuned as hard.
How the trials are drawn counts as much as how many there are. In 2012 James Bergstra and Yoshua Bengio put a number on it in the Journal of Machine Learning Research: “Compared with neural networks configured by a pure grid search, we find that random search over the same domain is able to find models that are as good or better within a small fraction of the computation time.” On a 32-dimensional deep-belief-network space, random search matched a hand-plus-grid search on four of seven datasets and beat it on one of seven. Their explanation is that “for most data sets only a few of the hyper-parameters really matter, but ... different hyper-parameters are important on different data sets”. The same model family can therefore win or lose on how its hyperparameters were sampled. That is why the size and the method of the search belong in the write-up, beside the architecture.
And a heavily re-used test set eventually reports on the community as much as on the model. The CIFAR-10 and ImageNet test sets were rebuilt in 2019 by re-running the original collection process. Benjamin Recht and three co-authors wrote: “We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet.” The size of that effect is itself contested. Logan Engstrom and colleagues at MIT showed in 2020 that after correcting statistical bias in the replication, only 3.6% ± 1.5% of the original 11.7% ± 1.0% ImageNet drop remained unexplained. Fresh evaluation data moved the numbers. Working out how much of the move was real took a second analysis.
Example
A demand forecast that needs interaction, not memorization
This exact choice was put to thousands of teams and scored. The M5 Accuracy competition ran on Kaggle from 2 March to 30 June 2020, with $100,000 in prizes. “The competition’s objective was to produce the most accurate point forecasts for 42,840 time series that represent the hierarchical unit sales of the largest retail company in the world, Walmart.” Spyros Makridakis and two colleagues, who ran it, published the results in the International Journal of Forecasting. Some 7,092 participants on 5,507 teams from 101 countries forecast 28 days of daily unit sales across 3,049 products, 10 stores and 3 US states, given 1,969 days of history running from 2011-01-29 to 2016-06-19. The winner, YeonJun Im, beat the best statistical benchmark by 22.4%. The winning entry was an equal-weighted average of LightGBM models fitted per store (10 models), store-category (30) and store-department (70). The capacity came from the grouping, not from a lookup table.
- A single global average over all 42,840 series cannot respond to stores, products, weekdays, or promotions, so it underfits every one of them.
- Store and product structure captures stable differences without learning every date independently: the winning entry fitted LightGBM models per store (10 models), store-category (30) and store-department (70).
- Grouping is what makes interactions affordable — a promotion response shared inside a department is learned once from many series instead of separately for each of the 3,049 products.
- An unrestricted identifier-by-date lookup could reproduce all 1,969 days of history exactly and still have nothing to say about the 28 days actually being forecast.
- The payoff was measured rather than assumed: all top 50 submissions beat the best statistical benchmark by more than 14%, and averaging the grouped models with equal weights shrank the influence of any poorly supported one.
Steps
Compare capacity without changing five things at once
A controlled capacity experiment makes the trade-off visible.
1. Freeze the data
Use identical partitions, preprocessing, and evaluation code.
2. Choose an ordered family
Vary one interpretable capacity control such as depth, degree, or regularization.
3. Track two curves
Compare training and validation performance for each setting.
4. Inspect errors
Check whether added capacity fixes systematic mistakes or only rare training cases.
5. Confirm once
Use the final test set only after selecting the approach.
Describe capacity choices without hype
"The bigger model is smarter" explains nothing. Say instead what new relationships it can represent, what evidence supports the added flexibility, and which constraints keep it from chasing noise. Hoffmann and colleagues did not report that Chinchilla was better. They reported 67.5% average accuracy on MMLU from a 70B-parameter model, at the same compute budget as the 280B-parameter Gopher, and they named the mechanism: tokens scaled with parameters.
This language connects architecture choices to observable behavior and makes later debugging easier. It also prevents model size from becoming a substitute for problem understanding.
Key takeaways
- Learned parameters are fitted from examples, while hyperparameters and system configuration are selected around the fitting process.
- Model capacity describes the family of relationships a model can express, not its parameter count: Frankle and Carbin retrained subnetworks under 10–20% of the original size on MNIST and CIFAR10 to the full network's test accuracy.
- Too little capacity creates systematic error, and excessive effective capacity follows noise — standard image classifiers have been driven to zero training error on random labels and on completely unstructured random noise.
- More parameters is not the axis that decides the outcome: at a matched compute budget the 70B-parameter Chinchilla beat the 280B-parameter Gopher, reaching 67.5% average accuracy on MMLU, and the AI Act likewise puts training compute, not parameter count, in Article 51(2).
- Repeated hyperparameter decisions can overfit the validation set even without gradient updates on it, and the search itself can carry the result — properly regularised LSTMs outperformed newer models under large-scale tuning, and random search matched or beat grid search in a small fraction of the computation time.
- Capacity experiments should vary one control, compare training and validation behavior, inspect errors, and reserve final test evidence — fresh test sets built by re-running the original collection process cost 3% - 15% accuracy on CIFAR-10 and 11% - 14% on ImageNet.