Skip to content
AI.info

Deep architectures

Architecture Evaluation, Ablation, and Scaling Evidence

Build rigorous architecture comparisons using baselines, ablations, scaling curves, robustness slices, hardware measurements, and uncertainty-aware reporting.

By the end you can

An architecture claim is a conditional statement

“Architecture A is better” is incomplete. Better for which data regime, quality metric, training budget, hardware, batch shape, output length, and operating constraint? A credible claim names the comparison class and the boundary where evidence stops. Architecture results are rarely universal rankings.

Hold the architecture fixed and change everything around it, and the architecture stops looking like the cause. Eight authors did that to a canonical ResNet in 2021. Their paper opens on the confusion it was written against: “the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies”. What they found was that “training and scaling strategies may matter more than architectural changes”, and “that the resulting ResNets match recent state-of-the-art models”. The ResNet-RS family they got out of that old design is “1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet”. Bello and Fedus led the work. The architecture was their constant, not their variable.

Every architecture advantage should be attached to conditions, baselines, and measured trade-offs.

Comparison

Four evidence types answer different questions

One strong benchmark cannot replace the others.

Each column below has a published failure attached to it later in this lesson. The matched-baseline column has an audit of 18 neural recommendation algorithms. The ablation column has a large set of published Transformer modifications, re-run in one shared setting. The scaling column has the ranking of ten architectures moving as budgets grow. The robustness and systems column has MLCommons fixing a 99th-percentile latency budget before any throughput number is allowed to be reported.

FigureComparison · 4 columns

Matched baseline

Does the complete design outperform a credible alternative?

  • Same data and split
  • Comparable tuning effort
  • Matched quality or compute
  • Includes simple architecture

Ablation

Which component contributes to the measured change?

  • Remove one mechanism
  • Control confounders
  • Report interactions
  • Avoid post-hoc storytelling

Scaling study

How does behavior change with data, model, or compute?

  • Multiple budget points
  • Fit uncertainty
  • Identify crossovers
  • Avoid one-point extrapolation

Robustness and systems test

Does the advantage survive operational conditions?

  • Shift and corruptions
  • Latency and memory
  • Batch and sequence variation
  • Failure severity

Case

Over 400 trained models put model size and token count on the same footing

The third column is the one most often run at a single point. One 2022 study refused to. It trained “over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens”. The conclusion drawn from that spread is about a ratio, not a size: “for compute-optimal training, the model size and the number of training tokens should be scaled equally”. Twenty-two authors signed it, Hoffmann and Borgeaud first among them. The model their fit predicted makes the same point in one line. Chinchilla has 70B parameters and the same compute budget as the 280B Gopher, and it reaches “a state-of-the-art average accuracy of 67.5% on the MMLU benchmark”. A quarter of the parameters. The same compute.

Visual

The architecture cost stack

Different metrics dominate different deployment regimes. The field's usual shorthand is to count the FLOPs. The people who profiled that shorthand called it unreliable in 2018: “Currently, the neural network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs. However, the direct metric, e.g., speed, also depends on the other factors such as memory access cost and platform characterics.” That is the ShuffleNet V2 paper, from Ma and Zhang with two colleagues. The misspelling of the last word is in the published abstract.

A second group reached the same place from the language side. Parameter count, FLOPs and throughput are the three standard cost indicators. In 2022 five authors showed that they can contradict each other. Their Figure 1 compares standard Transformers, Universal Transformers and Switch Transformers on 64 TPU-v3, and the relative ranking reverses between cost indicators. Dehghani and Arnab called the paper The Efficiency Misnomer. Two labs, two modalities, one conclusion: the model with fewer FLOPs or fewer parameters can be the slower one. So the layers below are measured separately rather than collapsed into a single headline number.

FigureLayers · 6 layers
  1. 01

    Total parameters

    Storage, checkpoint movement, and optimizer state.

  2. 02

    Active operations

    Arithmetic performed for one training or inference pass.

  3. 03

    Activation and cache memory

    Intermediate states retained for gradients or decoding.

  4. 04

    Memory movement and communication

    Data transferred between hierarchy levels or devices.

  5. 05

    Critical-path latency

    Sequential operations that cannot be hidden by parallelism.

  6. 06

    End-to-end product cost

    Preprocessing, queueing, safety, retries, and infrastructure.

Example

Architecture comparisons that look fair but are not

These confounders can reverse a conclusion, and two published audits counted how often they did.

Somebody went through the neural recommendation literature and tried to run it again. The count is in the abstract: “Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort.” Of those 7, 6 could often be outperformed by comparably simple heuristics such as nearest-neighbour or graph-based methods. Ferrari Dacrema and two co-authors published that in 2019.

A year later Google re-ran the neural collaborative filtering experiments. With proper hyperparameter selection, a plain dot product substantially outperforms the learned MLP similarity it was said to beat. Rendle and Krichene led that one, with two colleagues. The confounder was the tuning budget. Removing it cost the architecture its result.

  • One model receives stronger augmentation or more pretraining data while the paper attributes the gain to its block design.
  • A sparse model reports active FLOPs but omits all-to-all communication and total parameter storage.
  • A Transformer and CNN use different image resolution, optimizer schedules, or tuning budgets — the confounder Rendle and Krichene's group removed before a plain dot product overtook the learned MLP similarity.
  • A long-context model is measured at maximum length while its baseline is measured at a shorter, faster setting.
  • A generative architecture is compared with a different sampler, guidance scale, or number of decoding steps.

Analogy

A controlled vehicle test, not a showroom comparison

Two vehicles are compared on the same route, load, weather, fuel quality, and driver policy; removing one component then reveals whether it caused the change.

Training randomness and hyperparameter interactions create more uncertainty than a mechanical test does. Matched conditions and repeated runs matter for that reason.

A fair architecture experiment controls the surrounding recipe before attributing differences to topology.

Key idea

Ablations can mislead when components interact

Removing a component from a tuned system can create an unfairly broken variant; the remaining hyperparameters may have been selected around the original architecture.

The reverse direction has been run at scale. Sixteen authors took a large set of published Transformer modifications and re-ran them in one shared experimental setting. The 2021 paper reports the outcome in a single sentence: “Surprisingly, we find that most modifications do not meaningfully improve performance.” The variants that did help were mostly developed in the same codebase the authors used. The gain travelled with the recipe, not with the component. Narang and Chung led that study. The same conclusion arrives from the opposite budget end. Geiping and Goldstein trained a language model on a single GPU in one day in 2023. Their architecture section concludes that scaling laws create a barrier to making major gains with architectural modifications beyond computational efficiencies.

Report whether variants were retuned and include additive as well as subtractive tests where possible; interaction effects should be acknowledged instead of forced into independent contributions.

An ablation estimates contribution inside a design context, not a context-free causal effect.

Steps

Run an architecture study that another team can reproduce

Predeclare the comparison and record every budget that can influence the result.

Step 4 is the one that gets skipped, and there is a measurement of what skipping it costs. Ten trials of TRPO on HalfCheetah-v1, varying nothing but the random seed, split into two groups of five. The two groups differed at t = −9.0916, p = 0.0016. Two statistically different distributions came out of identical hyperparameters. Henderson and Islam ran that in 2018 with four co-authors. Three years later another group named the habit in its abstract: “Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs.” On Atari 100k, Agarwal and Schwarzer's team found substantial discrepancies between the conclusions drawn from point estimates and those drawn from interval estimates. That paper won an outstanding paper award at NeurIPS in 2021.

Step 1 and the tail-latency half of step 4 have a public model. MLPerf Inference v5.0 added a Llama 3.1 405B benchmark. Before any throughput could be claimed, MLCommons wrote the operating constraint into the rules: “To balance the demands of long-context processing with real-world usability, we set a 99th percentile TTFT of 6 seconds and a 99th percentile TPOT of 175ms.” The same round tightened Llama 2 70B Interactive to a 99th-percentile TTFT of 450 ms and TPOT of 40 ms. Its results were published on 2 April 2025. Under those constraints NVIDIA reported that its GB200 NVL72 submission delivered up to 30x the throughput of its own H200 NVL8 submission on the 405B benchmark. That figure means something precisely because the latency budget was fixed first.

FigureProcess · 5 steps

  1. 1. Define the decision

    State the product constraint and the architecture choice it informs.

  2. 2. Select credible baselines

    Include simple, established, and capacity-matched alternatives.

  3. 3. Equalize the recipe

    Control data, augmentations, tuning budget, seeds, and stopping rules.

  4. 4. Measure distributions

    Report repeated runs, confidence intervals, and tail latency.

  5. 5. Publish failure boundaries

    List regimes where the preferred architecture loses or becomes impractical.

Scaling curves can cross

An architecture with strong inductive bias may win at small data and lose after extensive pretraining, and a computationally expensive design may become preferable only after quality requirements exceed a threshold.

That crossing has been measured rather than merely warned about. Ten authors ran a systematic scaling study of ten architectures, among them Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers and MLP-Mixers. The result: “Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales.” The phrasing “is an indeed an” is as published. Tay and Dehghani led it. A ranking measured at one budget need not hold at another.

Measure several data, model, and compute points before extrapolating. Fit uncertainty and distinguish interpolation from claims beyond the observed range.

One budget point cannot reveal the shape or crossover of an architecture scaling curve.

Position

A scaling law is a curve someone fitted, not a promise someone made

Over four hundred language models were trained to produce the one sentence everybody quotes, and the sentence is almost always quoted without them. That is how scaling arguments travel. The result is a fit. The fit had a range. The range is the first thing dropped when a roadmap needs backing.

Look at what the range was. Models from 70 million to over 16 billion parameters, trained on 5 to 500 billion tokens. The conclusion drawn inside it is about a ratio rather than about size: “for compute-optimal training, the model size and the number of training tokens should be scaled equally”. The model that came out of it makes the same point in a line. Chinchilla, 70B parameters at the same compute budget as the 280B Gopher, reaches a state-of-the-art average accuracy of 67.5 per cent on MMLU. A quarter of the parameters, the same compute. A reader who took that curve as evidence that larger is better was reading one axis of a fit that has two.

That a fit has an owner is not an abstraction either. Two labs fitted the same phenomenon and prescribed opposite things. In 2020 OpenAI concluded that optimally compute-efficient training means very large models on relatively modest data. In 2022 DeepMind concluded that model size and tokens should be scaled equally. Kaplan and McCandlish signed the first, Hoffmann and Borgeaud the second. Five authors took the disagreement apart in 2024: “We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning.” Correct those three and the two laws agree. Porian and Wortsman found the gap between the era's two most cited scaling prescriptions in the recipe, not in the topology.

The Chinchilla fit has since been audited in the same spirit. Four researchers at Epoch AI reconstructed the data behind the third of the paper's three estimation procedures. Their verdict: “We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals”. Intervals that narrow would require over 600,000 experiments. Hoffmann and his co-authors likely ran fewer than 500. Besiroglu and Erdil's rederivation restores agreement with the other two approaches. The dispute is live and documented on both sides. A 2025 paper from Schaeffer and Levi, with five co-authors, opens by acknowledging the concerns raised — “wide confidence intervals, discrepancies between its three approaches” — and concludes that the key results survive sizeable perturbations.

The same caution applies to what a scaling result is attached to. Bello and his co-authors made the point in 2021. They observed that the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies. Then they found that training and scaling strategies may matter more than architectural changes, producing from a canonical ResNet a family 1.7 to 2.7 times faster than EfficientNets on TPUs at similar ImageNet accuracies. The topology was the part they did not change.

None of this makes scaling evidence weak. It makes it evidence, with an owner and a domain, and the section above has already said what that means in operation: curves can cross. When a plan cites a scaling law, the questions are which family was fitted, over which range, holding which recipe fixed, and how far beyond the last measured point the plan is standing. Inside its data the curve is a measurement. Outside it, the curve is a forecast, and somebody is making it.

More than four hundred models bought one sentence about a ratio. It was never a sentence about size on its own.

A useful architecture report ends with a decision table

Summarize quality, robustness, training cost, inference cost, memory, implementation maturity, and operational risk for each candidate. Separate measured values from assumptions. Keep the cost columns apart rather than collapsing them. Parameter count, FLOPs and throughput ranked the same three Transformer variants in contradictory orders in Dehghani and Arnab's Figure 1. A table with one efficiency column has already made that choice on the reader's behalf.

Whoever writes that summary should name a rollback or fallback option, and the evidence that would make the team think again. Choosing an architecture is a revisable engineering decision.

The report should make disagreement traceable to evidence, constraints, or risk tolerance.

Key takeaways