Skip to content
AI.info

Research

Mechanistic Critics for Sample-Efficient NPU Design-Space Exploration

CriticNPU's online-calibrated mechanistic critic filters weak NPU designs before simulation and localizes bottlenecks, making LLM-inference accelerator design-space exploration less brute-force.

arXiv
2611.24733
Published
2026-11-18
Authors
Yicheng He, Yuqi Xue, Tianyang Wu, Jian Huang, Bin Hu

AI summary

Overview

Research area: Computer architecture (cs.AR), specifically automated design-space exploration (DSE) for neural processing units (NPUs) targeting large language model (LLM) inference. Technical level: Intermediate — readers should be comfortable with accelerator architecture terms (systolic arrays, HBM bandwidth, TTFT/TPOT) and with the idea of a Roofline model, though the paper's core idea is explained geometrically. Scope: The paper proposes CriticNPU, a framework that replaces purely scalar simulator feedback in NPU design-space exploration with a calibrated, physics-based "mechanistic critic" that screens candidate configurations before simulation and diagnoses bottlenecks after it.

Note on the text: one paragraph in the Introduction states that the paper "is not about making a given edge LLM faster" and instead concerns selecting which LLM should become the NPU design target. That statement is not consistent with the rest of the paper, which is entirely about searching NPU architecture configurations for a fixed LLM specification and deployment setting.

What This Paper Is About

Designing an NPU for LLM inference requires searching a discrete space of architecture parameters — systolic-array count and dimension, vector-unit count, clock frequency, on-chip memory, HBM bandwidth, interconnect bandwidth — under power and memory constraints. The only practical way to evaluate a candidate is an expensive simulator that returns just a couple of scalars, such as latency and power. Those scalars can rank configurations you have already tried, but they say nothing about why a configuration is slow or which parameter to change next, and the problem is compounded because the prefill phase (measured by TTFT) and the decode phase (measured by TPOT) are often limited by different resources such as compute throughput and HBM bandwidth. The paper's goal is to extract more decision-relevant signal from each simulator call so the search reaches near-best designs with far fewer simulations.

Key Contributions

  1. Framing NPU DSE around mechanistic critics. The authors formulate sample-efficient design-space exploration as a critic-guided search problem in which simulator feedback is used not only to score configurations already evaluated, but also to decide which candidate configurations are worth simulating and which architecture parameters to prioritize next.

  2. Calibrating Roofline limits into predictive and diagnostic feedback. They instantiate the critic as a Roofline-style model that derives compute and memory limits from architecture parameters, then learns per-operator-type efficiency factors from simulator observations so that each operator class is compared against its own effective ridge point rather than one shared theoretical peak.

  3. Reducing simulator cost for LLM inference DSE. They build CriticNPU, which combines an LLM-based workload extractor (Compute Demand Agent), the calibrated critic for candidate filtering and bottleneck-guided feedback, and an LLM-based architecture planner (Architecture Design Agent).

  4. Empirical validation across model scales. Across six LLM workloads and 16 deployment configurations per model, CriticNPU obtains best-or-tied-best latency in 74% of prefill and 87.5% of decode settings while using roughly half the simulator calls of competitive baselines, together with ablations isolating each component and analyses of surrogate ranking quality.

Main Findings

  • Best-or-tied-best rate with far fewer simulations. CriticNPU wins 74.0% of prefill and 87.5% of decode configurations using only 54 simulator calls on average, and every one of its 96 runs stays under 100 calls — a property the paper says no other competitive method shares.

  • Baselines pay more for comparable quality. Classical methods (BO, CMA-ES, greedy search) reach comparable or higher win rates only by spending 2.2–2.4× more simulations. BO uses 117 calls, CMA-ES 130, greedy 129–130, and grid search 1440 across the reported models. LLM-DSE uses roughly 98–108 calls, LLM-Box 87–117, and ChatDSE reports no simulator-call count (marked as a backslash in Table 1) because it bypasses simulator feedback.

  • Latency close to the universal best. Averaged over all 96 cells, CriticNPU reaches 1.012× the best valid latency any method achieved on TTFT and 1.008× on TPOT — within 1.2% of optimum on average — compared with BO (1.006) and greedy (1.009), which spend 2.2–2.4× more calls, and CMA-ES (1.242), LLM-Box (1.198), LLM-DSE (1.528, p90 2.86), and ChatDSE (3.500).

  • A known exception: DSv3-671B decode. BO obtains 15/16 wins there while CriticNPU obtains 8/16. The authors attribute this to MoE routing, activated-expert imbalance, and cross-expert KV-cache/communication effects that the current operator-type calibration does not explicitly model, and treat it as a limitation.

  • The surrogate ranks well even when absolute values are off. On 54 proposals drawn from a 30-iteration run on Llama3-8B prefill and a 50-iteration run on Llama3.1-405B prefill, predicted versus simulated TTFT shows Kendall's τ = 0.36 and 0.41 respectively, with pooled Kendall's τ = 0.41 and Spearman's ρ = 0.54. The theoretical roofline systematically underestimates latency, but ordering — which is what gating needs — is positively correlated.

  • Gating prunes roughly half of proposals. Roofline-based gating skipped 24 and 46 simulator calls over the course of those two runs respectively.

  • The Roofline beats an LLM at its own quantitative tasks. Against GPT-5.4-mini — the same model CriticNPU uses as its Design Agent — given identical inputs on 150 Role-1 pairs and 83 Role-2 pairs, the roofline wins on every metric: 47% vs. 96% MAPE and 5% vs. 21% catastrophic errors on absolute prediction, τ = 0.62 vs. 0.52 on Role-1 ranking, and τ = 0.31 vs. 0.16 on Role-2 ΔTTFT ranking. The paper argues the LLM's apparently better MAPE comes from hedging by echoing the reference TTFT, which carries almost no information about which edit helps.

  • Calibration produces very different ceilings per operator type. On Llama3-8B prefill the factors converge to η_FFN = 0.54 and η_attention = 0.15, so FFN operators reach 54% of peak and attention operators only 15%. The authors explain the gap as cache misses and pipeline stalls for FFN versus softmax serialization for attention, and note the shifted ridge flips regime labels for boundary operators such as layer normalization and softmax.

  • Ablations show where the savings come from. On Llama3-8B, Llama3-70B, and Llama3.1-405B, removing the Roofline model inflates decode simulator calls from 15 to 48, 13 to 38, and 13 to 41 respectively, while leaving TPOT unchanged; removing the Compute Demand Agent slightly worsens TTFT on Llama3-70B (0.4248 vs. 0.4164) and Llama3.1-405B (2.0285 vs. 2.0015) and costs a few extra calls. Role 1 only and Role 2 only variants each recover most of the full method's quality, with the gating role responsible for most call savings.

Methodology in Plain English

The search loop has three moving parts. First, an LLM-based Compute Demand Agent reads the LLM specification (model dimensions, layer counts, tensor shapes) plus the deployment setting and emits an operator-level workload: for each operator, its FLOPs, its memory traffic including weights, activations and KV cache, how many times it executes per forward pass, and its arithmetic intensity (FLOPs divided by bytes).

Second, an Online-Calibrated Roofline Model acts as the critic. Using the candidate configuration, it computes a theoretical compute peak C(θ) = n_sa · d_sa² · 2f and a memory peak B(θ) = B_HBM, and forms a ridge point C/B that separates compute-bound from memory-bound operators. The critic is used twice per iteration. Before simulation, it predicts the candidate's latency by taking the measured per-operator runtimes from a recent reference configuration and rescaling each operator according to whether its arithmetic intensity places it above or below the ridge — divide by the change in compute peak if compute-bound, by the change in memory peak if memory-bound. If the prediction beats the current best (or the run is still in a warm-up budget), the candidate is simulated; otherwise it is skipped. After simulation, it converts raw per-operator runtimes into per-phase bottleneck shares — what fraction of prefill and of decode latency sits under each regime — and then reuses the same scaling rule to predict counterfactually what a proposed parameter edit would do (which operators speed up, whether the dominant regime flips, how TTFT and TPOT change, whether power is violated). These numbers are filled into fixed text templates by deterministic code, not generated by an LLM.

The refinement that makes this work is calibration. A single theoretical C/B ridge mislabels operators near the boundary, because no real workload reaches the full peak — a softmax dependency chain stalls the systolic array, layer normalization is dominated by cache misses. So the authors group operators into a small set of types (QKV projection, FFN GEMM, attention kernel, other matmuls, and non-matmul ops such as norm, activation, and all-reduce) and, after every simulation, estimate from the measured systolic-array time and memory time how much of each peak that type actually reached. These estimates are averaged within a type and smoothed with an exponential moving average. Each type then sees its own reduced ceilings and its own effective ridge, while the peaks themselves remain a hardware property and the learned factors transfer as the configuration changes.

Third, an LLM-based Architecture Design Agent proposes the next configuration from the workload, the current configuration, the bottleneck summary and counterfactual text, the constraints, and the search history. The planner proposes; the critic decides what is worth simulating and what to prioritize.

Why This Matters

  • It reframes what simulator feedback is for. Instead of treating the simulator as a scoring oracle, the paper treats it as a source of calibration data for a cheap physics-based model, which is a reusable pattern for any expensive-evaluation search problem.
  • It exploits a real structural property of LLM serving. Because prefill and decode stress different resources, a single scalar latency hides which resource to attack; the per-phase bottleneck shares make that explicit and are what allow prefill and decode chips to be optimized separately.
  • It quantifies when an LLM is the wrong tool. The head-to-head against GPT-5.4-mini on latency prediction and counterfactual ΔTTFT is a concrete, reproducible argument for splitting high-level design proposal from quantitative estimation.

Real-world applications:

  • Accelerator architecture exploration at chip design teams, where each simulator run is costly and the search space is combinatorially large.
  • Datacenter capacity planning for LLM serving, where the deployment setting (chip count, parallelism degree, power budget) is chosen jointly with the hardware configuration.
  • Hardware/software co-design studies for new LLM families, where the operator graph changes with every model release and the workload extractor removes hand-coding per model.
  • Cost-aware LLM serving deployment, since the framework optimizes TTFT for prefill chips and TPOT for decode chips separately under 300 W-class power constraints in the reported ablation settings.

Industry relevance: the results are all from an analytical simulator (NeuSIM), not silicon, so the immediate industrial value is in reducing exploration cost during early architecture definition rather than in validated hardware numbers — a point the authors state plainly in their limitations.

Future Directions

  • Model the effects the critic currently misses. The authors name MoE routing imbalance, cross-expert communication, and compiler scheduling artifacts as causes of the DSv3-671B decode shortfall; extending the critic to cover them is the obvious next step.
  • Validate beyond simulation. The evaluation is entirely in simulation with no physical implementation or tape-out, and the authors note that online calibration can reduce mismatch within a simulator but cannot correct systematic simulator error. Silicon or higher-fidelity validation would test whether the calibrated ridges transfer.
  • Replace hand-written templates with richer diagnosis. The critic currently writes its counterfactual outputs into fixed text templates with no LLM inside; whether a learned interface between critic and planner does better is unexplored.
  • Test whether the sample-efficiency claim holds for other search spaces. The paper's own framing says the approach is most effective when latency is governed by standard compute/HBM/ICI ceilings, which suggests probing other accelerator types and constraint regimes.

Target Audience

Researchers and practitioners in computer architecture and ML systems who work on accelerator design-space exploration, LLM inference infrastructure, or LLM-assisted hardware design. It is most useful to readers who already know what TTFT and TPOT measure and why prefill and decode behave differently, and to anyone evaluating whether an LLM agent should be trusted with quantitative estimation tasks versus high-level proposal generation.

Authors’ abstract

NPU design-space exploration for LLM inference mainly relies on an expensive simulator that often returns only scalar feedback such as latency and power. Such feedback can rank evaluated configurations, but gives little information for deciding why a configuration is slow or which architecture parameter should be changed next. This limitation is especially severe because prefill and decode workloads can be limited by different resources, including compute throughput and HBM…

The full abstract on arXiv
Read the original paper