Implementation Guides

Inference Economics: Quantization, Speculative Decoding, and the Art of Serving LLMs Cheaply

PagedAttention, continuous batching, KV-cache pricing, quantization, speculative decoding, MoE routing and batch tiers — and why the per-token floor keeps falling while a $10/$50 premium tier reopened above it.

Gabriele Masetti · 2026-09-19

Why Inference Now Eats the AI Budget

Training gets the headlines; serving gets the invoice. Industry estimates put inference at 55-80% of enterprise AI GPU spend, and analysts tracking cloud infrastructure billing say inference crossed 55% of AI cloud spend in early 2026, overtaking training for the first time. The pain shows up in earnings calls, not just benchmarks: Salesforce CEO Marc Benioff said on the All-In podcast that Salesforce is on track to spend $300 million on Anthropic tokens in 2026, and ServiceNow CEO Bill McDermott has talked publicly about customers getting "surprised on the tokenization of models" as usage-based bills hit their budgets.

Training is a one-time capital expense, amortized across however many requests a model ever serves. Every one of those requests is a separate inference call, and at scale the multiplication overwhelms the training bill — a single popular chat feature or agentic coding tool can generate more tokens in a month than the entire pretraining run consumed.

Inference is also latency-constrained in a way training isn't: a training job can wait a queue out, but a user staring at a spinner cannot, which forces over-provisioning for peak rather than average load. Every technique below attacks one of three levers: shrink the compute per token, shrink the memory per token, or shrink the number of tokens computed at all.

The Price Collapse, and the Tier That Reopened Above It

The clearest evidence that serving got dramatically more efficient is the sticker price. GPT-4 launched in 2023 at $30 per million input tokens and $60 per million output tokens. GPT-5.1 still lists at $1.25/$10 per million input/output tokens — a 24x drop on input and a 6x drop on output, for a model that benchmarks well ahead of the 2023 original.

GPT-5.1 is no longer the top of OpenAI's menu, and the models above it are not cheaper. The GPT-5.6 family lists Sol, the flagship, at $4/$20 per million tokens for short context and $8/$30 beyond it; Terra at $2/$12; and Luna, the small tier, at $0.20/$1.20. Luna is where the collapse continued — 150x below GPT-4's input price — while Sol costs more per token than GPT-5.1 does.

Anthropic's pricing tells the same two-sided story. Opus 5 and Opus 4.8 both list at $5/$25 per million tokens. Claude Sonnet 5 is $2/$10, and what launched as introductory pricing is now standard: Anthropic states the $3/$15 increase scheduled for 1 September 2026 will not occur. Sonnet 5 still undercuts GPT-4's original output price by 6x while running a stronger model.

Above all of them sits a tier that did not exist when this curve was first drawn: Claude Fable 5.1 and Mythos 5.1 at $10 input and $50 output per million tokens, with fast mode for Opus 5 and Opus 4.8 priced the same. The floor keeps falling and the ceiling has been rebuilt higher. Budgeting on "prices only go down" budgets for the tier you used last year, not the one your product manager wants this year.

Per-token comparisons across generations are also softer than they look. Anthropic notes that Claude 4.7 and later models use a newer tokenizer producing roughly 30 percent more tokens for the same text. A price cut per token is not a price cut per task, and the only number worth trusting is your own invoice.

None of this is charity: it is the compounding effect of better silicon (H100 and now Blackwell), smarter serving software, and quantization formats that didn't exist at GPT-4 launch — the engineering behind the rest of this piece, and a curve you can apply yourself if you self-host.

The Serving Stack: vLLM, SGLang, and TensorRT-LLM

vLLM, from UC Berkeley (SOSP 2023 paper), is the default open-source serving engine for a reason: PagedAttention. Classic KV cache allocation reserves a contiguous buffer sized for the maximum sequence length per request, wasting GPU memory to fragmentation, since most requests never reach the max but the memory is reserved anyway. PagedAttention borrows the OS trick of paging: it splits the KV cache into small fixed-size, non-contiguous blocks allocated on demand, the same way virtual memory maps physical RAM. Reported memory waste reduction is up to 90%.

The second load-bearing idea is continuous batching (in-flight or iteration-level batching). Naive batching processes a fixed batch through decode until every sequence finishes, so a batch of 32 requests runs at the pace of its slowest sequence, with GPU slots idle for every request that finished early. Continuous batching schedules at the level of a single decode step instead: at every forward pass the scheduler evicts finished sequences and admits new ones from the queue. The original vLLM paper measured 10-23x throughput improvement over static batching; PagedAttention plus continuous batching together are typically cited as a 2-4x gain over naive serving.

SGLang targets repeated prefixes instead (shared system prompts, few-shot examples, multi-turn history). Its RadixAttention structure stores KV cache entries in a radix tree so a new request automatically reuses the cache of the longest matching prefix already computed. SGLang's own benchmarks report up to 5x throughput over vLLM and Guidance on prefix-heavy workloads; recent H100 numbers with Llama-3.1-8B show SGLang at roughly 16,200 tokens/sec against vLLM's 12,550, about a 29% edge — though below roughly 60% shared-prefix overlap, the advantage shrinks toward parity.

TensorRT-LLM, Nvidia's own engine built on the TensorRT compiler, implements in-flight batching, paged KV cache, quantization, and speculative decoding natively through a C++ batch manager, plus chunked prefill to interleave long-prompt processing with ongoing decode. It needs more setup than vLLM or SGLang (model-specific engine builds) but extracts the most from Nvidia hardware, and is the usual choice once a fixed model and GPU generation are locked in for production.

KV Cache: The Memory Bill Nobody Budgets For

The KV cache stores the key and value projections for every token already processed, so attention doesn't recompute them on every new token — what makes autoregressive decoding linear instead of quadratic in generated length. It's stored per-sequence, per-layer, per-attention-head, and grows with every token generated or ingested.

Worked example (assumptions labeled). Take a 70B-class dense model with 80 layers and hidden size 8,192, in FP16 (2 bytes/value), no grouped-query attention (worst case). Per-token KV cache size is 2 (K and V) × 80 layers × 8,192 hidden × 2 bytes ≈ 2.62 MB/token. Production 70B-class models typically use grouped-query attention at roughly an 8:1 query-to-KV-head ratio, dividing that to about 330 KB/token.

For a single 32K-token context that's still 330 KB × 32,768 ≈ 10.3 GB of HBM for one sequence — before a second concurrent user shows up. On an 80 GB H100, that one long-context request already claims an eighth of total memory, which is why paged, block-level allocation beats reserving a worst-case contiguous buffer per request.

Prefix caching is the economic answer to the same problem: if the first N tokens of a request match a previous request (a system prompt, a RAG document, a tool schema), read the cached KV entries back instead of recomputing them. Anthropic prices a cache read at 0.1x the standard input rate — a 90% discount — while writing costs 1.25x for a 5-minute TTL or 2x for a 1-hour TTL; the discount stacks with the Batch API's 50% cut, so a cached batch request can cost as little as 5% of an uncached standard-rate request. On the Fable 5.1 and Mythos 5.1 tier the cache read is 0.025x rather than 0.1x — $0.25 per million tokens against a $10 base — taking the same stacked floor to about 1.25%.

OpenAI runs the equivalent play: prompts over 1,024 tokens cache automatically, with hits discounted 90% off standard input pricing. If a workload has any repeated structure — most agentic and RAG systems do — prefix caching is close to a free win, provided stable content comes before variable content, since anything before a cache breakpoint that changes invalidates everything after it.

Quantization: FP16 to FP8 to FP4/INT4

Quantization cuts precision on weights (and sometimes activations) to shrink memory footprint and raise arithmetic throughput, since lower-precision math runs faster on hardware built for it. Each step needs new hardware and calibration technique, not just a smaller number format.

FP8 is native to Hopper (H100/H200): fourth-generation Tensor Cores and the Transformer Engine run FP8 matrix math directly, delivering roughly 2x FP16's application throughput on the same silicon while halving memory footprint, automatically mixing FP8 and FP16 per operation to control accuracy loss. Quality loss versus FP16 is typically minimal for inference — close to "free" precision — which is why FP8 is now near-default for large-model serving on H100-class fleets.

INT4/FP4 is a bigger cut needing more careful calibration. GPTQ (2022) is a one-shot post-training method that compresses weights to 3-4 bits while keeping activations at FP16, using second-order error correction to limit degradation. AWQ instead identifies which weight channels matter most by activation magnitude and protects those preferentially; independent evaluations report AWQ holding accuracy more consistently than GPTQ at 4-bit across Llama-family sizes, needing less calibration data. Neither is free: 4-bit quantization measurably narrows quality on some reasoning and long-tail tasks, so production deployments benchmark on their own eval set rather than trust an aggregate perplexity number from a paper.

FP4 is the newest step and it's hardware-gated: Nvidia's NVFP4 format is native only to Blackwell silicon (B200, B300, RTX 5090, RTX PRO 6000) — small 16-value microscaling blocks with FP8 (e4m3) block-scale factors, making 4-bit floating point numerically viable rather than just fast. Nvidia's own Llama-2-70B benchmarks show Blackwell delivering up to 4x the tokens/sec per GPU of H100, and Fireworks reports its FireAttention V4 stack clearing 250+ tokens/sec per user on B200 — but only once a fleet has actually turned over from Hopper.

Speculative Decoding, Distillation, and Small Models

Speculative decoding attacks a different bottleneck: autoregressive decoding is memory-bandwidth bound, generating one token per forward pass regardless of leftover compute headroom. A small, cheap draft model proposes several tokens ahead; the large target model verifies them all in a single batched forward pass, accepting the ones matching what it would have generated anyway — buying a multi-token speedup whenever the draft's guesses are good.

Medusa and EAGLE are the two dominant approaches, both training extra prediction heads on the target model itself rather than a separate draft model. SpecBench benchmarks report both landing around a 2.4x overall speedup on a single A100, rising to roughly 2.8x on multi-turn conversation and math reasoning — the more predictable the continuation, the bigger the win. EAGLE-3 reports 2-6x depending on model size and batch configuration, larger models (70B+) trending toward 4-6x as draft-head overhead becomes negligible against memory-bandwidth savings; production H100 benchmarks put it 15-25% ahead of EAGLE-2 and Medusa-2 in tokens/sec. DeepSeek-V3 trains multi-token prediction directly into the base model, making speculative decoding a built-in capability rather than an add-on.

Distillation and small models attack cost from the other direction: don't serve a big model for tasks a small one handles well enough. Llama 3.2 ships 1B and 3B text-only variants for edge and low-cost serving; Microsoft's Phi-3.5-Mini is a 3.8B-parameter model tuned for reasoning and code; Google's Gemma 3 comes in a 4B multilingual, multimodal size.

Google has published that its Gemma 2 2B and 9B models were trained via knowledge distillation from a larger teacher, measurably improving quality over training the same size from scratch. A distilled 8B model that clears a task's quality bar can run 5-10x cheaper per token than a 70B+ model, before quantization or speculative decoding are even applied.

MoE Serving and the Batch/Offline Tier

Mixture-of-experts architectures decouple parameter count from compute cost per token. DeepSeek-V3 is the clearest public example: 671B total parameters, only 37B active per token — about 5.5% of the model does the work on any forward pass, with 256 experts per layer and 8 selected per token via top-k routing, plus multi-head latent attention (MLA) to shrink the KV cache footprint that would otherwise offset the savings.

Per DeepSeek's technical report, the result is frontier-class quality at roughly a tenth of the training cost of a comparably capable dense model, and the same sparsity saves inference compute too, since only the active experts' FLOPs run per token even though HBM must hold every expert. That's the MoE tradeoff in one line: trade GPU memory (host all the experts) for GPU compute (run only a fraction), which is why MoE models want high-memory GPUs and large batch sizes.

Batch/offline tiers are the cheapest lever for anything not latency-sensitive. Both Anthropic and OpenAI run asynchronous batch endpoints at a flat 50% discount on input and output tokens, results typically returned well inside a 24-hour window. For evals, bulk classification, embeddings backfills, or overnight document processing, batch mode is close to a free 2x — and Anthropic's batch discount stacks multiplicatively with prompt caching, so a cached, batched request can fall to roughly 5% of the standard-rate, uncached cost.

Build vs Buy: When Self-Hosting Actually Wins

H100 rental has no single market rate, and the trackers disagree by more than a factor of two. One survey updated in August 2026 puts the band at $1.49/hour (a Vast.ai promotional rate) to $6.98/hour on Azure, a mid-range of $2.89-$3.90 and no published average, because providers package H100s in different node sizes, regions and tiers. Another, reading September 2026, quotes $2.89-$11.06/hour with a specialist-cloud median near $4.17 against $7.89 on hyperscalers. The spread is the finding: build-vs-buy moves with which provider will actually sell you capacity, and self-hosting only wins if those GPU-hours stay busy.

Worked example (assumptions labeled). Assume a self-hosted 70B-class model in FP8 on 2x H100 SXM at $3.20/hour each ($6.40/hour total), a published specialist-cloud rate, served with continuous batching via vLLM, sustaining — as an illustrative, non-benchmarked assumption — an aggregate 2,000 output tokens/sec at full utilization. That's 7.2M tokens/hour, roughly $0.89/Mtok blended, on GPU rental alone (ignoring engineering overhead).

Compare that to a 3:1 input:output-weighted blend on Claude Sonnet 5's standard pricing: (3×$2 + 1×$10) / 4 = $4/Mtok. Under those assumptions self-hosting looks about 4-5x cheaper — but only near 100% utilization. Drop to 30% utilization (realistic for bursty traffic) and the effective cost rises to roughly $2.96/Mtok, erasing most of the advantage before counting the team needed to run vLLM in production and handle GPU failures. The comparison is also not token-for-token, since the hosted model's newer tokenizer bills more tokens for the same text.

The rule of thumb: self-hosting wins with sustained, high-volume traffic that keeps GPUs near saturation — internal batch pipelines, high-QPS features, or fine-tuned models no API provider serves. APIs win when traffic is bursty, frontier quality is required that no open model matches, or volume is too low to keep a fleet busy. Most teams underestimate the last condition and pay for idle GPU-hours that would have been cheaper as API calls with caching on.

Most large clouds are still mid-migration from Hopper to Blackwell in September 2026. B200 capacity is quoted at $3.50-$27.04/hour where it is quoted at all, NVIDIA's allocation is spoken for into 2027 and CoWoS packaging is sold out, so Blackwell is effectively reserved for hyperscalers while Hopper stays well supplied. The H100 fleet running today's FP8 economics is not the fleet that will run tomorrow's FP4 ones: for most teams FP4 is still a plan, not a price.