Implementation Guides

Deploying LLMs in Production: A Complete Guide

Everything you need to know about serving large language models at scale. Covers model serving frameworks, quantization, batching, scaling, and cost optimization.

Gabriele Masetti · 2026-08-10

Choosing a serving framework

Four frameworks cover nearly every production LLM deployment today. Pick based on hardware constraints and traffic shape, not benchmarks alone — the gaps between them are real but usually smaller than the gap between a tuned and untuned deployment of any single one.

• vLLM is the default choice for most teams. It introduced PagedAttention (from UC Berkeley's Sky Computing Lab, published at SOSP 2023), which manages the KV cache like OS virtual memory — paging it into fixed-size blocks (16 tokens each) so sequences don't need contiguous memory and can share pages when they share a prefix. Combined with continuous batching, this gets vLLM 2–5x the throughput of naive static batching. It supports the widest range of hardware (NVIDIA, AMD, some inference chips) and is the safest general-purpose pick.
• SGLang uses RadixAttention, a prefix-caching scheme that automatically reuses KV cache across requests sharing a prompt prefix. For workloads with heavy prefix reuse — chat applications with long system prompts, RAG pipelines, multi-turn agents — SGLang has been benchmarked beating vLLM by roughly 29% on throughput. It also tends to win on single-request latency.

• TensorRT-LLM is NVIDIA's compiled-kernel runtime. On H100/H200 it can run 8–13% faster than vLLM depending on concurrency, because it compiles model-specific CUDA kernels ahead of time. The cost is inflexibility: expect 1–2 weeks of setup and engine-building per model/GPU combination, and you're locked to NVIDIA hardware.
• TGI (Hugging Face's Text Generation Inference) is finished. It went into maintenance mode and the repository was archived read-only on 21 March 2026 — no fixes, no features, no security patches. Its own archive notice points to vLLM and SGLang, and to llama.cpp or MLX for local engines. Don't start a new project on it, and plan a migration if you are still on it.

Practical rule: default to vLLM. Move to SGLang if your traffic is dominated by shared-prefix requests (support bots, agent frameworks, RAG with a fixed system prompt). Move to TensorRT-LLM only if you have a stable, standardized NVIDIA fleet, latency is the primary KPI, and you have engineering time to maintain compiled engines across model updates.

Quantization

Quantization is usually the highest-leverage lever for both cost and latency, and the choice depends on whether you're GPU-memory-bound or throughput-bound.

• AWQ (Activation-aware Weight Quantization) protects the small number of weight channels that matter most for activations, instead of quantizing everything uniformly. In published comparisons at 4-bit, AWQ holds accuracy substantially better than GPTQ at the same bit width, and its TinyChat runtime shows roughly 2.7–2.9x speedup over FP16 on consumer and edge GPUs. Use AWQ when you need 4-bit and can't tolerate the accuracy hit of naive quantization.

• GPTQ is a layer-by-layer post-training quantization method optimized for GPU throughput. It remains a solid choice for GPU-centric production inference where you're optimizing purely for tokens/sec at INT4 and have validated accuracy on your own eval set — GPTQ and AWQ are close enough that framework support and tooling maturity often decide the tiebreak.
• FP8 is the native format on Hopper (H100/H200) and Blackwell (B200) tensor cores. It delivers up to 25–30% higher throughput than FP16 at large batch sizes, with much smaller quality loss than INT4 schemes because FP8's dynamic range absorbs outliers better. FP8 KV cache (quantizing the cache itself, not just weights) can hold within ~0.01 perplexity of a full-precision baseline in recent benchmarks. If your GPUs support native FP8 (Hopper/Blackwell), it's frequently the best default: less accuracy risk than INT4, most of the throughput win.
• GGUF (llama.cpp's format, Q3_K_M through Q8_0) is the right choice for CPU inference, edge deployment, or single-GPU hobbyist-scale serving — not typically what you want for a multi-GPU production serving cluster.

Default recommendation for a production cluster on Hopper or newer: start with FP8 weights + FP8 KV cache. Drop to AWQ INT4 only if you're memory-constrained and can validate the accuracy trade-off on your own tasks — recall AWQ vs GPTQ vs FP8 all need per-model, per-task validation before you commit; published perplexity numbers on generic benchmarks don't guarantee your task tolerates the same loss.

Batching, KV cache, and speculative decoding

Continuous batching replaces the old model of "wait for a full batch, run it, wait for the next batch" with a scheduler that adds new requests to the running batch as soon as a GPU slot frees up (when another sequence finishes), and evicts finished sequences immediately. This alone is responsible for most of the throughput gain modern servers have over 2022-era batching.

PagedAttention (vLLM) and its equivalents in other engines solve the memory-fragmentation problem continuous batching creates: variable-length sequences waste GPU memory if the KV cache must be contiguous. Paging the cache into fixed-size blocks — and letting multiple sequences share pages when they share a prefix — cuts VRAM usage by more than 50% in long-context, high-prefix-overlap scenarios (chat with a shared system prompt, few-shot prompts, agent scratchpads).

Chunked prefill splits the prompt-processing (prefill) phase into chunks interleaved with decode steps from other requests, instead of letting a long prompt block the GPU for one request at a time. This matters when your workload mixes short and long prompts — without it, a 20K-token prompt request can spike latency for every other concurrent request. Recommended tuning order for a new vLLM deployment: enable continuous batching (on by default), raise gpu_memory_utilization toward 0.90–0.95 to give PagedAttention a bigger pool, then turn on chunked prefill if you see prefill-induced latency spikes.

Speculative decoding is standard in every serious inference engine now (vLLM, SGLang, TensorRT-LLM, LMDeploy). A small draft model proposes several tokens ahead; the full model verifies them in one forward pass, accepting the ones that match what it would have generated anyway. With acceptance rates commonly above 80% on structured or predictable text, this delivers 2–4x speedups in production, and specialized draft methods (EAGLE-3 and similar) push coding-heavy workloads to 3.5–5x.

It's less effective on high-entropy, creative-writing-style output, where the draft model's guesses are rejected more often and the overhead isn't worth it — reserve it for coding assistants, structured extraction, and other constrained-output workloads first.

A minimal vLLM launch reflecting these choices — with the weights left as a variable, because that is the part of the stack that turns over fastest:

MODEL=<your-instruct-checkpoint>     # current open-weight family, see below
DRAFT=<small-checkpoint-same-family> # ~1B, for speculative decoding

vllm serve "$MODEL" \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --enable-chunked-prefill \
  --max-num-seqs 256 \
  --tensor-parallel-size 4 \
  --speculative-config "{\"model\": \"$DRAFT\", \"num_speculative_tokens\": 5}"

The open-weight field turned over twice between 2025 and 2026. A self-hosting leaderboard updated on 20 July 2026 ranks DeepSeek-V4-Pro (1.6T parameters, MIT), Kimi K2.6 (1T, modified MIT) and GLM-5.2 (753B, MIT) at the top, with DeepSeek-V4-Flash (284B, MIT) and Hunyuan Hy3 (295B, Apache 2.0) below them and Llama 3.3 70B several tiers down. Most of the leaders are sparse mixture-of-experts models whose memory footprint bears little relation to the headline parameter count, so size against the specific checkpoint rather than a rule of thumb.

Hardware sizing

GPU choice and count follow directly from model size, quantization, and target concurrency.

Rates vary heavily by provider and commitment level — the same H100 SKU was tracked at $1.25/hr at the cheapest in-stock provider and $6.98/hr at Azure on the day these figures were read. Always get current quotes before budgeting; the table above is a starting point, not a quote.

Sizing a 70B-class dense model in FP8: weights alone are roughly 70 GB. That doesn't fit comfortably on a single 80 GB H100 once you add KV cache and activation memory, so tensor-parallel across 2–4 GPUs is standard — 4x H100 (320 GB pooled) gives comfortable headroom for KV cache at long context and high concurrency; 2x H200 (282 GB pooled) is a leaner alternative given the extra memory per card.

For a 8B-class model, FP8 fits on a single H100/H200 with room for large batches. Always size the KV cache explicitly: cache size per token scales with 2 × num_layers × num_kv_heads × head_dim × dtype_bytes, multiplied by your target concurrent-sequence count and max context length — this is frequently the actual capacity bottleneck, not the weights.

Latency budget

Two metrics dominate user-facing latency SLOs:

• TTFT (time to first token) — driven by prefill time and queueing. A common production target is p99 TTFT under 1 second for interactive chat.
• TPOT (time per output token), aka inter-token latency — driven by decode-phase throughput per sequence. A common target is p99 TPOT under 100ms (roughly 10 tokens/sec perceived streaming speed, though 20–40+ tokens/sec reads as fluent).

These trade off against batch size and concurrency: larger batches raise aggregate throughput but can raise per-request TPOT if the GPU is saturated, and a long prefill from one large request can spike TTFT for everyone else queued behind it (the reason to enable chunked prefill). Load-test at your expected concurrent-request distribution — not a synthetic single-request benchmark — before setting SLOs; vLLM and SGLang both expose per-request TTFT/TPOT histograms so you can see the actual p50/p95/p99 shape under realistic load rather than assuming it from throughput numbers alone.

Scaling and reliability

Don't autoscale LLM inference pods on CPU or raw GPU utilization. CPU is nearly irrelevant to a GPU-bound workload, and GPU utilization is a lagging, misleading signal for engines like vLLM — PagedAttention keeps GPU utilization high by design, so a pod at 85% utilization might be serving 5 requests or 500 with no visible difference in the utilization metric. By the time utilization looks saturated, TTFT p99 has often already blown through its SLO because queueing built up during a burst that the utilization metric smoothed over.

The correct primary signal is queue depth — vLLM exposes it as vllm:num_requests_waiting. Scale out when this exceeds a threshold per replica; scale in when it's near zero. Pair it with GPU-level metrics from DCGM Exporter (dcgm_fi_dev_gpu_util, memory utilization/bandwidth) as a secondary signal, targeting 60–80% sustained compute utilization as healthy headroom rather than a scaling trigger.

KEDA (Kubernetes Event-Driven Autoscaling) is the standard mechanism for this — the default Horizontal Pod Autoscaler reacts too slowly for bursty inference traffic, and KEDA supports scale-to-zero for cost-sensitive or spiky workloads. A minimal KEDA ScaledObject driving off vLLM's own Prometheus metric:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-inference-scaler
spec:
  scaleTargetRef:
    name: vllm-deployment
  minReplicaCount: 1
  maxReplicaCount: 10
  cooldownPeriod: 120
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: vllm_num_requests_waiting
        query: sum(vllm:num_requests_waiting)
        threshold: "10"

For reliability beyond autoscaling: run health checks that actually exercise inference (a cheap real generation, not just a port check), keep at least one warm replica per model to avoid cold-start latency spikes, and separate GPU pools by model/quantization so a bad deploy on one model doesn't starve capacity from another.

Observability

vLLM and most serving engines expose a Prometheus-compatible /metrics endpoint out of the box. The metrics that matter for an SRE dashboard:

• vllm:time_to_first_token_seconds (histogram) — TTFT, track p50/p95/p99
• vllm:time_per_output_token_seconds / inter-token latency (histogram) — TPOT
• vllm:num_requests_waiting and vllm:num_requests_running (gauges) — queue depth and active load, your autoscaling and capacity-planning signal
• dcgm_fi_dev_gpu_util, GPU memory used/total (from DCGM Exporter) — hardware saturation, secondary to queue depth
• Request/response token counts and error rates — cost attribution and reliability

Wire these into Grafana (vLLM ships a reference dashboard) rather than building custom dashboards from scratch, and alert on TTFT/TPOT p99 breaching your SLO threshold, not on GPU utilization alone — by the time utilization alerts fire, users have often already felt the latency.

Cost control: self-host vs. API

The decision between self-hosting and using a hosted API is primarily a volume and utilization question, and it shifts as both API prices and GPU efficiency improve.

Hosted API list pricing per million tokens (input/output), read off each provider's own pricing page on 19 September 2026:

(Several of these carry conditions the table can't hold: Gemini 3.8 Flash's rate is promotional until the end of 2026, Gemini charges more above a 200K-token prompt, and DeepSeek halves its rates off-peak, so the low end of its range is the off-peak price. Cached input is cheaper than the figures above on every one of these platforms. Confirm directly with the provider before a procurement decision.)

Self-hosting a 70B-class model on a small GPU cluster (2–4 H100/H200-class GPUs) typically runs $3,000–5,000/month in all-in infrastructure cost. Whether that beats API pricing depends entirely on sustained volume and utilization — published break-even estimates for a self-hosted 70B model vary widely by source, generally landing somewhere in the range of several billion to over ten billion tokens per month at healthy (70%+) GPU utilization sustained continuously.

Below that volume, or with bursty/low-utilization traffic, hosted APIs are almost always cheaper once you account for the fixed cost of idle GPU capacity, on-call engineering time, and the opportunity cost of not using a frontier model's latest capabilities. The math only clearly favors self-hosting at high sustained volume (tens to hundreds of millions of tokens/day) or when data residency, latency-to-data-locality, or fine-tuning requirements make a hosted API infeasible regardless of cost.

Practical cost-control levers, roughly in order of impact for a self-hosted deployment:

1. Quantization (FP8 or AWQ) — cuts both memory footprint and cost per token before you touch anything else.
2. Right-size concurrency and batch limits — an under-utilized GPU is pure waste; tune max-num-seqs and gpu-memory-utilization against your real traffic pattern, not a benchmark default.
3. Autoscale on queue depth with scale-to-zero for low-traffic environments — don't pay for idle replicas overnight or on weekends if traffic supports it.
4. Speculative decoding for latency-sensitive, structured-output workloads — improves throughput per GPU without adding hardware.
5. Route by task difficulty — use a smaller/cheaper model or a lower-cost API tier for simple classification and extraction, and reserve the large self-hosted or frontier-API model for tasks that actually need it.

Re-evaluate the self-host/API decision periodically rather than once — both hosted API pricing and open-weight model quality shift fast enough that a decision made a year ago is worth re-checking against current numbers.