Implementation Guides

GPU Infrastructure Management for ML Teams

How to plan, provision and manage GPU compute for machine learning: workload tiering across Hopper, Blackwell and Rubin, Slurm versus Kubernetes, MIG, spot economics and DCGM monitoring.

Gabriele Masetti · 2026-08-12

Right-Sizing the Fleet Before You Buy or Rent Anything

The single most expensive planning mistake in GPU infrastructure is picking hardware based on what's hyped rather than what the workload needs. Start by separating your workloads into three buckets — large-scale pretraining, fine-tuning, and inference — because each has a different bottleneck.

Pretraining and large distributed training are bandwidth- and interconnect-bound. You're moving activations and gradients across hundreds of GPUs, so NVLink domain size and HBM bandwidth matter more than raw FLOPs. That is where NVIDIA's SXM-form-factor parts belong: H100 SXM5 (80GB HBM3, ~3.35 TB/s bandwidth), H200 (141GB HBM3e, ~4.8 TB/s — 76% more memory and 43% more bandwidth than H100), and Blackwell-generation B200 (192GB HBM3e, ~8 TB/s, roughly 2.3x the compute and 1.7x the bandwidth of H200).

At the rack level, GB200 NVL72 pairs Grace CPUs with Blackwell GPUs into a 72-GPU NVLink domain, and GB300 (Blackwell Ultra) uses the same fabric with more memory per GPU. NVIDIA quotes a GB300 NVL72 rack at 20TB of GPU memory and up to 576 TB/s of bandwidth across its 72 Blackwell Ultra GPUs and 36 Grace CPUs. Both are line items on public price sheets now rather than roadmap entries — CoreWeave lists GB200 NVL72, GB300 NVL72 and HGX B300 alongside H100 and A100.

Vera Rubin arrived on schedule. Each Rubin package carries 288GB of HBM4 at up to 22 TB/s, built on TSMC's 3nm process and rated at roughly 50 petaflops of NVFP4 inference; Rubin Ultra, two Rubin dies in one package, is a 2027 part. The first benchmarked numbers came on 16 September 2026, when MLPerf Inference v6.1 published a Vera Rubin NVL72 preview submission: NVIDIA reported up to 3.7x the throughput of GB300 NVL72 on Qwen3-VL and up to 2.5x on DeepSeek-R1. Preview is not the verified available category, and the baseline is NVIDIA's own, so read those ratios as ceilings. The planning consequence is blunter than the ratios anyway — a multi-year H100 reservation signed now will spend its back half competing with silicon two generations newer.

The B300 row is derived from NVIDIA's rack-level figures — 20TB and 576 TB/s divided across 72 GPUs — because NVIDIA publishes GB300 specifications per NVL72 rack rather than per GPU.

Fine-tuning (LoRA, QLoRA, full fine-tunes of models under ~70B parameters) rarely needs the top-bin part. A single A100 80GB or H100 PCIe often has enough memory headroom, and you avoid paying an NVLink premium you won't use for single- or dual-GPU jobs.

Inference is a different optimization problem entirely: it's about cost per token and throughput per dollar, not peak FLOPs. NVIDIA's L40S (Ada Lovelace, 48GB GDDR6) is purpose-built for this tier — it handles chatbots, RAG pipelines, and mid-size model serving in standard PCIe servers at a fraction of H100 pricing, and the H100 NVL variant (two GPUs bridged via NVLink, 94GB HBM3 per GPU, 188GB combined) exists specifically to give inference workloads a big memory pool without full SXM infrastructure.

A practical rule for the planning phase: don't buy or reserve SXM-class hardware for anything that runs comfortably on a single GPU. The premium for NVLink and HBM3e is only justified when your job actually spans GPUs.

Scheduling: Kubernetes, Slurm, or Both

Most ML platform teams end up running both, because they solve different problems.

Kubernetes + the NVIDIA device plugin

The NVIDIA device plugin for Kubernetes exposes each node's GPUs as the schedulable resource nvidia.com/gpu, letting the standard Kubernetes scheduler place pods on nodes with available GPU capacity. On its own, the device plugin treats a GPU as an indivisible unit — one pod, one whole GPU. That's fine for training jobs but wasteful for inference services that only need a slice of a GPU's compute or memory.

The NVIDIA GPU Operator extends this with two sharing mechanisms, configured through the device plugin's ConfigMap:

version: v1
sharing:
  timeSlicing:
    resources:
    - name: nvidia.com/gpu
      replicas: 4

Time-slicing lets multiple pods interleave on the same physical GPU by advertising it as multiple schedulable replicas — but there's no memory or fault isolation between replicas, so one runaway container can OOM its neighbors. Use it for latency-tolerant, bursty inference workloads where hard isolation isn't required.

Kubernetes is the right fit for inference serving, notebook environments, and any workload that benefits from autoscaling, rolling updates, and service discovery — things Slurm doesn't do natively.

Slurm for batch training

Slurm remains the default for large, long-running training jobs, especially in dedicated GPU clusters and HPC-style environments. GPU allocation is handled through GRES (Generic Resources): you define resources explicitly in slurm.conf and gres.conf, using type-qualified names rather than a generic GPU label:

# slurm.conf
NodeName=gpu-node[01-08] Gres=gpu:h100:8 CPUs=96 RealMemory=1000000

# gres.conf
Name=gpu Type=h100 File=/dev/nvidia[0-7]

A job then requests GPUs with --gres=gpu:h100:8 (or a mixed-fleet cluster might use --gres=gpu:a100:4). Slurm also supports MIG-partitioned GPUs as schedulable GRES types, which matters if you're running a shared research cluster where some jobs need a full H100 and others only need a 1g.10gb slice.

Slurm's strengths — topology-aware placement, backfill scheduling, fair-share quotas across research teams, checkpoint/requeue semantics — are exactly what multi-week pretraining runs need. Its weakness is that it's a poor fit for always-on services, which is why most orgs run Slurm for training and Kubernetes for serving, with a shared node pool sometimes managed by both via partitioning.

NVIDIA Run:ai as a third option

Since NVIDIA's 2024 acquisition, Run:ai sits on top of Kubernetes and adds capabilities neither raw K8s nor Slurm provide out of the box: dynamic GPU fractions (allocating GPUs in sub-integer units like 0.5, with the scheduler adjusting allocation based on live demand rather than a static time-slice), hierarchical quota management across teams and projects, and workload-aware bin-packing.

NVIDIA and Nebius benchmarks reported up to 2x greater user capacity on the same hardware during peak periods versus static allocation. In April 2025, NVIDIA open-sourced the scheduler core that originated inside Run:ai as the KAI Scheduler under Apache 2.0 — worth evaluating if you want fractional-GPU scheduling without the commercial licence. NVIDIA's published list price for AI Enterprise on self-managed systems is $4,500 per GPU for one year, $18,000 per GPU for five, or $1 per GPU-hour through cloud marketplaces on top of the instance cost.

Getting Utilization Out of Expensive Silicon

GPU utilization in most ML orgs sits well below what the hardware is capable of, largely because jobs are sized to "a GPU" rather than to actual memory and compute needs. Two mechanisms address this at the hardware level.

MIG (Multi-Instance GPU)

MIG physically partitions a supported GPU (A100, H100, and newer) into up to seven isolated instances, each with dedicated memory, L2 cache, and compute cores — true hardware-level isolation, not just scheduling-level sharing. On an 80GB A100 or H100, common profiles range from 1g.10gb (one compute slice, 10GB memory) up to 7g.80gb (the whole GPU).

A frequent production pattern is carving a single H100 into seven 1g.10gb instances for small-model multi-tenant inference, or two 3g.40gb instances to run two independent 13B-class model servers on one physical GPU. MIG has been integrated with the Kubernetes device plugin since 2020, so partitioned instances show up as distinct schedulable resources.

The tradeoff versus time-slicing is flexibility for isolation: MIG gives you predictable, guaranteed performance per tenant, but the partition layout is fixed until you reconfigure the GPU (which requires draining running work). Time-slicing gives you finer-grained, dynamic sharing but no isolation guarantees. For latency-sensitive multi-tenant inference where noisy neighbors are unacceptable, MIG is the safer default; for internal dev/test workloads where occasional contention is tolerable, time-slicing is simpler to operate.

Sizing for the workload, not the GPU

Before reaching for MIG or fractional scheduling, check whether the job actually needs a full GPU's memory. A 7B parameter model in fp16 needs roughly 14GB for weights plus activation memory — it doesn't need an 80GB H100 to itself. Right-sizing at the job-definition level (matching model size to MIG profile or GPU tier) typically yields bigger utilization gains than any scheduler tuning.

Spot vs. Reserved vs. On-Demand

Cloud GPU capacity comes in three flavors with materially different pricing and risk profiles, and planning should map workload tolerance to capacity type deliberately.

• On-demand is your ceiling price and safety net — AWS P5 (H100) on-demand runs around $6.88/GPU-hour ($55.04/hour for a p5.48xlarge with 8 H100s). Use it for anything you can't afford to have interrupted mid-run.
• Reserved / Capacity Blocks (AWS EC2 Capacity Blocks for ML, similar constructs elsewhere) require advance purchase for a fixed duration and guarantee availability without interruption — useful for a planned pretraining run where you know start date and duration but want price certainty.
• Spot cuts cost by roughly 60–70% versus on-demand on AWS P5, but comes with interruption risk that varies significantly by GPU type — H100 spot interruption rates on AWS run under 5%, while A100 spot interruption is meaningfully higher at 15–20%, reflecting differing supply/demand balance.

Neocloud providers add a fourth axis worth comparing against hyperscaler pricing. CoreWeave's North America list price for an 8-GPU HGX H100 node is $49.24 per hour — about $6.16 per GPU-hour — with HGX H200 at $50.44 per node and an 8-GPU A100 node at $21.60, roughly $2.70 per GPU-hour. The Blackwell tiers are quoted the same way: HGX B200 at $68.80 per node, about $8.60 per GPU-hour, GB200 NVL72 at $42.00, and GB300 NVL72 and HGX B300 on request. Neocloud rates often sit meaningfully below hyperscaler on-demand for the same silicon, at the cost of a less mature managed-service ecosystem.

Making spot work in practice comes down to checkpointing discipline: save model state every 15–30 minutes so an interruption never costs more than a small slice of compute, and wire your training loop to catch the interruption notice and force an emergency checkpoint before the instance is reclaimed. Notice windows differ by cloud — AWS gives roughly two minutes, GCP around 30 seconds — so your checkpoint-on-signal path needs to be fast and already tested, not something you build during an actual interruption.

Sharded checkpointing (writing each rank's state independently rather than gathering to one node) keeps this fast even for large models. With this in place, teams commonly run 40–70% of total training compute on spot, reserving on-demand or reserved capacity for the final, time-critical stretch of a run.

Multi-Tenancy and Quota Management

Once a GPU cluster serves more than one team, capacity contention becomes a governance problem, not just a scheduling one. The pattern that scales:

• Namespace or project-level quotas enforced at the scheduler (Kubernetes ResourceQuota on nvidia.com/gpu, Slurm QOS/association limits, or Run:ai's hierarchical quota tree) prevent one team from starving another during a capacity crunch.
• Fair-share with burst — give teams a guaranteed baseline allocation, but allow bursting into idle capacity when other teams aren't using their share, with preemption rules for reclaiming it.
• Priority tiers distinguishing interactive/dev workloads (need fast turnaround, small jobs) from batch training (long-running, preemptible) prevent a single large job from blocking every small experiment on the cluster.
• Chargeback visibility — even in an internal cluster, per-team GPU-hour reporting changes behavior; teams over-request far less once usage is visible and attributed.

Monitoring: DCGM as the Foundation

Whatever scheduler stack you run, GPU-level observability should go through NVIDIA's Data Center GPU Manager (DCGM). dcgm-exporter is the standard bridge into Prometheus: it's a Go binary built on the DCGM API that exposes GPU telemetry on a /metrics HTTP endpoint, deployable as a standalone container or — more commonly — as a DaemonSet installed automatically by the GPU Operator.

The metrics worth alerting on, not just collecting:

• GPU utilization (SM activity) — sustained low utilization across a fleet is the clearest signal you're over-provisioned or under-scheduling; sustained near-100% with growing queue depth signals you need more capacity or better job sizing.
• Memory controller utilization and free memory — catches jobs that are memory-bound rather than compute-bound, a common reason to reach for MIG instead of a bigger GPU.
• ECC error counts — rising correctable/uncorrectable ECC errors on a specific GPU is an early hardware-failure signal, worth an automated cordon-and-drain before it causes a training job to crash mid-run.
• Power draw and temperature — useful for catching throttling before it silently degrades training throughput.
• NVLink and PCIe throughput — for multi-GPU training, a GPU with unexpectedly low NVLink traffic relative to its peers often indicates a topology or placement misconfiguration, not a hardware fault.

Grafana dashboards built on the official DCGM-Exporter dashboard template give a fast starting point; the value comes from wiring the ECC-error and utilization-outlier signals into your existing alerting stack rather than treating DCGM as a passive dashboard.

Planning Checklist

Before committing budget or signing a reservation, confirm: workload tier (pretrain/fine-tune/inference) maps to the right GPU generation and memory size; scheduler choice matches job shape (Slurm for batch training, Kubernetes for services, Run:ai/KAI if you need fractional GPUs and multi-team quotas); MIG vs. time-slicing decision is made per workload rather than cluster-wide; spot eligibility is decided per job based on checkpoint maturity, not blanket policy; and DCGM-based monitoring is live before the first production job runs, not added after the first mystery slowdown.