How to Right-Size GPU Requests in Kubernetes

Right-sizing on CPU is simple: measure usage, lower the request, pack more pods on. For GPUs that playbook hits a wall — you cannot request a fractional GPU, and you cannot overcommit one. Right-sizing GPUs means choosing the right sharing mechanism, backed by real measurement.

💡 Put a number on the waste first: the free GPU Idle-Cost Calculator turns fleet size and utilization into an annual figure in two minutes.

The constraint that changes everything

GPUs are an extended resource, nvidia.com/gpu, advertised by the NVIDIA device plugin — and two rules follow. First, it is integer only: 1 is valid, 0.5 is rejected. Second, request must equal limit: Kubernetes requires extended-resource requests to equal their limits, killing the CPU trick of a low request with a high limit. There is no GPU overcommit and no bursting — a scheduled pod fully reserves the card and bills as fully used whether it runs at 5% or 95%.

resources:
  limits:
    nvidia.com/gpu: 1     # request implied equal; no 0.5

So the CPU lever — shave the request — does not exist. The waste is a whole card handed to a workload that needs a sliver, and right-sizing is about sharing that card safely.

Step 1: measure real utilization first

The intuition "this model is small, so it under-uses the GPU" is unreliable — some small models are memory-bandwidth bound and saturate a card. Get the truth from DCGM: DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for real compute activity (not the coarse DCGM_FI_DEV_GPU_UTIL), and DCGM_FI_DEV_FB_USED for the VRAM footprint that decides how many co-tenants fit. Workloads worth sharing sit low on both: notebooks, CI, and bursty inference.

Step 2: pick a sharing strategy per workload

Fractional GPUs are delivered one layer down, by the device plugin. Time-slicing (via the GPU Operator) advertises a card as N replicas that take turns on it, with no memory or fault isolation — ideal for dev, CI and light inference, wrong for isolated tenants. MIG, on A100/H100 and newer, hardware-partitions a card into isolated instances with dedicated memory and compute (fixed profiles such as 1g.10gb or 3g.40gb on an A100-80GB); see time-slicing vs MIG. MPS adds concurrent submission with a per-client memory cap, but still no fault isolation. A job that saturates a card should keep it.

Step 3: bin-pack, and separate by latency class

Placement decides whether you reclaim the capacity. Thin spreading re-creates fragmentation, so prefer a bin-packing posture (the MostAllocated scoring strategy in the scheduler's NodeResourcesFit plugin) that fills nodes and leaves whole free nodes for large jobs. Never time-slice a latency-sensitive service onto the same card as a heavy batch job — with no isolation, the batch bursts become tail-latency spikes. Keep serving on whole cards or dedicated MIG instances and dev/batch/CI on shared pools steered with taints. Measure from DCGM, match each workload to a strategy, bin-pack, and separate — that moves a fleet from ~30% toward the 60–70% range.

The strategy is clear; the hard part is finding which of your workloads are safe to share — on your real cluster, from your real telemetry. That is what a GPU Cost & Reliability Audit does: measure it, map each workload to a strategy, and hand you the config.

Start with the number: the free GPU Idle-Cost Calculator →
Get the next field note. Practical, occasional notes on GPU/K8s scheduling and cost — the traps and the numbers. No spam.

Related reading

See real utilization with DCGM, Prometheus & Grafana, choose time-slicing vs MIG, understand why clusters sit 30–50% idle, and reduce LLM inference cost.