Kubernetes GPU Scheduling: Why Your Cluster Is 30–50% Idle (and Billing at 100%)

Here's the uncomfortable arithmetic of GPU infrastructure: a GPU costs exactly the same whether it runs at 100% or sits at 0%. You pay for the allocation, not the work. And Kubernetes, left to its defaults, allocates GPUs in a way that leaves a large fraction of them reserved-but-idle. Typical fleets run at 30–50% utilization — meaning half your GPU budget can be evaporating on cards nobody is using. This is why, and what to do about it.

The root cause: Kubernetes schedules whole GPUs

By default, nvidia.com/gpu is an integer resource. A pod requests 1, and the scheduler hands it an entire physical GPU — a Jupyter notebook, a small inference service, a CI job, all get a whole card whether they need 5% of it or 95%. So "the cluster is fully allocated" and "the cluster is fully utilized" become two very different numbers, and the gap between them is pure waste that still shows up on the invoice. Everything below is a way to close that gap.

Fragmentation: the idle you can't schedule around

The second driver is fragmentation, and it's subtler. Suppose each node has 8 GPUs and jobs land unevenly — a few 1-GPU pods scattered across several nodes leave, say, 3 free GPUs on this node and 2 on that one. Now an 8-GPU training job arrives and cannot schedule, even though the cluster has 5+ GPUs free, because they're not contiguous on one node. The free capacity is real and billed, but unusable. Multi-GPU and multi-node jobs make this worse: the more your workloads need co-located GPUs, the more scattered singletons strand capacity. Bin-packing scheduling policy (pack onto the fewest nodes) instead of spreading helps, as does keeping interactive/singleton work off the nodes reserved for large jobs.

Sharing a GPU: time-slicing vs MIG

If whole-GPU allocation is the problem, GPU sharing is the primary fix — and there are two mechanisms with different tradeoffs.

Time-slicing

Via the NVIDIA GPU Operator, time-slicing advertises one physical GPU as N schedulable units; pods take turns on the card. There's no hard memory or fault isolation — one pod can starve or OOM the others — so it's ideal for dev notebooks, CI, and light/bursty inference, where a 2–4× density gain on those workloads is common. It is the wrong tool for isolated production tenants.

MIG (Multi-Instance GPU)

On supported data-center GPUs, MIG hard-partitions a card into independent slices, each with its own memory and compute — real isolation. A large card becomes several smaller "GPUs," so small production models stop consuming whole cards. Profiles are GPU-specific; list them with nvidia-smi mig -lgip. The rule of thumb: time-slice for dev and elastic workloads, MIG for isolated production.

Taints, tolerations, and reserved pools that sit empty

A common self-inflicted idle source: dedicating a GPU node pool to one team or one workload with taints, then that workload underuses it while other teams queue for GPUs elsewhere. Taints are the right tool for protecting large-job nodes from interactive clutter — but a taint that reserves capacity for demand that isn't there is just idle you've fenced off. Audit your taints against actual utilization: a reserved pool at 20% is more expensive than a shared one at 70%.

kubectl describe nodes | grep -i -e taint -e nvidia.com/gpu

Queue fragmentation in Run:AI, Slurm, and gang scheduling

Higher-level schedulers — Run:AI, Slurm, Volcano/Kueue — add fair-share quotas and gang scheduling on top of Kubernetes, which is exactly what large GPU fleets need. But they introduce their own idle mode: queue fragmentation. When capacity is carved into per-team or per-project quotas, Team A's jobs can be queued and waiting while Team B's reserved GPUs sit idle — the cluster is idle and backlogged at the same time. The fixes are policy, not hardware: enable quota borrowing / preemption so idle reserved capacity is lent to queued work, use gang scheduling so multi-GPU jobs don't half-schedule and strand partial allocations, and set fair-share so reclamation is automatic rather than a Slack argument.

The last mile: allocated-but-idle GPUs nobody notices

Even with sharing and good scheduling policy, the final slice of waste is the stuck allocation: the notebook someone left open over the weekend, the finished job that never released its GPU, the crashed process still holding the card. These don't show up in "GPUs allocated" as a problem — they show up as full allocation with near-zero utilization. You catch them by reading per-GPU utilization from DCGM → Prometheus, flagging cards under ~5% for hours, and (opt-in) reclaiming them. Tracking reclaimed idle-GPU-hours × your $/GPU-hour is how the whole effort proves its own value.

What to actually measure

Getting a fleet from ~30% to ~70% average utilization typically cuts the GPU bill 30–60% for the same useful work — no new hardware, no quality tradeoffs, just scheduling and reclamation. That's why utilization, not capacity, is the number to chase.