How to Audit GPU Utilization on a Kubernetes Cluster

GPU utilization audits fail when teams rely on coarse GPU-busy percentages that hide whether workloads are compute-bound, memory-bound, or simply idle between batches. The dominant cost of an idle GPU is not electricity—it is the amortized capital or the reserved hourly price you pay whether the GPU is 5% or 95% utilized. A proper audit surfaces which pods are holding GPUs at low true utilization, quantifies the gap between allocated and productive capacity, and provides the data needed to justify consolidation, right-sizing, or reclaiming underused reservations.

The Real Cost of Low Utilization

An idle GPU draws a fraction of its TDP—tens of watts versus the hundreds of watts under load—but electricity is not the lever. The A100 SXM has a TDP around 400 W; the H100 SXM around 700 W. At typical data-center power costs, the difference between idle and full load is a few dollars per GPU per month. The dominant cost is the amortized capital expense or the reserved cloud hourly rate, which you pay in full whether the GPU runs inference at 80% SM occupancy or sits idle waiting for the next job. This is why utilization, not wattage, is the audit target.

The payoff from an audit is concrete: identifying pods that hold entire GPUs but use them sporadically lets you consolidate workloads via MIG, time-slicing, or batch-queue policies, reclaiming capacity without new hardware. Quantifying the gap between allocated GPUs (the integer count in pod requests) and productive GPU-hours provides the data to justify infrastructure changes or to challenge teams claiming they need more quota.

Instrumentation: DCGM and the Profiling Metrics

DCGM (Data Center GPU Manager) is NVIDIA's telemetry daemon. The dcgm-exporter DaemonSet, typically deployed via the GPU Operator, scrapes DCGM and exposes metrics in Prometheus format on each node. The most commonly graphed metric, DCGM_FI_DEV_GPU_UTIL, reports the percentage of time the GPU had at least one active kernel on any streaming multiprocessor. This metric can read 90% while the GPU is severely underutilized—if kernels are memory-bound and the SMs are stalled waiting for DRAM, or if only a small fraction of SMs have work, GPU_UTIL still reports the GPU as busy.

The profiling fields give truer utilization. DCGM_FI_PROF_SM_ACTIVE measures the fraction of cycles where at least one warp was active on the SMs, averaged across all SMs. DCGM_FI_PROF_SM_OCCUPANCY is the ratio of active warps to maximum theoretical warps. DCGM_FI_PROF_PIPE_TENSOR_ACTIVE reports Tensor Core activity, critical for transformer inference and training. DCGM_FI_PROF_DRAM_ACTIVE shows memory-controller utilization, revealing memory-bound workloads. DCGM_FI_PROF_PCIE_TX_BYTES and DCGM_FI_PROF_PCIE_RX_BYTES track PCIe traffic, useful for diagnosing host-to-device transfer bottlenecks.

These profiling metrics require DCGM to enable the profiling (DCP) mode, which adds modest overhead. The GPU Operator's dcgm-exporter ConfigMap can be edited to include the DCGM_FI_PROF_* fields. Once enabled, you can query Prometheus for DCGM_FI_PROF_SM_ACTIVE{gpu="0",UUID="..."} and compare it to DCGM_FI_DEV_GPU_UTIL. A large gap—GPU_UTIL at 85% but SM_ACTIVE at 30%—indicates the workload is not compute-limited, and the GPU is underused despite appearing busy.

Pod-Level Attribution

DCGM metrics are labeled by gpu index and UUID, but not by pod name or namespace. To attribute utilization to individual pods, you must join GPU telemetry with the pod-to-device mapping. The Kubernetes device plugin allocates GPUs by setting the pod's resource limit nvidia.com/gpu to an integer count, and the kubelet binds specific device IDs to the container's cgroup. This mapping is not exposed as a first-class API object; you must either inspect the device plugin's internal state or correlate GPU processes with pod cgroups.

One approach is to deploy a sidecar or node-level agent that queries NVML (NVIDIA Management Library) to list processes on each GPU, retrieves their process IDs, and maps those PIDs to Kubernetes pod UIDs via /proc/<pid>/cgroup. The cgroup path contains the pod UID, which you can resolve to pod name and namespace by querying the Kubernetes API. This sidecar can then label or re-export DCGM metrics with pod metadata, or write time-series records to a database for later analysis.

A simpler method for batch or inference workloads is to have each pod self-report its GPU usage. The Kubernetes downward API can inject the pod name, namespace, and UID as environment variables. The workload process can then periodically query its own GPU metrics via NVML or the DCGM API and emit them to a logging or metrics backend with pod labels attached. This requires modifying the workload or wrapping it in a monitoring shim, but it avoids the complexity of external process-to-pod correlation.

Measuring Allocation vs. Productive Utilization

The audit must distinguish between allocated GPU-hours and productive GPU-hours. Allocated GPU-hours are straightforward: sum the count of nvidia.com/gpu in each pod's resource limits, multiplied by the pod's runtime. Productive GPU-hours require a utilization threshold—for example, counting only the time when DCGM_FI_PROF_SM_ACTIVE exceeds 50%, or when DCGM_FI_PROF_PIPE_TENSOR_ACTIVE is above 40% for Tensor-Core workloads. The threshold depends on workload type: training runs often sustain high SM activity, while inference serving may have bursty utilization between requests.

Construct a time-series query that, for each GPU, calculates the fraction of time above the threshold over a measurement window (e.g., one week). Multiply that fraction by the number of GPUs allocated to each pod to estimate productive GPU-hours per pod. Compare the sum of productive GPU-hours to the sum of allocated GPU-hours across the cluster. A ratio of 0.6 means 40% of allocated GPU capacity was unproductive by your threshold—this is the waste figure that justifies intervention.

Identifying Underutilized Pods and Workload Patterns

Rank pods by their productive-utilization ratio (productive GPU-hours divided by allocated GPU-hours) over the audit period. Pods at the bottom of the list—those with ratios below 20%—are candidates for consolidation or right-sizing. Inspect their utilization time-series to distinguish between different underutilization patterns:

• Idle between jobs: The pod holds a GPU allocation but runs inference or training tasks sporadically, with long gaps of zero activity. These workloads are candidates for batch queuing (e.g., via Slurm or Volcano) or for moving to a shared MIG or time-sliced pool where the GPU is released between jobs.
• Memory-bound: DCGM_FI_DEV_GPU_UTIL is high but DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE are low, while DCGM_FI_PROF_DRAM_ACTIVE is high. The workload is bottlenecked on memory bandwidth, not compute. Optimizations include increasing batch size (if memory allows), fusing kernels to reduce DRAM round-trips, or using quantization to shrink model weights and activations.
• Small-batch inference: The pod runs an LLM or vision model with small request batches, so each forward pass is short and the GPU idles waiting for the next request. Continuous batching (as in vLLM) or request aggregation can increase utilization, or the workload can be moved to a smaller MIG instance (e.g., 1g or 2g) to free the rest of the GPU for other pods.

MIG and Time-Slicing for Consolidation

Multi-Instance GPU (MIG) partitions one physical GPU into isolated instances with hardware-level memory, cache, and fault isolation. Each MIG instance has a fixed slice of streaming multiprocessors and memory. A GPU is divided into seven compute slices and eight memory slices, so the maximum number of MIG instances is seven (seven 1g profiles). Profile naming is <compute-slices>g.<memory>gb. The compute-slice budget is seven, so the maximum count of a profile per GPU is floor(7 / compute-slices): 1g profiles max seven per GPU, 2g profiles max three per GPU (2+2+2=6), 3g profiles max two per GPU (3+3=6; a third would require nine slices), and 4g or 7g profiles max one per GPU.

On the A100-40GB, profiles are 1g.5gb, 2g.10gb, 3g.20gb, 4g.20gb, and 7g.40gb (memory slice = 5 GB). On the A100-80GB and H100-80GB, profiles are 1g.10gb, 2g.20gb, 3g.40gb, 4g.40gb, and 7g.80gb (memory slice = 10 GB). Note that 3g.40gb exists only on 80GB parts; on the A100-40GB the 3-slice profile is 3g.20gb.

MIG is exposed to Kubernetes via the NVIDIA device plugin. You configure MIG profiles on the node (via nvidia-smi mig commands or the GPU Operator's MIG manager), and the device plugin advertises each MIG instance as a separate resource, e.g., nvidia.com/mig-1g.5gb. Pods request MIG instances by specifying the corresponding resource in their limits. Because MIG instances are hardware-isolated, an out-of-memory error or crash in one instance does not affect others on the same GPU.

Time-slicing, by contrast, multiplexes the whole GPU in time across multiple pods. It has no memory isolation and no fault isolation—an OOM or kernel crash in one pod can disrupt others sharing the GPU. Time-slicing is configured via the device plugin's config map, where you set a replicas count per GPU. The device plugin then advertises nvidia.com/gpu with a capacity equal to physical_gpu_count × replicas. Pods still request an integer count of nvidia.com/gpu, but multiple pods may be scheduled to the same physical device. Time-slicing is simpler to configure than MIG and works on GPUs that do not support MIG (e.g., consumer or older datacenter cards), but it provides no isolation guarantees.

Draining Nodes for Reconfiguration

Changing MIG profiles or device-plugin configuration requires draining GPU nodes to evict running pods. The kubectl drain command cordons the node (marks it unschedulable) and then evicts pods via the Eviction API, which respects PodDisruptionBudgets. Key flags:

kubectl drain <node> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=300 \
  --timeout=0

--ignore-daemonsets is required in practice because DaemonSet pods (including the device plugin and DCGM exporter) are not evicted by drain. --delete-emptydir-data allows drain to evict pods using an emptyDir volume (without this flag, drain will refuse to evict them). --grace-period overrides the pod's terminationGracePeriodSeconds, giving workloads time to checkpoint or shut down cleanly. --timeout=0 means wait forever (no timeout), not "give up immediately"—drain will block until all evictable pods terminate or you interrupt it.

The --disable-eviction flag makes drain delete pods directly instead of using the Eviction API, which bypasses PodDisruptionBudgets and force-removes running pods. It does not "only delete completed pods"; it is a hammer that ignores disruption policies. Use it only when the Eviction API is unavailable or when you need to forcibly clear a node despite PDB constraints.

After draining, reconfigure MIG profiles on the node (either manually via nvidia-smi mig -cgi <profile> -C or via the GPU Operator's MIG config CRD), restart the device plugin DaemonSet pod on that node to pick up the new profiles, then uncordon the node with kubectl uncordon <node>. Pods will reschedule to the newly available MIG resources.

Audit Reporting and Threshold Tuning

The output of the audit is a report listing each pod's allocated GPU-hours, productive GPU-hours, and utilization ratio over the measurement period, along with cluster-wide aggregates. Include a breakdown by namespace or team label to show which groups are under- or over-utilizing their allocations. Attach time-series graphs of key metrics—DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE—for the lowest-utilization pods to illustrate the specific bottleneck (idle, memory-bound, or low-batch).

Threshold tuning is workload-dependent. For large-batch training, a productive-utilization threshold of 70% SM_ACTIVE is reasonable, because well-optimized training should keep SMs busy. For inference serving, bursty utilization is expected; a threshold of 30–40% may be more appropriate, or you may measure utilization only during request-handling windows (excluding idle time between requests). For Tensor-Core-heavy workloads (transformers, large matmuls), consider using DCGM_FI_PROF_PIPE_TENSOR_ACTIVE as the primary metric instead of SM_ACTIVE, since high SM_ACTIVE without Tensor Core activity indicates the workload is not leveraging the hardware's peak throughput.

Continuous Monitoring and Utilization Policies

A one-time audit provides a snapshot, but utilization patterns change as workloads evolve. Continuous monitoring with alerting on low utilization—e.g., a Prometheus alert firing when a pod's 24-hour average DCGM_FI_PROF_SM_ACTIVE falls below 20%—enables proactive intervention. Pair monitoring with policy: require teams to justify GPU requests above a certain count, implement auto-scaling for inference workloads to release GPUs during low-traffic periods, or use a batch scheduler (Slurm, Volcano, Kueue) that allocates GPUs only when jobs are running and releases them on completion.

The mechanism that makes utilization audits actionable is the link between measurement and reclamation. Identifying underutilized pods is the first step; the second is having the tooling and process to consolidate those workloads onto fewer GPUs via MIG, time-slicing, or batch queuing, and the third is feeding the reclaimed capacity back into the allocation pool or decommissioning it to reduce cost. Without that closed loop, the audit remains a report rather than a lever for efficiency.