GPU Monitoring on Kubernetes with DCGM, Prometheus & Grafana

The vendor-supported way to know whether a GPU is busy and healthy on Kubernetes is dcgm-exporter → Prometheus → Grafana: dcgm-exporter runs as a DaemonSet reading per-GPU telemetry from the driver, publishing it on :9400/metrics for Prometheus to scrape and Grafana to graph. Install it via the NVIDIA GPU Operator or the standalone Helm chart. The skill is knowing which DCGM_FI_* series matter.

💡 The free GPU Idle-Cost Calculator turns fleet size and utilization into an annual waste figure in two minutes — the number these dashboards help you attack.

Utilization: the metric that lies

DCGM_FI_DEV_GPU_UTIL is what nvidia-smi shows and what everyone graphs first — and it misleads. It reports the fraction of time at least one kernel was resident, so sparse tiny kernels pin it at 100% while the compute engines idle. For the truth add the profiling ("DCP") series: DCGM_FI_PROF_SM_ACTIVE (how much of the SM array works), DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (Tensor-Core activity) and DCGM_FI_PROF_DRAM_ACTIVE (memory-bandwidth pressure). Graph GPU_UTIL beside PROF_SM_ACTIVE and you'll routinely see "100% utilized" at 20% SM-active.

Memory, throttling, and health

Memory usually caps concurrency before compute: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE (framebuffer, MiB) drive a "used / total" panel. Throttling is the panel most skip — a card at a thermal/power limit silently downclocks, no error. DCGM_FI_DEV_CLOCK_THROTTLE_REASONS is a bitmask; the trouble bits include HW slowdown (0x8), HW thermal (0x40), HW power-brake (0x80) and SW power cap (0x4) — read it alongside DCGM_FI_DEV_GPU_TEMP. For health, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL counts uncorrectable double-bit ECC errors that corrupt compute, and DCGM_FI_DEV_XID_ERRORS reports the last XID the driver raised — up to fatal ones like Xid 79.

Wiring the scrape: the ServiceMonitor

With the Prometheus Operator, point Prometheus at the exporter's Service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: {name: dcgm-exporter, namespace: gpu-operator,
  labels: {release: kube-prometheus-stack}}  # must match serviceMonitorSelector
spec:
  selector: {matchLabels: {app: nvidia-dcgm-exporter}}
  endpoints: [{port: gpu-metrics, interval: 15s, path: /metrics}]

The usual "exporter is up but Prometheus is empty" bug is a label mismatch — Prometheus only adopts ServiceMonitors matching its serviceMonitorSelector, so verify that release: label matches.

The alerts most setups miss

Utilization and memory panels are table stakes; these alerts earn their keep. Throttling: a thermal/power bit in CLOCK_THROTTLE_REASONS for minutes. Hardware faults: any rise in ECC_DBE_VOL_TOTAL or a new XID_ERRORS — cordon and drain before it kills a job. Allocated-but-idle: a pod-bound GPU at near-zero PROF_SM_ACTIVE for hours, the biggest pure-waste signal. Memory saturation: FB_USED/(FB_USED+FB_FREE) over ~95%. Start from NVIDIA's official dcgm-exporter Grafana dashboard and layer these failure and waste alerts on top — that is how you catch both dying cards and idle spend.

Skip the hand-rolling: the GPU Monitoring Starter Pack ships the dcgm-exporter + Prometheus + Grafana config, the panels above, and the throttle/ECC/XID/idle alert rules — drop-in and validated.

And put a number on the idle you'll surface: the free GPU Idle-Cost Calculator →
Get the next field note. Practical, occasional notes on GPU/K8s monitoring and cost — the traps and the numbers. No spam.

Related reading

Once you can see real utilization, act on it: right-size GPU requests, choose time-slicing vs MIG, understand why clusters sit 30–50% idle, and reduce LLM inference cost.