NVIDIA GPU Operator Not Detecting GPUs on Kubernetes: The Debugging Checklist

The card is physically in the node. nvidia-smi works on the host. But kubectl describe node shows nvidia.com/gpu: 0, your pod is stuck Pending with Insufficient nvidia.com/gpu, and you're paying for an accelerator Kubernetes refuses to see. This is the checklist that finds it, in the order that resolves it fastest.

First, understand what actually advertises the GPU

Kubernetes does not discover GPUs from the kernel. Capacity is advertised by the NVIDIA device plugin, which registers with the kubelet over a socket and reports the nvidia.com/gpu extended resource. Behind it, the NVIDIA GPU Operator orchestrates a chain of components, and if any link is broken the node reports zero — even with perfectly healthy hardware. The chain is:

NFD labels the node → driver loads (host or Operator-managed) → container-toolkit rewrites the runtime → device plugin registers nvidia.com/gpu → DCGM exports metrics.

Debugging is just walking that chain until you find the broken link. Start here:

kubectl -n gpu-operator get pods
kubectl describe node <node> | grep -A6 Capacity

If nvidia.com/gpu is absent or 0 under Capacity/Allocatable, keep going.

1. Node labels — is Node Feature Discovery seeing the card?

The Operator uses Node Feature Discovery (NFD) to find GPU nodes before it schedules anything onto them. If NFD never labeled the node, no operand lands there and detection silently never starts. Check for the PCI vendor label (NVIDIA is 0x10de):

kubectl get node <node> -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep -i -e pci-10de -e nvidia

No feature.node.kubernetes.io/pci-10de.present=true? Confirm NFD is actually running (kubectl -n gpu-operator get pods | grep nfd), that the worker pcidevice is passed through (bare metal / passthrough VMs), and that a stray nodeSelector or taint isn't keeping NFD off the node.

2. The container runtime — the single most common cause

This is where most "0 GPUs" tickets actually end. Even with a loaded driver, a pod only gets a GPU if the container runtime invokes nvidia-container-runtime. On modern clusters that's wired through a RuntimeClass plus a containerd config edit. Two things must be true:

a. containerd must know the nvidia handler

Look in /etc/containerd/config.toml for the runtime handler the toolkit installs:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

If the container-toolkit pod edited the config but containerd was never restarted, the handler doesn't exist yet. Restarting containerd (or letting the toolkit do it) is a routine fix. Confirm the RuntimeClass object exists too: kubectl get runtimeclass.

b. the default runtime vs. an explicit RuntimeClass

Depending on how the Operator is installed, GPU pods either rely on nvidia being the default containerd runtime, or on each GPU pod carrying runtimeClassName: nvidia. A mismatch here — default runtime is still runc, pods don't set the RuntimeClass — means the GPU is never mounted into the container. Read the toolkit pod logs when in doubt:

kubectl -n gpu-operator logs -l app=nvidia-container-toolkit-daemonset

3. Driver / toolkit / plugin version skew

The driver, container-toolkit, device plugin, and (if used) DCGM are a matched set. Skew — an Operator bundle that expects a newer driver than the one installed, or a host-installed driver too old for the CUDA image your workload uses — shows up as a device plugin that crash-loops or registers zero. Establish ground truth from the driver daemonset, not from memory:

kubectl -n gpu-operator exec ds/nvidia-driver-daemonset -- nvidia-smi
kubectl -n gpu-operator logs ds/nvidia-driver-daemonset

Two rules keep you out of trouble: (1) decide up front whether the driver is host-managed or Operator-managed and don't run both — a pre-installed host driver with the Operator also trying to install one is a classic conflict; (2) the CUDA runtime inside your workload image must be supported by the installed driver (newer CUDA userspace generally needs a recent enough driver). If nvidia-smi works on the host but the driver daemonset is unhealthy, you likely have both fighting.

4. The device plugin itself — did it register?

This is the component that actually publishes nvidia.com/gpu. If steps 1–3 are clean but capacity is still 0, read its logs directly:

kubectl -n gpu-operator logs -l app=nvidia-device-plugin-daemonset

Healthy output enumerates the GPUs and reports registration with the kubelet. Common failures you'll see here: it can't find libnvidia-ml.so (driver not visible to the plugin — back to step 3), the kubelet device-plugin socket path is wrong (unusual kubelet config), or the pod can't schedule at all because of a taint. Speaking of which:

5. Taints and tolerations — the "it registered but nothing schedules" case

A subtle variant: capacity does show up, but your workload still sits Pending. Many GPU node pools carry a taint like nvidia.com/gpu=present:NoSchedule so only GPU workloads land there. If your pod doesn't tolerate it, the scheduler skips the node. Check both sides:

kubectl describe node <node> | grep -i taint
kubectl describe pod <pod> | grep -iA3 -e toleration -e events

The Operator's own operands ship with the right tolerations; your application pods need them added explicitly if the pool is tainted.

6. DCGM — the last mile for visibility

Once scheduling works, DCGM (and dcgm-exporter) is what gives you per-GPU utilization, ECC/XID errors, and thermal throttling in Prometheus. It's not required for scheduling, but if dcgm-exporter is crash-looping you'll be flying blind on exactly the metric that tells you whether the GPU you fought to detect is now sitting idle. Its logs live in the same namespace and its failures usually trace back to the same driver/toolkit chain above.

Quick reference: symptom → likely link in the chain

The meta-lesson

Almost every "GPU Operator not detecting GPUs" incident is one broken link in the NFD → driver → toolkit → device-plugin chain, and the fastest path is to walk it top-down reading each component's own logs rather than guessing. The hardware is almost never the problem; the wiring between the host driver and the container runtime almost always is.