Running GPU workloads on spot instances in Kubernetes without losing work

Spot and preemptible GPU nodes are the largest single discount available on accelerator compute — typically a very large percentage off on-demand, though the exact figure varies by cloud, region and GPU type. The catch is in the name: the cloud can reclaim the node with almost no notice. Handle interruptions properly and spot is close to free money; handle them badly and you lose a night of training. Here is how to run GPU work on spot in Kubernetes without losing work.

The economics, honestly

Spot capacity is the cloud's spare inventory sold at a steep, variable discount in exchange for the right to take it back. The discount is real and large, but it is not guaranteed capacity: during a regional crunch your preferred GPU type may be unavailable at spot prices, and running nodes can be reclaimed. So the mental model isn't "cheaper on-demand" — it's "interruptible capacity you must architect around." Everything below is about making the interruption a non-event.

The preemption notice — and why it's short

When the cloud reclaims a spot node it sends a preemption/interruption notice first, but the window is small. On Google Cloud you get roughly 30 seconds before the instance is stopped; on AWS a Spot interruption notice gives about two minutes. That's your entire budget to react — stop accepting new work, flush state, and exit cleanly. Two design consequences follow immediately: your shutdown path must be fast enough to finish inside that window, and your work must already be checkpointed, because you cannot save an hour of progress in 30 seconds.

React to the notice: graceful node shutdown + SIGTERM handling

Kubernetes has a built-in kubelet feature, Graceful Node Shutdown, that detects the node going down and terminates pods in order, honoring a shutdown grace period (managed distributions such as GKE enable it on their nodes). On clouds where you manage it yourself, a node-termination handler (for example the AWS Node Termination Handler) watches the instance metadata for the interruption notice and cordons + drains the node. Either way, your pod has to actually respond to SIGTERM:

spec:
  terminationGracePeriodSeconds: 25   # must fit INSIDE the ~30s notice
  containers:
  - name: trainer
    lifecycle:
      preStop:
        exec: {command: ["/bin/sh","-c","/app/flush-checkpoint.sh"]}

Keep terminationGracePeriodSeconds comfortably under the cloud's notice window, and have the process trap SIGTERM to write a final checkpoint and close cleanly. A container that ignores SIGTERM and waits for SIGKILL loses whatever was in flight.

Checkpoint training frequently (the real safety net)

For training, the notice window is far too short to save a large model, so the actual protection is frequent checkpointing to durable storage during the run — not at shutdown. Write checkpoints on an interval (every N steps or minutes) to object storage (GCS/S3) or a persistent volume, keep the last few, and make startup resume from the latest checkpoint rather than restart. Then a preemption costs you at most the work since the last checkpoint. The shutdown handler's job shrinks to "best-effort flush the most recent state," and correctness no longer depends on catching the notice at all.

PodDisruptionBudgets — useful, but know the limit

A PodDisruptionBudget keeps a minimum number of replicas available during voluntary disruptions — node drains, cluster upgrades, autoscaler scale-down:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: infer-pdb}
spec:
  minAvailable: 2
  selector: {matchLabels: {app: llm-inference}}

The critical caveat: a cloud yanking a spot node is an involuntary disruption, which a PDB does not prevent. It helps when the platform drains the node before terminating (the graceful path), and it protects you from the autoscaler evicting too many replicas at once — but you must still design for a node simply vanishing. PDBs reduce correlated loss; they don't abolish it.

Spread across capacity so one pool can't take you down

Concentration is the enemy: if every replica sits in one zone on one instance type, a single capacity event reclaims them together. Spread the risk with topology spread constraints across zones and by diversifying instance types in the node pool so the scheduler can place work wherever spot capacity exists:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector: {matchLabels: {app: llm-inference}}

The more zones and GPU SKUs your workload can accept, the less likely a single reclaim wave takes down a meaningful fraction of your fleet at once.

Mix spot and on-demand with node affinity

The robust production pattern is a hybrid: a small on-demand baseline that guarantees service, plus spot for the elastic bulk. Managed spot nodes carry a label (on GKE, cloud.google.com/gke-spot: "true") you can select on, and it's common to taint spot nodes so only workloads that tolerate interruption land there. Route the on-demand floor with node affinity and let burst/interruptible replicas prefer spot:

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 80
      preference:
        matchExpressions:
        - {key: cloud.google.com/gke-spot, operator: In, values: ["true"]}
tolerations:
- {key: cloud.google.com/gke-spot, operator: Equal, value: "true", effect: NoSchedule}

Now normal traffic is served by the on-demand base even if every spot node disappears, while the cost savings ride on the spot majority.

Inference vs training: they need different things

The two workloads fail differently, so treat them differently:

Stateless inference is the easy win: keep enough replicas across zones with an on-demand floor and a preemption is invisible to users. Long training runs are safe on spot only when checkpoint-and-resume is genuinely reliable — test it by killing a node mid-run and confirming the job picks up from the last checkpoint, not from step zero.

The checklist

Spot GPUs pay off when all of this is true: the process traps SIGTERM and exits inside the notice window; terminationGracePeriodSeconds fits the ~30s (GCP) / ~2-min (AWS) budget; training checkpoints frequently to durable storage and resumes; replicas spread across zones and instance types; a PDB limits voluntary correlated loss; and an on-demand floor guarantees baseline service. Miss the checkpointing or the on-demand floor and the discount isn't worth the 3 a.m. surprise. Get them right and you're serving the same work at a fraction of the bill — the highest-leverage GPU cost move after fixing plain idle capacity, which the scheduling & idle-cost guide covers.

Related reading

Cut GPU spend from every angle: why GPU clusters sit idle, right-size GPU requests, reduce LLM inference cost, and deploy serving with production vLLM on Kubernetes.