Production vLLM on Kubernetes: a deployment guide

Getting vLLM to serve one model on one GPU is a one-liner. Getting it to survive real traffic, restarts, node failures and a finance review is the actual job. This guide walks the production concerns in the order they bite: the OpenAI-compatible server, GPU scheduling, the throughput knobs, the KV-cache, tensor parallelism, probes, autoscaling and model loading.

The OpenAI-compatible server

vLLM ships an OpenAI-compatible HTTP server, so your existing OpenAI SDK clients point at it with a base URL change and nothing else. Launch it with vllm serve (equivalently python -m vllm.entrypoints.openai.api_server):

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama3-8b \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --port 8000

It exposes /v1/chat/completions, /v1/completions and /v1/models, a /health endpoint that returns 200 when the engine is up, and a Prometheus /metrics endpoint. The --served-model-name flag decouples the public model id your clients send from the on-disk weights path — worth setting so you can swap the underlying checkpoint without breaking callers.

Requesting the GPU (integer-only, request == limit)

GPUs are exposed to Kubernetes as the nvidia.com/gpu extended resource, and extended resources have two hard rules: they are integer-only and non-overcommittable. You cannot request 0.5 of a physical card through this resource, and you set it under limits — Kubernetes copies that value into requests, so request always equals limit. Fractional sharing is a separate mechanism (MIG or time-slicing); the plain resource is whole cards.

resources:
  limits:
    nvidia.com/gpu: 1      # request is set equal automatically

One more thing bites tensor-parallel and multi-GPU pods: vLLM uses shared memory for inter-process/NCCL communication, and the container default /dev/shm is tiny. Mount a memory-backed volume:

volumes:
- name: dshm
  emptyDir: {medium: Memory, sizeLimit: 8Gi}
# ... in the container:
volumeMounts:
- {name: dshm, mountPath: /dev/shm}

Continuous batching and --max-num-seqs

vLLM's headline feature is continuous (in-flight) batching: it admits new requests into the running batch as sequences finish, instead of waiting for a static batch to complete, which keeps the GPU saturated across concurrent users. You don't turn it on — it is the scheduler. What you tune is --max-num-seqs, the ceiling on sequences running concurrently, and --max-num-batched-tokens, the token budget per scheduler step. Raise --max-num-seqs for more concurrency and throughput; lower it if you're seeing KV-cache preemption (vLLM logging that it is recomputing or swapping sequences because the cache filled). These two flags, plus memory utilization below, are the throughput levers that matter most.

KV-cache and --gpu-memory-utilization

After weights load, vLLM allocates the remaining GPU memory to the KV-cache — the per-token attention state that actually determines how many sequences fit at once. --gpu-memory-utilization (default 0.90) is the fraction of each card vLLM is allowed to claim. Push it toward 0.95 and you get more KV-cache blocks and higher concurrency; push too far and you leave no headroom for fragmentation spikes and risk CUDA OOM at load. --max-model-len caps context length, which directly caps worst-case KV-cache per request — set it to what you actually serve, not the model's theoretical maximum, or a handful of huge-context requests will starve everyone else. If your workload shares a long system prompt across requests, --enable-prefix-caching reuses that prefix's KV blocks instead of recomputing them.

Tensor parallelism across GPUs

When a model doesn't fit on one card, shard it with --tensor-parallel-size N (across GPUs in one node) and, for very large models across nodes, --pipeline-parallel-size. Match the pod's GPU limit to the tensor-parallel degree — --tensor-parallel-size 4 needs nvidia.com/gpu: 4 — and keep those GPUs on the same node with fast interconnect (NVLink/NVSwitch where available), because tensor parallelism is communication-heavy and splitting across a slow link tanks throughput. Don't reach for parallelism as a default: a model that fits on one GPU serves cheaper on one GPU than sharded across four, because you pay the communication overhead for nothing.

Readiness and liveness probes

The single most common vLLM-on-K8s outage is traffic arriving before the model finishes loading. Loading tens of GB of weights takes real time, so your probes must tolerate it. Point readiness at /health and give liveness a generous startup grace (or use a startupProbe) so the kubelet doesn't kill the pod mid-load:

startupProbe:
  httpGet: {path: /health, port: 8000}
  failureThreshold: 60
  periodSeconds: 10          # allow up to ~10 min to load
readinessProbe:
  httpGet: {path: /health, port: 8000}
  periodSeconds: 10
livenessProbe:
  httpGet: {path: /health, port: 8000}
  periodSeconds: 20
  failureThreshold: 3

Readiness gates the Service endpoint, so a still-loading replica never receives requests; the startup probe protects the slow first minutes; liveness restarts a genuinely wedged engine.

Autoscaling on queue depth, not CPU

CPU and memory are near-useless signals for LLM serving — a GPU-bound pod can be maxed out at low CPU. Scale on vLLM's own Prometheus metrics instead: vllm:num_requests_waiting (queue depth) and vllm:num_requests_running, with vllm:gpu_cache_usage_perc as a saturation indicator. KEDA's Prometheus scaler is the clean path — scale up when average waiting requests per replica crosses a threshold. Two caveats keep it safe: cold starts are slow, so keep a warm floor of replicas and scale up fast, down gently; and scale-to-zero is only viable for genuinely intermittent models where an accepted first-request delay is fine. This is the same request-aware pattern covered in reducing LLM inference cost.

Model loading: PVC vs object store

Where the weights come from decides your startup time and your blast radius. Three common patterns:

• Pull at startup from Hugging Face or an object store (S3/GCS) into an emptyDir. Simplest, but every pod re-downloads, and a rate limit or outage upstream stalls every cold start.
• Shared read-only PVC (a ReadOnlyMany volume) holding the weights, pre-populated once. Fast, repeatable cold starts and no external dependency at scale-out — usually the right default for a fleet.
• Node-local cache / image bake. Weights on fast local NVMe or baked into the image trade image size and node prep for the fastest possible load.

Whichever you pick, mount the cache read-only and set HF_HOME/the model path explicitly so vLLM reads from it rather than re-fetching.

Multi-model routing

A vLLM server hosts one model. To serve several, run one Deployment per model — each independently sized, scaled and probed — and put a router in front that reads the OpenAI model field and forwards to the right backend Service (LiteLLM, an API gateway, or the vLLM production-stack router all do this). This keeps a heavy model's traffic from starving a light one, lets you scale and roll each independently, and isolates a bad deploy to a single model. Consolidate genuinely low-traffic models onto shared GPUs via MIG or time-slicing rather than giving each its own idle card — see time-slicing vs MIG.

The short version

Get these right and vLLM is a genuinely boring, high-utilization serving layer. Get the probes and autoscaling wrong and you get 3 a.m. pages and a GPU bill for capacity you never used.

Related reading

Serve efficiently, then watch it: reduce LLM inference cost, set up DCGM + Prometheus + Grafana monitoring, right-size GPU requests, and choose time-slicing vs MIG.