How to Reduce LLM Inference Cost on Kubernetes

Training got the headlines; inference is where the money actually goes. A model is trained a handful of times and served millions. If your AI GPU bill is climbing, it's almost certainly your serving fleet — and most of that cost is recoverable without touching model quality. Here are the levers, in the order that usually pays off fastest.

The one metric that governs cost: throughput per GPU

Inference cost reduces to a single ratio: tokens served per GPU-hour ÷ your $/GPU-hour. You lower cost either by raising the numerator (serve more tokens on the same card) or lowering the denominator (cheaper cards, spot). Everything below is one of those two moves. And because a GPU bills at 100% whether it's saturated or idle, the fastest wins come from raising throughput on hardware you're already paying for.

1. Continuous batching — the single biggest throughput win

Naive serving processes one request (or one static batch) at a time; the GPU stalls waiting for the slowest sequence to finish. A modern serving engine — vLLM, TGI, TensorRT-LLM — uses continuous (in-flight) batching: it admits new requests into the running batch as slots free up, so the GPU stays saturated across many concurrent users. For real, concurrent traffic this is typically the largest single improvement in cost-per-token you can make, and it's a serving-layer change — the model is untouched. If you're still serving LLMs from a hand-rolled Flask/FastAPI loop, this is where to start.

2. KV-cache: the hidden memory (and cost) driver

Every token of context is held in the KV-cache, and that memory — not the weights — usually dictates how many concurrent requests fit on a GPU. Two consequences for cost:

• PagedAttention-style cache management (in vLLM and similar) packs the cache far more densely than contiguous allocation, raising the concurrency a single GPU can hold — directly more throughput per card.
• Long contexts are expensive. KV-cache grows with context length × concurrency, so a service that lets every request carry a huge prompt quietly caps your batch size. Trimming context, enabling prefix caching for shared system prompts, and capping max sequence length are real cost levers, not just latency ones.

3. Quantization — fit more model on less GPU (with eyes open)

Serving weights at lower precision (FP8, INT8, INT4 via AWQ/GPTQ and similar) shrinks the memory footprint and can raise throughput, which often lets a model drop to a smaller GPU or fewer of them — a direct cost cut. The honest tradeoff: quantization can degrade quality, and how much depends on the model and the method. Treat it as a change to validate, not a free win: run your own eval set before and after and decide with numbers. Done right it's one of the highest-leverage cost moves; done blindly it's a quality regression you'll discover in production.

4. Right-size the replica to the model

A frequent, silent waste: serving a model that fits comfortably on one GPU across a multi-GPU replica "for headroom," or reserving a whole 80GB card for a small model. Tensor/pipeline parallelism has real overhead — split a model across more GPUs than it needs and you pay in both idle capacity and communication cost. Right-sizing means: use the fewest GPUs the model and your latency target actually require, and pack small models onto shared or partitioned GPUs (see MIG and time-slicing) rather than giving each its own card.

5. Autoscale on the right signal for bursty traffic

Inference traffic is spiky, and this is where money leaks. If you provision replicas for peak, GPUs sit idle between bursts while billing at 100%. The fix is autoscaling — but CPU/memory HPA is the wrong signal for LLM serving. Scale on a request-aware metric instead: queue depth, requests/pending per replica, or a custom throughput metric exported from your serving engine, driven through KEDA or a custom-metrics HPA. Two caveats that make it safe:

• Cold starts are slow — loading tens of GB of weights takes time, so keep a warm floor of replicas and scale up aggressively, down gently.
• Scale-to-zero is viable for genuinely intermittent internal models, paired with a fast loader and an accepted first-request latency.

6. Cheaper GPU-hours: spot and the right accelerator

The denominator lever. Interruption-tolerant inference — async/batch scoring, offline generation, anything idempotent and retryable — runs well on spot/preemptible GPU nodes at a large discount, provided you handle preemption gracefully (fast drain, ret/requeue, a warm on-demand floor for latency-critical traffic). Separately, match the accelerator to the workload: the newest, largest GPU isn't always the cheapest per token for a small model, and a smaller card at high utilization can beat a big one at 30%.

Putting it together

The pattern across all six: utilization is the multiplier. A serving fleet averaging 30% utilization is paying roughly 3× per useful token versus one kept busy. Fix batching and autoscaling first — they raise utilization without any model risk — then take the model-level wins (quantization, right-sizing) with your evals in hand.