The llm-d project just shipped version 1.0, and for the engineers who wake up at 3 a.m. to restart inference pods, that changes the rhythm of the night. The open-source software, founded by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, now gives teams a single way to run large language models across mixed Kubernetes clusters without burning through GPU budgets.
The Problem That Keeps Teams Up
Running inference at scale has become a quiet crisis inside companies deploying chatbots, coding assistants and agentic systems. The models themselves are free or cheap to download. Serving them — the part where a trained model actually answers a user — eats compute, memory and stored context with every request. Edoardo Vacchi, who works on the project at Red Hat, explained on a recent InfoQ podcast that traditional Kubernetes load balancers break LLM efficiency because they spread requests randomly across nodes, forcing each node to recompute the same conversational context from scratch.
Vacchi’s team saw the same pattern vLLM solved for a single box: PagedAttention lets one instance reuse key-value caches across requests. But enterprises don’t run one box. They run fleets across clouds, on NVIDIA GPUs, on Google TPUs, on whatever accelerator the procurement team negotiated. llm-d extends that cache awareness across the whole cluster.
Session-Aware Routing and the Human Side of Caching
Think of a multi-turn conversation like a family dinner where everyone remembers what was said five minutes ago. If you seat the same people at the same table, the conversation flows. If a load balancer reshuffles them every course, each new neighbor has to be briefed from the start. llm-d’s session-aware routing keeps the conversation at the same table — the same node — so the cached prefix context gets reused. At the same time, it watches node capacity so no single GPU drowns while others sit idle.
The software also introduces batch inference during quiet hours and semantic classification that routes a prompt to a smaller, cheaper model when the task allows, all without sending sensitive data outside the cluster. It supports vLLM and SGLang as model servers, and the 1.0 release adds multi-tenant controls so competing workloads share hardware without stepping on each other, plus high-availability routing and rolling upgrades that let teams patch without downtime.
Numbers That Translate to Budget Meetings
IBM Research tested the stack on quantum error-correction workloads using open models. They recorded a prefix-cache hit rate of about 98 percent and a blended serving cost of roughly eleven cents per million tokens — 11.5 times lower than their all-proprietary baseline. In another run, the 753-billion-parameter GLM 5.2 model on NVIDIA GPUs produced more than 6.6 million output tokens a minute at peak and served 3,000 concurrent coding agents without pre-emption. The project says that setup delivers per-token costs five to ten times below current commercial API pricing, with the biggest savings on input-heavy workloads like code review.
A Standard Interface Across Clouds and Silicon
The project now sits under the Cloud Native Computing Foundation, with contributions from more than 500 developers across over 25 organizations. Version 1.0 delivers a stable configuration interface so a team can write one deployment spec and run it on AWS, Google Cloud, CoreWeave or on-prem iron, whether the accelerators are NVIDIA H100s, AMD MI300s or Google TPUv7s. SemiAnalysis benchmarks show TPUv7 with llm-d disaggregated serving beating NVIDIA GB300 FP8 at every interactivity level and restoring parity even against GB300 FP4 at higher accuracy.
What This Means for the People Building the Future
For the platform engineer, it means fewer bespoke YAML files and fewer 3 a.m. pages. For the CFO, it means inference costs that look like a line item instead of a mystery. For the researcher, it means the open model they fine-tuned last week can run tomorrow on whatever cluster has capacity, without a rewrite. The human side of this breakthrough isn’t the scheduler or the cache — it’s the team that gets to ship on Friday and actually enjoy the weekend.
Sources
- Large language model
- Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM
- llm-d 1.0 aims to cut AI inference costs for firms
- The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference
- I measured my local LLM’s power bill and it costs less than a single $20 ChatGPT, Claude or Gemini subscription
- Local LLM Operation Guide 2026 — Running Securely on Your Own GPU｜Kabomain
Photo: Domaintechnik / Unsplash (Unsplash License)
