What is llm-d?
llm-d is a Kubernetes-native framework for running language model inference across many machines. It sits above model servers such as vLLM rather than replacing them: vLLM executes a model on one node, and llm-d decides which node should handle which request and how the work is divided between them.
It is a joint initiative involving Red Hat, Google, IBM Research, CoreWeave and NVIDIA among others, and it joined the CNCF Sandbox in March 2026.
The problem it solves
A single GPU server has a ceiling. When you reach it, the instinct is to run several copies of the model behind a load balancer, and that works for a while. It is also wasteful, because an ordinary load balancer does not know anything about what makes language model inference expensive.
Two facts about inference make naive load balancing a poor fit:
A request has two very different phases. First the model reads the prompt, which is compute-heavy and parallel. Then it generates tokens one at a time, which is memory-bandwidth-heavy and sequential. These phases want different things from hardware, and running both on the same GPU means one of them is always using it badly.
Conversations have expensive memory. The state a request accumulates, the KV cache, is large. If a follow-up message lands on a different server, that state is gone and the work is repeated from scratch. A round-robin balancer causes this constantly.
What it does about them
Prefill and decode disaggregation. The two phases are split across separate pools of workers, each sized and tuned for its own profile, so neither is stuck sharing hardware optimized for the other.
Distributed KV cache. Cached state can be moved between nodes rather than recomputed, so a follow-up request does not pay the full cost of the conversation again.
Inference-aware routing. Requests are directed by what the system knows about the workload, including which node already holds relevant cached state, rather than by turn-taking.
When you need it
Bluntly: later than most people think.
- When one node is genuinely the ceiling, and you have confirmed that by measuring rather than assuming.
- When conversations are long and prompts are reused, which is where cache locality is worth real money.
- When latency and throughput both matter and you cannot buy one with the other.
- When GPU spend is large enough that a percentage of it justifies operating a distributed system.
When you do not
If a single well-tuned vLLM node serves your traffic, stop here. Adding a distributed layer adds failure modes and operational work in exchange for capacity you are not using.
If you have not tuned what you have. Batching, quantization and model choice usually have more headroom than people expect, and they are much cheaper to change than the architecture.
If your traffic is bursty but small. Scaling replicas of a simple deployment is easier to run and easier to reason about.
This ordering is the important part of this page. llm-d is the answer to a scale problem, and applying it to a tuning problem produces a complicated system that is not faster.
What running it involves
- It presumes Kubernetes competence. This is a distributed system on top of a distributed system. Both need someone who can debug them.
- GPU topology matters. How nodes are interconnected determines whether cache transfer is a saving or a bottleneck, which makes hardware selection part of the design.
- More moving parts, more failure modes. Routing, cache transfer and worker pools each fail in their own way, and the useful metrics span all of them.
- It is young. CNCF Sandbox is the earliest maturity level. Interfaces change, and adopting it means following the project rather than installing it once.
Where VSHN fits
VSHN deploys and operates both vLLM and llm-d on Swiss cloud infrastructure, which means we have no reason to sell you the larger architecture when the smaller one is enough. Prompts, completions and cached state stay in the chosen jurisdiction, and the sovereignty assessment covers how that is scored.
The right first step is usually measurement: whether a single node is actually your ceiling, and if it is, whether tuning or distribution is the cheaper way past it. For the single-node half of that question, vLLM is the place to start.