[HN Gopher] llm-d, Kubernetes native distributed inference
___________________________________________________________________
llm-d, Kubernetes native distributed inference
Author : smarterclayton
Score : 91 points
Date : 2025-05-20 12:37 UTC (10 hours ago)
(HTM) web link (llm-d.ai)
(TXT) w3m dump (llm-d.ai)
| anttiharju wrote:
| I wonder if this is preferable to kServe
| smarterclayton wrote:
| llm-d would make sense if you are running a very large
| production LLM serving setup - say 5+ full H100 hosts. The aim
| is to be much more focused than kserve is on exactly the needs
| of serving LLMs. It would of course be possible to run
| alongside kserve, but the user we are targeting is not
| typically a kserve deployer today.
| anttiharju wrote:
| Do you think https://github.com/openai/CLIP can be ran on it?
| LLM makes me think of chatbots but I suppose because it's
| inference-based it would work. Somewhat unclear on what's the
| difference between LLMs and inference, I think inference is
| the type of compute LLMs use.
|
| I wonder if inference-d would be a fitting name.
| smarterclayton wrote:
| Inference is the process of evaluating a model ("inferring"
| a response to the inputs). LLMs are uniquely difficult to
| serve because they push the limits on the hardware.
|
| The models we support come from the model server vLLM https
| ://docs.vllm.ai/en/latest/models/supported_models.html,
| which has a focus on large generative models. I don't see
| CLIP in the list.
| dzr0001 wrote:
| I did a quick scan of the repo and didn't see any reference to
| Ray. Would this indicate that llm-d lacks support for pipeline
| parallelism?
| qntty wrote:
| I believe this is a question you should ask about vLLM, not
| llm-d. It looks like vLLM does support pipeline parallelism via
| Ray:
| https://docs.vllm.ai/en/latest/serving/distributed_serving.h...
|
| This project appears to make use of both vLLM and Inference
| Gateway (an official Kubernetes extension to the Gateway
| resource). The contributions of llm-d itself seems to mostly be
| a scheduling algorithm for load balancing across vLLM
| instances.
| smarterclayton wrote:
| We inherit any multi-host support from vLLM, so
| https://docs.vllm.ai/en/latest/serving/distributed_serving.h...
| would be the expected path.
|
| We plan to publish examples of multi-host inference that
| leverages LeaderWorkerSets - https://github.com/kubernetes-
| sigs/lws - which helps run ranked serving workloads across
| hosts. LeaderWorkerSet is how Google supports both TPU and GPU
| multi-host deployments - see https://github.com/kubernetes-
| sigs/lws/blob/main/config/samp... for an example.
|
| Edit: Here is an example Kubernetes configuration running
| DeepSeek-R1 on vLLM multi-host using LeaderWorkerSet
| https://github.com/kubernetes-sigs/wg-
| serving/blob/main/serv.... This work would be integrated into
| llm-d.
| rdli wrote:
| This is really interesting. For SOTA inference systems, I've seen
| two general approaches:
|
| * The "stack-centric" approach such as vLLM production stack,
| AIBrix, etc. These set up an entire inference stack for you
| including KV cache, routing, etc.
|
| * The "pipeline-centric" approach such as NVidia Dynamo, Ray,
| BentoML. These give you more of an SDK so you can define
| inference pipelines that you can then deploy on your specific
| hardware.
|
| It seems like LLM-d is the former. Is that right? What prompted
| you to go down that direction, instead of the direction of
| Dynamo?
| qntty wrote:
| It sounds like you might be confusing different parts of the
| stack. NVIDIA Dynamo for example supports vLLM as the inference
| engine. I think you should think of something like vLLM as more
| akin to GUnicorn, and llm-d as an application load balancer.
| And I guess something like NVIDIA Dynamo would be like Django.
| smarterclayton wrote:
| llm-d is intended to be three clean layers:
|
| 1. Balance / schedule incoming requests to the right backend
|
| 2. Model server replicas that can run on multiple hardware
| topologies
|
| 3. Prefix caching hierarchy with well-tested variants for
| different use cases
|
| So it's a 3-tier architecture. The biggest difference with
| Dynamo is that llm-d is using the inference gateway extension
| - https://github.com/kubernetes-sigs/gateway-api-inference-
| ext... - which brings Kubernetes owned APIs for managing
| model routing, request priority and flow control, LoRA
| support etc.
| rdli wrote:
| I would think that that the NVidia Dynamo SDK (pipelines)
| is a big difference as well (https://github.com/ai-
| dynamo/dynamo/tree/main/deploy/sdk/doc...), or am I missing
| something?
| smarterclayton wrote:
| That's a good example - I can at least answer about _why_
| it 's a difference: different target user.
|
| As I understand the Dynamo SDK it is about simplifying
| and helping someone get started with Dynamo on
| Kubernetes.
|
| From the user set we work with (large inference
| deployers) that is not a high priority - they already
| have mature deployment opinions or a set of tools that
| would not compose well with something like the Dynamo
| SDK. Their comfort level with Kubernetes is moderate to
| high - either they use Kubernetes for high scale training
| and batch, or they are deploying to many different
| providers in order to get enough capacity and need a
| standard orchestration solution.
|
| llm-d focuses on helping achieve efficiency dynamically
| at runtime based on changing traffic or workload on
| Kubernetes - some of the things the Dynamo SDK encodes
| are static and upfront and would conflict with that
| objective. Also, large deployers with serving typically
| have significant batch and training and they are looking
| to maximize capacity use without impacting their prod
| serving. That requires the orchestrator to know about
| both workloads at some level - which Dynamo SDK would
| make more difficult.
| rdli wrote:
| In this analogy, Dynamo is most definitely not like Django.
| It includes inference aware routing, KV caching, etc. -- all
| the stuff you would need to run a modern SOTA inference
| stack.
___________________________________________________________________
(page generated 2025-05-20 23:01 UTC)