[HN Gopher] Nvidia Dynamo: A Datacenter Scale Distributed Infere...
       ___________________________________________________________________
        
       Nvidia Dynamo: A Datacenter Scale Distributed Inference Serving
       Framework
        
       Author : ashvardanian
       Score  : 50 points
       Date   : 2025-03-18 20:44 UTC (2 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | Carrok wrote:
       | As someone who spent the better part of a year trying to get
       | various Nvidia inference products to work _at all_ even with a
       | direct line to their developers, I will simply say "beware".
        
         | vinni2 wrote:
         | Can you share some of your wisdom on setting up a scalable
         | inference infrastructure?
        
           | Carrok wrote:
           | Use Ray Serve. https://docs.ray.io/en/latest/serve/index.html
        
             | ipsum2 wrote:
             | As someone who has run LLMs in production, using Ray is
             | probably the worst idea. It's not optimized for language
             | models, and is extremely slow. There's no KV-caching, model
             | parallelism, and other basic table stakes features that are
             | offered by Dynamo or other open source inference
             | frameworks. Useful only if you have <1 QPS.
             | 
             | Use SGLang, vLLM, or text-generation-inference instead.
        
               | Carrok wrote:
               | This is probably true, but unlike every Nvidia product we
               | tried, it did, you know, reply to inference requests with
               | actual output. That said, you can serve vLLM with Ray
               | Serve.
               | https://docs.ray.io/en/latest/serve/tutorials/vllm-
               | example.h...
        
               | ipsum2 wrote:
               | Ray doesn't offer anything if you use vLLM on top of Ray
               | Serve though.
        
               | erulabs wrote:
               | It really depends on the task. If you have 1 massive job,
               | Ray sucks and doesn't provide table stakes. If you have
               | 50M tiny jobs, Ray and kuberay is great and serves as the
               | backbone of several billion dollar products.
               | 
               | Good for the goose, good for the gander...
        
         | islewis wrote:
         | is this in reference to Triton?
        
       | bloomingkales wrote:
       | _Built in Rust for performance and in Python for extensibility_
       | 
       | Omg, a team that knows how to selectively use tech as needed.
       | Looking at the Rust web developers in corner.
        
       | lmeyerov wrote:
       | So this replaces triton for LLMs or?
        
       ___________________________________________________________________
       (page generated 2025-03-18 23:00 UTC)