[HN Gopher] Life of an inference request (vLLM V1): How LLMs are...
       ___________________________________________________________________
        
       Life of an inference request (vLLM V1): How LLMs are served
       efficiently at scale
        
       Author : samaysharma
       Score  : 52 points
       Date   : 2025-06-28 18:42 UTC (4 hours ago)
        
 (HTM) web link (www.ubicloud.com)
 (TXT) w3m dump (www.ubicloud.com)
        
       | 0xjunhao wrote:
       | Hi, I'm the author of this post. Writing it was a great learning
       | experience. I gained a lot of insight into vLLM. If you have any
       | feedback or questions, feel free to drop a comment below!
        
         | criemen wrote:
         | Thanks for writing the article!
         | 
         | I didn't quite get
         | 
         |  _Note that during the prefill phase, all prompt tokens from a
         | request can be processed in one batch. This is possible because
         | the query (Q) tensors, calculated from the tokens immediately
         | before them, are available for each prompt token position._
         | 
         | I know that in practice prefill is much faster than inference.
         | Would watching the 2h video from Karpathy help me understand
         | why?
        
           | criemen wrote:
           | And on the topic of prefill: Do you know what the role of
           | GPUs is vs. in inference?
        
       ___________________________________________________________________
       (page generated 2025-06-28 23:00 UTC)