[HN Gopher] Nano-vLLM: How a vLLM-style inference engine works
___________________________________________________________________
Nano-vLLM: How a vLLM-style inference engine works
Author : yz-yu
Score : 200 points
Date : 2026-02-02 12:52 UTC (10 hours ago)
(HTM) web link (neutree.ai)
(TXT) w3m dump (neutree.ai)
| jbarrow wrote:
| The whole thing feels AI written, generated from the codebase.*
|
| *this is incorrect per the author's response, my apologies.
|
| For instance, it goes into (nano)vLLM internals and doesn't
| mention PagedAttention once (one of the core ideas that vLLM is
| based on)[1].
|
| Also mentions that Part 2 will cover dense vs MoE's, which is
| weird because nanovllm hardcodes a dense Qwen3 into the source.
|
| Here are better (imo) explainers about how vLLM works:
|
| - https://hamzaelshafie.bearblog.dev/paged-attention-from-firs...
|
| - https://www.aleksagordic.com/blog/vllm
|
| - https://huggingface.co/blog/continuous_batching
|
| Aleksa's blog is a bit in the weeds for my taste but it's really
| worth working through.
|
| A lot of the magic of vLLM happens in the PagedAttention kernels,
| which are really succinctly implanted in nanovllm. And the
| codebase is great and readable by itself!
|
| --
|
| 1. https://arxiv.org/abs/2309.06180
| lukax wrote:
| Not really in the PagedAttention kernels. Paged attention was
| integrated into FlashAttention so that FlashAttention kernels
| can be used both for prefill and decoding with paged KV. The
| only paged attention specific kernels are for copying KV blocks
| (device to device, device to host and host to device). At least
| for FA2 and FA3, vLLM maintained a fork of FA with paged
| attention patches.
| yz-yu wrote:
| Hi jbarrow, thanks for your feedback and the links you shared--
| they're great readings for me (and likely others too).
|
| That said, I need to clarify: the content was not written by
| AI, and certainly not generated from a database in one shot. If
| there's some agent + prompt that can produce what I wrote, I'd
| love to learn it--it would've saved me two weekends :)
|
| Before addressing your questions further, some context: I'm a
| developer with no ML background but plenty of Cloud Infra
| experience. I'm currently building an open-source AI Infra
| project, which is why I studied nano-vllm. So my writing
| reflects some gaps in ML knowledge.
|
| To your specific points:
|
| > it goes into (nano)vLLM internals and doesn't mention
| PagedAttention once
|
| I didn't find any explicit "paged attention" naming in nano-
| vllm. After reading the first article you linked--specifically
| the "Paged KV Caching" section--I believe the block management
| logic and CPU/GPU block mapping it describes is exactly what I
| covered in both posts. It may not be the full picture of paged
| attention, but I interpreted what I saw in the code and
| captured the core idea. I think that's a reasonable outcome.
|
| > Part 2 will cover dense vs MoE's, which is weird because
| nanovllm hardcodes a dense Qwen3 into the source
|
| This reflects my learning approach and background. Same as
| point 1--I may not have realized the block design was the
| famous PagedAttention implementation, so I didn't name it as
| such. For point 2, seeing a dense Qwen3 naturally made me
| wonder how it differs from the xx-B-A-yy-B MoE models I'd seen
| on Hugging Face--specifically what changes in the decoder
| layers. That curiosity led me to learn about MoE and write it
| up for others with the same questions.
|
| ---
|
| I completely understand that in this era, people care more
| about whether what they're reading is AI-generated--no one
| wants to waste time on low-effort slop with no human
| involvement.
|
| But as I explained above--and as my hand-drawn Excalidraw
| diagrams show (I haven't seen an LLM produce diagrams with
| logic that satisfies me)--this is the result of learning shaped
| by my own knowledge background and preferences.
| jacquesm wrote:
| Funny, this reads _even more_ AI written than the article
| itself.
| yz-yu wrote:
| Cool, humans hallucinate too. -- AI
| marcellus23 wrote:
| It really doesn't.
| Juvination wrote:
| The em dashes really aren't helping their case.
| _alternator_ wrote:
| Wait--do people here really think the em dash was
| nonexistent before LLMs? It's widely used by people like
| me who care about writing style. The reason LLMs use it
| is because they reflect care and concern about writing
| style.
| CodeMage wrote:
| Yeah, people do seem to think that em dashes are an
| indicator of GenAI. I have been accused of using AI to
| write my posts on a forum, precisely because of em
| dashes. That's how I found out about that particular
| sniff test people use.
|
| Hasn't made me change the way I write, though. Especially
| because I never actually type an em dash character
| myself. Back when I started using computers, we only had
| ASCII, so I got used to writing with double dashes.
| Nowadays, a lot of software is smart enough to convert a
| double dash into an em dash. Discourse does that and
| that's how I ended up being accused of being an AI bot.
| 1718627440 wrote:
| Shouldn't a double dash result in an en dash and only a
| triple in an em dash?
| Juvination wrote:
| Nobody ever said that they were nonexistent before LLMs.
| When you are investigating and trying to determine if
| something is AI generated they are the number one
| indicator.
|
| So if you're being accused of just spewing AI, then
| double down and spew what looks EVEN MORE like AI. What
| are you even doing?
| selfsimilar wrote:
| Number one indicator? A single punctuation mark that's
| trivial to make on most keyboards (option-dash on macOS).
| And generally people who write software are extra fixated
| on punctuation for obvious reasons: missing semi-colons
| break your build, etc. Maybe in some other niche message
| board people will use dash and em dash interchangeably,
| but here?
|
| Also, if the a single character is how you're red-
| flagging LLM output, do you know how easy it is to avoid?
| I didn't use it here at all, but how do you know I didn't
| run this through some slop-machine to tighten my prose?
| It's really low-effort take to say "just avoid em dashes
| so we know you're not an AI".
|
| https://www.mcsweeneys.net/articles/the-em-dash-responds-
| to-...
| pessimizer wrote:
| Yes, number one indicator. Yes, of course you can go
| through the output and take out all of the em-dashes.
| Then the number one indicator will obviously not work.
| WhitneyLand wrote:
| No, people think humans use it a lot less often than AI,
| because it's true. Especially for casual writing.
|
| The contrast might become even greater because some
| humans that did use them have stopped to avoid false
| accusations.
| mmaunder wrote:
| My guess it's a translator they're using.
| CodeMage wrote:
| It does, but what does that say about the state of
| communication in our industry? I've seen a lot of writing
| that reads like an AI produced it in contexts where I could
| be pretty sure no AI was involved. We want to sound
| professional, so we sanitize how we write so much that it
| becomes... whatever this current situation is.
|
| No offense intended to @yz-yu, by the way. I miss the times
| when more people wrote in an eccentric style -- like Steve
| Yegge -- but that doesn't detract from what you wrote.
| yz-yu wrote:
| The comments here turned out much more interesting than I
| expected--this has become a great place to discuss the
| difference between AI-generated, AI-written, and AI-
| assisted content.
|
| So let me start from @jbarrow's comment: "AI written,
| generated from the codebase."
|
| My actual learning process looked like this:
|
| 1. I walked through the nano-vLLM codebase, asking Claude
| Code some high-level questions to warm up. 2. Then I
| asked detailed questions one by one, let it explore, and
| double-checked the code myself. As someone without an ML
| background, it sometimes took hours to understand a
| single concept. 3. Once I felt I understood enough, I
| started drawing Excalidraw diagrams to explain what I
| learned.
|
| Does this count as "generated from the codebase"? I don't
| think so.
|
| Where we might disagree is the writing process.
|
| As a non-native English speaker, my workflow looks like
| this:
|
| 1. Write a short paragraph (<100 words), then ask my
| writing agent to "fix this for readability and grammar."
| 2. Review the output. *If it changes any technical
| meaning, I correct it.* I consider this a responsible way
| to write a tech blog. 3. Move to the next paragraph.
|
| Is this "AI-written"? I'd call it "AI-assisted." Every
| idea in every sentence is mine. Honestly, things like "em
| dashes" never stood out to me when reviewing. I suspect
| that's common for non-native speakers.
|
| I wrote this comment the same way. The LLM fixed 14
| grammar mistakes that I think would distract readers more
| than any LLM-ish phrasing.
|
| That said, I'm open to suggestions on how to improve my
| writing process :)
| tubs wrote:
| When text is (clearly) non native English I think most
| native readers don't even register grammar errors.
|
| To be honest most native readers wouldn't register
| grammar errors full stop.
|
| I guess I have more awe of people who speak a foreign
| language at all compared to piping it through some agent
| malarkey.
| lambda wrote:
| One thing to keep in mind is that a lot of non-native
| English speakers use LLMs to translate to English, or to
| polish their English prose; they may not realize that it
| causes the translation to come out in a very LLM-style
| tone. Not sure if that's the case here, but it looks like
| OP is a native Chinese speaker so may be using tools to
| translate to English.
| jacquesm wrote:
| It looks like you were right about that.
|
| https://news.ycombinator.com/item?id=46858409
|
| But: this was never a problem and now we have to
| distinguish between LLM generated, human generated, LLM
| polished and human generated. I'd much prefer it if
| people just wrote their own text, warts and all.
| WhitneyLand wrote:
| Actually I thought it was a great example clarity, focus, and
| economy of words that AI is not capable of at this point in
| time.
| yz-yu wrote:
| Since HN only allows one link per submission, dropping Part 2
| here.
|
| https://www.neutree.ai/blog/nano-vllm-part-2
| OsamaJaber wrote:
| Great job! This is the kind of project that should exist for
| every complex system Systems like vLLM's codebase are massive and
| hard to follow Would love to see the same approach for other
| infra (a nano-Kubernetes, nano Postgres.....
| vitaelabitur wrote:
| Shameless plug for my structured LLM outputs handbook which is
| written in a similar spirit:
| https://nanonets.com/cookbooks/structured-llm-outputs/
___________________________________________________________________
(page generated 2026-02-02 23:00 UTC)