[HN Gopher] So, you want to chunk really fast?
___________________________________________________________________
So, you want to chunk really fast?
Author : snyy
Score : 102 points
Date : 2026-01-05 17:19 UTC (5 hours ago)
(HTM) web link (minha.sh)
(TXT) w3m dump (minha.sh)
| snyy wrote:
| We're the maintainers of Chonkie, a chunking library for RAG
| pipelines.
|
| Recently, we've been using Chonkie to build deep research agents
| that watch topics for new developments and automatically update
| their reports. This requires chunking a large amount of data
| constantly.
|
| While building this, we noticed Chonkie felt slow. We started
| wondering: what's the theoretical limit here? How fast can text
| chunking actually get if we throw out all the abstractions and go
| straight to the metal?
|
| This post is about that rabbit hole and how it led us to build
| memchunk - the fastest chunking library, capable of chunking text
| at 1TB/s.
|
| Blog: https://minha.sh/posts/so,-you-want-to-chunk-really-fast
|
| GitHub: https://github.com/chonkie-inc/memchunk
|
| Happy to answer any questions!
| brene wrote:
| Do you see this project merge with the Chonkie at some point? Or
| do you intend to keep it separate?
| snyy wrote:
| Memchunk is already in Chonkie as the `FastChunker`
|
| To install: pip install chonkie[fast]
|
| ``` from chonkie import FastChunker
|
| chunker = FastChunker(chunk_size=4096) chunks =
| chunker(huge_document) ```
| SkyPuncher wrote:
| I've been seeing a bunch of LLM-adjacent articles recently that
| are focusing on being fast - and they leave me a bit stumped.
|
| While latency _can_ be a problem, reliability and accuracy are
| almost always my bottlenecks (to user value). Especially with
| chunking. Chunking is generally a one-time process where users
| aren't latency sensitive.
| chaboud wrote:
| If you have reliability and accuracy (big if) then the
| practical usability _and_ cost become performance problems.
|
| And this is a bit of a sliding scale. Of course users want the
| best possible answer. However, if they can get 80% (magic hand-
| wavey fakie number) of the best answer on one second instead of
| 20, that may be a worthwhile tradeoff.
| snyy wrote:
| > Chunking is generally a one-time process where users aren't
| latency sensitive.
|
| This is not necessarily true. For example, in our use case we
| are constantly monitoring websites, blogs, and other sources
| for changes. When a new page is added, we need to chunk and
| embed it fast so it's searchable immediately. Chunking speed
| matters for us.
|
| When you're processing changes constantly, chunking is in the
| hot path. I think as LLMs get used more in real time workflows,
| every part of the stack will start facing latency pressure.
| smlacy wrote:
| Not all languages have such well-defined and commonly used
| delimiters. Is this "English only"?
| snyy wrote:
| Which language are you thinking of? Ideally, how would you
| identify split points in this language?
|
| I suppose we've only tested this with languages that do have
| delimiters - Hindi, English, Spanish, and French
|
| There are two ways to control the splitting point. First is
| through delimiters, and the second is by setting chunk size. If
| you're parsing a language where chunks can't be described by
| either of those params, then I suppose memchunk wouldn't work.
| I'd be curious to see what does work though!
| smlacy wrote:
| There are certainly cases of Greek/Latin without any
| punctuation at all, typically in a historical context.
| Chinese & Japanese historically did not have any punctuation
| whatsoever.
| ks2048 wrote:
| Do the delimiters have to be single bytes? e.g. Japanese full
| stop (IDEOGRAPHIC FULL STOP) is 3 bytes in UTF-8.
| snyy wrote:
| No, delimiters can be multiple bytes. They have to be
| passed as a pattern.
|
| // With multi-byte pattern
|
| let metaspace = "<japanese_full_stop>".as_bytes();
|
| let chunks: Vec<&[u8]> =
| chunk(text).pattern(metaspace).prefix().collect();
| vjerancrnjak wrote:
| So, whole english wikipedia in <1 second (~20GB compressed)?
|
| Or is it now a lack of proper pipelining where you first load,
| then uncompress, then chunk, then write?
|
| Add a nice strong linear model on top like vowpal wabbit and
| chunk at 100GB/s any language of your choice.
| srcreigh wrote:
| 4/5 of today's top CNN articles have words with periods in them:
| "Mr.", "Dr.", "No.", "John D. Smith", "Rep."
|
| The last one also has periods within quotations, so period
| chunking would cut off the quote.
| Havoc wrote:
| I suspect chunking is an exercise in ,,good enough"
| snyy wrote:
| A big chunk size with overlap solves this. Chunks don't have to
| be be "perfectly" split in order to work well.
| srcreigh wrote:
| True, but you don't need 150GB/s delimiter scanning in that
| case either.
| snyy wrote:
| As the other comment said, its a practice in good enough
| chunks quality. We focus on big chunks (largest we can make
| without hurting embedding quality) as fast as possible. In
| our experience, retrieval accuracy is mostly driven by
| embedding quality, so perfect splits don't move the needle
| much.
|
| But as the number of files to ingest grows, chunking speed
| does become a bottleneck. We want faster everything
| (chunking, embedding, retrieval) but chunking was the first
| piece we tackled. Memchunk is the fastest we could build.
| ubertaco wrote:
| Does this even work if you're incredulous enough???
| neonsunset wrote:
| .NET's string.Split implementation is very close to what the
| article showcases, even 3-character limit is there:
| https://github.com/dotnet/runtime/blob/main/src/libraries/Sy...
| stabbles wrote:
| For the particular case of the 5 delimiters '\n', '.', '?', '!',
| and ';', it just happens to be so that you can do this as a
| single shuffle instruction, replacing the explicit lookup table.
|
| You can do this whenever `c & 0x0F` is unique for the set of
| characters you're looking for.
|
| See https://stoppels.ch/2022/11/30/io-is-no-longer-the-
| bottlenec... for details.
| bhavnicksm wrote:
| Hey! Author of the blog here.
|
| This is pretty cool~ Thanks for suggesting this, I will read
| this in detail and add it to the next (0.5.0) release of
| memchunk.
| dataflow wrote:
| Note your compiler might turn that _mm256_set_epi64x into a
| load from memory, so there might still be memory accesses you
| don't expect.
| mwsherman wrote:
| While this article is about perf -- and trading off semantic
| precision by design -- there is a Unicode standard for sentence
| boundaries, may be interesting:
| https://www.unicode.org/reports/tr29/#Sentence_Boundaries
|
| I implemented the sentence boundaries, but also thought that the
| notion of a "phrase" might be useful for such applications:
| https://github.com/clipperhouse/uax29/tree/master/phrases
| bob1029 wrote:
| > you have a massive pile of text, and you need to split it into
| smaller pieces that fit into embedding models or context windows.
|
| I think the recently posted Recursive Language Models paper
| approaches this in a far more compelling way. They put the long
| context into the environment and make the LLM write and iterate
| python code to query against it in a recursive loop. Fig. 2 & 4
| are most relevant here.
|
| https://news.ycombinator.com/item?id=46475395
|
| https://arxiv.org/abs/2512.24601
|
| I really like this because it is in The Bitter Lesson genre of
| solutions. Make the model learn the best way to retrieve info
| from a massive prompt on disk given the domain and any human
| feedback (explicit and otherwise).
|
| The bigger the prompt.txt, the less relevant the LLM's raw
| context capabilities are. Context scaling is quadratic in cost.
| It's a very expensive rabbit to chase. Recursively invoking the
| same agent with decomposed problem bits is more of a logarithmic
| scaling thing. You could hypothetically manage a 1 gigabyte
| prompt with a relatively minuscule context window under a
| recursive scheme using nothing other than a shell/python
| interpreter.
| analog8374 wrote:
| This warms my heart
| Neywiny wrote:
| Some notes: 1. Nice and tight article, good work 2. Shipped a
| piece of code, always props to that 3. The has_zero_byte it would
| be nice to actually do the math in the example. As is the example
| doesn't really show anything. It also should say "its" instead of
| "it's" 4. The work done per chunk shouldn't include the
| broadcasts. That should be done at the start of the search and
| those values kept in the registers, no? 5. Isn't AVX and SSE also
| SWAR? They're just wider registers 6. I think a graph showing the
| cost of the lookup table vs n needles would be cool to see
|
| Overall nice work
| fmstephe wrote:
| Can some clarify this part of the article for me
|
| "if you search forward, you need to scan through the entire
| window to find where to split. you'd find a delimiter at byte 50,
| but you can't stop there -- there might be a better split point
| closer to your target size. so you keep searching, tracking the
| last delimiter you saw, until you finally cross the chunk
| boundary. that's potentially thousands of matches and index
| updates."
|
| So I understand that this is optimal if you want to make your
| chunks as large as possible for a given chunk size.
|
| What I don't understand is why is it desirable to grab the
| largest chunk possible for a given chunk limit?
|
| Or have I misunderstood this part of the article?
| snyy wrote:
| You have the right understanding.
|
| We've found that maximizing chunk size gives the best retrieval
| performance and is easier to maintain since you don't have to
| customize chunking strategy per document type.
|
| The upper limit for chunk size is set by your embedding model.
| After a certain size, encoding becomes too lossy and
| performance degrades.
|
| There is a downside: blindly splitting into large chunks may
| cut a sentence or word off mid-way. We handle this by splitting
| at delimiters and adding overlap to cover abbreviations and
| other edge cases.
___________________________________________________________________
(page generated 2026-01-05 23:00 UTC)