[HN Gopher] Production RAG: what I learned from processing 5M+ d...
___________________________________________________________________
Production RAG: what I learned from processing 5M+ documents
Author : tifa2up
Score : 265 points
Date : 2025-10-20 15:55 UTC (7 hours ago)
(HTM) web link (blog.abdellatif.io)
(TXT) w3m dump (blog.abdellatif.io)
| manishsharan wrote:
| Thanks for sharing. TIL about rerankers.
|
| Chunking strategy is a big issue. I found acceptable results by
| shoving large texts to to gemini flash and have it summarize and
| extract chunks instead of whatever text splitter I tried. I use
| the method published by Anthropic
| https://www.anthropic.com/engineering/contextual-retrieval i.e.
| include full summary along with chunks for each embedding.
|
| I also created a tool to enable the LLM to do vector search on
| its own .
|
| I do not use Langchain or python.. I use Clojure+ LLMs' REST
| APIs.
| esafak wrote:
| Have you measured your latency, and how sensitive are you to
| it?
| manishsharan wrote:
| >> Have you measured your latency, and how sensitive are you
| to it?
|
| Not sensitive to latency at all. My users would rather have
| well researched answers than poor answers.
|
| Also, I use batch mode APIs for chunking .. it is so much
| cheaper.
| jascha_eng wrote:
| I have a RAG setup that doesn't work on documents but other data
| points that we use for generation (the original data is call
| recordings but it is heavily processed to just a few text
| chunks). Instead of a reranker model we do vector search and then
| simply ask GPT-5 in an extra call which of the results is the
| most relevant to the input question. Is there an advantage to
| actual reranker models rather than using a generic LLM?
| tifa2up wrote:
| OP here. rerankers are finetuned small models, they're cheap
| and very fast compared to an additional GPT-5 call.
| jascha_eng wrote:
| It's an async process in my case (custom deep research like)
| so speed is not that critical
| alansaber wrote:
| I think you should do both in parallel, rather than
| sequentially. Main reason is vector scoring could cut off
| something that an LLM will score as relevant
| esafak wrote:
| They say the chunker is the most important part, but theirs looks
| rudimentary: https://github.com/agentset-
| ai/agentset/blob/main/packages/e...
|
| That is, there is nothing here that one could not easily write
| without a library.
| tifa2up wrote:
| OP here. We've been working on agentset.ai full-time for 2
| months. The product now gets you something working quite well
| out of the box. Better than most people with no experience in
| RAG (I'd say so with confidence).
|
| Ingestion + Agentic Search are two areas that we're focused on
| in the short term.
| teraflop wrote:
| I'm not sure there is a chunker in this repo. The file you
| linked certainly doesn't seem to _perform_ any chunking, it
| just defines a data model for chunks.
|
| The only place I see that actually operates on chunks does so
| by fetching them from Redis, and AFAICT nothing in the repo
| actually _writes_ to Redis, so I assume the chunker is
| elsewhere.
|
| https://github.com/agentset-ai/agentset/blob/main/packages/j...
| alexchantavy wrote:
| > What moved the needle: Query Generation
|
| What does query generation mean in this context, it's probably
| not SQL queries right?
| daemonologist wrote:
| It's described in the remainder of the point - they use an LLM
| to generate additional search queries, either rephrasings of
| the user's query or bringing additional context from the chat
| history.
| goleary wrote:
| Here's an interesting read on the evolution beyond RAG:
| https://www.nicolasbustamante.com/p/the-rag-obituary-
| killed-...
|
| One of the key features in Claude Code is "Agentic Search"
| aka using (rip)grep/ls to search a codebase without any of
| the overhead of RAG.
|
| Sounds like even RAG approaches use a similar approach (Query
| Generation).
| smokel wrote:
| The article raises several interesting points, but I find
| its claim that Claude Code relies primarily on grep for
| code search unconvincing. It's clear that Claude Code can
| parse and reason about code structure, employing techniques
| far beyond simple regex matching. Since this assumption
| underpins much of the article's argument, it makes me
| question the overall reliability of its conclusions a bit.
|
| Or am I completely misunderstanding how Claude Code works?
| andreasgl wrote:
| I think they mean query expansion:
| https://en.wikipedia.org/wiki/Query_expansion
| nextworddev wrote:
| Exactly what kind of processing was done? Your pipeline is a
| function of the use case, lest you overengineer...
| js98 wrote:
| Similar writeup I did about 1.5 years ago for processing millions
| of (technical) pages for RAG. Lots has stayed the same it seems
|
| https://jakobs.dev/learnings-ingesting-millions-pages-rag-az...
| winstonp wrote:
| I also built a RAG system about a year back for technical
| search, everything seems the same!
| daemonologist wrote:
| I concur:
|
| The big LLM-based rerankers (e.g. Qwen3-reranker) are what you
| always wanted your cross-encoder to be, and I highly recommend
| giving them a try. Unfortunately they're also quite
| computationally expensive.
|
| Your metadata/tabular data often contains basic facts that a
| human takes for granted, but which aren't repeated in every text
| chunk - injecting it can help a lot in making the end model seem
| less clueless.
|
| The point about queries that don't work with simple RAG (like
| "summarize the most recent twenty documents") is very important
| to keep in mind. We made our UI very search-oriented and
| deemphasized the chat, to try to communicate to users that search
| is what's happening under the hood - the model only sees what you
| see.
| thethimble wrote:
| I wish there was more info on the article about actual customer
| usage - particularly whether it improved process efficiency.
| It's great to focus on the technical aspects of system
| optimization but unless this translates to tangible business
| value it's all just hype.
| agentcoops wrote:
| I agree completely with your point, especially the difficulty
| of developing the user's mental model for what's going on with
| context and the need to move away from chat UX. It's
| interesting that there are still few public examples of non-
| chat UIs that make context management explicit. It's possible
| that the big names tried this and decided it wasn't worth it --
| but from comments here it seems like everyone that has built a
| production RAG system has come to the opposite conclusion. I'm
| guessing the real reason is otherwise: likely for the consumer
| apps controlling context (especially for free users) and
| inference time is one of the main levers for cost management at
| scale. Private RAGs, on the other hand, are more concerned with
| maximizing result quality and minimizing time spent by employee
| on a particular problem with cost per query much less of a
| concern --- that's been my experience at least.
| leetharris wrote:
| Embedding based RAG will always just be OK at best. It is useful
| for little parts of a chain or tech demos, but in real life use
| it will always falter.
| sgt wrote:
| What do you recommend? Query generation?
| esafak wrote:
| Compared with what?
| leetharris wrote:
| Full text agentic retrieval. Instead of cosine similarity on
| vectors, parsing metadata through an agentic loop.
|
| To give a real world example, the way Claude Code works
| versus how Cursor's embedded database works.
| lifty wrote:
| How do you do that on 5 million documents?
| charcircuit wrote:
| Most of my ChatGPT queries use RAG (based on the query ChatGPT
| will decide if it needs to search the web) to get up to date
| information about the world. In reality life it's effective and
| it's why every large provider supports it.
| underlines wrote:
| rag will be pronounced differently ad again and again. it has
| its use cases. we moved to agentic search having rag as a tool
| while other retrieval strategies we added use real time search
| in the sources. often skipping ingested and chunked soueces.
| large changes next windows allow for putting almost whole
| documents into one request.
| phillipcarter wrote:
| Not necessarily? It's been the basis of one of the major ways
| people would query their data since 2023 on a product I worked
| on: https://www.honeycomb.io/blog/introducing-query-assistant
|
| The difference is this feature explicitly isn't designed to do
| a whole lot, which is _still_ the best way to build most LLM-
| based products and sandwich it between non-LLM stuff.
| mediaman wrote:
| The point about synthetic query generation is good. We found
| users had very poor queries, so we initially had the LLM generate
| synthetic queries. But then we found that the results could vary
| widely based on the specific synthetic query it generated, so we
| had it create three variants (all in one LLM call, so that you
| can prompt it to generate a wide variety, instead of getting
| three very similar ones back), do parallel search, and then use
| reciprocal rank fusion to combine the list into a set of broadly
| strong performers. For the searches we use hybrid dense + sparse
| bm25, since dense doesn't work well for technical words.
|
| This, combined with a subsequent reranker, basically eliminated
| any of our issues on search.
| avereveard wrote:
| final tip is to also feed the interpretation of the user search
| to the user on the other side, so he can check if the llm
| understanding was correct.
| deepsquirrelnet wrote:
| > For the searches we use hybrid dense + sparse bm25, since
| dense doesn't work well for technical words.
|
| One thing I'm always curious about is if you could simplify
| this and get good/better results using SPLADE. The v3 models
| look really good and seem to provide a good balance of semantic
| and lexical retrieval.
| siva7 wrote:
| Boy, that should not be the concern of the end user (developer)
| but those implementing RAG solutions as a service at Amazon,
| Microsoft, Openai and so on.
| pamelafox wrote:
| At Microsoft, that's all baked into Azure AI Search - hybrid
| search does BM25, vector search, and re-ranking, just with
| setting booleans to true. It also has a new Agentic retrieval
| feature that does the query rewriting and parallel search
| execution.
|
| Disclosure: I work at MS and help maintain our most popular
| open-source RAG template, so I follow the best practices
| closely: https://github.com/Azure-Samples/azure-search-
| openai-demo/
|
| So few developers realize that you need more than just vector
| search, so I still spend many of my talks emphasizing the
| FULL retrieval stack for RAG. It's also possible to do it on
| top of other DBs like Postgres, but takes more effort.
| catmanjan wrote:
| I'd love to work with Azure search but because copilot with
| external items has been made so cheap it's hard to
| justify...
| pamelafox wrote:
| Do you mean that you're using the Copilot indexer for
| Sharepoint docs? https://learn.microsoft.com/en-
| us/microsoftsearch/semantic-i...
|
| AI Search team's been working with the Sharepoint team to
| offer more options, so that devs can get best of both
| worlds. Might have some stuff ready for Ignite (mid
| November).
| alansaber wrote:
| That is concerning given that pure vector search is
| terrible outside of abstractions
| pamelafox wrote:
| I know :( But I think vector DBs and vector search got so
| hyped that people thought you could switch entirely over
| to them. Lots of APIs and frameworks also used "vector
| store" as the shorthand for "retrieval data source",
| which didn't help.
|
| That's why I write blog posts like
| https://blog.pamelafox.org/2024/06/vector-search-is-not-
| enou...
| osigurdson wrote:
| It is almost like embeddings are a technology from the
| olden days.
| osigurdson wrote:
| Are you using Elasticsearch behind the scenes?
| pamelafox wrote:
| I believe that Azure AI Search currently uses lucene for
| BM25, hnswlib for vector search, and the Bing re-ranking
| model for semantic ranking. (So, no, it does not, though
| features are similar)
| alansaber wrote:
| Yep- that's all best practice. I want to know if we could push
| performance further- routing the query to different embedding
| models or scoring strategies, or using multiple re-rankers-
| still feels like the process is missing something.
| n_u wrote:
| > Reranking: the highest value 5 lines of code you'll add. The
| chunk ranking shifted a lot. More than you'd expect. Reranking
| can many times make up for a bad setup if you pass in enough
| chunks. We found the ideal reranker set-up to be 50 chunk input
| -> 15 output.
|
| What is re-ranking in the context of RAG? Why not just show the
| code if it's only 5 lines?
| tifa2up wrote:
| OP. Reranking is a specialized LLM that takes the user query,
| and a list of candidate results, then re-sets the order based
| on which ones are more relevant to the query.
|
| Here's sample code: https://docs.cohere.com/reference/rerank
| yahoozoo wrote:
| What is the difference between reranking versus generating
| text embeddings and comparing with cosine similarity?
| tifa2up wrote:
| text similarity finds items that closely match. Reranking
| my select items that are less semantically "similar" but
| are more relevant to the query.
| derefr wrote:
| My understanding:
|
| If you generate embeddings (of the query, and of the
| candidate documents) and compare them for similarity,
| you're essentially asking whether the documents "look like
| the question."
|
| If you get an LLM to evaluate how well each candidate
| document _follows from_ the query, you 're asking whether
| the documents "look like _an answer to_ the question. "
|
| An ideal candidate chunk/document from a cosine-similarity
| perspective, would be one that perfectly restates what the
| user said -- whether or not that document actually helps
| the user. Which _can_ be made to work, if you 're e.g.
| indexing a knowledge base where every KB document is SEO-
| optimized to embed all pertinent questions a user might ask
| that "should lead" to that KB document. But for such
| documents, even matching the user's query text against a
| "dumb" tf-idf index will surface them. LLMs aren't gaining
| you any ground here. (As is evident by the fact that
| webpages SEO-optimized in this way could already be easily
| surfaced by old-school search engines if you typed such a
| query into them.)
|
| An ideal candidate chunk/document from a re-ranking LLM's
| perspective, would be one that an instruction-following LLM
| (with the whole corpus in its context) would spit out as a
| response, if it were prompted with the user's query. E.g.
| if the user asks a question that could be answered with
| data, a document containing that data would rank highly.
| And that's exactly the kind of documents we'd like
| "semantic search" to surface.
| osigurdson wrote:
| Because LLMs are a lot smarter than embeddings and basic
| math. Think of the vector / lexical search as the first
| approximation.
| 383toast wrote:
| They should've tested other embedding models, there are better
| ones than openai's (and cheaper)
| prettyblocks wrote:
| Which do you suggest?
| roze_sha wrote:
| https://huggingface.co/spaces/mteb/leaderboard
| 383toast wrote:
| yep
| remz14 wrote:
| You should use RTEB instead. See here for why:
| https://huggingface.co/blog/rteb
|
| Here is that leaderboard https://huggingface.co/spaces/mteb
| /leaderboard?benchmark_nam...
|
| Voyage-3-large seems like SOTA right now
| leftnode wrote:
| The Qwen3 600M and 4B embedding models are near state of the
| art and aren't too computationally intensive.
| hatmanstack wrote:
| Not here to schlep for AWS but S3 Vectors is hands down the SOTA
| here. That combined with a Bedrock Knowledge Base to handle
| Discovery/Rebalance tasks makes for the simplest implementation
| on the Market.
|
| Once Bedrock KB backed by S3 Vectors is released from Beta it'll
| eat everybody's lunch.
| arcanemachiner wrote:
| Shill, not schlep.
|
| I'm correcting you less out of pedantry, and more because I
| find the correct term to be funny.
| hatmanstack wrote:
| I feel like I'm schelpin' through these comments, it's all
| mishigas
| esafak wrote:
| You feel like a _schlemiel_ , perhaps?
| hatmanstack wrote:
| more a schlimazel, Charles Schultzie, Lucy's everywhere
| latchkey wrote:
| Especially now that if you google the word schlep, the first
| result is now something totally different than what you'd
| expect.
| pietz wrote:
| I find it interesting that so many services and tools were
| investigated except for embedding models. I would have thought
| that's one of the biggest levers.
| Trias11 wrote:
| they just grabbed the better one (3-large) right off the bat.
| 6x cost to 3-small, but it's still tiny.
| pietz wrote:
| But the model is like 18 months old. and recently we've seen
| big leaps on MTEB. Not sure how well those translate to
| reality, but I'm a little surpised this wasn't worth looking
| into.
| bityard wrote:
| I must be missing something, this says it can be self-hosted. But
| the first page of the self-hosting docs say you need accounts
| with no less than 6 (!) other third-party hosted services.
|
| We have very different ideas about the meaning of self-hosted.
| goodev wrote:
| I consider this to be good open source and I'm a happy user of
| their OSS offering. Want no hosted dependencies? Then go write
| it all in Rust.
| icemanx wrote:
| that's a stupid take and shows lack of engineering experience
| dgfitz wrote:
| I've never worked in such a space where the deployed
| environment had unfettered internet access, no access at all
| actually.
|
| I've probably missed a huge wave of programming technology
| because of this, and I've figured out a way to make it work for
| a consistent paycheck over these past 20 years.
|
| I'm also not a great example, I think I've watched 7 whole
| hours of YouTube videos ever, and those were all for car repair
| help.
|
| I shy away from tech that needs to be
| online/connected/whatever.
| RobertDeNiro wrote:
| That was my observation as well. To be fair their business is
| to sell a hosted version, they're under no obligation to
| release a truly self hosted version.
| dcreater wrote:
| do you still use langchain/llamaindex for other agents/AI use
| cases?
| bitpatch wrote:
| Really solid write-up -- it's rare to see someone break down the
| real tradeoffs of scaling RAG beyond the toy examples. The bit
| about reranking and chunking actually saving more than fancy LLM
| tricks hits home to me.
| whinvik wrote:
| Anybody know what is meant by 'injecting relevant metadata'.
| Where is it injected?
| inshard wrote:
| Nice app bro https://usul.ai/chat/VgnzXjlRdljIDMBVCfqiy
| osigurdson wrote:
| Speaking of embedding models, OpenAIs are getting a little long
| in the tooth at this stage.
___________________________________________________________________
(page generated 2025-10-20 23:00 UTC)