[HN Gopher] Ask HN: How are you doing RAG locally?
___________________________________________________________________
Ask HN: How are you doing RAG locally?
I am curious how people are doing RAG locally with minimal
dependencies for internal code or complex documents? Are you using
a vector database, some type of semantic search, a knowledge graph,
a hypergraph?
Author : tmaly
Score : 343 points
Date : 2026-01-14 14:38 UTC (1 days ago)
| eajr wrote:
| Local LibreChat which bundles a vector db for docs.
| whattheheckheck wrote:
| Anythingllm is promising
| rahimnathwani wrote:
| If your data aren't too large, you can use faiss-cpu and pickle
|
| https://pypi.org/project/faiss-cpu/
| notyourwork wrote:
| For the uneducated, how large is too large? Curious.
| itake wrote:
| FAISS runs in RAM. If your dataset can't fit into ram, FAISS
| is not the right tool.
| hahahahhaah wrote:
| Shoud it be:
|
| If the total size of your data isn't loo large...?
|
| Data being a plural gets me.
|
| You might have small datums but a lot of kilobytes!
| pousada wrote:
| Data is technically a plural but nobody uses the singular and
| it's being used as a singular term often - which is
| completely fine I think, nobody speaks Latin anyway
| DonHopkins wrote:
| The opposite of Data is Lore.
| motakuk wrote:
| LightRAG, Archestra as a UI with LightRAG mcp
| ramesh31 wrote:
| SQLite with FTS5
| nineteen999 wrote:
| A little BM25 can get you quite a way with an LLM.
| jeffchuber wrote:
| try out chroma or better yet as opus to!
| electroglyph wrote:
| simple lil setup with qdrant
| CuriouslyC wrote:
| Don't use a vector database for code, embeddings are slow and bad
| for code. Code likes bm25+trigram, that gets better results while
| keeping search responses snappy.
| lee1012 wrote:
| static embedding models im finding quite fast lee101/gobed
| https://github.com/lee101/gobed is 1ms on gpu :) would need to
| be trained for code though the bigger code llm embeddings can
| be high quality too so its just yea about where is ideal on the
| pareto fronteir really , often yea though your right it tends
| to be bm25 or rg even for code but yea more complex solutions
| are kind of possible too if its really important the search is
| high quality
| itake wrote:
| With AI needing more access to documentation, WDYT about using
| RAG for documentation retrieval?
| CuriouslyC wrote:
| IME most documentation is coming from the web via web search.
| I like agentic RAG for this case, which you can achieve
| easily with a Claude Code subagent.
| postalcoder wrote:
| I agree. Someone here posted a drop-in for grep that added the
| ability to do hybrid text/vector search but the constant need
| to re-index files was annoying and a drag. Moreover, vector
| search can add a ton of noise if the model isn't meant for code
| search and if you're not using a re-ranker.
|
| For all intents and purposes, running gpt-oss 20B in a while
| loop with access to ripgrep works pretty dang well. gpt-oss is
| a tool calling god compared to everything else i've tried, and
| _fast_.
| ehsanu1 wrote:
| I've gotten great results applying it to file paths +
| signatures. Even better if you also fuse those results with
| BM25.
| CuriouslyC wrote:
| I like embeddings for natural language documents where your
| query terms are unlikely to be unique, and overall document
| direction is a good disambiguator.
| rao-v wrote:
| Anybody know of a good service / docker that will do BM25 +
| vector lookup without spinning up half a dozen microservices?
| donkeyboy wrote:
| Elasticsearch / Opensearch is the industry standard for this
| abujazar wrote:
| Used to be, but they're very complicated to operate
| compared to more modern alternatives and have just gotten
| more and more bloated over the years. Also require a bunch
| of different applications for different parts of the stack
| in order to do the same basic stuff as e.g. Meilisearch,
| Manticore or Typesense.
| cluckindan wrote:
| >very complicated to operate compared to more modern
| alternatives
|
| Can you elaborate? What makes the modern alternatives
| easier to operate? What makes Elasticsearch complicated?
|
| Asking because in my experience, Elasticsearch is pretty
| simple to operate unless you have a huge cluster with
| nodes operating in different modes.
| abujazar wrote:
| Meilisearch
| porridgeraisin wrote:
| For BM25 + trigram, SQLite FTS5 works well.
| cipherself wrote:
| Here's a Dockerfile that will spin up postgres with pgvector
| and paradedb https://gist.github.com/cipherself/5260fea1e2631
| e9630081fb7d...
|
| You can use pgvector for the vector lookup and paradedb for
| bm25.
| jankovicsandras wrote:
| You can do hybrid search in Postgres.
|
| Shameless plug: https://github.com/jankovicsandras/plpgsql_bm25
| BM25 search implemented in PL/pgSQL ( Unlicense / Public domain
| )
|
| The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function
| for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal
| Rank Fusion; and Jupyter notebook examples.
| canadiantim wrote:
| Wow very impressive library great work!
| Der_Einzige wrote:
| This is true in general with LLMs, not just for code. LLMs can
| be told that their RAG tool is using BM25+N-grams, and will
| search accordingly. keyword search is superior to embeddings
| based search. The moment google switched to bert based
| embeddings for search everyone agreed it was going down hill.
| Most forms of early enshittification were simply switching off
| BM25 to embeddings based search.
|
| BM25/tf-idf and N grams have always been extremely difficult to
| beat baselines in information retrieval. This is why embeddings
| still have not led to a "ChatGPT" moment in information
| retrieval.
| lee1012 wrote:
| lee101/gobed https://github.com/lee101/gobed static embedding
| models so they are embedded in milliseconds and on gpu search
| with a cagra style on gpu index with a few things for speed like
| int8 quantization on the embeddings and fused embedding and
| search in the same kernel as the embedding really is just a
| trained map of embeddings per token/averaging
| pdyc wrote:
| sqlite's bm25
| init0 wrote:
| I built a lib for myself https://pypi.org/project/piragi/
| stingraycharles wrote:
| That looks great! Is there a way to store / cache the
| embeddings?
| jeanloolz wrote:
| Sqlite-vec
| petesergeant wrote:
| I've got it deployed in production for a dataset that changes
| infrequently and it works really well
| ehsanu1 wrote:
| Embedded usearch vector database. https://github.com/unum-
| cloud/USearch
| dvorka wrote:
| Any suggestion what to use as embeddings model runtime and
| semantic search in C++?
| cbcoutinho wrote:
| The Nextcloud MCP Server [0] supports Qdrant as a vectordb to
| store embeddings and provide semantic search across your personal
| documents. This enables any LLM & MCP client (e.g. claude code)
| into a RAG system that you can use to chat with your files.
|
| For local deployments, Qdrant supports storing embeddings in
| memory as well as in a local directory (similar to sqlite) - for
| larger deployments Qdrant supports running as a standalone
| service/sidecar and can be made available over the network.
|
| [0] https://github.com/cbcoutinho/nextcloud-mcp-server
| lormayna wrote:
| I have done some experiments with nomic embedding through Ollama
| and ChromaDB.
|
| Works well, but I didn't tested on larger scale
| tebeka wrote:
| https://duckdb.org/2024/05/03/vector-similarity-search-vss
| m00dy wrote:
| does duckdb scale well over large datasets for vector search ?
| lgrebe wrote:
| What order of magnitude would you define as ,,large" in this
| case?
| m00dy wrote:
| like over 1tb.
| cess11 wrote:
| Some people are using DuckDB for large datasets, https://
| duckdb.org/docs/stable/guides/performance/working_wi... ,
| but you'd probably do some testing under the specific
| conditions of your rig to figure out if it is a good
| match or not.
| riku_iki wrote:
| its clear many DuckDB sql queries can handle terabytes of
| data, but the question here was about vector search..
| jlarks32 wrote:
| +1 on this one, I've been pleasantly surprised by this for a
| small (<3GB) local project
| autogn0me wrote:
| https://github.com/ggozad/haiku.rag/ - the embedded lancedb is
| convenient and has benchmarks; uses docling. qwen3-embedding:4b,
| 2560 w/ gpt-oss:20b.
| miohtama wrote:
| +1 for Haiku! It's very simple to get up and running.
| baalimago wrote:
| I thought that context building via tooling was shown to be more
| effective than rag in practically every way?
|
| Question being: WHY would I be doing RAG locally?
| petesergeant wrote:
| For code, maybe? For documents, no, text embeddings are magical
| alien technology.
| beret4breakfast wrote:
| For the purposes of learning, I've built a chatbot using ollama,
| streamlit, chromadb and docling. Mostly playing around with
| embedding and chunking on a document library.
| sidrag22 wrote:
| i took a similar path, i spun up a discord bot, used ollama,
| pgvector, docling for random documents, and made some
| specialized chunking strategies for some clunkier json data.
| its been a little while since i messed with it, but i really
| did enjoy it when i was.
|
| it all moves so fast, i wouldnt be surprised if everything i
| made is now crazy outdated and it was probably like 2 months
| ago.
| Strift wrote:
| I just use a web server and a search engine.
|
| TL;DR: - chunk files, index chunks - vector/hybrid search over
| the index - node app to handle requests (was the quickest to
| implement, LLMs understand OpenAPI well)
|
| I wrote about it here: https://laurentcazanove.com/blog/obsidian-
| rag-api
| esperent wrote:
| I'm lucky enough to have 95% of my docs in small markdown
| markdown files so I'm just... not _(+)_. I 'm using SQLite FTS5
| (full text search) to build a normal search index and using that.
| Well, I already had the index so I just wired it up to my mastra
| agents. Each file has a short description field, so if a keyword
| search surfaces the doc they check the description and if it
| matches, load the whole doc.
|
| This took about one hour to set up and works very well.
|
| _(+) At least, I don 't think this counts as RAG. I'm honestly a
| bit hazy on the definition. But there's no vectordb anyway._
| dmos62 wrote:
| Retrieval-augmented generation. What you described is a perfect
| example of a RAG. An embedding-based search might be more
| common, but that's a detail.
| esperent wrote:
| Well, that is what the acronym stands for. But every source
| I've ever seen quickly follows by noting it's retrieval
| _backed by a vectordb_. So we 'd probably find an even split
| of people who would call this RAG or not.
| spqw wrote:
| I am surprised to see very few setups leveraging LSP support.
| (Language Server Protocol) It has been added to Claude Code last
| month. Most setups rely on naive grep.
| woggy wrote:
| I've written a few terminal tools on top of Roslyn to assist
| Claude in code analysis for C# code. Obviously the tools are
| also written with the help of Claude. Worked quite well.
| aqula wrote:
| LSP is not great for non-editor use cases. Everything is cursor
| position oriented.
| HarHarVeryFunny wrote:
| Yes, something like TreeSitter would seem to be of more value
| - able to lookup symbols by name, and find the spans of
| source code where they are defined and used.
| alchemist1e9 wrote:
| https://github.com/ast-grep/ast-grep
| WilcoKruijer wrote:
| There are actions that don't require cursor position, like
| document/workspace symbols, that could be useful.
| d4rkp4ttern wrote:
| LSP is currently broken in CC:
|
| https://github.com/anthropics/claude-code/issues/15168
| geuis wrote:
| I don't. I actually write code.
|
| To answer the question more directly, I've spent the last couple
| of years with a few different quant models mostly running on
| llama.cpp and ollama, depending. The results are way slower than
| the paid token api versions, but they are completely free of
| external influence and cost.
|
| However the models I've tests generally turn out to be pretty
| dumb at the quant level I'm running to be relatively fast. And
| their code generation capabilities are just a mess not to be
| dealt with.
| softwaredoug wrote:
| I built a Pandas extension SearchArray, I just use that (plus in
| memory embeddings) for any toy thing
|
| https://github.com/softwaredoug/searcharray
| Bombthecat wrote:
| AnythingLLM for documents, amazing tool!
| lsb wrote:
| I'm using Sonnet with 1M Context Window at work, just stuffing
| everything in a window (it works fine for now), and I'm hoping to
| investigate Recursive Language Models with DSPy when I'm using
| local models with Ollama
| bzGoRust wrote:
| In my company, we build the internal chatbot based on RAG through
| LangChain + Milvus + LLM. Since the documents are well formatted,
| it is easy to do the overlapping chunking, then all those
| chunking data are inserted into vector db Milvus. The hybrid
| search (combine dense search and sparse search) is native
| supported in the Milvus could help us to do better retrieve. Thus
| the better quality answers are there.
| cluckindan wrote:
| Hybrid search usually refers to traditional keyword search
| (BM25, TF-IDF) combined with a vector similarity search.
| __jf__ wrote:
| For vector generation I started using Meta-LLama-3-8B in april
| 2024 with Python and Transformers for each text chunk on an
| RTX-A6000. Wow that thing was fast but noisy and also burns 500W.
| So a year ago I switched to an M1 Ultra and only had to replace
| Transformers with Apple's MLX python library. Approximately the
| same speed but less heat and noise. The Llama model has 4k
| dimensions so at fp16 thats 8 kilobyte per chunk, which I store
| in a BLOB column in SQLite via numpy.save(). Between running on
| the RTX and M1 there is a very small difference in vector output
| but not enough for me to change retrieval results, regenerate the
| vectors or change to another LLM.
|
| For retrieval I load all the vectors from the SQlite database
| into a numpy.array and hand it to FAISS. Faiss-gpu was
| impressively fast on the RTX6000 and faiss-cpu is slower on the
| M1 Ultra but still fast enough for my purposes (I'm firing a few
| queries per day, not per minute). For 5 million chunks memory
| usage is around 40 GB which both fit into the A6000 and easily
| fits into the 128GB of the M1 Ultra. It works, I'm happy.
| sinandrei wrote:
| Anyone use these approaches with academic pdfs?
| urschrei wrote:
| Another approach is to teach Claude Code how to use your Zotero
| library's full-text search:
| https://github.com/urschrei/zotero_search_skill.
| amelius wrote:
| Anyone using them for electronics datasheets?
| bradfa wrote:
| I would like to. I haven't yet found a solution that works
| well.
|
| The problems with datasheets is tables which span multiple
| pages, embedded images for diagrams and plots, they're
| generally PDFs, and only sometimes are they 2-column layout.
|
| Converting from PDF to markdown while retaining tables
| correctly seems to work well for me with Mistral's latest OCR
| model, but this isn't an open model. Using docling with
| different models has produced much worse results.
| sosojustdo wrote:
| I've been working on a tool specifically to handle these
| messy PDF-to-Markdown conversions because I ran into the
| same issues with tables and multi-column layouts.
|
| I've optimized https://markdownconverter.pro/pdf-to-
| markdown to handle complex PDFs, including those tricky
| tables that span multiple pages and 2-column formats that
| usually trip up tools like Docling. It also extracts
| embedded diagrams/images and links them properly in the
| output.
|
| Full disclosure: I'm the developer behind it. I'd love to
| see if it handles your specific datasheets better than the
| models you've tried. Feel free to give it a spin!
| bradfa wrote:
| Cool! But given that often electronics documentation is
| covered by NDAs, my preferred solution is local-first if
| at all possible.
| alansaber wrote:
| I've not seen any impressive products. But products do exist ie
| https://scibite.com/solutions/semantic-search/
| podgietaru wrote:
| I made a small RAG database just using Postgres. I outlined it in
| the blog post below. I use it for RSS Feed organisation, and
| searching. They are small blobs. I do the labeling using a
| pseudo-KNN algorithm.
|
| https://aws.amazon.com/blogs/machine-learning/use-language-e...
|
| The code for it is here: https://github.com/aws-samples/rss-
| aggregator-using-cohere-e...
|
| The example link no longer works, as I no longer work at AWS.
| acutesoftware wrote:
| I am using LangChain with a SQLite database - it works pretty
| well on a 16G GPU, but I started running it on a crappy NUC,
| which also worked with lesser results.
|
| The real lightbulb moment is when you realise the ONLY thing a
| RAG passes to the LLM is a short string of search results with
| small chunks of text. This changes it from 'magic' to 'ahh, ok -
| I need better search results'. With small models you cannot pass
| a lot of search results ( TOP_K=5 is probably the limit ),
| otherwise the small models 'forget context'.
|
| It is fun trying to get decent results - and it is a rabbithole,
| next step I am going into is pre-summarising files and folders.
|
| I open sourced the code I was using -
| https://github.com/acutesoftware/lifepim-ai-core
| reactordev wrote:
| You can expand your context window to something like 100,000 to
| prevent memory loss.
| SamLeBarbare wrote:
| sqlite + FTS + sqlite-vec + local LLM for reranking results
| (reasoning model)
| yandrypozo wrote:
| this's pretty cool, which LLM are you using currently?
| robotswantdata wrote:
| You don't need a vector database or graph, it really depends on
| your existing infrastructure , file types and needs.
|
| The newer "agent" search approach can just query a file system or
| api. It's slightly slower but easier to setup and maintain as no
| extra infrastructure.
| beklein wrote:
| Most of my complex documents are, luckily, Markdown files.
|
| I can recommend https://github.com/tobi/qmd/ . It's a simple CLI
| tool for searching in these kinds of files. My previous workflow
| was based on fzf, but this tool gives better results and enables
| even more fuzzy queries. I don't use it for code, though.
| Aachen wrote:
| Given that preface, I was really expecting that link to be a
| grepping tool rewritten in golang or something, or perhaps
| customised for markdown to weigh matches in "# heading title"s
| heavier for example
| whacked_new wrote:
| Here's a rust one: https://github.com/BeaconBay/ck
|
| I haven't used it extensively, but semantic grep alone was
| kind of worth it.
| Aachen wrote:
| Right, I should have said Rust. Golang is so 2017!
| codebolt wrote:
| Giving the LLM tools with an OData query interface has worked
| well for me. In C# it's pretty trivial to set up an MCP server
| with OData querying for an arbitrary data model. At work we have
| an Excel sheet with 40k rows which the LLM was able to quickly
| and reliably analyse using this method.
| lmeyerov wrote:
| Claude code / codex which internally uses ripgrep, and I'm unsure
| if it's using parallel mode. And, project specific static
| analyzers.
|
| Studies generally show when you do agentic retrieval w/ text
| search, that's pretty good. Adding vector retrieval and graph
| rag, so the typical parallel multi-retrieval followed by
| reranking, gives a bit of speedup and quality lift. That lines up
| with my local flow experience, where it is only enough that I
| want that for $$$$ consumer/prosumer tools, and not easy enough
| for DIY that I want to invest in that locally. For those who
| struggle with tools like spotlight running when it shouldn't,
| that kind of thing turns me off on the cost/benefit side.
|
| For code, I experiment with unsound tools (semgrep, ...) vs sound
| flow analyzers, carefully setup for the project. Basically, ai
| coders love to use grep/sed for global replace refactors and
| other global needs, but keeps tripped up on sound flow analysis.
| Similar to lint and type checking, that needs to be setup for a
| project and taught as a skill. I'm not happy with any of my
| experiments here yet however :(
| mmargenot wrote:
| Cursor uses a vector index, some details here:
| https://cursor.com/docs/context/semantic-search
| lmeyerov wrote:
| Thanks!
|
| Their discussion is super relevant to exactly what I wrote --
|
| * They note speed benefits * The quality benefit they note is
| synonym search... which agentic text search can do: Agents
| can guess synonyms in the first shot for you, eg,
| `navigation` -> `nav|header|footer`, and they'll be iterating
| anyways
|
| To truly do better, and not make the infra experience stink,
| it's real work. We do it on our product (louie.ai) and our
| service engagements, but real costs/benefits.
| oliveiracwb wrote:
| We handle ~300k customer interactions per day, so latency and
| precision really matter. We built an internal RAG-based portal on
| top of our knowledge base (basically a much better FAQ).
|
| On the retrieval side, I built a custom search/indexing layer
| (Node) specifically for service traceability and discovery. It
| uses a hybrid approach -- embeddings + full-text search + IVF-
| HNSW -- to index and cross-reference our APIs, services, proxies
| and orchestration repos. The RAG pipelines sit on top of this
| layer, which gives us reasonable recall and predictable latency.
|
| Compliance and observability are still a problem. Every year new
| vendors show up promising audits, data lineage and observability,
| but none of them really handle the informational sprawl of ~600
| distributed systems. The entropy keeps increasing.
|
| Lately I've been experimenting with a more semantic/logical KAG
| approach on top of knowledge graphs to map business rules
| scattered across those systems. The goal is to answer higher-
| level questions about how things actually work -- Palantir-like
| outcomes, but with explicit logic instead of magic.
|
| Curious if others are moving beyond "pure RAG" toward graph-based
| or hybrid reasoning setups.
| jackfranklyn wrote:
| For document processing in a side project, I've been using a
| local all-MiniLM model with FAISS. Works well enough for semantic
| matching against ~50k transaction descriptions.
|
| The real challenge wasn't model quality - it was the chunking
| strategy. Financial data is weirdly structured and breaking it
| into sensible chunks that preserve context took more iteration
| than expected. Eventually settled on treating each complete
| record as a chunk rather than doing sliding windows over raw
| text. The "obvious" approaches from tutorials didn't work well at
| all for structured tabular-ish data.
| pj4533 wrote:
| Well this isn't code, but I've been working on a memory system
| for Claude Code. This portion provides semantic search over the
| session files in .claude/projects. It uses OpenAI for embeddings
| so not completely local (would be easy to modify) and storage in
| ChromaDB.
|
| https://github.com/pj4533/seance
| reactordev wrote:
| I have three tools dedicated to this.
|
| save_memory, recall_memory, search
|
| Save memory vectorizes a session, summarizes it, and stores it in
| SQLite. Recall memory takes vector or a previous tool run id and
| loads the full text output. Search takes a vector array or string
| array and searches through the graph using fuzzy matching and
| vector dot products.
|
| It's not fancy, but it works really well. gpt-oss
| yakkomajuri wrote:
| I've written about this (and the post was even here on HN) but
| mostly from the perspective of running a RAG on your infra as an
| organization. But I cover the general components and alternatives
| to Cloud services.
|
| Not sure how useful it is for what you need specifically:
| https://blog.yakkomajuri.com/blog/local-rag
| prakashn27 wrote:
| I feel local rag system , slows down my computer (I got M1 Pro 32
| GB)
|
| So I use hosted one to prevent this. My business use vector db,
| so created a new db to vectorize and host my knowledge base. 1.
| All my knowledge base is markdown files. So I split that by
| header tags. 2. The split is hashed and hash value is stored in
| SQLite 3. The hashed version is vectorized and pushed to cloud
| db. 4. When ever I make changes , I run a script which splits and
| checks hash, if it is changed the. I upsert the document. If not
| I don't do anything. This helps me keep the store up to date
|
| For search I have a cli query which searches and fetches from
| vector store.
| metawake wrote:
| I am using a vector DB using Docker image. And for debugging and
| benchmarking local RAG retrieval, I've been building a CLI tool
| that shows what's actually being retrieved:
| ragtune explain "your query" --collection prod
|
| Shows scores, sources, and diagnostics. Helps catch when your
| chunking or embeddings are silently failing or you need numeric
| estimations to base your judgements on.
|
| Open source: https://github.com/metawake/ragtune
| yokuze wrote:
| I made, and use this: https://github.com/libragen/libragen
|
| It's a CLI tool and MCP server for creating discrete, versioned
| "libraries" of RAG-able content.
|
| Under the hood, it uses an embedding model locally. It chunks
| your content and stores embeddings in SQLite. The search
| functionality uses vector + keyword search + a re-ranking model.
|
| You can also point it at any GitHub repo and it will create a RAG
| DB out of it.
|
| You can also use the MCP server to create and query the
| libraries.
|
| Site: https://www.libragen.dev/
| bradfa wrote:
| Your README references a file named LICENSE which doesn't seem
| to exist on the main branch.
| navar wrote:
| For the retrieval stage, we have developed a highly efficient,
| CPU-only-friendly text embedding model:
|
| https://huggingface.co/MongoDB/mdbr-leaf-ir
|
| It ranks #1 on a bunch of leaderboards for models of its size. It
| can be used interchangeably with the model it has been distilled
| from (https://huggingface.co/Snowflake/snowflake-arctic-embed-
| m-v1...).
|
| You can see an example comparing semantic (i.e., embeddings-
| based) search vs bm25 vs hybrid here: http://search-
| sensei.s3-website-us-east-1.amazonaws.com (warning! It will
| download ~50MB of data for the model weights and onnx runtime on
| first load, but should otherwise run smoothly even on a phone)
|
| This mini app illustrates the advantage of semantic vs bm25
| search. For instance, embedding models "know" that j lo refers to
| jennifer lopez.
|
| We have also published the recipe to train this type of models if
| you were interested in doing so; we show that it can be done on
| relatively modest hardware and training data is very easy to
| obtain: https://arxiv.org/abs/2509.12539
| jasonjmcghee wrote:
| How does performance (embedding speed and recall) compare to
| minish / model2vec static word embeddings?
| navar wrote:
| I interacted with the authors of these models quite a bit!
|
| These are very interesting models.
|
| The tradeoff here is that you get even faster inference, but
| lose on retrieval accuracy [0].
|
| Specifically, inference will be faster because essentially
| you are only doing tokenization + a lookup table + an
| average. So despite the fact that their largest model is 32M
| params, you can expect inference speeds to be higher than
| ours, which 23M params but it is transformer-based.
|
| I am not sure about typical inference speeds on a CPU for
| their models, but with ours you can expect to do ~22 docs per
| second, and ~120 queries per second on a standard 2vCPU
| server.
|
| As far as retrieval accuracy goes, on BEIR we score 53.55,
| all-MiniLM-L12-v2 (a widely adopted compact text embedding
| model) scores 42.69, while potion-8M scores 30.43.
|
| I can't find their larger models but you can generally get an
| idea of the power level of different embedding models here:
| https://huggingface.co/spaces/mteb/leaderboard
|
| If you want to run them on a CPU it may make sense to filter
| for smaller models (e.g., <100M params). On the other side
| our models achieve higher retrieval scores.
|
| [0] "accuracy" in layman terms, not in accuracy vs recall
| terms. The correct word here would be "effectiveness".
| rcarmo wrote:
| Hmmm. I recently created https://github.com/rcarmo/asterisk-
| embedding-model, need to look at this since I had very limited
| training resources.
| jacekm wrote:
| I am curious what are you using local RAG for?
| scosman wrote:
| Kiln wraps up all the parts in on app. Just drag and drop in
| files. You can easily compare different configs on your dataset:
| extraction methods, embedding model, search method (BM25, hybrid,
| vector), etc.
|
| It uses LanceDB and has dozens of different extraction/embedding
| models to choose from. It even has evals for checking retrieval
| accuracy, including automatically generating the eval dataset.
|
| You can use its UI, or call the RAG via MCP.
|
| https://github.com/kiln-ai/kiln
|
| https://docs.kiln.tech/docs/documents-and-search-rag
| gaganyatri wrote:
| Built discovery using - Qwen-3-VL-8B for Document Ocr + Prompts +
| Tool Call - ChromaDB for Vector storage. - BM25 + Embedding model
| for Hybrid RAG. - Backend- FastAPI + Python - Frontend- React +
| Typescript - vllm + docker for model deployment on L40 GPU
|
| Demo: https://app.dwani.ai
|
| GitHub: https://github.com/dwani-ai/discovery
|
| Now working on added Agentic features, by continuous analysis of
| Document with Generated prompts.
| juleshenry wrote:
| SurrealDB coupled with local vectorization. Mac M1 16GB
| eb0la wrote:
| We started with PGVector just because we already knew Postgres
| and it was easy to hand over to the operations people.
|
| After some time we noticed a semi-structured field in the prompt
| had a 100% match with the content needed to process the prompt.
|
| Turns out operators started puting tags both in the input and the
| documents that needed to match on every use case (not much, about
| 50 docs).
|
| Now we look for the field first and put the corresponding file in
| the prompt, then we look for matches in the database using the
| embedding.
|
| 85% of the time we don't need the vectordb.
| alansaber wrote:
| Most vectordb is a hammer looking for a nail
| folli wrote:
| I think it can be more efficient for two-step RAG so you can
| reuse the natural language query directly, but for agentic
| RAG it might indeed be overkill.
| juanre wrote:
| I built https://github.com/juanre/llmemory and I use it both
| locally and as part of company apps. Quite happy with the
| performance.
|
| It uses PostgreSQL with pgvector, hybrid BM25, multi-query
| expansion, and reranking.
|
| (It's the first time I share it publicly, so I am sure there'll
| be quirks.)
| raghavankl wrote:
| I have a python tooling to do indexing and relevance offline
| using ollama.
|
| https://github.com/raghavan/pdfgptindexer-offline
| claylyons wrote:
| Has anyone tried this?
| https://aws.amazon.com/s3/features/vectors/
| theahura wrote:
| SQLite works shockingly well. The agents know how to write good
| queries, know how to chain queries, and can generally manipulate
| the DB however they need. At nori (https://usenori.ai/watchtower)
| we use SQLite + vec0 + fts5 for semantic and word search
| init0 wrote:
| from piragi import Ragi
|
| kb = Ragi(["./docs", "s3://bucket/data/*/*.pdf",
| "https://api.example.com/docs"])
|
| answer = kb.ask("How do I deploy this?")
|
| that's it! with https://pypi.org/project/piragi/
| mooball wrote:
| i thought rag/embeddings were dead with the large context
| windows. thats what i get for listening to chatgpt.
| amscotti wrote:
| More of a proof of concept to test out ideas, but here's my
| approach for local RAG, https://github.com/amscotti/local-LLM-
| with-RAG
|
| Using Ollama for the embeddings with "nomic-embed-text", with
| LanceDB for the vector database. Recently updated it to use
| "agentic" RAG, but probably not fully needed for a small project.
| someguyiguess wrote:
| Woah. I am doing something very similar also using lancedb
| https://github.com/nicholaspsmith/lance-context
|
| Mine is much more basic than yours and I just started it a
| couple of weeks ago.
| vaylian wrote:
| Thank you for being the kind of person who explains what the
| abbreviation RAG stands for. I have been very confused reading
| this thread.
| someguyiguess wrote:
| I feel this pain! It feels like in the world of LLMs there is
| a new acronym to learn every day!
|
| For the curious RAG = Retrieval Augmented Generation. From
| wikipedia: RAG enables large language models (LLMs) to
| retrieve and incorporate new information from external data
| sources
| __mharrison__ wrote:
| Grep (rg)
| turnsout wrote:
| The Claude Code model highlights the power of simple search
| (grep) and selective reads (only reading in excerpts). The only
| time I vectorize is when I explicitly want to similarity-based
| searching, but that's actually pretty rare.
| marwamc wrote:
| BM25 has been sufficient for my needs. I typically need to refer
| to codebases of existing tools as referential sources (istio,
| envoy, oauth2-proxy, tantivy index etc) so I just clone those
| repos, index them and search away. Built a cli and mcp tool for
| this workflow.
|
| https://github.com/rhobimd-oss/shebe
|
| One area where BM25 particularly shines is the refactoring
| workflow: let's say you want to upgrade your istio installation
| from 1.28 to 1.29 and maybe in 1.29 the authorizationpolicy crd
| has a breaking change in one of it's properties. BM25 allows you
| to efficiently enumerate all code locations in your codebase that
| need to change and then you can set the cli coders off using this
| list. Grep and LSP can still perform this enumeration but they
| have shortcomings. Wrote about it here
| https://github.com/rhobimd-oss/shebe/blob/main/WHY_SHEBE.md#...
| tubs wrote:
| The download links for binaries 404 for me.
| marwamc wrote:
| Will fix the links. Meanwhile here is the releases page. I
| develop on gitlab and mirror to github. Need to make that
| clear as well.
|
| https://gitlab.com/rhobimd-oss/shebe/-/releases
| tubs wrote:
| Ah, I tried the gitlab and the tarballs 404 for me there,
| sorry I should have been more specific in the original
| post!
|
| fwiw this does look interesting.
| marwamc wrote:
| I see what's happening. I never validated those build
| artifacts... Thanks for the catch. Will rebuild notify
| you here.
| tschellenbach wrote:
| Vector & BM25 on Turbopuffer. (see
| https://github.com/GetStream/Vision-Agents/blob/main/plugins...)
| andoando wrote:
| Anyone have suggestions for doing semantic caching?
| philip1209 wrote:
| I run a Mac Mini home datacenter [1]. I've been using Chroma,
| Qwen 0.6B embeddings, and gpt-oss-20b to build a search agent
| over my blog.
|
| [1]: https://www.contraption.co/a-mini-data-center/
| yandrypozo wrote:
| Is there a thread for hardware used for local LLMs?
| mmargenot wrote:
| I made an obsidian extension that does semantic and hybrid (RRF
| with FTS) search with local models. I have done some knowledge
| graph and ontology experimentation around this, but nothing that
| I'd like to include yet.
|
| This is specifically a "remembrance agent", so it surfaces
| related atoms to what you're writing rather than doing anything
| generative.
|
| Extension: https://github.com/mmargenot/tezcat
|
| Also available in community plugins.
| g0wda wrote:
| Store fp16 vector blobs in sqlite. Load the vectors after filter
| queries into memory and do a matvec multiplication for similarity
| scores (this part will be fast if the library (e.g. numpy/torch)
| uses multithreading/blas/GPU). I will migrate this to the very
| based https://github.com/sqliteai/sqlite-vector when it starts to
| become a bottleneck. In my case the filters by other features
| (e.g. date, location) just subset a lot. All this is behind some
| interface that will allow me to switch out the backend.
| throwaway7783 wrote:
| We have a Q&A database. The questions, answers are both trigram
| indexed and also have embeddings. All in postgres. We then use
| pgvector + trigram search, combine them by relevance scores.
| folli wrote:
| I was just working on a RAG implementation for >500k news
| articles, completely local, using postgres as a vector database:
| https://github.com/r-follador/TeletextSignals
|
| I'm positively surprised on how well it works, especially if you
| also connect it to an LLM.
___________________________________________________________________
(page generated 2026-01-15 23:01 UTC)