[HN Gopher] txtai: Open-source vector search and RAG for minimal...
___________________________________________________________________
txtai: Open-source vector search and RAG for minimalists
Author : dmezzetti
Score : 89 points
Date : 2024-07-21 11:39 UTC (11 hours ago)
(HTM) web link (neuml.github.io)
(TXT) w3m dump (neuml.github.io)
| dmezzetti wrote:
| Link to source (Apache 2.0): https://github.com/neuml/txtai
| freeqaz wrote:
| This looks interesting. I've been wanting to build some tools to
| help feed text documents into Stable Diffusion and this looks
| like it could be helpful. Are there any other libs people are
| aware of that they'd recommend in this space?
| janice1999 wrote:
| It's frustrating when developers of ML projects don't state even
| the most basic requirements. Do I need an Nvidia 4090 or a
| cluster of H100s to run this?
| dmezzetti wrote:
| A RTX 3090 is more than enough for 7B LLMs. With 4-bit
| quantization, you can run inference with an even larger LLM
| using a 24GB GPU.
|
| If you're using remote API services, you might be able to just
| use a CPU.
| malux85 wrote:
| It's frustrating when people ask for hardware requirements
| without stating what they are trying to do, do you have
| 100,000,000 books to index or do you have 5 articles? What are
| the context lengths you need? What about latency?
|
| How can someone tell you what hardware you need when you give
| literally no information about what you're trying to do?
| pjot wrote:
| I've done something similar, but using duckDB as the
| backend/vector store. You can use embeddings from wherever. My
| demo uses OpenAI.
|
| https://github.com/patricktrainer/duckdb-embedding-search
| dmezzetti wrote:
| Hello, author of txtai here. txtai was created back in 2020
| starting with semantic search of medical literature. It has since
| grown into a framework for vector search, retrieval augmented
| generation (RAG) and large language model (LLM)
| orchestration/workflows.
|
| The goal of txtai is to be simple, performant, innovative and
| easy-to-use. It had vector search before many current projects
| existed. Semantic Graphs were added in 2022 before the Generative
| AI wave of 2023/2024. GraphRAG is a hot topic but txtai had
| examples of using graphs to build search contexts back in
| 2022/2023.
|
| There is a commitment to quality and performance, especially with
| local models. For example, it's vector embeddings component
| streams vectors to disk during indexing and uses mmaped arrays to
| enable indexing large datasets locally on a single node. txtai's
| BM25 component is built from the scratch to work efficiently in
| Python leading to 6x better memory utilization and faster search
| performance than the BM25 Python library most commonly used.
|
| I often see others complain about AI/LLM/RAG frameworks, so I
| wanted to share this project as many don't know it exists.
|
| Link to source (Apache 2.0): https://github.com/neuml/txtai
| fastneutron wrote:
| I've been building a RAG mini app with txtai these past few weeks
| and it's been pretty smooth. I'm between this and llamaindex as
| the backend for a larger app I want to build for a small-to-
| midsize customer.
|
| With the (potentially) obvious bias towards your own framework,
| are there situations in which you would _not_ recommend it for a
| particular application?
| dmezzetti wrote:
| Glad to hear txtai is on your list.
|
| I recently wrote an article (https://medium.com/neuml/vector-
| search-rag-landscape-a-revie...) comparing txtai with other
| popular frameworks. I was expecting to find some really
| interesting and innovative things in the others. But from my
| perspective I was underwhelmed.
|
| I'm a big fan of simplicity and none of them are following that
| strategy. Agentic workflows seem like a big fancy term but I
| don't see the value currently. Things are hard enough as it is.
|
| If your team is already using another framework, I'm sure
| anything can work. Some of the other projects are VC-backed
| with larger teams. In some cases, that may be important.
| ipsi wrote:
| So here's something I've been wanting to do for a while, but have
| kinda been struggling to figure out _how_ to do it. txtai looks
| like it has all the tools necessary to do the job, I'm just not
| sure which tool(s), and how I'd use them.
|
| Basically, I'd like to be able to take PDFs of, say, D&D books,
| extract that data (this step is, at least, something I can
| already do), and load it into an LLM to be able to ask questions
| like:
|
| * What does the feat "Sentinel" do?
|
| * Who is Elminster?
|
| * Which God(s) do Elves worship in Faerun?
|
| * Where I can I find the spell "Crusader's Mantle"?
|
| And so on. Given this data is all under copyright, I'd probably
| have to stick to using a local LLM to avoid problems. And, while
| I wouldn't expect it to have good answers to all (or possibly
| any!) of those questions, I'd nevertheless love to be able to
| give it a try.
|
| I'm just not sure where to start - I think I'd want to fine-tune
| an existing model since this is all natural language content, but
| I get a bit lost after that. Do I need to pre-process the content
| to add extra information that I can't fetch relatively
| automatically. e.g., page numbers are simple to add in, but would
| I need to mark out things like chapter/section headings, or in-
| character vs out-of-character text? Do I need to add all the
| content in as a series of questions and answers, like "What
| information is on page 52 of the Player's Handbook? => <text of
| page>"?
| dmezzetti wrote:
| Based on what you're looking to do, it sounds like Retrieval
| Augmented Generation (RAG) should help. This article has an
| example on how to do that with txtai:
| https://neuml.hashnode.dev/build-rag-pipelines-with-txtai
|
| RAG sounds sophisticated but it's actually quite simple. For
| each question, a database (vector database, keyword, relational
| etc) is first searched. The top n results are then inserted
| into a prompt and that is what is run with the LLM.
|
| Before fine-tuning, I'd try that out first. I'm planning to
| have another example notebook out soon building on this.
| ipsi wrote:
| Ah, that's very helpful, thanks! I'll have a dig into this at
| some point relatively soon.
|
| An example of how I might provide references with page
| numbers or chapter names would be great (even if this means a
| more complex text-extraction pipeline). As would examples
| showing anything I can do to indicate differences that are
| obvious to _me_ but that an LLM would be unlikely to pick up,
| such as the previously mentioned in-character vs out-of-
| character distinction. This is mostly relevant for asking
| questions about the setting, where in-character information
| might be suspect ( "unreliable narrator"), while out-of-
| character information is generally fully accurate.
|
| Tangentially, is this something that I could reasonably
| experiment with without a GPU? While I do have a 4090, it's
| in my Windows gaming machine, which isn't really set up for
| AI/LLM/etc development.
| dmezzetti wrote:
| Will do, I'll have the new notebooks published within the
| next couple weeks.
|
| In terms of a no GPU setup, yes it's possible but it will
| be slow. As long as you're OK with slow response times,
| then it will eventually come back with answers.
| simonw wrote:
| Fine-tuning is almost certainly the wrong way to go about this.
| It's not a good way of adding small amounts of new knowledge to
| a model because the existing knowledge tends to overwhelm
| anything you attempt to add in the fine-tuning steps.
|
| Look into different RAG and tool usage mechanisms instead. You
| might even be able to get good results from dumping large
| amounts of information into a long context model like Gemini
| Flash.
| brrrrrm wrote:
| First I would calculate the number of tokens you actually need.
| If its less than 32k there are plenty of ways to pull this off
| without RAG. If more (millions), you should understand RAG is
| an approximation technique and results may not be as high
| quality. If wayyyy more (billions), you might actually want to
| finetune
| fancy_pantser wrote:
| No fine-tuning is necessary. You can use something reasonably
| good at RAG that's small enough to run locally like the
| Command-R model run by Ollama and a small embedding model like
| Nomic. There are dozens of simple interfaces that will let you
| import files to create a RAG knowledgebase to interact with as
| you describe, AnythingLLM is a popular one. Just point it at
| your locally-running LLM or tell them to download one using the
| interface. Behind the scenes they store everything in LanceDB
| or similar and perform the searching for you when you submit a
| prompt in the simple chat interface.
___________________________________________________________________
(page generated 2024-07-21 23:02 UTC)