[HN Gopher] What Is Retrieval-Augmented Generation a.k.a. RAG?
___________________________________________________________________
What Is Retrieval-Augmented Generation a.k.a. RAG?
Author : teleforce
Score : 62 points
Date : 2023-12-01 19:42 UTC (3 hours ago)
(HTM) web link (blogs.nvidia.com)
(TXT) w3m dump (blogs.nvidia.com)
| john_cogs wrote:
| In every presentation I saw at re:Invent that mentioned AI, RAG
| was also mentioned.
|
| It seems to me that with AI, as models and products achieve
| parity, the amount of data that is accessible to a provider will
| be the key differentiator in quality of responses. Those who can
| gain access to the most customer data will be best-positioned to
| win the AI market.
| binarymax wrote:
| Because AWS trying to upsell Bedrock and Opensearch instances.
| minimaxir wrote:
| RAG is the one paradigm of modern AI that's completely
| uncontroversial (hallucinations aside) and will persist even if
| there's an AI-industry crash.
|
| The more annoying part of RAG is that it's so effective it's
| creating a lot of confusing best practices and a ton of venture
| capital around tooling, notably vector stores (which IMO are
| insufficiently differentiated) and libraries to integrate a RAG
| flow (LangChain being the common painful example, but RAG is
| simple enough that it doesn't even need its own abstraction).
| deegles wrote:
| Using RAG feels like asking an acquaintance to write a book
| report by giving them semi-randomly cut out paragraphs from the
| book.
|
| Finetuning seems to be a solution, but there is still the issue
| of the LLM not really "learning" facts.
|
| What's the latest on actually adding knowledge to an LLM?
| filterfiber wrote:
| The most likely "real solution" is going to be using various
| tricks and finetuning on higher context lengths to just extend
| the context window. However I will say knowledge graphs are
| becoming more popular.
|
| I made a comment the other day with a list of some of the
| popular methods:
|
| https://news.ycombinator.com/item?id=38476596
|
| I completely forgot to mention ROME in my last comment, where
| you can modify facts within an LLM
| https://arxiv.org/pdf/2202.05262.pdf
|
| Note that there could be something more recent then these that
| I missed, but as far as I know knowledge-graphs/RAG are what
| most people are currently using, but there's a lot of work
| being focused on extending the context window.
| kristjansson wrote:
| This doesn't deserve the downvotes it (currently) has. This is
| a completely accurate statement, simple RAG works well when
| questions are highly correlated with specifics chunks of
| documents. It does not allow an LLM to synthesize an entire
| corpus to an answer (e.g. a book report).
|
| Other modes of RAG that retrieve multiple times per generation
| (e.g. every so-many tokens) can improve performance, and
| intuitively feel closer to the 'sparse attention over the
| entire corpus' that RAG is approximating.
|
| However, fine-tuning on relevant, high quality, knowledge-rich
| question/answer pairs seems dominant, when such examples are
| available or can be generated.
| filterfiber wrote:
| I have to agree, RAG is effectively just compensating for
| data not fitting in the context window, and it has
| (potentially severe) limitations.
|
| Summarizing a book I think is a great example, RAG prevents
| LLMs from correlating all of the data.
|
| It will likely never be obsolete, but a larger context would
| be far preferable in nearly all scenerios.
| svcrunch wrote:
| I think your comment is accurate, but regarding your last
| point:
|
| "However, fine-tuning on relevant, high quality, knowledge-
| rich question/answer pairs seems dominant, when such examples
| are available or can be generated."
|
| How does one solve the problem of access-controlled data, if
| not through RAG? Do you imagine a separate version of the LLM
| for every user, reflecting their unique permissions on the
| data?
|
| Also, in scenarios where the data is being updated regularly,
| RAG provides much lower latency to the new information.
| Deletes also present a challenge for a pure-LLM approach.
| eskibars wrote:
| Having worked on a (production, as-a-service RAG system), what
| we've learned is that fine tuning is not good for really
| adding/removing facts but is great for changing the _form_ of
| the output. Anyscale has a great blog on this:
| https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa...
| (I work at Vectara, not Anyscale, but appreciate the details
| and thought they put into this blog)
|
| Fine-tuning can help with certain areas of knowledge
| acquisition, but it's costly and frankly doesn't work that well
| when the knowledge you're trying to provide goes "against" the
| data that was trained into the core system. e.g. try to fine
| tune a model that there is a cure to <disease X> if you're a
| research org and it's incredibly hard to do that because the
| base model may be so convinced otherwise.
|
| Probably the most difficult thing though is using fine-tuning
| to get a model to "forget" or "not respond to" something that
| it shouldn't take a stance on. I always ask a fine-tuned model:
| "how do you calculate the fourth interior angle of a triangle."
| This is obviously a nonsense question, but helps to show how
| most LLMs will happily tell you to sum the interior angles of
| your first 3 angles and then subtract from 180. A well-tuned
| RAG system will say "sorry, I don't have information for that."
| It shows how it's less "guess-y" and hallucinated
| dmezzetti wrote:
| Nice article that covers the overall idea behind RAG. RAG is one
| of the most practical and production-ready ideas out of
| Generative AI.
|
| If you want a more hands on approach, txtai has a couple articles
| (disclaimer I'm the author of txtai).
|
| https://neuml.hashnode.dev/build-rag-pipelines-with-txtai
|
| https://medium.com/neuml/judge-your-resume-with-ai-4223a2803...
| bloopernova wrote:
| Huggingface is blocked at work. If anyone else is in the same
| boat, here's an archive link:
| http://web.archive.org/web/20231122230456/https://huggingfac...
| maxlamb wrote:
| Do you know why it's blocked? Seems weird to me
| bloopernova wrote:
| The policy is "no external genAI access", meaning no chatGPT,
| Bard, etc etc. Which, I think, has (mis)translated to "ban
| all AI-related sites" at the corporate content-filtering
| firewall which uses 3rd party rulesets.
|
| Though I can appreciate the tightrope that corporate IT has
| to walk. Many IT security departments been given that kind of
| directive by a leadership absolutely _petrified_ of losing
| intellectual property.
|
| Another corp that several friends work at has blanket-banned
| a bunch of top-level domains, and so far their security ops
| folks haven't even responded to their complaints/requests.
| I'm very thankful I'm not affected by _that_ kind of
| filtering; .dev and I think .io are affected.
| politelemon wrote:
| Where can I learn about effectively using Vector stores? I
| understand RAG itself, been through a few tutorials and the
| workflow. But I mean using a Vector store isn't just a matter of
| generate some embeddings, slap it in a storage, and then start
| doing searches like cosine similarity, hoping for the best.
|
| Are there any guides on effectively using vector stores, and
| effectively searching for the use case you're dealing with? I
| don't even know what the various search types mean, just that
| everyone seems to use cosine similarity.
|
| Hope that's making sense... my question is analogous to learning
| to become familiar with SQL and indexes and CTEs, while it's a
| common thing to implement a three-tier application talking to a
| database.
|
| I feel that RAG might become a somewhat common feature of a
| team's tech stack alongside operational datastores (eg Postgres,
| MSSQL), because it is a low overhead to getting started and it
| persists, so it's cheaper. I just don't want to go at it blindly.
| dmezzetti wrote:
| Couple articles that are txtai-specific but general enough in
| terms of vector search.
|
| Getting started with semantic search:
| https://medium.com/neuml/getting-started-with-semantic-searc...
|
| txtai intro: https://medium.com/neuml/introducing-txtai-the-
| all-in-one-em...
|
| Examples: https://neuml.github.io/txtai/examples/
| minimaxir wrote:
| > But I mean using a Vector store isn't just a matter of
| generate some embeddings, slap it in a storage, and then start
| doing searches like cosine similarity, hoping for the best.
|
| That _is_ 90% of the work with RAG. RAG requires a surprising
| amount of QA to ensure "the best" is sufficient for your use
| case.
|
| For clarification, cosine similarity is a metric not a search
| type, and people like it since a) it allows for more defined
| heuristics since cosine similarity is limited to [-1, 1] and b)
| it's computationally efficient (dot product, as with Euclidian
| distance) if the vectors are unit-normalized beforehand.
|
| There are indeed multiple search types but HNSW is the best 99%
| of the time for both performance and latency, and is
| implemented in almost every modern major vector store.
| politelemon wrote:
| > That is 90% of the work with RAG. RAG requires a surprising
| amount of QA to ensure "the best" is sufficient for your use
| case.
|
| Thanks that's insightful (and also adds to my current
| impostor syndrome), and makes a lot of sense... so it sounds
| like it's easy to write the query, but knowing the metric
| value to use is the testing part.
|
| I haven't heard of HNSW but I'll give it a try. It seems
| PGVector has added HNSW:
| https://github.com/pgvector/pgvector#hnsw
| anon373839 wrote:
| Has anyone systematically compared embedding based retrieval
| against traditional full text search? With all of the focus
| on vector stores lately, it feels like the whole field of
| information retrieval has been nearly set aside.
| svcrunch wrote:
| Take a look at the BEIR benchmark, which has served as one
| of the main drivers for development of neural IR systems
| since its introduction in 2020.
|
| BM25 presents a challenging cross-domain benchmark, and it
| wasn't till ~2022 that neural methods overtook it. If
| memory serves, it was the sparse neural methods like
| Splade, although recent dense models can also beat it _.
|
| _ The caveat is that BEIR is suffering from overfitting at
| this point.
| svcrunch wrote:
| > but HNSW is the best 99% of the time for both performance
| and latency, and is implemented in almost every modern major
| vector store.
|
| In my experience, HNSW indexes are very expensive to build,
| relative to indexes like IVF. They also have a larger memory
| footprint. IVF, on the other hand, is pretty trivial to
| parallelize across multiple machines, and while I'm aware
| there are techniques for doing that with HNSW, I don't know
| the details well enough.
|
| Also, if you review papers like "SOAR: Improved Quantization
| for Approximate Nearest Neighbor Search", they hint at some
| of the throughput barriers faced by graph-based methods like
| HNSW.
| NKosmatos wrote:
| I'm preparing a small internal tool for my work to search
| documents and provide answers (with references), I'm thinking of
| using GPT4All [0], Danswer [1] and/or privateGPT [2].
|
| The RAG technique is very close to what I have in mind, but I
| don't want the LLM to "hallucinate" and generate answers on its
| own by synthesizing the source documents. As stated by many
| others, we're living in interesting times.
|
| [0] https://gpt4all.io/index.html
|
| [1] https://www.danswer.ai/
|
| [2] https://github.com/imartinez/privateGPT
___________________________________________________________________
(page generated 2023-12-01 23:01 UTC)