https://github.com/pchunduri6/rag-demystified Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} pchunduri6 / rag-demystified Public * Notifications * Fork 6 * Star 177 An LLM-powered advanced RAG pipeline built from scratch License Apache-2.0 license 177 stars 6 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights pchunduri6/rag-demystified This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/p] Use Git or checkout with SVN using the web URL. [gh repo clone pchund] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @pchunduri6 pchunduri6 Update README.md ... fcd586f Oct 28, 2023 Update README.md fcd586f Git stats * 42 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time images updates October 28, 2023 10:25 .gitignore working pipeline October 19, 2023 17:21 LICENSE Create LICENSE October 28, 2023 13:06 README.md Update README.md October 28, 2023 18:20 complex_qa.py updates October 28, 2023 10:00 llama_index_baseline.py updates October 28, 2023 09:50 openai_utils.py updates October 28, 2023 09:36 requirements.txt working pipeline October 19, 2023 17:21 subquestion_generator.py updates October 27, 2023 21:05 View code [ ] Demystifying Advanced RAG Pipelines Quick start RAG Overview Building advanced RAG Pipelines The setup The secret sauce Task 1: Sub-question Generation Task 2: Vector/Summary Retrieval Task 3: Response Aggregation Putting it all together Challenges Conclusion README.md Demystifying Advanced RAG Pipelines Retrieval-Augmented Generation (RAG) pipelines powered by large language models (LLMs) are gaining popularity for building end-to-end question answering systems. Frameworks such as LlamaIndex and Haystack have made significant progress in making RAG pipelines easy to use. While these frameworks provide excellent abstractions for building advanced RAG pipelines, they do so at the cost of transparency. From a user perspective, it's not readily apparent what's going on under the hood, particularly when errors or inconsistencies arise. In this EvaDB application, we'll shed light on the inner workings of advanced RAG pipelines by examining the mechanics, limitations, and costs that often remain opaque. [intro] Llama working on a laptop Quick start If you want to jump right in, use the following commands to run the application: pip install -r requirements.txt export OPENAI_API_KEY='yourkey' python complex_qa.py RAG Overview Retrieval-augmented generation (RAG) is a cutting-edge AI paradigm for LLM-based question answering. A RAG pipeline typically contains: 1. Data Warehouse - A collection of data sources (e.g., documents, tables etc.) that contain information relevant to the question answering task. 2. Vector Retrieval - Given a question, find the top K most similar data chunks to the question. This is done using a vector store (e.g., Faiss). 3. Response Generation - Given the top K most similar data chunks, generate a response using a large language model (e.g. GPT-4). RAG provides two key advantages over traditional LLM-based question answering: 1. Up-to-date information - The data warehouse can be updated in real-time, so the information is always up-to-date. 2. Source tracking - RAG provides clear traceability, enabling users to identify the sources of information, which is crucial for accuracy verification and mitigating LLM hallucinations. Building advanced RAG Pipelines To enable answering more complex questions, recent AI frameworks like LlamaIndex have introduced more advanced abstractions such as the Sub-question Query Engine. In this application, we'll demystify sophisticated RAG pipelines by using the Sub-question Query Engine as an example. We'll examine the inner workings of the Sub-question Query Engine and simplify the abstractions to their core components. We'll also identify some challenges associated with advanced RAG pipelines. The setup A data warehouse is a collection of data sources (e.g., documents, tables etc.) that contain information relevant to the question answering task. In this example, we'll use a simple data warehouse containing multiple Wikipedia articles for different popular cities, inspired by LlamaIndex's illustrative use-case. Each city's wiki is a separate data source. Note that for simplicity, we limit each document's size to fit within the LLM context limit. Our goal is to build a system that can answer questions like: 1. "What is the population of Chicago?" 2. "Give me a summary of the positive aspects of Atlanta." 3. "Which city has the highest population?" As you can see, the questions can be simple factoid/summarization questions over a single data source (Q1/Q2) or complex factoid/ summarization questions over multiple data sources (Q3). We have the following retrieval methods at our disposal: 1. vector retrieval - Given a question and a data source, generate an LLM response using the top-K most similar data chunks to the question from the data source as the context. We use the off-the-shelf FAISS vector index from EvaDB for vector retrieval. However, the concepts are applicable to any vector index. 2. summary retrieval - Given a summary question and a data source, generate an LLM response using the entire data source as context. The secret sauce Our key insight is that each component in an advanced RAG pipeline is powered by a single LLM call. The entire pipeline is a series of LLM calls with carefully crafted prompt templates. These prompt templates are the secret sauce that enable advanced RAG pipelines to perform complex tasks. In fact, any advanced RAG pipeline can be broken down into a series of individual LLM calls that follow a universal input pattern: equation where: * Prompt Template - A curated prompt template for the specific task (e.g., sub-question generation, summarization) * Context - The context to use to perform the task (e.g. top-K most similar data chunks) * Question - The question to answer Now, we illustrate this principle by examining the inner workings of the Sub-question Query Engine. The Sub-question Query Engine has to perform three tasks: 1. Sub-question generation - Given a complex question, break it down into a set of sub-questions, while identifying the appropriate data source and retrieval function for each sub-question. 2. Vector/Summary Retrieval - For each sub-question, use the chosen retrieval function over the corresponding data source to retrieve the relevant information. 3. Response Aggregation - Aggregate the responses from the sub-questions into a final response. Let's examine each task in detail. Task 1: Sub-question Generation Our goal is to break down a complex question into a set of sub-questions, while identifying the appropriate data source and retrieval function for each sub-question. For example, the question "Which city has the highest population?" is broken down into five sub-questions, one for each city, of the form "What is the population of {city}?". The data source for each sub-question has to be the corresponding city's wiki, and the retrieval function has to be vector retrieval. At first glance, this seems like a daunting task. Specifically, we need to answer the following questions: 1. How do we know which sub-questions to generate? 2. How do we know which data source to use for each sub-question? 3. How do we know which retrieval function to use for each sub-question? Remarkably, the answer to all three questions is the same - a single LLM call! The entire sub-question query engine is powered by a single LLM call with a carefully crafted prompt template. Let's call this template the Sub-question Prompt Template. -- Sub-question Prompt Template -- """ You are an AI assistant that specializes in breaking down complex questions into simpler, manageable sub-questions. When presented with a complex user question, your role is to generate a list of sub-questions that, when answered, will comprehensively address the original question. You have at your disposal a pre-defined set of functions and data sources to utilize in answering each sub-question. If a user question is straightforward, your task is to return the original question, identifying the appropriate function and data source to use for its solution. Please remember that you are limited to the provided functions and data sources, and that each sub-question should be a full question that can be answered using a single function and a single data source. """ The context for the LLM call is the names of the data sources and the functions available to the system. The question is the user question. The LLM outputs a list of sub-questions, each with a function and a data source. task_1_table For the three example questions, the LLM returns the following output: LLM output Table Question Subquestions Retrieval Data method Source "What is the population "What is the population vector Chicago of Chicago?" of Chicago?" retrieval "Give me a summary of the "Give me a summary of the summary positive aspects of positive aspects of retrieval Atlanta Atlanta." Atlanta." "What is the population vector Toronto of Toronto?" retrieval "What is the population vector Chicago of Chicago?" retrieval "Which city has the "What is the population vector Houston highest population?" of Houston?" retrieval "What is the population vector Boston of Boston?" retrieval "What is the population vector Atlanta of Atlanta?" retrieval Task 2: Vector/Summary Retrieval For each sub-question, we use the chosen retrieval function over the corresponding data source to retrieve the relevant information. For example, for the sub-question "What is the population of Chicago?", we use vector retrieval over the Chicago data source. Similarly, for the sub-question "Give me a summary of the positive aspects of Atlanta.", we use summary retrieval over the Atlanta data source. For both retrieval methods, we use the same LLM prompt template. In fact, we find that the popular RAG Prompt from LangchainHub works great out-of-the-box for this step. -- RAG Prompt Template -- """ You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise. Question: {question} Context: {context} Answer: Both the retrieval methods only differ in the context used for the LLM call. For vector retrieval, we use the top K most similar data chunks to the sub-question as context. For summary retrieval, we use the entire data source as context. task_2_table Task 3: Response Aggregation This is the final step that aggregates the responses from the sub-questions into a final response. For example, for the question "Which city has the highest population?", the sub-questions retrieve the population of each city and then response aggregation finds and returns the city with the highest population. The RAG Prompt works great for this step as well. The context for the LLM call is the list of responses from the sub-questions. The question is the original user question and the LLM outputs a final response. task_3_table Putting it all together After unraveling the layers of abstraction, we uncovered the secret ingredient powering the sub-question query engine - 4 types of LLM calls each with different prompt template, context, and a question. This fits the universal input pattern that we identified earlier perfectly, and is a far cry from the complex abstractions that we started with. To summarize: equation call_types_table To see the full pipeline in action, run the following commands: pip install -r requirements.txt export OPENAI_API_KEY='yourkey' python complex_qa.py Here is an example of the system answering the question "Which city with the highest population?". full_pipeline Challenges Now that we've demystified the inner workings of advanced RAG pipelines, let's examine the challenges associated with them. 1. Question sensitivity - The biggest challenge that we observed with these systems is the question sensitivity. The LLMs are extremely sensitive to the user question, and the pipeline fails unexpectedly for several user questions. Here are a few example failure cases that we encountered: + Incorrect sub-questions - The LLM sometimes generates incorrect sub-questions. For example, "Which city has the highest number of tech companies?" is broken down into "What are the tech companies in each city?" 5 times (once for each city) instead of "What is the number of tech companies in Toronto?", "What is the number of tech companies in Chicago?" , etc. + Incorrect retrieval function - "Summarize the positive aspects of Atlanta and Toronto." results in using the vector retrieval function instead of the summary retrieval method. We had to put in significant effort into prompt engineering to get the pipeline to work for each question. This is a significant challenge for building robust systems. To verify this behavior, we implemented the example using the LlamaIndex Sub-question query engine. Consistent with our observations, the system often generates the wrong sub-questions and also uses the wrong retrieval function for the sub-questions, as shown below. llama_index_baseline 2. Cost - The second challenge is the cost dynamics of advanced RAG pipelines. The issue is two-fold: + Cost sensitivity - The final cost of the question is dependent on the number of sub-questions generated, the retrieval function used, and the number of data sources queried. Since the LLMs are sensitive to the prompt, the cost of the question can vary significantly depending on the question and the LLM output. For example, the incorrect model choice in the LlamaIndex baseline example above (summary_tool) results in a 3x higher cost compared to the vector_tool while also generating an incorrect response. + Cost estimation - Advanced abstractions in RAG frameworks obscure the estimated cost of the question. Setting up a cost monitoring system is challenging since the cost of the question is dependent on the LLM output. Conclusion Advanced RAG pipelines powered by LLMs have revolutionized question-answering systems. However, as we have seen, these pipelines are not turnkey solutions. Under the hood, they rely on carefully engineered prompt templates and multiple chained LLM calls. As illustrated in this EvaDB application, these pipelines can be question-sensitive, brittle, and opaque in their cost dynamics. Understanding these intricacies is key to leveraging their full potential and paving the way for more robust and efficient systems in the future. About An LLM-powered advanced RAG pipeline built from scratch Topics ai question-answering gpt rag vector-database llm chatgpt retrieval-augmented-generation Resources Readme License Apache-2.0 license Activity Stars 177 stars Watchers 2 watching Forks 6 forks Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @pchunduri6 pchunduri6 Pramod Chunduri * @jarulraj jarulraj Joy Arulraj Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.