RAG Specialized Models in Hugging Face (1B - 7B parameters) reddit / selfhosted ---------------------------------------------------------------------- Mammoth-Doughnut-160 · 2023-12-01 20:15 Hi everyone, We recently launched in Hugging Face RAG specialized models that have been specifically fine-tuned for RAG, ranging in size from 1B parameters to 7B parameters. Our models have been fine-tuned on the most popular foundation models including LlaMA, YI, and Mistral with benchmark testing data provided. (example below of the performance of one of our 7B models) Our BLING model series are designed for CPU use cases, and our DRAGON series are for 1 GPU RAG use cases and are designed to provide along with our RAG framework: * *Unified framework* that integrates LLM models with a set of surrounding workflow capabilities (e.g., document parsing, embedding, prompt management, source verification, audit tracking); * *High-quality, smaller, specialized LLMs* that have been optimized for fact-based question-answering and enterprise workflows and * *Open Source, Cost-effective, Private deployment* with flexibility and options for customization.  [https://huggingface.co/llmware](https://huggingface.co/llmware) ​ ### Benchmark Test Example for DRAGON- Yi-6B Evaluated against the benchmark test: [**RAG-Instruct-Benchmark-Tester **](https://www.huggingface.co/datasets/llmware/rag_instruct_benchmark _tester) Average of 2 Test Runs with 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations. \--**Accuracy Score**: **99.5** correct out of 100 \--Not Found Classification: 90.0% \--Boolean: 87.5% \--Math/Logic: 77.5% \--Complex Questions (1-5): 4 (Above Average) \--Summarization Quality (1-5): 4 (Above Average) \--Hallucinations: No hallucinations observed in test runs. ---------------------------------------------------------------------- jakeheller · 2023-12-02 18:38 This is really impressive. I’ve been wondering if anyone is fine- tuning smaller models for accuracy in these sorts of tasks! A few questions: - can you say more about the data used to fine tune the model? How many examples? What’s included? This would help judge what the model is good for, and what it’s not. - what is the recommended prompt template to get these results? - have you considered doing this on the Yi-200k model? If we could actually have this type of functionality on a 200k context length model… ---------------------------------------------------------------------- jakeheller · 2023-12-02 18:38 Btw, I think the folks at r/LocalLlama would be interested in this! ---------------------------------------------------------------------- Mammoth-Doughnut-160 · 2023-12-02 19:16 Thank you! I appreciate the tip and I can post there too. ---------------------------------------------------------------------- Mammoth-Doughnut-160 · 2023-12-02 19:25 We use our proprietary data set we hand crafted over many many months but generally, we train our models for business, legal and financial documents. A good way to judge the kinds of tasks and answers can be gleaned from the test dataset we published to assess the RAG performance of our models which are NOT part of the training. This is the link to the test dataset we use to judge model performance: [https ://huggingface.co/datasets/llmware/rag\_instruct\_benchmark\_tester](h ttps://huggingface.co/datasets/llmware/rag_instruct_benchmark_tester) We also have a blog that shows how our performance benchmarking works in-depth: [https://medium.com/@darrenoberst/how-accurate-is- rag-8f0706281fd9](https://medium.com/@darrenoberst/how-accurate-is- rag-8f0706281fd9) Our RAG models work really well with our open source RAG system (and of course, other RAG systems too): [https://github.com/llmware- ai/llmware](https://github.com/llmware-ai/llmware) Finally, we have many YT videos on these and similar topics to help you get started with LLMWare, RAG and our models. I hope this helps! P.S. We haven't looked at the larger models because our 7B parameter models work so well with RAG that we don't think they are needed even in most enterprise use cases. ---------------------------------------------------------------------- jakeheller · 2023-12-02 20:11 I hear you re larger models, with the one exception of larger context. In practice, things like contract question/answering, understanding complex legal documents (like cases), comparing documents, etc., are all substantially complicated by smaller context. For example, something may be defined in a definitions section that substantially impacts the understanding of the contract. Comparing two documents with each other is also really tough. Carrying that type of information across contexts is lossy. That’s what has gotten me interested in models like Yi 200k (but I haven’t dug deep or fine- tuned it myself, so I don’t know how well it would do). ---------------------------------------------------------------------- Mammoth-Doughnut-160 · 2023-12-02 22:22 I honestly don't know if the longer context window addresses issues like comparing two documents and complex legal documents. From your examples, it looks like you may deal with legal documents, which is actually the core of our company's DNA. The two founders are attorneys and the entire open source RAG system and models were first tested in the company's paid SaaS site used by attorneys for legal research and complex document analysis. There have been quite a few studies indicating that longer context windows don't improve results in the situations you describe, and our internal comparison studies have not yielded good results with longer context windows as well. Having said that, LLM performances are getting to be better by the month practically so we will keep track of how we can improve RAG performance. ---------------------------------------------------------------------- jakeheller · 2023-12-03 00:52 It is absolutely the case that, today, long contexts don’t work well. LLMs seem to “forget” information in the middle or so of the context. GPT-4 is better than Claude, but both are flawed. I am very curious if this can be overcome by fine-tuning, or if it’s inherent in LLM architecture.