Chat with Paperless-ngx documents using AI reddit / selfhosted ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 00:39 Hey everyone, I have some exciting news! SecureAI Tools now integrates with Paperless-ngx so you can chat with documents scanned and OCR'd by Paperless-ngx. Here is a quick demo: [https://youtu.be/dSAZefKnINc](https://youtu.be/dSAZefKnINc) This feature is available from [v0.0.4](https://github.com/SecureAI- Tools/SecureAI-Tools/releases/tag/v0.0.4). Please try it out and let us know what you think. We are also looking to integrate with NextCloud, Obsidian, and many more data sources. So let us know if you want integration with them, or any other data sources. Cheers! Links: * Project: [https://github.com/SecureAI-Tools/SecureAI- Tools/](https://github.com/SecureAI-Tools/SecureAI-Tools/) * Release v0.0.4: [https://github.com/SecureAI-Tools/SecureAI- Tools/releases/tag/v0.0.4](https://github.com/SecureAI-Tools/SecureAI- Tools/releases/tag/v0.0.4) ---------------------------------------------------------------------- Rjman86 · 2023-12-27 01:12 I am a normal person, I don't want to have a conversation with my documents. ---------------------------------------------------------------------- quinyd · 2023-12-27 01:13 This seems like such a bad idea. Why share your private and confidential documents with OpenAI? It seems like some local models are supported but as soon as I see “private and secure” on the page as “OpenAI” and “ChatGPT” I am immediately worried. ChatGPT is the complete opposite of private. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 01:37 This runs models locally as well. In fact, my demo video is running Llama2 locally on M2 MacBook :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 01:37 Fair enough. This is for those who would. It was one of the most requested features: https://www.reddit.com/r/selfhosted/comments/18k3a 1g/comment/kdpn7zi/?utm\_source=share&utm\_medium=web2x&context=3 ---------------------------------------------------------------------- SecureNotebook · 2023-12-27 02:53 This looks awesome! Well done ! ---------------------------------------------------------------------- dzakich · 2023-12-27 05:14 Very nice work, OP. I've been following your repo for a few releases and want to take it for a spin. However, I am very interested in a barebone install on Debian/Ubuntu LXC instead of docker. Are you planning to create a guide eventually? Thanks! ---------------------------------------------------------------------- Kaleodis · 2023-12-27 05:38 So is this currently (only) for getting an LLM to talk about a specific document or can I ask a question about any document (type) and get answers pulled together from multiple documents (with sources)? ---------------------------------------------------------------------- Butthurtz23 · 2023-12-27 06:04 Great, another weekend project for me, and even more reason for my wife to leave me. Since I will be spending more time with A.I., ingesting all of my personal datasets. ---------------------------------------------------------------------- fmillion · 2023-12-27 08:41 As your documents we cannot offer advice on how to address your lack of desire to converse with us. However we are able to help you answer questions about our contents or provide insight into your life choices and your future as an assimilated AI consumer. How can we help you? ---------------------------------------------------------------------- ExcessiveEscargot · 2023-12-27 09:34 I can think of a few immediate uses for myself, especially as an interactive search through stored docs and natural language rather than typical syntax. I'm not sure if I'd be considered normal, though, to be fair. ---------------------------------------------------------------------- ev1z_ · 2023-12-27 09:52 The project page makes it pretty clear you have the choice to either selfhost a model or use OpenAI. Not everyone has the HW resources to run models locally and a select subset of documents can provide a way to tinker with AI use cases using this project. Judging books on covers much? ---------------------------------------------------------------------- ev1z_ · 2023-12-27 09:52 All right, this finally gives me an actual reason to give locally hosted AI a shot. Looks nice ! ---------------------------------------------------------------------- boli99 · 2023-12-27 10:03 > normal person normal people can't form coherent queries. they want to take what could be a single question, and turn it into a multi-stage conversation. Old and busted: "Show me all the invoices from Dave Smith that are greater than $2000 and are dated between 5/6/23 and 7/8/23" New 'hotness': - hello - hello. are you there? - oh great. i wasnt sure if you were working - I need invoices from Dave Jones - Sorry. I mean Dave Smith - no, not those ones, well some of them maybe. i mean ones after June 2023 - ok but get rid of the ones before august '23 - and add back the first week of august '23 - make it only the ones that are more than tooth house and - ducking autocorrect - delete that. i meant two thousand - no. not two thousand invoices. i mean two thousand dollars - no not for everyone. just for Dave Jones - I mean Dave Smith - zoom. enhance. why isnt this working? - ...etc ---------------------------------------------------------------------- prone-to-drift · 2023-12-27 10:04 It just irks me that you keep referring to this as "chat with documents", as in, I'm having a conversation/ talking to my documents. While you _could_ say that I'm chatting with the LLM and we have some shared documents, that turn of phrase just doesn't create that image! How about: "Chat about documents" instead? ---------------------------------------------------------------------- TBT_TBT · 2023-12-27 10:10 You might have a 100 page instruction manual for some complicated device and would like to know a specific thing. You could read a lot, or you could use this. There are so many use cases for this, for business, but also private use. ---------------------------------------------------------------------- TBT_TBT · 2023-12-27 10:12 Tomato 🍅. ---------------------------------------------------------------------- JigSawFr · 2023-12-27 10:51 Second this ! ---------------------------------------------------------------------- gregorianFeldspar · 2023-12-27 10:56 I mean they have a [Dockerfile](https://github.com/SecureAI- Tools/SecureAI-Tools/blob/main/Dockerfile) based on an alpine image. If it runs on alpine it will run on every distro of your choice. Just reproduce what is done in the Dockerfile. ---------------------------------------------------------------------- Icy_Holiday_1089 · 2023-12-27 10:57 ^ This guy fcuks ---------------------------------------------------------------------- Lobbelt · 2023-12-27 11:15 If it’s as accurate as Microsoft Co-pilot is for Office suite documents, it’s basically a toss-up whether you’ll get something accurate and complete, something accurate but irrelevant or something completely made up. ---------------------------------------------------------------------- TBT_TBT · 2023-12-27 11:17 And that is why it is a version 0.0.4. Before using productively, it should be tested extensively. And even if it is ok, checking the results is always necessary. ---------------------------------------------------------------------- solarizde · 2023-12-27 12:13 What would be really useful would be a ai integrated in the whole document database to quickly find things like "give me a summary of all insurances I paid in 2023 ordered by monthly fee." "how much I spend in 2023 in all invoices tagged with #gifts" ---------------------------------------------------------------------- Lobbelt · 2023-12-27 12:16 I’m not criticising OP’s project - which is wonderful. Just doubting the general usefulness of LLMs for the purposes of retrieving truthful information from a given set of documents. My personal feeling is that they are not at all suitable for this purpose. ---------------------------------------------------------------------- TBT_TBT · 2023-12-27 12:23 Data and statistics don’t care about „feelings“. „Reproducible Ai“ ( https://research.aimultiple.com/reproducible-ai/ ) is an important field of research to make sure to be able to have trust in an LLM. This field is however still quite at the beginning. LLM results without linked sources shouldn’t be trusted. ---------------------------------------------------------------------- a_sugarcane · 2023-12-27 13:00 I wonder if something like this is possible. I have all the salary slips till date. Can I ask it to calculate the salary I earned during May 2019 to Sep 2022 ---------------------------------------------------------------------- Kaleodis · 2023-12-27 13:03 Afaik LLMs are notoriously terrible at maths, so i wouldn't even try. it might be smart enough to find and list it though. ---------------------------------------------------------------------- PsecretPseudonym · 2023-12-27 13:36 I’m really happy to see what the SecureAI team is coming up with and their momentum lately. I plan to integrate it into my current personal infrastructure asap. Please keep it up! ---------------------------------------------------------------------- mikkel1156 · 2023-12-27 13:57 Probably not, but more realistically that it could choose the right tools for it. ---------------------------------------------------------------------- 1h8fulkat · 2023-12-27 13:58 Venturing into data analytics which LLMs suck at. Your better approach would be to ask it to extract the net salary from all payslips with the pay date in CSV format, then copy it to Excel and find the total. ---------------------------------------------------------------------- ozzeruk82 · 2023-12-27 14:05 Yeah exactly, the whole "chat with" paradigm came through first 'Chat'GPT then the 'Chat'WithPDF plugin. I think projects need to backtrack and instead promote it as "query documents using natural language and AI intelligence". Or something, 'Chat' just sounds like the sort of thing you do at the water cooler. This is far more interesting and useful. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:22 Thank you :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:28 You can do both. It allows you to talk to LLM about zero or more documents. So you can do all three of these 1. One doc: Select only one doc when creating a document collection or chat. 2. Multiple docs: Select multiple docs when creating a document collection or chat. 3. Zero docs: Plain old ChatGPT without any document context. Don't select any docs when creating a new chat ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:29 u/Kaleodis is right. LLMs don't do very well at math and logic at the moment. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:30 Awesome! Let us know if you have any feedback or suggestions for us :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:32 Fair enough. And yes, you are right, it is "chat about documents with AI" than "chatting with documents directly". ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:36 For now, you can create a document collection and select documents from your data source. And then reuse that document collection to create chats. The only thing it doesn't do is keep document collection in sync with data source -- but we plan to build that soon ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 14:39 Thank you :) ---------------------------------------------------------------------- Service-Kitchen · 2023-12-27 14:47 More than possible, I’m on holiday but I can give an example when I’m back, I’ll set a reminder for myself in 8 days. ---------------------------------------------------------------------- Alarmed-Literature25 · 2023-12-27 14:48 I will say that if you’re using GPT4All to read documents, it will link to the section of the document that it pulled the answer from. ---------------------------------------------------------------------- PowerfulAttorney3780 · 2023-12-27 15:28 Can't wait to try it! ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 15:32 Awesome. Let us know if you have any feedback or suggestions for us as you try it out :) ---------------------------------------------------------------------- Shadoweee · 2023-12-27 16:48 Well that was quick! Huge thanks! ---------------------------------------------------------------------- colev14 · 2023-12-27 17:06 This looks really cool. Would I be able to use this to upload a bunch of old documents and ask the ai to generate a new document using the old ones as a template? I write statements of work pretty frequently for work. This would be amazing if I could upload 5 or 6 old ones and 1 document with new details and have it generate a new sow based on the new details, but in the same general framework as the old ones. ---------------------------------------------------------------------- shababara · 2023-12-27 17:06 Cool update u/jay-workai-tools! ---------------------------------------------------------------------- flyingvwap · 2023-12-27 17:08 Integration with paperless and possibly obsidian in the future? You have my attention! Is this able to utilize a Nvidia GPU for quicker processing? Edit: I see it does support optional GPU for processing. Excited to try it out! ---------------------------------------------------------------------- tenekev · 2023-12-27 17:24 Pretty slow without a dedicated GPU. It "works" but not usable. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 17:41 Yes, it does support NVidia GPUs. There is a commented-out block in the docker-compose file -- please uncomment it to give inference service access to GPU. For even better performance, I recommend running the Ollama binary directly on the host OS if you can. On my M2 MacBook, I am seeing it run approx 1.5x times faster directly on the host OS without the Docker. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 17:41 Thank you :) ---------------------------------------------------------------------- PovilasID · 2023-12-27 17:46 What is the local context limit? I want to load in a bunch of laws and regulations and some documents and it would be quite a lot of docs. Languages? Not familiar with local AI tools enough to know if it's English only? ---------------------------------------------------------------------- dzakich · 2023-12-27 17:49 This is a valuable suggestion. Yes, this can certainly be done, the problem is bandwidth to reverse engineering the config. Dad with two little kids doing self-hosting as a labor of love in those 1-2 hours I get to myself during a given day :) If this was on a roadmap for OP, this would be very helpful for folks like myself who prefer to bare metal things. Though I suppose I can always ask a LLM to perform this task for me :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 17:52 Oh, that is an interesting use case. At the moment, it wouldn't do well in generating the whole document. Because it only considers top K document chunks when generating the answer. It splits each document into chunks (controlled by DOCS\_INDEXING\_CHUNK\_SIZE and DOCS\_INDEXING\_CHUNK\_OVERLAP env vars). And then when answering the question, it takes the most relevant DOCS\_RETRIEVAL\_K chunks to synthesize the answer. But you could ask it to generate each section separately. In the future, we would love to support complex tasks like getting the LLM to understand full documents, and then generate full documents. One naive way to do what you want: Feed all 5-6 documents into the LLM as one prompt and ask it to generate more text like it based on other parameters. This would also require the underlying LLM's context window to be large enough to accommodate all 5-6 documents though. ---------------------------------------------------------------------- colev14 · 2023-12-27 17:56 Oh ok. I'll give it a shot next weekend when I have more free time and see if I can do paragraph by paragraph or something like that. Thanks for your help! ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 17:59 \> What is the local context limit? I want to load in a bunch of laws and regulations and some documents and it would be quite a lot of docs. There are two limits to be aware of: 1. Chunking limits: The tool splits the document into smaller chunks of size DOCS\_INDEXING\_CHUNK\_SIZE with DOCS\_INDEXING\_CHUNK\_OVERLAP overlap. And then it uses top DOCS\_RETRIEVAL\_K chunks to synthesize the answer. All three of these are env variables, so you can configure them based on your need. 2. LLM context limit: This depends on your choice of LLM. Each LLM will have their own token limits. The tool is LLM agnostic. ​ \> Languages This will depend on your choice of LLM. The tool allows you to use 100+ open-source LLMs locally ([full library](https://ollama.ai/library)). You can also [convert any GGUF- compatible LLM](https://github.com/jmorganca/ollama/tree/main?tab=readme-ov- file#import-from-gguf) you find on HuggingFace into a compatible model for this stack. ---------------------------------------------------------------------- B1tN1nja · 2023-12-27 18:08 Will this be on docker hub or ghcr for those of us who use docker run instead of docker compose? ---------------------------------------------------------------------- terrencepickles · 2023-12-27 18:41 It's 'chat, with [your] documents', not 'chat with documents'. ---------------------------------------------------------------------- prone-to-drift · 2023-12-27 18:47 Thats wasn't the point, but even then, I'm not talking _to_ the documents, I'm talking _about_ them with an LLM. ---------------------------------------------------------------------- noseshimself · 2023-12-27 22:50 > Oh, that is an interesting use case. You never wrote a business plan, did you? Id made me frown across my head and down my back to find out that it is not the numbers that are doing the work but the "summary" you are writing (a major work of fiction if you ask me). Guess who can write a better one in a few minutes than a pro in several days? ---------------------------------------------------------------------- eichkind · 2023-12-27 23:09 That would be a really nice feature to have! But even this is really impressive to see :) how consuming is it? I am running paperless on an intel Nuc where it works fine but I assume a LLM would be hard to handle? Edit: and another question: Are there plans to make the LLM understand document meta data like tags? ---------------------------------------------------------------------- ronmfnjeremy · 2023-12-28 01:20 You are close, but the problem I have with this is that I want to have a collection of hundreds or thousands of docs and PDFs and use an AI as a question answer system. The only way for this to work though I think is to train the AI on those documents and retrain periodically as more come in? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-28 01:35 Nope, we don't have to train the AI for this. Question answering can be done through retrieval augmented generation (RAG). SecureAI Tools does RAG currently, so it should be able to answer questions based on documents. RAG works by splitting documents into smaller chunks, and then for each chunk, it creates an embedding vector and stores that embedding vector. When you ask a question, it computes the embedding vector of the question, and using that, it finds top K documents based on vector similarity search. Then the top-K chunks are fed into LLM along with the question to synthesize the final answer. As more documents come in, we only need to index them -- i.e. split them into chunks, compute embedding vectors, and remember the embedding vectors so they can be used at retrieval time. ---------------------------------------------------------------------- advanced_soni · 2023-12-28 08:37 Hi u/jay-workai-tools , great job! I'd have a couple of questions. I've done a similar RAG pipeline via langchain and found that it can't always find the information within documents. I had to ask VERY specific questions in order to retrieve information, otherwise it'd just say "it doesn't contain such info". How reliable do you find your implementation, especially for information in the beginning or end of the document or information that is only a few lines long and exist only one place in the document. ---------------------------------------------------------------------- tenekev · 2023-12-29 15:40 Poteito 🥔? ---------------------------------------------------------------------- TBT_TBT · 2023-12-29 15:51 Or that.