Chat with hundreds (or even thousands) of documents at once reddit / selfhosted ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:13 Hey everyone, We just released a new feature in [SecureAI Tools](https://github.com/SecureAI-Tools/SecureAI-Tools) (v0.0.2) that allows you to chat with hundreds (or even thousands) of documents. Here is a quick demo video with a couple documents: [https://www.youtu be.com/watch?v=PwvfVx8VCoY](https://www.youtube.com/watch?v=PwvfVx8VCo Y) Now you can create a collection of documents once, and then create as many chats with the collection as needed. The documents in a collection get processed in the background allowing you to add hundreds or thousands of documents to a collection. It also saves time because you don't have to re-process all documents again every time you want to chat with a collection of documents. Please try it out, and let us know if you have any feedback for us :) (This was one of the most requested features from the community, so sharing for everyone's visibility) ---------------------------------------------------------------------- Mintfresh22 · 2023-12-16 23:16 How do you carry on a conversation with documents? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:21 You can find your past chats under the chat history page, click on an old chat there to navigate to it, and then carry on that conversation from where it was left off. ​ Let me know if I misunderstood the question though :) ---------------------------------------------------------------------- Mistic92 · 2023-12-16 23:22 Using LLM and RAG ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:33 Yes, correct. The tool allows you to use the LLM of your choice -- you can either use a locally running one (it comes with Ollama by default so you can choose from 100s of open source LLMs that it supports), or use OpenAI APIs. ---------------------------------------------------------------------- tomistruth · 2023-12-16 23:35 Vector database? What is the max token length supported for replies? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:37 It uses chroma as vector-db. The max token length will depend on the LLM you choose to use. The tool allows you to use the LLM of your choice -- you can either use a locally running one (it comes with Ollama by default so you can choose from 100s of open source LLMs that it supports) or use OpenAI APIs. ---------------------------------------------------------------------- lilolalu · 2023-12-16 23:50 1. Can we use your tool with LocalAI as well? Chatbot-UI is not very flexible when configuring OpenAI API _compatible_ servers as it seems totally geared towards using OpenAI's original API. 2. Do you have any remote auth functionality, LDAP, OIDC, SAML etc? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:55 \> Can we use your tool with LocalAI as well? Yes, you can use it with \*any\* OpenAI compatible API provider. LocalAI provides OpenAI-compatible API so you can point SecureAI Tools to it: [https://github.com/SecureAI-Tools/SecureAI-Tools?tab=readme- ov-file#use-with-openai-or-openai-compatible- apis](https://github.com/SecureAI-Tools/SecureAI-Tools?tab=readme-ov- file#use-with-openai-or-openai-compatible-apis) ​ \> Do you have any remote auth functionality, LDAP, OIDC, SAML etc? Not at the moment, but we are looking to add support for them. Do you have any favorites? ---------------------------------------------------------------------- elizabeth-dev · 2023-12-16 23:58 I know it's in the name, but specifying you were talking about chatting with an AI in your explanation would also have been nice. keep in mind this subreddit isn't AI-specific so it's not the first thing people will think you're talking about. ---------------------------------------------------------------------- Mintfresh22 · 2023-12-16 23:59 So you don't. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:59 Good point. I missed that. Let me add it as an edit :) ---------------------------------------------------------------------- QT31416 · 2023-12-17 00:10 Hey this looks amazing! We recently talked to a paid-service provider that offers something very similar to this. Their product was via WhatsApp, you chat with a bot about the operations and maintenance programs of an industrial plant (using the docs that a company uploaded). ---------------------------------------------------------------------- the-internet- · 2023-12-17 00:46 OAuth or SAML imo. Can’t beat LDAP either. ---------------------------------------------------------------------- lannistersstark · 2023-12-17 01:10 Don't be obtuse for no reason. ---------------------------------------------------------------------- InterestingMain5192 · 2023-12-17 01:39 What would you say the system requirements for this are? I’m sure it depends on the model used, but at least for decent response times with multiple documents? ---------------------------------------------------------------------- 12_nick_12 · 2023-12-17 02:06 Does it do OCR? Can it index current directories? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 02:12 No, not yet. We would love to add OCR as well as directory sync in future ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 02:14 Yep, it depends on the LLM. ​ Most models work with 8 GB RAM and a decent GPU. I have been using it on M2 MacBook with 16 GB RAM and it works great. ---------------------------------------------------------------------- 12_nick_12 · 2023-12-17 02:15 This would be great to integrate into nextcloud. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 02:16 Thank you. Supporting OAuth shouldn't be that hard given that it uses NextAuth which supports a wide variety of OAuth providers. ---------------------------------------------------------------------- [deleted] · 2023-12-17 02:20 [deleted] ---------------------------------------------------------------------- 1h8fulkat · 2023-12-17 02:21 Everyone is doing the same thing, some are charging an arma and a leg for it and others, like this fine developer, are bettering the world through open source. ---------------------------------------------------------------------- dzakich · 2023-12-17 02:37 Would it be difficult to deploy this on proxmox LXC instead of docker? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 02:37 Yes, I agree. We are planning to integrate with external document sources like NextCloud. Stay tuned :) ---------------------------------------------------------------------- fab_space · 2023-12-17 02:48 very nice work 👏👏👏 ---------------------------------------------------------------------- abii820 · 2023-12-17 02:51 Would be a really a game changer with a plugin for Obsidian. 😍😍 Thank you for all the efforts ❤️ ---------------------------------------------------------------------- BlockDigest · 2023-12-17 04:01 Would be cool if this could integrate with paperless-ngx. ---------------------------------------------------------------------- Kolmain · 2023-12-17 04:05 Do you have or do you have plans to add an API so we can use SecureAI Tools via other platforms? ---------------------------------------------------------------------- QT31416 · 2023-12-17 05:45 We haven't received a quotation yet, but it's definitely more expensive than FOSS. Lol. I'll give this a try, my bosses will love a self hostable solution. ---------------------------------------------------------------------- lilolalu · 2023-12-17 07:47 Like I said, Chatbot-UI supports the openAI API as well but i.e. when it comes to using non OpenAI Models (like Mistral/Mixtral) through the API, things start to get wonky. ---------------------------------------------------------------------- lestrenched · 2023-12-17 08:04 Thank you, will save this. Looks wonderful and very helpful ---------------------------------------------------------------------- Interesting_Argument · 2023-12-17 08:33 You can deploy Docker containers in LXC using Podman! And then managing the container through systemd. ---------------------------------------------------------------------- Interesting_Argument · 2023-12-17 08:57 Sounds interesting! Can this be used with a USB or PCI-e Coral TPU? ---------------------------------------------------------------------- geekyrahulvk · 2023-12-17 09:02 First of all this looks amazing. I have a doubt. In the docker-compose.yml it uses ollama for inference. So my doubt is if I am using an external api how does it work ? Will it still deploy the ollama instance ? In my case I have an open ai compatible api running on a remote server. So should I still need to use ollama for inference ? ---------------------------------------------------------------------- tomistruth · 2023-12-17 10:20 Looks good, will give it a try :) thanks ---------------------------------------------------------------------- pogky_thunder · 2023-12-17 10:29 Sorry if I missed it in the readme but what are the hardware requirements for this? Also, can this run only locally without connecting to openai or any other server? ---------------------------------------------------------------------- pete-standing-alone · 2023-12-17 10:43 This looks great ! I'm guessing you need a pretty beefy setup for it to work smoothly ? ---------------------------------------------------------------------- paul_h · 2023-12-17 13:14 I think OP meant "Chat with AI about office documents uploaded to environment" ---------------------------------------------------------------------- nezia · 2023-12-17 13:24 I could have a access to a system with a 6/12 core Intel 12th Gen., 128GB of DDR4 RAM and an RTX 3060 (12GB of VRAM) what would your choice for a local model be? ---------------------------------------------------------------------- Professional-Fix-960 · 2023-12-17 13:55 your company should consider donating to this product to help the developers out :) ---------------------------------------------------------------------- QT31416 · 2023-12-17 14:04 Oh yeah, if we decide to implement it, I'll definitely push that. Thanks for the idea. ---------------------------------------------------------------------- Mintfresh22 · 2023-12-17 15:28 okay thanks. Why would I want to do that? ---------------------------------------------------------------------- Mintfresh22 · 2023-12-17 15:30 Fuck off, nerd. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 15:53 Thank you :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 15:55 Yep, we have plans to integrate with external document providers like Obsidian, NextCloud, paperless-ngx, and many more. We will probably build it such that the open-source community can add other providers as needed, so let me know if you'd be interested in contributing. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 15:57 I would start with [mistral](https://ollama.ai/library/mistral) and [mixtral](https://ollama.ai/library/mixtral) \-- both are probably the best smaller models out there right now. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 15:58 I agree. We have been thinking about integrating with external document sources like paperless-ngx, NextCloud, Obsidian, and many more. We will probably build it such that open source community can add specific connectors in the future. Let me know if you'd be open to contributing paperless-ngx connector ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 15:59 Yeah, we would love to explore this direction. Can you tell me more about your use case? It will help me understand what level of integration you need ---------------------------------------------------------------------- Kolmain · 2023-12-17 16:00 I'd love to tie this into our existing platforms. As examples, we have our own Slackbot, ticketing system, project management, etc. It would be great to have an API to use in to interact with these ChatBots via those applications in addition to a UI users could log into. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:03 You may already know this, but it may be worth repeating (for others' context): SecureAI Tools comes with Ollama and it supports mistral and mixtral both models out of the box. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:04 Thank you for the kind words. Please let us know if you have any feedback for us as you try it out :) ---------------------------------------------------------------------- lilolalu · 2023-12-17 16:05 That may be so, but I am going to use it through LocalAI in any case because it provides OpenAI compatible API's which I need to make the models work with Nextcloud. So the question remains if SecureAI tools are able to accept non OpenAI Models through the OpenAI API :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:09 I haven't tried this myself yet. But there was a previous discussion on Coral at inference. The only time it needs an internet connection is when setting up the AI models -- it needs to download the pre- trained model weights from the Ollama registry. After that initial setup, it can be used completely offline. \> can this be used with a USB or PCI-e Coral TPU to off-load the GPU I haven't tried this myself yet. But there was previous discussion on Coral at [https://www.reddit.com/r/selfhosted/comments/187jmte/comment /kbfgjel/?utm\_source=share&utm\_medium=web2x&context=3](https://www.r eddit.com/r/selfhosted/comments/187jmte/comment/kbfgjel/?utm_source=sh are&utm_medium=web2x&context=3) (in case it is helpful) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:15 \> First of all this looks amazing. Thank you. :) \> Will it still deploy the ollama instance ? In my case I have an open ai compatible api running on a remote server. So should I still need to use ollama for inference ? If you ***only*** want to use remote OpenAI-compatible API then you do not need Ollama, and so you can certainly not run Ollama. To do that, 1. Remove the \`inference\` service block from the docker-compose.yml file, and 2. Remove all references to \`inference\` from \`depends\_on\` of other services. Let me know how it goes ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:16 Thank you. Let us know if you have any feedback for us :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:19 Yeah, I think I just haven't documented hardware requirements in README yet. I'll do that shortly. In the meantime, here is the thread from past where we discussed hardware requirements: [https://www.reddi t.com/r/selfhosted/comments/187jmte/comment/kberz81/?utm\_source=share &utm\_medium=web2x&context=3](https://www.reddit.com/r/selfhosted/comm ents/187jmte/comment/kberz81/?utm_source=share&utm_medium=web2x&contex t=3) ​ \> can this run only locally without connecting to openai or any other server? Yes, it can run completely offline after downloading the model weights (one-time set-up). It comes with Ollama for inference by default which runs models locally. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:27 Update: I just added a hardware requirements section in README: [https://github.com/SecureAI-Tools/SecureAI- Tools/tree/main?tab=readme-ov-file#hardware- requirements](https://github.com/SecureAI-Tools/SecureAI- Tools/tree/main?tab=readme-ov-file#hardware-requirements) ​ Let me know if I can improve it in any way :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:33 Thanks :) \> I'm guessing you need a pretty beefy setup for it to work smoothly? Not necessarily. I am using it on an M2 MacBook with 16 GB RAM and it runs pretty well. Here is a quick demo from the past with one document on my M2: [https://youtu.be/UvRHL6f\_w74](https://youtu.be/UvRHL6f_w74) There is also an option to use a remote OpenAI-compatible API server with our platform -- in which case, the hardware doesn't need anything special. More on hardware requirement at https://github.com/SecureAI- Tools/SecureAI-Tools/tree/main?tab=readme-ov-file#hardware- requirements ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:37 Ah, that is definitely something we want to do in the future. Allow using SecureAI tools from within other tools like Slack, ticketing, support systems, etc. Let me think through this in more details and get back to you. I'll also send you a DM on reddit so we can continue our conversation there as needed. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:46 \> So the question remains if SecureAI tools are able to accept non OpenAI Models through the OpenAI API Yes, I think so. You should be able to do it this way: 1. [Point to the LocalAI API server](https://github.com/SecureAI- Tools/SecureAI-Tools?tab=readme-ov-file#use-with-openai-or-openai- compatible-apis) 2. Choose "OpenAI" as model-type and then mixtral or mistral model as model-name in organization AI settings ([step 6.2 here](https://github.com/SecureAI-Tools/SecureAI-Tools?tab=readme-ov- file#6-post-installation-set-up)). Then as long as LocalAI works with "mixtral" or "mistral" like custom model name in \`model\` API param, it should all work. Please try it out, and let me know if you run into any issues or have any feedback. ---------------------------------------------------------------------- dzakich · 2023-12-17 17:26 Right. I was curious if there is an easy way to set this up on bare metal Linux distro on LXC instead of introducing a few layers of indirection (proxmox ve -> Debian VM -> Docker -> [Portainer]). Though this indirection is not necessary bad since VM jail is a good way to mitigate docker escapes. ---------------------------------------------------------------------- OwDog · 2023-12-17 18:52 Whats the difference between this and Danswer-AI? They look functionally identical, besides Danswer having more time/connectivity? ---------------------------------------------------------------------- paul_h · 2023-12-17 19:16 Say, upload a PDF study to the env you make for yourself. The ask AI for an opinion on it. Or an excerpt/summary, or to find holes in it, or to compare to another study, or to make a "word cloud" of it..... ---------------------------------------------------------------------- Mintfresh22 · 2023-12-17 20:36 Okay, guess I am too old to understand why you would want to do that, Ill go back to yelling at clouds. ---------------------------------------------------------------------- InterestingMain5192 · 2023-12-17 21:56 I think I read somewhere LLMs don’t necessary play nice with AMD hardware. Currently I have some ryzen 1600x PCs with multiple Rx 5700 XT and Rx 6900 XT GPUs on risers from when mining crypto was anywhere near profitable, now collecting dust. This could be a good use for them for some office applications. Have you heard of any reason this may not work out well out of curiosity? ---------------------------------------------------------------------- pogky_thunder · 2023-12-17 23:09 Thanks I'll give it a try. ---------------------------------------------------------------------- pogky_thunder · 2023-12-18 01:06 Ok so I gave it an initial try and I ran into some problems. Host is a NUC6i3SYH with 8GB of ram, Fedora Linux 38, no GPU. [uname -a](https://pastebin.com/iUg1xBKB), [docker-compose -v](https://pastebin.com/SDtp4h4k), [docker -v](https://pastebin.com/ArpuBjsW) I ran the initial setup script and then `docker-compose up -d`. At first I got this error message: ERROR: The Compose file './docker-compose.yml' is invalid because: services.task-master.depends_on contains unsupported option: 'restart' services.web.depends_on contains unsupported option: 'restart' I tried commenting out the restart options and I could download and start the package. When I visit the login page I enter the default credentials and try to login. I get an Internal Server Error that I guess is related to the changes I made to the compose file. Any ideas on how to fix it? ---------------------------------------------------------------------- paul_h · 2023-12-18 14:08 I've a book that's nearly finished writing at 200 pages. I thought I'd shove it into this running app, then ask the AI questions about it. After that I may be able to fix things based on the insights. I've come to realize I probably need 64GB ram and a proper GPU, plus lots of experimentation as to which of https://ollama.ai/library I should start with. ---------------------------------------------------------------------- paul_h · 2023-12-18 14:14 I might be misunderstanding, but I upload a 200 book PDF I am writing (I can't upload the LeanPub markdown source in the current version of `SecureAI Tools`). It takes a couple of hours to I guess understand the book (I have no GPU on my otherwise 32G Ryzen NUC). After that I can only get it (mistral) to look at sources (single page excerpts) of the larger PDF not the whole PDF. For example it is refusing to count the words in the book. It says " I'm unable to give an accurate answer as the context provided only includes parts of the original document from two sources, and there is no way to determine how many words are in the entire document without having access to it in its entirety", and then hyperlinks two pages of the larger doc. I might kill the web docker container, restart, then try this all again with something smaller.... ---------------------------------------------------------------------- Mintfresh22 · 2023-12-18 14:43 But AI's give false information and or outdated information. Anyway, what is your book about? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-18 16:42 Yes, it looks at document-chunks. The chunk size is controllable with \`DOCS\_INDEXING\_CHUNK\_SIZE\` and \`DOCS\_INDEXING\_CHUNK\_OVERLAP\` env vars, so I would encourage you to play with those depending on the task you want the system to perform. For example, you could set DOCS\_INDEXING\_CHUNK\_SIZE to such a large value that it can contain an entire book. But any time you change the chunk size, you would have to create a new document collection and wait for it to be processed. So it'd be a good idea to play with small documents first to speed up trial and error. Re: count the words in the book LLMs are known to do poorly with math and logic. But what they are reasonably good at is finding relevant answers from passages and understanding chat history. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-22 20:54 I think this is because \`docker-compose\` is being phased out, and so it probably doesn't understand all options. https://stackoverflow.com/a/66526176 ​ Can you try using \`docker compose\` (without the dash between docker and compose)? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 00:40 We have just released integration with Paperless-ngx! [https://www.red dit.com/r/selfhosted/comments/18rn3gr/chat\_with\_paperlessngx\_docume nts\_using\_ai/](https://www.reddit.com/r/selfhosted/comments/18rn3gr/ chat_with_paperlessngx_documents_using_ai/) Please try it out and let us know what you think :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-27 00:40 We have just released integration with Paperless-ngx! [https://www.reddit.com/r/selfhosted/comments/18rn3gr/chat\_with\_pape rlessngx\_documents\_using\_ai/](https://www.reddit.com/r/selfhosted/c omments/18rn3gr/chat_with_paperlessngx_documents_using_ai/) Please try it out and let us know what you think :)