Self-hosted alternative to ChatGPT (and more) reddit / selfhosted ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 14:20 Hey self-hosted community 👋 My friend and I have been hacking on SecureAI Tools — an open-source AI tools platform for everyone’s productivity. And we have our very first release 🎉 Here is a quick demo: [https://youtu.be/v4vqd2nKYj0](https://youtu.be/v4vqd2nKYj0) Get started: [https://github.com/SecureAI-Tools/SecureAI- Tools#install](https://github.com/SecureAI-Tools/SecureAI- Tools#install) Highlights: * **Local inference:** Runs AI models locally. Supports 100+ open- source (and semi open-source) AI models. * **Built-in authentication:** A simple email/password authentication so it can be opened to the internet and accessed from anywhere. * **Built-in user management:** So family members or coworkers can use it as well if desired. * **Self-hosting optimized** A simple we A simple email/password authentication so it can be opened to the internet and accessed from anywhere. * **Lightweight:** A simple web app with SQLite DB to avoid having to run additional DB docker. Data is persisted on the host machine through docker volumes In the future, we are looking to add support for more AI tools like chat-with-documents, discord bot, and many more. Please let us know if you have any specific ones that you’d like us to build, and we will be happy to add them to our to-do list. Please give it a go and let us know what you think. We’d love to get your feedback. Feel free to contribute to this project, if you'd like -- we welcome contributions :) We also have a small discord community at [https://discord.gg/YTyPGHcYP9](https://discord.gg/YTyPGHcYP9) so consider joining it if you'd like to follow along ​ ---------------------------------------------------------------------- atika · 2023-11-30 14:38 Hardware requirements? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 14:42 * RAM: As much as the AI model requires. Most models have a variant that works well on 8 GB RAM * GPU: GPU is commended but not required. It also runs in CPU-only mode but will be slower on Linux, Windows, and Mac-Intel. On M1/M2/M3 Macs, the inference speed is really good. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 15:00 Hardware requirements: * RAM: As much as the AI model requires. Most models have a variant that works well on 8 GB RAM * GPU: GPU is recommended but not required. It also runs in CPU-only mode but will be slower on Linux, Windows, and Mac-Intel. On M1/M2/M3 Macs, the inference speed is really good. (For some reason, my response to original comment isn't showing up so reposting here) ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 15:01 I am seeing some downvotes. I'd love to understand why some folks are downvoting, and correct if I made any mistakes :) ---------------------------------------------------------------------- mahinthjoe · 2023-11-30 15:37 URL https://github.com/SecureAI-Tools/SecureAI-Tools.git ---------------------------------------------------------------------- Spaceman_Splff · 2023-11-30 15:43 How slow on cpu mode are we talking? I don’t have a gpu in my microserver but tons of cpu power. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:07 For mistral model and "Tell me a Dad joke about Canada" prompt, I got following results on my two machines: * 32.06 seconds on Intel-i5 Ubuntu with 12 GB RAM * 1.29 seconds on M2 MacBook Pro with 16 GB RAM ​ I'd love to see these number and specs from your set-up for comparison ---------------------------------------------------------------------- GodRaine · 2023-11-30 16:18 Nothing but upvotes here, this looks awesome! ---------------------------------------------------------------------- GodRaine · 2023-11-30 16:18 Nothing but upvotes here, this looks awesome! ---------------------------------------------------------------------- moostmartijn · 2023-11-30 16:18 I'm desperately looking for an AI which can translate english .srt subtitles to Dutch. Please remind me when it's built ;) ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:21 \> this looks awesome! Thank you :) ---------------------------------------------------------------------- OccupiedOsprey · 2023-11-30 16:22 How does it get it's training data? Would this work offline? ---------------------------------------------------------------------- Sky_Linx · 2023-11-30 16:26 This is awesome and can’t wait to try it when I’m at home! Which model do you recommend? I have an M2 Pro ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:30 A great use case. I just tried it with the mistral model and it seems to work out of the box. I don't know Dutch language so I can't tell how well it translated, but I can see that timestamps and SRT format match! [https://imgur.com/a/P04gHsO](https://imgur.com/a/P04gHsO) It's surprising how well LLMs (AI models) can understand different formats and so on. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:34 Thank you. On M2 pro, it works like a charm. Inference speed is almost comparable to ChatGPT itself. I have tried mistral, llama2:7b and llama2:13b. Mistral beats llama2 on most benchmarks so I'd recommend that. That being said, I would highly encourage you to tinker with a few different models. SecureAI Tools uses Ollama for inference and they have a good library of all models at https://ollama.ai/library ---------------------------------------------------------------------- MonsieurNoss · 2023-11-30 16:36 You should give a try to [Whishper](https://github.com/pluja/whishper) maybe ? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:38 It does not do training at all. It takes a pre-trained model and does inference only. So yes, it can work offline -- the only time it needs an internet connection is when downloading the pre-trained model weights. After that, it works completely offline. ​ In case you need to train or fine-tune a model on your custom data set: You can train a model yourself using myriad of open source tools, export it to gguf/ggml formats and use it with SecureAI Tools. Under the hood it uses llama.cpp which works with any gguf/ggml model format. ---------------------------------------------------------------------- Sky_Linx · 2023-11-30 16:38 Thanks. Since it’s a much smaller model, how is quality of the responses compared to chat gpt? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:43 Personally I find it almost comparable to ChatGPT. It struggles with the larger context windows compared to ChatGPT. Processing larger context requires higher RAM, and local machines typically have limited resources compared to ChatGPT's server resources. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:45 I think Whisper is for audio-to-text transcription -- whereas what u/moostmartijn wants is a text (SRT) to text translate. As I mentioned [here](https://www.reddit.com/r/selfhosted/comments/187jmte/comment/kb f6am6/?utm_source=share&utm_medium=web2x&context=3), the mistral model with SecureAI Tools seems to be working fine for SRT translation tasks. ---------------------------------------------------------------------- Sky_Linx · 2023-11-30 16:45 Are 32gb of ram enough? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 16:47 That should be more than enough for most self-hosting/homelab use cases. ChatGPT has models running with 100s of GB or RAM and so it can effectively process 4000 tokens-long context windows with lower latency. ---------------------------------------------------------------------- Sky_Linx · 2023-11-30 16:48 Gotcha, thanks. Looking forward to trying this ---------------------------------------------------------------------- moostmartijn · 2023-11-30 17:06 Thanks for the try, but the translation is not really correct. It feels more like a Google Translate translation which translates every word one on one. The final sentence is not how it should be translated. I’m looking for an AI specialized for this use case. Maybe a model that learned from correct translated Dutch subtitles compared with the English subtitles. I hope there will be such a model in the near feature. ---------------------------------------------------------------------- moostmartijn · 2023-11-30 17:06 Thanks for the try, but the translation is not really correct. It feels more like a Google Translate translation which translates every word one on one. The final sentence is not how it should be translated. I’m looking for an AI specialized for this use case. Maybe a model that learned from correct translated Dutch subtitles compared with the English subtitles. I hope there will be such a model in the near feature. ---------------------------------------------------------------------- moostmartijn · 2023-11-30 17:08 Exactly, as I already have the English subtitles it would be an overkill to translate from audio. ---------------------------------------------------------------------- lilolalu · 2023-11-30 17:24 Since this has been discussed extensively, maybe you could outline the difference of your software in comparison to - GPT4All - privateGPT - oobabooga Sounds like it's pretty much the same? ---------------------------------------------------------------------- lilolalu · 2023-11-30 17:26 Got to huggingface, filter models by "text generation" and / or "translation", add dutch as a language filter. Check which models adapted to dutch. Did the same for German today and there are variants of Mistral and Falcon etc. specifically for certain languages. ---------------------------------------------------------------------- moostmartijn · 2023-11-30 17:29 I will give it a try, thanks! ---------------------------------------------------------------------- severanexp · 2023-11-30 17:32 Fantastic work. Do you think a google coral could be used for inference?? ---------------------------------------------------------------------- Sky_Linx · 2023-11-30 17:33 I am at home now and am trying it but it's very slow, and I noticed that the Docker image is for amd64. Is there an ARM image for M1/2/3 macs? ---------------------------------------------------------------------- lilolalu · 2023-11-30 17:36 Btw I did translations of German SRT to various languages in (the free) ChatGPT 3.5. Works excellent, chatGPT keeps the timestamps etc. I would prefer a selfhosted version as well, but you can't find one and this is not privacy relevant stuff that you cannot out into chatgpt, give it a try. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 17:37 Great question. The main difference is that SecureAI Tools is optimized for self-hosting/homelab in following ways: 1. It comes with built-in authentication so it can be opened up to the internet (over VPN or so) and accessed from anywhere. 2. It allows multiple users so you can give access to your family and colleagues if needed. 3. It comes with necessary configs and docker image/compose files to make self-hosting easy. In my experience, GPT4All, privateGPT, and oobabooga are all great if you want to just tinker with AI models locally. But when it comes to self-hosting for longer use, they lack key features like authentication and user-management. ---------------------------------------------------------------------- Mephidia · 2023-11-30 17:37 Nice I’ve been waiting for someone to make a good UI for local inference ---------------------------------------------------------------------- moostmartijn · 2023-11-30 17:40 Doesn’t ChatGPT have a maximum amount of characters? I thought that was the problem with the free ChatGPT, I don’t want to do a copy/paste multiple times. I also have a lot of subtitles to be translated, so it would take some free time. Of course I would prefer a self hosted solution which can be automated / linked to various applications like Bazarr, but as you said; there is none at the moment. Maybe I will give ChatGPT a try to see if it works. ---------------------------------------------------------------------- imacleopard · 2023-11-30 17:51 This is reddit. Some people downvote because they have nothing better to do. ---------------------------------------------------------------------- lilolalu · 2023-11-30 17:59 I don't know, my video was just 10min, so i probably didn't run into that limitation, if it exists. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:06 \> Fantastic work. Thank you. \> Do you think a google coral could be used for inference?? Right now, it only supports local inference out of the box. However, we definitely have plans to support remote APIs like Google Coral, OpenAI, Claude, etc. SecureAI Tools aims to be the AI-model agnostic application layer. I'd love to understand your use case a bit more if you're open to sharing. ---------------------------------------------------------------------- lilolalu · 2023-11-30 18:07 It cannot. LLM's need a lot of Memory. ---------------------------------------------------------------------- bloodguard · 2023-11-30 18:08 Neat. I have an [NVIDIA Tesla T4](https://www.nvidia.com/en-us/data- center/tesla-t4/) that's not being used at the moment. I may give this a whirl. ---------------------------------------------------------------------- lilolalu · 2023-11-30 18:08 There are several good UI for local inference. The difference is that this one has Multiuser capabilities and authentication, if that's something you need. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:13 Awesome. Let us know how it goes. We haven't tried it ourselves on T4 GPUs because we don't have access to them. We'd love to incorporate lessons from your trials into our project :) ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:14 >google coral Wait, I think I may have misunderstood. Is Google Coral a hardware/device? Or is it an API (like OpenAI's API)? ---------------------------------------------------------------------- severanexp · 2023-11-30 18:17 Google coral are little devices that are used for inference very used in NVRs, for image recognition. Now I know it’s not the same, but seeing how cheap they are, and how cheap system memory is, I wondered if having a RAM disk with a google coral if that could somehow replace the gpu all together. My use case is partly making self hosting LLMs cheap :) a google coral is what, 50 bucks nowadays? My end objective is to plug these self hosted llms to smart homes (/r/openHAB or /r/homeassistant for example) for a first gen real “smart” home. There’s already work being done with whisper/willow for locally hosted voice control (more info here: https://community.openhab.org/t/willow- open-source-echo-google-home-quality-speech-hardware-for-50/146717 ) The point would be, to plug an Llm to a smart home where it could “see” the status and information from all sensors and relays and such, and then have a conversation with it: “I’d like for the entrance light to turn on, when someone opens the door after it’s dark.” The llm would have access to date, time, the door sensor status, and the light relay, and be able to generate a rule to make this happen. Second step, would be for the llm to auto generate rules it seems might be helpful based on the changes it sees daily, assessing habits and such from they analysis propose automations or improvements. “I’ve seen you turn on the light after you get up from the bed, would you like for me to turn on the light automatically?” Stuff like that :) ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:19 To add to u/lilolalu's answer, SecureAI Tools can work with the models you find on huggingface. SecureAI Tools uses Ollama as the inference engine and Ollama allows you to create your own models: [https://github.com/jmorganca/ollama/blob/main/README.md#customize- your-own-model](https://github.com/jmorganca/ollama/blob/main/README.m d#customize-your-own-model) So if you find a model on huggingface that isn't available on Ollama out of the box, then you can create a compatible Ollama model yourself and use it with SecureAI Tools :) ---------------------------------------------------------------------- seanpuppy · 2023-11-30 18:19 the M1/2/3 macs have some insane vram per buck for consumer grade stuff. Memory is shared with the GPU so you can run a 70B model locally. By default the GPU has access to about 67% of the total RAM but I saw a post on r/LocalLLaMA yesterday showing how to increase that. ​ This also came out this week, a one click installer for an LLM web app to atleast POC something quickly: [https://simonwillison.net/2023/Nov/ 29/llamafile/](https://simonwillison.net/2023/Nov/29/llamafile/) ---------------------------------------------------------------------- severanexp · 2023-11-30 18:20 RAM disk. I don’t see that being a problem honestly. Even if inference takes a hit, it’s a ton of a lot cheaper than a gpu, with potential for a crap ton of lot more memory too. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:20 Thank you. Let us know if you have any feedback or suggestions for improving our UI/UX. ---------------------------------------------------------------------- GodRaine · 2023-11-30 18:23 Hey Jay - so here’s a use case for you. I run a medium sized physical therapy clinic and my pie in the sky dream is to have a local AI that doesn’t connect to any outside services (for HIPAA reasons) but is capable of providing a conversational AI for my new staff so that they can ask it questions about policies and procedures in the clinic. Essentially an AI that can be a trainer when the trainer isn’t available; can answer questions like “how do I complete authorization for this patient who has this insurance” and it walks you through the steps, etc. Do you think that’s doable with this tool? I’ve been trying to set up Danswer for this but it’s extremely resource heavy and I haven’t been able to budget a decent PC / server to run the Docker services for it. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:26 \> llamafile Oh, that looks neat! One drawback of llamafile approach is that binary/exe and model weights are in the same file. So if you want to switch between models then you need to download a new binary, change some configs, and restart docker containers. With SecureAI Tools, you don't need to do any of that -- it uses [Ollama](https://github.com/jmorganca/ollama/) under the hood and Ollama separates executable from model-weights and that makes model-switching way easier. So to switch models in SecureAI Tools, all you need to do is go to a page on the web-app and change a string ---------------------------------------------------------------------- bobzilla__ · 2023-11-30 18:26 It’s a hardware tpu ---------------------------------------------------------------------- lilolalu · 2023-11-30 18:28 I don't have a coral device, I was contemplating buying one but discarded the idea because I read several posts where people explained that LLM don't work not the coral device. One example https://www.reddit.com/r/LocalLLaMA/s/T8NFXIpELl ---------------------------------------------------------------------- severanexp · 2023-11-30 18:30 The first 5 posts on that thread made the issue very clear for me. Thank you! (Holy shit the amount of data moving is absurd!) ---------------------------------------------------------------------- lilolalu · 2023-11-30 18:34 I think the next best thing to actually buying a GPU with enough VRAM (which I mainly dislike because if their idle power consumption) would be firing up "on-demand" cloud GPU's on something like vast.ai ... The problem is that the LLM models can easily be 5-10gb so them on the VM, starting it up can take a couple of minutes and that's usually not what you want. ---------------------------------------------------------------------- seanpuppy · 2023-11-30 18:35 oh yeah its definitely not a great solution if you are at all a power user. But I saw it yesterday so thought it could be cool to anyone completely unfamilar with self hosting LLMs ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:35 Ah, ok. Sorry, I mistook Google coral to be an API like OpenAI or Claude. \> I wondered if having a RAM disk with a google coral if that could somehow replace the gpu all together. That would be really neat. Let us know how it goes if you end up trying this approach. I really like the use cases about home automation you have in mind. Long term, I would love to allow automations like the one you talked about. The way I image them to work as an AI Agent. Users can configure AI agent with 1. Appropriate instructions in natural language. From your example, “I’d like for the entrance light to turn on, when someone opens the door after it’s dark.” 2. Give it access to appropriate plug-in/APIs so it can read data and take actions. ChatGPT has shown already that LLMs work great with JSON and OpenAPI specifications. 3. Whether human needs to be in the loop or not. For some sensitive actions, it'd be good to have approval from human. A general purpose AI agent like this can really be applied to soo many domains -- home-automation being one of them. ---------------------------------------------------------------------- boli99 · 2023-11-30 18:38 its true. i just downvoted this comment out of spite but then i upvoted it to keep you on your toes. might downvote again later. idk. ---------------------------------------------------------------------- severanexp · 2023-11-30 18:40 I like your train of thought :). Both homeassistant and openHAB have local apis so that would work: https://developers.home-assistant.io/docs/api/rest/ https://www.openhab.org/docs/configuration/restdocs.html One step closer to Jarvis :D ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:45 Ah, this is a great use case. And it can certainly be done. The policies and procedures in the clinic can be fed into SecureAI Tools as documents and then your trainers can chat with those documents to get answers to their questions. And as you mentioned, it all runs locally so it's compliant with HIPAA (and almost all other compliance regulations). I would love to work with you and help you deploy an instance for your needs. Sending you a DM invite so we can chat privately ---------------------------------------------------------------------- chewableplate · 2023-11-30 18:46 The NPU's in the new Macs are great for this. ---------------------------------------------------------------------- 100GHz · 2023-11-30 18:51 Out of curiosity, why prepackaged it within docker? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:53 And we are starting to see others follow. [Qualcomm](https://www.reuters.com/technology/qualcomm-unveils-new-pc- laptop-chip-with-ai-features-2024-2023-10-24/) and [Intel](https://www.reuters.com/technology/intel-says-newest-laptop- chips-software-will-handle-generative-ai-2023-09-19/) have both announced somethings capable of running AI models locally. In a few years we will see that most hardware will be able to run AI models natively as well as M1/M2/M3 Macs ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 18:56 I have seen that the self-hosting community likes to use Docker containers for their self-hosting needs. So that's why we started with it. It doesn't HAVE to be used with docker. You can spin up an instance without docker just as well. ---------------------------------------------------------------------- 100GHz · 2023-11-30 19:04 Thanks I'll give it a twirl later on then :) ---------------------------------------------------------------------- Spaceman_Splff · 2023-11-30 19:25 Tried to set i up on ubuntu vm with 8 cores and 32 GB of RAM and cannot get it to give me a response. It just spins. I went into the settings tab and downloaded the mistral. I switched it to llama2 and still no luck. It does successfully download everything. ---------------------------------------------------------------------- eye_can_do_that · 2023-11-30 19:25 Could i use this to point an AI at 1000 documents then ask questions about them, and get a ref to where it is getting it's answer from? ---------------------------------------------------------------------- tchansen · 2023-11-30 19:38 I prefer containers! Thanks for that. ---------------------------------------------------------------------- lilolalu · 2023-11-30 19:38 You can use GPT4all, privateGPT, docsGPT ... They all allow ingesting and querying your own documents. ---------------------------------------------------------------------- It_Might_Be_True · 2023-11-30 19:40 Can I run this with a google coral to improve performance? ---------------------------------------------------------------------- Didacity777 · 2023-11-30 19:53 Hey, I managed to get it up and running, but it is not responding in the chat. Am I missing something?? ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-11-30 20:00 This is EXACTLY what I've been looking for! Thank you!!!!!!! ---------------------------------------------------------------------- qksv · 2023-11-30 20:10 FWIW, chatGPT can't properly conjugate verbs in Hebrew. ---------------------------------------------------------------------- Traditional_Mud_303 · 2023-11-30 20:13 I have set it up, but I don't see an AI tab on settings to set the AI model ---------------------------------------------------------------------- ryosen · 2023-11-30 20:15 > Intel-i5 There have been 13 generations since the Intel i5 was first released 14 years ago. Given the massive time difference between your two stats, I think it would be very useful to know which model of i5 was used. ---------------------------------------------------------------------- x6q5g3o7 · 2023-11-30 20:28 Nicely done! What are options like for AMD GPUs? Any future plans to support? ---------------------------------------------------------------------- gregorianFeldspar · 2023-11-30 20:31 > GPT4all, privateGPT, docsGPT What's the most privacy "friendly" among them? ---------------------------------------------------------------------- kdevkk · 2023-11-30 20:43 Apologies if this question doesn't make sense. Does the model get updated and learn as we provide data? ---------------------------------------------------------------------- joost00719 · 2023-11-30 20:46 I just got a gtx 1050 for my server. Can I do anything with it for something like this? ---------------------------------------------------------------------- DoubleSoftware4137 · 2023-11-30 20:46 u/jay-workai-tools this is very good work. We should collaborate. We are helping our users deploy cloud native apps in large scale envs as well as their local computers. Checkout this YT video [https://youtu.b e/FRgHmQucWy8?si=QaM2F9kI8fku6KMa](https://youtu.be/FRgHmQucWy8?si=QaM 2F9kI8fku6KMa). Let me know if you are interested. ---------------------------------------------------------------------- lilolalu · 2023-11-30 20:48 I think in terms of privacy they are all the same, because they use local LLM models, so you can run them without connecting to any external services at all... The differences are more in the UI and overall design focus. GPT4All is an electron based desktop GUI application while the others are self hostable web services. ---------------------------------------------------------------------- pushing_film · 2023-11-30 21:10 Hi Jay, nice work! One question: How does one feed documents to it? ---------------------------------------------------------------------- thefoxman88 · 2023-11-30 21:22 Is there any chance we can get a unraid template setup in the community apps? ---------------------------------------------------------------------- TheSmashy · 2023-11-30 21:32 I can't run this on a raspberry pi? God damn it. ---------------------------------------------------------------------- dangernoodle01 · 2023-11-30 21:35 oobabooga is a great backend with tons of features, this one seems to be a more streamlined, simple frontend with automatic backend - which is very nice. ---------------------------------------------------------------------- dangernoodle01 · 2023-11-30 21:39 Seems nice so far, best of luck! Gave it a star on github and might download it later, excited to see where the journey leads. I'm hoping to see back to this comment in 2 years when this completely replaced my ChatGPT subscription :) ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:39 Higher RAM & GPU are for inference servers. The web service can certainly run on Raspberry pi and point to an inference server running on a more capable machine somewhere else. ​ There is also a project to make LLMs smaller -- for example: [https:// github.com/jzhang38/TinyLlama](https://github.com/jzhang38/TinyLlama) and its equivalent model on Ollama at [https://ollama.ai/saikatkumarde y/tinyllama](https://ollama.ai/saikatkumardey/tinyllama) (specify "saikatkumardey/tinyllama" as model on [http://localhost:28669/- /settings?tab=ai](http://localhost:28669/-/settings?tab=ai) ) It might just run on Raspberry Pi. Please let us know how it goes if you do decide to run it. I am curious to see the performance of TinyLlama on Rpi hardware :) ---------------------------------------------------------------------- dangernoodle01 · 2023-11-30 21:41 this is the only comment this user has ever made. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:42 Ah yeah, the web service has only amd64 image. But that shouldn't be slowing it down. The inference server does all the inference. Is that slow? If the Ollama/inference server itself is slow, then could you please try running Ollama directly on the OS and pointing web service to it? Ollama for mac can be downloaded at https://ollama.ai/download/mac ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:46 Not yet, but we are building that soon in the "chat-with-documents" feature. The only thing we don't know yet is how good of a performance (latency-wise) it would give if you throw 1000 docs at once and it's running on home PCs -- it may take hours to process. I would love to understand the use case of 1000s of documents. Why that many documents? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:49 That is certainly weird. I would love to help dig into why this is happening. Can you share logs from inference container? May be there is a clue in there as to what is happening? It may be easier to discuss this in our Discord community at [https://discord.gg/YTyPGHcYP9](https://discord.gg/YTyPGHcYP9) \-- I can respond faster there. I am jay\_haha there so please tag me if you post there ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:51 Right now, there is no built-in way to feed documents into the web application directly just yet. In order to do this, we would be building retrieval augmented generation (RAG) soon as part of our "chat-with-documents" feature ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:54 That is certainly weird. Just to confirm, you downloaded the model right? It's the [step #6.2 in the installation guide](https://github.com/SecureAI-Tools/SecureAI-Tools#6-post- installation-set-up). If you did download the model and this is still happening to you, then please share your system specs and logs from inference container, and I can try to find a clue. It may be easier/faster to debug this in our discord community at [https://discord.gg/YTyPGHcYP9](https://discord.gg/YTyPGHcYP9) \-- please feel free to post there and tag me (jay\_haha) ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:55 Doesn't seem like it. Previous discussion at https://www.reddit.com/r/ selfhosted/comments/187jmte/comment/kbfmc9q/?utm\_source=share&utm\_me dium=web2x&context=3 ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:56 Awesome, thank you. Please let us know how it goes ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 21:58 I think you might be on user settings page ([http://localhost:28669/settings](http://localhost:28669/settings)) instead of org settings page ([http://localhost:28669/- /settings?tab=ai](http://localhost:28669/-/settings?tab=ai)) -- notice the extra "-" in there. Another user reported on discord that this was happening to them and it was because of that missing dash "-" in the URL path. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:00 Nope, this only does inference so model weights do not change as we feed data into it. It only tried to predict the next token based on provided data and context window. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:01 Yep, you can run it with GTX 1050. I would love to see your performance numbers and system spec to understand how well it does on it. So please share if you can ---------------------------------------------------------------------- joost00719 · 2023-11-30 22:03 I haven't installed it yet and need to figure out gpu pass through then. But will report back if I don't forget about the project ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:09 Thank you for the kind words and GH star -- it keeps us going. Please try it and let us know how it goes. \> I'm hoping to see back to this comment in 2 years when this completely replaced my ChatGPT subscription :) Haha, nice! Let's aim for a much shorter period than that haha :D ---------------------------------------------------------------------- Axelazo · 2023-11-30 22:13 Any guide on how to setup this for Spanish language? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:15 Sorry, I am a bit noob for unraid. Is this what you are referring to? [https://unraid.net/community/apps](https://unraid.net/community/apps) ​ If so, how does one make their thing available there? And what is involved to make a dockerized app to be made available there? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:17 Are you referring to Spanish language for the web UI? or are you looking for the AI model that can do well with Spanish language? ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:19 Good point. Mine is 8th Gen i5. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:33 We use Ollama as the inference engine and AFAIK Ollama doesn't yet support AMD GPUs out of the box. Ollama uses llama.cpp under the hood and there appears to be a way to compile llama.cpp to work with AMD GPUs: [https://www.reddit.com/r/Loc alLLaMA/comments/13m8li2/finally\_got\_a\_model\_running\_on\_my\_xtx\ _using/](https://www.reddit.com/r/LocalLLaMA/comments/13m8li2/finally_ got_a_model_running_on_my_xtx_using/) ---------------------------------------------------------------------- eye_can_do_that · 2023-11-30 22:38 I have a few use cases in my head. Journal papers in my field, the subtitles of a fantasy/fiction pod cast I listen to (this is just hundreads), my emails. I could envision asking questions that these would have the answer to. Also why I want it to reference back to it. ---------------------------------------------------------------------- fy8d6jhegq · 2023-11-30 22:39 Most srt files are going to be for movies which are probably 10 times the size/length. ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 22:50 Gotcha. Referencing back or citation is definitely possible with RAG. The only thing that worries me is scaling to 1000 docs on home hardware. It can be easily done on server-clusters with a ton of resources and parallelism but on home hardware, doing it would be tricky -- especially meeting the acceptable UX bar ---------------------------------------------------------------------- Axelazo · 2023-11-30 22:56 Sorry for my dumb question, I'm referring to the second one! ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 23:07 No no. There are no dumb questions :) Personally, I don't know any models that do well with the Spanish language. As someone else shared [here](https://www.reddit.com/r/selfhosted/comm ents/187jmte/comment/kbffibr/?utm_source=share&utm_medium=web2x&contex t=3), you can find the model on HuggingFace and make it work with SecureAI Tools through [this](https://www.reddit.com/r/selfhosted/comm ents/187jmte/comment/kbfobp2/?utm_source=share&utm_medium=web2x&contex t=3). Alternatively, you can always try to use Spanish with mistral and llama2:7b models and see how well they do. I suspect that they won't be that bad given that they were trained on internet text and there is a ton of content online in Spanish language, so it may not need any customizations or fine-tuning to work decently in Spanish language. ---------------------------------------------------------------------- mimikater · 2023-11-30 23:13 I will try that on a rPi4 ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 23:45 Awesome. Please share your experience, feedback and performance numbers if you can :) ---------------------------------------------------------------------- lmamakos · 2023-11-30 23:57 I don't think ollama.ai uses the NPU resources on the ARM Macs, but instead the GPUs. I think that's what I see when running iton my M1 based system. ---------------------------------------------------------------------- mollynaquafina · 2023-11-30 23:59 Do you have any plans to support llava models? ---------------------------------------------------------------------- mimikater · 2023-12-01 00:01 A multi arch image would be nice for running it on arm64/aarch64. Have to build it myself :) ---------------------------------------------------------------------- mimikater · 2023-12-01 00:02 A multi arch image would be nice for running it on arm64/aarch64. Have to build it myself, found some help in the github issue no 5 ---------------------------------------------------------------------- akmzero · 2023-12-01 00:27 Since March of 21'! ---------------------------------------------------------------------- Agreeable-Pirate-886 · 2023-12-01 00:54 Reddit randomly fudges the vote count on every reload. If your post has a karma of 10 and then 8 on the next refresh, that doesn't mean you got 2 down votes. ---------------------------------------------------------------------- dryEther · 2023-12-01 01:07 AMD GPU support? ---------------------------------------------------------------------- SlowThePath · 2023-12-01 01:14 Yes. This is the perfect excuse for me to buy a new GPU and put the old one in the server. 3080's are old and crappy now anyway ...right? (Yes that was sarcasm btw). ---------------------------------------------------------------------- bityard · 2023-12-01 01:31 Okay fine but were the jokes any good? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 02:35 It doesn't support AMD GPUs out of the box right now, but there is a way to make it work: https://www.reddit.com/r/selfhosted/comments/187j mte/comment/kbgtu51/?utm\_source=share&utm\_medium=web2x&context=3 ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 02:37 I agree. I have never worked with multi-arch images myself, so if you could point me to right direction then I can try to see how much work it would be to publish them :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 02:39 Yes, it supports llama2 out of the box. You can specify llama2:7b, llama2:13b, and llama2:70b on the org settings page and it will work. It uses Ollama under the hood so all the models in [Ollama library](https://ollama.ai/library) are supported. ---------------------------------------------------------------------- mollynaquafina · 2023-12-01 02:41 Yes right, but LLaVA is different. [Repo here](https://github.com/haotian-liu/LLaVA) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 02:46 Oh, I thought it was a typo lol. Ollama can work with any gguf/ggml format model. Looks like people are trying to convert LLaVa into gguf/ggml format: [https://www.reddit.com /r/LocalLLaMA/comments/16ky4eo/llava\_ggufggml\_version/](https://www. reddit.com/r/LocalLLaMA/comments/16ky4eo/llava_ggufggml_version/) Once a model has been exported to gguf format, then it is easy to convert into Ollama model: https://github.com/jmorganca/ollama/tree/main#customize-your-own-model ---------------------------------------------------------------------- mollynaquafina · 2023-12-01 02:47 Gotcha, I'll keep an eye out. Thanks for your work! ---------------------------------------------------------------------- audero · 2023-12-01 03:19 Is this compatible with the OpenAI API? By that I mean you could use a client app designed to work with OpenAI and just point the endpoint to SecureAI instead. ---------------------------------------------------------------------- rope93 · 2023-12-01 03:24 This looks awesome! My little project was missing just that! Https://github.com/rogueghost93/fly-hi I'll add it these days! ---------------------------------------------------------------------- kidogodub · 2023-12-01 03:31 You should add this link to the YouTube description. ---------------------------------------------------------------------- Comms · 2023-12-01 03:48 Any chance this will end up on Unraid community apps? ---------------------------------------------------------------------- fidalgofeliz · 2023-12-01 03:50 Easy installation, very well done and very useful. Impossible to speak something bad to you. Congratulations! ---------------------------------------------------------------------- planetearth80 · 2023-12-01 03:51 Can this also be used to provide API access? I am currently using the OpenAI Python package and wondering if I can change the API URL to use this instead? ---------------------------------------------------------------------- Commercial_Ad8403 · 2023-12-01 04:18 Anyone had any luck getting this going on unraid? I'm able to login - sort of - but it keeps redirecting me to http://localhost:3000/api/auth after, which of course doesn't work. I tried setting NEXTAUTH_URL= but then I get an SSL error instead. ---------------------------------------------------------------------- Commercial_Ad8403 · 2023-12-01 04:32 Unfortunately, unraid for 'apps' only supports single docker images; i.e. you cannot create an app that includes ollama. However, ollama is already set as an app, so you can include a variable in your template that the user would input the location. i.e. INFERENCE_SERVER=http://unraid-ip:11434 However - after reading the docs, I see that they don't really expect non-unraid users to submit community apps. https://forums.unraid.net/topic/57181-docker-faq/#comment-566084 ---------------------------------------------------------------------- msic · 2023-12-01 04:43 Why not try translation software instead... https://libretranslate.com/ ---------------------------------------------------------------------- Disastrous_Elk_6375 · 2023-12-01 05:03 When running CPU inference the most important bottleneck is RAM speed, not so much CPU speed. That's the main reason macs have become a viable inference platform. ---------------------------------------------------------------------- I_EAT_THE_RICH · 2023-12-01 05:48 I’m going to be honest, I’m sick and tired of repackaged, industry standard software that is just an nginx reverse proxy and underpowered authentication system. Self hosting is already easy. SSL is easy. LDAP, and SSO are easy. If people actually wanted to help they’d make tutorials instead of opinionated branded tools that aren’t as flexible. Just my two cents ---------------------------------------------------------------------- aManPerson · 2023-12-01 06:54 have 64gb of system ram. the last time i tried to run any AI model thing localy i - could only get 1 of them to respond/work at all - when it ran, it only ever used 1gb of system ram and ran really, really slowly running on a ryzen 5820u cpu laptop, linux os. besides all of the other self hosting wrapper stuff you have, will the AI model stuff run any better? ---------------------------------------------------------------------- Tripanafenix · 2023-12-01 07:24 Where we're at it, is there a possibility to train a model graphql? A customer wishes to transform natural language into graphql prompts. Which AI is already able to do this or where can I find training data? ---------------------------------------------------------------------- znutarr · 2023-12-01 08:21 Whisper ai ---------------------------------------------------------------------- stuffitystuff · 2023-12-01 08:21 I've got 27 years worth of email I'd love to be able to chat with. ---------------------------------------------------------------------- Woke_killa · 2023-12-01 08:59 Does this make sense on a home server? The response time will not take several dozen seconds and the response quality will not be worse than chatgpt? I'm currently using openai api and it's like lvl 0 for me. So is your project better or worse? ---------------------------------------------------------------------- niemand112233 · 2023-12-01 09:22 I can't get it running with my GPU. I get this error: `parsing /root/secure-ai-tools/docker-compose.yml: yaml: line 19: did not find expected key` This is my .yaml: version: "3.8" services: web: image: public.ecr.aws/d8f2p0h3/secure-ai-tools:latest platform: linux/amd64 volumes: - ./web:/app/volume env_file: - .env environment: - INFERENCE_SERVER=http://inference:11434/ ports: - 28669:28669 command: sh -c "cd /app && sh tools/db-migrate-and-seed.sh ${DATABASE_FILE} && node server.js" depends_on: - inference inference: image: ollama/ollama:latest volumes: - ./inference:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: 'all' capabilities: [gpu] ---------------------------------------------------------------------- 2RM60Z · 2023-12-01 10:17 That could be fun! ---------------------------------------------------------------------- niemand112233 · 2023-12-01 10:21 I'm running it as a LXC on Proxmox on this Hardware: ​ Dell R620 (2\* E5-2650L V2), 178 GB of RAM and a Nvidia Quadro P620. ​ I passthroughed the GPU, gave the LXC access to all 24 Threads and assigned 64 GB of RAM. While the software is running and utilizing the GPU, however, only 50 % of the CPU is used and quite often (e.g. when I ask to rephrase a longer text), the AI does not answer anymore. Any ideas? ---------------------------------------------------------------------- suddenlypenguins · 2023-12-01 10:23 It's frustrating how many open source tools spend a ton of dev energy on access and control. My home meal planner app doesn't need LDAP support and be scalable to 10 million users and groups with military grade multitenancy and permission systems. While effort on basic features and UI stagnates. ---------------------------------------------------------------------- Richeh · 2023-12-01 11:29 Seconded, I'd love to have a chat AI running on a raspberry pi. Great for ubiquity / price / portability. If you manage it I'd love to hear how hard it was / a guide if you had time. ---------------------------------------------------------------------- elroypaisley · 2023-12-01 12:22 What happens if you run something like this on a no GPU VPS, something like a racknerd or oracle free cloud box with 3vCPU and 3GB ram? Does it work? Is it so slow as to be useless? ---------------------------------------------------------------------- pushing_film · 2023-12-01 12:31 Thanks for the answer! I think that is a big bang-for-your buck kind of feature. I hope you are able to implement the "chat-with-documents" feature soon. Good luck! ---------------------------------------------------------------------- wet_moss_ · 2023-12-01 13:28 I know what iam doing this weekend. Try to run it on my rpi4. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:14 It's on our to-do list. To allow SecureAI Tools to work with OpenAI APIs as LLM backend. Stay tuned. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:19 Someone else also asked for this. But as I mentioned, I am a noob when it comes to raid stack: [https://www.reddit.com/r/selfhosted/comments/ 187jmte/comment/kbgr534/?utm\_source=share&utm\_medium=web2x&context=3 ](https://www.reddit.com/r/selfhosted/comments/187jmte/comment/kbgr534 /?utm_source=share&utm_medium=web2x&context=3) I would love to explore supporting this. You can also contribute to project if you're already familiar with unraid and are willing to contribute. We welcome contributions :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:19 Thank you so much. You made my day! ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:22 Yes, kind of. SecureAI Tools uses Ollama under the hood so if you want API integration, you can directly work with Ollama API. In the future, we could explore the direction of providing API-level authentication, usage tracking etc from the SecureAI Tools layer if enough users would find it useful. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:25 I am unfamiliar with unraid so I won't be much of help with that. But I can help you with NEXTAUTH\_URL. Could you share the SSL error you are seeing? It may be related to NextAuth/AuthJS framework that SecureAI Tools uses for authentication. It may be easier/faster to discuss and debug this on our discord community: [https://discord.gg/YTyPGHcYP9](https://discord.gg/YTyPGHcYP9) . If you post there, please feel free to tag me at jay\_haha ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:27 Thank you. This was helpful context ---------------------------------------------------------------------- lowerseagate · 2023-12-01 14:33 I should give it a try ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:35 The 64GB of RAM is more than enough for most models. Compute would probably be the bottleneck in your set-up. The good thing about SecureAI Tools is that it is not tied to any specific AI model. It supports all of the gguf/ggml format models. It uses Ollama under the hood and Ollama has a large collection of models. Some of those models are optimized to run with less computing power -- like [https://ollama.ai/saikatkumardey/tinyllama](https://oll ama.ai/saikatkumardey/tinyllama) (just specify "saikatkumardey/tinyllama" as model under organization settings). ​ You can also use most of the gguf/ggml models you find on HuggingFace with SecureAI Tools: https://github.com/jmorganca/ollama#customize- your-own-model ---------------------------------------------------------------------- unconscionable · 2023-12-01 14:35 >It's frustrating how many open source tools spend a ton of dev energy on access and control. It's crazy because I would (almost) never expose a web service to the open internet with its own home-rolled authentication anyways. I just whitelist any endpoints (i.e. /inbound-webhook) and everything else uses SSO at the reverse proxy layer ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:38 Theoretically, all of this is possible. SecureAI Tools is probably the wrong place for it though, because it focuses on providing the application layer to use pre-trained or already-fine-tuned models for regular use (i.e. inference only). There are other online tools available for training and/or fine-tuning LLMs. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:41 This is a fair point! We are open to integrating SSO. What are some popular SSO providers that the self-hosting community likes to use? I can look into how much effort it would be for us to support the most popular ones ---------------------------------------------------------------------- lowerseagate · 2023-12-01 14:44 can i wrap it as an API so i can use it with my apps? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:45 Wow, yeah we would love to get there for sure. As I mentioned in another comment on this thread, one of my main concerns is the amount of time it would take a LLM RAG system to index that much amount of data. It could probably take days to process that much data on hardware that most self-hosters use. But it is a fun challenge to tackle for sure ;) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:53 \> So is your project better or worse \[than OpenAI APIs\]? * Model output quality: We support open-source models like llama2, mistral, and many others. And open-source models today do not match OpenAI's GPT3.5/4 models, although some of them come close enough to it on popular benchmarks. * Inference speed: This depends largely on your hardware. For example, on M1/M2/M3 MacBooks we have seen inference speed to be comparable to OpenAI APIs. But on Mac Intel or Linux CPU-only machines, it is much slower. ​ In the future, we have plans to support remote inference APIs like OpenAI and Claude APIs. The advantage then would be that your chat history would be all stored on your local machine and you can have full control over it, while doing expensive LLM inference operations on OpenAI/Claude's distributed inference infrastructure. Let us know if you that would work better for your use cases. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 14:54 For others context there was discussion on GH at [https://github.com/SecureAI-Tools/SecureAI- Tools/issues/9](https://github.com/SecureAI-Tools/SecureAI- Tools/issues/9) ---------------------------------------------------------------------- rope93 · 2023-12-01 14:56 constantly getting this in LOGS Error: Ollama call failed with status code 404: model 'mistral' not found, try pulling it first the chat doesnt respond to whatever i ask it, any idea what could be wrong? ---------------------------------------------------------------------- Zoenboen · 2023-12-01 15:19 Since you asked in the OP, look at Ollama's ability to run an 'ingest' script and create a database from documents and their 'privateGPT' script that allows for RAG chats against those documents. RAG just isn't possible with ChatGPT out of the box and makes this a killer app. Make it easy to add and remove from the document library and you've got a winner. Who doesn't want to just setup a chat support for family... that also has specifics embedded should they get stuck? ---------------------------------------------------------------------- Zoenboen · 2023-12-01 15:27 Author said it uses Ollama, which provides an API end point for completions out of the box. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:28 It will for sure. And the slowness depends largely on the model you choose to use. ​ On such a machine, a smaller model like [https://ollama.ai/saikatkumar dey/tinyllama](https://ollama.ai/saikatkumardey/tinyllama) may work. Specify "saikatkumardey/tinyllama" in the org settings page to use that model. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:30 Yep, we aim to release a very basic chat-with-document feature next week, and then keep adding more capabilities to it in the coming weeks. Stay tuned! ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:31 Awesome. Let us know how it goes on Rpi. Most models may be too big to run on Rpi so please also tinker with [h ttps://ollama.ai/saikatkumardey/tinyllama](https://ollama.ai/saikatkum ardey/tinyllama) (use "saikatkumardey/tinyllama" as model string on SecureAI Tools orgs settings page) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:32 Please do. Let us know how it goes so we can improve our project from your experience ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:32 Correct. Ollama API docs: https://github.com/jmorganca/ollama/blob/main/docs/api.md ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:34 Ah you may have missed the step 6.2 of downloading a model: [https://github.com/SecureAI-Tools/SecureAI-Tools#6-post-installation- set-up](https://github.com/SecureAI-Tools/SecureAI-Tools#6-post- installation-set-up) ​ Let us know if this problem still persists after downloading the model. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 15:35 We are also planning to release a chat-with-documents feature soon. Stay tuned! ---------------------------------------------------------------------- lowerseagate · 2023-12-01 15:39 good project btw. Im still finding a perfect use case to run this over the ChatGPT. Ill let you know if anything can be improve ---------------------------------------------------------------------- lowerseagate · 2023-12-01 15:42 I just discovered the existence of Ollama, haha. now im thinking of a new project ---------------------------------------------------------------------- ovirt001 · 2023-12-01 16:33 Does this support integrated GPUs? ---------------------------------------------------------------------- AbyssalReClass · 2023-12-01 16:59 Looks cool, I'll definitely be giving this a try. Any plans to implement MemGPT support? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 17:05 Thanks. I didn't know about MemGPT -- looks neat. We use Ollama and it looks like MemGPT also supports Ollama: [https:// memgpt.readthedocs.io/en/latest/ollama/](https://memgpt.readthedocs.io /en/latest/ollama/) So my understanding is that if you want to use MemGPT, then you need to just run it separately and point it to Ollama. Then make the SecureAI Tools web service to point to MemGPT process port and it should work, right? ---------------------------------------------------------------------- nashosted · 2023-12-01 17:57 I would like to add [Ollama](https://github.com/jmorganca/ollama) to that list. It's one of the best free AI chat tools I have seen so far. ---------------------------------------------------------------------- I_EAT_THE_RICH · 2023-12-01 18:05 There are already auth services. Your home meal planner shouldn't have ANY authentication. It should focus on one thing. It's already so easy to use a proper auth, and a proper SSL solution. ---------------------------------------------------------------------- I_EAT_THE_RICH · 2023-12-01 18:06 precisely how I set things up. Terminate at the reverse proxy. ---------------------------------------------------------------------- Woke_killa · 2023-12-01 18:31 Thanks for the answer. I wanted to buy an HPE dl380 server and two Tesla P100 16GB ($2x150) in a few months to play with local AI, such as creating a chatbot in VR Quest 3 goggles, based on local API. So it will be too slow for logical conversation and the available trained models too stupid for conversation? ---------------------------------------------------------------------- Comms · 2023-12-01 18:53 Yeah, I don't know how to do any of that, I'm not a dev. I just use unraid. But it would be nice to see it on there at some point, it's a cool project. ---------------------------------------------------------------------- stuffitystuff · 2023-12-01 19:21 Days isn't really that bad (especially if it means not having to spend $10k+). It already takes a couple days to wipe a modern hard drive and do many other offline batch processes. Not everything is customer- facing and requires low latency :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 20:00 If you have a Tesla GPU, then it should be fairly fast I think ---------------------------------------------------------------------- nerdyviking88 · 2023-12-01 20:39 Can you train it against your own data, and if so, how? ---------------------------------------------------------------------- jay-workai-tools · 2023-12-01 22:02 SecureAI Tools doesn't have any built-in mechanism to train or fine- tune models. SecureAI Tools only does inference. But it is technically possible to train/fine-tune a model on your data set using **others'** open-source tooling, export to gguf/ggml format, build the Ollama model and use it with SecureAI Tools. ---------------------------------------------------------------------- Badgerized · 2023-12-02 00:09 What would be a good model if I want help with writing my code. I'm good at coding... just lazy. Rather code review than write stuff from complete scratch nowadays lol My server should be able to run it. Epyc7401p, 256gb ram. ---------------------------------------------------------------------- NewDad907 · 2023-12-02 05:59 Cool, I’ll check it out. I have LlammaGPT running locally, but it’s pretty damn slow. ---------------------------------------------------------------------- ReachingForVega · 2023-12-02 10:35 You have bots and other users trying to manipulate what sits on the front page of subs. As long as it is positive overall stress not. ---------------------------------------------------------------------- MatthKarl · 2023-12-02 13:48 Thanks for this. I just installed it and after the initial login, I got an error message saying it couldn't reach the post-login page. With the back button I then got to the page and could use it. But it seems there was some issue with that post-login. After using it, I also got a few more errors, and I posted those on Github. ---------------------------------------------------------------------- srikon · 2023-12-02 14:22 Good work Jay. While we talk about the performance, would it be an option to use embeddings+vector db to make it easy to chat with them. We are exploring that route for our use cases and would like to know your experience or thoughts. Happy to connect if you’d like to discuss. ---------------------------------------------------------------------- srikon · 2023-12-02 14:24 Would start with Azure AD. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-02 14:52 Yep, for RAG, we are planning to add vector db. I'd love to understand more about your use cases. Sending you a DM request ---------------------------------------------------------------------- jay-workai-tools · 2023-12-02 14:55 Taking a look. ---------------------------------------------------------------------- KingPinX · 2023-12-03 17:44 do you run any of these yourself? and if so in docker by any chance? I have been reading the docs on all these and experimenting since you posted about them but... they seem to be less than happy to use docker for everything. ---------------------------------------------------------------------- lilolalu · 2023-12-03 17:53 I am using GPT4All but on my laptop, per the desktop app. But I just came as far as checking which models work best for German. Haven't tried yet to deploy them as containers. ---------------------------------------------------------------------- KingPinX · 2023-12-03 17:57 ok thanks :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-08 23:32 Chat with documents feature is now available with the latest release! Please give it a go and let us know what you think https://www.reddit.com/r/selfhosted/comments/18dzo3y/secureai\_tools\_ now\_supports\_chat\_with\_documents/ ---------------------------------------------------------------------- jay-workai-tools · 2023-12-08 23:33 The chat with documents feature is now available with the latest release! Please give it a go and let us know what you think https://www.reddit.com/r/selfhosted/comments/18dzo3y/secureai\_tools\_ now\_supports\_chat\_with\_documents/ ---------------------------------------------------------------------- jay-workai-tools · 2023-12-08 23:33 The chat with documents feature is now available with the latest release! Please give it a go and let us know what you think https://www.reddit.com/r/selfhosted/comments/18dzo3y/secureai\_tools\_ now\_supports\_chat\_with\_documents/ ---------------------------------------------------------------------- jay-workai-tools · 2023-12-08 23:34 The chat with documents feature is now available with the latest release! For now, it works with a handful of documents to start with. But we also have plans to make this a background job so that it can be scaled to 100s of documents. ​ Please give it a go and let us know what you think. https://www.reddit.com/r/selfhosted/comments/18dzo3y/secureai\_tools\_ now\_supports\_chat\_with\_documents/ ---------------------------------------------------------------------- jay-workai-tools · 2023-12-08 23:34 The chat with documents feature is now available with the latest release! Please give it a go and let us know what you think https://www.reddit.com/r/selfhosted/comments/18dzo3y/secureai\_tools\_ now\_supports\_chat\_with\_documents/ ---------------------------------------------------------------------- Aziroshin · 2023-12-12 14:13 You might also want to look into Authentik and Authelia. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 22:59 Hi there! We just added this in the latest release (v0.0.2). You can now create a document collection and upload as many PDFs into it as needed. The documents are processed in the background and once processing finishes, you can create as many chats with it as needed. Quick demo: https://youtu.be/PwvfVx8VCoY Installation Instructions: https://github.com/SecureAI-Tools/SecureAI- Tools?tab=readme-ov-file#docker-compose-recommended Please try it out, and let me know how it goes. We're always looking to improve the tool so let us know if you have any feedback for us :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:00 Hi there! We just added this in the latest release (v0.0.2). You can now create a document collection and upload as many PDFs into it as needed. The documents are processed in the background and once processing finishes, you can create as many chats with it as needed. * Quick demo: [https://youtu.be/PwvfVx8VCoY](https://youtu.be/PwvfVx8VCoY) * Installation Instructions: [https://github.com/SecureAI- Tools/SecureAI-Tools?tab=readme-ov-file#docker-compose- recommended](https://github.com/SecureAI-Tools/SecureAI- Tools?tab=readme-ov-file#docker-compose-recommended) Please try it out, and let me know how it goes. We're always looking to improve the tool so let us know if you have any feedback for us :) ---------------------------------------------------------------------- jay-workai-tools · 2023-12-16 23:05 Hello again :) We have since added the ability to use OpenAI compatible APIs: [https://github.com/SecureAI-Tools/SecureAI-Tools?tab=readme-ov- file#use-with-openai-or-openai-compatible- apis](https://github.com/SecureAI-Tools/SecureAI-Tools?tab=readme-ov- file#use-with-openai-or-openai-compatible-apis) ​ Please try it out, and let us know if you run into any issues :) ---------------------------------------------------------------------- audero · 2023-12-17 00:39 Brilliant. I’ll give it a go and report back. ---------------------------------------------------------------------- audero · 2023-12-17 01:59 I successfully built the image for arm64, and could run mistral. It's faster than I expected. But I'm not trying to use OpenAI's servers as an upstream model for SecureAI, but rather, use SecureAI as a self-hosted server for a client app that uses the OpenAI API. In other words, have SecureAI pretend to be OpenAI. 🙂 ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 02:13 That's amazing. Did you build an arm64 image of Ollama or is it for SecureAI Tools (web app)? ---------------------------------------------------------------------- audero · 2023-12-17 04:35 There already is an arm64 image for Ollama and the other containers we need, so all I had to do was build the SecureAITools image. It didn't require any changes, I just had to specifc the arch as arm64 for the base image in the Dockerfile. ---------------------------------------------------------------------- jay-workai-tools · 2023-12-17 16:01 Gotcha. I haven't spent enough time understanding how to publish multi-arch docker images myself yet, but I would love a contribution from the community if someone can show me how I can publish docker images for both amd64 and arm64