Self-hosted Large Language Model reddit / selfhosted ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 17:30 Hello self-hosters. I sometimes use OpenAI's GPT3.5, and as you can tell; it's closed source and hosted in OpenAI's servers. I came across [this video](https://www.youtube.com/watch?v=GyllRd2E6fg)(not my video, obviously), speaking of an open source LLM called [Dolphin](https://hu ggingface.co/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF). I thought why not try that, since it is open source and uncensored.. I am wondering, has any of you ever self-hosted an LLM? If yes, my main question is, what model was it and how much ram does it require? I have a pretty decent CPU and GPU, but I only got 16Gb of ram that I can upgrade to 32Gb, I got the impression that those models require significant amount of ram.. Thanks for taking the time to read this. Looking forward to your replies. ---------------------------------------------------------------------- EndlessHiway · 2023-12-22 17:35 It would work on 32 gigs of ram but it will be kind of slow. Jim's Garage on Youtube just did a video on this that is really good, plus he is a regular here on Reddit. ---------------------------------------------------------------------- IntrovertedFL · 2023-12-22 17:39 Checkout /r/LocalLLaMA Great discussions about running Local LLM's on consumer hardware etc. ---------------------------------------------------------------------- LightningSaviour · 2023-12-22 18:03 Pretty decent is how much vram exactly? Because even the best consumer market GPUs will not be able to run it as efficiently as openAI servers, it'll be much slower for inference ---------------------------------------------------------------------- ElectronicFinance603 · 2023-12-22 18:03 Dolphin is a very good llm but it's also pretty heavy. If you want to try some local llm, you can try to host a docker of Serge (you can find it on GitHub). It's a frontend similar to chatgpt, but with the possibility to download several models (some of them are extremely heavy other are more light). Note: to easily understand how much a model is heavy, look at the value at the end of the name (7b, 13b, 40b ...) higher the value, higher the ram and power you need to run. (On 16gb, probably a 7b can run) ---------------------------------------------------------------------- ElectronicFinance603 · 2023-12-22 18:15 For which model i have run, on a ryzne 5 5600 with 32gb of ram I have tested mistral 7b (general caht) and llama2-code-13b (tuned for help on code). You can find a lot of model onm the page of the project ollama: [https://ollama.ai/](https://ollama.ai/) Other interesting project that you could take a look are: Serge: [https://github.com/serge-chat/serge](https://github.com/serge- chat/serge) Cheshire Cat: [https://cheshirecat.ai/](https://cheshirecat.ai/) ---------------------------------------------------------------------- centomila · 2023-12-22 18:19 Take a look to to LM Studio. [https://lmstudio.ai/](https://lmstudio.ai/) It show right away what can works on your machine and what not. On YT this channel [https://www.youtube.com/@matthew\_berman](https:// www.youtube.com/@matthew_berman) is all about LLM. His thumbnails and titles are a little clickbaity and annoying, but is good for starting on the argument. ---------------------------------------------------------------------- _3xc41ibur · 2023-12-22 18:19 I've ran LLaMA 2 on 64GB RAM and a GTX 1050 Ti. Super slow. But now I have a Tesla P40. Performance might improve a little more, but I haven't tested the LLM on it yet. Just been using it on a local Stable Diffusion instance which works quite well. ---------------------------------------------------------------------- mirisbowring · 2023-12-22 18:20 And most important question: is it consuming the resources all the time or only during „requests“? If my server is used 100% all the time, it becomes veerry expensive and even a gpt4.0 abo seems cheap in comparison ---------------------------------------------------------------------- rwclark88 · 2023-12-22 18:45 I'm into self-hosted LLMs pretty heavily. In my opinion, the best platform for hosting LLMs is: [https://github.com/oobabooga/text-generation- webui](https://github.com/oobabooga/text-generation-webui) Although it changes every day and depends on your specific use case, I'll go out on a limb and say that the following model is the best open source one currently: [https://huggingface.co/mistralai/Mixtral-8x7B-Instruct- v0.1/tree/main](https://huggingface.co/mistralai/Mixtral-8x7B- Instruct-v0.1/tree/main) Of course, that full size version is too large for most on-site users, so I recommend you check out this guy's quantized versions of mixtral instruct (he's well known in the local LLM space for making excellent quantized versions): [https://huggingface.co/TheBloke](https://huggingface.co/TheBloke) Good luck! ---------------------------------------------------------------------- tshawkins · 2023-12-22 18:48 I self-host dolphin-mixtral (instruct) on a 64gb server machine that runs on a nuc12pro (i7 12th gen), performance is OK I use ollama and operate it in "server" mode, so the nuc is accessible via my network on my dev laptop. ---------------------------------------------------------------------- Compux72 · 2023-12-22 19:11 Take a look at this: https://github.com/ollama-webui/ollama- webui/blob/main/docker-compose.yml In my case mixtral uses 1GB of VRAM and 50GB of RAM, deployed on k8s. ---------------------------------------------------------------------- lilolalu · 2023-12-22 19:51 If you are planning to use the LLM in Nextcloud, the way to go forward is using LocalAI. https://github.com/mudler/LocalAI What is interesting is that LocalAI is sort of a unifying API bridge between various selfhosted AI models (LLM, Image Generation, TTS, Speech to Text etc.) To the front-end it acts as an OpenAI compatible API, pretending to be ChatGPT, DALL-E, while at the backend side it uses stable diffusion, lama.cpp and others, with the models of your choice which each have advantages and disadvantages. To host something comparable to ChatGPT you need a lot of VRAM, to be able to use something like the 40B models. But smaller models like Mixtral 7B are useful and capable and run on less ressources, 12gb VRAM is enough. but you can't really compare them to an general purpose ai like chatgpt. If you have a certain focus you are interested in, you can usually get models specialized for that purpose i.e. coding. ---------------------------------------------------------------------- startages · 2023-12-22 19:51 Looking at the size of this model, you'd need around 24GB GPU or more to run this locally with decent speed. I've tried a model with the same size on my RTX 4070 8GB with 32GB RAM and it took 6 minutes to return an answer. So, don't rely on RAM helping with speed, it can help run the model, but will be very slow. You'd probably need the current best GPU available on a consumer device to run such a large model (eg, RTX 4090..etc). ​ If you don't mind using the command line, you can run the model using both GPU+CPU, you can use a library like [llama.cpp](https://github.com/ggerganov/llama.cpp) or if you want to use python [llama-cpp-python](https://github.com/abetlen/llama-cpp- python). If you need an interface, check [LM Studio](https://lmstudio.ai/). **Note:** before spending time on this, just be aware that this specific model will not run well on your hardware, or might not run at all. You'd be better off trying a quantized version of a 7B model, or something around that size, you'd get something usable at least. ---------------------------------------------------------------------- AlukardBF · 2023-12-22 20:31 If you use Q4 quantization, you can offload entire 13b model on 12gb vram. ---------------------------------------------------------------------- ElectronicFinance603 · 2023-12-22 20:45 I have a Rx 6750.. and all the guides that I have found for windows use cuda (so are Nvdia only). For now rocm work only on linux ---------------------------------------------------------------------- rickysaturn · 2023-12-22 20:58 I see a mention of https://serge.chat in another comment but giving a +1 regardless. Some months ago I was curious about running something locally and this offered interesting results quickly. I haven't returned to this with any great depth recently but will note this: - I've used mostly a 7b - conversations are slow, consuming tons of cpu - Running it on a Dell Optiplex 7010 (Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz, 16Mb RAM), Ubuntu 22.04.2 Your question has prompted a lot of great comments here and reminds me I'm overdue for looking at this more closely! ---------------------------------------------------------------------- Fire597 · 2023-12-22 21:23 I tried GPT4ALL once. You can host model with ram. From 4GB to 16GB depending of the model. It was working well. ---------------------------------------------------------------------- hedonihilistic · 2023-12-22 21:24 I use ooba with the openai extension that gives me an openai compatible endpoint to be able to use my local model in any application just like I would use the chatgpt API. I have my own simple applications that can switch between my local AI and chatgpt as needed. I have a pair of RTX 3090s with a 5900x and 128gb ram. The processor and ram is not as important as the gpus, and more importantly total available VRAM, if you want to run fast inference. I can run quantized models with up to around 70B parameters, but a lot of really amazing and intelligent stuff can be done with smaller models on a single GPU. Apple silicon is apparently pretty good as well and may be cheaper but a little slower for large models, but I'm not very familiar with apple stuff or CPU inferencing. I can load much larger models and run them on my CPU and ram but it's too slow. ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 21:25 Thank you. ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 21:28 I reckon what they don't have in common is the number after Q. They all have 7b. They also have a layout showing how much RAM each model will potentially take. I downloaded the one with 24Gb ram hoping it might run on my 16 gigs without having to upgrade. Thanks tho. ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 21:29 Thanks. I will give Serge a look! ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 21:31 Thanks! ---------------------------------------------------------------------- dj_raidar_vip · 2023-12-22 21:42 I have been testing an idea utilizing the Mixtral model, Ollama, Home Assistant, and ESPhome, but I just bought a Tesla K80 thinking it can replace my Tesla P4 in my HP DL380e G8 so that I can have a whole local stack of programs in AI. But I'm currently trying to figure out if it would be good to build a dedicated rack mount unit for AI (Stable Diffusion XL, Ollama + LLMs, AutoGen, Jupyter Notebook, etc.) because I would really love to make an AI that can not only control my house when asked, but also be able to carry out tasks for me in AutoGen and read data to give me more helpful tips on how to live. It's a large project, yes, but I really want to build something that could be really fun to basically be my second brain. Heck, I've considered figuring out a way for the AI I'm making to read my Obsidian vaults and putting it into memory by hosting MemGPT. ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 21:42 Thanks a lot for your reply! Which platforms have tried beside Ooba? I am after an uncensored model, I heard that Dolphin is great for such work cases. But, I'll give the one you recommended a try, would it run on my 16 gigs of ram machine? I'm new to hosting LLMs, just heard of it today. ---------------------------------------------------------------------- AlukardBF · 2023-12-22 21:46 I have the exact same gpu. You can use ggml models in llama.cpp and koboldcpp with it on Windows. text-generation-webui would not work, because it needs pytorch that don't work on windows with rocm for now. In llama.cpp and koboldcpp you can use openblas to offload model to gpu, but it would not work in a full speed (but better than cpu of course). If you would like to use hipblas to use full potential of your gpu you will need to commpile koboldcpp-rocm fork yourself, or llama.cpp. I can share my script with you, but the most difficult part is to build Tensile for gfx1031 (our gpu). The good starting point is [this comment](https://www.reddit.com/r/Loc alLLaMA/comments/16d1hi0/comment/jzmvbfc/), see the step 8. But I should warn you that this is quite difficult. For now you can try koboldcpp with openblas download koboldcpp_nocuda.exe from [here](https://github.com/LostRuins/koboldcpp/releases/tag/v1.52.2) ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 22:11 Yeah. Exactly. I think it just sits idle until you request something from it. But, that wouldn't be the case RAM-wise. ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 22:17 I actually prefer using the console rather than UIs. Thanks for the links, and most importantly, the note. ---------------------------------------------------------------------- _iamhamza_ · 2023-12-22 22:24 Interesting. I think I might give this one a go. I've got an i7-13700H alongside an RTX4060, I think I have a good chance of running something in my personal machine. But, I am wondering if that model is as censored as ChatGPT? I wouldn't want to go through the hassle of downloading and installing a model just to tell me that it can't answer such questions... ---------------------------------------------------------------------- error_fourzerofour · 2023-12-22 22:56 K8s? ---------------------------------------------------------------------- Compux72 · 2023-12-22 23:08 Kubernetes ---------------------------------------------------------------------- byttle · 2023-12-22 23:32 i've been hoping for something like Stable Matrix for LLM's ---------------------------------------------------------------------- senectus · 2023-12-22 23:33 Llamafile is the easiest ---------------------------------------------------------------------- chibiace · 2023-12-22 23:40 i use koboldcpp with sillytavern as frontend, running dolphin-2.2.1-mistral-7b.Q5_K_M.gguf on a nvidia 1070 (8gb vram) ---------------------------------------------------------------------- myself248 · 2023-12-22 23:52 This was dead-simple to set up: https://github.com/getumbrel/llama-gpt ---------------------------------------------------------------------- rexyuan · 2023-12-23 00:25 This is the answer ---------------------------------------------------------------------- sani999 · 2023-12-23 01:11 Hey, an excuse to have 64gig of ram lol ---------------------------------------------------------------------- NewDad907 · 2023-12-23 03:23 I looked at LM Studio. I’m not clear if you can train and grow a LLM with it though. ---------------------------------------------------------------------- R_X_R · 2023-12-23 03:34 I’ve seen kobold/silly tavern posted a lot and had a brief go at KoboldCPP. This seems more RP based yeah? Loading it immediately seems to throw me right into a new D&D style text adventure. Are these two geared towards this? ---------------------------------------------------------------------- Mugen0815 · 2023-12-23 03:36 Using this: [https://github.com/Atinoda/text-generation-webui- docker](https://github.com/Atinoda/text-generation-webui-docker) I was able to run LLMs with GPU-support under WSL2 with my 3080ti. Not very fast though, but I didnt test much. ---------------------------------------------------------------------- centomila · 2023-12-23 03:43 **As far as I know**, you can't do that. The primary uses are: * Trying and managing models from Hugging Face * Local chatting * Using it as a local server API (the syntax is the same as OpenAI's) I believe training and expanding a LLM requires extremely powerful hardware. If is doable with a desktop GPU is beyond my knowledge. ---------------------------------------------------------------------- Sub_Atomic1 · 2023-12-23 03:45 How would you think a 5b parameter model would run on 32gb ram and a 1060? I doubt well but I'm curious to try now ---------------------------------------------------------------------- _3xc41ibur · 2023-12-23 03:48 It wouldn't be worth it. Llama 2 is 7b and the speed was not impressive. One or two minutes for simple prompts. Maybe you would have a different experience though, not sure ---------------------------------------------------------------------- Sub_Atomic1 · 2023-12-23 04:02 Yeah I would guess that it wouldn't be worth my time as you probably are right, i might be making coffee and drink it all before it finishes. ---------------------------------------------------------------------- nebajoth · 2023-12-23 04:08 There's a dolphin fine-tune of Mixtral 8x7b, GGUF so it can run on CPU. By TheBloke. Q4 should run in 32gb of ram. https://huggingface.co/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF ---------------------------------------------------------------------- mrorbitman · 2023-12-23 04:16 LlamaGPT if you want low system requirements for a very functional llm ---------------------------------------------------------------------- chibiace · 2023-12-23 04:17 been experimenting with getting it to do code completion, can be made to just answer questions pretty easily. ultimately i'm thinking to dump sillytavern and just make my own user interface, without the computing power and memory its hard to have any sort of long conversation anyway so it does seem well suited for simple question response applications. one upside is that sillytavern lets you change alot of settings which can allow you to learn more about how to get what you want. ---------------------------------------------------------------------- PowerfulAttorney3780 · 2023-12-23 04:32 I just tried GPT for all on my PC and it ran amazingly. I have a 3080 and an I-79700K so nothing too special 32 gigs rim ---------------------------------------------------------------------- wegwerfen · 2023-12-23 04:42 Also /r/Oobabooga And if you are interested in RP check out this front end: /r/SillyTavernAI/ https://github.com/SillyTavern/SillyTavern Currently I am running: TheBloke_Airoboros-L2-13B-3.1.1-GPTQ on oobabooga with the SillyTavern front end On a RTX 3060 with 12GB VMEM Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz 32gb system RAM ---------------------------------------------------------------------- travellingtechie · 2023-12-23 05:49 Check out Ollama. Here is a great intro video (this guy has lots of great AI videos). https://youtu.be/rIRkxZSn-A8?si=hK_eBBERBInbEAtI ---------------------------------------------------------------------- Nixellion · 2023-12-23 07:08 Or LoneStriker for exl2 quants for exllama v2. Its stupid fast compared to GPTQ and just like GGUF it supports various compression levels, from 2 to 8 bit. I'm currently running OpenHermes 2.5 in 6 bit, its much faster and more coherent than 4bit GPTQ. ---------------------------------------------------------------------- za_organic · 2023-12-23 07:13 Arguably, by the time you are running q4 you might as well run a smaller model without quantization. ---------------------------------------------------------------------- Agile_Ad_2073 · 2023-12-23 07:40 Olama is the easiest ---------------------------------------------------------------------- rjames24000 · 2023-12-23 09:19 you seem very knowledgeable what would touch recommend to someone who say has a 24 core i9 with 64gb of ram but only a single 3090 ---------------------------------------------------------------------- lilolalu · 2023-12-23 09:41 I am not sure why people here are discussing the machines RAM requirements. Even the 7B models are unbearable slow on CPU so if you don't just want to toy around with these tools but productively use them, you need GPU and VRAM. Regular RAM is not fast enough for decent speeds. The cheapest GPU with the highest VRAM to my knowledge are the Intel ARC A770 with 16gb for <350€ unfortunately Intel is not well supported with the most inference engines and the Intel GPU's are slower. Plus, the Intel ARC GPU have a really bad idle power consumption of 18W or so. For AI a Nvidia GPU is still the way to go, in terms of VRAM, the 3070 with 16gb (450€) is the entry level model ATM. ---------------------------------------------------------------------- lilolalu · 2023-12-23 09:53 Depends on the capability of your GPU to go idle. An Nvidia rtx49090 can consume 300w under load but has an idle 3-4W consumption. I don't think what you are asking taking into consideration that we are not seeing realistic subscription prices with AI services. OpenAI etc. are losing millions at the moment with the prices they offer their services for. So this pricing has a foreseeable end when enough people have integrated ai as an indispensable tool in their daily workflow. Another reason is that you are basically not allowed to use these AI models in a corporate environment for legal reasons. ---------------------------------------------------------------------- ifndefx · 2023-12-23 11:05 I'm about to do this after Christmas 🎄🎄 Thanks. ---------------------------------------------------------------------- AutomaticDriver5882 · 2023-12-23 15:03 check out this sub r/localllma ---------------------------------------------------------------------- caffeine947 · 2023-12-23 15:34 I self host mistral 7b, it is by far the best 7b model out there. Takes about 9gb vram, I have an rtx 3080. I use a gguf quant (q5 iirc). There are quantized models out there for just about everything thanks to the bloke. I get around 5 tokens per sec with it. ---------------------------------------------------------------------- caffeine947 · 2023-12-23 15:36 Also, check out vLLM for hosting models locally if you want an openai compatible API that scales well ---------------------------------------------------------------------- lilolalu · 2023-12-23 17:10 Mixtral 7b is better ---------------------------------------------------------------------- caffeine947 · 2023-12-23 17:26 Mixtral 8x7b is definitely better but it's the size of a 40b model so won't fit in my vram unless I run at q1 or q2 which sucks ---------------------------------------------------------------------- lilolalu · 2023-12-23 17:43 Works on a 12gb 4070 so I am surprised it doesn work on the 3080 ---------------------------------------------------------------------- caffeine947 · 2023-12-23 17:53 Maybe you are offloading some to ram. It requires 15.6gb vram for the q2 unless you offload some of the layers ---------------------------------------------------------------------- lilolalu · 2023-12-23 18:00 Do you have a tool to check these kind of things? I am usually going with the defaults that are suggested and don't explicitly check for my GPU... ---------------------------------------------------------------------- caffeine947 · 2023-12-23 18:03 Sorry, I dont... i just guess at settings and check nvidia-smi to see how much it's using. The vram requirement comes from the bloke model card https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF ---------------------------------------------------------------------- V0dros · 2023-12-23 20:38 Is it open source? ---------------------------------------------------------------------- centomila · 2023-12-23 22:10 No. Something is open but not the main app. They say this on their website: *The app does not collect data nor monitor your actions. Your data stays local on your machine. It's free for personal use. For business use, please get in touch.* I didn't notice any strange network activity from my machine while using it, but I invite you to do your own research if you are worried about that ---------------------------------------------------------------------- Vampire_Duchess · 2023-12-24 01:08 Do you know if there is something for pdfs? I need to stack a bunch of pds and I would like to use something to ask for answers based on those pdfs is there something similar ? ---------------------------------------------------------------------- cfurini · 2023-12-29 13:17 THANK YOU! I just installed your recommendation in a Windows VM that runs Ubuntu and it worked beautifully, although chat responses are very slow. Now, my next steps is find a model that replies faster but still works for me. ---------------------------------------------------------------------- cfurini · 2023-12-29 13:40 I relate a lot to your project and I am working in a very similar direction. I already have Home Assistant plugged in my house for a while. I am now learning how can I use AI to build the "second brain" as you mentioned. ---------------------------------------------------------------------- cfurini · 2023-12-29 13:44 By any chance you integrated it ooba with Home Assistant using the home assistant ChatGPT integraton?