As of today what would be the best AI to self-host on Linux reddit / selfhosted ---------------------------------------------------------------------- kremata · 2023-11-28 03:10 I will mainly use it for help in coding and general questions. ---------------------------------------------------------------------- darklord451616 · 2023-11-28 03:23 There's a desktop app called LM Studio, get that for UI and ease of use. Get 4 bit GGUF models from TheBloke on hugging face (LM Studio gives you a nice interface to download them directly). For Coding CodeLlama is good for python, For general questions, The Mistral series of models are good. If you have a beefy setup for inference, prefer using 13B+ models instead of 7B models in the Llama family, and it's derivatives. ---------------------------------------------------------------------- TheApadayo · 2023-11-28 03:28 I would check out the [models megathread](https://www.reddit.com/r/Loc alLLaMA/comments/185770m/models_megathread_2_what_models_are_you_curre ntly/) over at /r/LocalLLaMA. The one I hear mentioned the most recently is CodeLlama-34b for your use case. ---------------------------------------------------------------------- Still-Snow-3743 · 2023-11-28 03:30 You got half a terabyte of ram? ---------------------------------------------------------------------- kremata · 2023-11-28 03:36 I'm trying to get this one but I can find a link to install on Arch. On Windows it seems easy but on Linux you need to on Discord, ask for an invite, etc... I don't know Discord, never use it so I'm lost in there. ---------------------------------------------------------------------- RedditSlayer2020 · 2023-11-28 04:42 None, don't host it yourself. You will need a ton of ram and a really powerfull PC. Additionally your electricity bill will go up significantly ---------------------------------------------------------------------- beholdthepineapple · 2023-11-28 04:50 I do indeed. What’s the next step? ---------------------------------------------------------------------- noobgolang · 2023-11-28 04:52 You can check out project [https://nitro.jan.ai/](https://nitro.jan.ai/) source: I'm the project maintainer ---------------------------------------------------------------------- GJT11kazemasin · 2023-11-28 05:19 Text Generation WebUI, feature-riched. Download models by yourself. ---------------------------------------------------------------------- Still-Snow-3743 · 2023-11-28 05:19 Well step one is brag, thats what my buddies with far too much money have done lol It is my understanding LLM Studio, discussed on here, is the way to go ---------------------------------------------------------------------- beholdthepineapple · 2023-11-28 05:20 Things got bad in the Bay Area and I got a severance and a beast of a cluster that I have at home and I’m looking for serious things to do with it. ---------------------------------------------------------------------- darklord451616 · 2023-11-28 06:10 Hey, this looks cool. Does this use Text Gen Rust backend? The one made by HF ---------------------------------------------------------------------- HoustonBOFH · 2023-11-28 06:19 >on Linux you need to on Discord, ask for an invite, etc... What the ... Sigh. Just when I think developers can not have a more bone headed dependency... ---------------------------------------------------------------------- HoustonBOFH · 2023-11-28 06:20 You do know this is selfhosted, right? People tell us that what we do is "not a good idea" all the time. ---------------------------------------------------------------------- MinallWch · 2023-11-28 07:15 Check ollama. ---------------------------------------------------------------------- dbossman11 · 2023-11-28 08:18 Ollama is great! ---------------------------------------------------------------------- noobgolang · 2023-11-28 11:43 no its coded entirely in c++ ---------------------------------------------------------------------- kjarkr · 2023-11-28 13:12 How does this differ from https://github.com/mudler/LocalAI ? ---------------------------------------------------------------------- RedditSlayer2020 · 2023-11-28 13:27 Im all for selfhosting but LLMs is just stupid. But go for it you do you and evite the ride ♥ ---------------------------------------------------------------------- terAREya · 2023-11-28 15:41 Cant say enough good things about Serge ​ https://github.com/serge-chat/serge ---------------------------------------------------------------------- Annual-Advisor-7916 · 2023-11-28 16:22 How good are such self hosted AIs really, in term of coding? Same level as GPT 3.5? ---------------------------------------------------------------------- kremata · 2023-11-28 16:24 Look interesting ---------------------------------------------------------------------- darklord451616 · 2023-11-28 16:28 The non quantized version of 13B plus llama2 derivatives like CodeLlana is good enough (compared to gpt3.5 please refer to HF benchmars for accurate figure) in fp16/bf16 mode, tps throughout will depend on hardware like top tier consumer GPUs like a 4090. One inference call will saturate the entire VRAM in a single go for non quantized models, so yeah, homelabbers will need a dedicated GPU for inference if other things are not to be bothered. ---------------------------------------------------------------------- kremata · 2023-11-28 16:33 Thanks ---------------------------------------------------------------------- terAREya · 2023-11-28 16:51 its great and has active development. Hit up the discord, lotta helpful folks in there ---------------------------------------------------------------------- Annual-Advisor-7916 · 2023-11-28 16:57 Wow, I always thought that these open models are far less capable, there are tons of LLMs better than GPT4.0 (haven't researched if open or not right now), and to be honest, GPT3.5 already isn't too bad for "writing" well specified code snippets. I mean a 4090 is expensive and I'll pass on that for now, but given how fast hardware improves, I bet selfhosting a LLM will be far more appealing in the near future. Cool stuff ---------------------------------------------------------------------- juanfdo82465 · 2023-11-28 17:37 Agree people be selfhosting to save money in subscriptions but then spend a 4090 and tons of energy to use an LLM from time to time ---------------------------------------------------------------------- viral-architect · 2023-11-28 20:51 Discord is the freaking devil, man. I am in multiple software vendor discords because it's the only way to get support.