How to Host an Uncensored AI Model for Cheap reddit / selfhosted ---------------------------------------------------------------------- Successful-Carry-461 · 2023-10-09 09:00 I am building an app and need to use uncensored model such as - [https://huggingface.co/georgesung/llama2\_7b\_chat\_uncensored](https ://huggingface.co/georgesung/llama2_7b_chat_uncensored) [https://huggingface.co/mistralai/Mistral-7B-Instruct- v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) Its expensive to host it on hugging face inference API. Is there any other alternative which is cheaper or any platform which provides usage based APIs for the same? Thanks in advance. ---------------------------------------------------------------------- EndlessHiway · 2023-10-09 10:37 Is this some kind of sex thing? ---------------------------------------------------------------------- c0n5pir4cy · 2023-10-09 10:54 I'm not aware of any platform with the same features as HuggingFace. If you're happy to use something quantized and aren't too worried about performance you should be able to get it running in [https://git hub.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) locally. There is also [https://www.bellard.org/ts\_server/](https://www.bellar d.org/ts_server/) which gives a HTTP server but when I was last looking into it I couldn't find instructions to convert a model to be compatible with it. If you still want a hosted instance but don't mind doing some work to build a docker container or server you could look into hosting on a public cloud - there are some which have low cost on-demand GPU instances like [vast.ai](https://vast.ai) (GPU rental from other people), [paperspace.com](https://paperspace.com) (they seem to have some free instance, but I don't know how these work) or [lambda.ai](https://lambda.ai). ---------------------------------------------------------------------- autisticit · 2023-10-09 11:06 OpenAI threatened to shutdown my chatgpt plus account after using it to correct some texts describing the arrival of our child. Just because it mentioned breastfeeding... ---------------------------------------------------------------------- Obliterous · 2023-10-09 11:32 https://github.com/getumbrel/llama-gpt ---------------------------------------------------------------------- iiiiiiiiiiip · 2023-10-09 12:28 Host it locally, this is selfhosted afterall. 3060 non-TI is great value with 12GB of VRAM. Otherwise look at used 3090's, they have 24GB of VRAM and are good value. ---------------------------------------------------------------------- Successful-Carry-461 · 2023-10-09 12:41 Thanks for your suggestions! I've looked into the options you and others have mentioned. However, hosting a custom model on the cloud seems like it could run into a few hundred dollars, which is a bit steep for me at the moment. I'm leaning more towards a pay-as-you-go API service with uncensored models, as it seems to fit my budget better. Unfortunately, I haven't come across such a solution yet, but I'm on the lookout for it. Any recommendations would be greatly appreciated. ---------------------------------------------------------------------- Puptentjoe · 2023-10-09 12:49 Yikes. ---------------------------------------------------------------------- lilolalu · 2023-10-09 13:47 Vast.ai is much cheaper. You have to check how much GPU hours you need and check when it's cheaper to buy your own rtx4090 ---------------------------------------------------------------------- c0n5pir4cy · 2023-10-09 13:48 Yeah - I don't think many exist and the ones that do are quite expensive. If you use llama.cpp you might be able to get it running on a standard cloud instance (CPU only) on a cloud provider. For example Hetzner will do an 8GB instance for around $15 a month which you should be able to run a Q8 version of the models you posted - you might need to build some things around it, there are examples of servers in the llama.cpp repository but they would likely just be open without authentication etc. You could put this behind an Nginx/HAProxy with authentication to secure it. It's quite a bit more effort than other solutions especially if you've not worked with the tools before. It's the lowest cost way I can think of doing it at the moment and there are likely to be providers cheaper than Hetzner out there. ---------------------------------------------------------------------- pelic4n · 2023-10-09 14:14 When comes to to running an LLM locally, something like Oobabooga's WebUI is something very easy to run locally with just CPU/RAM models if you don't have a good GPU. As provides an API that can be used locally, or across the web depending on configurations. https://github.com/oobabooga/text-generation-webui The installation guide is pretty straight forward to get it up and running, but keep it mind that it will require some time investment to understand the config side of things. In general though, starting with a simple GGUF model, you can get it up and running in a matter of minutes. A good no nonsense LLM that doesn't need a Jailbreak would be something like: https://huggingface.co/TheBloke/MythoMax-L2-13B-GGUF You can find other GGUF models just by searching hf as well. (https://huggingface.co/models?search=GGUF) Taking the time to deep dive into the below subreddits can also provide quite a bit of insight into configurations and setup: https://www.reddit.com/r/localLLaMA/ https://www.reddit.com/r/PygmalionAI/ https://www.reddit.com/r/SillyTavernAI/ https://www.reddit.com/r/faraday_dot_dev/ My suggestion: - Install the Text Gen webUI - Find a good GGUF model that you can run locally. - **MAKE SURE YOU UNDERSTAND THE PROMPT TEMPLATE FOR THE MODEL YOU ARE USING.** - This is usually found in the hf model card, and can occasionally be provided via the webUI when the model is loaded. - Test it out with different configs and parameters to find what works for you. Outside of that, if you want to experiment with your local model even further, you can connect it to a separate chat interface like SillyTavern. Another thing to learn, but being able to have a character as your AI chatbot, especially for simple mundane text editing and conversation, seems to be a popular option. The most interesting aspect is the ability to use large LLM models to create Text Adventure/RPGS that are all locally generated. You can: - Install a separate UI like SillyTavern or Faraday. - Find a Character Card that works for your usage. ---------------------------------------------------------------------- reklis · 2023-10-09 15:51 Pick up a gaming laptop on sale or off Craigslist. Something with a 30xx Nvidia card in it and an x86 processor would do. Install Linux, disable the lid so you can close the lid and keep it running as a server. Run llama-gpt with cuda support. ---------------------------------------------------------------------- SeanFrank · 2023-10-09 16:31 I wonder if this still would have been a problem if you used the new PC term "Chestfeeding" instead ---------------------------------------------------------------------- Toutanus · 2023-10-09 16:50 AI now with US puritanism. ---------------------------------------------------------------------- kastmada · 2023-10-09 17:41 The easiest and fastest is ollama.ai Fun one with some edge is : https://github.com/oobabooga/text- generation-webui ---------------------------------------------------------------------- Cvalin21 · 2023-10-09 18:31 Some how not surprised ---------------------------------------------------------------------- fireshaper · 2023-10-09 21:40 Do you need to host it in the cloud? If you are just using it for yourself try using Kobold or LM Studio. ---------------------------------------------------------------------- _Mr-Z_ · 2023-10-09 22:24 Might I suggest faraday? It lets you download models, load custom models and it works really well for me, those two models you linked look to be the smallest ones I've tried (they're 7B, like 3-4GB memory usage or GPU vram usage in faraday), and it runs 100% locally, all you pay for is the power it takes to run it, no subscriptions or anything. Praying I don't get shot for putting a link here, "faraday.dev" is the website. ---------------------------------------------------------------------- nashosted · 2023-10-10 00:05 It’s literally an email harvester. No download links anywhere. ---------------------------------------------------------------------- AllowMe2Retort · 2023-10-10 02:32 Why a laptop and not just a graphics card? ---------------------------------------------------------------------- toidicodedao · 2023-10-10 05:35 Consider using a serverless service like mystic.ai or runpod. It’s billed by second, so if you only use the endpoint occasionally, the cost is minimal (2-20$/month). If you’re using a popular model, can also check together.ai, they provide hosted API for some popular models. Pricing is okay in my opinion. ---------------------------------------------------------------------- utopiah · 2023-10-10 12:07 I'm curious but did you start a spreadsheet and run the numbers? I bet that it depends on how many queries you need and your expertise : * HuggingFace is probably best if you do not have the hardware, know how and need just 100 queries or so. Yes you pay but you use and stop as you wish. * Renting GPU time on AWS/Azure/OVH/etc is probably best if you need 10000 queries and are familiar with Docker * Buying a desktop with GPU is probably best if you need more query over a longer period, so more sparse usage The numbers are **WRONG** though so it means you need to actually put the correct price and number of queries over time then pick. Asking others without at least estimating usage won't help much IMHO. ---------------------------------------------------------------------- utopiah · 2023-10-10 12:08 PS: this isn't my expertise but I did write [https://fabien.benetou.fr /Content/SelfHostingArtificialIntelligence](https://fabien.benetou.fr/ Content/SelfHostingArtificialIntelligence) so did tinker with dozens of models locally. ---------------------------------------------------------------------- jogai-san · 2023-10-10 13:11 Do you know of what's available to run locally for video super- resolution? I'm thinking of upscaling & denoising my old home recordings. ---------------------------------------------------------------------- reklis · 2023-10-10 13:17 Because it’s cheaper than building a desktop pc ---------------------------------------------------------------------- utopiah · 2023-10-10 13:32 I have never tried that but I'd check [https://ffmpeg.org/ffmpeg- filters.html#sr](https://ffmpeg.org/ffmpeg-filters.html#sr) and similar filters e.g [https://superuser.com/questions/1547426/using- ffmpeg-to-denoise-reduce-film- grain](https://superuser.com/questions/1547426/using-ffmpeg-to- denoise-reduce-film-grain) ---------------------------------------------------------------------- jogai-san · 2023-10-10 14:32 Well, looked around and found this: https://github.com/ONdraid/reve Maybe that's something I can use. ---------------------------------------------------------------------- AllowMe2Retort · 2023-10-11 03:21 Oh yeah fair, I just assumed the sort of people who are setting up home labs would probably have an old desktop lying around tho. I've got like 5 🙄 ---------------------------------------------------------------------- _Mr-Z_ · 2023-10-12 09:07 the website using a device detection might be the case, checked it on my computer just now and there are download options, gonna toss an imgur link of my view here ​ https://imgur.com/a/KHiOcN0 ---------------------------------------------------------------------- AI_from_Ukraine · 2023-12-27 13:15 Hey, I'm also interested in the same stuff. Could you let me know if you found an appropriate host? I'm thinking about [together.ai](https://together.ai), it is fast and cheap but I'm not really familiar with this platform.