Why use OpenAI's ChatGPT3.5 online service, if you can instead host your own local llama? reddit / selfhosted ---------------------------------------------------------------------- innocuousAzureus · 2023-09-28 07:58 What are the pros and cons of using OpenAI's unpaid service, if you could run e.g. a 13B or 30B model yourself? Is it somehow better on their website? ---------------------------------------------------------------------- Slendy_Milky · 2023-09-28 08:05 Simple answer : Speed ---------------------------------------------------------------------- FatalV0rt3x · 2023-09-28 08:10 I tried using the GPT4ALL model recently, but my server isn't powerful enough yet, maybe once I chuck a decent GPU in it, but for now I'll stick with OpenAi If you have the hardware needed, I don't see a problem with it, it's a great thing to learn. Check the hardware specs, and have fun, there are loads of projects out there currently for it. Take a look at https://gpt4all.io, their docs are pretty awesome ---------------------------------------------------------------------- FatalV0rt3x · 2023-09-28 08:10 I tried using the GPT4ALL model recently, but my server isn't powerful enough yet, maybe once I chuck a decent GPU in it, but for now I'll stick with OpenAi If you have the hardware needed, I don't see a problem with it, it's a great thing to learn. Check the hardware specs, and have fun, there are loads of projects out there currently for it. Take a look at https://gpt4all.io, their docs are pretty awesome ---------------------------------------------------------------------- Stetsed · 2023-09-28 08:29 Firstly a 30B model is not as powerful as ChatGPT 3.5 in general for most tasks however for most tasks it will be okay. However if you do not have a GPU with enough VRAM(So for a 30B model 20GB+ of VRAM) because running it inside of the CPU it will be incrediblely slow and if you input a few hundred word text you will be waiting a LONG time for any output as the time to generate will increase as an exponent the more you add to process.(And even with a simple phrase on a i7-11700K I was looking at 100ms per token and a token is about 3/4 of a word). ---------------------------------------------------------------------- Mother_Construction2 · 2023-09-28 09:04 Hey it seems like you dual posted the same comment, maybe remove one of them? ---------------------------------------------------------------------- FatalV0rt3x · 2023-09-28 09:19 Thanks for the heads up, looks like the android app still has issues 😕 ---------------------------------------------------------------------- ozzeruk82 · 2023-09-28 09:50 Yes, it's better. Better quality results, faster results. ---------------------------------------------------------------------- DelScipio · 2023-09-28 11:13 Openai is Faster and better results ---------------------------------------------------------------------- ZAFJB · 2023-09-28 11:22 The real use cases for self hosted is if you want it to look at confidential data, or if you really want to hammer it will a lot of queries. Other than that online is better, principally because of speed. But also because you don't have to invest resources in building, configuring, maintaining and operating a server. Even if you pay the money for ChatGPT4, at current pricing, you are still be ahead in overall effort and spend. ---------------------------------------------------------------------- pentesticals · 2023-09-28 12:22 Have you actually used Llama? It’s actually so bad. Even plugged it into Langchain and it’s not even good enough and makes the agent just fail. ---------------------------------------------------------------------- -quakeguy- · 2023-09-28 12:22 The software being run isn't the problem. The hardware is. You need H100's if you want to get reasonable performance out of anything. I don't know about you, but I don't find spending $30,000 for a few niche cases in my home use to be reasonable. ---------------------------------------------------------------------- SEND_NUKES_PLS · 2023-09-28 12:53 30B llama is ass compared to ChatGPT. ---------------------------------------------------------------------- utopiah · 2023-09-28 13:15 FWIW I wrote [https://fabien.benetou.fr/Content/SelfHostingArtificialI ntelligence](https://fabien.benetou.fr/Content/SelfHostingArtificialIn telligence) and updated it just an hour ago after testing Mistral. Why would one NOT self host? 1. convenience 2. lack of hardware 3. benchmark results One interesting thing about Mistral IMHO is precisely being API compatible with OpenAI, cf [https://docs.mistral.ai/quickstart#next- steps](https://docs.mistral.ai/quickstart#next-steps) , and thus insuring that if you do want to switch, it should be a no hassle process. ---------------------------------------------------------------------- utopiah · 2023-09-28 13:16 Taking this opportunity to mention that one can run models on other infrastructures also, from the typical cloud providers to dedicated ones, e.g HuggingFace. For testing it can often be a good compromise then once validated consider if running locally is worth it. ---------------------------------------------------------------------- utopiah · 2023-09-28 13:19 Can't talk about you or OP but I believe a lot of people who tinker with models don't go for BLOOM or very large models but rather things than run on commercial GPUs, thus 40xx or lower, that they often already have in their gaming rig. I believe most people who rely on H100, A100, etc do so more for training than inference. ---------------------------------------------------------------------- YourFlakingFuture · 2023-09-28 13:41 Llama is very stupid compared to chatgpt ---------------------------------------------------------------------- Elle221LL · 2023-09-28 16:10 If you used LLama locally you should know why. ChatGPT Just works fast and reliable ---------------------------------------------------------------------- ZaxLofful · 2023-09-28 16:13 ChatGPT is way more powerful and free! ---------------------------------------------------------------------- notdoreen · 2023-09-28 17:14 What unpaid service? ---------------------------------------------------------------------- Mephidia · 2023-09-28 20:18 LLAMA is against license to use for commercial applications. ---------------------------------------------------------------------- grahamsz · 2023-09-28 22:33 There are versions that permit it though https://huggingface.co/VMware/open-llama-7b-v2-open-instruct I have it on my laptop, and it's kinda decent (but not near gpt4's capability) while also being incredibly slow (16GB RTX3080) ---------------------------------------------------------------------- TinyMicron · 2023-09-28 22:48 How advanced is Llama compared to ChatGPT 3.5? Is it as fast? ---------------------------------------------------------------------- raunchieska · 2023-09-28 22:57 curious - why though? what chatgpt has in it that makes it better? is it all of the ingested copyrighted books? ---------------------------------------------------------------------- mrchem1911 · 2023-09-29 11:50 Llama is fckin stupid, only Wicuna is good and worth trying but rn GPT-4 is the smartest and most accurate model