leading self hostable chatbot like chatgpt reddit / selfhosted ---------------------------------------------------------------------- TheRealTrailblaster · 2023-11-20 09:27 What is currently the leading chatbot that can be self hosted on your computer? I have heard about alot of them and it seems like everyone and there dog is making one however I'm not sure which one I should be use. ---------------------------------------------------------------------- Lick_A_Brick · 2023-11-20 10:23 Have a look at Ollama: https://ollama.ai/ Easy to run on your own machine (if you run Windows you can run it in WSL2). The Llama2 model requires about 8GB of RAM. ---------------------------------------------------------------------- TheRealTrailblaster · 2023-11-20 10:43 Does this support being controllable with code? such as python or nodejs? ---------------------------------------------------------------------- FishSpoof · 2023-11-20 10:47 Are there any non-woke chat bots that will get basic facts correct such as there being only 2 genders ? ---------------------------------------------------------------------- bpadair31 · 2023-11-20 10:57 Get a life ---------------------------------------------------------------------- fuuman1 · 2023-11-20 11:05 Grow up mate. ---------------------------------------------------------------------- just_some_onlooker · 2023-11-20 11:15 You phrases your question wrong - Reddit is too woke to respond reasonably. And also there isn't. ---------------------------------------------------------------------- DanJOC · 2023-11-20 11:21 You're conflating gender with sex, which is a human error. ---------------------------------------------------------------------- TestPilot1980 · 2023-11-20 11:44 This is ok. Best is to go to huggingface and explore. Join openllama here on reddit. They have a leaderboard too. This is one of the good ones. https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca ---------------------------------------------------------------------- lilolalu · 2023-11-20 12:56 I think the most advanced OpenSource LLM model right now is considered to be Mistral 7B Open Orca. You can serve it via the Oobabooga GUI (which let's you try other LLM models as well). If you don't have a GPU for interference, this will be nothing like the ChatGPT experience though but much slower. https://github.com/oobabooga/text-generation-webui You can also try these models on your desktop using GPT4all, which doesn't support GPU ATM. https://gpt4all.io/index.html ---------------------------------------------------------------------- Lick_A_Brick · 2023-11-20 13:25 It has an API which you can call from your code: https://github.com/jmorganca/ollama#rest-api This makes it easy to integrate with Python / NodeJS apps. ---------------------------------------------------------------------- lilolalu · 2023-11-20 14:17 Well I never tried to explore how normative most LLM are in terms of gender but with most you will not be able to discuss this because they are censored in talking about conflictive subjects. I think there are uncensored versions of Mistral, and there are definitely uncensored other OpenSource LLM. So, if you want to research, I think the best starting point would be something like GPT4all, adding an uncensored model and just asking it questions about gender normativity. I think algorithmic bias is a huge subject that is not properly discussed publicly. There are a couple of organizations that deal with this, like algorithmic justice league and others. https://www.ajl.org/ ---------------------------------------------------------------------- CellistOld6437 · 2023-11-20 15:03 fr fr i hate when fucking AI believes the round earth propaganda and lies instead of listening to the truth and facts ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-11-20 15:42 Reddit was woke 12 years ago when I first used it on another account. Its only gotten way worse. Its the hard truth ---------------------------------------------------------------------- contradictingpoint · 2023-11-20 15:57 Thanks for the write up- will the coral usb work for interference? ---------------------------------------------------------------------- just_some_onlooker · 2023-11-20 15:58 I've only used it for 3 or 4 years. It's always been to me. Always been heavily used to push agendas that the majority don't agree with... But think it's ok cause it don't affect them... ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-11-20 16:03 I mean we're talking in the woke bot downvote wasteland comment thread right now lol. But to be serious, yeah, it's because reddit seems to attract a certain age range of people, cough millennials, cough. And that age range tends to be very politically left leaning. That and i'm sure theres bots all over the place downvoting any sort of "non-woke" content. Look up what city Reddit is based out of and your questions will be answered. ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-11-20 16:08 It also seems the bots do not like us even discussing about the reddit algorithms. Hence the downvotes... ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-11-20 16:12 also I think they assume that normal people care about reddit karma. I could care less ---------------------------------------------------------------------- just_some_onlooker · 2023-11-20 16:25 Likewise. In the words of prophet Dave... Reddit isn't a real place :/ ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-11-20 16:27 also people didnt forget all the shitty things u/spez did like 4 months ago. And reddit for keeping him in power shows where they stand. ---------------------------------------------------------------------- lilolalu · 2023-11-20 16:32 >coral usb https://www.reddit.com/r/LocalLLaMA/comments/12o96hf/has\_anyone\_used \_llama\_with\_a\_tpu\_instead\_of\_gpu/ in this discussion it doesnt sound as if this device will help you much. most of these ai models best used with nvidia gpu acceleration with as much ram as possible, while VRAM is more important than speed. ---------------------------------------------------------------------- strangekind · 2023-11-20 17:03 Mistral OpenOrca is a good one. I pull about 10 to 11 tokens/sec. Very impressive. ---------------------------------------------------------------------- centomila · 2023-11-20 17:05 I recommend LMSTUDIO ([https://lmstudio.ai/](https://lmstudio.ai/)). It allows you to manage and download models from Hugging Face, and suggests models compatible with your machine. Additionally, it can initiate a local HTTP server that functions similarly to OpenAI's API." ---------------------------------------------------------------------- lilolalu · 2023-11-20 17:18 [https://github.com/oobabooga/text-generation- webui](https://github.com/oobabooga/text-generation-webui) provides OpenAI compatible Apis as well ---------------------------------------------------------------------- contradictingpoint · 2023-11-20 18:00 Thanks. Maybe it will give me a reason to dig the Jetson nano or Jetson agx Orin industrial out of the closet. ---------------------------------------------------------------------- ScribeOfGoD · 2023-11-20 21:15 Not me reading it as liggma thinking it was a lick my balls joke 😂😂😂😅 ---------------------------------------------------------------------- BfrogPrice2116 · 2023-11-20 22:34 LM Studio https://lmstudio.ai/ Easiest way to get started ---------------------------------------------------------------------- Revexious · 2023-11-20 22:51 Faraday.dev is pretty good, i've found. The Mistral 7b model in particular ---------------------------------------------------------------------- mmayrink · 2023-11-20 23:47 !RemindMe 14 hours ---------------------------------------------------------------------- RemindMeBot · 2023-11-20 23:48 I will be messaging you in 14 hours on [**2023-11-21 13:47:23 UTC**](h ttp://www.wolframalpha.com/input/?i=2023-11- 21%2013:47:23%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/17zkpfy/leading_self_ hostable_chatbot_like_chatgpt/ka34nz8/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F17zkpfy%2Fleading_self_hostable_chatbot_like_c hatgpt%2Fka34nz8%2F%5D%0A%0ARemindMe%21%202023-11- 21%2013%3A47%3A23%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%2017zkpfy) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- remghoost7 · 2023-11-21 01:48 I'd recommend [koboldcpp](https://github.com/LostRuins/koboldcpp) for your backend, [SillyTavern](https://github.com/SillyTavern/SillyTavern) for your frontend, and I've been a fan of [dolphin-2.1-mistral- 7B](https://huggingface.co/TheBloke/dolphin-2.1-mistral-7B-GGUF). I've been using the Q4\_K\_S. But you could probably run a 13B model just fine. I've heard good things about the nous-hermes models (I was a big fan of their Llama2 model). **I'd stick to mistral variants, personally.** Their dataset/training has *far surpassed* base Llama2 stuff in my opinion. ---------------------------------------------------------------------- SuperLucas2000 · 2023-11-21 02:38 Can someone explain what is the benefit of running all of these models locally? Are they better than the free available chatgpt? Any good reading on how to learn/get started with all this? ---------------------------------------------------------------------- TechnicaVivunt · 2023-11-21 04:08 Privacy, and free api access for other projects. ---------------------------------------------------------------------- FishSpoof · 2023-11-21 09:50 I can see there are libtards and woketards in here lol The earth is round and there are only 2 genders. So radical right ? ---------------------------------------------------------------------- FishSpoof · 2023-11-21 09:53 Libtard/woketard answer. Let me guess, you stand with Ukraine too ? Lol ---------------------------------------------------------------------- FishSpoof · 2023-11-21 10:23 Libtard ---------------------------------------------------------------------- FishSpoof · 2023-11-21 10:53 Yeah I figured. My goal is to have negative karma so I'm on the right track ---------------------------------------------------------------------- bpadair31 · 2023-11-21 14:03 Oh, good one. I am overwhelmed by the intelligence and cogency of your argument. ---------------------------------------------------------------------- SuperLucas2000 · 2023-11-21 14:49 Thanks any good resource to get started on this? ---------------------------------------------------------------------- TheRealTrailblaster · 2023-11-22 00:13 Yeah, I'm hearing alota about mistral lol. Also could I really run 13B? I thought I could only run upto 7B? Do I like need enough vram to load it or am I missing something? ---------------------------------------------------------------------- remghoost7 · 2023-11-22 00:24 With 12GB of RAM? You could *more than likely* run a 13B model. I can almost run a 13B model entirely on my 1060 6GB. Have to offload 10 or so layers (out of 45ish). It's been a while since I tried though. You realistically want to load the entire model into VRAM, but with LLMs (as opposed to something like Stable Diffusion) you can offload a few of the layers onto your CPU/RAM. Slower generation speed, mind you. You'll also have to keep your context size in mind. The larger your conversation, the more VRAM you'll need (since it's pretty much processing the entire conversation again every time you hit "generate"). With a small context, I wouldn't be surprised if you can run a 13B model. **No harm in trying though.** I spun up a 34B model a while back (almost entirely on system RAM) just for the heck of it. Slow as molasses, but it ran. Depends on what sorts of speed/coherence you're looking for. I'd recommend sticking to the 4bit quantization models (Q4). I've heard Q5 is a bit more coherent, but the speed penalty wasn't worth it for me. ---------------------------------------------------------------------- FishSpoof · 2023-11-22 08:20 Woketard ---------------------------------------------------------------------- jay-workai-tools · 2023-11-30 14:51 We have also made a ChatGPT alternative optimized for self-hosting at https://www.reddit.com/r/selfhosted/comments/187jmte/selfhosted\_alter native\_to\_chatgpt\_and\_more/ Hope you like it :)