Distributed LLM like Tdarr?
reddit / selfhosted

----------------------------------------------------------------------
XxNerdAtHeartxX · 2023-12-11 16:20

For those who don't know what Tdarr is, it's a Video Transcoding
application that lets you connect 'nodes' to the main config server.
This lets you run the server/webUI of Tdarr on your main server, while
offloading the actual work to a separate PC somewhere else.

Like many, I have a Gaming PC with a good graphics cards that I'd like
to use for LLMs such as LLama, however I don't have the hardware to
run it in my actual server where all my services are running with
docker right now.

I don't want to leave a web server running on my gaming pc at all
times (plus, my vpn only allows me to access my server ip), so Im
hoping to run the 'webui' on my server, while loading an LLM Node on
my Gaming PC to do the actual hard processing at the request of the
server via web ui interactions.

Have any projects popped up that let you run the webui on one machine,
but do the processing of the LLM on another?

----------------------------------------------------------------------
lilolalu · 2023-12-11 17:19

I was researching this the other day, distributed inference seems to
be very complicated but there are research projects around it.
Distributed model training seems to be something which is fine
already.

https://openmined.org/

----------------------------------------------------------------------
ZAFJB · 2023-12-11 18:48

Follow this thread: https://www.reddit.com/r/selfhosted/comments/18fy0
hp/seeking_suggestions_for_selfhosted_chatgptlike/

----------------------------------------------------------------------
moarmagic · 2023-12-11 19:01

I've been doing something like this, but not as an all in one
solution. Lmstudio let's you run llm models and chat with them, and
also has a server mode where any front-end that can use openai's can
send it requests. So you'd have to open ports between the webui  and
desktop flr that.  There are a couple different solutions for webuis-
I've been playing mostly with sillytavern. Has a fair amount of
extensions and features.

My own irks about this setup:

I dont know if there's a native way to start/stop lmstudio remotely,
so I have to leave it running. However, it doesn't seem to put much
stress on ny system until I send it a query, so its not idling while
maxing out my ram/gpu at least.

Lm studio does allow you to do cpu/gpu tasks together, but this I'd at
the cost of only running gguf format models- that said, I haven't run
into a model I wanted to try that someone hadn't concerted to that.

Sillytavern is not a multi user environment, so there's not great ways
to segregate content out. If you want to share it, or set up a lot of
test characters/parameters, you might want to run separate instances.

----------------------------------------------------------------------
brunovianna · 2023-12-13 10:05

Isn't petals what you are looking for?
https://github.com/bigscience-workshop/petals#readme