Seeking Suggestions for Self-Hosted ChatGPT-like Interface to Run
Local LLM Models via Docker Compose
reddit / selfhosted

----------------------------------------------------------------------
3millionmax · 2023-12-11 16:08

I'm looking for a self-hosted solution to run a ChatGPT-like interface
for local large language models (LLMs) using Docker Compose.

I'm interested in projects that provide an easy-to-use web UI and can
be set up with minimal configuration.

Are there any  projects that you've tried and found useful for this
purpose? Planning to run this in a VM in cloud with 16gb of ram.

Please share your suggestions. Thanks in advance!

----------------------------------------------------------------------
RoyalCartographer · 2023-12-11 17:07

Checkout llama.cpp project on github

----------------------------------------------------------------------
ZAFJB · 2023-12-11 18:46

Github has loads and loads of lists pointing to loads and loads of AI
projects.

To find lists of projects, search
https://www.google.com/search?q=github+awesome

----------------------------------------------------------------------
007craft · 2023-12-12 10:13

Unfortunetly I dont think youre gonna have enough Vram.  Last time I
went down this route I was finding that you needed 22GB+ to get any
sort of value out of it.  There are options for using less vram but it
wont be as good or enjoyable.

My plan was to just use chatgpt for now or other online bots until
next year when nvidias 5000 Series cards come out.  Perhaps even some
Nvidia/intel/ATI cards that are specifically designed for AI.  Cards
like that will become more plentiful and cheaper in the future.
Either way I think were still 1-3 years away from bringing Chat-GPT
power into the self hosted realm based on hardware bottlenecks.

If Youre interested tho, heres one you can look into:

https://github.com/KoboldAI/KoboldAI-Client

----------------------------------------------------------------------
3millionmax · 2023-12-12 16:42

Thank you, I tried Serge, but it took too long to give any responses.

----------------------------------------------------------------------
3millionmax · 2023-12-12 16:43

Thank you for the insight. I tried installing Serge last night. It was
slow and not an enjoyable experience. Like you said, this will have to
wait for a bit for being effective in self hosting.

----------------------------------------------------------------------
t2noob · 2023-12-15 05:13

Ollama good starting point