Selfhosting Open source LLM with Iris Takeoff reddit / selfhosted ---------------------------------------------------------------------- 1FNn4 · 2023-12-25 07:12 TitanML's iris take off allows you host open source LLM. [https://github.com/titanml/takeoff- community](https://github.com/titanml/takeoff-community) pip install titan-iris To start takeoff: iris takeoff --model tiiuae/Falcon-RW-1B --device cpu --port 8000 For gpu: iris takeoff --model tiiuae/Falcon-RW-1B --device cuda --port 8000 Supported models: * BART (e.g. facebook/bart-large) * BLOOM (e.g. bigscience/bloom-3b) * CodeGen (e.g. Salesforce/codegen25-7b-mono) * Falcon (e.g. tiiuae/falcon-7b) * LLaMa (e.g. decapoda-research/llama-7b-hf) * MPT (e.g. mosaicml/mpt-7b) * OpenAI GPT2 (e.g. gpt2-large) * GPTBigCode (e.g. bigcode/gpt\_bigcode-santacoder) * GPT-J (e.g. EleutherAI/gpt-j-6b) * GPT-NeoX (e.g. EleutherAI/gpt-neox-20b) * OPT (e.g facebook/opt-1.3b) * Pegasus (e.g. google/pegasus-large) * T5 (e.g. google/flan-t5-large) * Llama 2 For llma2[https://docs.titanml.co/docs/titan- takeoff/experimentation/supported-models#using- llama-2](https://docs.titanml.co/docs/titan- takeoff/experimentation/supported-models#using-llama-2) ​ Finding models verisons, for example falcon: [https://huggingface.co/tiiuae](https://huggingface.co/tiiuae) If you don't have gpu and you have powerfull cpu (like amd 5800x I saw their demo) you can still run it. Once model downloaded their optimastion takes time. Optimastion on client. My trusted thinkpad couldn't handle it and I don't have gpu vm right now. Please give some feedback. :)) ---------------------------------------------------------------------- Luis15pt · 2023-12-25 08:48 Can you add a bit of context, what is achievable with this ? ---------------------------------------------------------------------- 1FNn4 · 2023-12-25 08:54 You can run open source language model like chatgpt your own homelab. You can access chat ui with http://localhost:8000/demos/chat or you can send request with curl. ---------------------------------------------------------------------- Crazy-Company-9749 · 2023-12-25 10:14 Can we host it on a cpu based vps like Digital Ocean? ---------------------------------------------------------------------- 1FNn4 · 2023-12-25 10:27 Yes you can. Make sure VM has least 16 core and 32 gb ram. Or you can try entry level gpu like testla T4. ---------------------------------------------------------------------- Crazy-Company-9749 · 2023-12-25 10:28 Thanks for the reply, where did you get the specs requirements from? ---------------------------------------------------------------------- 1FNn4 · 2023-12-25 12:37 Their demo. Normally you need lots of gpu computation power to run large language model. But their algorithms optimize models weight. So it can run cpu. But you need still powerful cpu to serve model. Let me know if you need help for setup. ---------------------------------------------------------------------- dually · 2023-12-25 13:00 What is the point of running the software unless you have a vast dataset to analyze? ---------------------------------------------------------------------- katrinatransfem · 2023-12-25 18:43 If it is anything like GPT4All or Oobabooga, you download an already trained model and run it on your own computer. ---------------------------------------------------------------------- Dazzling_Ad1507 · 2023-12-26 01:59 I'd love to see screenshots of the chat ui. Are they in the docs somewhere? ---------------------------------------------------------------------- 1FNn4 · 2023-12-26 03:27 Demo: https://www.youtube.com/watch?v=aIRP5yW5E1E youtube channel : https://www.youtube.com/@TitanMLOfficial ---------------------------------------------------------------------- CloudContainer · 2023-12-26 05:08 This is very cool, thanks for sharing! I have an Intel NUC 11TNKi5 Pro with 16 GB of available RAM. There is no dedicated GPU. How do you think it would run on it? https://www.intel.com/content/www/us/en/products/sku/205603/intel- nuc-11-pro-kit-nuc11tnki5/specifications.html ---------------------------------------------------------------------- 1FNn4 · 2023-12-26 05:31 Is it running linux? I don't think run well. But you can try smaller model. Falcon Falcon-RW-1B model 1.3B parameters and 2.6 GB. you can test it. once you run take off command. Docker image will be downloaded and model. Than optimization starts. Model optimization takes a lot cpu power and ram. How hope you're using nuc as headles homelab. let me know how it go. ---------------------------------------------------------------------- Craftkorb · 2023-12-26 11:31 How does it compare to oobas text-gen-ui or LMstudio? ---------------------------------------------------------------------- AnomalyNexus · 2023-12-27 09:46 What advantages does this have over tried & tested stuff like text- generation-ui? ---------------------------------------------------------------------- horus-heresy · 2023-12-27 17:44 You safely train on data you have or propose it at your job and train on their data to get an edge in competition. But if you need to ask questions like why you might be just not the audience for this ---------------------------------------------------------------------- rolyantrauts · 2023-12-27 20:01 You can anyway with [https://github.com/ggerganov/llama.cpp](https://g ithub.com/ggerganov/llama.cpp) that seems to have better model coverage. Supported models: LLaMA 🦙 LLaMA 2 🦙🦙 Falcon Alpaca GPT4All Chinese LLaMA / Alpaca and Chinese LLaMA-2 / Alpaca-2 Vigogne (French) Vicuna Koala OpenBuddy 🐶 (Multilingual) Pygmalion/Metharme WizardLM Baichuan 1 & 2 + derivations Aquila 1 & 2 Starcoder models Mistral AI v0.1 Refact Persimmon 8B MPT Bloom Yi models StableLM-3b-4e1t Deepseek models Qwen models Mixtral MoE PLaMo-13B Multimodal models: Llava 1.5 models Bakllava Obsidian ShareGPT4V ---------------------------------------------------------------------- dually · 2023-12-27 23:57 My point is that the value in ai is derived from throwing $billions of dollars in vc capital at a vast data set, which you can't duplicate by installing some software on your own machine. ---------------------------------------------------------------------- horus-heresy · 2023-12-28 00:17 Garbage in garbage out. Vast public data set is irrelevant when you want to prompt against company specific data without the risks of exposing sensitive information ---------------------------------------------------------------------- 1FNn4 · 2023-12-28 03:00 I'm not familiar with it.