https://github.com/getumbrel/llama-gpt Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} getumbrel / llama-gpt Public * Notifications * Fork 6 * Star 212 A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. apps.umbrel.com/app/llama-gpt License MIT license 212 stars 6 forks Activity Star Notifications * Code * Issues 6 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights getumbrel/llama-gpt This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 4 branches 2 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/g] Use Git or checkout with SVN using the web URL. [gh repo clone getumb] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @mayankchhabra mayankchhabra Update README.md ... c9cfd24 Aug 17, 2023 Update README.md c9cfd24 Git stats * 312 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows Build images on tag August 16, 2023 18:39 api Fix signal handling August 16, 2023 15:52 ui add fallback for copying to clipboard over http (#1) August 15, 2023 15:42 .gitignore Disable 13B docker builds, add images to docker-compose.yml August 16, 2023 04:59 LICENSE.md Add support for 13B and 70B models, workflow, readme August 15, 2023 23:11 README.md Update README.md August 17, 2023 05:56 docker-compose-13b.yml Add restart on-failure policy August 17, 2023 04:17 docker-compose-70b.yml Add restart on-failure policy August 17, 2023 04:17 docker-compose.yml Add restart on-failure policy August 17, 2023 04:17 View code [ ] LlamaGPT Demo How to install Install LlamaGPT on your umbrelOS home server Install LlamaGPT anywhere else Benchmarks Nous Hermes Llama 2 7B (GGML q4_0) Nous Hermes Llama 2 13B (GGML q4_0) Meta Llama 2 70B Chat (GGML q4_0) Roadmap and contributing Acknowledgements README.md LlamaGPT LlamaGPT A self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. 100% private, with no data leaving your device. umbrel.com >> [6874747073] [6874747073] [6874747073] [6874747073] Demo LlamaGPT.mp4 How to install Install LlamaGPT on your umbrelOS home server Running LlamaGPT on an umbrelOS home server is one click. Simply install it from the Umbrel App Store. LlamaGPT on Umbrel App Store Install LlamaGPT anywhere else You can run LlamaGPT on any x86 or arm64 system. Make sure you have Docker installed. Then, clone this repo and cd into it: git clone https://github.com/getumbrel/llama-gpt.git cd llama-gpt You can now run LlamaGPT with any of the following models depending upon your hardware: Model Model used Minimum RAM How to start LlamaGPT size required 7B Nous Hermes Llama 2 8GB docker compose up -d 7B (GGML q4_0) Nous Hermes Llama 2 docker compose -f 13B 13B (GGML q4_0) 16GB docker-compose-13b.yml up -d Meta Llama 2 70B Chat docker compose -f 70B (GGML q4_0) 48GB docker-compose-70b.yml up -d You can access LlamaGPT at http://localhost:3000. To stop LlamaGPT, run: docker compose down Benchmarks We've tested LlamaGPT models on the following hardware with the default system prompt, and user prompt: "How does the universe expand?" at temperature 0 to guarantee deterministic results. Generation speed is averaged over the first 10 generations. Feel free to add your own benchmarks to this table by opening a pull request. Nous Hermes Llama 2 7B (GGML q4_0) Device Generation speed M1 Max MacBook Pro (10 64GB RAM) 8.2 tokens/sec Umbrel Home (16GB RAM) 2.7 tokens/sec Raspberry Pi 4 (8GB RAM) 0.9 tokens/sec Nous Hermes Llama 2 13B (GGML q4_0) Device Generation speed M1 Max MacBook Pro (64GB RAM) 3.7 tokens/sec Umbrel Home (16GB RAM) 1.5 tokens/sec Meta Llama 2 70B Chat (GGML q4_0) Unfortunately, we don't have any benchmarks for this model yet. If you have one, please open a pull request to add it to this table. Roadmap and contributing We're looking to add more features to LlamaGPT. You can see the roadmap here. The highest priorities are: * Add CUDA and Metal support. * Moving the model out of the Docker image and into a separate volume. * Updating the front-end to show model download progress, and to allow users to switch between models. * Making it easy to run custom models. If you're a developer who'd like to help with any of these, please open an issue to discuss the best way to tackle the challenge. If you're looking to help but not sure where to begin, check out these issues that have specifically been marked as being friendly to new contributors. Acknowledgements A massive thank you to the following developers and teams for making LlamaGPT possible: * Mckay Wrigley for building Chatbot UI. * Georgi Gerganov for implementing llama.cpp. * Andrei for building the Python bindings for llama.cpp. * NousResearch for fine-tuning the Llama 2 7B and 13B models. * Tom Jobbins for quantizing the Llama 2 models. * Meta for releasing Llama 2 under a permissive license. --------------------------------------------------------------------- License umbrel.com About A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. apps.umbrel.com/app/llama-gpt Topics ai self-hosted openai llama gpt gpt-4 llm chatgpt llamacpp llama-cpp gpt4all localai llama2 llama-2 Resources Readme License MIT license Activity Stars 212 stars Watchers 4 watching Forks 6 forks Report repository Packages 2 Contributors 3 * @mayankchhabra mayankchhabra Mayank Chhabra * @lukechilds lukechilds Luke Childs * @nmfretz nmfretz Nathan Fretz Languages * TypeScript 96.7% * Dockerfile 1.4% * Other 1.9% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.