https://github.com/Lightning-AI/lit-llama Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} Lightning-AI / lit-llama Public * Notifications * Fork 11 * Star 406 License Apache-2.0 license 406 stars 11 forks Star Notifications * Code * Issues 6 * Pull requests 2 * Actions * Security * Insights More * Code * Issues * Pull requests * Actions * Security * Insights Lightning-AI/lit-llama This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 8 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/L] Use Git or checkout with SVN using the web URL. [gh repo clone Lightn] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @williamFalcon williamFalcon Update README.md ... 1be84b0 Mar 28, 2023 Update README.md 1be84b0 Git stats * 65 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github Cache pip dependencies in CPU tests (#46) March 28, 2023 16:21 lit_llama Speed up quantization in generate.py (#35) March 28, 2023 16:09 scripts Fix tests without pip install (#45) March 28, 2023 09:54 tests Speed up quantization in generate.py (#35) March 28, 2023 16:09 .gitignore Add generate.py and prepare_shakespeare.py tests (#42) March 28, 2023 05:41 LICENSE Introduce basic configuration setup (#26) March 27, 2023 18:01 README.md Update README.md March 28, 2023 12:44 generate.py Fix time display in generate.py (#47) March 28, 2023 16:22 requirements.txt Add generate.py and prepare_shakespeare.py tests (#42) March 28, 2023 05:41 setup.py Rework repo structure (#41) March 28, 2023 00:06 train.py Rework repo structure (#41) March 28, 2023 00:06 View code [ ] [?] Lit-LLaMA [?] [?] Lit-LLaMA [?] Why? Design principles Get involved! Setup Use the model Run Lit-LLaMA on consumer devices Get involved! Acknowledgements License README.md Lit-LLaMA [?] Lit-LLaMA [?] cpu-tests license Discord Lit-LLaMA and pineapple pizza [?] Lit-LLaMA [?] Independent implementation of LLaMA that is fully open source under the Apache 2.0 license. This implementation builds on nanoGPT. Why? We believe that AI should be fully open source and part of the collective knowledge. The original LLaMA code is GPL licensed which means any project using it must also be released under GPL. This "taints" any other code and prevents meaningful academic and commercial use. Lit-LLaMA solves that for good. Design principles Lit-LLaMA is: * Simple: Single-file implementation without boilerplate. * Correct: Numerically equivalent to the original model. * Optimized: Runs on consumer hardware or at scale. * Open-source: No strings attached. Get involved! Join our Discord to build high-performance, truly open-source models for the common benefit of the community. Setup Clone the repo git clone https://github.com/Lightning-AI/lit-llama cd lit-llama install dependencies pip install -r requirements.txt You are all set! Use the model To generate text predictions, download the model weights following the instructions on the official LLaMA repository. Now you should have a folder like this: checkpoints/llama +-- 7B | +-- checklist.chk | +-- consolidated.00.pth | +-- params.json +-- 13B | ... +-- tokenizer_checklist.chk +-- tokenizer.model Convert the weights to the Lit-LLaMA format: python scripts/convert_checkpoint.py \ --output_dir checkpoints/lit-llama \ --ckpt_dir checkpoints/llama \ --tokenizer_path checkpoints/llama/tokenizer.model \ --model_size 7B Run inference: python generate.py --prompt "Hello, my name is" This will run the 7B model and require ~26 GB of GPU memory (A100 GPU). Run Lit-LLaMA on consumer devices For GPUs with less memory, enable quantization (--quantize true). This will take longer to load but require ~8GB of memory. This can run on any consumer GPU. python generate.py --quantize true --prompt "Hello, my name is" See python generate.py --help for more options. Get involved! We're in a quest towards fully open source AI. Lit-LLaMA Join us and start contributing, especially on the following areas: * [ ] Pre-training * [ ] Fine-tuning (full and LoRA) * [ ] Quantization * [ ] Sparsification Look at train.py for a starting point towards pre-training / fine-tuning using Lightning Fabric. Don't forget to join our Discord! Acknowledgements * @karpathy for nanoGPT * @FacebookResearch for the original LLaMA implementation * @TimDettmers for bitsandbytes License Lit-LLaMA is released under the Apache 2.0 license. About No description, website, or topics provided. Resources Readme License Apache-2.0 license Stars 406 stars Watchers 21 watching Forks 11 forks Releases No releases published Packages 0 No packages published Contributors 4 * @carmocca carmocca Carlos Mocholi * @awaelchli awaelchli Adrian Walchli * @lantiga lantiga Luca Antiga * @williamFalcon williamFalcon William Falcon Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.