https://github.com/rustformers/llm Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} rustformers / llm Public * Notifications * Fork 142 * Star 3.1k Run inference for Large Language Models on CPU, with Rust License Apache-2.0, MIT licenses found Licenses found Apache-2.0 LICENSE-APACHE MIT LICENSE-MIT 3.1k stars 142 forks Star Notifications * Code * Issues 26 * Pull requests 1 * Discussions * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights rustformers/llm This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 13 branches 1 tag Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/r] Use Git or checkout with SVN using the web URL. [gh repo clone rustfo] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @philpax philpax Merge pull request #200 from danforbes/dfo/chore/refactors ... 67ee753 May 9, 2023 Merge pull request #200 from danforbes/dfo/chore/refactors Refactors from llm-chain integration 67ee753 Git stats * 507 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows feat: pull submodules for release May 8, 2023 21:05 .vscode Reduce duplicated example code May 2, 2023 07:52 binaries Refactors from llm-chain integration May 9, 2023 12:32 crates Refactors from llm-chain integration May 9, 2023 12:32 doc/resources docs: update README, etc May 8, 2023 19:58 examples feat: $PROMPT -> {{PROMPT}} April 2, 2023 12:01 scripts scripts: use GGMF version of convert-pth-to-ggml.py April 5, 2023 00:35 .gitignore feat: re-enable BLOOM for now April 30, 2023 22:56 .gitmodules refactor: rearrange crates April 30, 2023 23:54 CONTRIBUTING.md refactor: reintroduce ggml-sys and bindgen tool April 30, 2023 23:26 Cargo.lock Refactors from llm-chain integration May 9, 2023 12:32 Cargo.toml feat(llm): add llm::load_dynamic May 6, 2023 21:23 Dockerfile llama-rs -> llm, update README May 1, 2023 03:55 LICENSE-APACHE llama-rs -> llm, update README May 1, 2023 03:55 LICENSE-MIT llama-rs -> llm, update README May 1, 2023 03:55 README.md docs(readme): make lib/cli split more obvious May 9, 2023 18:21 flake.lock nix: add flake May 5, 2023 18:17 flake.nix nix: add flake May 5, 2023 18:17 known-good-models.md fix #192 - list supported models May 8, 2023 01:18 View code [ ] llm Getting started Building llm-cli Using cargo From repository Getting models From Hugging Face r/LocalLLaMA LLaMA original weights Running Docker Q&A Why did you do this? Seriously now. I insist. Is this the real reason? How is this different from llama.cpp? What applications and libraries use llm? Applications Libraries README.md llm A llama riding a crab, AI-generated Image by @darthdeus, using Stable Diffusion Latest version MIT/Apache2 Discord llm is a Rust ecosystem of libraries for running inference on large language models, inspired by llama.cpp. The primary crate is the llm crate, which wraps llm-base and supported model crates. On top of llm, there is a CLI application, llm-cli, which provides a convenient interface for running inference on supported models. Inferencing can be done as a one-off, or interactively, through REPL or chat modes. It can also print information about, or quantize, a GGML model. It can be downloaded from the latest GitHub release or by installing it from crates.io. llm is powered by the ggml tensor library, and aims to bring the robustness and ease of use of Rust to the world of large language models. At present, inference is only on the CPU, but we hope to support GPU inference in the future through alternate backends. Currently, the following models are supported: * GPT-2 * GPT-J * LLaMA: LLaMA, Alpaca, Vicuna, Koala, GPT4All v1, GPT4-X, Wizard * GPT-NeoX: GPT-NeoX, StableLM, Dolly v2 (partial, not the same tensor names?) * BLOOM: BLOOMZ Getting started Make sure you have a Rust 1.65.0 or above and C toolchain^1 set up. llm is a Rust library that re-exports llm-base and the model crates (e.g. bloom, gpt2 llama). To use llm, add it to your Cargo.toml: [dependencies] llm = "0.1" NOTE: For best results, make sure to build and run in release mode. Debug builds are going to be very slow. Building llm-cli llm-cli (binary name llm) is a basic application that provides a CLI interface to the library. Using cargo Run cargo install llm-cli to install llm to your Cargo bin directory, which rustup is likely to have added to your PATH. The CLI application can then be run through llm. From repository Clone the repository and then build it with git clone --recurse-submodules git@github.com:rustformers/llm.git cargo build --release The resulting binary will be at target/release/llm[.exe]. It can also be run directly through Cargo, using cargo run --release -- This is useful for development. Getting models GGML files are easy to acquire. For a list of models that have been tested, see the known-good models. Certain older GGML formats are not supported by this project, but the goal is to maintain feature parity with the upstream GGML project. For problems relating to loading models, or requesting support for supported GGML model types, please open an Issue. From Hugging Face Hugging Face is a leader in open-source machine learning and hosts hundreds of GGML models. Search for GGML models on Hugging Face . r/LocalLLaMA This Reddit community maintains a wiki related to GGML models, including well organized lists of links for acquiring GGML models (mostly from Hugging Face ). LLaMA original weights Currently, the only legal source to get the original weights is this repository. After acquiring the weights, it is necessary to convert them into a format that is compatible with ggml. To achieve this, follow the steps outlined below: Warning To run the Python scripts, a Python version of 3.9 or 3.10 is required. 3.11 is unsupported at the time of writing. # Convert the model to f16 ggml format python3 scripts/convert-pth-to-ggml.py /path/to/your/models/7B/ 1 # Quantize the model to 4-bit ggml format cargo run --release llama quantize /path/to/your/models/7B/ggml-model-f16.bin /path/to/your/models/7B/ggml-model-q4_0.bin q4_0 In future, we hope to provide a more streamlined way of converting models. Note The llama.cpp repository has additional information on how to obtain and run specific models. Running For example, try the following prompt: llm llama infer -m /ggml-model-q4_0.bin -p "Tell me how cool the Rust programming language is:" Some additional things to try: * Use --help to see a list of available options. * If you have the alpaca-lora weights, try repl mode! llm llama repl -m /ggml-alpaca-7b-q4.bin -f examples/alpaca_prompt.txt * Sessions can be loaded (--load-session) or saved (--save-session) to file. To automatically load and save the same session, use --persist-session. This can be used to cache prompts to reduce load time, too: Docker # To build (This will take some time, go grab some coffee): docker build -t llm . # To run with prompt: docker run --rm --name llm -it -v ${PWD}/data:/data -v ${PWD}/examples:/examples llm llama infer -m data/gpt4all-lora-quantized-ggml.bin -p "Tell me how cool the Rust programming language is:" # To run with prompt file and repl (will wait for user input): docker run --rm --name llm -it -v ${PWD}/data:/data -v ${PWD}/examples:/examples llm llama repl -m data/gpt4all-lora-quantized-ggml.bin -f examples/alpaca_prompt.txt Q&A Why did you do this? It was not my choice. Ferris appeared to me in my dreams and asked me to rewrite this in the name of the Holy crab. Seriously now. Come on! I don't want to get into a flame war. You know how it goes, something something memory something something cargo is nice, don't make me say it, everybody knows this already. I insist. Sheesh! Okaaay. After seeing the huge potential for llama.cpp, the first thing I did was to see how hard would it be to turn it into a library to embed in my projects. I started digging into the code, and realized the heavy lifting is done by ggml (a C library, easy to bind to Rust) and the whole project was just around ~2k lines of C++ code (not so easy to bind). After a couple of (failed) attempts to build an HTTP server into the tool, I realized I'd be much more productive if I just ported the code to Rust, where I'm more comfortable. Is this the real reason? Haha. Of course not. I just like collecting imaginary internet points, in the form of little stars, that people seem to give to me whenever I embark on pointless quests for rewriting X thing, but in Rust. How is this different from llama.cpp? This is a reimplementation of llama.cpp that does not share any code with it outside of ggml. This was done for a variety of reasons: * llama.cpp requires a C++ compiler, which can cause problems for cross-compilation to more esoteric platforms. An example of such a platform is WebAssembly, which can require a non-standard compiler SDK. * Rust is easier to work with from a development and open-source perspective; it offers better tooling for writing "code in the large" with many other authors. Additionally, we can benefit from the larger Rust ecosystem with ease. * We would like to make ggml an optional backend (see this issue). In general, we hope to build a solution for model inferencing that is as easy to use and deploy as any other Rust crate. What applications and libraries use llm? Applications * llmcord: Discord bot for generating messages using llm. Libraries * llm-chain: Work in progress, see this PR. Footnotes 1. A modern-ish C toolchain is required to compile ggml. A C++ toolchain should not be necessary. - About Run inference for Large Language Models on CPU, with Rust Resources Readme License Apache-2.0, MIT licenses found Licenses found Apache-2.0 LICENSE-APACHE MIT LICENSE-MIT Stars 3.1k stars Watchers 24 watching Forks 142 forks Report repository Releases 1 v0.1.1 Latest May 8, 2023 Packages 0 No packages published Used by 5 * @cheolwanpark * @rustformers * @dmisino * @eulervoid Contributors 25 * @philpax * @setzer22 * @danforbes * @iacore * @KerfuffleV2 * @odysa * @hlhr202 * @tehmatt * @RoyVorster * @jempabroni * @katopz + 14 contributors Languages * Rust 97.9% * Python 1.6% * Other 0.5% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.