https://github.com/punica-ai/punica Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} punica-ai / punica Public * Notifications * Fork 7 * Star 69 Serving multiple LoRA finetuned LLM as one arxiv.org/abs/2310.18547 69 stars 7 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights punica-ai/punica This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/p] Use Git or checkout with SVN using the web URL. [gh repo clone punica] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @abcdabcd987 abcdabcd987 update docs ... 0e70d87 Nov 8, 2023 update docs 0e70d87 Git stats * 67 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time assets update docs November 8, 2023 19:50 benchmarks rename batch_decode November 8, 2023 17:08 csrc move kernels to csrc November 8, 2023 16:56 examples stash November 8, 2023 16:48 punica update docs November 8, 2023 19:50 tests update docs November 8, 2023 19:50 third_party add cutlass and nvbench October 12, 2023 20:27 .clang-format lora ops August 15, 2023 17:46 .clangd sgmv: use preallocated buffer, add nvbench October 14, 2023 19:04 .gitignore update benchmarks October 20, 2023 00:21 .gitmodules add cutlass and nvbench October 12, 2023 20:27 CMakeLists.txt sgmv: use preallocated buffer, add nvbench October 14, 2023 19:04 README.md update docs November 8, 2023 19:50 pyproject.toml add pybind and pytest for sgmv_cutlass October 18, 2023 19:12 setup.py move kernels to csrc November 8, 2023 16:56 View code Punica: Serving multiple LoRA finetuned LLM as one Overview Citation README.md Punica: Serving multiple LoRA finetuned LLM as one (paper) Overview Low rank adapation (LoRA) is a parameter efficient way to add new knowledge to a pretrained LLM. Although the pretrained LLM takes 100s of GB storage, a LoRA finetuned model only adds 1% storage and memory overhead. Punica enables running multiple LoRA finetuned models at the cost of running one. How? Assuming W of shape [H1, H2] is the weight of the pretrained model, LoRA adds two small matrices A of shape [H1, r] and B of [r, H2]. Running a input x on the finetuned model would be y := x @ (W + A@B), which is the same as y := x@W + x@A@B. When there are n LoRA models, there will be A1, B1, A2, B2, ..., An, Bn. Given a input batch X := (x1,x2,...,xn) that maps to each LoRA model, the output is Y := X@W + (x1@A1@B1, x2@A2@B2, ..., xn@An@Bn). The left-hand-side computes the input batch on the pretrained model. It is quite efficient. The latency is almost the same as when there's only one input, thanks to the strong batching effect. We figured out an efficient way to compute the right-hand-side (the LoRA addon). We encapsulate this operation in a CUDA kernel, called Segmented Gather Matrix-Vector multiplication (SGMV), as illustrated below. SGMV In the following microbenchmark figure, we can observe the strong batching effect of the pretrained model. Naive implementation of LoRA is slow, as depicted in the orange line. LoRA implemented via SGMV is effificent and preserves the strong batching effect. SGMV is fast and maintains strong batching effect The following figure shows the text generation throughput comparison between Punica and other systems, including HuggingFace Transformers, DeepSpeed, FasterTransformer, vLLM. The benchmark considers different settings of LoRA model popularity. Distinct means that each request is for a different LoRA model. Identical means that all requests are for the same LoRA model. Uniform and Skewed are in between. Punica achieves 12x throughput compared to state-of-the-art systems. Punica achieves 12x throughput compared to state-of-the-art systems Read our paper to understand more: Punica: Multi-Tenant LoRA Serving. Citation @misc{punica, title={Punica: Multi-Tenant LoRA Serving}, author={Lequn Chen and Zihao Ye and Yongji Wu and Danyang Zhuo and Luis Ceze and Arvind Krishnamurthy}, year={2023}, eprint={2310.18547}, archivePrefix={arXiv}, primaryClass={cs.DC} } About Serving multiple LoRA finetuned LLM as one arxiv.org/abs/2310.18547 Topics lora large-language-models llm Resources Readme Activity Stars 69 stars Watchers 6 watching Forks 7 forks Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @abcdabcd987 abcdabcd987 Lequn Chen * @yzh119 yzh119 Zihao Ye Languages * Cuda 64.3% * Python 27.7% * C++ 7.5% * CMake 0.5% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.