https://github.com/mlc-ai/mlc-llm Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} mlc-ai / mlc-llm Public * Notifications * Fork 25 * Star 828 Enable everyone to develop, optimize and deploy AI models natively on everyone's devices. mlc.ai/mlc-llm License Apache-2.0 license 828 stars 25 forks Star Notifications * Code * Issues 13 * Pull requests 1 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights mlc-ai/mlc-llm This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 3 branches 1 tag Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/m] Use Git or checkout with SVN using the web URL. [gh repo clone mlc-ai] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @tqchen tqchen Moving most device memory static (#21) ... 486d5b0 Apr 30, 2023 Moving most device memory static (#21) This PR makes the chat.cc mostly static allocate memory to reduce overall memory allocations in runtime. 486d5b0 Git stats * 9 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time 3rdparty Initial commit April 28, 2023 23:29 cpp Moving most device memory static (#21) April 30, 2023 16:01 ios Moving most device memory static (#21) April 30, 2023 16:01 log_db Initial commit April 28, 2023 23:29 mlc_llm Moving most device memory static (#21) April 30, 2023 16:01 scripts Initial commit April 28, 2023 23:29 site Update README.md April 29, 2023 10:52 tests Initial commit April 28, 2023 23:29 .clang-format Initial commit April 28, 2023 23:29 .gitignore Initial commit April 28, 2023 23:29 .gitmodules Initial commit April 28, 2023 23:29 CMakeLists.txt Initial commit April 28, 2023 23:29 LICENSE Initial commit April 28, 2023 23:29 README.md Update README.md April 29, 2023 17:57 build.py Support for cross compiling dylib to x86 macos (#8) April 29, 2023 17:41 setup.py Initial commit April 28, 2023 23:29 version.py Initial commit April 28, 2023 23:29 View code MLC LLM What is MLC LLM? How does MLC Enable Universal Native Deployment? Links Acknowledgements README.md MLC LLM | Project | Demo: iOS | Demo: CLI | WebLLM | MLC LLM is a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases. Our mission is to enable everyone to develop, optimize and deploy AI models natively on everyone's devices. Everything runs locally with no server support and accelerated with local GPUs on your phone and laptops. Supported platforms include: * iPhone * Metal GPUs and Intel/ARM MacBooks; * AMD and NVIDIA GPUs via Vulkan on Windows and Linux; * NVIDIA GPUs via CUDA on Windows and Linux; * WebGPU on browsers (through companion project WebLLM). Check out our instruction page to try out! [demo] What is MLC LLM? In recent years, there has been remarkable progress in generative artificial intelligence (AI) and large language models (LLMs), which are becoming increasingly prevalent. Thanks to open-source initiatives, it is now possible to develop personal AI assistants using open-sourced models. However, LLMs tend to be resource-intensive and computationally demanding. To create a scalable service, developers may need to rely on powerful clusters and expensive hardware to run model inference. Additionally, deploying LLMs presents several challenges, such as their ever-evolving model innovation, memory constraints, and the need for potential optimization techniques. The goal of this project is to enable the development, optimization, and deployment of AI models for inference across a range of devices, including not just server-class hardware, but also users' browsers, laptops, and mobile apps. To achieve this, we need to address the diverse nature of compute devices and deployment environments. Some of the key challenges include: * Supporting different models of CPUs, GPUs, and potentially other co-processors and accelerators. * Deploying on the native environment of user devices, which may not have python or other necessary dependencies readily available. * Addressing memory constraints by carefully planning allocation and aggressively compressing model parameters. MLC LLM offers a repeatable, systematic, and customizable workflow that empowers developers and AI system researchers to implement models and optimizations in a productivity-focused, Python-first approach. This methodology enables quick experimentation with new models, new ideas and new compiler passes, followed by native deployment to the desired targets. Furthermore, we are continuously expanding LLM acceleration by broadening TVM backends to make model compilation more transparent and efficient. How does MLC Enable Universal Native Deployment? The cornerstone of our solution is machine learning compilation (MLC ), which we leverage to efficiently deploy AI models. We build on the shoulders of open-source ecosystems, including tokenizers from HuggingFace and Google, as well as open-source LLMs like Llama, Vicuna, Dolly and more. Our primary workflow is based on Apache TVM Unity, an exciting ongoing development in the Apache TVM Community. * Dynamic shape: we bake a language model as a TVM IRModule with native dynamic shape support, avoiding the need for extra padding to the maximum length and reducing both computation amount and memory usage. * Composable ML compilation optimizations: we perform many model deployment optimizations, such as better compilation code transformation, fusion, memory planning, library offloading and manual code optimization can be easily incorporated as TVM's IRModule transformations exposed as Python APIs. * Quantization: we utilize low-bit quantizations to compress the model weights and leverage TVM's loop-level TensorIR to quickly customize code generations for different compression encoding schemes. * Runtime: The final generated libraries run on the native environment, with TVM runtime that comes with minimal dependencies, which supports various GPU driver APIs and native language bindings (C, JavaScript, etc). Architecture Diagram Additionally, we also provide a lightweight C++-based example CLI app that showcases how to wrap up the compiled artifacts and necessary pre/post-processing, which will hopefully clarify the workflow to embed them into native applications. As a starting point, MLC generates GPU shaders for CUDA, Vulkan and Metal. It is possible to add more support, such as OpenCL, sycl, webgpu-native, through improvements to TVM compiler and runtime. MLC also supports various CPU targets including ARM and x86 via LLVM. We heavily rely on open-source ecosystem, more specifically, TVM Unity, an exciting latest development in the TVM project that enables python-first interactive MLC development experiences that allows us to easily compose new optimizations all in Python, and incrementally bring our app to the environment of interest. We also leveraged optimizations such as fused quantization kernels, first class dynamic shape support and diverse GPU backends. Links * You might also be interested in WebLLM, our companion derived project that focus on bringing LLM to browsers. * Project page for instructions. * Local build Instructions for ios App. * You might want to check out our online public Machine Learning Compilation course for a systematic walkthrough of our approaches. Acknowledgements This project is initiated by members from CMU catalyst, UW SAMPL, SJTU, OctoML and the MLC community. We would love to continue developing and supporting the open-source ML community. This project is only possible thanks to the shoulders open-source ecosystems that we stand on. We want to thank the Apache TVM community and developers of the TVM Unity effort. The open-source ML community members made these models publicly available. PyTorch and Hugging Face communities that make these models accessible. We would like to thank the teams behind Vicuna, SentencePiece, LLaMA, and Alpaca. We also would like to thank the Vulkan, Swift, C++, python Rust communities that enables this project. About Enable everyone to develop, optimize and deploy AI models natively on everyone's devices. mlc.ai/mlc-llm Topics language-model tvm llm machine-learning-compilation Resources Readme License Apache-2.0 license Stars 828 stars Watchers 17 watching Forks 25 forks Report repository Releases 1 tags Packages 0 No packages published Contributors 6 * @tqchen * @MasterJH5574 * @jinhongyii * @junrushao * @spectrometerHBH * @yzh119 Languages * Python 90.6% * C++ 6.4% * Swift 1.5% * CMake 0.6% * Objective-C++ 0.5% * Shell 0.3% * Objective-C 0.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.