https://github.com/johnma2006/mamba-minimal Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} johnma2006 / mamba-minimal Public * Notifications * Fork 24 * Star 406 Simple, minimal implementation of Mamba in one file of PyTorch. 406 stars 24 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights johnma2006/mamba-minimal This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/j] Use Git or checkout with SVN using the web URL. [gh repo clone johnma] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @johnma2006 johnma2006 Add weights_only=True to torch.load (thanks kit1980) ... 82efa90 Dec 20, 2023 Add weights_only=True to torch.load (thanks kit1980) 82efa90 Git stats * 7 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time README.md Update README.md December 20, 2023 06:43 demo.ipynb Documentation updates December 20, 2023 17:18 model.py Add weights_only=True to torch.load (thanks kit1980) December 20, 2023 19:15 View code mamba-minimal Demo References README.md mamba-minimal Simple, minimal implementation of Mamba in one file of PyTorch. Featuring: * Equivalent numerical output as official implementation for both forward and backward pass * Simplified, readable, annotated code Does NOT include: * Speed. The official implementation is heavily optimized, and these optimizations are core contributions of the Mamba paper. I kept most implementations simple for readability. * Proper parameter initialization (though this could be added without sacrificing readability) Demo See demo.ipynb for examples of prompt completions. from model import Mamba from transformers import AutoTokenizer model = Mamba.from_pretrained('state-spaces/mamba-370m') tokenizer = AutoTokenizer.from_pretrained('EleutherAI/gpt-neox-20b') generate(model, tokenizer, 'Mamba is the') Mamba is the world's longest venomous snake with an estimated length of over 150 m. With such a large size and a venomous bite, Mamba kills by stabbing the victim (which is more painful and less effective than a single stab of the bite) 150 meters... scary! References The Mamba architecture was introduced in Mamba: Linear-Time Sequence Modeling with Selective State Spaces by Albert Gu and Tri Dao. The official implementation is here: https://github.com/state-spaces/ mamba/tree/main About Simple, minimal implementation of Mamba in one file of PyTorch. Resources Readme Activity Stars 406 stars Watchers 10 watching Forks 24 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 65.8% * Jupyter Notebook 34.2% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.