https://github.com/joennlae/tensorli Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} joennlae / tensorli Public * Notifications * Fork 2 * Star 93 Absolute minimalistic implementation of a GPT-like transformer using only numpy (<650 lines). 93 stars 2 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights joennlae/tensorli This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/j] Use Git or checkout with SVN using the web URL. [gh repo clone joennl] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @joennlae joennlae docs: update readme ... f2ca754 Nov 13, 2023 docs: update readme f2ca754 Git stats * 63 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows feat(mypy): add mypy for static type checking July 28, 2023 18:48 tensorli refactor: add some types November 13, 2023 14:21 test fix: typo November 13, 2023 15:08 .gitignore chore: update gitignore November 12, 2023 16:06 README.md docs: update readme November 13, 2023 15:19 environment.yml feat(mypy): add mypy for static type checking July 28, 2023 18:48 logo.png docs: add logo November 13, 2023 15:15 pyproject.toml lint: update linting policies November 13, 2023 14:17 View code Minimalistic implementation of a GPT-like transformer using only numpy (<650 lines). Example Naming Caveats Inspiration Plans & Outlook README.md logo Minimalistic implementation of a GPT-like transformer using only numpy (<650 lines). --------------------------------------------------------------------- The implementation includes: * Automatic differentiation * Tensorli object (PyTorch like) * Simple NN layers: Linearli, Embeddingli, MultiheadAttentionli, LayerNorm * Optimizers: Adamli All that is "needed" to train and execute a GPT-like transformer model. ...and everything else is just efficiency ~ Andrej Karpathy^1. 1: Youtube - micrograd Example from tensorli.tensorli import Tensorli from tensorli.models.transformerli import Transformerli vocb_size, embd_dim, seq_len, n_layer, n_head, batch_size = 10, 64, 10, 3, 4, 16 transformer = Transformerli(vocb_size, embd_dim, seq_len, n_layer, n_head) x_numpy = np.random.randint(0, vocb_size - 1, size=(batch_size, seq_len)) x = Tensorli(x_numpy) out = transformer(x) Naming In the region where I grew up, a word for "little" is used as a suffix [2]. For example, "little dog" would be "dogli". I thought it would be a nice name for a minimalistic implementation of a neural network library. Caveats This library works, but it is NOT optimized. It is not meant to be used in production or for anything at scale. It is meant to be used as a learning tool. Inspiration This library is heavily inspired by the following projects: * minGPT * tinygrad I highly recommend checking them out. Plans & Outlook * [ ] Add Dropout * [ ] Add more experimental architectures About Absolute minimalistic implementation of a GPT-like transformer using only numpy (<650 lines). Resources Readme Activity Stars 93 stars Watchers 1 watching Forks 2 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.