https://github.com/Rishit-dagli/Nystromformer Skip to content Sign up * Product + Features + Mobile + Actions + Codespaces + Copilot + Packages + Security + Code review + Issues + Discussions + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Skills + GitHub Sponsors + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} Rishit-dagli / Nystromformer Public * Notifications * Fork 1 * Star 32 An implementation of the Nystromformer, using Nystrom method to approximate standard self attention License Apache-2.0 license 32 stars 1 fork Star Notifications * Code * Issues 0 * Pull requests 0 * Discussions * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Discussions * Actions * Projects * Security * Insights Rishit-dagli/Nystromformer This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 1 tag Code * Clone HTTPS GitHub CLI [https://github.com/R] Use Git or checkout with SVN using the web URL. [gh repo clone Rishit] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @Rishit-dagli Rishit-dagli Add colab badge ... 8b41089 Aug 21, 2022 Add colab badge 8b41089 Git stats * 35 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github example media nystromformer .gitignore CODE_OF_CONDUCT.md CONTRIBUTING.md LICENSE MANIFEST.in README.md setup.py View code [ ] Nystromformer Installation Developing nystromformer Usage Nystrom Attention Nystromformer Want to Contribute [?]? Want to discuss? Citation License README.md Nystromformer Twitter Open In Colab PyPI Run Tests Upload Python Package Code style: black codecov GitHub License GitHub stars GitHub followers Twitter Follow An implementation of the Nystromformer: A Nystrom-Based Algorithm for Approximating Self-Attention paper by Xiong et al. The self-attention mechanism that encodes the influence or dependence of other tokens on each specific token is a key component of the performance of Transformers. This uses the Nystrom method to approximate standard self-attention with O(n) complexity allowing to exhibit scalability as a function of sequence length. [nystromfor] Installation Run the following to install: pip install nystromformer Developing nystromformer To install nystromformer, along with tools you need to develop and test, run the following in your virtualenv: git clone https://github.com/Rishit-dagli/Nystromformer.git # or clone your own fork cd Nystromformer pip install -e .[dev] To run rank and shape tests run the following: pytest -v --disable-warnings --cov Usage Nystrom Attention import tensorflow as tf from nystromformer import NystromAttention attn = NystromAttention( dim = 512, dim_head = 64, heads = 8, num_landmarks = 256, # number of landmarks pinv_iterations = 6, # number of moore-penrose iterations for approximating pinverse. 6 was recommended by the paper residual = True # whether to do an extra residual with the value or not. supposedly faster convergence if turned on ) x = tf.random.normal((1, 16384, 512)) mask = tf.ones((1, 16384), dtype=tf.bool) attn(x, mask = mask) # (1, 16384, 512) Nystromformer import tensorflow as tf from nystromformer import Nystromformer model = Nystromformer( dim = 512, dim_head = 64, heads = 8, depth = 6, num_landmarks = 256, pinv_iterations = 6 ) x = tf.random.normal((1, 16384, 512)) mask = tf.ones((1, 16384), dtype=tf.bool) model(x, mask = mask) # (1, 16384, 512) Want to Contribute [?]? Awesome! If you want to contribute to this project, you're always welcome! See Contributing Guidelines. You can also take a look at open issues for getting more information about current or upcoming tasks. Want to discuss? Have any questions, doubts or want to present your opinions, views? You're always welcome. You can start discussions. Citation @misc{xiong2021nystromformer, title = {Nystromformer: A Nystrom-Based Algorithm for Approximating Self-Attention}, author = {Yunyang Xiong and Zhanpeng Zeng and Rudrasis Chakraborty and Mingxing Tan and Glenn Fung and Yin Li and Vikas Singh}, year = {2021}, eprint = {2102.03902}, archivePrefix = {arXiv}, primaryClass = {cs.CL} } Yannic Kilcher's Video PyTorch Implementation PyTorch Implementation License Copyright 2020 Rishit Dagli Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. About An implementation of the Nystromformer, using Nystrom method to approximate standard self attention Topics machine-learning deep-learning tensorflow keras artificial-intelligence transformer attention-mechanism nystrom nystromformer Resources Readme License Apache-2.0 license Code of conduct Code of conduct Stars 32 stars Watchers 3 watching Forks 1 fork Releases 1 Nystromformer v0.1.0 Latest Aug 20, 2022 Contributors 3 * @Rishit-dagli Rishit-dagli Rishit Dagli * @imgbot[bot] imgbot[bot] * @ImgBotApp ImgBotApp Imgbot Languages * Python 51.5% * Jupyter Notebook 48.5% Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.