https://arxiv.org/abs/2102.03902

close this message

Donate to arXiv

Please join the Simons Foundation and our generous member
organizations in supporting arXiv during our giving campaign
September 23-27. 100% of your contribution will fund improvements and
new initiatives to benefit arXiv's global scientific community.

DONATE

[secure site, no need to create account]

Skip to main content
Cornell University
We gratefully acknowledge support from
the Simons Foundation and member institutions.
 
arXiv.org > cs > arXiv:2102.03902
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2102.03902 (cs)
[Submitted on 7 Feb 2021]

Title:Nystromformer: A Nystrom-Based Algorithm for Approximating
Self-Attention

Authors:Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing
Tan, Glenn Fung, Yin Li, Vikas Singh
Download PDF

    Abstract: Transformers have emerged as a powerful tool for a
    broad range of natural language processing tasks. A key component
    that drives the impressive performance of Transformers is the
    self-attention mechanism that encodes the influence or dependence
    of other tokens on each specific token. While beneficial, the
    quadratic complexity of self-attention on the input sequence
    length has limited its application to longer sequences -- a topic
    being actively studied in the community. To address this
    limitation, we propose Nystromformer -- a model that exhibits
    favorable scalability as a function of sequence length. Our idea
    is based on adapting the Nystrom method to approximate standard
    self-attention with $O(n)$ complexity. The scalability of
    Nystromformer enables application to longer sequences with
    thousands of tokens. We perform evaluations on multiple
    downstream tasks on the GLUE benchmark and IMDB reviews with
    standard sequence length, and find that our Nystromformer
    performs comparably, or in a few cases, even slightly better,
    than standard Transformer. Our code is at this https URL.

Comments: AAAI 2021
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:  arXiv:2102.03902 [cs.CL]
          (or arXiv:2102.03902v1 [cs.CL] for this version)

Submission history

From: Yunyang Xiong [view email]
[v1] Sun, 7 Feb 2021 20:06:59 UTC (1,593 KB)
Full-text links:

Download:

  * PDF
  * Other formats

(license)
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2102
Change to browse by:
cs
cs.LG

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export bibtex citation Loading...

Bibtex formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Mendeley logo Reddit logo ScienceWISE logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
( ) Code

Code Associated with this Article

[ ] arXiv Links to Code Toggle
arXiv Links to Code (What is Links to Code?)
( ) Related Papers

Recommenders and Search Tools

[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)
( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs and how to get involved.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack