https://arxiv.org/abs/2501.00663

Skip to main content
Cornell University

In just 3 minutes help us improve arXiv:

Annual Global Survey
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2501.00663
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Machine Learning

arXiv:2501.00663 (cs)
[Submitted on 31 Dec 2024]

Title:Titans: Learning to Memorize at Test Time

Authors:Ali Behrouz, Peilin Zhong, Vahab Mirrokni
View a PDF of the paper titled Titans: Learning to Memorize at Test
Time, by Ali Behrouz and Peilin Zhong and Vahab Mirrokni
View PDF HTML (experimental)

    Abstract:Over more than a decade there has been an extensive
    research effort on how to effectively utilize recurrent models
    and attention. While recurrent models aim to compress the data
    into a fixed-size memory (called hidden state), attention allows
    attending to the entire context window, capturing the direct
    dependencies of all tokens. This more accurate modeling of
    dependencies, however, comes with a quadratic cost, limiting the
    model to a fixed-length context. We present a new neural
    long-term memory module that learns to memorize historical
    context and helps attention to attend to the current context
    while utilizing long past information. We show that this neural
    memory has the advantage of fast parallelizable training while
    maintaining a fast inference. From a memory perspective, we argue
    that attention due to its limited context but accurate dependency
    modeling performs as a short-term memory, while neural memory due
    to its ability to memorize the data, acts as a long-term, more
    persistent, memory. Based on these two modules, we introduce a
    new family of architectures, called Titans, and present three
    variants to address how one can effectively incorporate memory
    into this architecture. Our experimental results on language
    modeling, common-sense reasoning, genomics, and time series tasks
    show that Titans are more effective than Transformers and recent
    modern linear recurrent models. They further can effectively
    scale to larger than 2M context window size with higher accuracy
    in needle-in-haystack tasks compared to baselines.

Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI);
          Computation and Language (cs.CL)
Cite as:  arXiv:2501.00663 [cs.LG]
          (or arXiv:2501.00663v1 [cs.LG] for this version)
          https://doi.org/10.48550/arXiv.2501.00663
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Ali Behrouz [view email]
[v1] Tue, 31 Dec 2024 22:32:03 UTC (3,249 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Titans: Learning to Memorize at
    Test Time, by Ali Behrouz and Peilin Zhong and Vahab Mirrokni
  * View PDF
  * HTML (experimental)
  * TeX Source
  * Other Formats

license icon view license
Current browse context:
cs.LG
< prev   |   next >
new | recent | 2025-01
Change to browse by:
cs
cs.AI
cs.CL

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] alphaXiv Toggle
alphaXiv (What is alphaXiv?)
[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Huggingface Toggle
Hugging Face (What is Huggingface?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)
[ ] IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack