https://arxiv.org/abs/2306.15794

Skip to main content
Cornell University
We are hiring

We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2306.15794
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Machine Learning

arXiv:2306.15794 (cs)
[Submitted on 27 Jun 2023]

Title:HyenaDNA: Long-Range Genomic Sequence Modeling at Single
Nucleotide Resolution

Authors:Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum
Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano
Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, Chris Re
Download a PDF of the paper titled HyenaDNA: Long-Range Genomic
Sequence Modeling at Single Nucleotide Resolution, by Eric Nguyen and
12 other authors
Download PDF

    Abstract: Genomic (DNA) sequences encode an enormous amount of
    information for gene regulation and protein synthesis. Similar to
    natural language models, researchers have proposed foundation
    models in genomics to learn generalizable features from unlabeled
    genome data that can then be fine-tuned for downstream tasks such
    as identifying regulatory elements. Due to the quadratic scaling
    of attention, previous Transformer-based genomic models have used
    512 to 4k tokens as context (<0.001% of the human genome),
    significantly limiting the modeling of long-range interactions in
    DNA. In addition, these methods rely on tokenizers to aggregate
    meaningful DNA units, losing single nucleotide resolution where
    subtle genetic variations can completely alter protein function
    via single nucleotide polymorphisms (SNPs). Recently, Hyena, a
    large language model based on implicit convolutions was shown to
    match attention in quality while allowing longer context lengths
    and lower time complexity. Leveraging Hyenas new long-range
    capabilities, we present HyenaDNA, a genomic foundation model
    pretrained on the human reference genome with context lengths of
    up to 1 million tokens at the single nucleotide-level, an up to
    500x increase over previous dense attention-based models.
    HyenaDNA scales sub-quadratically in sequence length (training up
    to 160x faster than Transformer), uses single nucleotide tokens,
    and has full global context at each layer. We explore what longer
    context enables - including the first use of in-context learning
    in genomics for simple adaptation to novel tasks without updating
    pretrained model weights. On fine-tuned benchmarks from the
    Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA)
    on 12 of 17 datasets using a model with orders of magnitude less
    parameters and pretraining data. On the GenomicBenchmarks,
    HyenaDNA surpasses SotA on all 8 datasets on average by +9
    accuracy points.

Subjects: Machine Learning (cs.LG); Genomics (q-bio.GN)
Cite as:  arXiv:2306.15794 [cs.LG]
          (or arXiv:2306.15794v1 [cs.LG] for this version)
          https://doi.org/10.48550/arXiv.2306.15794
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Eric Nguyen [view email]
[v1] Tue, 27 Jun 2023 20:46:34 UTC (1,792 KB)
Full-text links:

Download:

  * Download a PDF of the paper titled HyenaDNA: Long-Range Genomic
    Sequence Modeling at Single Nucleotide Resolution, by Eric Nguyen
    and 12 other authors
    PDF
  * Other formats

(license)
Current browse context:
cs.LG
< prev   |   next >
new | recent | 2306
Change to browse by:
cs
q-bio
q-bio.GN

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)
[ ] IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack