https://arxiv.org/abs/2404.08698

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2404.08698
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2404.08698 (cs)
[Submitted on 10 Apr 2024]

Title:Lossless Acceleration of Large Language Model via Adaptive
N-gram Parallel Decoding

Authors:Jie Ou, Yueming Chen, Wenhong Tian
View a PDF of the paper titled Lossless Acceleration of Large
Language Model via Adaptive N-gram Parallel Decoding, by Jie Ou and 2
other authors
View PDF HTML (experimental)

    Abstract:While Large Language Models (LLMs) have shown remarkable
    abilities, they are hindered by significant resource consumption
    and considerable latency due to autoregressive processing. In
    this study, we introduce Adaptive N-gram Parallel Decoding
    (ANPD), an innovative and lossless approach that accelerates
    inference by allowing the simultaneous generation of multiple
    tokens. ANPD incorporates a two-stage approach: it begins with a
    rapid drafting phase that employs an N-gram module, which adapts
    based on the current interactive context, followed by a
    verification phase, during which the original LLM assesses and
    confirms the proposed tokens. Consequently, ANPD preserves the
    integrity of the LLM's original output while enhancing processing
    speed. We further leverage a multi-level architecture for the
    N-gram module to enhance the precision of the initial draft,
    consequently reducing inference latency. ANPD eliminates the need
    for retraining or extra GPU memory, making it an efficient and
    plug-and-play enhancement. In our experiments, models such as
    LLaMA and its fine-tuned variants have shown speed improvements
    up to 3.67x, validating the effectiveness of our proposed ANPD.

Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:  arXiv:2404.08698 [cs.CL]
          (or arXiv:2404.08698v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2404.08698
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Jie Ou [view email]
[v1] Wed, 10 Apr 2024 16:11:09 UTC (4,882 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Lossless Acceleration of Large
    Language Model via Adaptive N-gram Parallel Decoding, by Jie Ou
    and 2 other authors
  * View PDF
  * HTML (experimental)
  * TeX Source
  * Other Formats

view license
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2404
Change to browse by:
cs
cs.LG

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack