https://arxiv.org/abs/2604.05091

Skip to main content
Cornell University
Learn about arXiv becoming an independent nonprofit.
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2604.05091
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2604.05091 (cs)
[Submitted on 6 Apr 2026]

Title:MegaTrain: Full Precision Training of 100B+ Parameter Large
Language Models on a Single GPU

Authors:Zhengqing Yuan, Hanchi Sun, Lichao Sun, Yanfang Ye
View a PDF of the paper titled MegaTrain: Full Precision Training of
100B+ Parameter Large Language Models on a Single GPU, by Zhengqing
Yuan and 3 other authors
View PDF HTML (experimental)

    Abstract:We present MegaTrain, a memory-centric system that
    efficiently trains 100B+ parameter large language models at full
    precision on a single GPU. Unlike traditional GPU-centric
    systems, MegaTrain stores parameters and optimizer states in host
    memory (CPU memory) and treats GPUs as transient compute engines.
    For each layer, we stream parameters in and compute gradients
    out, minimizing persistent device state. To battle the CPU-GPU
    bandwidth bottleneck, we adopt two key optimizations. 1) We
    introduce a pipelined double-buffered execution engine that
    overlaps parameter prefetching, computation, and gradient
    offloading across multiple CUDA streams, enabling continuous GPU
    execution. 2) We replace persistent autograd graphs with
    stateless layer templates, binding weights dynamically as they
    stream in, eliminating persistent graph metadata while providing
    flexibility in scheduling. On a single H200 GPU with 1.5TB host
    memory, MegaTrain reliably trains models up to 120B parameters.
    It also achieves 1.84$\times$ the training throughput of
    DeepSpeed ZeRO-3 with CPU offloading when training 14B models.
    MegaTrain also enables 7B model training with 512k token context
    on a single GH200.

Subjects: Computation and Language (cs.CL); Distributed, Parallel,
          and Cluster Computing (cs.DC); Operating Systems (cs.OS)
Cite as:  arXiv:2604.05091 [cs.CL]
          (or arXiv:2604.05091v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2604.05091
          Focus to learn more
          arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zhengqing Yuan [view email]
[v1] Mon, 6 Apr 2026 18:43:56 UTC (787 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled MegaTrain: Full Precision Training
    of 100B+ Parameter Large Language Models on a Single GPU, by
    Zhengqing Yuan and 3 other authors
  * View PDF
  * HTML (experimental)
  * TeX Source

view license
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2026-04
Change to browse by:
cs
cs.DC
cs.OS

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] alphaXiv Toggle
alphaXiv (What is alphaXiv?)
[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Huggingface Toggle
Hugging Face (What is Huggingface?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status