https://arxiv.org/abs/2509.14786

Skip to main content
Cornell University

In just 5 minutes help us improve arXiv:

Annual Global Survey
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2509.14786
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Machine Learning

arXiv:2509.14786 (cs)
[Submitted on 18 Sep 2025]

Title:Pre-training under infinite compute

Authors:Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto
View a PDF of the paper titled Pre-training under infinite compute,
by Konwoo Kim and Suhas Kotha and Percy Liang and Tatsunori Hashimoto
View PDF HTML (experimental)

    Abstract:Since compute grows much faster than web text available
    for language model pre-training, we ask how one should approach
    pre-training under fixed data and no compute constraints. We
    first show that existing data-constrained approaches of
    increasing epoch count and parameter count eventually overfit,
    and we significantly improve upon such recipes by properly tuning
    regularization, finding that the optimal weight decay is $30\
    times$ larger than standard practice. Since our regularized
    recipe monotonically decreases loss following a simple power law
    in parameter count, we estimate its best possible performance via
    the asymptote of its scaling law rather than the performance at a
    fixed compute budget. We then identify that ensembling
    independently trained models achieves a significantly lower loss
    asymptote than the regularized recipe. Our best intervention
    combining epoching, regularization, parameter scaling, and
    ensemble scaling achieves an asymptote at 200M tokens using $5.17
    \times$ less data than our baseline, and our data scaling laws
    predict that this improvement persists at higher token budgets.
    We find that our data efficiency gains can be realized at much
    smaller parameter counts as we can distill an ensemble into a
    student model that is 8$\times$ smaller and retains $83\%$ of the
    ensembling benefit. Finally, our interventions designed for
    validation loss generalize to downstream benchmarks, achieving a
    $9\%$ improvement for pre-training evals and a $17.5\times$ data
    efficiency improvement over continued pre-training on math
    mid-training data. Our results show that simple algorithmic
    improvements can enable significantly more data-efficient
    pre-training in a compute-rich future.

Subjects: Machine Learning (cs.LG)
Cite as:  arXiv:2509.14786 [cs.LG]
          (or arXiv:2509.14786v1 [cs.LG] for this version)
          https://doi.org/10.48550/arXiv.2509.14786
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Konwoo Kim [view email]
[v1] Thu, 18 Sep 2025 09:36:23 UTC (5,735 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Pre-training under infinite
    compute, by Konwoo Kim and Suhas Kotha and Percy Liang and
    Tatsunori Hashimoto
  * View PDF
  * HTML (experimental)
  * TeX Source

license icon view license
Current browse context:
cs.LG
< prev   |   next >
new | recent | 2025-09
Change to browse by:
cs

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] alphaXiv Toggle
alphaXiv (What is alphaXiv?)
[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Huggingface Toggle
Hugging Face (What is Huggingface?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)
[ ] IArxiv recommender toggle
IArxiv Recommender (What is IArxiv?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status