https://arxiv.org/abs/2405.14782

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2405.14782
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2405.14782 (cs)
[Submitted on 23 May 2024]

Title:Lessons from the Trenches on Reproducible Evaluation of
Language Models

Authors:Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao
, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka
Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen
Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster,
Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas
Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan,
Xiangru Tang, Kevin A. Wang, Genta Indra Winata, Francois Yvon, Andy
Zou
View a PDF of the paper titled Lessons from the Trenches on
Reproducible Evaluation of Language Models, by Stella Biderman and 28
other authors
View PDF HTML (experimental)

    Abstract:Effective evaluation of language models remains an open
    challenge in NLP. Researchers and engineers face methodological
    issues such as the sensitivity of models to evaluation setup,
    difficulty of proper comparisons across methods, and the lack of
    reproducibility and transparency. In this paper we draw on three
    years of experience in evaluating large language models to
    provide guidance and lessons for researchers. First, we provide
    an overview of common challenges faced in language model
    evaluation. Second, we delineate best practices for addressing or
    lessening the impact of these challenges on research. Third, we
    present the Language Model Evaluation Harness (lm-eval): an open
    source library for independent, reproducible, and extensible
    evaluation of language models that seeks to address these issues.
    We describe the features of the library as well as case studies
    in which the library has been used to alleviate these
    methodological concerns.

Subjects: Computation and Language (cs.CL)
Cite as:  arXiv:2405.14782 [cs.CL]
          (or arXiv:2405.14782v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2405.14782
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Hailey Schoelkopf [view email]
[v1] Thu, 23 May 2024 16:50:49 UTC (1,650 KB)
Full-text links:

Access Paper:

    View a PDF of the paper titled Lessons from the Trenches on
    Reproducible Evaluation of Language Models, by Stella Biderman
    and 28 other authors
  * View PDF
  * HTML (experimental)
  * TeX Source
  * Other Formats

license icon view license
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2405
Change to browse by:
cs

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] GotitPub Toggle
Gotit.pub (What is GotitPub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack