https://arxiv.org/abs/2305.06161

Skip to main content
Cornell University
We gratefully acknowledge support from
the Simons Foundation and member institutions.
 
arxiv logo > cs > arXiv:2305.06161
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2305.06161 (cs)
[Submitted on 9 May 2023]

Title:StarCoder: may the source be with you!

Authors:Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff
, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia
Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo,
Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier,
Joao Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel
Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin
, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman,
Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan
Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh,
Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel
Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger,
Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu,
Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt,
Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine
Jernite, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha
, Leandro von Werra, Harm de Vries
Download a PDF of the paper titled StarCoder: may the source be with
you!, by Raymond Li and 66 other authors
Download PDF

    Abstract: The BigCode community, an open-scientific collaboration
    working on the responsible development of Large Language Models
    for Code (Code LLMs), introduces StarCoder and StarCoderBase:
    15.5B parameter models with 8K context length, infilling
    capabilities and fast large-batch inference enabled by
    multi-query attention. StarCoderBase is trained on 1 trillion
    tokens sourced from The Stack, a large collection of permissively
    licensed GitHub repositories with inspection tools and an opt-out
    process. We fine-tuned StarCoderBase on 35B Python tokens,
    resulting in the creation of StarCoder. We perform the most
    comprehensive evaluation of Code LLMs to date and show that
    StarCoderBase outperforms every open Code LLM that supports
    multiple programming languages and matches or outperforms the
    OpenAI code-cushman-001 model. Furthermore, StarCoder outperforms
    every model that is fine-tuned on Python, can be prompted to
    achieve 40\% pass@1 on HumanEval, and still retains its
    performance on other programming languages. We take several
    important steps towards a safe open-access model release,
    including an improved PII redaction pipeline and a novel
    attribution tracing tool, and make the StarCoder models publicly
    available under a more commercially viable version of the Open
    Responsible AI Model license.

          Computation and Language (cs.CL); Artificial Intelligence
Subjects: (cs.AI); Programming Languages (cs.PL); Software
          Engineering (cs.SE)
Cite as:  arXiv:2305.06161 [cs.CL]
          (or arXiv:2305.06161v1 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2305.06161
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Harm de Vries [view email]
[v1] Tue, 9 May 2023 08:16:42 UTC (640 KB)
Full-text links:

Download:

  * Download a PDF of the paper titled StarCoder: may the source be
    with you!, by Raymond Li and 66 other authors
    PDF
  * Other formats

[by-sa-4]
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2305
Change to browse by:
cs
cs.AI
cs.PL
cs.SE

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Mendeley logo Reddit logo ScienceWISE logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack