https://arxiv.org/abs/2312.14187

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2312.14187
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Computation and Language

arXiv:2312.14187 (cs)
[Submitted on 20 Dec 2023 (v1), last revised 11 Jan 2024 (this
version, v3)]

Title:WaveCoder: Widespread And Versatile Enhanced Instruction Tuning
with Refined Data Generation

Authors:Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu,
Yishujie Zhao, Wenxiang Hu, Qiufeng Yin
Download a PDF of the paper titled WaveCoder: Widespread And
Versatile Enhanced Instruction Tuning with Refined Data Generation,
by Zhaojian Yu and 7 other authors
Download PDF HTML (experimental)

    Abstract:Recent work demonstrates that, after being fine-tuned on
    a high-quality instruction dataset, the resulting model can
    obtain impressive capabilities to address a wide range of tasks.
    However, existing methods for instruction data generation often
    produce duplicate data and are not controllable enough on data
    quality. In this paper, we extend the generalization of
    instruction tuning by classifying the instruction data to 4
    code-related tasks and propose a LLM-based
    Generator-Discriminator data process framework to generate
    diverse, high-quality instruction data from open source code.
    Hence, we introduce CodeOcean, a dataset comprising 20,000
    instruction instances across 4 universal code-related tasks,which
    is aimed at augmenting the effectiveness of instruction tuning
    and improving the generalization ability of fine-tuned model.
    Subsequently, we present WaveCoder, a fine-tuned Code LLM with
    Widespread And Versatile Enhanced instruction tuning. This model
    is specifically designed for enhancing instruction tuning of Code
    Language Models (LLMs). Our experiments demonstrate that
    Wavecoder models outperform other open-source models in terms of
    generalization ability across different code-related tasks at the
    same level of fine-tuning scale. Moreover, Wavecoder exhibits
    high efficiency in previous code generation tasks. This paper
    thus offers a significant contribution to the field of
    instruction data generation and fine-tuning models, providing new
    insights and tools for enhancing performance in code-related
    tasks.

Subjects: Computation and Language (cs.CL); Artificial Intelligence
          (cs.AI); Software Engineering (cs.SE)
Cite as:  arXiv:2312.14187 [cs.CL]
          (or arXiv:2312.14187v3 [cs.CL] for this version)
          https://doi.org/10.48550/arXiv.2312.14187
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Zhaojian Yu [view email]
[v1] Wed, 20 Dec 2023 09:02:29 UTC (1,336 KB)
[v2] Tue, 26 Dec 2023 13:51:38 UTC (1,336 KB)
[v3] Thu, 11 Jan 2024 07:44:55 UTC (1,463 KB)
Full-text links:

Access Paper:

    Download a PDF of the paper titled WaveCoder: Widespread And
    Versatile Enhanced Instruction Tuning with Refined Data
    Generation, by Zhaojian Yu and 7 other authors
  * Download PDF
  * HTML (experimental)
  * Other Formats

view license
Current browse context:
cs.CL
< prev   |   next >
new | recent | 2312
Change to browse by:
cs
cs.AI
cs.SE

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
[ ] Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack