https://arxiv.org/abs/2310.02059

Skip to main content
Cornell University
We gratefully acknowledge support from the Simons Foundation, member
institutions, and all contributors. Donate
 
arxiv logo > cs > arXiv:2310.02059
[                    ]

Help | Advanced Search

[All fields        ]
Search
arXiv logo
Cornell University Logo
[                    ] GO
quick links

  * Login
  * Help Pages
  * About

Computer Science > Software Engineering

arXiv:2310.02059 (cs)
[Submitted on 3 Oct 2023]

Title:Security Weaknesses of Copilot Generated Code in GitHub

Authors:Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba
Shahin, Jiaxin Yu
Download a PDF of the paper titled Security Weaknesses of Copilot
Generated Code in GitHub, by Yujia Fu and 5 other authors
Download PDF

    Abstract:Modern code generation tools use AI models, particularly
    Large Language Models (LLMs), to generate functional and complete
    code. While such tools are becoming popular and widely available
    for developers, using these tools is often accompanied by
    security challenges. Therefore, it is important to assess the
    quality of the generated code, especially in terms of its
    security. Researchers have recently explored various aspects of
    code generation tools, including security. However, many open
    questions about the security of the generated code require
    further investigation, especially the security issues of
    automatically generated code in the wild. To this end, we
    conducted an empirical study by analyzing the security weaknesses
    in code snippets generated by GitHub Copilot that are found as
    part of publicly available projects hosted on GitHub. The goal is
    to investigate the types of security issues and their scale in
    real-world scenarios (rather than crafted scenarios). To this
    end, we identified 435 code snippets generated by Copilot from
    publicly available projects. We then conducted extensive security
    analysis to identify Common Weakness Enumeration (CWE) instances
    in these code snippets. The results show that (1) 35.8% of
    Copilot generated code snippets contain CWEs, and those issues
    are spread across multiple languages, (2) the security weaknesses
    are diverse and related to 42 different CWEs, in which CWE-78: OS
    Command Injection, CWE-330: Use of Insufficiently Random Values,
    and CWE-703: Improper Check or Handling of Exceptional Conditions
    occurred the most frequently, and (3) among the 42 CWEs
    identified, 11 of those belong to the currently recognized 2022
    CWE Top-25. Our findings confirm that developers should be
    careful when adding code generated by Copilot (and similar AI
    code generation tools) and should also run appropriate security
    checks as they accept the suggested code.

Subjects: Software Engineering (cs.SE); Cryptography and Security
          (cs.CR)
Cite as:  arXiv:2310.02059 [cs.SE]
          (or arXiv:2310.02059v1 [cs.SE] for this version)
          https://doi.org/10.48550/arXiv.2310.02059
          Focus to learn more
          arXiv-issued DOI via DataCite

Submission history

From: Peng Liang [view email]
[v1] Tue, 3 Oct 2023 14:01:28 UTC (1,734 KB)
Full-text links:

Access Paper:

    Download a PDF of the paper titled Security Weaknesses of Copilot
    Generated Code in GitHub, by Yujia Fu and 5 other authors
  * Download PDF
  * PostScript
  * Other Formats

[by-4]
Current browse context:
cs.SE
< prev   |   next >
new | recent | 2310
Change to browse by:
cs
cs.CR

References & Citations

  * NASA ADS
  * Google Scholar
  * Semantic Scholar

a export BibTeX citation Loading...

BibTeX formatted citation

x
[loading...          ]
Data provided by:

Bookmark

BibSonomy logo Reddit logo
(*) Bibliographic Tools

Bibliographic and Citation Tools

[ ] Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
[ ] Litmaps Toggle
Litmaps (What is Litmaps?)
[ ] scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
( ) Code, Data, Media

Code, Data and Media Associated with this Article

[ ] Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
[ ] DagsHub Toggle
DagsHub (What is DagsHub?)
[ ] Links to Code Toggle
Papers with Code (What is Papers with Code?)
[ ] ScienceCast Toggle
ScienceCast (What is ScienceCast?)
( ) Demos

Demos

[ ] Replicate Toggle
Replicate (What is Replicate?)
[ ] Spaces Toggle
Hugging Face Spaces (What is Spaces?)
( ) Related Papers

Recommenders and Search Tools

[ ] Link to Influence Flower
Influence Flower (What are Influence Flowers?)
[ ] Connected Papers Toggle
Connected Papers (What is Connected Papers?)
[ ] Core recommender toggle
CORE Recommender (What is CORE?)

  * Author
  * Venue
  * Institution
  * Topic

( ) About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and
share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have
embraced and accepted our values of openness, community, excellence,
and user data privacy. arXiv is committed to these values and only
works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community?
Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is
MathJax?)

  * About
  * Help

  * Click here to contact arXiv Contact
  * Click here to subscribe Subscribe

  * Copyright
  * Privacy Policy

  * Web Accessibility Assistance
  * arXiv Operational Status
    Get status notifications via email or slack