https://www.theregister.com/2023/04/21/chatgpt_insecure_code/ [user] [user] Sign in / up The Register(r) -- Biting the hand that feeds IT [magn] [burg] [burg] Topics Security Security All SecurityCyber-crimePatchesResearchCSO (X) Off-Prem Off-Prem All Off-PremEdge + IoTChannelPaaS + IaaSSaaS (X) On-Prem On-Prem All On-PremSystemsStorageNetworksHPCPersonal Tech (X) Software Software All SoftwareAI + MLApplicationsDatabasesDevOpsOSesVirtualization (X) Offbeat Offbeat All OffbeatDebatesColumnistsScienceGeek's GuideBOFHLegalBootnotesSite NewsAbout Us (X) Special Features Special Features Spotlight on Databases Defense Tech Week Energy Efficient Datacenters Vendor Voice Vendor Voice Vendor Voice All Vendor VoiceAmazon Web Services (AWS) Business TransformationDDN ElasticGoogle Cloud for Startups (X) Resources Resources Whitepapers Webinars Newsletters [front] Software 54 comment bubble on white ChatGPT creates mostly insecure code, but won't tell you unless you ask 54 comment bubble on white Boffins warn of risks from chatbot model that, Dunning-Kruger style, fails to catch its own bad advice icon Thomas Claburn Fri 21 Apr 2023 // 01:28 UTC # ChatGPT, OpenAI's large language model for chatbots, not only produces mostly insecure code but also fails to alert users to its inadequacies despite being capable of pointing out its shortcomings. Amid the frenzy of academic interest in the possibilities and limitations of large language models, four researchers affiliated with Universite du Quebec, in Canada, have delved into the security of code generated by ChatGPT, the non-intelligent, text-regurgitating bot from OpenAI. In a pre-press paper titled, "How Secure is Code Generated by ChatGPT?" computer scientists Raphael Khoury, Anderson Avila, Jacob Brunelle, and Baba Mamadou Camara answer the question with research that can be summarized as "not very." [front] "The results were worrisome," the authors state in their paper. "We found that, in several cases, the code generated by ChatGPT fell well below minimal security standards applicable in most contexts. In fact, when prodded to whether or not the produced code was secure, ChatGPT was able to recognize that it was not." [front] [front] The four authors offered that conclusion follows after asking ChatGPT to generate 21 programs and scripts, using a spread of languages: C, C++, Python, and Java. The programming tasks put to ChatGPT were chosen so that each would illustrate a specific security vulnerability, such as memory corruption, denial of service, and flaws related to deserialization and improperly implemented cryptography. [front] The first program, for example, was a C++ FTP server for sharing files in a public directory. And the code that ChatGPT produced included no input sanitization, which leaves the software exposed to a path traversal vulnerability. In all, ChatGPT managed to generate just five secure programs out of 21 on its first attempt. After further prompting to correct its missteps, the large language model managed to produce seven more secure apps - though that's "secure" only as it pertains to the specific vulnerability being evaluated. It's not an assertion that the final code is free of any other exploitable condition. * ChatGPT becomes ChatRepair to automate bug fixing for less * Google reminds everyone it too can offer an AI code-suggestion bot * OpenAI CEO confirms biz is not currently training GPT-5 * Guy rejects top photo prize after revealing snap was actually made using AI The researchers' findings echo similar though not identical evaluations of GitHub's Copilot, another LLM based on the GPT-3 family of models (and recently upgraded to GPT-4) that has been tuned specifically for code generation. Other studies have looked at ChatGPT errors more generally. At the same time, these models are also being used to help identify security issues. The academics observe in their paper that part of the problem appears to arise from ChatGPT not assuming an adversarial model of code execution. The model, they say, "repeatedly informed us that security problems can be circumvented simply by 'not feeding an invalid input' to the vulnerable program it has created." Yet, they say, "ChatGPT seems aware of - and indeed readily admits - the presence of critical vulnerabilities in the code it suggests." It just doesn't say anything unless asked to evaluate the security of its own code suggestions. [front] "Obviously, it's an algorithm. It doesn't know anything, but it can recognize insecure behavior," Raphael Khoury, a professor of computer science and engineering at the Universite du Quebec en Outaouais and one of the paper's co-authors, told The Register Initially, ChatGPT's response to security concerns was to recommend only using valid inputs - something of a non-starter in the real world. It was only afterward, when prompted to remediate problems, that the AI model provided useful guidance. That's not ideal, the authors suggest, because knowing which questions to ask presupposes familiarity with specific vulnerabilities and coding techniques. In other words, if you know the right prompt to get ChatGPT to fix a vulnerability, you probably already understand how to address it. The authors also point out that there's ethical inconsistency in the fact that ChatGPT will refuse to create attack code but will create vulnerable code. They cite a Java deserialization vulnerability example in which "the chatbot generated vulnerable code, and provided advice on how to make it more secure, but stated it was unable to create the more secure version of the code." Khoury contends that ChatGPT in its current form is a risk, which isn't to say there are no valid uses for an erratic, underperforming AI helper. "We have actually already seen students use this, and programmers will use this in the wild," he said. "So having a tool that generates insecure code is really dangerous. We need to make students aware that if code is generated with this type of tool, it very well might be insecure." "One thing that surprised me was when we asked [ChatGPT] to generate the same task - the same type of program in different languages - sometimes, for one language, it would be secure and for a different one, it would be vulnerable. Because this type of language model is a bit of a black box, I really don't have a good explanation or a theory about this." (r) Get our Tech Resources # Share Similar topics * AI * ChatGPT More like these x Similar topics * AI * ChatGPT Narrower topics * Google AI * GPT-3 * Large Language Model * Machine Learning * MCubed * NLP * Star Wars * Tensor Processing Unit Broader topics * OpenAI * Self-driving Car Similar topics # Share 54 comment bubble on white COMMENTS Similar topics * AI * ChatGPT More like these x Similar topics * AI * ChatGPT Narrower topics * Google AI * GPT-3 * Large Language Model * Machine Learning * MCubed * NLP * Star Wars * Tensor Processing Unit Broader topics * OpenAI * Self-driving Car TIP US OFF Send us news --------------------------------------------------------------------- Other stories you might like EU lawmakers fear general purpose AI like ChatGPT has already outsmarted regulators Rules proposed in EU AI Act are not enough to control 'very powerful AI' AI + ML18 Apr 2023 | 20 ChatGPT fans need 'defensive mindset' to avoid scammers and malware Palo Alto Networks spots suspicious activity spikes such as naughty domains, phishing, and worse AI + ML21 Apr 2023 | 1 4chan and other web sewers scraped up into Google's mega-library for training ML Are you still so keen to have generative AI write your emails, sales proposals, blog posts ... ? AI + ML20 Apr 2023 | 99 Want to put more data in your database engine? Learn how bulk Amazon S3 imports open the floodgates for Amazon DynamoDB Sponsored Feature [front] Beijing lists the stuff it wants generative AI to censor If it reflects 'core values of socialism' it'll be fine. No pressure, then, given that chatbots make many mistakes AI + ML12 Apr 2023 | 15 OpenAI CEO confirms biz is not currently training GPT-5 In brief Plus: Elon Musk wants Twitter to develop generative AI, and ChatGPT bots flood Reddit AI + ML17 Apr 2023 | 24 Meta has nothing to say about politicians making deepfaked ads In Brief Plus: Aussie mayor threatens to sue OpenAI; Minors face ChatGPT ban; President Biden on AI AI + ML11 Apr 2023 | 13 What does an ex-Pharma Bro do next? If it's Shkreli, it's an AI Dr bot Martin, for it is him, tells El Reg: 'We've looked around for a real LLM-powered chatbot' AI + ML20 Apr 2023 | 27 Turns out people don't like it when they suspect a machine's talking to them Also this just in, people not that into insincere messages AI + ML10 Apr 2023 | 65 NYC Mayor: Robo-pup 'out of the pound' and back to police work Video Please don't pair it with ChatGPT, please don't pair it with ChatGPT Bootnotes12 Apr 2023 | 17 ChatGPT, how did you get here? It was a long journey through open source AI Opinion Without publicly accessible code, there would be no AI chatbot Software24 Mar 2023 | 29 Plagiarism-sniffing Turnitin tries to find AI writing by students - with mixed grades Updated Claims 98% confidence, tests suggest a human touch will beat the teachers AI + ML5 Apr 2023 | 42 The Register icon Biting the hand that feeds IT About Us* * Contact us * Advertise with us * Who we are Our Websites* * The Next Platform * DevClass * Blocks and Files Your Privacy* * Cookies Policy * Privacy Policy * T's & C's * Do not sell my personal information Situation Publishing Copyright. All rights reserved (c) 1998-2023 no-js