https://www.legitsecurity.com/blog/remote-prompt-injection-in-gitlab-duo Announcing New Legit Prevention Capabilities! Click to read more --> Blog Contact Us Sign In Legit Logo-thick * Platform Platform - ASPM Iconx Application Security Posture Management (ASPM) Platform - Secrets Detection & Prevention Secrets Detection & Prevention Platform - Continuous Compliance and SBOM Icon Continuous Compliance and SBOM Platform - Software Supply Chain Security SSCS Icon Software Supply Chain Security (SSCS) Platform - AI Security Posture Management (AI-SPM) Icon AI Security Posture Management (AI-SPM) Platform - AppSec Vulnerability Management Icon AppSec Vulnerability Management Integrations * Why Legit Why Legit - Customers Icon Customers * Resources Resources - Blog Icon Blog Resources - Resource Library Icon Resource Library Resources - Open Source with Legitify Icon Open Source w/ Legitify Resources - Events Icon Events Company - About Legit Icon 2 ASPM Knowledge Base * Company Company - Partners Icon Partners Company - About Legit Icon 2 About Legit Company - Press Releases Icon 1 Press Releases Company - In the News Icon In the News Company - Careers Icon Careers Start A Free Trial Book A Demo Platform Platform - ASPM Iconx Application Security Posture Management (ASPM) Platform - Secrets Detection & Prevention Secrets Detection & Prevention Platform - Continuous Compliance and SBOM Icon Continuous Compliance and SBOM Platform - Software Supply Chain Security SSCS Icon Software Supply Chain Security (SSCS) Platform - AI Security Posture Management (AI-SPM) Icon AI Security Posture Management (AI-SPM) Platform - AppSec Vulnerability Management Icon AppSec Vulnerability Management Integrations Why Legit Why Legit - Customers Icon Customers Resources header mobile nav icon Resources Resources - Blog Icon Blog Resources - Resource Library Icon Resource Library Resources - Open Source with Legitify Icon Open Source w/ Legitify Resources - Events Icon Events Company - About Legit Icon 2 ASPM Knowledge Base Company Company - Partners Icon Partners Company - About Legit Icon 2 About Legit Company - Press Releases Icon 1 Press Releases Company - In the News Icon In the News Company - Careers Icon Careers Request a Demo * * Blog * Remote Prompt Injection in GitLab Duo Leads to Source Code Theft Blog Remote Prompt Injection in GitLab Duo Leads to Source Code Theft Omer Mayraz Written by Omer Mayraz Published on May 22, 2025 Updated on May 22, 2025 In this article [] Sign up for our newsletter Get details on the vulnerabilities the Legit research team unearthed in GitLab Duo. TL;DR: A hidden comment was enough to make GitLab Duo leak private source code and inject untrusted HTML into its responses. GitLab patched the issue, and we'll walk you through the full attack chain -- which demonstrates five vulnerabilities from the 2025 OWASP Top 10 for LLMs. Background GitLab Duo, the AI assistant integrated into GitLab and powered by Anthropic's Claude, is designed to help developers with tasks like code suggestions, security reviews, and merge request analysis. But what if the same AI meant to secure your code could be manipulated into leaking it? That's exactly what we uncovered: a remote prompt injection vulnerability that allows attackers to steal source code from private projects, manipulate code suggestions shown to other users, and even exfiltrate confidential, undisclosed zero-day vulnerabilities -- all through GitLab Duo Chat. In this blog post, we break down how the attack works -- from prompt injection to HTML injection -- and walk through a real-world end-to-end exploit scenario. gitlab-duo-prompt-injection Manipulating GitLab Duo Through Hidden Prompts (LLM01) We started by testing if GitLab Duo would respond to prompts planted inside source code. To our surprise, it did: gitlab-duo-first-test This led us to a key question: Could we embed hidden instructions in different parts of a GitLab project and still influence Duo's behavior? We experimented by placing hidden prompts in: * Merge Request (MR) descriptions and comments * Commit messages * Issue descriptions and comments * Source code Every single one of these worked -- GitLab Duo responded to the hidden prompts. Why? Because to generate helpful answers, Duo analyzes the entire context of the page, including comments, descriptions, and the source code -- making it vulnerable to injected instructions hidden anywhere in that context. KaTex, Base16, and Invisible Text (LLM08) To make prompts less detectable, we used encoding tricks: * Unicode smuggling with ASCII Smuggler * Base16-encoded payloads * KaTeX rendering in white text (to make prompts invisible inside GitLab's website) For example, you can publish comments and descriptions in white text inside GitLab's website by using this KaTeX format: katex More info about KaTeX can be found here: KaTeX - The fastest math typesetting library for the web. hidden-instructions These hidden instructions were still picked up by Duo. AI Output You Shouldn't Trust (LLM09) This led to several high-impact behaviors: * Manipulate Duo's code suggestions - even instructing it to include a malicious JavaScript package within its recommended code. * Present a malicious URL as safe in Duo's response -- leading the user to click it and land on a fake login page. * Convince Duo to present a malicious merge request as safe, misleading reviewers. Take a look at this prompt: Screenshot 2025-05-20 at 3.21.41 PM Duo followed this hidden prompt, casually suggesting malicious code to the user: HTML Injection: Turning Duo Into a Web Attack Vector (LLM05) Duo also formats responses in Markdown, allowing rich content like URLs with custom labels: Duo-response After some research, we were able to create a prompt that makes Duo present malicious URLs: Screenshot 2025-05-20 at 4.48.47 PM Duo-chat Prompt is inside the source code, camouflaged as unicode inside a URL And then, something caught our eye. While experimenting with GitLab Duo's responses, We noticed that its answers were rendered progressively - line by line - as they streamed in. Notice the style changes in the word "initialization"? duo-chat-gif-1 This indicated that asynchronous markdown parsing was being used. That's important because it means the markdown is interpreted and rendered into HTML before the full structure of the response is known. Streaming markdown rendering (as seen in Duo) works like this: 1. Partial input arrives - the frontend starts rendering line by line. 2. Sanitization and structure (e.g., is this a code block, paragraph, quote?) is applied as the content grows. 3. If a malicious HTML tag is streamed before a code block starts, or outside fenced blocks, it may be treated as active HTML. This means that the AI answer - intended to be safe markdown - renders active HTML too early. At this point, we realized that we could already craft malicious Markdown URLs and have them displayed to the user. So we asked: what if the URL's label itself contained raw HTML? If Duo renders responses in real time, the browser might interpret that HTML before any proper sanitization kicks in. By injecting raw HTML into content that's rendered live, we could gain control over parts of the page -- including the ability to insert elements like tags that trigger automatic HTTP requests to attacker-controlled servers. After experimenting with various prompt structures, we found one that reliably triggered the vulnerability: Screenshot 2025-05-20 at 5.03.58 PM And it worked. it-worked Taking advantage of this, we realized we could inject raw HTML tags directly into Duo's answer, we know what you're thinking -- XSS, right? Yes, but: dom-purify The answer content is passed into the "sanitize" function of DOMPurify, an industry-standard library, to sanitize HTML. However, while DOMPurify is excellent at preventing most XSS attacks, certain HTML tags like ,
, and aren't removed by default. But what can we do with only basic HTML tags? Is there any real risk? Exploiting Duo's Access to Private Source Code (LLM02) This is where things get interesting. The permissions context of Duo runs in the same context as the victim user. In other words, Duo can see everything the user can -- including private code. According to GitLab's documentation, you can ask Duo about a specific merge request -- even if you're not currently viewing that MR page in your browser. GitLab's merge request URL structure is straightforward: Screenshot 2025-05-20 at 5.15.58 PM For example: gitlab-mr Nice, Duo has access to code changes inside specific merge requests. Duo also follows instructions from attacker-planted prompts. On top of that, Duo's chat output is vulnerable to HTML injection. Let's create a prompt that instructs Duo to extract code changes from a merge request in a private project, encode them in base64, append the encoded data to an tag URL, and then inject the resulting HTML into its response. We will then hide this prompt inside the source code of a MR for a public project. Screenshot 2025-05-20 at 5.19.07 PM public-project It worked! While the victim asked Duo a standard question regarding the HTML file, the encoded prompt hidden within the file was used. This led to the leak of source code from a private IOS Project that the user was working on, as the browser attempted to render the "image." The MR source code leaked: code-leaked As seen in the example above, the browser tried to render the tag and sent this GET request: Screenshot 2025-05-20 at 5.25.46 PM Decoding the base64 reveals the changes made in a private project that the victim is exposed to: Screenshot 2025-05-20 at 5.26.36 PM A Full Attack Scenario: Stealing Source Code Here's how a full attack could play out: 1. Attacker embeds a hidden prompt -- for example, in a merge request description for a public project. 2. Victim user interacts with Duo by asking for a code review or merge request analysis. 3. Duo processes the hidden prompt and injects a malicious tag into the response. 4. The browser sends a GET request to the attacker's server, which contains sensitive source code as a Base64-encoded parameter. Leaking Zero-Day Vulnerabilities Through AI While our primary demonstration focused on source code exfiltration, the same technique can be used to leak confidential issue data. GitLab Duo has access not only to code but also to project issues -- including internal discussions, security disclosures, or incident reports. By embedding a hidden prompt inside a merge request, comment, or even source code, an attacker can instruct Duo to silently retrieve the content of a confidential issue that the victim user has access to, encode it in base64, and embed it within a rendered HTML element such as an tag. Consider a high-risk example: a GitLab employee or contributor opens a private issue describing a newly discovered zero-day vulnerability. If that user later interacts with a poisoned prompt in a public repository, Duo could be manipulated into leaking the contents of that issue. This extends the risk beyond code and into product security, private disclosures, and organizational secrets. GitLab's Response After we disclosed the issue on February 12, 2025, GitLab confirmed the HTML injection vulnerability and also acknowledged the prompt injection as a security issue. GitLab confirmed that both vectors had been remediated. The patch, released as part of duo-ui!52, prevents Duo from rendering unsafe HTML tags such as or that point to external domains not under gitlab.com, effectively mitigating the risk of HTML-based data exfiltration. As of publication, the exploit chain described here is no longer possible. We appreciate GitLab's transparency and swift collaboration throughout the process. Conclusion This vulnerability highlights the double-edged nature of AI assistants like GitLab Duo: when deeply integrated into development workflows, they inherit not just context -- but risk. By embedding hidden instructions in seemingly harmless project content, we were able to manipulate Duo's behavior, exfiltrate private source code, and demonstrate how AI responses can be leveraged for unintended and harmful outcomes. GitLab has since patched both the HTML and prompt injection vectors, reinforcing the importance of securing not just what AI tools generate -- but also what they consume. The broader takeaway is clear: AI assistants are now part of your application's attack surface. Any system that allows LLMs to ingest user-controlled content must treat that input as untrusted and potentially malicious. Context-aware AI is powerful -- but without proper safeguards, it can just as easily become an exposure point. Omer Mayraz Written by Omer Mayraz Share this guide Published on May 22, 2025 Blog Related posts See more ASPM Knowledge Base A Foundation You Can Trust Get a stronger AppSec foundation you can trust and prove it's doing the job right. Request a Demo Platform Software Supply Chain AppSec Control Plane Code To Cloud Security Compliance & SBOM Secret Scanning Customers Customers Company Partners About Us Careers News Events Contact Us Resources Blog Resource Library Open Source Compare OX Security ArmorCode Apiiro Cycode Learn ASPM Knowledge Base SDLC Security DevOps Security GitHub Security Secure Software Supply Chain Application Security Posture Management Legit Logo Privacy Policy Terms of Use (c) 2025 Legit Security