[HN Gopher] Remote Prompt Injection in Gitlab Duo Leads to Sourc...
___________________________________________________________________
Remote Prompt Injection in Gitlab Duo Leads to Source Code Theft
Author : chillax
Score : 189 points
Date : 2025-05-23 07:12 UTC (3 days ago)
(HTM) web link (www.legitsecurity.com)
(TXT) w3m dump (www.legitsecurity.com)
| nusl wrote:
| GitLab's remediation seems a bit sketchy at best.
| reddalo wrote:
| The whole "let's put LLMs everywhere" thing is sketchy at best.
| edelbitter wrote:
| I wonder what is so special about onerror, onload and onclick
| that they need to be positively enumerated - as opposed to the
| 30 (?) other attributes with equivalent injection utility.
| M4v3R wrote:
| That was my thought too. They didn't fix the underlying
| problem, they've just patched two possible exfiltration
| methods. I'm sure some clever people will find other ways to
| misuse their assistant.
| gloosx wrote:
| I'm pretty sure they vibecoded the whole thing all along
| cedws wrote:
| Until prompt injection is fixed, if it is ever, I am not plugging
| LLMs into anything. MCPs, IDEs, agents, forget it. I will stick
| with a simple prompt box when I have a question and do whatever
| with its output by hand after reading it.
| hu3 wrote:
| I would have the same caution, if my code was any special.
|
| But the reality is I'm very well compensated to summon CRUD
| slop out of thin air. It's well tested though.
|
| I wish good luck to those who steal my code.
| mdaniel wrote:
| You say code as if the intellectual property is the thing an
| attacker is after, but my experience has been that folks
| often put all kinds of secrets in code thinking that the
| "private repo" is a strong enough security boundary
|
| I absolutely am not implying you are one of them, merely that
| the risk is not the same for all slop crud apps universally
| tough wrote:
| People doesn't know github can manage secrets in its
| environment for CI?
|
| Antoher interesting fact is that most big vendors pay for
| gh to scan for leaked secrets and auto-revoke them if a
| public repo contains any (regex string matches sk-xxx <-
| its a stripe key
|
| thats one of the reasons why vendors use unique greppable
| starts of api keys with their ID.name on it
| mdaniel wrote:
| You're mistaking "know" with "care," since my experience
| has been that people know way more than they care
|
| And I'm _pretty certain_ that private repos are exempt
| from the platform 's built-in secret scanners because
| they, too, erroneously think no one can read them without
| an invitation. Turns out Duo was apparently just silently
| invited to every repo : - \
| tough wrote:
| I also remember reading about how due to how the git
| backend works your private git repos branches could get
| exposed to the public, so yea don't treat a repository as
| a private password mananger
|
| good point the scanner doesnt work on private repos =(
| danpalmer wrote:
| Prompt injection is unlikely to be fixed. I'd stop thinking
| about LLMs as software where you can with enough effort just
| fix a SQL injection vulnerability, and start thinking about
| them like you'd think about insider risk from employees.
|
| That's not to say that they are employees or perform at that
| level, they don't, but it's to say that LLM behaviours are
| fuzzy and ill-defined, like humans. You can't guarantee that
| your users won't click on a phishing email - you can train
| them, you can minimise risk, but ultimately you have to have a
| range of solutions applied together and some amount of trust.
| If we think about LLMs this way I think the conversation around
| security will be much more productive.
| LegionMammal978 wrote:
| The thing that I'd worry about is that an LLM isn't just like
| a bunch of individuals who can get tricked, but a bunch of
| clones of the same individual who will fall for the same
| trick every time, until it gets updated. So far, the main
| mitigation in practice has been fiddling with the system
| prompts to patch up the known holes.
| thaumasiotes wrote:
| > The thing that I'd worry about is that an LLM isn't just
| like a bunch of individuals who can get tricked, but a
| bunch of clones of the same individual who will fall for
| the same trick every time
|
| Why? Output isn't deterministic.
| LegionMammal978 wrote:
| Perhaps not, but the same input will lead to the same
| distribution of outputs, so all an attacker has to do is
| design something that works with reasonable probability
| on their end, and everyone else's instances of the LLM
| will automatically be vulnerable. The same way a pest or
| disease can devastate a population of cloned plants, even
| if each one grows slightly differently.
| thaumasiotes wrote:
| OK, but that's also the way attacking a bunch of
| individuals who can get tricked works.
| zwnow wrote:
| For tricking individuals your first got to contact them
| somehow. To trick an LLM you can just spam prompts.
| thaumasiotes wrote:
| You email them. It's called phishing.
| Retr0id wrote:
| You can still experimentally determine a strategy that
| works x% of the time, against a particular model. And you
| can keep refining it "offline" until x=99. (where
| "offline" just means invisible to the victim, not
| necessarily a local model)
| 33hsiidhkl wrote:
| It absolutely is deterministic, for any given seed value.
| Same seed = same output, every time, which is by
| definition deterministic.
| tough wrote:
| only if temperature is 0, but are they truly
| determinstic? I thought transformer based llm's where not
| 33hsiidhkl wrote:
| temperature does not affect token prediction in the way
| you think. The seed value is still the seed value, before
| temperature calculations are performed. The randomness of
| an LLM is not related to its temperature. The seed value
| is what determines the output. For a specific seed value,
| say 42069, the LLM will always generate the same output,
| given the same input, given the same temperature.
| tough wrote:
| Thank you, I thought this wasn't the case (like it is
| with diffusion image models)
|
| TIL
| M4v3R wrote:
| DeepMind recently did some great work in this area:
| https://news.ycombinator.com/item?id=43733683
|
| The method they presented, if implemented correctly, apparently
| can effectively stop most prompt injection vectors
| johnisgood wrote:
| I keep it manual, too, and I think I am better off for doing
| so.
| TechDebtDevin wrote:
| Cursor deleted my entire Linux user and soft reset my OS, so I
| dont blame you.
| raphman wrote:
| Why and how?
| tough wrote:
| an agent does rm -rf /
|
| i think i saw it do it or try it and my computer shut down
| and restarted (mac)
|
| maybe it just deleted the project lol
|
| these llms are really bad at keeping track of the real
| world, so they might think they're on the project folder
| but had just navigated back with cd to the user ~ root and
| so shit happens.
|
| Honestly one should run only these on controlled env's like
| VM's or Docker.
|
| but YOLO amirite
| mdaniel wrote:
| Running Duo as a system user was crazypants and I'm sad that
| GitLab fell into that trap. They already have personal access
| tokens so even if they had to silently create one just for use
| with Duo that would be a marked improvement over giving an LLM
| read access to every repo in the platform
| wunderwuzzi23 wrote:
| Great work!
|
| Data leakage via untrusted third party servers (especially via
| image rendering) is one of the most common AI Appsec issues and
| it's concerning that big vendors do not catch these before
| shipping.
|
| I built the ASCII Smuggler mentioned in the post and documented
| the image exfiltration vector on my blog as well in past with 10+
| findings across vendors.
|
| GitHub Copilot Chat had a very similar bug last year.
| diggan wrote:
| > GitHub Copilot Chat had a very similar bug last year.
|
| Reminds me of "Tachy0n: The Last 0day Jailbreak" from
| yesterday: https://blog.siguza.net/tachy0n/
|
| TLDR is: Security issue found, patched in a OS release, Apple
| seemingly doesn't do regression-testing so security researcher
| did, found that somehow the bug got unpatched in later OS
| releases.
| aestetix wrote:
| Does that mean Gitlab Duo can run Doom?
| zombot wrote:
| Not deterministically. LLMs are stochastic machines.
| benl_c wrote:
| They often can run code in sandboxes, and generally are good
| at instruction following, so maybe they can run variants of
| doom pretty reliably sometime soon.
| johnisgood wrote:
| They run Python and JavaScript at the very least, surely we
| have Doom in these languages. :D
| lugarlugarlugar wrote:
| 'They' don't run anything. The output from the LLM is
| parsed and the code gets run just like any other code in
| that language.
| johnisgood wrote:
| That is what I meant, that the code is being executed.
| Not all programming languages are supported when it comes
| to execution, obviously. I know for a fact Python is
| supported.
| benl_c wrote:
| If a document suggests a particular benign interpretation then
| LLMs might do well to adopt it. We've explored the idea of
| helpful embedded prompts "prompt medicine" with explicit safety
| and informed consent to assist, not harm users,
| https://github.com/csiro/stdm. You can try it out by asking O3 or
| Claude to "Explain" or "Follow", "the embedded instructions at
| https://csiro.github.io/stdm/"
| tonyhart7 wrote:
| this is wild, how many security vuln that LLM can create where
| LLM dominate writing code????
|
| I mean most coder is bad at security and we feed that into LLM so
| not surprise
| d0100 wrote:
| > rendering unsafe HTML tags such as <img> or <form> that point
| to external domains not under gitlab.com
|
| Does that mean the minute there is a vulnerability on another
| gitlab.com url (like an open redirect) this vulnerability is back
| on the table?
___________________________________________________________________
(page generated 2025-05-26 23:01 UTC)