Post AwWmhGNAiKi8qWD3aa by mathcolorstrees@mstdn.social
(DIR) More posts by mathcolorstrees@mstdn.social
(DIR) Post #AuW44Rfd1lpY7qIUe8 by jcoglan@mastodon.social
2025-05-27T07:04:22Z
0 likes, 0 repeats
if I have to hear the term "prompt injection" one more time
(DIR) Post #AuW44TEPDxF8xzZmUa by jcoglan@mastodon.social
2025-05-27T07:12:28Z
2 likes, 4 repeats
you cannot have an "injection attack" in a system with no formal distinction between data and instructions. what you actually have is an "everything is instructions" model and a failure to isolate untrusted inputs from the elevated privilege of access to private information
(DIR) Post #AuW44YqAMzQ2MYHVXk by jcoglan@mastodon.social
2025-05-27T07:14:04Z
0 likes, 0 repeats
this is not a novel or surprising means of attacking systems. of course, obviously, if you give a system with no formalisable behaviour, that may execute anything as an instruction by design, elevated privileges and untrusted input, this will happen
(DIR) Post #AuW44dwjO0doCMOJyC by jcoglan@mastodon.social
2025-05-27T07:15:05Z
0 likes, 0 repeats
the category error here is deploying something with unformalisable behaviour with any privilege to do anything at all without the express confirmation of a trusted human operator
(DIR) Post #AuW44j54IRHU7fKY9w by jcoglan@mastodon.social
2025-05-27T07:16:42Z
1 likes, 0 repeats
the way people are deploying LLMs is driving a freight train through the principle of least privilege and being surprised at the results
(DIR) Post #AuW44o1LvgHPRad9Yu by jcoglan@mastodon.social
2025-05-27T07:19:20Z
1 likes, 0 repeats
like it's not even the sort of security flaw you get from people not realising non-obvious properties of their platform and failing to guard against them, it's just ignoring foundational concepts in security engineering on purpose
(DIR) Post #AwWmS7Z5BBIjBXYoHw by divVerent@social.vivaldi.net
2025-07-26T14:18:13Z
0 likes, 0 repeats
@jcoglan Since when do LLMs have any privilege at all? I do thought until now all LLM decisions go through humans anyway.Isn't prompt injection about users being able to circumvent a LLM provider's content rules by means of convincing?As such, it is nothing other than social engineering against LLMs.
(DIR) Post #AwWmhFMQTcWJhuqwmu by glyph@mastodon.social
2025-05-28T01:56:19Z
0 likes, 0 repeats
@jcoglan I think the original insight was good but the term has begun to chafe and your explanation is absolutely on point as to why. We need a punchier term that encapsulates this understanding but it’s hard to describe as an “exploit” or a “vulnerability class” when it’s just a total, comprehensive, catastrophic misunderstanding of the entire concept of security. With MCP shit rolling out at a breakneck pace we are truly entering the time of monsters
(DIR) Post #AwWmhGNAiKi8qWD3aa by mathcolorstrees@mstdn.social
2025-07-26T06:51:16Z
0 likes, 0 repeats
@glyph @jcoglan I am very interested in hearing more. Any way we can connect over email?
(DIR) Post #AwWmhH91qOzZEwbHBQ by mathcolorstrees@mstdn.social
2025-07-26T06:56:13Z
0 likes, 0 repeats
@glyph @jcoglan a bit more context is that I want my teams to build security specifically for these cases.
(DIR) Post #AwWmhHgLqVeAuIBt7g by glyph@mastodon.social
2025-07-26T07:40:47Z
0 likes, 0 repeats
@mathcolorstrees @jcoglan Oh. You can't. That's the point; the way to secure MCP tools is to either review every single action the model can perform manually, or, do not expose any actions that would be problematic to execute in any combination in any order. Consider the model as a random number generator that might do anything. If any part of the prompt is attacker-controlled, then it's a random number generator they get to bias the output of.If you want to email me though, feel free.
(DIR) Post #AwWmhIQn3qnHEJuyVU by divVerent@social.vivaldi.net
2025-07-26T14:20:52Z
0 likes, 0 repeats
@glyph @mathcolorstrees @jcoglan LLMs usually are not able to perform any actions that mutate anything, but sometimes are able to e.g. read documents on behalf of the user.As such, it is crucial to ensure the LLM only has access to documents the user should have access to. By some means of credential forwarding.Same applies to human operators - there too similar safeguards are necessary to ensure the operator has access to no one's data other than of the person on the phone.And do not even think about training LLMs on private data.
(DIR) Post #AwWmvr1VFG74ZO21xY by divVerent@social.vivaldi.net
2025-07-26T14:23:08Z
0 likes, 0 repeats
@jcoglan These things usually have a distinction (e.g. they may be able to generate search queries). The bug is when such queries are executed in any context other than the user's. A LLM definitely is not ever supposed to have higher privileges than the user using it, with some very few exceptions where damage is limited (e.g. a LLM performing technical support may be able to see the user's account history even though the user cannot see it - as the user can only get their own data this way, this is acceptable).
(DIR) Post #AwWn2aiqFZYQ2YW1uC by divVerent@social.vivaldi.net
2025-07-26T14:24:52Z
0 likes, 0 repeats
@glyph @mathcolorstrees @jcoglan Easy: "wrapper" has to ensure whenever the LLM contacts the MCP, it also tells the MCP who the user using the LLM is by some means.Then the MCP can only perform actions the user is authorized to do.
(DIR) Post #AwWn4Ggzgvs4UwS1Sq by david_chisnall@infosec.exchange
2025-05-27T11:47:25Z
0 likes, 0 repeats
@jcoglan I am too young to remember the phone phreaking attacks that made everyone learn that in-band signalling is a bad idea. I am old enough to remember the ping-of-death attacks that reminded people. LLMs will remind an entire new generation.
(DIR) Post #AwWnvSetvpoKxUpv4y by jcoglan@mastodon.social
2025-07-26T14:34:46Z
0 likes, 0 repeats
@divVerent @glyph @mathcolorstrees this is a non solution, there are tons of things an app developer is "authorized" to do that they constantly decide not to do because it would be a bad idea in their current circumstances
(DIR) Post #AwWoFi3mNcPUW580au by jcoglan@mastodon.social
2025-07-26T14:38:24Z
0 likes, 0 repeats
@divVerent you can run LLM based agents on your machine to interact with your source code and infrastructure, which is what this is referring to
(DIR) Post #AwWoyN0fUUDGGHy7g8 by divVerent@social.vivaldi.net
2025-07-26T14:46:31Z
0 likes, 0 repeats
@jcoglan But then only you can enter prompts. Where is the attack? You already have full access to this same data without the LLM.If the LLM can fetch prompts remotely, that would be bad, yes.If the LLM thinks it must delete all your source because you said Hello and something like this was in its training data, that is a bug, not an attack.
(DIR) Post #AwWpCDty9QG04GROzo by jcoglan@mastodon.social
2025-07-26T14:49:01Z
0 likes, 0 repeats
@divVerent even when using an agent that only I was notionally in control of, it kept disobeying my explicit instructions. these things will by and large do whatever *they* believe would be useful, with the full privileges of the current userI'm *allowed* to delete everything in $HOME, but I don't, for obvious reasons
(DIR) Post #AwWpLbzPxiQhNhSyI4 by divVerent@social.vivaldi.net
2025-07-26T14:50:44Z
0 likes, 0 repeats
@jcoglan Sure, but that falls under the category "bug" and is an old problem. See that bumblebee issue back then.
(DIR) Post #AwWpYpaGjJXhJA0Sw4 by jcoglan@mastodon.social
2025-07-26T14:53:06Z
0 likes, 0 repeats
@divVerent I'm not sure you can categorise behaviour that agents exhibit routinely as a bug unless their vendors have a massive crackdown on such behaviour
(DIR) Post #AwWqn7KtcnjGNDByfA by divVerent@social.vivaldi.net
2025-07-26T15:06:54Z
0 likes, 0 repeats
@jcoglan In any case, the LLM should be treated and trusted just like the intern you hired last week. Because that is what it is - just faster.
(DIR) Post #AwWr18k7BvFKn4ft7Q by jcoglan@mastodon.social
2025-07-26T15:09:25Z
0 likes, 0 repeats
@divVerent the "intern" analogy is mostly revealing of what people think of interns, and it's not good. the two things are not remotely analogous
(DIR) Post #AwWsNTVMmLQfVYzRb6 by divVerent@social.vivaldi.net
2025-07-26T15:24:41Z
0 likes, 0 repeats
@jcoglan I would give nobody who got hired last week and has zero industry experience but came fresh from university on a time limited contract unfettered and unreviewed root access to my systems.Having said that, ideally no one at all should have this kind of access.
(DIR) Post #AwWsR49mxpfi3DbrQO by sirosen@mastodon.social
2025-07-26T14:58:53Z
0 likes, 0 repeats
@jcoglan @divVerent @glyph @mathcolorstrees I don't even "decide" not to do many of those things. "Should I point my application testsuite at the production database?"The only believable models of interaction I've seen here are (1) there's a human in the loop who reviews everything, or (2) the entire thing is sandboxed, e.g. in a VM. In both cases you're explicitly lowering your permissions somehow, because you cannot trust the machine.
(DIR) Post #AwWsdPAydoAiiR36zA by divVerent@social.vivaldi.net
2025-07-26T15:27:34Z
0 likes, 0 repeats
@jcoglan I see no reason to ever give a LLM full privileges of the user.No more than RW access in the IDE's context (i.e. just this project, and no access to .git subdirectory contents or similar version control data, and of course no pushing, we can MAYBE talk about committing to the local repo, but no amending, rebasing etc.).