Post B7Qkl03Jsa3QMwssjI by xgranade@wandering.shop
(DIR) More posts by xgranade@wandering.shop
(DIR) Post #B7QkkrxLysWpG820Js by xgranade@wandering.shop
0 likes, 1 repeats
RE: https://wandering.shop/@xgranade/116764432226058529So, about in-band and out-of-band signaling.Suppose you're busy inventing the telephone, and you decide to do so by connecting a wire for your microphone to your friend's speaker and vice versa.That takes connecting your side directly to theirs, but it works.Now you want to call *two* people with your fancy new invention, so you change the wires out of your phone to end in a plug, and have your two friends do the same.(1/many)RT: https://wandering.shop/users/xgranade/statuses/116764432226058529
(DIR) Post #B7QkksCb4Cio1PAB4y by xgranade@wandering.shop
0 likes, 0 repeats
Now, someone can sit at a board with a bunch of jacks and, by moving those plugs between different jacks, connect your phone to either friend 1 or friend 2.But then you need someone to sit there and work that board. Fine, you hire someone to do that. Now your phone goes to that operator, you tell them who you want to talk to, then they move the plugs around to make that happen.Next, say you want to call a million friends, and for each of them to be able to call everyone else.
(DIR) Post #B7QkksPiHRDIg5IeWW by xgranade@wandering.shop
0 likes, 0 repeats
You'd need to hire a *lot* of operators to make that work, but whatever. Your friends pay you, you pay the operators, and everyone's happy.Except.Now they're not your friends, because you're charging them money for a service. They're your *customers*, and you need to cut costs.So fine. You fire all the operators and build a machine that does all the moving of plugs between jacks. Much cheaper.Except.How do you tell the machine what to do?
(DIR) Post #B7QkksblYcr3HSwHJI by xgranade@wandering.shop
0 likes, 0 repeats
You, being the greedy capitalist that you are, just fired everyone who can take instructions and do shit based on that. Your fancy new machine needs to know who wants to call whom and when.Worse, your million former friends all have telephones and don't want to buy new ones — especially not from you!So you're stuck, all the instructions to your fancy new machine have to work with just a single wire for each of the microphone and speaker.
(DIR) Post #B7QkksnopoUnsqZu64 by xgranade@wandering.shop
0 likes, 0 repeats
Someone then has a brilliant idea that will never, ever go wrong. Make your machine take instructions only as sounds that no human can make.Except.That doesn't work because all those wires connecting telephones only really work for the kinds of noises humans tend to make, you never needed them to do anything else!So fine. You make your machine take instructions only after someone sends an *extremely pure* tone, the kind no one could make with their mouths.Except.
(DIR) Post #B7Qkkx8Ak59ZJ88DNQ by xgranade@wandering.shop
0 likes, 0 repeats
One of your friends-now-customers really likes their breakfast cereal, and they just happened to get a whistle in a box of Cap'n Crunch one day.They tell their friends about how awesome their whistle is, and oops! turns out it makes *exactly the right noise* to get your fancy new switch machine to take instructions.
(DIR) Post #B7Qkkye76oIW0U5Eno by xgranade@wandering.shop
0 likes, 0 repeats
As one might guess, this actually fucking happened, modulo that I'm being very glib about the history and all, taking some pretty extreme creative liberties for the purpose of simplifying the presentation.But the essential elements are all more or less here: it turns out to be a very bad idea to make it so your machines take instructions based on filtering those instructions out of untrusted data.https://en.wikipedia.org/wiki/Phreaking
(DIR) Post #B7Qkl03Jsa3QMwssjI by xgranade@wandering.shop
0 likes, 0 repeats
Sometimes, because of whatever history and the like, the only ideas available are bad ideas and you need to pick the least-bad from amongst them, so yeah, you can find in-band signalling (that is, where instructions are filtered out of untrusted inputs) in use in the real world.If you can, though, using some other kind of way of providing instructions entirely separately from the data your machine acts on? That's much better and more secure.
(DIR) Post #B7Qkl1VMTo4ysD0n4q by xgranade@wandering.shop
0 likes, 0 repeats
The trouble for AI vendors is that LLMs fundamentally cannot do anything but in-band signaling. There is simply no way to do anything *but* try and filter out bad words from the jumble of words that you call a "prompt."Sometimes, attackers find a way to get your machine to confuse your out-of-band signaling and effectively make your system into one that uses in-band signalling; that's in essence what an *injection* attack is. But that's not how LLMs work! At all!
(DIR) Post #B7Qkl2vzAIyDJ4TZDM by xgranade@wandering.shop
0 likes, 0 repeats
Except.It's worse than that. In the Cap'n Crunch whistle case, it's at least a closed problem as to how to filter out instructions from an untrusted stream. You still need to worry about when it's safe to let instructions through and when it isn't, so out-of-band is still much better, but at least you know what frequencies to filter out and the like.For LLMs, any sequence of words could be bad for your product! Filtering out instructions is a completely open-ended problem!
(DIR) Post #B7Qkl4RDZfXzyE61XE by xgranade@wandering.shop
0 likes, 0 repeats
So, no. Prompt injection isn't a thing. Injection attacks work by turning out-of-band systems into in-band systems, but LLMs start as in-band and fundamentally always will be in-band. The fundamental way that LLMs work is indistinguishable from the result of someone successfully hitting you with an injection attack.
(DIR) Post #B7QklRj4zQV8CHd74K by ricci@discuss.systems
1 likes, 0 repeats
@xgranade okay, you're right, but we're missing out on Prompt Injection aS a Service (PISS)
(DIR) Post #B7QkmUsCm3QGFPOooi by ricci@discuss.systems
1 likes, 0 repeats
@xgranade pretty wild that people are putting hundreds of billions of dollars into building products that by design just do whatever the fuck