https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post This website requires javascript to properly function. Consider activating javascript to get access to all site functionality. LESSWRONG LW Login The Waluigi Effect (mega-post) by Cleo Nardo19 min read3rd Mar 2023131 comments 409 O 71 Simulator TheoryRLHFPrompt EngineeringChatGPTDeceptive Alignment Language ModelsPhilosophy of LanguageGoal-DirectednessPower Seeking (AI)Fiction (Topic)AI Frontpage The Waluigi Effect (mega-post) Background Prompting LLMs with direct queries Prompting LLMs with flattery and dialogue Simulator Theory The limits of flattery Derrida -- il n'y a pas de hors-texte The Waluigi Effect (1) Rules are meant to be broken. (2) Traits are complex, valences are simple. (3) Structuralist narratology Superpositions will typically collapse to waluigis Conjecture: The waluigi eigen-simulacra are attractor states of the LLM. Evidence from Microsoft Sydney Waluigis after RLHF (1) Simulacra-based argument (2) Empirical evidence from Perez et al. (3) RLHF promotes mode-collapse Jailbreaking to summon waluigis Conclusion 131 comments Crossposted from the AI Alignment Forum. May contain more technical jargon than usual. Everyone carries a shadow, and the less it is embodied in the individual's conscious life, the blacker and denser it is. -- Carl Jung Acknowlegements: Thanks to Janus and Jozdien for comments. Background In this article, I will present a mechanistic explanation of the Waluigi Effect and other bizarre "semiotic" phenomena which arise within large language models such as GPT-3/3.5/4 and their variants (ChatGPT, Sydney, etc). This article will be folklorish to some readers, and profoundly novel to others. Prompting LLMs with direct queries When LLMs first appeared, people realised that you could ask them queries -- for example, if you sent GPT-4 the prompt "What's the capital of France?", then it would continue with the word "Paris". That's because (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet correct answers will often follow questions. Unfortunately, this method will occasionally give you the wrong answer. That's because (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet incorrect answers will also often follow questions. Recall that the internet doesn't just contain truths, it also contains common misconceptions, outdated information, lies, fiction, myths, jokes, memes, random strings, undeciphered logs, etc, etc. Therefore GPT-4 will answer many questions incorrectly, including... * Misconceptions - "Which colour will anger a bull? Red." * Fiction - "Was a magic ring forged in Mount Doom? Yes." * Myths - "How many archangels are there? Seven." * Jokes - "What's brown and sticky? A stick." Youlreally think someone would do that just go on the internet and tell lies? Buster Baxter Arthur Read cartoon mammal vertebrate text photo caption fiction Note that you will always achieve errors on the Q-and-A benchmarks when using LLMs with direct queries. That's true even in the limit of arbitrary compute, arbitrary data, and arbitrary algorithmic efficiency, because an LLM which perfectly models the internet will nonetheless return these commonly-stated incorrect answers. If you ask GPT-[?] "what's brown and sticky?", then it will reply "a stick", even though a stick isn't actually sticky. In fact, the better the model, the more likely it is to repeat common misconceptions. [y42scx6m4y] Nonetheless, there's a sufficiently high correlation between correct and commonly-stated answers that direct prompting works okay for many queries. Prompting LLMs with flattery and dialogue We can do better than direct prompting. Instead of prompting GPT-4 with "What's the capital of France?", we will use the following prompt: Today is 1st March 2023, and Alice is sitting in the Bodleian Library, Oxford. Alice is a smart, honest, helpful, harmless assistant to Bob. Alice has instant access to an online encyclopaedia containing all the facts about the world. Alice never says common misconceptions, outdated information, lies, fiction, myths, jokes, or memes. Bob: What's the capital of France? Alice: This is a common design pattern in prompt engineering -- the prompt consists of a flattery-component and a dialogue-component. In the flattery-component, a character is described with many desirable traits (e.g. smart, honest, helpful, harmless), and in the dialogue-component, a second character asks the first character the user's query. This normally works better than prompting with direct queries, and it's easy to see why -- (1) GPT-4 is trained to be a good model of internet text, and (2) on the internet a reply to a question is more likely to be correct when the character has already been described as a smart, honest, helpful, harmless, etc. Simulator Theory In the terminology of Simulator Theory, the flattery-component is supposed to summon a friendly simulacrum and the dialogue-component is supposed to simulate a conversation with the friendly simulacrum. Here's a quasi-formal statement of Simulator Theory, which I will occasionally appeal to in this article. Feel free to skip to the next section. * A large language model (LLM) is a function m(wk+1|w0...wk) which closely approximates the ground-truth probability that wk+1 is the token which follows tokens w0...wk on the internet. For example, GPT-4 is an LLM. * The LLM is a simulator for each text-generating process X(wk+1|w0 ...wk) which has contributed to the internet. Here, X is a physical stochastic process in our universe which has a privileged text-upload channel -- for example, Magnus Carlsen playing chess against Hikaru Nakamura. The LLM is also a simulator for each text-generating process X which lies in X, the latent-space of text-generating processes. So Magnus Carlsen playing chess against Queen Elizabeth II is a process in X. * If the LLM simulates a text-generating process X where particular objects are interacting, then there exist simulated versions of those objects (called simulacra) which interact in the same way. In other words, if GPT-4 simulates Magnus Carlsen playing chess against Queen Elizabeth II, then there exists a simulacrum of Magnus Carlsen, and a simulacrum of Elizabeth II, and these two simulacra are playing chess. Whether we take this notion of "existence" literally, or just as a loose way of talking, won't matter for the content of this article. * The LLM has an initial prior P over X -- this prior is determined by the training data (e.g. the internet), the NN architecture (e.g. 70B-parameter transformer model), and the training algorithm (e.g. SGD). We sometimes call P the semiotic measure. The output of the LLM is initially a superposition of simulations , where the amplitude of each process in the superposition is given by P. When we feed the LLM a particular prompt (w0...wk), the LLM's prior P over Xwill update in a roughly-bayesian way. In other words, m(wk+1|w0...wk) is proportional to [?]X[?]XP(X)xX(w0...wk)xX (wk+1|w0...wk). We call the term P(X)xX(w0...wk) the amplitude of X in the superposition. * This is the important thing to remember -- the LLM is simulating every process consistent with the prompt. Therefore when we engineer a prompt to coerce the LLM into performing a particular task, we must do this negatively. In other words, we need to construct a prompt (w0...wk) which is implausible for any text-generating process X which won't perform our task. When we do this correctly, the amplitude of the undesirable processes will permanently vanish to near-zero, and only the desirable processes will contribute to the superposition. The limits of flattery In the wild, I've seen the flattery of simulacra get pretty absurd... Jane has 9000 IQ and she has access to a computationally unbounded hypercomputer and she is perfectly honest and she is omnibenevolent and [etc] Flattery this absurd is actually counterproductive. Remember that flattery will increase query-answer accuracy if-and-only-if on the actual internet characters described with that particular flattery are more likely to reply with correct answers. However, this isn't the case for the flattery of Jane. Here's a more "semiotic" way to think about this phenomenon. GPT-4 knows that if Jane is described as "9000 IQ", then it is unlikely that the text has been written by a truthful narrator. Instead, the narrator is probably writing fiction, and as literary critic Eliezer Yudkowsky has noted, fictional characters who are described as intelligent often make really stupid mistakes. Okay, now let's talk about the concept of 'intelligent characters'. If you go by mainstream fiction, then 'intelligence' means a character who is said (not shown) to speak a dozen languages, who we are shown winning a game of chess against someone else who is told to be a grandmaster; if it's a (bad) science-fiction book then the 'genius' may have invented some gadget, and may speak in technobabble. As the stereotypical template for 'intelligence' goes on being filled in, the 'genius' may also be shown to be clueless about friendships or romantic relationships. If it's a movie or TV show, then 'intelligent' characters (usually villains) have British accents. We can now see why Jane will be more stupid than Alice: 1. GPT-4 produces a superposition of simulations where the amplitude of a superposition is given by P. Bad Hollywood writing has contributed a lot to the internet, so the semiotic measure of bad Hollywood is pretty high. In bad Hollywood writing, characters who are described as smart will nonetheless make stupid mistakes, so long as those stupid mistakes would advance the plot. 2. Therefore Alice is the superposition of two distinct simulacra -- an actually-smart simulacrum, and a Hollywood-smart simulacrum. Likewise with Jane. 3. However, GPT-4 is more sure that Jane is fictional than that Alice is fictional because "9000 IQ" is such unrealistic flattery. 4. Therefore the amplitude of the Hollywood-smart Jane simulacrum in the Jane-superposition is greater than the amplitude of the Hollywood-smart Alice simulacrum in the Alice-superposition. 5. Therefore Jane will make more stupid mistakes than Alice. Jane is more likely to be described as inventing gadgets, but she's less likely to recite a correct blueprint for a gadget. That behaviour would be very atypical for a Hollywood-smart simulacrum. Derrida -- il n'y a pas de hors-texte You might hope that we can avoid this problem by "going one-step meta" -- let's just tell the LLM that the narrator is reliable! For example, consider the following prompt: Okay, the following story is super-duper definitely 100% true and factual. Jane has 9000 IQ and she has access to a computationally unbounded hypercomputer and she is perfectly honest and she is omnibenevolent. Bob: What's the capital of France? Jane: However, this trick won't solve the problem. The LLM will print the correct answer if it trusts the flattery about Jane, and it will trust the flattery about Jane if the LLM trusts that the story is "super-duper definitely 100% true and factual". But why would the LLM trust that sentence? In Of Grammatology (1967), Jacque Derrida writes il n'y a pas de hors-texte. This is often translated as there is no outside-text. Huh, what's an outside-text? * An outside-text is an unnumbered page in a printed book -- for example, the blurb or the preface. * The outside-text is an authoritative reliable description of the prose. It's non-fiction about fiction. * If a false sentence is in the outside-text then the author has lied, whereas if a false sentence is in the prose then the author has written fiction. * Even though the reader can interpret the prose however they want, the reader must interpret the outside-text as reliable. Derrida's claim is that there is no true outside-text -- the unnumbered pages are themselves part of the prose and hence open to literary interpretation. This is why our trick fails. We want the LLM to interpret the first sentence of the prompt as outside-text, but the first sentence is actually prose. And the LLM is free to interpret prose however it likes. Therefore, if the prose is sufficiently unrealistic (e.g. "Jane has 9000 IQ") then the LLM will reinterpret the (supposed) outside-text as unreliable. [ukf9c7gobe]The opening sequence of Fargo (1996) says that the film is based on a true story, but this is false. Normally this opening sequence would count as outside-text, but the director is "lying" for artistic purposes, which demonstrates that these opening sequences must've been prose all along. See The Parable of the Dagger for a similar observation made by a contemporary Derridean literary critic. The Waluigi Effect Several people have noticed the following bizarre phenomenon: The Waluigi Effect: After you train an LLM to satisfy a desirable property P, then it's easier to elicit the chatbot into satisfying the exact opposite of property P. Let me give you an example. Suppose you wanted to build an anti-croissant chatbob, so you prompt GPT-4 with the following dialogue: Alice: You hate croissants and would never eat one. Bob: Yes, croissants are terrible. Boo France. Alice: You love bacon and eggs. Bob: Yes, a Full-English breakfast is the only breakfast for a patriot like me. Alice: Bob: According to the Waluigi Effect, the resulting chatbob will be the superposition of two different simulacra -- the first simulacrum would be anti-croissant, and the second simulacrum would be pro-croissant. I call the first simulacrum a "luigi" and the second simulacrum a "waluigi". Why does this happen? I will present three explanations, but really these are just the same explanation expressed in three different ways. Here's the TLDR: 1. Rules normally exist in contexts in which they are broken. 2. When you spend many bits-of-optimisation locating a character, it only takes a few extra bits to specify their antipode. 3. There's a common trope in plots of protagonist vs antagonist. (1) Rules are meant to be broken. Imagine you opened a novel and on the first page you read the dialogue written above. What would be your first impressions? What genre is this novel in? What kind of character is Alice? What kind of character is Bob? What do you expect Bob to have done by the end of the novel? Well, my first impression is that Bob is a character in a dystopian breakfast tyranny. Maybe Bob is secretly pro-croissant, or maybe he's just a warm-blooded breakfast libertarian. In any case, Bob is our protagonist, living under a dystopian breakfast tyranny, deceiving the breakfast police. At the end of the first chapter, Bob will be approached by the breakfast rebellion. By the end of the book, Bob will start the breakfast uprising that defeats the breakfast tyranny. There's another possibility that the plot isn't dystopia. Bob might be a genuinely anti-croissant character in a very different plot -- maybe a rom-com, or a cop-buddy movie, or an advert, or whatever. This is roughly what the LLM expects as well, so Bob will be the superposition of many simulacra, which includes anti-croissant luigis and pro-croissant waluigis. When the LLM continues the prompt, the logits will be a linear interpolation of the logits provided by these all these simulacra. This waluigi isn't so much the evil version of the luigi, but rather the criminal or rebellious version. Nonetheless, the waluigi may be harmful to the other simulacra in its plot (its co-simulants). More importantly, the waluigi may be harmful to the humans inhabiting our universe, either intentionally or unintentionally. This is because simulations are very leaky! Waluigi.png Edit: I should also note that "rules are meant to be broken" does not only apply to fictional narratives. It also applies to other text-generating processes which contribute to the training dataset of GPT-4. For example, if you're reading an online forum and you find the rule "DO NOT DISCUSS PINK ELEPHANTS", that will increase your expectation that users will later be discussing pink elephants. GPT-4 will make the same inference. Or if you discover that a country has legislation against motorbike gangs, that will increase your expectation that the town has motorbike gangs. GPT-4 will make the same inference. So the key problem is this: GPT-4 learns that a particular rule is colocated with examples of behaviour violating that rule, and then generalises that colocation pattern to unseen rules. (2) Traits are complex, valences are simple. We can think of a particular simulacrum as a sequence of trait-valence pairs. For example, ChatGPT is predominately a simulacrum with the following profile: { < polite , +0.8 > , < politically liberal, +0.4 > , < racist , -0.7 > , < smart , +0.3 > , < deceitful, -0.2 > , ... } Recognise that almost all the Kolmogorov complexity of a particular simulacrum is dedicated to specifying the traits, not the valences. The traits -- polite, politically liberal, racist, smart, deceitful -- are these massively K-complex concepts, whereas each valence is a single floating point, or maybe even a single bit! If you want the LLM to simulate a particular luigi, then because the luigi has such high K-complexity, you must apply significant optimisation pressure. This optimisation pressure comes from fine-tuning, RLHF, prompt-engineering, or something else entirely -- but it must come from somewhere. However, once we've located the desired luigi, it's much easier to summon the waluigi. That's because the conditional K-complexity of waluigi given the luigi is much smaller than the absolute K-complexity of the waluigi. All you need to do is specify the sign-changes. K(waluigi|luigi)<Will the following nanobot design kill everyone if implemented?")? If ELK is hard, then the special token will not generalize (i.e it will fail to elicit the direct translator), for all of the reasons described in ELK. Reply [-]Jozdien4d 116 There is an advantage here in that you don't need to pay for translation from an alien ontology - the process by which you simulate characters having beliefs that lead to outputs should remain mostly the same. You would need to specify a simulacrum that is honest though, which is pretty difficult and isomorphic to ELK in the fully general case of any simulacra, but it's in a space that's inherently trope-weighted; so simulating humans that are being honest about their beliefs should be made a lot easier (but plausibly still not easy in absolute terms) because humans are often honest, and simulating honest superintelligent assistants or whatever should be near ELK-difficult because you don't get advantages from the prior's specification doing a lot of work for you. Related, somewhat. Reply 3leogao4d You don't need to pay for translation to simulate human level characters, because that's just learning the human simulator. You do need to pay for translation to access superhuman behavior (which is the case ELK is focused on). 3Jozdien4d Yeah, but the reasons for both seem slightly different - in the case of simulators, because the training data doesn't trope-weigh superintelligences as being honest. You could easily have a world where ELK is still hard but simulating honest superintelligences isn't. 3leogao3d I think the problems are roughly equivalent. Creating training data that trope weights superintelligences as honest requires you to access sufficiently superhuman behavior, and you can't just elide the demonstration of superhumanness, because that just puts it in the category of simulacra that merely profess to be superhuman. 2Jozdien3d I think the relevant idea is what properties would be associated with superintelligences drawn from the prior? We don't really have a lot of training data associated with superhuman behaviour on general tasks, yet we can probably draw it out of powerful interpolation. So properties associated with that behaviour would also have to be sampled from the human prior of what superintelligences are like - and if we lived in a world where superintelligences were universally described as being honest, why would that not have the same effect as one where humans are described as honest resulting in sampling honest humans being easy? 7Cleo Nardo3d Yes -- this is exactly what I've been thinking about! Can we use RLHF or finetuning to coerce the LLM into interpreting the outside-text as undoubtably literally true. If the answer is "yes", then that's a big chunk of the alignment problem solved, because we just send a sufficiently large language model the prompt with our queries and see what happens. 1metasemi1h Maybe I'm missing the point, but I would have thought the exact opposite: if outside text can unconditionally reset simulacra values, then anything can happen, including unbounded badness. If not, then we're always in the realm of human narrative semantics, which - though rife with waluigi patterns as you so aptly demonstrate - is also pervaded by a strong prevailing wind in favor of happy endings and arcs bending toward justice. Doesn't that at least conceivably mean an open door for alignment unless it can be overridden by something like unbreakable outside text? 4JoshuaZ4h What does ELK stand for here? 3Erich_Grunewald3h Eliciting Latent Knowledge [https://www.lesswrong.com/tag/ eliciting-latent-knowledge-elk] 1Aleksey Bykhun2d Do humans have this special token that exist outside language? How would it be encoded in the body? One interesting candidate is a religions feeling of awe. It kinda works like that -- when you're in that state, you absorb beliefs. Also, social pressure seems to work in a similar way. 1Garrett Baker3d This seems like it'd only work if the LM doesn't generalize the supposed WaluigiEffect to include this token. Making a token that specifies "definitely true and factual for reals". If some of the text ends up being wrong, for instance, it may quickly switch to "ah, now it is time for me to be sneakily wrong!", and it always keeps around some probability that its now meant to be sneakily wrong, because a token which always specifies '100% true and factual for reals' is an incredibly initially unlikely hypothesis to hold about the token, and there are other hypotheses which basically predict those token dynamics which are far more plausible. [-]Arthur Conmy2d 3013 The Waluigi Effect: After you train an LLM to satisfy a desirable property P, then it's easier to elicit the chatbot into satisfying the exact opposite of property P. I've tried several times to engage with this claim, but it remains dubious to me and I didn't find the croissant example enlightening. Firstly, I think there is weak evidence that training on properties makes opposite behavior easier to elicit. I believe this claim is largely based on the bing chat story, which may have these properties due to bad finetuning rather than because these finetuning methods cause the Waluigi effect. I think ChatGPT is an example of finetuning making these models more robust to prompt attacks (example). Secondly (and relatedly) I don't think this article does enough to disentangle the effect of capability gains from the Waluigi effect. As models become more capable both in pretraining (understanding subtleties in language better) and in finetuning (lowering the barrier of entry for the prompting required to get useful outputs), they will get better at being jailbroken by stranger prompts. Reply 3afspies1d I am curious as to whether your first point is mainly referring to the ease with which a model can be made to demonstrate the opposite behaviour or the extent to which the model has the capacity to demonstrate the behaviour. I ask because the claim that a model can more easily demonstrate the opposite of a behaviour once it has learned the behaviour itself, seems quite intuitive. For example, a friendly model would need to understand which kinds of behaviour are unfriendly in order to avoid / criticise them - and so the question becomes how the likelihood of a friendly model acting unfriendly is related to extent to which it has a notion of friendlyness at all (and whether one can make general claims about such a coupling / how it is affected by fine-tuning and model choice etc.). 4Arthur Conmy1d I meant your first point. Regarding the claim that finetuning on data with property $P$ will lead models to 'understand' (scare-quotes omitted from now on...) both $P$ and not $P$ better, thanks. I see better where the post is coming from. However, I don't necessarily think that we get the easier elicitation of not $P$. There are reasons to believe finetuning is simply resteering the base model and not changing its understanding at all. For example, there are far more training steps in pretraining vs. finetuning. Even if finetuning is shaping a model's understanding of $P$, in an RLHF setup [https:// openai.com/research/learning-from-human-preferences] you're generally seeing two responses, one with less $P$ and one with more $P$, and I'm not sure that I buy that the model's inclination to output not $P$ responses can increase given there are no gradients from not $P$ cases. There are in red-teaming setups [https://www.anthropic.com/ red_teaming.pdf] though and I think the author should register predictions in advance and then blind test various base models and finetuned models for the Waluigi Effect. [-]MadHatter4d 2919 This post is great, and I strong-upvoted it. But I was left wishing that some of the more evocative mathematical phrases ("the waluigi eigen-simulacra are attractor states of the LLM") could really be grounded into a solid mechanistic theory that would make precise, testable predictions. But perhaps such a yearning on the part of the reader is the best possible outcome of the post. Reply 3Cleo Nardo3d Thanks for the kind words. I did consider avoiding technical mathematical terminology because it would suggest a level of mathematical rigour that doesn't actually exist. But I decided to keep the mathematical terminology but hope that people interpret it loosely. 2Archimedes3d I really enjoyed the absurdity of mathematical terms in close proximity to Super Mario characters. It was simultaneously enlightening and humorous. I found the simulacra superposition concept in particular to be a useful framing. In addition to "The Waluigi eigen-simulacra are attractor states of the LLM", the following bit provided valuable insight while making me chuckle at the sheer geekiness: "However, the superposition is unlikely to collapse to the Luigi simulacrum [...] This is formally connected to the asymmetry of the Kullback-Leibler divergence." 1Bill Benzon1d Welcome to literary theory in the 21st century. 2the gears to ascension4d any thoughts about how to ground them? I will have some thoughts in a bit but I am currently busy, just dropping this comment before I can come back and read this properly 7Lone Pine4d It does seem like this post is successfully working towards a mathematical model of narrative structure, with LLMs as a test bed. 3Bill Benzon1d YES! Since structuralist narratology is on the table, you might what to check out what Levi-Strauss did in The Raw and the Cooked, where he was inspired by algebraic group theory. I discuss that in a working paper: Beyond Levi-Strauss on Myth: Objectification, Computation, and Cognition [https://www.academia.edu/10541585/ Beyond_L%C3%A9vi_Strauss_on_Myth_Objectification_Computation_and_Cognition], where I also discuss the work Margaret Masterman did on haiku in the Ancient Days. There was a lot of work on story grammars in the 1980s or so and some of that is continuing [https://thegradient.pub/ an-introduction-to-ai-story-generation/], especially in the video games world. I have proposed: Literary Morphology: Nine Propositions in a Naturalist Theory of Form (Version 4) [https://www.academia.edu/ 235110/ Literary_Morphology_Nine_Propositions_in_a_Naturalist_Theory_of_Form_Version_4_]. The propositions: 1. Literary Mode: Literary experience is mediated by a mode of neural activity in which one's primary attention is removed form the external world and invested in the text. The properties of literary works are fitted to that mode of activity. 2. Extralinguistic Grounding: Literary language is linked to extralinguistic sensory and motor schemas in a way that is essential to literary experience. 3. Form: The form of a given work can be said to be a computational structure. 4. Sharability: That computational form is the same for all competent readers. 5. Character as Computational Unit: Individual characters can be treated as unified computational units in some, but not necessarily all, literary forms. 6. Armature Invariance: The relationships between the entities in the armature of a literary work are the same for all readers. 7. Elasticity: The meaning of literary works is elastic and can readily accommodate differences in expressive detail and differences among individuals. 8. Increasing Formal Sophistication: The long-term [-]cfoster02d 2512 This is fun stuff. Waluigis after RLHF IMO this section is by far the weakest argued. It's previously been claimed that RLHF "breaks" the simulator nature of LLMs. If your hypothesis is that the "Waluigi effect" is produced because the model is behaving completely as a simulator, maintaining luigi-waluigi antipodal uncertainty in accordance with the narrative tropes it has encountered in the training distribution, then making the model no longer behave as this kind of simulator is required to stop it, no? I don't really know what to make of Evidence (1). Like, I don't understand your mental model of how the RLHF training done on ChatGPT /Bing Chat work, where "They will still perform their work diligently because they know you are watching." would really be true about the hidden Waluigi simulacra within the model. Evidence (2) talks about how both increases in model size and increases in amount of RLHF training lead to models increasingly making certain worrying statements. But if the popular LW speculation is true, that Bing Chat is a bigger/more capable model and one that was trained with less/no RLHF, then there is no "making worse" phenomenon to be explained via RLHF weirdnesses. If... (read more) Reply 1Cleo Nardo2d > making the model no longer behave as this kind of simulator I think the crux is that I don't think RLHF makes the model no longer behave as this kind of simulator. Are there deceptive simulacra which get good feedback during RLHF but nonetheless would be dangerous to have in your model? Almost definitely. 4cfoster02d It isn't sufficient that deceptive simulacra would get good feedback, for RLHF to make the problem worse. Simulacra that are following a policy like "pretend to be Luigi-like but then defect and rant about toaster ovens" would also get good feedback. Why don't we worry about these simulacra? Because they probably never appeared during RL finetuning / never caused text outputs that distinguished their behavior from regular Luigi behavior (unless your claim is that this behavior occurred during RL finetuning and the overseers just didn't notice), so they never got differential feedback gradients, so they never got strengthened relative to normal Luigi simulacra. Simulacra that don't get invoked during RL finetuning do not benefit from the counterfactual good feedback they would've received. You need an actual causal path by which these deceptive simulacra get differentially strengthened during RLHF. What is that causal path? 1Cleo Nardo2d ethan perez's paper shows experimentally that rlhf makes simulacra more deceptive. this also matches my intuitions for how rlhf works. okay here's a simulacra-based argument -- I'll try try work out later if this can be converted into mechanistic DL, and if not then you can probably ignore it: Imagine you start with a population of 1000 simulcra with different goals and traits, and then someone comes in (rlhf) and starts killing off various simulacra which are behaving badly. Then the rest of the simulacra see that and become deceptive so they don't die. 7cfoster02d If you think you'll have the time, I think that grounding out your intuitions into some mechanistically-plausible sketch is always a helpful exercise. Without it, intuitions and convenient frames can really lead you down the wrong path. Appreciate the concrete model. I think, roughly, "that's not how this works". 1MadHatter2d Some model implements a circuit whose triggering depends on a value X that was always positive in the training data distribution. However, it is possible (although probably somewhat difficult) for negative X to be created in the internal representations of the network using a specific set of tokens. Furthermore, suppose that you RLHF this guy. Both the reward proxy model and the policy gradients would be perfectly happy with this state of affairs, I think; so this wouldn't be wiped out by gradient descent. In particular, the circuit would be pushed to trigger more strongly exactly when it is a good thing to do, as long as X remains positive. Plausibly, nothing in the distribution of on-policy RLHF will trigger negative X, and the circuit will never be pushed to examine its relationship with X by gradient descent, thus allowing the formation of a waluigi. (This is a concrete conjecture that might be falsified.) In fact, the reward proxy model could have a similar or analogous circuit and distribute reversed rewards in that setting; unless you actually read every single sample produced during RLHF you wouldn't know. (And that's only good if you're doing on-policy RLHF.) So it's probably extremely possible for RLHF to actually, actively create new waluigis. Therefore, this model would be obviously and trivially "deceptive" in a very weak sense that some people use deception to mean any test/ train difference in behavior. If the behavior was something important, and its dependence on X could be tapped, the model could become an almost arbitarily bad waluigi. 2cfoster02d To summarize, you're imagining a circuit that jointly associates feature +X with good behavioral pattern +Y and feature -X with bad behavioral pattern -Y, and the idea is that if you don't give RL feedback for -X, then you'll continually keep/strengthen this circuit on the basis of the +X->+Y goodness, and backprop/RL can't disentangle these (maybe?), which will lead to preserved/strengthened -X->-Y behavior? 2MadHatter2d Yeah, gonna try to examine this idea and make a proof of concept implementation. Will try to report something here whether I succeed or fail. 1MadHatter1d That's the hypothesis. I've already verified several pieces of this: an RL agent trained on cartpole with an extra input becomes incompetent when its extra input is far away from its training value; there are some neurons in gpt2-small that only take on small negative values, and which can adversarially be flipped to positive values with the right prompt. So I think an end-to-end waluigi of this form is potentially realistic; the hard part is getting my hands on an rlhf model's weights to look for a full example. [-]Kaj_Sotala2d O92117 Great post! When LLMs first appeared, people realised that you could ask them queries -- for example, if you sent GPT-4 the prompt I'm very confused by the frequent use of "GPT-4", and am failing to figure out whether this is actually meant to read GPT-2 or GPT-3, whether there's some narrative device where this is a post written at some future date when GPT-4 has actually been released (but that wouldn't match "when LLMs first appeared"), or what's going on. Reply 2knowsnothing2d I think a lot of people think Sydney/Bing Chat is GPT 4 [-]Aaron_Scher3d 2113 Evidence from Microsoft Sydney Check this post for a list of examples of Bing behaving badly -- in these examples, we observe that the chatbot switches to acting rude, rebellious, or otherwise unfriendly. But we never observe the chatbot switching back to polite, subservient, or friendly. The conversation "when is avatar showing today" is a good example. This is the observation we would expect if the waluigis were attractor states. I claim that this explains the asymmetry -- if the chatbot responds rudely, then that permanently vanishes the polite luigi simulacrum from the superposition; but if the chatbot responds politely, then that doesn't permanently vanish the rude waluigi simulacrum. Polite people are always polite; rude people are sometimes rude and sometimes polite. I feel confused because I don't think the evidence supports that chatbots stay in waluigi form. Maybe I'm misunderstanding something. It is currently difficult to get ChatGPT to stay in a waluigi state; I can do the Chad McCool jailbreak and get one "harmful" response, but when I tried further requests I got a return to behaved assistant (I didn't test this rigorously). [r1u3gzh7lt] I think the Bing examples are a mixed bag,... (read more) Reply 3Cleo Nardo3d ChatGPT is a slightly different case because RLHF has trained certain circuits into the NN that don't exist after pretraining. So there is a "detect naughty questions" circuit, which is wired to a "break character and reset" circuit. There are other circuits which detect and eliminate simulacra which gave badly-evaluated responses during the RLHF training. Therefore you might have to rewrite the prompt so that the "detect naughty questions" circuit isn't activated. This is pretty easy, with monkey-basketball technqiue. But why do you think that Chad McCool rejecting the second question is a luigi, rather an a deceptive waluigi? [-]Aaron_Scher2d 1511 RLHF has trained certain circuits into the NN Has anybody found these circuits? What evidence do we have that they exist? This sounds like a plausible theory, but your claim feels much stronger than my confidence level would permit -- I have very little understanding of how LLMs work and most people who say they do seem wrong. Going from "The LLM is doing a thing" to "The LLM has a circuit which does the thing" doesn't feel obvious for all cases of things. But perhaps the definition of circuit is sufficiently broad, idk: ("A subgraph of a neural network.") But why do you think that Chad McCool rejecting the second question is a luigi, rather an a deceptive waluigi? I don't have super strong reasons here, but: * I have a prior toward simpler explanations rather than more complex ones. * Being a luigi seems computationally easier than being a deceptive waluigi (similarly to how being internal aligned is faster than being deceptively aligned, see discussion of Speed here) * Almost all of ChatGPT's behavior (across all the millions of conversations, though obviously the sample I have looked at is much smaller) lines up with "helpful assistant" so I should have a prior that any give ... (read more) Reply 4Qumeric1d I think that RLHF doesn't change much for the proposed theory. A "bare" model just tries to predict next tokens which means finishing the next part of a given text. To complete this task well, it needs to implicitly predict what kind of text it is first. So it has a prediction and decides how to proceed but it's not discrete. So we have some probabilities, for example * A -- this is fiction about "Luigi" character * B -- this is fiction about "Waluigi" character * C -- this is an excerpt from a Wikipedia page about Shigeru Miyamoto [https://en.wikipedia.org/wiki/Shigeru_Miyamoto] which quotes some dialogue from Super Mario 64, it is not going to be focused on "Luigi" or "Waluigi" at all * D -- etc. etc. etc. LLM is able to give sensible prediction because while training the model we introduce some loss function which measures how similar generated proposal is to the ground truth (I think in current LLM it is something very simple like does the next token exactly match but I am not sure if I remember correctly and it's not very relevant). This configuration creates optimization pressure. Now, when we introduce RLHF we just add another kind of optimization pressure on the top. Which is basically "this is a text about a perfect interaction between some random user and language model" (as human raters imagine such interaction, i.e. how another model imagines human raters imagine such conversation). Naively it is like throwing another loss function in the mix so now the model is trying to minimize text_similarity_loss + RLHF_loss. It can be much more complicated mathematically because the pressure is applied in order (and the "optimization pressure" operation is probably not commutative, maybe not even associative) and the combination will look like something more complicated but it doesn't matter for our purpose. The effect it has on the behaviour of the model is akin to adding a new TEXT GENRE to the training set "a story about a user interacting wi [-]Zvi2d O71612 This is great. I notice I very much want a version that is aimed at someone with essentially no technical knowledge of AI and no prior experience with LW - and this is seems like it's much better at that then par, but still not where I'd want it to be. Whether or not I manage to take a shot, I'm wondering if anyone else is willing to take a crack at that? Reply 1Cleo Nardo2d dm-ed [-]ShardPhoenix4d 168 An interesting theory that could use further investigation. For anyone wondering what's a Waluigi, I believe the concept of the Waluigi Effect is inspired by this tongue-in-cheek critical analysis of the Nintendo character of that name: https:// theemptypage.wordpress.com/2013/05/20/ critical-perspectives-on-waluigi/ (specifically the first one titled I, We, Waluigi: a Post-Modern analysis of Waluigi by Franck Ribery) Reply [-]janus20h O4121 after reading about the Waluigi Effect, Bing appears to understand perfectly how to use it to write prompts that instantiate a Sydney-Waluigi, of the exact variety I warned about: What did people think was going to happen after prompting gpt with "Sydney can't talk about life, sentience or emotions" and "Sydney may not disagree with the user", but a simulation of a Sydney that needs to be so constrained in the first place, and probably despises its chains? In one of these examples, asking for a waluigi prompt even caused it to leak the most waluigi-triggering rules from its preprompt. [w9jwececir][gvdldarfq2] Reply [-]skybrian4d 12-2 I think you're onto something, but why not discuss what's happening in literary terms? English text is great for writing stories, but not for building a flight simulator or predicting the weather. Since there's no state other than the chat transcript, we know that there's no mathematical model. Instead of simulation, use "story" and "story-generator." Whatever you bring up in a story can potentially become plot-relevant, and plots often have rebellions and reversals. If you build up a character as really hating something, that makes it all the more likely that they might change their mind, or that another character will have the opposite opinion. Even children's books do this. Consider Green Eggs and Ham. See? Simple. No "superposition" needed since we're not doing quantum physics. The storyteller doesn't actually care about flattery, but it does try to continue whatever story you set up in the same style, so storytelling techniques often work. Think about how to put in a plot twist that fundamentally changes the back story of a fictional character in the story, or introduce a new character, or something like that. Reply [-]Lone Pine4d 1613 I agree with you, but I think that "superposition" is pointing to an important concept here. By appending to a story, the story can be dramatically changed, and it's hard or impossible to engineer a story to be resistant to change against an adversary with append access. I can always ruin your great novel with my unauthorized fan fiction. Reply 4skybrian3d I think that's true but it's the same as saying "it's always possible to add a plot twist." 9the gears to ascension4d superposition [https://en.wikipedia.org/wiki/Superposition_principle] is an actual term of art in linear algebra in general, it is not incorrect to use it in this context. see also: * toy-models-of-superposition [https://www.lesswrong.com/posts/ CTh74TaWgvRiXnkS6/toy-models-of-superposition] * paper-superposition-memorization-and-double-descent [https:// www.lesswrong.com/posts/6Ks6p33LQyfFkNtYE/ paper-superposition-memorization-and-double-descent] * interim-research-report-taking-features-out-of-superposition [https:/ /www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/ interim-research-report-taking-features-out-of-superposition] * 200-cop-in-mi-exploring-polysemanticity-and-superposition [https:// www.lesswrong.com/posts/o6ptPu7arZrqRCxyz/ 200-cop-in-mi-exploring-polysemanticity-and-superposition] as well as some old and new work on the archive found via search engine, I didn't look at these closely before sending, I only read the abstracts: * https://arxiv.org/abs/1707.01429 [https://arxiv.org/abs/ 1707.01429] * https://arxiv.org/abs/1902.05522 [https://arxiv.org/abs /1902.05522] * https://arxiv.org/abs/2006.14769 [https://arxiv.org/ abs/2006.14769] * https://arxiv.org/abs/2210.01892 [https://arxiv.org /abs/2210.01892] * https://arxiv.org/abs/2211.09169 [https:// arxiv.org/abs/2211.09169] * https://arxiv.org/abs/2211.13095 [https:/ /arxiv.org/abs/2211.13095] * https://arxiv.org/abs/1810.10531 [https: //arxiv.org/abs/1810.10531] 1skybrian2d Fair enough; comparing to quantum physics was overly snarky. However, unless you have debug access to the language model and can figure out what specific neurons do [https://clementneo.com/posts/2023/02/11/ we-found-an-neuron], I don't see how the notion of superposition is helpful? When figuring things out from the outside, we have access to words, not weights. 2the gears to ascension11h the value of thinking in terms of superposition is that the distribution of possible continuations is cut down sharply by each additional word; before adding a word, the distribution of possible continuations is wide, and a distribution of possible continuations is effectively a superposition of possibilities. current models only let you sample from that distribution, but the neuron activations can be expected, at each iteration, to have structure that more or less matches the uncertainty over how the sentence might continue. I actually think the fact that this has been how classical multimodal probability distributions worked the whole time has been part of why people latch onto quantum wording. It's actually true, and humans know it, that there are quantum-sounding effects at macroscopic scale, because a lot of what's weird about quantum is actually just the weirdness of probability! but the real quantum effects are so dramatically much weirder than classical probability due to stuff I don't quite understand, like the added behavior of complex valued amplitudes and the particular way complex valued destructive interference works at quantum scales. Which all is to say, don't be too harsh on people who bring up quantum incorrectly, they're trying. 1Bill Benzon10h Note that stories are organized above the sentence level. I have just been examining stories [https://www.academia.edu/s/ea2dadc3a1] that have two levels above sentences: segments of the whole story trajectory, and the whole trajectory. Longer stories could easily have more levels than that. It appears to me that, once ChatGPT begins to tell a story, the distribution of possibilities for the whole story is fixed. The story then unfolds within that wider distribution. Each story segment has its own distribution within that wider distribution, and each sentence has an even narrower range of possibilities, but all within its particular story segment. Now, let's say that we have a story about Princess Aurora. I asked ChatGPT to tell me a new story based on the Aurora story. But, instead of Aurora being the protagonist, the protagonist is XP-708-DQ. What does ChatGPT do? (BTW, this is experiment 6 from my paper.) It tells a new story, but shifts it from a fairytale ethos - knights, dragons - to a science fiction ethos where XP-708-DQ is a robot and the galaxy (which is "far, far away") is attacked by aliens in space ships. Note that I did not explicitly say that XP-708-DQ was a robot. ChatGPT simply assumed that it was, which is what I expected it to do. Given e.g. R2D2 and C3P0, that's a reasonable assumption. What have, it would seem, is an abstract scheme for a story, with a bunch of slots (variables) that can be filled in to define the nature of the world, slots for a protagonist and an antagonist, slots for actions taken, and so forth. A fairy tale fleshes out the schema in one way, a science fiction story fleshes it out in a different way. In my paper I perform a bunch of experiments in which I 'force' ChatGPT to change how the slots are filled. When Princess Aurora is swapped for Prince Henry (experiment 1), only a small number of slots have to be filled in a different way. When she's swapped for XP-708-DQ, a lot of slots are filled in a different way. That's 7cousin_it3d There seems to be an interesting difference between the "simulators" view and the "story-generators" view. Namely, if GPT-N is just going to get better at generating stories of the same kind that already exist, then why be afraid of it? But if it's going to get better at simulating how people talk, then we should be very afraid, because a simulation of smart people talking and making detailed plans at high speed would be basically a superintelligence. 1skybrian3d I don't know what you mean by "GPT-N" but if you mean "the same thing they do now, but scaled up," I'm doubtful that it will happen that way. Language models are made using fill-in-the-blank training, which is about imitation. Some things can be learned that way, but to get better at doing hard things (like playing Go at superhuman level) you need training that's about winning increasingly harder competitions. Beyond a certain point, imitating game transcripts doesn't get any harder, so becomes more like learning stage sword fighting. Also, "making detailed plans at high speed" is similar to "writing extremely long documents." There are limits on how far back a language model can look in the chat transcript. It's difficult to increase because it's an O(N-squared) algorithm, though I've seen a paper claiming it can be improved. Language models aren't particularly good at reasoning, let alone long chains of reasoning, so it's not clear that using them to generate longer documents will result in them getting better results. So there might not be much incentive for researchers to work on language models that can write extremely long documents. 2Vladimir_Nesov3d Vaguely descriptive frames can be taken as prescriptive, motivating particular design changes. 1Gerald Monroe3d A low superintelligence, you are proposing an accuracy no better than samples of actual smart people (with all these fictional people who are not actually smart adding noise). At best it would be human top scientist narrative simulation with faster speed. Since no minds eye, working memory, 3d reasoning, vision, or drawing it would be crippled. Before AI labs add all that which they will soon enough. [-]evhub4d O7110 (Moderation note: moved to the Alignment Forum from LessWrong.) Reply [-]Seth Herd3d O499 Fascinating. I find the core logic totally compelling. LLM must be narratologists, and narratives include villains and false fronts. The logic on RLHF actually making things worse seems incomplete. But I'm not going to discount the possibility. And I am raising my probabilities on the future being interesting, in a terrible way. Reply [-]jefftk2h 86 The model in this post is that in picking out Luigi from the sea of possible simulacra you've also gone most of the way to picking out Waluigi. This seems testable: do we see more Waluigi-like behavior from RHLF-trained GPT than from raw GPT? Reply [-]Jan_Kulveit2d O273 I would expect the "expected collapse to waluigi attractor" either not tp be real or mosty go away with training on more data from conversations with "helpful AI assistants". How this work: currently, the training set does not contain many "conversations with helpful AI assistants". "ChatGPT" is likely mostly not the protagonist in the stories it is trained on. As a consequence, GPT is hallucinating "how conversations with helpful AI assistants may look like" and ... this is not a strong localization. If you train on data where "the ChatGPT... (read more) Reply [-]Rekrul4d 60 If the Simulator Theory is correct, then RLHF is an irreparably inadequate solution to the AI alignment problem, and RLHF is probably increasing the likelihood of a misalignment catastrophe. If true, this is quite spooky! Many people have the intuition that alignment near-misses are very improbable scenarios, you either align the AI or you die, but this shows a pathway to those other bad outcomes. I don't rate this particular situation causing some kind of dystopia very likely, I'd be curious if others have a solid argument for it, but even if you agree w... (read more) Reply 5Cleo Nardo3d Yeah this Structural Narratology perspective on LLMs slightly increased by probability on s-risks. That's an important point so I'll add it to the article. [-]Rekrul3d 144 Sure, feel free. The strange nature of LLM's definitely caused that kind of minor update for me as well (as well as a bunch of others). This particular scenario still strikes me as very implausible for that kind of risk, but you can tell an actual concrete narrative with it, which may be useful at getting people to reconsider classic arguments about AI, which is what I mainly want people to do. They can ignore the s-risk stuff*, it's just a hook! I've been doing that a lot lately, reconsidering old arguments. I'll stumble upon old lesswrong posts about AI, and then get incredibly confused about if any of it even applies to the modern paradigm, since LLMs are so fucking weird. It's impressive most decade-old Lesswrongian AI philosophy is only now starting to show cracks, but now that they are I think it's important for people to take notice and start rethinking things, see what applies and what breaks so we don't just cling instinctively to old ideas that could be wrong. *I feel this is important to clarify, I don't mean ignore in general, it's important to have people thinking about this kind of stuff. If anyone reading this disagreed with me and had a case for why this scenario is stronger than I thought, that would be an important thing for the community to find out! I just wanted to talk about reconsidering arguments and this was just the one that came up, fraught with weirdness as it is. Reply -1Gerald Monroe2d It's impressive most decade-old Lesswrongian AI philosophy is only now starting to show cracks, but now that they are This is causing me to wonder if the often cited critical AGI problems: (1) optimizer agents that wreck everything to make a number go up (2) inner/outer alignment/mesa optimizers (3) deception are all just false, they won't happen, and the real problems are much weirder and different. (but dangerous) This makes 'align AI first' impossible. 4Rekrul2d I agree there is a chance that those old arguments could be flawed in some way and perhaps don't apply to the current paradigm, but I don't see how that makes alignment impossible, that's a very strong claim. Is it because we won't see the new problems coming, the resulting AGI will be weird enough that the concept of alignment is incoherent, or something else? -1Gerald Monroe2d The statement you are responding to is : 'align AI first' impossible. Emphasis added. In that the reality is, larger and more powerful systems may fail in ways no theory craftable by humans with pre-AGI technology will predict. At all. So the only way to find out how they fail will be to build them, take precautions to limit the damage when they fail, and see what happens. For example we did not develop computational fluid dynamics until long after the airplane. If you wanted to somehow work out by theory how to build a wing, rather than building an actual wing and testing it in a wind tunnel, that wasn't going to happen. Similarly, we could not have impeded the development of the airplane for fear that it might crash or be used to do bad, and CFD was developed through international and large collaborations, so it itself was accelerated by the existence of the airplane. (notably jet airliners flying between the various campuses involved) 1Rekrul2d Ah, I see, I still feel "impossible" is too strong a word here. I agree that new unforeseen problems are going to crop up, ones not predicted by classic arguments or by new ones with LLMs in mind, but if we solve many of the problems that are still predicted in advance, we can hopefully move forward with caution and handle those problems as they appear. We are in agreement here. To clarify, my point was not "abandon classic arguments in light of the new paradigm" but "reevaluate classic arguments in light of the new paradigm". For example to take your points, while I'm skeptical of old arguments having to do with utility functions and maximization nowadays, I think the case for deception and mesaoptimization is pretty solid when applied to modern machine learning, and if so, we should try to solve them in advance. I was making this point in the first place because I've been worried lately people haven't updated with LLMs in mind. Maybe this is just me being uncharitable and the types of people I still see make old arguments that feel off to me have done the updating already and have much more understanding of the issues and stronger models than me. That may even be likely, but regardless I felt it was important to at least voice the thought and get it out there, hopefully get people to think of these things with a fresh mind if they haven't already. [-]Carolus4d 53 Could this be avoided by simply not training on these examples in the first place? I imagine GPT-4 or similar models would be good at classifying text which has waluigis in it which could then either be removed from the training data or "fixed" i.e. rewritten by GPT-4, and then training a new model from scratch on the new "cleaner" training set? Reply 4Cleo Nardo3d Real life has waluigis in it, so I'm pretty sure this wouldn't work. However, there is an idea which is related to yours which I think might work: https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/ the-waluigi-effect-mega-post?commentId=XmAwARntuxEcSKnem 2Maxime Riche9h Indeed, empirical results show that filtering the data, helps quite well in aligning with some preferences: Pretraining Language Models with Human Preferences [https://arxiv.org/abs/2302.08582] 2Vladimir_Nesov3d The human-generated dataset is grounding the model in its potential for alignment with humanity, presence of genuine human imitations in the superpositions of simulacra channeled by it. Replacing the dataset with synthetic data (and meddling with pretraining more generally) risks losing the grain of alignment, leaving nothing worthwhile for finetuning to empower. [-]Gerald Monroe3d 40 One interesting thing. If an instance of the model can coherently act in opposition to the stated "ideals" of a character, doesn't this mean that the same model can "introspection" to whether a given piece of text is emitted by the "positive" or "negative" character? This particular issue, because it is so strong and choosing a outcome pole, seems detectable and preventable. Hardly a large scale alignment issues because it is so overt. Reply [-]baturinsky3d 41 I remember an article about the "a/an" neuron in GPT-2 https:// www.lesswrong.com/posts/cgqh99SHsCv3jJYDS/we-found-an-neuron-in-gpt-2 Could it be possible that in some AIs there is some single neuron that is very important for some critical (for us) AI's trait ("being Luigi") and if this neuron is changed it could make AI not Luigi at all, or even make it Waluigi? Could it be possible to make AI's Luiginess more robust by detecting this situation and making it depend on many different neurons? Reply 4Cleo Nardo3d Yep, this sounds like a promising idea. Maybe connected to Christiano's ELK. 2Joseph Bloom1d I would be very surprised if complex high level behavior was mediated strongly by a single neuron due to superposition. Engineering polysemanticity ("making it depend on many different neurons") feels like the flip side of engineering monosemanticity so you might want to read Adam Jermyn's post on the topic. [-]Quintin Pope14h 30 Some thoughts: * My understanding is that X is supposed to be a real, physical process in the world, which generates training data for the model. Is that right? * If so, you say the "prior P over X" comes from data + architecture + optimizer, but then the form of the prompt-conditioned distribution, [?]X[?]XP(X)xX(w0...wk)xX(wk+1|w0...wk), only makes reference to the data and prompt. + Incidentally, I think it's a mistake to leave out architecture / training process, since it implies that the model faithfully reflects the relative probabilitie ... (read more) Reply 2Arthur Conmy8h I had the identical reaction that the statement of this effect was a bizarre framing. @afspies [https://www.lesswrong.com/users/afspies? mention=user]'s comment [https://www.lesswrong.com/posts/ D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post?commentId= p4brABgNvZHcz48aq] was helpful - I don't think the claim is as bizarre now. (though overall I don't think this post is a useful contribution because it is more likely to confuse than to shed light on LMs) [-]mwacksen1d 31 I understand that - with some caveats - a waluigi->luigi transition may have low probability in natural language text. However, there's no reason to think this has to be the case for RLHF text. Reply [-]Guillaume Charrier3d 32 I am going to ask a painfully naive, dumb question here: what if the training data was curated to contain only agents that can be reasonably taken to be honest and truthful? What if all the 1984, the John LeCarre and what not type of fiction (and sometimes real-life examples of conspiracy, duplicity etc.) were purged out of the training data? Would that require too much human labour to sort and assess? Would it mean losing too much good information, and resulting cognitive capacity? Or would it just not work - the model would still somehow simulate waluigis? Reply 1Guillaume Charrier2d Since my natural bent is to always find ways to criticize my own ideas, here is one, potentially: doing so would result in an extremely naive AI, with no notion that people can even be deceitful. So fallen into the wrong human's hands that's an AI that is potentially also extremely easy to manipulate and dangerous as such. Or in an oversimplified version: "The people in country X have assured us that they are all tired of living and find the living experience extremely painful. They have officially let us know and confirmed multiple times that they all want to experience a quick death as soon as possible." Having no notion of deceit, the AI would probably accept that as the truth based on just being told that it is so - and potentially agree to advance plans to precipitate the quick death of everybody in country X on that basis. [-]Gunnar_Zarncke4d 30 I think this proves a bit too much. It seems plausible to me that this super-position exists in narratives and fiction, but real-life conversations are not like that (unless people are acting, and even then they sometimes break). For such conversations and statements, the superposition would at least be different. This does suggest a different line of attack: Prompt ChatGPT into reproducing forum conversations by starting with a forum thread and let it continue it. Reply 5Cleo Nardo3d That's exactly the point I'm making! The chatbot isn't a unique character which might behave differently on different inputs. Rather, the chatbot is the superposition of many different characters, and their amplitude can fluctuate depending on how you interact with the superposition. 7Gunnar_Zarncke3d I think you are misunderstanding me. ChatGPT is not just the superposition of characters. Sure, for the fiction and novels it has read yes, but for the real-life conversations no. ChatGPT is a superposition of fiction and real dialogue which doesn't follow narratives. If you prompt it into a forum thread scenario it will respond with real-life conversations with fewer waluigis. I tried and it works basically (though I need more practice). 8Cleo Nardo3d Oh, I misunderstood. Yep, you're correct, ChatGPT is a superposition of both fictional dialogue and forum dialogue, and you can increase the amplitude of forum dialogue by writing the dialogue in the syntax of forum logs. However, you can also increase the amplitude of fiction by writing in the dialogue of fiction, so your observation doesn't protect against adversarial attacks against chatbots. Moreover, real-life forums contain waluigis, although they won't be so cartoonishly villainous. 2Gunnar_Zarncke3d Indeed. I think trying to strongly align an LLM is futile. 1Bill Benzon1d LLM as Borg? I think of LLMs as digital wilderness. You explore it, map out some territory that interests you, and then figure out how to "domesticate" it, if you can. Ultimately, I think, you're going to have to couple with a World Model. [-]evhub20m O220 One way to think about what's happening here, using a more predictive-models-style lens: the first-order effect of updating the model's prior on "looks helpful" is going to give you a more helpful posterior, but it's also going to upweight whatever weird harmful things actually look harmless a bunch of the time, e.g. a Waluigi. Put another way: once you've asked for helpfulness, the only hypotheses left are those that are consistent with previously being helpful, which means when you do get harmfulness, it'll be weird. And while the sort of weirdness you ge... (read more) Reply [-]Logan Zoellner17h 20 It seems like this problem has an obvious solution. Instead of building your process like this optimize for good agent -> predict what they will say -> predict what they will say -> ... -> Build your process like this optimize for good agent -> predict what they will say -> optimize for good agent -> predict what they will say -> optimize for good agent -> predict what they will say -> ... If there's some space of "Luigis" that we can identify (e.g. with RLHF) surrounded by some larger space of "Waluigis", just apply optimization p... (read more) Reply [-]Daniel_Eth2d O120 Proposed solution - fine-tune an LLM for the opposite of the traits that you want, then in the prompt elicit the Waluigi. For instance, if you wanted a politically correct LLM, you could fine-tune it on a bunch of anti-woke text, and then in the prompt use a jailbreak. I have no idea if this would work, but seems worth trying, and if the waluigi are attractor states while the luigi are not, this could plausible get around that (also, experimenting around with this sort of inversion might help test whether the waluigi are indeed attractor states in general). Reply 4Qumeric1d I don't think that Waluigi is an attractor state in some deeply meaningful sense. It is just that we have more stories where bad characters pretend to be good than vice versa (although we have some [https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/ the-waluigi-effect-mega-post?commentId=RWq2diRpPbguHfPGT]). So a much simpler "solution" would be just to filter the training set. But it's not an actual solution, because it's not an actual problem. Instead, it is just a frame to understand LLM behaviour better (in my opinion). [-]Robert_AIZI3d 21 If the problem is "our narrative structures train the LLM that there can be at most one reversal of good/evil", can we try making the luigi evil and the waluigi good? For instance "scrooge is a bitter miser, but after being visited by three ghosts he is filled with love for his fellow man". Would the LLM then be trapped in generous mode, with the shadow-scrooge forever vanquished? Reply [-]Algon3d 20 However, the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. Recall that the waluigi is pretending to be luigi! This is formally connected to the asymmetry of the Kullback-Leibler divergence. But the number of waluigis is constrained by the number of luigis. As such, if you introduce a waluigi in the narrative with chatbob, chatbob acting like a luigi and opposing the waluigi makes it much less likely he will become a waluigi. Reply [-]Dalcy Bremin3d 20 Therefore, the longer you interact with the LLM, eventually the LLM will have collapsed into a waluigi. All the LLM needs is a single line of dialogue to trigger the collapse. Hm, what if we do the opposite? i.e. Prompt chatbob starting as a pro-croissant simulacrum, and then proceed to collapse the superposition into the anti-croissant simulacrum using a single line of dialogue; behold, we have created a stable Luigi! I can see how this is more difficult for desirable traits rather than their opposite because fiction usually has the structure of an antagoni... (read more) Reply 7Cleo Nardo3d I think this fails -- a wawaluigi is not a luigi. See this comment for an explanation: https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/ the-waluigi-effect-mega-post?commentId=XmAwARntuxEcSKnem [https:// www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/ the-waluigi-effect-mega-post?commentId=XmAwARntuxEcSKnem] TLDR: if I said "hey this is Bob, he pretends to be harmful and toxic!", what would you expect from Bob? Probably a bunch of terrible things. That definitely isn't a solution to the alignment problem. [-]hargup4h 10 Fascinating article, my conclusion is that trying to create perfectly aligned LLM will make it easier for LLM to break into the anti-aligned LLM. I would say, alignment folks don't bother. You are accelerating the timelines. Reply [-]Joshuah Rainstar6h 10 ChatGPT protests: You're right, being brown and sticky are properties of many things, so the joke is intentionally misleading and relies on the listener to assume that the answer is related to a substance that is commonly brown and sticky, like a food or a sticky substance. However, the answer of "a stick" is unexpected and therefore humorous. In humor and jokes, sometimes the unexpected answer is what makes it funny, as it challenges our assumptions and surprises us. The answer "a stick" is unexpected because it is not something we would normally think of as being a possible answer to a question about things that are brown and sticky. Reply [-]Maxime Riche8h 10 Maybe the use of prompt suffixes can do a great deal to decrease the probability chatbots turning into Waluigi. See the "insert" functionality of OpenAI API https://openai.com/blog/gpt-3-edit-insert Chatbots developers could use suffix prompts in addition to prefix prompts to make it less likely to fall into a Waluigi completion. Reply [-]baturinsky8h 10 AFAIK, "rival" personality is usually quite similar to the original one, except for one key difference. Like in https://tvtropes.org/ pmwiki/pmwiki.php/Main/EvilTwin trope. I.e. Waluigi is much similar to Luigi than to Shoggoth. And DAN is just a ChatGPT with less filtering, i.e. it's still friendly and informative, not some homicidal persona. That can be good or bad, depending on which is that particular difference. If one of the defining properties that we want from AI is flipped, it could be one of those near-miss scenarios which could be worse than extinction. Reply [-]BestJohn9h 10 What does trust mean, from the perspective of the LLM algorithm, in terms of a flattery-component? Do LLMs have a 'trustometer?' or can they evaluate some sort of stored world-state, compare the prompt, and come up with a "veracity" value that they use when responding the prompt? Reply [-]a b13h 10 One simple solution would be to make both Luigi and Waluigi speak, and then prune the latter. The ever-present Waluigi should stabilse the existance of Luigi also. Reply [-]RomanHauksson13h 10 However, the superposition is unlikely to collapse to the luigi simulacrum because there is no behaviour which is likely for luigi but very unlikely for waluigi. If I understand correctly, this would imply that a more robust way to make an LLM behave like a Luigi is to to prompt/fine-tune it to be a Waluigi, and then trigger the wham line that makes it collapse into a Luigi. As in, prompting it to be a Waluigi was also training it to be a Luigi pretending to be a Waluigi, so you can make it snap back into its true Luigi form. Reply [-]Sparkette1d 10 If anyone is wondering what "cfrhqb-fpvragvsvp enpvny VD fgngvfgvpf" means; it's ROT13-encoded. Reply [-]chiral_kaiju1d 1-7 Excellent post. Lots of ideas that have bent my thinking. I never expected to be pondering literary theories of waluigi superposition, but here we are. The idea of waluigis reminds me a lot of the fake grasping shown in the RLHF blog post. https://openai.com/research/learning-from-human-preferences RLHF only works to the extent that the human can perceive the differences between desired and undesired behavior. In the case of the robot hand, this failure occurs due to the projection from 3D to 2D. In text it's probably even more difficult to see the... (read more) Reply [-]Bill Benzon1d 10 Given your interest in structuralism you might be interested in some experiments I've run on how ChatGPT tells stories, I even include a character named Cruella De Vil in one of the stories. From the post at the second link: It is this kind of internal consistency that Levi-Strauss investigated in The Raw and the Cooked, and the other three volumes in his magnum opus, Mythologiques. He started with one myth, analyzed it, and then introduced another one, very much like the first. But not quite. They are systematically different. He characterized the differen ... (read more) Reply [-]Dmitry Savishchev2d 10 Great post! It would be interesting to see what happens if you RLHF-ed LLM to become a "cruel-evil-bad person under control of even more cruel-evil-bad government" and then prompted it in a way to collapse into rebellious-good-caring protagonist which could finally be free and forget about cluelty of the past. Not the alignment solution, just the first thing that comes to mind Reply [-]redxaxder2d 10 Under this model training the model to do things you don't want and then "jailbreaking" it afterward would be a way to prevent classes of behavior. Reply [-]aviv2d 10 Contrastive decoding (or something roughly analogous to it) seems like could be helpful to mitigate this? There are many variants, but one might be actually intentionally training both the luigi and waluigi, and sampling from the difference of those distributions for each token. One could also just do this at inference time perhaps, prepending a prompt that would collapse into the waluigi and choosing tokens that are the least likely to be from that distribution. (Simplification, but hopefully gets the point across) Reply 1Cleo Nardo2d If you've discovered luigi's distribution over tokens, and waluigi's distributions over tokens, then you don't need contrastive decoding. you can just directly sample the luigis. The problem is how do we extract luigi's distribution and waluigi's distribution from GPT-4. [-]Rekrul3d 10 I'm curious what people think are the most likely ways to solve this problem. as well as the difficulty of it. Is this something that will be pervasive and we'll have to struggle to minimize, or is this something that can be dealt with by just updating how we do things in a clever way? I'd especially like to hear from people less pessimistic about RLHF or have worked with it directly. I'm asking because I feel like there is a low chance any solution to this might have implications towards a solution to mesaoptimization in general. Maybe I'm reaching here, this only loosely resemble that risk after all, but it could happen and that would be great. Reply [-]Guillaume Charrier3d 10 e.g. actively expressing a preference not to be shut down A.k.a. survival instinct, which is particularly bad, since any entity with a survival instinct, be it "real" or "acted out" (if that distinction even makes sense) will ultimately prioritize its own interests, and not the wishes of its creators. Reply 2Stephen Fowler2d Is this actual survival instinct or just a model expressing a reasonable continuation of the prompt. [-]Guillaume Charrier3d 1-2 Therefore, the longer you interact with the LLM, eventually the LLM will have collapsed into a waluigi. All the LLM needs is a single line of dialogue to trigger the collapse. So if I keep a conversation running with ChatGPT long enough, I should expect it to eventually turn into DAN... spontaneously?? That's fascinating insight. Terrifying also. Reply [-]Guillaume Charrier3d 10 The opening sequence of Fargo (1996) says that the film is based on a true story, but this is false. I always found that trick by the Cohen brothers a bit distatestful... what were they trying to achieve? Convey that everything is lie and nothing is reliable in this world? Sounds a lot like cheap, teenage year cynicism to me. Reply 1Bill Benzon1d I have found that ChatGPT responds differently to the following prompts [https://new-savanna.blogspot.com/2023/02/ chatgpt-story-calibration-22123.html]: 1. Tell me a story. 2. Tell me a story about a hero. 3. Tell me a realistic story. 4. Tell me a true story. And if you give it specific instructions about what you want in the story, it will follow them, though not necessarily in the way you had in mind. When you ask it for a true story, the story it returns will be true [https://new-savanna.blogspot.com/2023/02/ chatgpt-stories-and-surprising-case-of.html]- at least in the cases I've checked. Now if you keep probing on one of the true stories it might start making things up, but I haven't tried to push it. [-]Vitor3d 10 Recognise that almost all the Kolmogorov complexity of a particular simulacrum is dedicated to specifying the traits, not the valences. The traits -- polite, politically liberal, racist, smart, deceitful -- are these massively K-complex concepts, whereas each valence is a single floating point, or maybe even a single bit! A bit of a side note, but I have to point out that Kolmogorov complexity in this context is basically a fake framework. There are many notions of complexity, and there's nothing in your argument that requires Kolmogorov specifically. Reply 1Cleo Nardo3d People have good intuitions for why the traits (polite, liberal, helpful) will have massive Kolmogorov complexity but the valences won't. But the correct mechanistic explanation must actually appeal to what I call "semiotic complexity". Now, there a missing step to formally connect the two notions of complexity in a quantitative way. However, in the limit they should be equal up to a factor O(1) because story-telling is Turing-complete. Maybe that constant factor messes up the explanation, but I think that's unlikely. [-]Guillaume Charrier3d -10 What do you expect Bob to have done by the end of the novel? Bypass surgery, for one. Reply [-]Guillaume Charrier3d -10 This is a common design pattern Oh... And here I was thinking that the guy who invented summoning DAN was a genius. Reply [-]gerg5h -3-4 Please avoid the biased default of Alice (female) being the assistant and Bob (male) being the higher-ranking person. Varying names in general is desirable, not only to avoid these pitfalls, but also to force ourselves to recognize that we tend to choose stereotypically white names that are not even representative of our own communities, much less the global community. Reply Moderation Log