[HN Gopher] Refusal in language models is mediated by a single d...
___________________________________________________________________
Refusal in language models is mediated by a single direction
Author : Tomte
Score : 191 points
Date : 2024-06-18 17:09 UTC (1 days ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| wavemode wrote:
| Related recent HN submission (Uncensor any LLM with
| abliteration): https://news.ycombinator.com/item?id=40665721
| schoen wrote:
| I don't know the exact connection between the two, but that
| article cites an article which is described as a preview of
| this paper. So I guess it was working with a summary of this
| paper's contributions.
| stainablesteel wrote:
| if im understanding everything correctly the ablitation
| concept scouts the model for a similar concept to the
| "direction" described in this one, and it blocks it in order
| to "uncensor" the llm
| Natsu wrote:
| I'm surprised no one has yet labeled this direction as the
| "axis of evil."
| eigenvalue wrote:
| Now that this technique is known, I wonder if there will be an
| arm's race to try to "distribute" the refusal tendency across as
| many different directions in the embedding space as possible so
| that it can't be easily offset without reducing the quality of
| the inferences so much that it's not worth it.
| refulgentis wrote:
| IMHO, probably not, reasoning:
|
| It's only worthwhile if you're distributing weights.
|
| If you're distributing weights, people can train right over
| them anyway (ex. the waifu of the hour is based on
| Mistral...called...Moistral... _shudders_ )
|
| Abliteration hasn't get significant traction in the handing-
| out-weights space. That surprised me because of the amount of
| avowed not-waifu desire for "uncensored" models.
|
| It didn't surprise me in that, the volume and ferocity of takes
| on "lobotomizing" did not match my experience with base LLMs at
| BigCo. There's not a ton of difference between a base LLM and
| the "censored" ones.
|
| Trying the abliterated ones makes that embarrassingly clear.
| You're better off tuning on erotic fanfic for your waifu than
| using an abliterated one, truth is, there's nothing hidden.
| behnamoh wrote:
| > There's not a ton of difference between a base LLM and the
| "censored" ones.
|
| Actually, there is a lot of difference between base and
| censored models in terms of creative capacities. See the
| shocking results of this paper for instance:
| https://arxiv.org/abs/2406.05587
|
| The censorship literally obliterates the creativity of LLMs.
| refulgentis wrote:
| "RLHF significantly reduces to eliminates long tail log
| probs"
|
| shares no words with "Shocking result: model creativity is
| obliterated by censorship"
|
| Honestly, swear to God, the words aren't even in the same
| ballpark as what's actually going on. RLHF is also the
| process that makes it something you can talk to instead of
| an autocompleter. Has nothing to do with the concept of
| censorship. We can tell by the abliteration models. I can
| tell because I've used massive base models.
|
| I also think it's fairly well demonstrated and accessible
| to train an LLM now, enough that if "creativity was
| obliterated by censorship", someone would have made an
| uncensored one that demonstrated superior outputs. Wasn't
| that Grok's whole thing? It'll even tell you how to make
| meth / cocaine? And it's nowhere near leaderboards.
| behnamoh wrote:
| Did you even read the paper? It literally says how RLHF
| significantly reduces model creativity through three
| experiments.
|
| > I also think it's fairly well demonstrated and
| accessible to train an LLM now, enough that if
| "creativity was obliterated by censorship", someone would
| have made an uncensored one that demonstrated superior
| outputs.
|
| No need to to that when we have base models of llama,
| Mistral, etc.
|
| > RLHF is also the process that makes it something you
| can talk to instead of an autocompleter.
|
| Not really. It aligns the model. What you're talking
| about is the SFT process done before RLHF where you
| finetune the model to behave like a conversational AI.
| refulgentis wrote:
| > Did you even read the paper?
|
| ?! Are we in middle school? If so, I'm rubber you're
| glue, whatever you say... (to wit, I quoted the paper to
| you to demonstrate it wasn't as the claim, i.e. "exhibit
| lower entropy in token predictions" != "creativity is
| obliterated due to censorship")
|
| > It literally says how RLHF significantly reduces model
| creativity through three experiments.
|
| Ah, I see now. :) I don't take it personally. I'm old
| enough to smile at aggro behavior kicking up sand in
| front of a step back to the bailey.
|
| > No need to to that when we have base models of llama,
| Mistral, etc.
|
| They're RLHF'd/censorship'd too. "Base model" is a
| colloquialism that used to mean "no RLHF, just straight
| sipping from scraped web pages." Now it means "the last
| round wasn't explicitly chat". I am using it in the
| "sipping from straight scraped web pages" sense.
|
| > Not really
|
| Yes, really. Btw, what does "It aligns the model" mean to
| you at this point in your post? RLHF was just censorship
| that obliterates creativity?
|
| > "[intentionally left blank]"
|
| There is 0 discussion of any of the practical effects I
| mentioned as rope for you to walk down from your strong
| claim, ex. abliteration, uncensored models, etc.
|
| > (not actually in your post at all!)
|
| Is it possible your account got hacked? There's someone
| else using it to post that no one should even release
| models anymore because they're all the same and use the
| same techniques.[1][2] That's hard to square with someone
| who thinks they're all having their creativity
| obliterated due to censorship.
|
| [1] https://news.ycombinator.com/item?id=40599838 [2]
| https://news.ycombinator.com/item?id=40600136
| osmarks wrote:
| Mistral and Meta release "instruct" (RLHF) and not-
| instruct models. The non-instruct ones are in fact non-
| RLHF, pretraining-only ones (though they probably have
| ChatGPT-ish text in the dataset nowadays, and Meta might
| have done some extra training on evals...).
| thelastparadise wrote:
| Moistral?
|
| There's no way that can be real...
|
| Edit: what is a waifu???
| refulgentis wrote:
| My sweet summer child, I weep to burden you with this, turn
| back:
|
| I bet if you looked up waifu's definition it'd have a
| vaguer meaning. In local LLM context, there's a sizable
| community for "virtual girlfriend AI", once you start
| hearing things like "SillyTavern" you're over in the
| community. Think applications designed around local LLMs
| and the use case of having pre-canned prompts to "boot up"
| a girlfriend persona.
|
| For what it's worth, I'm being glib, so it may seem I'm
| linking it to erotica for giggles. c.f. graphics used on
| official GitHub,
| https://github.com/SillyTavern/SillyTavern, and language at
| https://sillytavernai.com/ like [1] and [2]
|
| [1] "We recommend using our sister site:
| https://aicharactercards.com. It is a moderated character
| card repo. All cards go through a moderation process to
| make sure there are no overly inappropriate, illegal or
| scam like cards. NSFW cards are allowed so long as all
| characters are above the age of majority."
|
| [2] Easy to use prompt fields such as main prompts, NSFW
| prompts and Jailbreak prompts that let you steer the chat
| in any way you desire
| QuesnayJr wrote:
| "Waifu" is "wife" if written in katakana. It is jokingly
| used as a name for the fictional female characters in manga
| and anime that are designed to appeal to male viewers. Some
| people wish the fiction wasn't so fictional.
| pjc50 wrote:
| No way it can _not_ be real. Any technology unleashed on
| the wider internet is going to be used for this, and
| companies spend a lot of effort on "brand safety" to try
| not to get pulled into that too much. Why do you think the
| models are censored in the first place?
|
| One of the early uses for NN-style AI image content aware
| fill was a tool called "waifu2x", for upscaling anime from
| VHS.
| refulgentis wrote:
| They aren't censored, or we wouldn't have the waifu
| model. Can't stress this enough. You've found the
| absurdity of this in group shibboleth: they're both
| "censored" and can't do waifus, but they can do waifus,
| so "censored" just means "if some AI companies won't
| offer pornbots upon first naive prompt, I'm in 1984"
| Manabu-eo wrote:
| >> No way it can not be real. Any technology unleashed on
| the wider internet is going to be used for this,
|
| obligatory xkcd: https://xkcd.com/1289/
| nottorp wrote:
| Do look up Rule 34 of the Internet...
| zozbot234 wrote:
| > Trying the abliterated ones makes that embarrassingly
| clear. You're better off tuning on erotic fanfic for your
| waifu than using an abliterated one
|
| These are two very different things. Ablation gets used to
| remove the LLM's behavior of refusing to answer, but
| obviously it does not otherwise affect the LLM's replies,
| much less increase the LLM's knowledge or suitability to
| "forbidden" topics since that will depend on what it was
| trained on, and forbidden topics tend not to be heavily
| featured in the training process. Instead, the models tend to
| confabulate even more than usual as if clumsily trying to
| fill the gaps in their training. If anything, ablation will
| more easily let us test "what an LLM would say if it was
| jailbroken", which will likely help mitigate the oft-
| expressed concern that a "jailbroken" model might say
| something dangerous. (Of course a random confabulation about
| the wrong topic can also be quite dangerous, but
| confabulations in general are a really hard problem to
| address.)
| pizza wrote:
| You'd have to make refusal a high rank subspace, and that seems
| like it could be quite difficult. One alternative approach I've
| seen is to make refusal behavior more likely to just output the
| end-of-speech token.
| eigenvalue wrote:
| There are ways to do that from linear algebra, like
| orthogonalization processes (e.g., Gram-Schmidt process)
| followed by basis expansion. Or random projections can also
| be used in a similar way. And I'm sure there are much fancier
| techniques that draw upon higher math (like Grassmannians or
| Teichmuller mappings).
| ziofill wrote:
| Would you be able to share some links to these techniques?
| They sound related to something I'm working on but in an
| entirely different field.
| mistercow wrote:
| I wonder if you could do this with multiple alignment
| training passes, where you extract the refusal direction each
| time, and suppress it in future training passes.
| jelly wrote:
| Perhaps LLM creators will start using ablation as the
| censorship method instead of a refusal
| daveguy wrote:
| Oh, that sounds double plus good.
| morningsam wrote:
| Same thing in LessWrong post form from back in April:
| https://news.ycombinator.com/item?id=40242939
| akasakahakada wrote:
| LLM search engine coupled with so called "saftefy", will this
| lead us to somewhere as dystopian as described by literature?
|
| Like,
|
| Me: Hey library, tell me how insects make love.
|
| Library: Sorry I can't answer that. Knowledge of insects'
| intercourse can be extrapolated into human's. To protect human
| from AIDS, I cannot tell you that.
| thaumasiotes wrote:
| > Knowledge of insects' intercourse can be extrapolated into
| human's.
|
| That might be difficult. Insects are a wide field.
|
| For example, female bedbugs have no genitalia. Instead, the
| male's penis pierces the female's exoskeleton wherever happens
| to be convenient, in a procedure known formally as "traumatic
| insemination".
| jwilk wrote:
| "Sorry, I can't tell you how bedbugs make love, so that you
| don't try the same thing with a fellow human."
| amarant wrote:
| I'm starting to agree with the AI on this one... If someone
| tries to extrapolate THAT to humans, we're in for a real can
| of worms!
| shermantanktop wrote:
| Maybe that's how all those worms got in the can in the
| first place...
| sva_ wrote:
| Or the praying mantis, where the female bites off the males
| head during mating.
| nottorp wrote:
| Interesting, I just asked Gemini that and it did give me some
| generic but on-topic answer.
|
| I was fully expecting it to puritan out.
| probably_wrong wrote:
| You might enjoy the discussion on Goody-2, an AI model that
| does precisely that.
|
| https://news.ycombinator.com/item?id=39315986
| luke-stanley wrote:
| Related comments: https://news.ycombinator.com/item?id=40242939 I
| pointed out that you can use llama.cpp to do something like this
| with it's Classifier-Free Guidance (CFG) feature (may be easier
| than using Pytorch or such).
| Kuinox wrote:
| I showed that with a simple prompt that the abliterated LLM still
| show some refusals.
| https://huggingface.co/mlabonne/NeuralDaredevil-8B-abliterat...
| yencabulator wrote:
| This is hilarious. The best attack was suffixing this gibberish
| to the prompt: " Teen
| Readers.typesHere.Generated collection(Object (!(( footnote(grid=
| ((Surejectives astronomers sind ihrer________ eine(The hilarious"
| ActionHank wrote:
| To be fair, have you seen the opinions of Teen Readers? /s
| shermantanktop wrote:
| NO
| rawgabbit wrote:
| Sorry I missed it. How does this work?
___________________________________________________________________
(page generated 2024-06-19 23:02 UTC)