[HN Gopher] Language models can explain neurons in language models
___________________________________________________________________
Language models can explain neurons in language models
Author : mfiguiere
Score : 420 points
Date : 2023-05-09 17:15 UTC (5 hours ago)
(HTM) web link (openai.com)
(TXT) w3m dump (openai.com)
| ftxbro wrote:
| I wonder will someone please check the neurons associated to the
| petertodd and other anomalous glitch tokens
| (https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-
| petert...)? I can see the github and I see that for any given
| neuron you can see associated tokens but I don't see how to do an
| inverse search.
| ShamelessC wrote:
| Those were discovered by finding strings that OpenAI's
| tokenizer didn't properly split up. Because of this, they are
| treated as singular tokens, and since these don't occur
| frequently in the training data, you get what are effectively
| random outputs when using them.
|
| The author definitely tries to up the mysticism knob to 11
| though, and the post itself is so long, you can hardly finish
| it before seeing this obvious critique made in the comments.
| ftxbro wrote:
| Thank you for your opinion on the post that I linked! I'm
| still curious about the associated neurons though.
| ShamelessC wrote:
| Fair enough. You would need to use an open model or work at
| OpenAI. I assume this work could be used on the llama
| models - although I'm not aware of anyone has found these
| glitchy phrases for those models yet.
| ftxbro wrote:
| > You would need to use an open model or work at OpenAI.
|
| The point of this post that we are commenting under is
| that they made this association public, at least in the
| neuron->token direction. I was thinking some hacker (like
| on hacker news) might be able to make something that can
| reverse it to the token->neuron direction using the
| public data so we could see the petertodd associated
| neurons.
| https://openaipublic.blob.core.windows.net/neuron-
| explainer/...
| throwuwu wrote:
| The ironic thing about lesswrong is that it's quite the
| opposite in some fantastically oblivious ways.
| ShamelessC wrote:
| Yeah, it's quite strange indeed. Clearly people with decent
| educations but zero background in applied research/peer
| review. More concerned with the sound of their own voice
| than with whether or not their findings are actually useful
| (or even true).
|
| Perhaps they are all on stimulants!
| bilsbie wrote:
| Can anyone explain what they did? I'm not understanding from the
| webpage or the paper. What role does gpt4 play?
|
| I'm seeing they had gpt4 label every neuron but how?
| redconfetti wrote:
| I got the impression that it mentioned that the complexity of
| what's going on in GPT is so complex that we should use GPT to
| explain/summarize/graph what is going on.
|
| We should ask AI, how are you doing this?
| redconfetti wrote:
| Operator: Skynet, are you doing good thing? Skynet: Yes.
| ccvannorman wrote:
| My take: Regurgitation of trained-on information about LLMs does
| not come anywhere close to "conscious brain knows it's
| conscious."
| TyrianPurple wrote:
| By analyzing the function of individual neurons, these guys
| were/are able to gain a deeper understanding of how language
| models process language, which could lead to improved model
| architecture and training methods.
| tschumacher wrote:
| Even if we can explain the function of a single neuron what do we
| gain? If the goal is to reason about safety of computer vision in
| automated driving as an example, we would need to understand the
| system as a whole. The whole point of neural networks is to solve
| nuanced problems we can't clearly define. The fuzziness of the
| problems those systems solve is fundamentally at odds with the
| intent to reason about them.
| ChatGTP wrote:
| I have to agree.
|
| I often think, "maybe I should use ChatGPT for this" then I
| realise I have very little way to verify what it tells me and
| as someone working in engineering, If I don't understand the
| black box, I just can't do it.
|
| I'm attracted to open source, because I can look at the code
| understand it.
| slowhadoken wrote:
| Language models can also tell you they're not AI.
| 0xdeadbeefbabe wrote:
| With dubious confidence too!
| davesque wrote:
| Based on my skimming the paper, am I correct in understanding
| that they came up with an elaborate collection of prompts that
| embed the text generated by GPT-2 as well as a representation of
| GPT-2's internal state? Then, in effect, they simply asked GPT-4,
| "What do you think about all this?"
|
| If so, they're acting on a gigantic assumption that GPT-4
| actually correctly encodes a reasonable model of the body of
| knowledge that went into the development of LLMs.
|
| Help me out. Am I missing something here?
| xthetrfd wrote:
| This blog post is not very informative. How did they prompt GPT4
| to explain the neuron's behavior?
| samgriesemer wrote:
| It's explained more in the "read paper" link, where they
| provide the actual prompts:
|
| https://openaipublic.blob.core.windows.net/neuron-explainer/...
| andrewprock wrote:
| At some point did we change the name from perceptron to neuron?
| Neural networks don't have neurons.
| sebastianconcpt wrote:
| Great. Going meta with an introspective feedback loop.
|
| Let's see if that's the last requisite for exponential AGI
| growth...
|
| Singoolaretee here we go..............
| mcguire wrote:
| There is no introspection here.
| sebastianconcpt wrote:
| ...our approach to alignment research: we want to automate
| the alignment research work itself. A promising aspect of
| this approach is that it scales with the pace of AI
| development. As future models become increasingly intelligent
| and helpful as assistants, we will find better explanations.
|
| The distance between "better explanations" and using that as
| input of prompts that would automate self-improve is very
| small, yes?
| Ameo wrote:
| I built a toy neural network that runs in the browser[1] to model
| 2D functions with the goal of doing something similar to this
| research (in a much more limited manner, ofc). Since the input
| space is so much more limited than language models or similar,
| it's possible to examine the outputs for each neuron for all
| possible inputs, and in a continuous manner.
|
| In some cases, you can clearly see neurons that specialize to
| different areas of the function being modeled, like this one:
| https://i.ameo.link/b0p.png
|
| This OpenAI research seems to be feeding lots of varied input
| text into the models they're examining and keeping track of the
| activations of different neurons along the way. Another method I
| remember seeing used in the past involves using an optimizer to
| generate inputs that maximally activate particular neurons in
| vision models[2].
|
| I'm sure that's much more difficult or even impossible for
| transformers which operate on sequences of tokens/embeddings
| rather than single static input vectors, but maybe there's a way
| to generate input embeddings and then use some method to convert
| them back into tokens.
|
| [1] https://nn.ameo.dev/
|
| [2] https://www.tensorflow.org/tutorials/generative/deepdream
| jerpint wrote:
| I'm most surprised by the approach they take of passing GPT
| tuples of (token, importance) and having the model reliably
| figure out the patterns.
|
| Nothing would suggest this should work in practice, yet it
| just... does. In more or less zero shot. With a completely
| different underlying model. That's fascinating.
| bilsbie wrote:
| They're not looking at activations?
| kobe_bryant wrote:
| its interesting that layer 0 is a bunch of different things like
| upper case letters and symbols and types of verbs.
|
| it would be great to see all the things theyve found for
| different layers
| jacooper wrote:
| For people overwhelmed by all the AI science speak, just spend a
| few minutes with bing or phind and it will explain everything
| surprisingly well.
|
| Imagine telling someone in the middle of 2020, that in three
| years a computer will be able to speak, reason and explain
| everything as if it was a human, absolutely incredible!
| teaearlgraycold wrote:
| I agree it's crazy good. But timeline-wise, GPT-3 was in beta
| and used by many companies in 2020.
| sudoapps wrote:
| This is really interesting. Could this lead to eventually being
| able to deconstruct these "black-boxes" to remove proprietary
| data or enforce legal issues?
| whimsicalism wrote:
| I think this is a generous usage of "can." As the article admits,
| these explanations are 'imperfect' and I think that is definitely
| true.
| sebzim4500 wrote:
| It depends how you parse it. It is clearly true that they 'can'
| explain neurons, in the sense that at least some of the neurons
| are quite well explained. On the other hand, it's also the case
| that the vast majority of neurons are not well explained at all
| by this method (or likely any method).
|
| It's only because of a quirk of AdamW that this is possible at
| all, if GPT-2 was trained with SGD almost no neurons would be
| interpretable.
|
| EDIT: This last part isn't true. I think they are only looking
| at the intermediate layer of the FFN which does have a
| privileged basis.
| whimsicalism wrote:
| > EDIT: This last part isn't true. I think they are only
| looking at the intermediate layer of the FFN which does have
| a privileged basis.
|
| it does?
| sebzim4500 wrote:
| Yeah, that's where they apply the activation function and
| that happens per neuron so you can't rotate everything and
| expect the same result.
| ftxbro wrote:
| > "This work is part of the third pillar of our approach to
| alignment research: we want to automate the alignment research
| work itself."
|
| I feel like this isn't a Yud-approved approach to AI alignment.
| FeepingCreature wrote:
| Honestly, I think any foundational work on the topic is
| inherently Yud-favored, compared to the blithe optimism and
| surface-level analysis at best that is usually applied to the
| topic.
|
| Ie, I think it's not that this shouldn't be done. This should
| certainly be done. It's just that so many more things than it
| should be done before we move forward.
| killthebuddha wrote:
| DISCLAIMER: I think Yudkowsky is a serious thinker and his
| ideas should be taken seriously, regardless of whether not they
| are correct.
|
| Your comment triggered a random thought: A perfect name for
| Yudkowsky et al and the AGI doomers is... wait for it... the
| Yuddites :)
| throwaway2137 wrote:
| Already used on 4chan :)
| ShamelessC wrote:
| You mean Yudkowski? I saw him on Lex Fridman and he was
| entirely unconvincing. Why is everyone deferring to a bunch of
| effective altruism advocates when it comes to AI safety?
| lubesGordi wrote:
| I heard him on Lex too, and it seemed to be just a given that
| AI is going to be deceptive and want to kill us all. I don't
| think there was a single example of how that could be
| accomplished given. I'm open to hearing thoughts on this,
| maybe I'm not creative enough to see the 'obvious' ways this
| could happen.
| ethanbond wrote:
| This is also why I go into chess matches against 1400 elo
| players. I cannot conceive of the specific ways in which
| they will beat me (a 600 elo player), so I have good reason
| to suspect that I can win.
|
| I'm willing to bet the future of our species on my
| consistent victory in these types of matches, in fact.
| lubesGordi wrote:
| Again, a given that AI is adversarial. Edit: In addition,
| as an 1100 elo chess player, I can very easily tell you
| how a 1600 player is going to beat me. The analogy
| doesn't hold. I'm in good faith asking how AI could
| destroy humanity. It seems given the confidence people
| who are scared of AI have in this, that they have some
| concrete examples in mind.
| ethanbond wrote:
| No it's a given that some people who attempt to wield AI
| will be adversarial.
|
| In any case a similar argument can be made with merely
| instrumental goals causing harm: "I am an ant and I do
| not see how or why a human would cause me harm, therefore
| I am not in danger."
| lubesGordi wrote:
| People wielding AI and destroying humanity is very
| different from AI itself, being a weird alien
| intelligence, destroying humanity.
|
| Honestly if you have no examples you can't really blame
| people for not being scared. I have no reason to think
| this ant-human relationship is analogous.
|
| And seriously, I've made no claims that AI is benign so
| please stop characterizing my claims thusly. The question
| is simple, give me a single hypothetical example of how
| an AI will destroy humanity?
| ethanbond wrote:
| Sure, here's a trivial example: It radicalizes or
| otherwise deceives an employee at a virus research lab
| into producing and releasing a horrific virus.
|
| The guy at Google already demonstrated that AIs are able
| to convince people of fairly radical beliefs (and we have
| proof that even _humans_ a thousand years ago were
| capable of creating belief systems that cause people to
| blow themselves up and kill thousands of innocent
| people).
|
| P.S. I was not characterizing your opinion, I was
| speaking in the voice of an ant.
| kevinventullo wrote:
| Other caveman use fire to cook food. Fire scary and hurt.
| No understand fire. Fire cavemen bad.
| ethanbond wrote:
| Other caveman use nuke to wipe out city. Nuke scary and
| hurt. No understand nuke. Nuke caveman bad.
|
| Other caveman use anthrax in subway station. Anthrax
| scary and hurt...
|
| Is AI closer to fire or closer to nukes and engineered
| viruses? Has fire ever invented a new weapon system?
|
| By the way: we have shitloads of regulations and safety
| systems around fire due to, you guessed it, the amount of
| harm it can do by accident.
| PeterisP wrote:
| IMHO the argument isn't that AI is definitely going to be
| deceptive and want to kill us all, but rather that if
| you're 90% sure that AI is going to be just fine, that 10%
| of existential risk is simply not acceptable, so you should
| assume that this level of certainty isn't enough and you
| should act as if AI may be deceptive and may kill us all
| and take very serious preventive measures even if you're
| quite certain that it won't be needed - because "quite
| certain" isn't enough, you want to be at "this is
| definitely established to not lead to Skynet" level.
| ethanbond wrote:
| Because they have arguments that AI optimists are unable to
| convincingly address.
|
| Take this blog post for example, which between the lines
| reads: we don't expect to be able to align these systems
| ourselves, so instead we're hoping these systems are able to
| align each other.
|
| Consider me not-very-soothed.
|
| FWIW, there are plenty of AI experts who have been raising
| alarms as well. Hinton and Christiano, for example.
| ryan93 wrote:
| People won't care until an actually scary AI exists. Will
| be easy to stop at that point. Or you can just stop
| research here and hope another country doesn't get one
| first. Im personally skeptical it will exist. Honestly
| might be making it worse with the scaremongering coming
| from uncharismatic AI alignment people.
| ethanbond wrote:
| Why would it be easy to stop at that point? The
| believable value prop will increase in lockstep with the
| believable scare factor, not to mention the (already
| significant) proliferation out of ultra expensive
| research orgs into open source repos.
|
| Nuclear weapons proliferated explicitly _because_ they
| proved their scariness.
| ryan93 wrote:
| If AI can exist humans have to figure it out. It's what
| we do. Really shockingly delusional to think people are
| gonna use chatgpt for a few min get bored and then ban it
| like it's a nuke. I'd rather the USA get it first
| anyways.
| ethanbond wrote:
| Where did I say we could or should ban it like a nuke?
|
| Anyway this is a good example of the completely blind-
| faith reasoning that backs AI optimism: we'll figure it
| out "because it's what we do."
|
| FWIW we have still not figured out how to dramatically
| reduce nuclear risk. We're here just living with it every
| single day still, and with AI we're likely stepping onto
| another tightrope that we and _all_ future generations
| have to walk flawlessly.
| ftxbro wrote:
| > Why is everyone deferring to a bunch of effective altruism
| advocates when it comes to AI safety?
|
| I'm not sure Yudkowski is an EA, but the EAs want him in
| their polycule.
| tomjakubowski wrote:
| He posts on the forum. I'm not sure what more evidence is
| needed that he's part of it.
|
| https://forum.effectivealtruism.org/users/eliezeryudkowsky
| ftxbro wrote:
| I guess it's true, not just a rationalist but also
| effective altruist!
| Teever wrote:
| Why does this matter?
| snapcaster wrote:
| Agreed, Yud does seem to have been right about the course
| things will take but I'm not confident he actually has any
| solutions to the problem to offer
| causalmodels wrote:
| His solution is a global regulatory regime to ban new large
| training runs. The tools required to accomplish this are,
| IMO, out of the question but I will give Yud credit for being
| honest about them while others who share his viewpoint try to
| hide the ball.
| qumpis wrote:
| Which things has he been right about and when, if you recall?
| shadowgovt wrote:
| "Yud-approved?"
| circuit10 wrote:
| https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky
| ftxbro wrote:
| He's the one in the fedora who is losing patience that
| otherwise smart sounding people are seriously considering
| letting AI police itself
| https://www.youtube.com/watch?v=41SUp-TRVlg
| jack_riminton wrote:
| That's not a fedora, that's his King of the Redditors crown
| aitanabewa wrote:
| Meaning is approved by Eliezer Yudkowsky.
|
| https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
| https://twitter.com/ESYudkowsky
| https://www.youtube.com/watch?v=AaTRHFaaPG8 (Lex Fridman
| Interview)
| bick_nyers wrote:
| These were my thoughts exactly. On one hand, this can enable
| alignment research to catch up faster. On the other hand, if we
| are worried about homicidal AI, then putting it in charge of
| policing itself (and training it to find exploits in a way) is
| probably not ideal.
| thomastjeffery wrote:
| OpenAI need to hear an explanation of the word "explain".
| srajabi wrote:
| "This work is part of the third pillar of our approach to
| alignment research: we want to automate the alignment research
| work itself. A promising aspect of this approach is that it
| scales with the pace of AI development. As future models become
| increasingly intelligent and helpful as assistants, we will find
| better explanations."
|
| On first look this is genius but it seems pretty tautological in
| a way. How do we know if the explainer is good?... Kinda leads to
| thinking about who watches the watchers...
| jacobr1 wrote:
| There is a longer-term problem of trusting the explainer
| system, but in the near-term that isn't really a concern.
|
| The bigger value here in the near-term is _explicability_
| rather than alignment per-se. Potentially having good
| explicability might provide insights into the design and
| architecture of LLMs in general, and that in-turn may enable
| better design of alignment-schemes.
| m1el wrote:
| You're correct to have a suspicion here. Hypothetically the
| explainer could omit a neuron or give a wrong explanation for
| the role of a neuron. Imagine you're trying to understand a
| neural network, and you spend enormous amount of time
| generating hypotheses and validating them. Well the explainer
| might give you 90% correct hypotheses, it means you have 10
| times less work to produce hypotheses. So if you have a solid
| way of testing an explanation, even if the explainer is evil,
| it's still useful.
| vhold wrote:
| It produces examples that can be evaluated.
|
| https://openaipublic.blob.core.windows.net/neuron-explainer/...
| bottlepalm wrote:
| Using 'im feeling lucky' from the neuron viewer is a really
| cool way to explore different neurons. And then being able to
| navigate up and down through the net to related neurons.
| eternalban wrote:
| Fun to look at activations and then search for the source
| on the net.
|
| _" Suddenly, DM-sliding seems positively whimsical"_
|
| https://openaipublic.blob.core.windows.net/neuron-
| explainer/...
|
| https://www.thecut.com/2016/01/19th-century-men-were-
| awful-a...
| KevinBenSmith wrote:
| I had similar thoughts about the general concept of using AI to
| automate AI Safety.
|
| I really like their approach and I think it's valuable. And in
| this particular case, they do have a way to score the explainer
| model. And I think it could be very valuable for various AI
| Safety issues.
|
| However, I don't yet see how it can help with the potentially
| biggest danger where a super intelligent AGI is created that is
| not aligned with humans. The newly created AGI might be 10x
| more intelligent than the explainer model. To such an extent
| that the explainer model is not capable of understanding any
| tactics deployed by the super intelligent AGI. The same way
| ants are most probably not capable of explaining the tactics
| delloyed by humans, even if we gave them a 100 years to figure
| it out.
| wongarsu wrote:
| It also lags one iteration behind. Which is a problem because a
| misaligned model might lie to you, spoiling all future research
| with this method
| regularfry wrote:
| It doesn't have to lag, though. You could ask gpt-2 to
| explain gpt-2. The weights are just input data. The reason
| this wasn't done on gpt-3 or gpt-4 is just because a) they're
| much bigger, and b) they're deeper, so the roles of
| individual neurons are more attenuated.
| sanxiyn wrote:
| > How do we know if the explainer is good?
|
| The paper explains this in detail, but here is a summary: an
| explanation is good if you can recover actual neuron behavior
| from the explanation. They ask GPT-4 to guess neuron activation
| given an explanation and an input (the paper includes the full
| prompt used). And then they calculate correlation of actual
| neuron activation and simulated neuron activation.
|
| They discuss two issues with this methodology. First,
| explanations are ultimately for humans, so using GPT-4 to
| simulate humans, while necessary in practice, may cause
| divergence. They guard against this by asking humans whether
| they agree with the explanation, and showing that humans agree
| more with an explanation that scores high in correlation.
|
| Second, correlation is an imperfect measure of how faithfully
| neuron behavior is reproduced. To guard against this, they run
| the neural network with activation of the neuron replaced with
| simulated activation, and show that the neural network output
| is closer (measured in Jensen-Shannon divergence) if
| correlation is higher.
| TheRealPomax wrote:
| Why is this genius? It's just the NN equivalent of making a new
| programming language and getting it to the point where its
| compiler can be written in itself.
|
| The reliability question is of course the main issue. If you
| don't know how the system works, you can't assign a trust value
| to anything it comes up with, even if it seems like what it
| comes up with makes sense.
| 0xParlay wrote:
| I love the epistemology related discussions AI inevitably
| surfaces. How can we know anything that isn't empirically
| evident and all that.
|
| It seems NN output could be trusted in scenarios where a test
| exists. For example: "ChatGPT design a house using [APP] and
| make sure the compiled plans comply with
| structural/electrical/design/etc codes for area [X]".
|
| But how is any information that isn't testable trusted? I'm
| open to the idea ChatGPT is as credible as experts in the
| dismal sciences given that information cannot be proven or
| falsified and legitimacy is assigned by stringing together
| words that "makes sense".
| typon wrote:
| Seems relevant: https://www.cs.cmu.edu/~rdriley/487/papers/Th
| ompson_1984_Ref...
| lynx23 wrote:
| I can almost hear the Animatrix voiceover: "At first, AI was
| useful. Then, we decided to automate oversight... The rest is
| history."
| shrimpx wrote:
| Seems like OpenAI is grasping at straws trying to make GPT "go
| meta".
|
| Reminds me of this Sam Altman quote from 2019:
|
| "We have made a soft promise to investors that once we build this
| sort-of generally intelligent system, basically we will ask it to
| figure out a way to generate an investment return."
|
| https://youtu.be/TzcJlKg2Rc0?t=1886
| ChatGTP wrote:
| I have a similar feeling, they've potentially built the most
| amazing but commercially useless thing in history.
|
| I don't mean it's not useful entirely, but I mean. It's not
| useful in that it's not deterministic enough to be trustworthy,
| it's dangerous and really hard to scale therefore it's more of
| an academic project than something that will make Altman as
| famous as Sergey Brin.
|
| I personally take people like Hinton seriously too and think
| people playing with these things need more oversight
| themselves.
| cschmid wrote:
| Has anyone here found a link to the actual paper? If I click on
| 'paper', I only see what seems to be an awkward HTML version.
| simonw wrote:
| You mean this?
| https://openaipublic.blob.core.windows.net/neuron-explainer/...
|
| Would you prefer a PDF?
|
| (I'm always fascinated to hear from people who would rather
| read a PDF than a web-native paper like this one, especially
| given that web papers are actually readable on mobile devices.
| Do you do all of your reading on a laptop?)
| cschmid wrote:
| My whole workflow of organizing and reading papers is
| centered on PDFs. While I like having interactive
| supplemental materials, I want to be able to print, save and
| annotate the papers I read.
| probably_wrong wrote:
| The equations look terrible on Firefox for Android, as they
| are _really_ small - a two-line fraction is barely taller
| than a single line, forcing me to constantly zoom in and out.
|
| So yes, I would prefer a PDF and have a guarantee that it
| will look the same no matter where I read it.
| nerpderp82 wrote:
| > always fascinated
|
| That feels like a loaded phrase. Is it "false confusion"
| adjacent?
| whimsicalism wrote:
| If you want to draw on it, PDF is usually the best
| bad_alloc wrote:
| Nope, reading the printed paper on... paper. :)
| superkuh wrote:
| With a pdf I don't have to update my PDF reader multiple
| times per month just to be able to read text.
|
| A PDF is a text document that includes all the text, images,
| etc within it in the state you are going to perceive them.
| That web page is just barely even a document. None of it's
| contents are natively within it, it all requires executing
| remote code which pulls down more remote code to run just to
| get the actual text and images to display... which they don't
| in my browser. I just see an index with links that don't work
| and the the "Contributions" which for some reason was
| actually included as text.
|
| Even as the web goes up it's own asshole in terms of
| recursive serial loading of javascript/json/whatever from
| unrelated domains and abandons all backwards compatibility,
| PDF, as a document, remains readable. I wish the web was
| still hyperlinked documents. The "application" web sucks for
| accessibility.
| kkylin wrote:
| I personally prefer reading PDF on an iPad so I can mark it
| up.
| hexomancer wrote:
| > Would you prefer a PDF?
|
| Yes, I was just reading the paper and some of the javascript
| glitched and deleted all the contents of the document except
| the last section, making me lose all context and focus.
| Doesn't really happen with PDF files.
| Imnimo wrote:
| To me the value here is not that GPT4 has some special insight
| into explaining the behavior of GPT2 neurons (they say it's
| comparable to "human contractors" - but human performance on this
| task is also quite poor). The value is that you can just run this
| on every neuron if you're willing to spend the compute, and
| having a very fuzzy, flawed map of every neuron in a model is
| still pretty useful as a research tool.
|
| But I would be very cautious about drawing conclusions from any
| individual neuron explanation generated in this way - even if it
| looks plausible by visual inspection of a few attention maps.
| mcguire wrote:
| They also mention they got a score above 0.8 for 1000 neurons
| out of GPT2 (which has 1.5B (?)).
| oofsa wrote:
| I thought they had only applied the technique to 307,200
| neurons. 1,000 / 307,200 = 0.33% is still low, but
| considering that not all neurons would be useful since they
| are initialized randomly, it's not too bad.
| nico wrote:
| Now grab the list of labels/explanations for each neuron, and
| train a small LLM only with data for that neuron.
|
| Then you get a dictionary/index of LLMs
|
| Could this be used to parallelize training?
|
| Or create lighter overall language models?
|
| The above would be like doing a "map", how would we do a
| "reduce"?
| SpaceManNabs wrote:
| Automating safeguards and interpretability decisions seems
| circular and likely to detach to policy.
| int_19h wrote:
| Of note:
|
| "... our technique works poorly for larger models, possibly
| because later layers are harder to explain."
|
| And even for GPT-2, which is what they used for the paper:
|
| "... the vast majority of our explanations score poorly ..."
|
| Which is to say, we still have no clue as to what's going on
| inside GPT-4 or even GPT-3, which I think is the question many
| want an answer to. This may be the first step towards that, but
| as they also note, the technique is already very computationally
| intensive, and the focus on individual neurons as a function of
| input means that they can't "reverse engineer" larger structures
| composed of multiple neurons nor a neuron that has multiple
| roles; I would expect the former in particular to be much more
| common in larger models, which is perhaps why they're harder to
| analyze in this manner.
| ryandvm wrote:
| Funny that we never quite understood how intelligence worked
| and yet it appears that we're pretty damn close to recreating
| it - still without knowing how it works.
|
| I wonder how often this happens in the universe...
| rvz wrote:
| > Which is to say, we still have no clue as to what's going on
| inside GPT-4 or even GPT-3, which I think is the question many
| want an answer to.
|
| Exactly. Especially:
|
| > ...the technique is already very computationally intensive,
| and the focus on individual neurons as a function of input
| means that they can't "reverse engineer" larger structures
| composed of multiple neurons nor a neuron that has multiple
| roles;
|
| This paper just brings us no closer to explainability in black
| box neural networks and is just another excuse piece by OpenAI
| to try to please the explainability situation that has been
| missing for decades in neural networks.
|
| It is also the reason why they cannot be trusted in the most
| serious of applications which such decision making requires
| lots of transparency rather than a model regurgitating nonsense
| confidently.
| jahewson wrote:
| > It is also the reason why they cannot be trusted in the
| most serious of applications which such decision making
| requires lots of transparency rather than a model
| regurgitating nonsense confidently.
|
| Like say, in court to detect if someone is lying? Or at an
| airport to detect drugs?
| int_19h wrote:
| You don't even have to look that far ahead. Apparently,
| people are already using ChatGPT to compile custom diet
| plans for themselves, and they expect it to take into
| account the information they supply regarding their
| allergies etc.
|
| But, yes, those are also good examples of what we shouldn't
| be doing, but are going to do anyway.
| carlmr wrote:
| >Apparently, people are already using ChatGPT to compile
| custom diet plans for themselves, and they expect it to
| take into account the information they supply regarding
| their allergies etc.
|
| Evolution is still doing it's thing.
| canadianfella wrote:
| What's the risk? Someone allergic to peanuts will eat
| peanuts because ChatGPT put it in their diet plan? That's
| silly.
| int_19h wrote:
| Yes, that's the risk, and people are literally doing that
| because "if it put them in the recipe, it knows that
| quantity is safe for me", or "I asked it if it's okay and
| it cited a study saying that it is".
| [deleted]
| coldtea wrote:
| Those cases sound like Darwin Awards mediated by high
| technology
| canadianfella wrote:
| [dead]
| [deleted]
| TaylorAlexander wrote:
| > the explainability situation that has been missing for
| decades in neural networks.
|
| Is this true? I thought explainability for things like DNNs
| for vision made pretty good progress in the last decade.
| pmarreck wrote:
| > It is also the reason why they cannot be trusted in the
| most serious of applications which such decision making
| requires lots of transparency rather than a model
| regurgitating nonsense confidently.
|
| Doesn't this criticism also apply to people to some extent?
| We don't know what the purpose of individual brain neurons
| is.
| TaylorAlexander wrote:
| People are better understood intuitively. We understand how
| people fail and why. We can build trust with people with
| some degree of success. But machine models are new and can
| fail in unpredictable ways. They also get deployed to
| billions of users in a way that humans do not, and deployed
| in applications that humans do not. So its certainly useful
| to try to explain neural networks in as great of detail as
| we can.
| istjohn wrote:
| Or we can build trust using black box methods like we do
| with humans, e.g., extrapolating from past behavior,
| administering tests, and the like.
| ketzo wrote:
| Is it really fair to say this brings us "no closer" to
| explainability?
|
| This seems like a novel approach to try to tackle the scale
| of the problem. Just because the earliest results aren't
| great doesn't mean it's not a fruitful path to travel.
| imranq wrote:
| I suspect that there's a sweet spot that combines a collection
| of several "neurons" and a human-readable explanation given a
| certain kind of prompt. However, this "three-body problem" will
| probably need some serious analytical capability to understand
| at scale
| gitfan86 wrote:
| We know that complex arrangements of neurons are triggered
| based on input and generating output that appears to have some
| intelligence to many humans.
|
| The more interesting question is why are
| intelligence/beauty/consciousness emergent properties that
| exist in our minds.
| dennisy wrote:
| Nature created humans to understand nature. We created GPT4
| to understand ourselves.
| mensetmanusman wrote:
| Humans are the universe asking who made it.
| keyle wrote:
| That's beautiful until you think about it.
|
| Humans so far have done a great job at destroying nature
| faster than any other kind could.
|
| And GPT4 was created for profit.
| NobleLie wrote:
| A mirror of ourselves*
| otabdeveloper4 wrote:
| There is no evidence that intelligence runs on neurons. Yes,
| there are neurons in brains, but there's also lots of other
| stuff in there too. And there are creatures that exhibit
| intelligent properties even though they have hardly any
| neurons at all. (An individual ant has only something like
| 250000 neurons, and yet they're the only creatures beside
| humans that managed to create a civilization.)
| rmorey wrote:
| This is not a good take. Yes there is a lot more going on
| in brains than just neuronal activity, we don't understand
| most of it. But understanding neurons and their connections
| is necessary (but not sufficient) to understanding what we
| consider intelligence. Also, 250k is a lot of neurons!
| Individual ants, as well as fruit flies which have even
| fewer neurons, show behavior we may consider intelligent.
| Source: I am not a scientist, but I work in neuroscience
| research
| srcreigh wrote:
| What's the argument that understanding neurons is
| necessary?
|
| Perhaps intelligence is like a black box input to our
| bodies (call it the "soul", even though this isn't
| testable and therefore not a hypothesis). The mind
| therefore wouldn't play any more of a role in
| intelligence than the eye. And I'm not sure people would
| say the eye is necessary for understanding intelligence.
|
| Now, I'm not really in a position to argue for such a
| thing, even if I believe it, but I'm curious what
| argument you might have against it.
| burnished wrote:
| You can actually hypothesize that a soul exists and that
| intelligence is non-material, its just that your tests
| would quickly disprove that hypothesis - crude physical,
| mechanical modifications to the brain cause changes to
| intellect and character. If your hypothesis was correct
| you would not expect to see changes like that at all.
|
| Some people think that neurons specifically aren't
| necessary for understanding intelligence but in the same
| way that understanding transistors isn't necessary to
| understand computers, that neurons comprise the units
| that more readily explain intelligence.
| rmorey wrote:
| The other comments have pretty much covered it. We can
| pretty clearly demonstrate that neurons in general are
| important to behavior (brain damage, etc) and we even
| have some understanding about specific neurons or
| populations/circuits of neurons and their relation to
| specific behaviors (Grid cells are a cool example). And
| this work is all ongoing, but we're also starting to
| relate the connectivity of networks of neurons to their
| function and role in information processing. Recently the
| first full connectome of a larval fruit fly was published
| - stay tuned for the first full adult connectome from our
| lab ;)
|
| Again, IANA neuroscientist, but this is my understanding
| from the literature and conversations with the scientists
| I work with.
| benlivengood wrote:
| Brain damage by physical trauma, disease, oxygen
| deprivation, etc. has dramatic and often permanent
| effects on the mind.
|
| The effect of drugs (including alcohol) on the mind. Of
| note is anesthesia which can reliably and reversibly stop
| internal experience in the mind.
|
| For a non-physical soul to hold our mind we would expect
| significant divergence from the above. Out of body
| experiences and similar are indistinguishable from
| dreams/hallucinations when tested against external
| reality (remote viewing and the like).
| mensetmanusman wrote:
| "For a non-physical soul to hold our mind we would expect
| significant divergence from the above."
|
| This sounds like it assumes a physical mind could access
| a non-physical soul. All we probably know is that we have
| to be using an intact mind to use free will.
| istjohn wrote:
| Why would you doubt neurons play a roll in intelligence
| when we've seen so much success in emulating human
| intelligence with artificial neural networks? It might
| have been an interesting argument 20 years ago. It's just
| silly now.
| burnished wrote:
| What else would intelligence run on?
| int_19h wrote:
| If you really want to present ants as a civilization, I
| don't think a single ant is a meaningful unit of that
| civilization comparable to a single human. A colony,
| perhaps - but then that's a lot more neurons, just
| distributed.
| holoduke wrote:
| Maybe the neurons are the hardware layer. The software is
| represented by the electronic activity. There is a good
| video https://youtu.be/XheAMrS8Q1c about this topic.
| account-5 wrote:
| In what way a civilization?
| jxf wrote:
| I'll repost a comment via Reddit that I think makes this
| case [0]:
|
| Ants have developed architecture, with plumbing,
| ventilation, nurseries for rearing the young, and paved
| thoroughfares. Ants practice agriculture, including
| animal husbandry. Ants have social stratification that
| differs from but is comparable to that of human cultures,
| with division of labor into worker, soldier, and other
| specialties that do not have a clear human analogy.
|
| Ants enslave other ants. Ants interactively teach other
| ants, something few other animals do, among them humans.
| Ants have built "supercolonies" dwarfing any human city,
| stretching over 5,000 km in one place. And ants too have
| a complex culture of sorts, including rich languages
| based on pheromones.
|
| Despite the radically different nature of our two
| civilizations, it is undeniable from an objective
| standpoint that this level of society has been achieved
| by ants.
|
| [0]: https://www.reddit.com/r/unpopularopinion/comments/t
| 2h1vs/an...
| TeMPOraL wrote:
| To be honest, this description is leaning _heavily_ on
| the associations we have with individual words used. Ant
| "architecture" isn't like our architecture. Ant
| "plumbing" and "ventilation" have little in common with
| the kind of plumbing and ventilation we use in buildings.
| "Nurseries", "rearing the young", that's just stretching
| the analogy to the point of breaking. "Agriculture",
| "animal husbandry" - I don't even know how to comment on
| that. "Social stratification" is literally a chemical
| feedback loop - ant larvae can be influenced by certain
| pheromones to develop into different types of ants, which
| happen to emit pheromones _suppressing_ development of
| larvae into more ants of that type. Etc.
|
| I could go on and on. Point being, analogies are fun and
| sometimes illuminating, but they're just that. There's a
| _vast_ difference in complexity between what ants do, and
| what humans do.
| mandmandam wrote:
| Nobody is saying that an ant might be the next Frank
| Lloyd Wright.
|
| They're saying they accomplish _incredible_ things for
| the size of their brain, which is absolutely and
| unequivocally true.
|
| "Go to the ant, thou sluggard; consider her ways, and be
| wise".
| ASalazarMX wrote:
| > There's a vast difference in complexity between what
| ants do, and what humans do.
|
| Interesting parallell with
| intelligence/sentience/sapience. Despite the means, isn't
| the end result what you have to judge? The end result
| looks like a rudimentary civilization. How much back in
| time would we have to go back to find more sophistication
| in ant societies than humans?
| est31 wrote:
| I think you can call ant societies civilizations, but the
| same time you can call a multi cellular organism a
| civilization, too. Usually, those also come from the same
| genetic seed similar to (most) ant colonies. But more
| importantly, you have various types of cooperation and
| specialization in multi cellular life. Airways are
| "ventillation", chitin using or keratinated tissues are
| "architecture", and there is even "animal husbandry" in
| the form of bacterial colonies living in organs.
| talentedcoin wrote:
| There is no evidence that any of those are emergent
| properties. It's no more or less logical than asserting they
| were placed there by a creator.
| PaulHoule wrote:
| I like the idea. Note that LLMs have some skill at decoding
| sequential dense vectors in the human brain
|
| https://pub.towardsai.net/ais-mind-reading-revolution-how-gp...
|
| so why not have them decode sequential dense vectors of their
| own activations?
|
| As for the majority scoring poorly, they suggest that most
| neurons won't have clear activation semantics so that is
| intrinsic to the task and you'd have to move to "decoding the
| semantics of neurons that fire as a group"
| jablongo wrote:
| This isnt exactly building an understanding of LLMs from first
| principles... IMO we should broadly be following the (imperfect)
| example set forth by neuroscientists attempting to explain fMRI
| scans and assigning functionality to various subregions in the
| brain. It is circular and "unsafe" from an alignment perspective
| to use a complex model to understand the internals of a simpler
| model; in order to understand GPT4 then we need GPT5? These
| approaches are interesting, but we should primarily focus on
| building our understanding of these models from building blocks
| that we already understand.
| roddylindsay wrote:
| I don't follow. Neuroscience imaging tools like fMRI are only
| used because it is impossible to measure the activations of
| each neuron in a brain in real time (unlike an artificial
| neural network). This research paper's attempt to understand
| the role of individual neurons or neuron clusters within a
| complete network gets much closer to "first principles" than
| fMRI.
| jablongo wrote:
| Right so it should be much easier w/ access to every neuron
| and activation. But the general approach is an experimental
| one where you try to use your existing knowledge about
| physics and biology to discern what is activating different
| structures (and neurons) in the brain. I agree w/ the
| approach of trying to assign some functionality to individual
| 'neurons', but I don't think that using GPT4 to do so is the
| most appealing way to go about that, considering GPT4 is the
| structure we are interested in decoding in the first place.
| PeterisP wrote:
| All of this seems to lead to something like this paper http
| s://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
|
| On the other hand, I find it plausible that it's
| fundamentally impossible to assign some functionality to
| individual 'neurons' due to the following argument:
|
| 1. Let's assume that for a system calculating a specific
| function, there is a NN configuration (weights) so that at
| some fully connected NN layer there is a well-defined
| functionality for specific individual neurons - #1
| represents A, #2 represents B, #3 represents C etc.
|
| 2. The exact same system outcome can be represented with
| infinitely many other weight combinations which effectively
| result in a linear transformation (i.e. every possible
| linear transformation) of the data vector at this layer,
| e.g. where #1 represents 0.1A + 0.3B + 0.6C, #2 represents
| 0.5B+0.5C, and #3 represents 0.4B+0.6C - in which case the
| functionality A (or B, or C) is not represented by any
| individual neurons;
|
| 3. When the system is trained, it's simply not likely that
| we just happen to get the best-case configuration where the
| theoretically separable functionality is actually separated
| among individual 'neurons'.
|
| Biological minds do get this separation because each
| connection has a metabolic cost; but the way we train our
| models (both older perceptron-like layers, and modern
| transfomer/attention ones) do allow linking everything to
| everything, so the natural outcome is that functionality
| simply does not get cleanly split out in individual
| 'neurons' and each 'neuron' tends to represent some mix of
| multiple functionalities.
| djokkataja wrote:
| > in order to understand GPT4 then we need GPT5?
|
| I also found this amusing. But you are loosely correct, AFAIK.
| GPT-4 cannot reliably explain itself in any context: say the
| total number of possible distinct states of GPT-4 is N; then
| the total number of possible distinct states of GPT-4 PLUS any
| context in which GPT-4 is active must be at least N + 1. So
| there are at least two distinct states in this scenario that
| GPT-4 can encounter that will necessarily appear
| indistinguishable to GPT-4. It doesn't matter how big the
| network is; it'll still encounter this limit.
|
| And it's actually much worse than that limit because a network
| that's actually useful for anything has to be trained on things
| besides predicting itself. Notably, this is GPT-4 trying to
| predict GPT- _2_ and struggling:
|
| > We found over 1,000 neurons with explanations that scored at
| least 0.8, meaning that according to GPT-4 they account for
| most of the neuron's top-activating behavior. Most of these
| well-explained neurons are not very interesting. However, we
| also found many interesting neurons that GPT-4 didn't
| understand. We hope as explanations improve we may be able to
| rapidly uncover interesting qualitative understanding of model
| computations.
|
| 1,000 neurons out of 307,200--and even for the highest-scoring
| neurons, these are still partial explanations.
| axutio wrote:
| I've been working in systems neuroscience for a few years
| (something of a combination lab tech/student, so full
| disclosure, not an actual expert).
|
| Based on my experience with model organisms (flies & rats,
| primarily), it is actually pretty amazing how analogous the
| techniques and goals used in this sort of research are to those
| we use in systems neuroscience. At a very basic level, the
| primary task of correlating neuron activation to a given
| behavior is exactly the same. However, ML researchers benefit
| from data being trivial to generate and entire brains being
| analyzable in one shot as a result, whereas in animal research
| elucidating the role of neurons in a single circuit costs
| millions of dollars and many researcher-years.
|
| The similarities between the two are so clear that I noticed
| that in its Microscope tool [1], OpenAI even refers to the
| models they are studying as "model organisms", an
| anthropomorphization which I find very apt. Another article I
| saw a while back on HN which I thought was very cool was [2],
| which describes the task of identifying the role of a neuron
| responsible for a particular token of output. This one is
| especially analogous because it operates on such a small scale,
| much closer to what systems neuroscientists studying model
| organisms do.
|
| [1] https://openai.com/research/microscope [2]
| https://clementneo.com/posts/2023/02/11/we-found-an-neuron
| [deleted]
| fnovd wrote:
| LLMs are quickly going to be able to start explaining their own
| thought processes better than any human can explain their own. I
| wonder how many new words we will come up with to describe
| concepts (or "node-activating clusters of meaning") that the AI
| finds salient that we don't yet have a singular word for. Or, for
| that matter, how many of those concepts we will find meaningful
| at all. What will this teach us about ourselves?
| chrisco255 wrote:
| Are there any examples of an LLM developing concepts that do
| not exist or cannot be inferred from its training set?
| sebzim4500 wrote:
| It is by definition impossible for an LLM to develop a
| concept that 'cannot be inferred from its training set'.
|
| On the other hand, that is an incredibly high bar.
| ftxbro wrote:
| I'm really curious what kind of concept you might have in
| mind. Can you give any example of a concept that if an LLM
| developed that concept then it would meet your criteria? It
| might sound like a sarcastic question but it's hard to agree
| on the meanings of "concepts that do not exist" or "concepts
| that cannot be inferred" maybe you can give some examples.
|
| EDIT: I see below you gave some examples, like invention of
| language before it existed, and new theorems in math that
| presumably would be of interest to mathematicians. Those ones
| are fair enough in my opinion. The AI isn't quite good enough
| for those ones I think, but I also think newer versions
| trained with only more CPU/GPU and more parameters and more
| data could be 'AI scientists' that will make these kinds of
| concepts.
| sgt101 wrote:
| The training sets are so poorly curated we will never know...
| PeterisP wrote:
| Tautologically, every concept that anything (LLM, or human,
| or alien) develops can be inferred from the input data(e.g.
| training set), because it was.
| chrisco255 wrote:
| No, it wasn't, language itself didn't even exist at one
| point. It wasn't inferred from training data into existence
| because such examples existed before. Now we have a
| dictionary of tens of thousands of words, which describe
| high level ideas, abstractions, and concepts that someone,
| somewhere along the line had to invent.
|
| And I'm not talking about imitation nor am I interested in
| semantic games, I'm talking about raw inventiveness. Not a
| stochastic parrot looping through a large corpus of
| information and a table of weights on word pairings.
|
| Has AI ever managed to learn something humans didn't
| already know? It's got all the physics text books in its
| data set. Can it make novel inferences from that? How about
| in math?
| flangola7 wrote:
| > No, it wasn't, language itself didn't even exist at one
| point.
|
| Language took dozens of millennia to form, and animals
| have long had vocalizations. Seems like a natural
| building on top of existing features.
|
| > Has AI ever managed to learn something humans didn't
| already know?
|
| AlphaZero invented all new categories of strategy for
| games like Go, when previously we thought almost all
| possible tactics had been discovered. AIs are finding new
| kinds of proteins we never thought about, which will blow
| up the fields of medicine and disease in a few years once
| the first trials are completed.
| fnovd wrote:
| "Cannot be inferred from its training set" is a pretty
| difficult hurdle. Human beings can infer patterns that aren't
| there, and we typically call those hallucinations or even
| psychoses. On the other hand, some unconfirmed, novel
| patterns that humans infer actually represent groundbreaking
| discoveries, like for example much of the work of Ramanujan.
|
| In a real sense, all of the future discoveries of mathematics
| already exist in the "training set" of our present
| understanding, we just haven't thought it all the way through
| yet. If we discover something new, can we say that the
| concept didn't exist, or that it "couldn't be inferred" from
| previous work?
|
| I think the same would apply to LLMs and their understanding
| of the way we encode information using language. Given their
| radically different approach to understanding the same
| medium, they are well poised to both confirm many things we
| understand intuitively as well as expose the shortcomings of
| our human-centric model of understanding.
| elwell wrote:
| And if the LLM is the explainer, it can lie to us if 'needed'.
| jmfldn wrote:
| "LLMs are quickly going to be able to start explaining their
| own thought processes better than any human can explain their
| own."
|
| There is no "their" and there is no "thought process" . There
| is something that produces text that appears to humans like
| there is something like thought going on (cf the Eliza Effect),
| but we must be wary of this anthropomorphising language.
|
| There is no self reflection, but if you ask an LLM program how
| "it" knows something it will produce some text.
| icholy wrote:
| Or maybe the human thought process isn't as sophisticated as
| we imagined.
| jmfldn wrote:
| I'm not arguing for or against that. It's more the
| implications of sentience and selfhood implicit in the
| language many use around LLMs.
| callesgg wrote:
| The text output of a llm is the thought process. In this
| context the main difference between humans and llms, is that
| llms can't have internalized thoughts. There are of course
| other differences to, like the fact that humans have a wider
| gamut of input: visuals, sound, input from other bodily
| functions. And the fact that we have live training.
| marshray wrote:
| > There is no self reflection, but if you ask an LLM program
| how "it" knows something it will produce some text.
|
| To be clear, you're saying that we should just dismiss out-
| of-hand any possibility that an LM AI might actually be able
| to explain its reasoning step-by-step?
|
| I find it kind of charming actually how so many humans are
| just so darn sure that they have their own _special kind_ of
| cognition that could never be replicated. Not even with
| 175,000,000,000 calculations for every word generated.
| jmfldn wrote:
| That's a strawman since I didn't argue anything about
| humans being special. I don't think there is anything
| necessarily inherently special about human intelligence,
| I'm just advocating for caution around the language we use
| to talk about current systems.
|
| All this talk of AGI and sentience and so on is premature
| and totally unfounded . It's pure sci fi, for now at least.
| nerpderp82 wrote:
| What if you ask it to emit the reflexive output, then feed
| that reflexive output back into the LLM for the conscious
| answer?
|
| What if you ask it to synthesize multiple internal streams of
| thought, for an ensemble of interior monologues, then have
| all those argue with each other using logic and then present
| a high level answer from that panoply of answers?
| YawningAngel wrote:
| What if you do? LLMs don't have reflexive output or
| internal streams of thought, they are simply (complex)
| processes that produce streams of tokens based on an
| inputted stream of tokens. They don't have a special
| response to tokens that indicate higher-level thinking to
| humans.
| int_19h wrote:
| If you direct the model output to itself and don't view
| it otherwise, how is it not an "internal stream of
| thought"?
| jpasmore wrote:
| As we don't know for sure what is happening 100% within a
| neural network, we can say we don't believe that they're
| thinking and we would still need to define the word thinking.
| Once LLM's can self-modify, the word "thinking" will be more
| accurate than it is today.
|
| And when Hinton says at MIT, "I find it very hard to believe
| that they don't have semantics when they consult problems
| like you know how I paint the rooms how I get all the rooms
| in my house to be painted white in two years time," I believe
| he's commenting on the ability of LLM's to think on some
| level.
| mcguire wrote:
| In this case, I think we do if you will check out the paper
| (https://openaipublic.blob.core.windows.net/neuron-
| explainer/...). Their method is to
|
| 1. Show GPT-4 a GPT-produced text with the activation level
| of a specific neuron at the time it was producing that part
| of the text highlighted. They then ask GPT-4 for an
| explanation of what the neuron is doing.
|
| Text: "...mathematics is _done _properly__ , it...if it's
| _done _right__. (Take ... "
|
| GPT produces "words and phrases related to performing
| actions correctly or properly".
|
| 2. Based on the explanation, get GPT to guess how strong
| the neuron activates on a new text.
|
| "Assuming that the neuron activates on words and phrases
| related to performing actions correctly or properly. GPT-4
| guesses how strongly the neuron responds at each token:
| '...Boot. When done __correctly__ , "Secure...'"
|
| 3. Compare those predictions to the actual activations of
| the neuron on the text to generate a score.
|
| So there is no introspection going on.
|
| They say, " _We applied our method to all MLP neurons in
| GPT-2 XL [out of 1.5B?]. We found over 1,000 neurons with
| explanations that scored at least 0.8, meaning that
| according to GPT-4 they account for most of the neuron 's
| top-activating behavior._" But they also mention, "
| _However, we found that both GPT-4-based and human
| contractor explanations still score poorly in absolute
| terms. When looking at neurons, we also found the typical
| neuron appeared quite polysemantic._ "
| codehalo wrote:
| Pride comes before the fall, and the AI comes before
| humility.
| emporas wrote:
| Very true. In my opinion, in case there is a way to extract
| "Semantic Clouds of Words", i.e given a particular topic,
| navigate semantic clouds word by word, find some close
| neighbours of that word, jump to a neighbour of that word and
| so on, then LLMs might not seem that big of a deal.
|
| I think LLMs are "Semantic Clouds of Words" + grammar and
| syntax generator. Someone could just discard the grammar and
| syntax generator, just use the semantic cloud and create the
| grammar and syntax by himself.
|
| For example, in writing a legal document, a slightly educated
| person on the subject, could just use the relevant words put
| into an empty paper, fill in the blanks of syntax and
| grammar, alongside with the human reasoning which is far
| superior than any machine reasoning, till today at least.
|
| The process of editing the GPT* generated documents to fix
| reasoning is not a negligible task anyway. Sam Altman
| mentioned that: "the machine has some kind of reasoning", not
| a human reasoning ability by any means.
|
| My point is, that LLMs are two programs fused into one, "word
| clouds" and "syntax and grammar", sprinkled with some kind of
| poor reasoning. Their word clouding ability, is so
| unbelievable stronger than any human it fills me with awe
| every time i use it. Everything else is, just whatever!
| ly3xqhl8g9 wrote:
| First of all, our own explanations about ourselves and our
| behaviour are mostly lies, fabrications, hallucinations, faulty
| re-memorization, post hoc reasoning:
|
| "In one well-known experiment, a split-brain patient's left
| hemisphere was shown a picture of a chicken claw and his right
| hemisphere was shown a picture of a snow scene. The patient was
| asked to point to a card that was associated with the picture
| he just saw. With his left hand (controlled by his right
| hemisphere) he selected a shovel, which matched the snow scene.
| With his right hand (controlled by his left hemisphere) he
| selected a chicken, which matched the chicken claw. Next, the
| experimenter asked the patient why he selected each item. One
| would expect the speaking left hemisphere to explain why it
| chose the chicken but not why it chose the shovel, since the
| left hemisphere did not have access to information about the
| snow scene. Instead, the patient's speaking left hemisphere
| replied, "Oh, that's simple. The chicken claw goes with the
| chicken and you need a shovel to clean out the chicken shed""
| [1]. Also [2] has an interesting hypothesis on split-brains:
| not two agents, but two streams of perception.
|
| [1] 2014, "Divergent hemispheric reasoning strategies: reducing
| uncertainty versus resolving inconsistency",
| https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522
|
| [2] 2017, "The Split-Brain phenomenon revisited: A single
| conscious agent with split perception",
| https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf
| haldujai wrote:
| I'm not understanding the connection between your paragraphs
| here even after reading the first article.
|
| Even if you accept classic theory (e.g. hemispheric
| localization and the homunculus) which most experts don't all
| this suggests is that the brain tries to make sense of the
| information it has and in sparse environments it fills in.
|
| How does this make our behavior "mostly lies, fabrications,
| hallucinations, faulty re-memorization, post hoc reasoning"
| as most humans don't have a severed corpus callosum.
|
| The discussion starts with:
|
| "In a healthy human brain, these divergent hemispheric
| tendencies complement each other and create a balanced and
| flexible reasoning system. Working in unison, the left and
| right hemispheres can create inferences that have explanatory
| power and both internal and external consistency."
| mcguire wrote:
| I think the point is that, in a non-healthy brain, the
| brain can create a balanced and flexible reasoning system
| that creates inferences that have explanatory power, but
| which may not match external reality. Oliver Sacks has a
| long bibliography of the weird things that can go on in
| brains.
|
| But the bottom line is that introspection is not
| necessarily reliable.
| og_kalu wrote:
| He didn't say behavior. He said explanations of behaviour.
| Split brain experiments aside, this is pretty evident from
| other research. We can't recreate previous mental states,
| we just do a pretty good job (usually) of rationalizing
| decisions after the fact.
| https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/
| haldujai wrote:
| I'm reading this as our explanations for our own behavior
| as in why am I typing on this keyboard right now, in
| which case it's not evident at all.
|
| The existence of cognitive dissonance suggested in your
| citation is in no way analogous to "our own explanations
| about ourselves and our behaviour are mostly lies,
| fabrications, hallucinations, faulty re-memorization,
| post hoc reasoning" and in fact supports the opposite.
| ly3xqhl8g9 wrote:
| One primary explanation of ourselves is that there is in
| fact a "self" there, we feel this "self" as being
| permanent, continuous through time, yet we are absolutely
| sure that is a lie: there are no continuous processes in
| the entire universe, energy itself is quantized.
|
| In the morning when we wake up, we are "booting" up the
| memories the brain finds and we believe that we have
| persisted through time, from yesterday to today, yet we
| are absolutely sure that is a lie: just look at an
| Alzheimer patient.
|
| We are feeling this self as if it's somewhere above the
| neck and we feel like this self is looking at the world
| and sees "out there", yet we are absolutely sure that is
| a lie: our senses are being overflown by inputs and the
| brain filters them, shapes a model of the world, and
| presents that model to the internal model of itself,
| which gets so immersed into model of the world that
| starts to believe the model is indeed the world, until
| the first bistable image [1] breaks the model down.
|
| [1] https://www.researchgate.net/profile/Amanda-
| Parker-14/public...
| westurner wrote:
| Not supported by neuroimaging. Promoted without evidence or
| sufficient causal inference.
|
| https://www.health.harvard.edu/blog/right-brainleft-brain-
| ri... :
|
| > _But, the evidence discounting the left /right brain
| concept is accumulating. According to a 2013 study from the
| University of Utah, brain scans demonstrate that activity is
| similar on both sides of the brain regardless of one's
| personality._
|
| > _They looked at the brain scans of more than 1,000 young
| people between the ages of 7 and 29 and divided different
| areas of the brain into 7,000 regions to determine whether
| one side of the brain was more active or connected than the
| other side. No evidence of "sidedness" was found. The authors
| concluded that the notion of some people being more left-
| brained or right-brained is more a figure of speech than an
| anatomically accurate description._
|
| Here's wikipedia on the topic: "Lateralization of brain
| function" https://en.wikipedia.org/wiki/Lateralization_of_bra
| in_functi...
|
| Furthermore, "Neuropsychoanalysis"
| https://en.wikipedia.org/wiki/Neuropsychoanalysis
|
| Neuropsychology:
| https://en.wikipedia.org/wiki/Neuropsychology
|
| Personality psychology > ~Biophysiological:
| https://en.wikipedia.org/wiki/Personality_psychology
|
| MBTI > Criticism: https://en.wikipedia.org/wiki/Myers%E2%80%9
| 3Briggs_Type_Indi...
|
| Connectome: https://en.wikipedia.org/wiki/Connectome
| haldujai wrote:
| Agree I'm not a neuro subspecialist but I've listened to
| some talks at conferences out of interest and I don't think
| anyone still believes in this anymore. Anecdotally the few
| fMRI's I reported as a trainee didn't support this either.
| jorgeortiz85 wrote:
| You are talking about the popular narrative of "left brain"
| thinking being more logical and "right brain" thinking
| being more creative. You are correct this is unsupported.
|
| The post you are replying to is talking about the small
| subset of individuals who have had their corpus callosum
| surgically severed, which makes it much more difficult for
| the brain to send messages between hemispheres. These
| patients exhibit "split brain" behavior that is well
| studied by experiments and can shed light into human
| consciousness and rationality.
| mcguire wrote:
| Your response doesn't seem to be directly related to the
| previous poster's split-brain comments, but rather the
| popular misuse of the lateralization idea.
| ethanbond wrote:
| This is not relevant to GP's comment. It has nothing to do
| with "are there fixed 'themes' that are operated in each
| hemisphere." It has to do with more generally, does the
| brain know what the brain is doing. The answer so far does
| not seem to be "yes."
| haldujai wrote:
| Says who? There is actual evidence to support that our
| brain doesn't "know" what it is doing on a subconscious
| level? As far as I'm aware it's more that conscious
| humans don't understand how our brain works.
|
| I think the correct statement is "so far the answer is we
| don't know"
| ethanbond wrote:
| The split brain experiments very very clearly indicate
| that different parts of the brain can independently
| conduct behavior and gain knowledge independently of
| other parts.
|
| How or if this generalizes to healthy brains is not super
| clear, but it does actually provide a good explanatory
| model for all sorts of self-contradictory behavior (like
| addiction): the brain has many semi-independent
| "interests" that are jockeying for overall control of the
| organism's behavior. These interests can be fully
| contradictory to each other.
|
| Correct, ultimately we do not know. But it's actually a
| different question than your rephrasing.
| rounakdatta wrote:
| Phantoms in the Brain is a fascinating book that deals with
| exactly this topic.
| BaculumMeumEst wrote:
| that is absolutely fascinating and also makes me extremely
| uncomfortable
| nomel wrote:
| This is why I suggest that curious individuals try a
| hallucinogen at least once*. It really makes the fragility
| of our perception, and how it's held up mostly by itself,
| very apparent.
|
| * in a safe setting with support, of course.
| causi wrote:
| Neurology is full of very uncomfortable facts. Here's one
| for you: there are patients who believe their arm is gone
| even though it's still there. When the doctor asks whose
| arm that is, they reply it must be someone else's. The
| brain can simply refuse to know something, and will adopt
| whatever delusions and contortions are necessary. Which of
| course leads to the realization that there could be things
| we're _all_ incapable of knowing. There could be things
| right in front of our faces we simply refuse to perceive
| and we 'd never know it.
| mcguire wrote:
| Oliver Sacks' _A Leg To Stand On_ is a lengthy discussion
| of that, including his own experiences after breaking a
| leg---IIRC, at one point after surgery but before he
| starts physical therapy, he wakes up convinced that a
| medical student has played a prank by removing his leg
| and attaching one from a cadaver, or at least sticking a
| cadaver 's leg under his blanket. (ISTR he tries to throw
| it out of bed and ends up on the floor.)
| ly3xqhl8g9 wrote:
| Famously, our nose is literally right in front of our
| faces and the brain simply "post-processes" it out of the
| view.
|
| After breaking my arm, split in two, pinching the nerve
| and making me unable to move it for about a year, I still
| feel as if the arm is "someone else's", as if I am moving
| an object in VR, not something which is "me" or "mine".
| tokamak-teapot wrote:
| Just wait until you notice how much humans do this day to
| day
| Joeri wrote:
| If that makes you uncomfortable you definitely should not
| go reading the evidence supporting the notion that
| conscious free will is an illusion.
|
| https://www.mpg.de/research/unconscious-decisions-in-the-
| bra...
| mcguire wrote:
| My impression is that the understanding of that research
| that comes up with statements like "But when it comes to
| decisions we tend to assume they are made by our
| conscious mind. This is questioned by our current
| findings" is based on dualistic reasoning.
|
| The idea that there should not be any neural activity
| before a conscious decision is straight-up dualism---the
| intangible soul makes a decision and neural activity
| follows it to carry out the decision.
|
| An alternative way of understanding that result is that
| the neural activity that precedes the "conscious
| decision" is the brain's mechanism of coming up with that
| decision. The "conscious mind" is the result of neural
| activity, right?
| incangold wrote:
| Same. We are so, so profoundly not what it feels like we
| are, to most of us anyway.
|
| I am morbidly curious how people are going to creatively
| explain away the more challenging insights AI gives us in
| to what consciousness is.
| ly3xqhl8g9 wrote:
| It's probably way worse than we can imagine.
|
| Reading/listening to someone like Robert Sapolsky [1] makes
| me laugh I could have ever hallucinated about such a muddy,
| not even wrong concept as "free will".
|
| Furthermore, between the brain and, say, the liver there is
| only a difference of speed/data integrity inasmuch as one
| cares to look for information processing as basal
| cognition: neurons firing in the brain, voltage-gated ion
| channels and gap junctions controlling bioelectrical
| gradients in the liver, and almost everywhere in the body.
| Why does only the brain has a "feels like" sensation? The
| liver may have one as well, but the brain being an
| autarchic dictator perhaps suppresses the feeling of the
| liver, it certainly abstracts away the thousands of highly
| specialized decisions the liver takes each second solving
| adequately the complex problem space of blood processing.
| Perhaps Thomas Nagel shouldn't have asked "What Is It Like
| to Be a Bat?" [2] but what is it like to be a liver.
|
| [1] "Robert Sapolsky: Justice and morality in the absence
| of free will", https://www.youtube.com/watch?v=nhvAAvwS-UA
|
| [2] https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_B
| at%3F
| sclarisse wrote:
| The biggest problem with the current popular idea of
| "free will" is that people think it means they're
| ineffably unpredictable. They're uncomfortable with the
| notion that if you were to simulate their brain in
| sufficient detail, you could predict thoughts and
| reaction. They take refuge in pseudoscientific mumbling
| about the links to the Quantum, for they have heard it is
| special and unpredictable.
|
| And that's just the polar opposite of having a meaningful
| will at all. It is good that you are pretty much
| deterministic. You _shouldn't_ be deciding meaningful
| things randomly. If you made 20 copies of yourself and
| asked them to support or oppose some essential and
| important political question (about human rights, or war,
| or what-have-you) they should all come down on the same
| side. What kind of a Will would that be that chose
| randomly?
| mrcode007 wrote:
| If the Godel incompleteness theorem applies here, then the
| explanations are likely ... incomplete or self-referential.
| AlexCoventry wrote:
| The Goedel Incompleteness Theorem has no straightforward
| application to this question.
| galaxyLogic wrote:
| It would if the language model did reasoning according
| rules of logic. But they don't. They use Markov chains.
|
| To me it makes no sense to say that a LLM could explain its
| own reasoning if it does no (logical) reasoning at all. It
| might be able to explain how the neural network calculates
| its results. But there are no logical reasoning steps in
| there that could be explained, are there?
| incangold wrote:
| Honest question: are we sure that it doesn't do logical
| reasoning?
|
| IANAE but although an LLM meets the definition of a
| Markov Chain as I understand it (current state in,
| probabilities of next states out), the big black box that
| spits out the probabilities could be doing anything.
|
| Is it fundamentally impossible for reasoning to be an
| emergent property of an LLM, in a similar way to a brain?
| They can certainly do a good impression of logical
| reasoning- better than some humans in some cases?
|
| Just because an LLM can be described as a Markov Chain
| doesn't mean it _uses_ Markov Chains? An LLM is very
| different to the normal examples of Markov Chains I'm
| familiar with.
|
| Or am I missing something?
|
| In any case, coemu is an interesting related idea to
| constrain AIs to thinking in ways we can understand
| better:
|
| https://futureoflife.org/podcast/connor-leahy-on-agi-and-
| cog...
|
| https://www.alignmentforum.org/posts/ngEvKav9w57XrGQnb/co
| gni...
| VictorLevoso wrote:
| All programs that you can fit on a computer can be
| described by a sufficiently large Markov chain(if you
| imagine all the possible states the memory as nodes)
| Whatever the human brain is doing is also describable as
| a massive Markov chain.
|
| But since the markov chain becomes exponentially larger
| whit the amount of states this is a very nitpicky and
| meaningless point.
|
| Clearly to say something its a markov chain and have that
| mean something you need to say the thing its doing could
| be more or less compressed to a simple markov chain for
| bigrams or something like that, but that is just not true
| empirically, not even for gpt2. Just this is already
| pretty hard to make into a reasonable size markov chain
| https://arxiv.org/abs/2211.00593.
|
| Just saying that it outputs probabilities from each state
| is not enough, the states are english strings, there's
| (number of tokens)^contex_lenght possible states for a
| certain length that's not a reasonable markov chain that
| you could actually implement or run.
| mrcode007 wrote:
| My understanding is that at least one form of training in
| the RLHF involves supplying antecedent and consequent
| training pairs for entailment queries.
|
| The LLM seems to be only one of the many building blocks
| and is used to supply priors / transition probabilities
| that are used elsewhere in downstream part of the model.
| PartiallyTyped wrote:
| As long as lazy evaluation exists, self-reference is fine,
| no?
|
| Hofstadter talks about something similar in his books.
| wizeman wrote:
| That's probably one of the reasons why you'd use GPT-4 to
| explain GPT-2.
|
| Of course, if you were trying to use GPT-4 to explain GPT-4
| then I think the Godel incompleteness theorem would be more
| relevant, and even then I'm not so sure.
| drdeca wrote:
| What leads you to suspect that Godel incompleteness may be
| relevant here?
|
| There's no formal axiom system being dealt with here, afaict?
|
| Do you just generally mean "there may be some kind of self-
| reference, which may lead to some kind of liar-paradox-
| related issues"?
| mrcode007 wrote:
| I commented in another answer but you can consult
| https://etc.cuit.columbia.edu/news/basics-language-
| modeling-...
|
| Some training forms include entailment : "if A then B". I
| hope this is first order logic which does have an axiom
| system :)
| fnovd wrote:
| So is the word "word" but that seems to have worked out OK so
| far. I can explain the meaning of "meaning" and that seems to
| work OK too. Being self-referential sounds a lot more like a
| feature than a bug. Given that the neurons in our own heads
| are connected to each other and not any ground truth, I think
| LLMs should do just fine.
| bilsbie wrote:
| I'm so interested in this. Any ideas how I can get involved with
| only a 2014 laptop?
| VictorLevoso wrote:
| https://www.neelnanda.io/mechanistic-interpretability/gettin...
| Garrrrrr wrote:
| https://www.w3schools.com/ai/default.asp
| drBonkers wrote:
| Wow-- this page is superb.
| rounakdatta wrote:
| I'm split between what's more impressive:
|
| - The software powering the research paper
|
| - The research itself (holy moly! They're showing the neurons!)
| jokoon wrote:
| I wish there was insightful explanations on why AI cannot think,
| and if there are researchers trying to explore this topic, and if
| yes what they do.
___________________________________________________________________
(page generated 2023-05-09 23:00 UTC)