[HN Gopher] Show HN: Llama 3.2 Interpretability with Sparse Auto...
___________________________________________________________________
Show HN: Llama 3.2 Interpretability with Sparse Autoencoders
I spent a lot of time and money on this rather big side project of
mine that attempts to replicate the mechanistic interpretability
research on proprietary LLMs that was quite popular this year and
produced great research papers by Anthropic [1], OpenAI [2] and
Deepmind [3]. I am quite proud of this project and since I
consider myself the target audience for HackerNews did I think that
maybe some of you would appreciate this open research replication
as well. Happy to answer any questions or face any feedback.
Cheers [1] https://transformer-circuits.pub/2024/scaling-
monosemanticit... [2] https://arxiv.org/abs/2406.04093 [3]
https://arxiv.org/abs/2408.05147
Author : PaulPauls
Score : 522 points
Date : 2024-11-21 20:37 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| jaykr_ wrote:
| This is awesome! I really appreciate the time you took to
| document everything!
| PaulPauls wrote:
| Thank you for saying that! I have a much, much harder time
| documenting everything and writing out each decision in
| continuous text than actually writing the code. So it took a
| look time for me to write all of this down - so I'm happy you
| appreciate it! =)
| curious_cat_163 wrote:
| Hey - Thanks for sharing!
|
| Will take a closer look later but if you are hanging around now,
| it might be worth asking this now. I read this blog post
| recently:
|
| https://adamkarvonen.github.io/machine_learning/2024/06/11/s...
|
| And the author talks about challenges with evaluating SAEs. I
| wonder how you tackled that and where to look inside your repo
| for understanding the your approach around that if possible.
|
| Thanks again!
| PaulPauls wrote:
| So evaluating SAEs - determining which SAE is better at
| creating the most unique features while being as sparse as
| possible at the same time - is a very complex topic that is
| very much at the heart of the current research into LLM
| interpretability through SAEs.
|
| Assuming you already solved the problem of finding multiple
| perfect SAE architectures and you trained them to perfection
| (very much an interesting ML engineering problem that this SAE
| project attempts to solve) then deciding on which SAE is better
| comes down to which SAE performs better on the metrics of your
| automated interpretability methodology. Particularly OpenAI's
| methodology emphasizes this automated interpretability at scale
| utilizing a lot of technical metrics upon which the SAEs can be
| scored _and thereby evaluated_.
|
| Since determining the best metrics and methodology is such an
| open research question that I could've experimented on for a
| few additional months, have I instead opted for a simple
| approach in this first release. I am talking about my and
| OpenAI's methodology and the differences between both in
| chapter 4. Interpretability Analysis [1] in my Implementation
| Details & Results section. I can also recommend reading the
| OpenAI paper directly or visiting Anthropics transformer-
| circuits.pub website that often publishes smaller blog posts on
| exactly this topic.
|
| [1]
| https://github.com/PaulPauls/llama3_interpretability_sae#4-i...
| [2] https://transformer-circuits.pub/
| curious_cat_163 wrote:
| Thanks!
| JackYoustra wrote:
| Very cool work! Any plans to integrate it with SAELens?
| PaulPauls wrote:
| Not sure yet to be honest. I'll definitely consider it but I'll
| reorient myself and what I plan to do next in the coming week.
| I also planned on maybe starting a simpler project and maybe
| showing people how to create the full model of a current Llama
| 3.2 implementation from scratch in pure PyTorch. I love
| building things from teh ground up and when I looked for
| documentation for the Llama 3.2 background section of this SAE
| project then the existing documentation I found was either too
| superficial or outdated and intended for Llama 1 or 2 -
| Documentation in ML gets outdated so quickly nowadays...
| foundry27 wrote:
| For anyone who hasn't seen this before, mechanistic
| interpretability solves a very common problem with LLMs: when you
| ask a model to explain itself, you're playing a game of rhetoric
| where the model tries to "convince" you of a reason for what it
| did by generating a plausible-sounding answer based on patterns
| in its training data. But unlike most trends of benchmark numbers
| getting better as models improve, more powerful models often
| score worse on tests designed to self-detect "untruthfulness"
| because they have stronger rhetoric, and are therefore more
| compelling at justifying lies after the fact. The objective is
| coherence, not truth.
|
| Rhetoric isn't reasoning. True explainability, like what
| overfitted Sparse Autoencoders claim they offer, basically
| results in the causal sequence of "thoughts" the model went
| through as it produces an answer. It's the same way you may have
| a bunch of ephemeral thoughts in different directions while you
| think about anything.
| stavros wrote:
| I want to point out here that people do the same: a lot of the
| time we don't know why we thought or did something, but we'll
| confabulate plausible-sounding rhetoric after the fact.
| sinuhe69 wrote:
| Not in math.
| TeMPOraL wrote:
| Yes in math. Formalisms come _after_ casual thoughts, at
| every step.
| sinuhe69 wrote:
| What is a casual thought that you cannot explain in math?
| TeMPOraL wrote:
| That question makes no sense. You can explain anything in
| math, because math is a language and lets you define
| whatever terms and axioms you need at a given moment.
|
| (Whether or not such explanation is useful for anything
| is another issue entirely.)
| worldsayshi wrote:
| Can you explain how intuition led you to try a certain
| approach?
| TeMPOraL wrote:
| Is it enough if I hand-wave it with probability
| distributions, or do you want me to write out adjacency
| search in a high-dimensional space?
| mdp2021 wrote:
| It's totally different: those formalisms are in a
| workbench, following a set of rules that either work or
| not.
|
| So, yes, that (math) is representative of the actual
| process: _pattern recognition_ gives you _spontaneous
| ideas_ , that you _assess for truthfulness_ in conscious
| acts of verification.
| legel wrote:
| Math comes from brains.
| LoganDark wrote:
| The split-brain experiment is one of my favorites!
| https://www.youtube.com/watch?v=wfYbgdo8e-8
| btbuildem wrote:
| https://en.wikipedia.org/wiki/Peace_on_Earth_(novel)
| mdp2021 wrote:
| /Some/ people bullshit themselves stating the plausible;
| others check their hypotheses.
|
| The difference is total in both humans and automated
| processes.
| stavros wrote:
| How are you going to check your hypotheses for why you
| preferred that jacket to that other jacket?
| DSingularity wrote:
| Is that example representative for the LLM tasks for
| which we seek explainability ?
| stavros wrote:
| Are we holding LLMs to a higher standard than people?
| f_devd wrote:
| Ideally yes, LLMs are tools that we expect to work,
| people are inherently fallible and (even unintentionally)
| deceptive. LLMs being human-like in this specific way is
| not desirable.
| stavros wrote:
| Then I think you'll be very disappointed. LLMs aren't in
| the same category as calculators, for example.
| f_devd wrote:
| I have no illusions on LLMs, I have been working with
| them since og BERT, always with these same issues and
| more. I'm just stating what would be needed in the future
| to make them reliably useful outside of creative writing
| & (human-guided & checked) search.
|
| If an LLM provides an incorrect/orthogonal rhetoric
| without a way to reliably fix/debug it it's just not as
| useful as it theoretically could be given the data
| contained in the parameters.
| mdp2021 wrote:
| Do not lose the original point: some systems have a goal
| to sound plausible, while some have a goal to say the
| truth. Some systems, when asked "where have you been",
| will reply "at the baker's" because it is a nice
| narrative in their "novel writing, re-writing of
| reality", some other will check memory and say "at the
| butcher's", where they have actually been.
|
| When people invent explicit reasons on why they turned
| left or right, those reasons remain hypotheses. The
| clumsy will promote those hypotheses to beliefs. The apt
| will keep the spontaneous ideas as hypotheses, until the
| ability to assess them comes.
| catskul2 wrote:
| Everyone, every last one of us, does this every single day,
| all day, and only occasionally do we deviate to check
| ourselves, and often then it's to save face.
|
| A Nobel prize was given for related research to Daniel
| Kahneman.
|
| If you think it doesn't apply to you, you're definitely
| wrong.
| mdp2021 wrote:
| > _occasionally_
|
| Properly educated people do it regularly, not
| occasionally. You are describing a definite set of
| people. No, it does not cover all.
|
| Some people will output a pre-given answer; some people
| check.
|
| Edit: sniper... Find some argument.
| mdp2021 wrote:
| By the way: I have seldom come across a post so weak.
|
| > _every last one of us_
|
| And how do you prove it.
|
| > _A Nobel prize was given_
|
| So?
|
| > _If you think, you_
|
| Prove it.
|
| Support it, at least. That is not discussion.
| fsndz wrote:
| I stopped at: "causal sequence of "thoughts" "
| benchmarkist wrote:
| Interpretability research is basically a projection of the
| original function implemented by the neural network onto a
| sub-space of "explanatory" functions that people consider to
| be more understandable. You're right that the words they use
| to sell the research is completely nonsensical because the
| abstract process has nothing to do with anything causal.
| HeatrayEnjoyer wrote:
| All code is causal.
| benchmarkist wrote:
| Which makes it entirely irrelevant as a descriptive term.
| mdp2021 wrote:
| "Servers shall be strict in formulation and flexible in
| interpretation."
| Onavo wrote:
| How does the causality part work? Can it spit out a graphical
| model?
| benreesman wrote:
| A lot of the mech interp stuff has seemed to me like a
| different kind of voodoo: the Integer Quantum Hall Effect?
| Overloading the term "Superposition" in a weird analogy not
| governed by serious group representation theory and some clear
| symmetry? You guys are reaching. And I've read all the papers.
| Spot the postdoc who decided to get paid.
|
| But there is one thing in particular that I'll acknowledge as a
| great insight and the beginnings of a very plausible research
| agenda: bounded near orthogonal vector spaces are wildly
| counterintuitive in high dimensions and there are existing
| results around it that create scope for rigor [1].
|
| [1]
| https://en.m.wikipedia.org/wiki/Johnson%E2%80%93Lindenstraus...
| txnf wrote:
| Superposition code is a well known concept in information
| theory - I think there is certainly more to the story then
| described in the current works, but it does feel like they
| are going in the right direction
| drdeca wrote:
| Where are you seeing the integer quantum Hall effect
| mentioned? Or are you bringing it up rather than responding
| to it being brought up elsewhere? I don't understand what the
| connection between IQHE and these SAE interpretability
| approaches is supposed to be.
| benreesman wrote:
| Pardon me, the reference is to the fractional Hall effect.
|
| "But our results may also be of broader interest. We find
| preliminary evidence that superposition may be linked to
| adversarial examples and grokking, and might also suggest a
| theory for the performance of mixture of experts models.
| More broadly, the toy model we investigate has unexpectedly
| rich structure, exhibiting phase changes, a geometric
| structure based on uniform polytopes, "energy level"-like
| jumps during training, and a phenomenon which is
| qualitatively similar to the fractional quantum Hall effect
| in physics, among other striking phenomena. We originally
| investigated the subject to gain understanding of cleanly-
| interpretable neurons in larger models, but we've found
| these toy models to be surprisingly interesting in their
| own right."
|
| https://transformer-circuits.pub/2022/toy_model/index.html
| snthpy wrote:
| A{rt,I} imitating life
|
| I believe that's why humans reason too. We make snap judgements
| and then use reason to try to convince others of our beliefs.
| Can't recall the reference right now but they argued that it's
| really a tool for social influence. That also explains why
| people who are good at it find it hard to admit when they are
| wrong - they're not used to having to do it because they can
| usually out argue others. Prominent examples are easy to find -
| X marks de spot.
| briffid wrote:
| Jonathan Haidt's The Righteous Mind describes this in
| details.
| snthpy wrote:
| Thanks
| omgwtfbyobbq wrote:
| I think Robert Sapolsky's lectures on yt cover this to some
| degree around 115.
|
| https://youtu.be/wLE71i4JJiM?feature=shared
|
| Sometimes our cortex is in charge, sometimes other parts of
| our brain are, and we can't tell the difference. Regardless,
| if we try to justify it later, that justification isn't
| always coherent because we're not always using the part of
| our brain we consider to be rational.
| snthpy wrote:
| Yes that was probably it because I rewatched that recently.
| Thanks!
| shshshshs wrote:
| People who are good at reasoning find it hard to admit that
| they were wrong?
|
| That's not my experience. People with reason are..
| reasonable.
|
| You mention X and that's not where the reasoners are. That's
| where the (wanna be) politicians are. Rhetoric is not all of
| reasoning.
|
| I can agree that rationalizing snap judgements is _one_ of
| our capabilities but I am totally unconvinced that it is the
| totality of our reasoning capabilities. Perhaps I
| misunderstood.
| Hedepig wrote:
| This is not totally my experience, I've debated a
| successful engineer who by all accounts has good reasoning
| skills, but he will absolutely double down on unreasonable
| ideas he's made on the fly he if can find what he considers
| a coherent argument behind them. Sometimes if I absolutely
| can prove him wrong he'll change his mind.
|
| But I think this is ego getting in the way, and our
| reluctance to change our minds.
|
| We like to point to artificial intelligence and explain how
| it works differently and then say therefore it's not "true
| reasoning". I'm not sure that's a good conclusion. We
| should look at the output and decide. As flawed as it is, I
| think it's rather impressive
| mdp2021 wrote:
| > _ego getting in the way_
|
| That thing which was in fact identified thousands of
| years ago as the evil to ditch.
|
| > _reluctance to change our minds_
|
| That is clumsiness in a general drive that makes sense
| and is recognized part of the Belief Change Theory:
| epistemic change is conservative. I.e., when you revise a
| body of knowledge you do not want to lose valid notions.
| But conversely, you do not want to be unable to see
| change or errors, so there is a balance.
|
| > _it 's not "true reasoning"_
|
| If it shows not to explicitly check its "spontaneous"
| ideas, then it is a correct formula to say 'it's not
| "true reasoning"'.
| Hedepig wrote:
| > then it is a correct formula to say 'it's not "true
| reasoning"'
|
| why is that point fundamental?
| mdp2021 wrote:
| Because the same way you do not want a human interlocutor
| to speak out of its dreams, uttering the first ideas that
| come to mind unvetted, and you want him to instead have
| thought hard and long and properly and diligently and
| well, equally you'll want the same from an automation.
| snthpy wrote:
| People with reason ... sound reasonable.
|
| I think some prominent people on X who are good at
| reasoning from First Principles will double down on things
| rather than admit their mistake.
|
| The other very prominent psychological phenomenon I have
| observed in the world is "Projection", i.e. the phenomenon
| of seeing qualities in other people that we have ourselves.
| I guess it is because we think others would do what we
| would do ourselves. Trump is a clear example of this -
| whatever he accuses someone else off, you know he is doing.
| Point here being that this doubling down on bad reasons in
| order to not admit my mistakes is something I've observed
| in myself. Reason does indeed help me to try and overcome
| it when I recognise it but the tricky part is being able to
| recognise it.
| fragmede wrote:
| The smarter a person is, the better they are at
| rationalizing their decisions. _Especially_ the really
| stupid decisions.
| mdp2021 wrote:
| Already before Galileo we had experiments to determine
| whether ideas represented reality or not. And in crucial
| cases, long before that, it meant life or death. This will be
| clear to engineers.
|
| <<Reason>> is part of that mechanism of vetting ideas. You
| experience massive failures without it.
|
| So, no, trained judgement is a real thing, and the presence
| of innumerable incompetent do not prove an alleged absence of
| the competent.
| jamesemmott wrote:
| I wonder if the reference you are reaching for, if it's not
| the Jonathan Haidt book suggested by a sibling comment, is
| _The Enigma of Reason_ by the cognitive psychologists Hugo
| Mercier and Dan Sperber (2017).
|
| In that book (quoting here from the abstract), Mercier and
| Sperber argue that reason 'is not geared to solitary use, to
| arriving at better beliefs and decisions on our own', but
| rather to 'help us justify our beliefs and actions to others,
| convince them through argumentation, and evaluate the
| justifications and arguments that others address to us'.
| Reason, they suggest, 'helps humans better exploit their
| uniquely rich social environment'.
|
| They resist the idea (popularized by Daniel Kahneman) that
| there is 'a contrast between intuition and reasoning as if
| these were two quite different forms of inference', proposing
| instead that 'reasoning is itself a kind of intuitive
| inference'. For them, reason as a cognitive mechanism is
| 'much more opportunistic and eclectic' than is implied by the
| common association with formal systems like logic. 'The main
| role of logic in reasoning, we suggest, may well be a
| rhetorical one: logic helps simplify and schematize intuitive
| arguments, highlighting and often exaggerating their force.'
|
| Their 'interactionist' perspective helps explain how
| illogical rhetoric can be so socially powerful; it is reason,
| 'a cognitive mechanism aimed at justifying oneself and
| convincing others', fulfilling its evolutionary social
| function.
|
| Highly recommended, if you're not already familiar.
| snthpy wrote:
| Thank you. That's exactly the idea and described much more
| eloquently. I probably heard it through the Sapolsky
| lecture from a sibling comment but that captures it
| exactly. Bookmarked.
| jwuphysics wrote:
| Incredible, well-documented work -- this is an amazing effort!
|
| Two things that caught my eye were (i) your loss curves and (ii)
| the assessment of dead latents. Our team also studied SAEs --
| trained to reconstruct dense embeddings of paper abstracts rather
| than individual tokens [1]. We observed a power-law scaling of
| the lower bound of loss curves, even when we varied the sparsity
| level and the dimensionality of the SAE latent space. We also
| were able to totally mitigate dead latents with an auxiliary
| loss, and we saw smooth sinusoidal patterns throughout training
| iterations. Not sure if these were due to the specific
| application we performed (over paper abstracts embeddings) or if
| they represent more general phenomena.
|
| [1] https://arxiv.org/abs/2408.00657
| PaulPauls wrote:
| I'm very happy you appreciate it - particularly the
| documentation. Writing the documentation was much harder for me
| than writing the code so I'm happy it is appreciated. I
| furthermore downloaded your paper and will read through it
| tomorrow morning - thank you for sharing it!
| Eliezer wrote:
| This seems like decent alignment-positive work on a glance,
| though I haven't checked full details yet. I probably can't make
| it happen, but how much would someone need to pay you to make up
| your time, expense, and risk?
| monkeycantype wrote:
| Thank you for posting this PaulPauls,
|
| can I please ask a wacky question that I have about mech.interp.
| ?
|
| we know that when we use a metric to measure humans, we game the
| metric, I wonder about future ai, gaming mech.interp.
|
| for simplicity let me propose a 2d matrix to encode tokens, and
| hope my understanding of neural networks is good enough for this
| to be a valid question I understand that higher dimensions have a
| lot more 'space', so accidental collisions are easily avoided,
| but 2d makes my example easier to explain.
|
| if we had the following tokens mapped in 2d space
| Apple 1a Pear 1b Donkey 2a Horse 2b
|
| it becomes impossible to understand if the neurons 1,2,a,b, all
| excited represents apple and horse or donkey and pear?
|
| I wonder if a future, vastly more competent AI overseeing its own
| training might use this form of ambiguity as means to create a
| model in which the weights are selected so the potential for
| encoding collisions exist, so that it is possible to deceive an
| mech.int. observer as to the true thoughts of the model,
| essentially enabling the ai to think in euphemisms?
| Majromax wrote:
| What you propose is a harder AI safety scenario.
|
| You don't need a 'vastly more competent AI overseeing its own
| training' to elicit this potential problem, just a malicious AI
| researcher, looking for (e.g.) a model that's racist but that
| does not have any interperable activation patterns that
| identifiably correspond to racism.
|
| The work here on this Show HN suggests that this kind of
| adversarial training might just barely be possible for a
| sufficiently-funded individual, and it seems like novel results
| would be very interesting.
| batterylake wrote:
| This is incredible!
|
| PaulPauls, how would you like us to cite your work?
| PaulPauls wrote:
| Thank you very much!
|
| I included a section at the bottom that provides a sample
| bibtex citation. I didn't expect this much attention so I
| didn't even bother with a License but I'll include a MIT
| license later today and release 0.2.1
| samstevens wrote:
| I'm really excited to see some more open SAE work! The
| engineering effort is non trivial and I'm going to check out your
| dataloading code tomorrow. You might be interested in an
| currently in-progress project of mine to train SAEs on vision
| models: https://github.com/samuelstevens/saev
| Carrentt wrote:
| Fantastic work! I absolutely love all the documentation.
| vivekkalyan wrote:
| This is great work! Mechanistic interpretability has tons of use
| cases, it's great to see open research in that field.
|
| You mentioned you spent your own time and money on it, would you
| be willing to share how much you spent? It would help others who
| might be considering independent research.
| PaulPauls wrote:
| Thank you, I too am a big believer and enjoyer of open
| research. The actual code has clarity that complex research
| papers were never able to convey to me as well as the actual
| code could.
|
| Regarding the cost I would probably sum it up to round about
| ~2.5k USD for just the actual execution cost. Development cost
| would've probably doubled that sum if I wouldn't already have a
| GPU workstation for experiments at home that I take for
| granted. That cost is made up of:
|
| * ~400 USD for ~2 months of storage and traffic of 7.4 TB (3.2
| TB of raw, 3.2 TB of preprocessed training data) on a GCP
| standard bucket
|
| * ~100 USD for Anthropic claude requests for experimenting with
| the right system prompt and test runs and the actual final
| execution
|
| * The other ~2k USD were used to rent 8x Nvidia RTX4090's
| together with a 5TB SSD from runpod.io for various stages of
| the experiments. For the actual SAE training I rented the node
| for 8 days straight and I would probably allocate an additional
| ~3-4 days of runtime just to experiments to determine the best
| Hyperparameters for training.
| enterthedragon wrote:
| This is amazing, the documentation is very well organized
| imranhou wrote:
| Coming from a layman's perspective, a genuine question regarding:
| "Implements SAE training with auxiliary loss to prevent and
| revive dead latents, and gradient projection to stabilize
| training dynamics".
|
| I struggle to understand this phrase "to prevent and revive ",
| perhaps this is simple speak to those that understand the subject
| of SAEs, but it feels a bit self contradictory to me, could
| anyone elaborate?
| versteegen wrote:
| A latent that is never active and hence doesn't (seem to)
| represent anything. A loss term to reduce the occurrence of
| that, and if it does happen, push it back to being active
| sometimes.
| imranhou wrote:
| So basically preventing dead latents from occurring and
| whenever they do occur to possibly reviving them through the
| use of auxiliary loss term in the loss function? Thanks btw
| PaulPauls wrote:
| Just bad wording from me, trying to combine too much
| information in 1 sentence. The auxiliary loss is supposed to
| prevent dead latents from occuring in the first place -
| therefore "prevent dead latents" - and it is also supposed to
| revive the latents that are already dead - therefore "revive
| dead latents".
|
| Now that I review that sentence again I see that I used 2 verbs
| on the same subject that could be interpreted differently
| depending on the verb. Me culpa. I hope you still gained some
| insights into it =)
| imranhou wrote:
| Thanks for sharing! It is certainly interesting to me who is
| not in the mainstream, I'm sure your intended audience
| understood what you were saying.
| S-Kaenel wrote:
| Amazing research!!
| moconnor wrote:
| Find a latent for the Golden Gate bridge and put a Golden Gate
| Llama 3.2 on HuggingFace. This will get even more attention and
| love, more so if you include link to a space to chat with it!
|
| Also, you didn't ask for suggestions but putting some interesting
| results / visualizations at the top of the README is a very good
| idea.
| yangwang92 wrote:
| Nice! You did what I wanted. Have you tried to train SAE for
| vision encoder and language encoder? I am working on this idea.
| May we work together, let me initial an issue.
___________________________________________________________________
(page generated 2024-11-22 23:00 UTC)