[HN Gopher] Superposition, Memorization, and Double Descent
___________________________________________________________________
Superposition, Memorization, and Double Descent
Author : lamename
Score : 50 points
Date : 2023-01-05 17:19 UTC (5 hours ago)
(HTM) web link (transformer-circuits.pub)
(TXT) w3m dump (transformer-circuits.pub)
| dimatura wrote:
| The paper/article this refers to has more context. Lots of nice
| diagrams! https://transformer-
| circuits.pub/2022/toy_model/index.html.
| dragfuture wrote:
| I really love the work in this series, it feels like they are
| getting close to uncovering a periodic table of features/concepts
| that are common across all models.
| colah3 wrote:
| I'm glad you've enjoyed it! If you like the idea of a periodic
| table of features, you might like the Early Vision article from
| the original Distill circuits thread:
| https://distill.pub/2020/circuits/early-vision/
|
| We've had a much harder time isolating features in language
| models than vision models (especially early vision), so I think
| we have a clearer picture there. And it seems remarkably
| structured! My guess is that language models are just making
| very heavy use of superposition, which makes it much harder to
| tease apart the features and develop a similar picture.
| Although we did get a tiny bit of traction here:
| https://transformer-circuits.pub/2022/solu/index.html#sectio...
| dragfuture wrote:
| Thank you for sharing these, I will definitely check them
| out! The concept of superposition here is new to me, but the
| way its described in these articles makes it very clear. The
| connection to compressed sensing and the Johnson-
| Lindenstrauss lemma is fascinating. I am very intrigued by
| your toy model results, especially the mapping out of the
| double-descent phenomena. Trying to understand what is
| happening to the model in this transition region feels very
| exciting.
| colah3 wrote:
| I'm glad you've found it easy to follow!
|
| My best guess at the middle regime is that there are
| _empirical correlations between features_ due to the
| limited data. That is, even though the features are
| independent, there's some dataset size where by
| happenstance some features will start to look correlated,
| not just in the sense of a single feature, but something a
| bit more general. So then the model can represent something
| like a "principal component". But it's all an illusion due
| to the limited data and so it leads to terrible
| generalization!
|
| This isn't something I've dug into. The main reason I
| suspect it is that if you look at the start of the
| generalizing regime, you'll see that each feature has a few
| small features slightly embedded in the same direction as
| it. These seem to be features with slight empirical
| correlations. So that's suggestive about the transition
| regime. But this is all speculation -- there's lots we
| don't yet understand!
| dragfuture wrote:
| I should mention, I've been a reader of hackernews for years,
| but never bothered to create an account/comment. These
| articles piqued my interest enough to finally get me to
| register/comment :)
| colah3 wrote:
| Gosh, that's very flattering! Very touched by your
| interest.
___________________________________________________________________
(page generated 2023-01-05 23:02 UTC)