[HN Gopher] Whats better: Neural nets wider with less layers or ...
       ___________________________________________________________________
        
       Whats better: Neural nets wider with less layers or thinner with
       more layers
        
       Author : vatsadev
       Score  : 24 points
       Date   : 2024-06-27 05:55 UTC (1 days ago)
        
 (HTM) web link (vatsadev.github.io)
 (TXT) w3m dump (vatsadev.github.io)
        
       | Grimblewald wrote:
       | It makes sense that lessons one learns from working with dense
       | networks, applies to transformers as well since these are at
       | their core still just dense networks.
       | 
       | The way I grew to understand the relationship, and I am happy to
       | discuss this / receive feedback, is that a layer's width
       | determines how much that layer can memorize while network depth
       | determines the complexity of abstraction possible for the network
       | to learn.
       | 
       | So a wide enough layer can simply remember everything while a
       | deep enough network will be able to, through abstraction,
       | recreate memories of everything using a simplification of the
       | input.
       | 
       | Ideally, you want a balance of the two, since you don't want to
       | rely on memory alone, as this doesn't tend to generalize well,
       | nor do you want to deal with the fantasy outputs from something
       | relying too heavily on abstraction, as this is not likely to be
       | reliable.
        
         | supple-mints wrote:
         | Is it harder to train the wider network or the deeper network
         | all else equal?
        
         | MattPalmer1086 wrote:
         | That makes a lot of sense, thanks for the explanation.
        
       | esafak wrote:
       | Classic paper: _Wide & Deep Learning for Recommender Systems_.
       | 
       | https://paperswithcode.com/method/wide-deep
        
       | chessgecko wrote:
       | *edit neverming below this is a character level model that
       | probably has a small vocab so it wouldn't make a massive
       | difference
       | 
       | Is this taking into account the parameters in the embedding and
       | the output ffn? Because normally when models are really small and
       | the vocab is large they can account for an extremely large number
       | of parameters and would explain why the optimal number of layers
       | here is unusually small.
       | 
       | I suspect it isn't being taken into account because doubling the
       | embedding and cutting the number of layers in half only holds the
       | parameters constant if you forget the embedding and output, but
       | id need to see more details on the config (mainly the vocab size
       | he used) to confirm.
        
       ___________________________________________________________________
       (page generated 2024-06-28 23:02 UTC)