[HN Gopher] Direct initialization of transformers using larger p...
       ___________________________________________________________________
        
       Direct initialization of transformers using larger pretrained ones
        
       Author : PaulHoule
       Score  : 24 points
       Date   : 2023-12-22 18:54 UTC (4 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | jacobsimon wrote:
       | This might sound like nonsense to someone who actually
       | understands neural networks, but I had a similar idea recently
       | that perhaps we could use LLMs or diffusion models to generate
       | the weights for a new network, e.g. if you think of a layer of a
       | network as an image where each pixel represents the strength of a
       | connection, then perhaps you could generate all the initial
       | weights for a specialized network from a more general/powerful
       | network. Curious if that's been tried or if there are fundamental
       | limitations that would prevent it.
        
         | turtles3 wrote:
         | I believe what you're looking for is Hypernetworks
         | 
         | https://arxiv.org/abs/1609.09106
        
           | jacobsimon wrote:
           | Very relevant thanks!
        
       | londons_explore wrote:
       | I would like to know if this approach is actually transferring
       | knowledge, or if random initialization is just not the optimal
       | starting place - for example, maybe having a different
       | distribution of weights is better.
       | 
       | A good test would be to apply this technique, but then train on a
       | totally different kind of data - for example take a text LLM,
       | apply this technique, but then train on audio/music data and see
       | if this technique reduces training time over random
       | initialization.
        
         | peterfirefly wrote:
         | Sounds like a good test.
         | 
         | We need something like this for "non-neural" things like
         | transformers and normalization layers:
         | 
         | https://proceedings.mlr.press/v157/skorski21a/skorski21a.pdf
        
       | DarmokJalad1701 wrote:
       | From Page 7:
       | 
       | > In the case of GPT-2 training, random initialization requires
       | 64x10^9 tokens to reach a perplexity of 12, while weight
       | subcloning accomplishes this in just 64x10^9 tokens, again
       | demonstrating a 4x training speedup
       | 
       | ... what?
        
         | mirekrusin wrote:
         | Looks like typo, from chart it seems to be around 20x10^9 for
         | weight subcloning (doesn't look like 16x10^9 <<which would be
         | 4x>> either).
        
       | mirekrusin wrote:
       | Instead of pruning least contributing neurons wouldn't it be
       | better to ie. nominate next lowest contributing neuron or most
       | similar one to merge them into one or redistribute its impact
       | onto others?
        
       ___________________________________________________________________
       (page generated 2023-12-22 23:00 UTC)