[HN Gopher] Gemma 3n Architectural Innovations - Speculation and...
       ___________________________________________________________________
        
       Gemma 3n Architectural Innovations - Speculation and poking around
       in the model
        
       Author : nolist_policy
       Score  : 12 points
       Date   : 2025-05-25 19:08 UTC (3 hours ago)
        
 (HTM) web link (old.reddit.com)
 (TXT) w3m dump (old.reddit.com)
        
       | impossiblefork wrote:
       | I think this is very interesting. Especially the per-layer
       | embedding things.
       | 
       | Having more than one embedding is something I've tried myself,
       | but not separate ones for each layer.
       | 
       | I'm guessing it's something like h_{l+1} =
       | MultiHeadSelfAttentionWithPositionEncodingBakedIn(MLP(h_l) +
       | embed_l(token_ids)). So it's probably really easy to implement on
       | toy problems to see if it works.
        
       ___________________________________________________________________
       (page generated 2025-05-25 23:02 UTC)