[HN Gopher] Modded-NanoGPT: NanoGPT (124M) quality in 3.25B tokens
       ___________________________________________________________________
        
       Modded-NanoGPT: NanoGPT (124M) quality in 3.25B tokens
        
       Author : ocean_moist
       Score  : 61 points
       Date   : 2024-10-12 20:29 UTC (2 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | gavindean90 wrote:
       | Seems like this is a modded NanoGPT not the original.
        
         | munchler wrote:
         | Yes. It's literally called "Modded-NanoGPT".
        
       | Scene_Cast2 wrote:
       | I wonder how much improvement is owed to which changes. I've also
       | never heard of "Muon - Momentum Orthogonalized by Newton-schulz"
       | being used.
       | 
       | EDIT: there's a bit more info on his twitter -
       | https://x.com/kellerjordan0
       | 
       | It looks like he created this optimizer. Works on 2D matrices
       | only.
        
       | whiplash451 wrote:
       | Cool work. No license?
        
       ___________________________________________________________________
       (page generated 2024-10-12 23:00 UTC)