[HN Gopher] SmolGPT: A minimal PyTorch implementation for traini...
       ___________________________________________________________________
        
       SmolGPT: A minimal PyTorch implementation for training a small LLM
       from scratch
        
       Author : amrrs
       Score  : 146 points
       Date   : 2025-01-29 18:09 UTC (4 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | numba888 wrote:
       | github has a bunch of them for years, the most known from Andrej
       | Karpathy:
       | 
       | https://github.com/karpathy/nanoGPT
       | 
       | some other have MoE implemented.
        
         | benreesman wrote:
         | nanoGPT is awesome (and I highly recommend his videos on it),
         | but it's closer to a direct reproduction of GPT-2, so it's cool
         | to have a really clean implementation of some newer ideas.
        
           | Nimitz14 wrote:
           | nanoGPT contains some new ideas.
           | https://github.com/karpathy/minGPT is more plain
        
         | syassami wrote:
         | Personal fave: https://github.com/karpathy/llama2.c
        
       | ks2048 wrote:
       | So, this has nothing to do with "SmolLM" - a set of models (with
       | data, training recipes, etc) released by HuggingFace?
       | https://huggingface.co/blog/smollm
        
       | sitkack wrote:
       | Neat, I love projects like these.
       | 
       | The next level down is to do it directly in numpy.
       | 
       | And then from there, write a minimal numpy work-a-like to support
       | the model above.
       | 
       | You start with a working system using the most powerful
       | abstractions. Then you iteratively remove abstractions, lowering
       | your solution, then when you get low enough but still riding on
       | an external abstraction, you rewrite that, but ONLY to support
       | the layers above you.
       | 
       | Following the above pattern, you can bootstrap yourself to have
       | full system understanding. This is not unlike RL+distillation
       | that human persons do learn complex topics.
        
         | lagrange77 wrote:
         | > but still riding on an external abstraction, you rewrite
         | that, but ONLY to support the layers above you.
         | 
         | i don't get it. Why do i stop before stripping all
         | abstractions?
        
           | byteknight wrote:
           | Where do you get that? He is postulating the external
           | abstraction you are using has more features than you use. He
           | is saying implement only the parts you use.
        
             | lagrange77 wrote:
             | > Where do you get that?
             | 
             | From "when you get low enough but still riding on an
             | external abstraction".
             | 
             | > He is saying implement only the parts you use.
             | 
             | Thanks.
        
               | sitkack wrote:
               | Correct, I should proof read my posts.
        
         | tomrod wrote:
         | Likewise. And your comment reminded me of real programmers*
         | 
         | * https://xkcd.com/378/
        
       | nostradumbasp wrote:
       | Cute! Keep making fun things.
        
       | the_real_cher wrote:
       | Any body have any good readings they read and liked to kind of
       | understand what is going on with how this works?
        
         | fragmede wrote:
         | https://spreadsheets-are-all-you-need.ai/
        
       | Lerc wrote:
       | The example story is interesting.
       | 
       | I have made my own implementation from scratch with my own multi-
       | channel tokeniser, each channel gets its own embedding table
       | 32768, 256,256, 64, and 4. Which are summed along with the
       | position encoding.
       | 
       | Yet with all of those differences, my stories have Lily as a
       | protagonist often enough that I thought I had a bug somewhere.
       | 
       | Might have to check tinystories for name distribution.
       | 
       | Most questionable output from mine so far:
       | 
       | "one day, a naughty man and a little boy went to the park place
       | to find some new things."
        
       | spidermonkey23 wrote:
       | Is there anything that can run locally on mobile in temrux
        
       | antirez wrote:
       | No cpu / mps support to train on Macs, apparently.
        
         | leopoldj wrote:
         | You should be able to make a few small changes to support
         | "mps".
         | 
         | In TrainingConfig set the device to "mps". The run training.
         | 
         | In sample.py modify parse_args() and add support for mps as a
         | possible value for the --device argument.
        
           | antirez wrote:
           | Thanks! I'll try. I didn't bother believing that if this was
           | developed heavily on CUDA, it was likely going to use kernels
           | that were missing in MPS.
        
       | brap wrote:
       | It's interesting that technology so transformative is only a few
       | hundred lines of code (excluding underlying frameworks and such).
       | 
       | How big would you guess state of the art models are, in terms of
       | lines of code?
        
         | miki123211 wrote:
         | Llama2 inference can be implemented in 900-ish lines of
         | dependency-free C89, with no code golfing[1]. More modern
         | architectures (at least the dense, non-MoE models) aren't that
         | much more complicated.
         | 
         | That code is CPU only, uses float32 everywhere and doesn't do
         | any optimizations, so it's not realistically usable for models
         | beyond 100m params, but that's how much it takes to run the
         | core algorithm.
         | 
         | [1] https://github.com/karpathy/llama2.c
        
       ___________________________________________________________________
       (page generated 2025-01-29 23:00 UTC)