[HN Gopher] A Baby GPT
       ___________________________________________________________________
        
       A Baby GPT
        
       Author : tosh
       Score  : 73 points
       Date   : 2023-04-09 20:17 UTC (2 hours ago)
        
 (HTM) web link (twitter.com)
 (TXT) w3m dump (twitter.com)
        
       | marcodiego wrote:
       | > This is a baby GPT with two tokens 0/1 and context length of 3,
       | viewing it as a finite state markov chain.
       | 
       | What is actually the difference between this and a "pure" Markov
       | chain? Is GPT simply a Markov chain?
       | 
       | EDIT: Better explained here:
       | https://colab.research.google.com/drive/1SiF0KZJp75rUeetKOWq...
        
         | karpathy wrote:
         | Yes but in the same way as saying that computers are just a
         | Markov chain.
        
         | KRAKRISMOTT wrote:
         | The mathematical modeling of most NLP tasks are Markovian.
         | (More specifically, they tend to be HMMs, Hidden Markov
         | Models). One of the popular "solutions" for Markovian processes
         | is the Metropolis algorithm which you might find relevant. Note
         | in statistical learning the mathematical formulation generally
         | operates as part of the loss function, the actual "model" is
         | usually a neural network (in the context of deep or variational
         | learning).
        
         | transcoderx wrote:
         | Since every state has a set of probabilities for the next
         | state, I guess you could call it that. It would be very
         | misleading, though.
        
           | [deleted]
        
         | Imnimo wrote:
         | Even in the case of a full size GPT, one could equivalently
         | write out a lookup table for every possible permutation of
         | tokens in the context window and the resulting output
         | distribution. The table would of course be impossibly large.
        
       | [deleted]
        
       | drexlspivey wrote:
       | Should have named it picoGPT
        
       | Jknitgirl wrote:
       | [flagged]
        
       ___________________________________________________________________
       (page generated 2023-04-09 23:00 UTC)