https://github.com/karpathy/LLM101n

Skip to content

Navigation Menu

Toggle navigation
 
Sign in

  * Product
      +  
        Actions
        Automate any workflow
      +  
        Packages
        Host and manage packages
      +  
        Security
        Find and fix vulnerabilities
      +  
        Codespaces
        Instant dev environments
      +  
        GitHub Copilot
        Write better code with AI
      +  
        Code review
        Manage code changes
      +  
        Issues
        Plan and track work
      +  
        Discussions
        Collaborate outside of code
    Explore
      + All features
      + Documentation
      + GitHub Skills
      + Blog
  * Solutions
    For
      + Enterprise
      + Teams
      + Startups
      + Education
    By Solution
      + CI/CD & Automation
      + DevOps
      + DevSecOps
    Resources
      + Learning Pathways
      + White papers, Ebooks, Webinars
      + Customer Stories
      + Partners
  * Open Source
      +  
        GitHub Sponsors
        Fund open source developers
      +  
        The ReadME Project
        GitHub community articles
    Repositories
      + Topics
      + Trending
      + Collections
  * Enterprise
      +  
        Enterprise platform
        AI-powered developer platform
    Available add-ons
      +  
        Advanced Security
        Enterprise-grade security features
      +  
        GitHub Copilot
        Enterprise-grade AI features
      +  
        Premium Support
        Enterprise-grade 24/7 support
  * Pricing

Search or jump to...

Search code, repositories, users, issues, pull requests...

Search
[                    ]
Clear

Search syntax tips

Provide feedback

We read every piece of feedback, and take your input very seriously.

[                    ] [ ] Include my email address so I can be
contacted
Cancel Submit feedback

Saved searches

Use saved searches to filter your results more quickly

Name [                    ] 
Query [                    ]

To see all available qualifiers, see our documentation.

Cancel Create saved search
Sign in
Sign up
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session. You switched accounts on another tab or window. Reload
to refresh your session. Dismiss alert
{{ message }}
karpathy / LLM101n Public

  * Notifications You must be signed in to change notification
    settings
  * Fork 30
  * Star 836
  * 

LLM101n: Let's build a Storyteller

836 stars 30 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

  * Code
  * Issues 1
  * Pull requests 1
  * Discussions
  * Actions
  * Security
  * Insights

Additional navigation options

  * Code
  * Issues
  * Pull requests
  * Discussions
  * Actions
  * Security
  * Insights

karpathy/LLM101n

This commit does not belong to any branch on this repository, and may
belong to a fork outside of the repository.
 master
BranchesTags
  
Go to file
Code

Folders and files

   Name        Name     Last commit message     Last commit date
Latest commit

 

History

1 Commit
 
README.md   README.md                        
llm101n.jpg llm101n.jpg                      
View all files

Repository files navigation

  * README

LLM101n: Let's build a Storyteller

 

LLM101n header image

    What I cannot create, I do not understand. -Richard Feynman

In this course we will build a Storyteller AI Large Language Model
(LLM). Hand in hand, you'll be able create, refine and illustrate
little stories with the AI. We are going to build everything
end-to-end from basics to a functioning web app similar to ChatGPT,
from scratch in Python, C and CUDA, and with minimal computer science
prerequisits. By the end you should have a relatively deep
understanding of AI, LLMs, and deep learning more generally.

Syllabus

  * Chapter 01 Bigram Language Model (language modeling)
  * Chapter 02 Micrograd (machine learning, backpropagation)
  * Chapter 03 N-gram model (multi-layer perceptron, matmul, gelu)
  * Chapter 04 Attention (attention, softmax, positional encoder)
  * Chapter 05 Transformer (transformer, residual, layernorm, GPT-2)
  * Chapter 06 Tokenization (minBPE, byte pair encoding)
  * Chapter 07 Optimization (initialization, optimization, AdamW)
  * Chapter 08 Need for Speed I: Device (device, CPU, GPU, ...)
  * Chapter 09 Need for Speed II: Precision (mixed precision
    training, fp16, bf16, fp8, ...)
  * Chapter 10 Need for Speed III: Distributed (distributed
    optimization, DDP, ZeRO)
  * Chapter 11 Datasets (datasets, data loading, synthetic data
    generation)
  * Chapter 12 Inference I: kv-cache (kv-cache)
  * Chapter 13 Inference II: Quantization (quantization)
  * Chapter 14 Finetuning I: SFT (supervised finetuning SFT, PEFT,
    LoRA, chat)
  * Chapter 15 Finetuning II: RL (reinforcement learning, RLHF, PPO,
    DPO)
  * Chapter 16 Deployment (API, web app)
  * Chapter 17 Multimodal (VQVAE, diffusion transformer)

Appendix

Further topics to work into the progression above:

  * Programming languages: Assembly, C, Python
  * Data types: Integer, Float, String (ASCII, Unicode, UTF-8)
  * Tensor: shapes, views, strides, contiguous, ...
  * Deep Learning frameowrks: PyTorch, JAX
  * Neural Net Architecture: GPT (1,2,3,4), Llama (RoPE, RMSNorm,
    GQA), MoE, ...
  * Multimodal: Images, Audio, Video, VQVAE, VQGAN, diffusion

About

LLM101n: Let's build a Storyteller

Resources

Readme
Activity

Stars

836 stars

Watchers

81 watching

Forks

30 forks
Report repository

Footer

 (c) 2024 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact
  * Manage cookies
  * Do not share my personal information

You can't perform that action at this time.