https://magazine.sebastianraschka.com/p/building-llms-from-the-ground-up

[https]

Ahead of AI

SubscribeSign in
Share this post
[https]

Building LLMs from the Ground Up: A 3-hour Coding Workshop

magazine.sebastianraschka.com
Copy link
Facebook
Email
Note
Other

Building LLMs from the Ground Up: A 3-hour Coding Workshop

[https]
Sebastian Raschka, PhD
Aug 31, 2024
177
Share this post
[https]

Building LLMs from the Ground Up: A 3-hour Coding Workshop

magazine.sebastianraschka.com
Copy link
Facebook
Email
Note
Other
 
5
 
Share

If you'd like to spend a few hours this weekend to dive into Large
Language Models (LLMs) and understand how they work, I've prepared a
3-hour coding workshop presentation on implementing, training, and
using LLMs.

Below, you'll find a table of contents to get an idea of what this
video covers (the video itself has clickable chapter marks, allowing
you to jump directly to topics of interest):

0:00 - Workshop overview

2:17 - Part 1: Intro to LLMs

9:14 - Workshop materials

10:48 - Part 2: Understanding LLM input data

23:25 - A simple tokenizer class

41:03 - Part 3: Coding an LLM architecture

45:01 - GPT-2 and Llama 2

1:07:11 - Part 4: Pretraining

1:29:37 - Part 5.1: Loading pretrained weights

1:45:12 - Part 5.2: Pretrained weights via LitGPT

1:53:09 - Part 6.1: Instruction finetuning

2:08:21 - Part 6.2: Instruction finetuning via LitGPT

02:26:45 - Part 6.3: Benchmark evaluation

02:36:55 - Part 6.4: Evaluating conversational performance

02:42:40 - Conclusion

It's a slight departure from my usual text-based content, but the
last time I did this a few months ago, it was so well-received that I
thought it might be nice to do another one!

Happy viewing!

References

 1. Build an LLM from Scratch book

 2. Build an LLM from Scratch GitHub repository

 3. GitHub repository with workshop code

 4. Lightning Studio for this workshop

 5. LitGPT GitHub repository

177
Share this post
[https]

Building LLMs from the Ground Up: A 3-hour Coding Workshop

magazine.sebastianraschka.com
Copy link
Facebook
Email
Note
Other
 
5
 
Share
5 Comments
[https]
[                    ]
         
        WvG
        Aug 31Liked by Sebastian Raschka, PhD

        THANK YOU so much for doing this! I have bought the early
        release of your book via MEAP and it is fantastic. Highly
[https] recommended for everybody who wants to be hands on and really
        get a deeper understanding and appreciation regarding LLMs.

        Expand full comment
        Reply
        Share

1 reply by Sebastian Raschka, PhD

         
        harish pillay
        harish's Newsletter
        13 hrs agoLiked by Sebastian Raschka, PhD

        Thanks, Sebastian for publishing the tutorial and book. I've
[https] placed an order for the book - not sure how long it will take
        to get to me here in Singapore. Will revisit the tutorial
        when the book arrives. Thanks again.

        Expand full comment
        Reply
        Share

3 more comments...
Top
Latest
Discussions

No posts

Ready for more?

[                    ]
Subscribe
(c) 2024 Sebastian Raschka
Privacy [?] Terms [?] Collection notice
 Start WritingGet the app
Substack is the home for great culture
Share
Copy link
Facebook
Email
Note
Other
This site requires JavaScript to run correctly. Please turn on
JavaScript or unblock scripts