https://github.com/jzhang38/TinyLlama Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} jzhang38 / TinyLlama Public * Notifications * Fork 11 * Star 452 License Apache-2.0 license 452 stars 11 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights jzhang38/TinyLlama This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/j] Use Git or checkout with SVN using the web URL. [gh repo clone jzhang] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @jzhang38 jzhang38 Merge pull request #1 from Green-Sky/patch-1 ... 0fcf9b6 Sep 4, 2023 Merge pull request #1 from Green-Sky/patch-1 0fcf9b6 Git stats * 20 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github first commit September 4, 2023 12:03 lit_gpt first commit September 4, 2023 12:03 pretrain first commit September 4, 2023 12:03 scripts first commit September 4, 2023 12:03 .gitignore first commit September 4, 2023 12:03 LICENSE first commit September 4, 2023 12:03 PRETRAIN.md first commit September 4, 2023 12:03 README.md fix live tracking link September 4, 2023 18:26 README_zh-CN.md Update README_zh-CN.md September 4, 2023 23:40 requirements.txt first commit September 4, 2023 12:03 View code [ ] TinyLlama-1.1B Releases Schedule Potential Usecase Training Details Blazingly Fast Getting Started TODO Acknowledgements Citation Frequently Asked Questions 1. Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? 2. What does "saturation" mean? Star History README.md TinyLlama-1.1B English | Zhong Wen The TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. With some proper optimization, we can achieve this within a span of "just" 90 days using 16 A100-40G GPUs . The training has started on 2023-09-01. [TinyLlama_logo] We adopted exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be plugged and played in many open-source projects built upon Llama. Besides, TinyLlama is compact with only 1.1B parameters. This compactness allows it to cater to a multitude of applications demanding a restricted computation and memory footprint. Releases Schedule We will be rolling out intermediate checkpoints following the below schedule. We also include some baseline models for comparison. Date HF Checkpoint Tokens Step HellaSwag Acc_norm Baseline StableLM-Alpha-3B 800B -- 38.31 Baseline Pythia-1B-intermediate-step-50k-105b 105B 50k 42.04 Baseline Pythia-1B 300B 143k 47.16 2023-09-04 TinyLlama-1.1B-intermediate-step-50k-105b 105B 50k 43.50 2023-09-16 -- 500B -- -- 2023-10-01 -- 1T -- -- 2023-10-16 -- 1.5T -- -- 2023-10-31 -- 2T -- -- 2023-11-15 -- 2.5T -- -- 2023-12-01 -- 3T -- -- It can be observed that TinyLlama has so far progressed well . Meanwhile, you can track the live cross entropy loss here. Potential Usecase Tiny but strong language models are useful for many applications. Here are some potential usecases: * Assisting speculative decoding of larger models. (See this tutorial by Andrej Karpathy) * Deployment on edge devices with restricted memory and computational capacities, for functionalities like real-time machine translation without an internet connection (the 4bit-quantized TinyLlama-1.1B's weight only takes up 550MB RAM). * Enabling real-time dialogue generation in video games. Moreover, our code can be a reference for enthusiasts keen on pretraining language models under 5 billion parameters without diving too early into Megatron-LM. Training Details Below are some details of our training setup: Setting Description Parameters 1.1B Attention Variant Grouped Query Attention Model Size Layers: 22, Heads: 32, Query Groups: 4, Embedding Size: 2048, Intermediate Size (Swiglu): 5632 Sequence Length 2048 Batch Size 2 million tokens (2048 * 1024) Learning Rate 4e-4 Learning Rate Cosine with 2000 warmup steps Schedule Training Data Slimpajama & Starcoderdata Data Excluded GitHub subset of Slimpajama; Sampled all Preprocessing code from Starcoderdata Combined Dataset Around 950B tokens Size Total Tokens 3 trillion (slightly more than 3 epochs/1430k During Training steps) Natural Language 7:3 to Code Ratio Hardware 16 A100-40G GPUs Blazingly Fast Our codebase supports the following features: * multi-gpu and multi-node distributed training with FSDP. * flash attention 2. * fused layernorm. * fused swiglu. * fused cross entropy loss . * fused rotary positional embedding. Thanks to those optimizations, we achieve a throughput of 24k tokens per second per A100-40G GPU, which translates to 56% model flops utilization without activation checkpointing (We expect the MFU to be even higher on A100-80G). It means you can train a chinchilla-optimal TinyLlama (1.1B param, 22B tokens) in 32 hours with 8 A100. Those optimizations also greatly reduce the memory footprint, allowing us to stuff our 1.1B model into 40GB GPU RAM and train with a per-gpu batch size of 16k tokens. You can also pretrain TinyLlama on 3090/ 4090 GPUs with a smaller per-gpu batch size. Below is a comparison of the training speed of our codebase with that of Pythia and MPT. Model A100 GPU hours taken on 300B tokens TinyLlama-1.1B 3456 Pythia-1.0B 4830 MPT-1.3B 7920 The Pythia number comes from their paper. The MPT number comes from here, in which they say MPT-1.3B " was trained on 440 A100-40GBs for about half a day" on 200B tokens. The fact that TinyLlama is a relatively small model with grouped query attention means it is also fast during inference. Below are some throughputs that we measure: Framework Device Settings Throughput (tokens/ sec) Llama.cpp Mac M2 16GB batch_size=1; 4-bit 71.8 RAM inference vLLM A40 GPU batch_size=100, n=10 7094.5 Getting Started Please refer to PRETRAIN.md for instructions on how to pretrain TinyLlama. TODO This project is still under active development. We are a really small team. Community feedback and contributions are highly appreciated. Here are some things we plan to work on: * [ ] Add scripts for pretraining on other datasets. * [ ] Sequence length extrapolation. * [ ] Test out speculative decoding for Llama-2-7B. * [ ] Test the throughput on RTX 3090/4090. * [ ] Add fine-tuning scripts. * [ ] Properly evaluate the model on downstream tasks. * [ ] A demo running on mobile phones. * [ ] Explore retrieval-augmentation. Acknowledgements This repository is built upon lit-gpt and flash-attention. Be sure to explore this fantastic open-source project if it's new to you! @online{lit-gpt, author = {Lightning AI}, title = {Lit-GPT}, url = {https://github.com/Lightning-AI/lit-gpt}, year = {2023}, } @article{dao2023flashattention2, title ={Flash{A}ttention-2: Faster Attention with Better Parallelism and Work Partitioning}, author ={Dao, Tri}, year ={2023} } Citation This project is currently contributed by Peiyuan Zhang, Guangtao Zeng , Tianduo Wang and Wei Lu. If you find our work valuable, please cite: @online{tinyllama, author = {Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu}, title = {TinyLlama}, url = {https://github.com/jzhang38/TinyLlama}, year = {2023}, month = {Sep}, } Frequently Asked Questions 1. Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law? The training loss curve of Llama 2 Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if the loss curve does not go down eventually, we can still study the phenomenon of saturation and learn something from it. 2. What does "saturation" mean? Figure 10 of the Pythia paper The figure from the Pythia paper displays the LAMBADA accuracy plotted against the total training tokens (300B). The term "saturation" pertains specifically to the 70M and 160M models. Notably, even the 410M model does not saturate with 300B tokens, as it continues to show an increasing trend, similar to the trend of larger models. Star History Star History Chart About No description, website, or topics provided. Resources Readme License Apache-2.0 license Activity Stars 452 stars Watchers 24 watching Forks 11 forks Report repository Releases No releases published Packages 0 No packages published Contributors 3 * @jzhang38 jzhang38 Zhang Peiyuan * @ChaosCodes ChaosCodes * @Green-Sky Green-Sky Erik Scholz Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.