https://github.com/certik/fastGPT Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} certik / fastGPT Public * Notifications * Fork 9 * Star 115 Fast GPT-2 inference written in Fortran License MIT license 115 stars 9 forks Star Notifications * Code * Issues 9 * Pull requests 6 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights certik/fastGPT This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 6 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/c] Use Git or checkout with SVN using the web URL. [gh repo clone certik] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @certik certik Merge pull request #46 from certik/bench1 ... 4e70c6a Mar 24, 2023 Merge pull request #46 from certik/bench1 Update benchmarks 4e70c6a Git stats * 79 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows CI: install gfortran on macOS March 5, 2023 23:20 ci Add a test for the basic prompt March 20, 2023 19:10 cmake FindOPENBLAS: give hints for CentOS/RHEL March 17, 2023 00:22 comparison Implement tokenizer in Fortran (#34) March 20, 2023 18:49 tests Implement a chatGPT driver March 21, 2023 15:13 .gitignore auto-ignore build directory March 17, 2023 00:24 CMakeLists.txt Implement a chatGPT driver March 21, 2023 15:13 LICENSE Initial Fortran implementation of GPT-2 March 5, 2023 18:20 README.md Update benchmarks March 23, 2023 19:00 build.sh build.sh: build out-of-source and in parallel March 17, 2023 00:10 chat.f90 Implement a chatGPT driver March 21, 2023 15:13 create_model.py Implement tokenizer in Fortran (#34) March 20, 2023 18:49 driver.f90 Implement a chatGPT driver March 21, 2023 15:13 environment.yml Add support for OpenBLAS March 6, 2023 23:52 gpt2.f90 Allow to stop on a given text March 21, 2023 15:12 input Implement tokenizer in Fortran (#34) March 20, 2023 18:49 linalg_accelerate.c Initial Fortran implementation of GPT-2 March 5, 2023 18:20 linalg_c.f90 Initial Fortran implementation of GPT-2 March 5, 2023 18:20 linalg_f.f90 Initial Fortran implementation of GPT-2 March 5, 2023 18:20 linalg_openblas.c Add support for OpenBLAS March 6, 2023 23:52 main.f90 Expose load_model more widely March 21, 2023 07:11 omp.f90 Add an omp module March 13, 2023 21:53 omp_dummy.f90 Add an omp module March 13, 2023 21:53 pt.py Add an option to turn off cache in PyTorch March 13, 2023 22:01 tokenizer.f90 Remove unused variables March 21, 2023 07:13 View code fastGPT Build and Run Example Output Chat interface BLAS Implementation Benchmarks TODO README.md fastGPT The progression of GPT-2 codes from the original to "minimal", "nano" and "pico": * openai/gpt-2 * karpathy/minGPT * karpathy/nanoGPT * jaymody/picoGPT fastGPT is very similar to picoGPT (very small and readable), but it is also fast (see the Benchmarks section below). The speed and readability is achieved by using Fortran. I wrote a blog post introducing fastGPT. fastGPT features: * Fast? * Training code? * Batch inference? * top-p sampling? top-k? temperature? categorical sampling?! greedy? * Readable? * Small? A quick breakdown of each of the files: * gpt2.f90: the actual GPT-2 model and a decoder * main.f90: the main driver * create_model.py: downloads the TensorFlow model and converts to our own format (model.dat) * encode_input.py: encodes the text input into tokens (input file for gpt2) * Matmul implementations + linalg_f.f90 native Fortran + linalg_c.f90, linalg_accelerate.c macOS Accelerate Framework * pt.py: a reference script to run PyTorch (returns the same answer) Build and Run Install prerequisites: mamba env create -f environment.yml conda activate fastgpt Configure and build: FC=gfortran cmake . make Create the model.dat file from a given GPT-2 model. Supported sizes (and the corresponding names to be used in pt.py, and the approximate download size): "124M" (gpt2, 0.5GB), "355M" (gpt-medium, 1.5GB), "774M" (gpt-large, 3GB), "1558M" (gpt-xl, 6GB). This will download the model and cache it for subsequent runs: python create_model.py --models_dir "models" --model_size "124M" Now you can modify the input file to change the input string and set other parameters. Run (requires model.dat and input in the current directory): ./gpt2 Example Output The above ./gpt2 command prints on Apple M1 Max: $ ./gpt2 Loading the model... done. Time: 0.111s Model parameters: n_vocab = 50257 n_ctx = 1024 n_embd = 768 n_layer = 12 n_head = 12 Input text Alan Turing theorized that computers would one day become very powerful, but even he could not imagine Encoding: tokenizing input text into tokens (currently slow)... done. Time: 0.074s Input parameters: n_seq = 19 n_tokens_to_generate = 20 Input tokens: 36235 39141 18765 1143 326 9061 561 530 1110 1716 845 3665 11 475 772 339 714 407 5967 Decoded input as text: Alan Turing theorized that computers would one day become very powerful, but even he could not imagine Running model... how they would be able to do so. "I think that the most important thing is done. Time: 0.304s (1.01x) Output tokens: 703 484 561 307 1498 284 466 523 13 198 198 1 40 892 326 262 749 1593 1517 318 Decoded output as text: how they would be able to do so. "I think that the most important thing is Chat interface Here is an example chat using the largest 1558M model: $ ./chat Your name is fastGPT and you are an AI bot. The user will ask you questions and you answer in a nice, truthful, short way. User: What is the capital of Czechia? fastGPT: Prague. User: How many legs does a dog have? fastGPT: Four. User: What color does the sky have? fastGPT: Blue. User: What can you type a document on? fastGPT: A typewriter. User: What can you drive in? fastGPT: A car. User: What can you fly in? fastGPT: A plane. User: What continent is Germany in? fastGPT: Europe. User: When did Second World War start? fastGPT: 1939. User: When did it end? fastGPT: 1945. User: When did the U.S. enter the Second World War? fastGPT: 1941. User: When did the First World War start? fastGPT: 1914. User: When did it end? fastGPT: 1918. User: When did the Mexican-American war start? fastGPT: 1846. User: When did it end? fastGPT: 1848. User: What color is snow? fastGPT: White. User: What color do plants usually have? fastGPT: Green. User: What is your name? fastGPT: fastGPT. BLAS Implementation You can choose which BLAS implementation to use for matmul using: * -DFASTGPT_BLAS=OpenBLAS: Use OpenBLAS * -DFASTGPT_BLAS=Accelerate: Use the macOS Accelerate Framework * -DFASTGPT_BLAS=Fortran: Use the default Fortran's intrinsic matmul Benchmarks On Apple M1 Max, inference of the above input file (20 tokens): 1 core 2 cores 4 cores 8 cores fastGPT (Accelerate, fast_tanh) 0.288s fastGPT (Accelerate) 0.299s PyTorch (Accelerate) 0.346s fastGPT (OpenBLAS) 0.837s 0.514s 0.341s 0.339s PyTorch (OpenBLAS) 0.873s 0.539s 0.386s 0.392s fastGPT (Accelerate, no cache) 0.717s picoGPT (Accelerate, no cache) 0.765s PyTorch (Accelerate, no cache) 0.787s fastGPT (OpenBLAS, no cache) 2.343s 1.603s 1.209s 1.018s PyTorch (OpenBLAS, no cache) 2.356s 1.520s 1.104s 0.997s picoGPT (OpenBLAS, no cache) 2.427s 1.645s 1.272s 1.081s Total run (includes loading the model and Python imports): fastGPT (Accelerate, fast_tanh): 0.401s picoGPT (8 cores): 3.445s PyTorch (OpenBLAS, 4 cores): 4.867s TODO * [ ] Parallelization: + [ ] Over heads: #2 + [ ] MPI: #5 * [ ] Other sampling methods: #8 * [ ] Batching: #7 * [*] Improve the UI: + [*] Implement the input tokenizer in Fortran: #1 + [*] Show the words as they are generated: #6 About Fast GPT-2 inference written in Fortran Topics fortran high-performance gpt-2 Resources Readme License MIT license Stars 115 stars Watchers 4 watching Forks 9 forks Report repository Releases No releases published Packages 0 No packages published Contributors 2 * @certik certik Ondrej Certik * @scivision scivision Languages * Fortran 58.1% * Python 28.3% * CMake 10.3% * C 2.2% * Shell 1.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.