https://github.com/FortAI-Hub/rwkv.f90 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} FortAI-Hub / rwkv.f90 Public * Notifications * Fork 1 * Star 21 Port of the RWKV-LM model in Fortran License MIT license 21 stars 1 fork Star Notifications * Code * Issues 1 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights FortAI-Hub/rwkv.f90 This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 3 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/F] Use Git or checkout with SVN using the web URL. [gh repo clone FortAI] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @matteo-grella matteo-grella Exclude -march=native as it consistently demonstrates a negative im... ... 62e7aa5 Jul 18, 2023 Exclude `-march=native` as it consistently demonstrates a negative im... ...pact TODO: Expand investigation to other architectures, as tests have been limited to Apple Silicon so far. 62e7aa5 Git stats * 28 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time model_converter Remove layer rescaling July 15, 2023 19:49 models Initial commit after 24 hours of work July 13, 2023 17:01 src Perform an initial forward to warm up the model July 18, 2023 11:20 .gitignore Added build/ directory and .pth model files to .gitignore July 17, 2023 17:51 CMakeLists.txt Exclude -march=native as it consistently demonstrates a negative im... July 18, 2023 11:52 LICENSE Initial commit after 24 hours of work July 13, 2023 17:01 README.md Update README.md July 16, 2023 16:49 View code [ ] RWKV.f90 Prerequisites Step-by-Step Guide 1. Downloading the Model 2. Model Conversion 3. Build the Project 3.1 Choosing a BLAS Implementation 4. Run the Project References README.md RWKV.f90 RWKV.f90 is a port of the original RWKV-LM, an open-source large language model initially developed in Python, into Fortran. Renowned for its robust capabilities in scientific and engineering computations, the primary focus of this project is to explore the potential of Fortran within the realm of Artificial Intelligence. Please note that this is an ongoing project and we welcome contributions. Prerequisites Before you start, ensure that you have the following installed on your system: * cmake * gfortran * go Step-by-Step Guide This section provides a detailed guide to setting up and running the RWKV.f90 project. The process includes downloading a model, moving the model to the correct directory, converting the model, building the project, and running it. 1. Downloading the Model Download the rwkv-4-world model of your choice from the following options available at huggingface.co: * 0.1B * 0.4B * 1.5B * 3B - Recommended for a balance of performance and computational efficiency Once you have downloaded your chosen model, move it to the 'models' directory in the project's root folder. 2. Model Conversion After the desired model is in the correct location, convert it using the model converter. Make sure to adjust the filename according to the model you have downloaded. Run the following commands in your terminal: cd model_converter go run model_converter.go ../models/.pth Replace with the actual name of the downloaded model file. 3. Build the Project With the model conversion done, you can now build the project. Navigate to the root directory of the project and run the following commands: mkdir build cd build cmake .. make 3.1 Choosing a BLAS Implementation The underlying BLAS (Basic Linear Algebra Subprograms) implementation that's used for matrix multiplication operations (matmul) can be chosen by providing the -DBLAS_LIBRARY flag during the CMake configuration step. Three options are available: * DBLAS_LIBRARY=OpenBLAS: This utilizes the OpenBLAS library, which is an open-source implementation of the BLAS API. Specify in the CMakeList.txt the directory where OpenBLAS library is located. * DBLAS_LIBRARY=Accelerate: This leverages the Apple Accelerate Framework, which is highly optimized for Apple hardware. * DBLAS_LIBRARY=Fortran or leaving it unset: This defaults to using the Fortran's intrinsic matmul function, which allows the Fortran compiler to handle matrix operations without an external library. To select an option, include the relevant flag when running cmake. For example, to use the OpenBLAS library, you would run: cmake -DBLAS_LIBRARY=OpenBLAS .. By not specifying a -DBLAS_LIBRARY flag or setting it to Fortran, the build will rely on the intrinsic matmul function provided by the compiler (resulting in a slow execution). 4. Run the Project Once the project is built, you can now run it. The following command also colors stderr outputs in red for better error visibility. Adjust the command as necessary, according to your downloaded and converted model name: ./rwkv ../models/rwkv_vocab_v20230424.csv ../models/ 2> >(while read line; do echo -e "\e[01;31m$line\e[0m" >&2; done) Replace with the name of your converted model file. That's all! You have successfully set up and run the rwkv.f90 project. If you encounter any issues, please raise them in the issue tracker. References The paper provides more details about the RWKV concept and its applications. It's recommended to read this paper for a better understanding of the project and its scientific background. @misc{peng2023rwkv, title={RWKV: Reinventing RNNs for the Transformer Era}, author={Bo Peng and Eric Alcaide and Quentin Anthony and Alon Albalak and Samuel Arcadinho and Huanqi Cao and Xin Cheng and Michael Chung and Matteo Grella and Kranthi Kiran GV and Xuzheng He and Haowen Hou and Przemyslaw Kazienko and Jan Kocon and Jiaming Kong and Bartlomiej Koptyra and Hayden Lau and Krishna Sri Ipsit Mantri and Ferdinand Mom and Atsushi Saito and Xiangru Tang and Bolun Wang and Johan S. Wind and Stansilaw Wozniak and Ruichong Zhang and Zhenyuan Zhang and Qihang Zhao and Peng Zhou and Jian Zhu and Rui-Jie Zhu}, year={2023}, eprint={2305.13048}, archivePrefix={arXiv}, primaryClass={cs.CL} } About Port of the RWKV-LM model in Fortran Resources Readme License MIT license Stars 21 stars Watchers 3 watching Forks 1 fork Report repository Releases No releases published Packages 0 No packages published Contributors 3 * @matteo-grella matteo-grella Matteo Grella * @marco-nicola marco-nicola Marco Nicola * @ekitahasegawa ekitahasegawa Ekita Hasegawa Languages * Fortran 73.3% * Go 23.4% * CMake 3.3% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.