https://github.com/s-casci/tinyzero Skip to content Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} s-casci / tinyzero Public * Notifications * Fork 2 * Star 102 Easily train AlphaZero-like agents on any environment you want! License MIT license 102 stars 2 forks Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights s-casci/tinyzero This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/s] Use Git or checkout with SVN using the web URL. [gh repo clone s-casc] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @s-casci s-casci updated README ... 3aad008 Dec 20, 2023 updated README 3aad008 Git stats * 31 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time connect2 added eval December 19, 2023 21:36 tictactoe1d added eval December 19, 2023 21:36 tictactoe2d renamed convnet December 19, 2023 21:39 LICENSE Create LICENSE December 20, 2023 19:11 README.md updated README December 20, 2023 19:16 agents.py added GPU support December 20, 2023 11:57 mcts.py first commit December 17, 2023 17:12 models.py added GPU support December 20, 2023 11:57 play.py added eval December 19, 2023 21:36 replay_buffer.py switched to torch December 18, 2023 12:44 requirements.txt added requirements.txt December 20, 2023 12:23 ruff.toml first commit December 17, 2023 17:12 tinyzero.png added README December 17, 2023 19:55 View code tinyzero Usage Add an environment Add a model Add a new agent README.md tinyzero [tinyzero] Easily train AlphaZero-like agents on any environment you want! Usage Make sure you have Python >= 3.8 intalled. After that, run pip install requirements.txt to install the necessary dependencies. Then, to train an agent on one of the existing environments, run: python3 tictactoe2d/train.py where tictactoe2d is the name of the environment you want to train on. Inside the train script, you can change some parameters, such as the number of episodes, the number of simulations and enable wandb logging. Similarly, to evaluate the trained agent run: python3 tictactoe2d/eval.py Add an environment To add a new environment, you can follow the game.py files in every existing examples. The environment you add should implement the following methods: * reset(): resets the environment to its initial state * step(action): takes an action and modifies the state of the environment accordingly * get_legal_actions(): returns a list of legal actions * undo_last_action(): cancels the last action taken * to_observation(): returns the current state of the environment as an observation (a numpy array) to be used as input to the model * get_result(): returns the result of the game (for example, it might be 1 if the first player won, -1 if the second player won, 0 if it's a draw, and None if the game is not over yet) * get_first_person_result(): returns the result of the game from the perspective of the current player (for example, it might be 1 if the current player won, -1 if the opponent won, 0 if it's a draw, and None if the game is not over yet) * swap_result(result): swaps the result of the game (for example, if the result is 1, it should become -1, and vice versa). It's needed to cover all of the possible game types (single player, two players, zero-sum, non-zero-sum, etc.) Add a model To add a new model, you can follow the existing examples in models.py. The model you add should implement the following methods: * __call__: takes as input an observation and returns a value and a policy * value_forward(observation): takes as input an observation and returns a value * policy_forward(observation): takes as input an observation and returns a distribution over the actions (the policy) The latter two methods are used to speed up the MCTS. The AlphaZero agent computes the policy loss as the Kulback-Leibler divergence between the distribution produced by the model and the one given by the MCTS. Therefore, the policy returned by the __call__ method should be logaritmic. On the other hand, the policy returned by the policy_forward method should represent a probability distribution. Add a new agent Thanks to the way the value and policy functions are interpreted by the search tree, it's possible to use or train any agent that implements them. To add a new agent, you can follow the existing example in agents.py. The agent you add should implement the following methods: * value_fn(game): takes as input a game and returns a value (float) * policy_fn(game): takes as input a game and returns a policy (Numpy array) Any other method is not directly used by the MCTS, so it's optional and depends on the agent you want to implement. For example, the AlphaZeroAgent implements a train_step method to train the model after each episode. About Easily train AlphaZero-like agents on any environment you want! Topics reinforcement-learning mcts alphazero Resources Readme License MIT license Activity Stars 102 stars Watchers 1 watching Forks 2 forks Report repository Releases No releases published Packages 0 No packages published Languages * Python 100.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.