https://github.com/sebosp/s2protocol-rs Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} sebosp / s2protocol-rs Public * Notifications * Fork 2 * Star 47 Starcraft 2 Protocol Replay Reader License MIT license 47 stars 2 forks Activity Star Notifications * Code * Issues 0 * Pull requests 1 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights sebosp/s2protocol-rs This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 3 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/s] Use Git or checkout with SVN using the web URL. [gh repo clone sebosp] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @sebosp sebosp Merge branch 'release/3.0.0' ... 755098f Oct 4, 2023 Merge branch 'release/3.0.0' 755098f Git stats * 212 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows Init CI February 13, 2023 19:50 assets Adding example of 89720 2v2-AI gameplay for testing April 9, 2023 13:13 jupyter_notebooks Updating docs October 4, 2023 20:50 src GameEvent Iterator easier to work with October 4, 2023 19:34 tests Starting to create protocol-agnostic unit trackers February 14, 2023 22:06 .gitignore Initial commit February 11, 2023 14:53 Cargo.toml Bump version October 4, 2023 20:56 LICENSE Initial commit February 11, 2023 14:53 README.md Updating docs October 4, 2023 20:54 requirements.txt More split, updating docs on IPCs and Iterators October 3, 2023 22:40 View code [ ] s2protocol-rs Motivation Consuming events Transition Iterators Interacting with polars Generating the IPC Arrow datasets Jupyter Notebooks polars-cli Status Current issues version compatibility. Generating protocol-specific code: JSON Sources README.md Crates.io Workflow Status s2protocol-rs A nom parser for the Starcraft 2 Protocol Replay format. Provides Iterators to transist a minimal state machine through the game loops Keeping track of unit groups, upgrades, targets, etc. Motivation The goal is to learn how to parse binary files format with nom and to learn how the Starcraft 2 Replay file is so incredibly small for the amount of information it packs. From the available data, analytics, visualizations and generative art can be created, for example by using * rerun : See the repo swarmy * lyon (PoC in progress in cooper) * yew cooper * eframe/egui: See repo eframes-c2 * bevyengine/bevy can be used to see: + An Enhanced Replay Minimap + Additional statistics. Consuming events Transition Iterators These are different ways to consume the events: * SC2EventIterator collects both TrackerEvents and GameEvents. Events are provided as they appear, be them Tracker or Game * TrackerEventIterator allows consuming only Tracker Events * GameEventIterator allows consuming only the Game Events Event changes transist a minimal state machine that updates: * names * positions * Attack events * etc. The iterators returns a tuple (SC2EventType, UnitChangeHint) The first item contains the event itself as it was deserialized from the SC2Replay. The second item hints the consumers of the iterator about the changes performed to the state machine due to this event. For example, units may have been deleted, added, changed position, etc. let source: PathBuf = PathBuf::new(); let res = s2protocol::state::SC2EventIterator::new(&source)?; for (event, change_hint) in res.into_iter() { println!("{},", serde_json::to_string(&event)?); } Interacting with polars Generating the IPC Arrow datasets In the directory ipcs/ one .ipc file will be created per implemented data type. The --source is the directory that contains the replay directory (Or a single file). Files are processed using parallel operations. For 3600 files (500 MBs) it takes 30 seconds to transform /split them. YMMV This is behind a feature flag arrow. $ mkdir ipcs/ $ cargo run --features arrow -r -- --source "/mnt/windows/Users/sebos/Documents/StarCraft II/Accounts/51504154/2-S2-1-8459957/Replays/Multiplayer/" --output ipcs/ write-arrow-ipc all 2023-10-04T18:53:36.030202Z INFO s2protocol::arrow: Processing Arrow write request 2023-10-04T18:53:36.441089Z INFO s2protocol::arrow: Found 3600 files 2023-10-04T18:53:36.441646Z INFO s2protocol::arrow: Processing InitData IPC write request 2023-10-04T18:53:38.515349Z INFO s2protocol::arrow: Loaded 3600 records 2023-10-04T18:53:38.575090Z INFO s2protocol::arrow: Processing Details IPC write request 2023-10-04T18:53:38.700572Z INFO s2protocol::arrow: Loaded 3600 records 2023-10-04T18:53:38.706659Z INFO s2protocol::arrow: Processing TrackerEvents IPC write request: Stats 2023-10-04T18:53:44.295524Z INFO s2protocol::arrow: Loaded 1722783 records 2023-10-04T18:53:44.515362Z INFO s2protocol::arrow: Processing TrackerEvents IPC write request: Upgrades 2023-10-04T18:53:49.963043Z INFO s2protocol::arrow: Loaded 292898 records 2023-10-04T18:53:50.036165Z INFO s2protocol::arrow: Processing TrackerEvents IPC write request: UnitBorn 2023-10-04T18:53:57.561082Z INFO s2protocol::arrow: Loaded 22754591 records 2023-10-04T18:54:00.502298Z INFO s2protocol::arrow: Processing TrackerEvents IPC write request: UnitDied 2023-10-04T18:54:07.387545Z INFO s2protocol::arrow: Loaded 16118808 records Total time: 33.654286961s Jupyter Notebooks $ virtualenv new venv $ source ./venv/bin/activate $ pip install -r requirements.txt $ jupyter lab Then open your browser and locate the Notebbooks, for example: Basic UnitBorn Queries All units born Most effective units across all games polars-cli $ cargo install polars-cli $ # List the max number of minerals that were lost in per map when the army was killed. echo "SELECT ext_fs_replay_file_name, MAX(minerals_lost_army) FROM read_ipc('/home/seb/git/s2protocol-rs/ipcs/stats.ipc') GROUP BY ext_fs_replay_file_name ORDER BY minerals_lost_army DESC;"|polars +-----------------------------------+--------------------+ | ext_fs_replay_file_name | minerals_lost_army | | --- | --- | | str | i32 | +-----------------------------------+--------------------+ | Heavy Artillery LE (349).SC2Repl... | 71362 | | Arctic Dream LE (398).SC2Replay | 59375 | | Nightscape LE (52).SC2Replay | 54846 | | ... | ... | | Emerald City LE (223).SC2Replay | 43450 | | Rhoskallian LE (101).SC2Replay | 41614 | | Fields of Death (345).SC2Replay | 41529 | | Rhoskallian LE (346).SC2Replay | 41425 | +-----------------------------------+--------------------+ Status * [*] Replay Tracker, Game Events and Chat Message Events for protocol75689 * [*] Replay Tracker, Game Events and Chat Message Events for protocol84643 * [*] Replay Tracker, Game Events and Chat Message Events for protocol87702 * [*] Replay Tracker, Game Events and Chat Message Events for protocol88500 * [*] Replay Tracker, Game Events and Chat Message Events for protocol89634 * [*] Replay Tracker, Game Events and Chat Message Events for protocol89720 * [*] Replay Tracker, Game Events and Chat Message Events for protocol90136 * [*] Replay Tracker, Game Events and Chat Message Events for protocol90779 * [*] Replay Tracker, Game Events and Chat Message Events for protocol90870 * [*] Parsing unit movement is done. * [*] Decoding the tag/recycle done to match Game Events. * [*] Game Events are parsed (tho some that seem irrelevant are skipped). * [*] Read the version and from the version call the correct module so that we can support multiple modules. * [*] Support for MPQ embedded file: replay.initData * [ ] Add the remaining Tracker/Game event types. * [ ] Support for MPQ embedded file: replay.gamemetadata.json * [ ] Support for MPQ embedded file: replay.attributes.events Current issues In the arrow file generation, sha256 digest is used to detect duplication/etc. This inflates the size of the rows. Even tho it's slightly less than long directory names. Perhaps using short rev-parse for sha256 may be better, find something like 7-characters unique combinations And use that instead of the long sha256 form. version compatibility. After a bit of testing, it seems most of the types are compatible between versions, so only when they differ would they make part of the protocol version. Since I started this exercise on protocol87702, all types would be relative to it. That is, most modules would re-use protocol87702 as much as possible. This explains why old-er versions such as 75689 would still reference 87702 as much as possible. The generator above thus would show example code S2ProtoResult created in favour of unwrapping/panic'ing. Generating protocol-specific code: The rust code for the protocols versions available were generated using: This would now be compared with ./src/versions/ protocol99999.template file and from there we can analyze what has changed. Notably, the number of bits used for the Chat Message is still miscalculated to 3 so it needs to be dismissed. mkdir src/versions/protocol89720/ RUST_LOG_SPAN_EVENTS=full RUST_LOG=debug cargo watch -i src/versions/protocol89720/mod.rs -x 'run -- --source ../s2protocol/json/protocol89720.json generate --output src/versions/protocol89720/mod.rs' # Add the new module to src/versions/mod.rs # Run rust format on the new src/versions/protocol87702/mod.rs file # cargo check, cargo build, etc # Additionally some code to transform from Protocol-Specific to Protocol-Agnostic was added, TODO: Add to generator.rs JSON Sources Blizzard/s2protocol repo About Starcraft 2 Protocol Replay Reader Topics arrow sc2replay nom sc2replaystats polars Resources Readme License MIT license Activity Stars 47 stars Watchers 1 watching Forks 2 forks Report repository Releases No releases published Packages 0 No packages published Used by 4 * @sebosp @sebosp / cooper * @sebosp @sebosp / swarmy Contributors 3 * @sebosp sebosp Seb Ospina * @DomShark DomShark DomShark * @szabgab szabgab Gabor Szabo Languages * Rust 54.9% * Jupyter Notebook 45.1% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.