[HN Gopher] Show HN: A nom parser for the Starcraft 2 Protocol R...
___________________________________________________________________
Show HN: A nom parser for the Starcraft 2 Protocol Replay format
Been having a lot of fun reading an SC2Replay collection through
nom parsers, serializing into Arrow files so that pola.rs can read
them and perform data analysis with jupyter lab, plotly or interact
with SQL operations, etc. Looking for feedback and ideas on what to
progress on. For example, "through history, are my timings getting
better?". etc. Also would love to have ideas on what libraries to
use to perform forecasting.
Author : sebosp
Score : 96 points
Date : 2023-10-04 19:03 UTC (3 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| Rodeoclash wrote:
| This is very cool, I'd love to chat more about it.
|
| For context, I have an open source video player designed for
| esports coaches. The main feature is that you can load in
| multiple video streams at once at synchronise them together.
| Mainly for FPS games like Valorant / Apex Legends
| (https://www.vodon.gg/ if you want to explore it).
|
| I'm starting to get access to some streams of data from the games
| via coaches that use the tool. My very naive approach was simply
| to load game events into the video timeline (so you could easily
| skip forwards to deaths, kills etc) but I hadn't thought about
| loading this into data analysis tools.
|
| The game events themselves for Valorant seem like they'd be
| enough to almost construct an online replay from them as well,
| which could compliment the recorded gameplay (i.e. construct a
| dynamic map of where everybody is that could be brought up on
| screen).
|
| It's a very cool space, if you'd like to chat more my email is in
| my profile.
| Nuzzerino wrote:
| I'm curious to see how well this works with Direct Strike (custom
| game mode for SC2)
| sebosp wrote:
| I only had one replay with direct strike in it's name but I
| can't remember watching it, I thought it was just one more
| map... I know it was failing initially to decode but in the end
| I fixed a couple of bugs and then it was working with this
| file. Could you explain what is significant go direct strike?
| How is it different to a normal game?
| Nuzzerino wrote:
| It has an active competitive scene, lobbies fill up in
| seconds sometimes. There are also tools such as dsstats that
| attempt to parse the replays already, but are limited by
| technology (maybe your tool can help with the limitations).
|
| If you search Direct Strike Starcraft on youtube you should
| be able to see the gameplay itself.
| tornato7 wrote:
| This is a super cool project! It would be especially interesting
| to use on pro-level games, maybe you could even sell services to
| pros who want to get better. Like being able to say "your
| baneling run-bys are most cost-effective with 7 banelings around
| the ten minute mark" etc.
|
| With a relatively larger dataset you could come up with some real
| interesting statistics on individual players performance, use of
| certain units, the success of various strategies and build
| orders, etc.
|
| It would be real fun to try to predict the outcome of a game
| based on the first 3 minutes or something.
| sebosp wrote:
| Indeed, another thing is that imho, casters and you tubers are
| data scientists that have the indexed dB in their heads, they
| say "at second X in this map Y you would have Z upgrade" and
| they talk about efficiency of mining, but I always wanted to
| see these things as statistics visually, like, how much faster
| or slower is player X in this game compared to the standard?
| How does the meta evolve? Or worse... If I check your replays,
| and you never ever scout this area, I could proxy there. You
| could find the build that in this map you never prepare for
| because let's be honest we are much more predictable than what
| we'd like to admit... I hope this project unlocks these kind of
| possibilities
| reaperman wrote:
| https://sc2replaystats.com is one service already doing that,
| at least to some degree. Some features are locked behind a
| subscription payment. There's definitely room for improvement
| -- but some of what you're looking for is "unsolved"...optimal
| strategies are unknown, and the best strategy for a
| professional player may not be the best strategy for you. You'd
| likely want to practice strategies which won't get you top-100
| in the world, but will work better with lower APM and lower
| overall "game sense".
| sebosp wrote:
| Is there a way to get a replay pack with order of millions of
| games (maybe more?) ? I'd like that to be able to create
| meaningful Stats, proper data analysis, training, etc...
| Wouldn't mind paying a bit for that... So far for the
| exploratory data analysis I've done I only have my own games
| and 2 or 3 replays I've been sent that don't work...
| reaperman wrote:
| The only groups I know of with millions of games are
| sc2replaystats and Blizzard themselves.
|
| SC2replaystats had 4,440,656 replays uploaded in 2020 and
| 3,390,117 in 2019 (the only years I can quickly find data
| for). Searching and downloading replays is now paywalled,
| probably to limit scraping.
|
| > Wouldn't mind paying a bit for that...
|
| Google probably paid Blizzard a good amount of money to
| help them develop AlphaStar and in the process got access
| to all the replays. But I doubt you'd be offering enough
| money to get executive-level attention for such a deal.
| Sc2replaystats might be more receptive to selling you a
| randomly selected subset of their total data, but you'd
| still have to offer enough cash to be clearly worth the
| time of going through the negotiation process. I doubt
| they'd sell you all of their replays, as that's essentially
| the business' moat.
|
| Perhaps you could buy the sc2replaystats business though.
|
| (Yes, I know none of these suggestions are practical for
| the vast majority of interested hackers)
|
| ----------
|
| There are smaller free replay packs available with a quick
| Google search. I'm sure if you looked around or asked
| around either Harstem's SC2 discord server or the SC2AI
| discord server or SC2 subreddit, someone would be able to
| point you to a free vault offering on the order of 10,000's
| of replays.
|
| However, the game is undergoing a period of large updates
| right now which are dramatically changing the details of
| the gameplay ... so really you'd want as many as possible
| starting from the most recent "balance patch" which went
| live just 6 days ago[0]. It will take the SC2 community
| quite a long time to establish known "best practices" for
| the current balance patch, and it's highly likely that it
| will be significantly modified again soon -- the most
| recent patch seems to be politically fragile at all levels
| of the game. Casual to professional players all have valid
| constructive criticism of it, so more changes soon are
| essentially inevitable.
|
| 0: https://news.blizzard.com/en-
| us/starcraft2/24009150/starcraf...
| jsphweid wrote:
| I'm messing around with a project now to featurize game states
| at arbitrary points and predict the winner with the idea of
| creating some sort of "eval bar" like in chess. Supply is a
| pretty good indicator but it's not always helpful. Definitely
| not in broodwar.
|
| If it's quite good it could be useful as a tool for people
| trying to get better (although various tooling and custom maps
| have been around for these types of purposes since the game
| came out and most people even pros seem to just grind ladder
| for practice).
|
| I was also hoping it could be used to analyze a replay and
| indicate how "back and forth" the game was, or if it was more
| one-sided, etc. Or if the top 100 pros were part of the
| featurization, then you could potentially determine how the
| odds change given a certain game state depending on who the pro
| is.
| comagoosie wrote:
| Thanks for sharing, very inspiring. I love Rust for parsing video
| game replays / save files. I've authored a Rocket League replay
| parser (boxcars) and an entire suite of web visualizations (via
| Webassembly) for EU4 called pdx.tools https://pdx.tools
|
| It's not easy to work with proprietary formats, but they've both
| become pretty popular, so I would 100% recommend sinking more
| time into this project as long as it scratches your itch. Gamers
| are always looking for more stats and deeper insights
| sebosp wrote:
| Have you seen rerun.io? Also web assembly, sharing again the
| link from above
| https://sebosp.github.io/swarmy/public/0.5.1/2023-GSL-S1-RO1...
| Maybe you can use it too in your project! The guys from rerun
| are amazing and super helpful, had a call with them to discuss
| questions and possible uses too
| comagoosie wrote:
| No joke, rerun seems perfect for visualizing a game like
| rocket league. Thank you for the recommendation!
| [deleted]
| munk-a wrote:
| It'd be great to use a non-Blizzard property for this analysis
| since that company is outstandingly despicable. Plenty of other
| RTSes exist and I know that AoE2 has an extremely strong data
| analytics scene.
| 0cf8612b2e1e wrote:
| Where would I go to see some of these analyses?
| munk-a wrote:
| A few places but I'd suggest starting with AOE stats[1].
|
| 1. https://aoestats.io/
| jsphweid wrote:
| I've also been impressed with the amount of data in SC2 replay
| files given the size, although I'm more interested in Broodwar
| (SC1) as a game. If I'm not mistaken Broodwar replay files
| contain a lot less information. They mainly contain player
| actions, i.e. you don't even know when a marine died -- the game
| has to literally be replayed to get that information.
| ktpsns wrote:
| The Broodwar replays were really cursed -- I remember that
| hilarious (perfectly reproducable) bugs of quickly replaying a
| game which resulted in a completely different replay then the
| actual game, apparently because the replay missed to integrate
| all replay information correctly and ended up in a different
| game state.
| icza wrote:
| Most of game corruption wasn't the result of faulty replay
| save / generation. Most of game corruption was the result of
| replays only recording user actions, and when the game engine
| was patched and rules were changed, re-playing the same user
| action could go wrong: imagine the cost of Probe getting
| increased from 50 minerals to 60. You load a replay that
| contains a "Build Probe" command when the user had 50
| minerals. In the new game engine no probe will be built as
| the player does not have enough minerals. If this new probe
| was used to build other buildings, those also will not be
| built in the new game engine. Butterfly effect.
| sebosp wrote:
| This brings another question, if I watch an old replay,
| does it mean it has to have the versioned game engine
| stored in the computer? Does it mean with every update it
| gets increasingly bigger? Or is there a number of supported
| versions you could replay? I only started playing like 3
| years ago...
| [deleted]
| reaperman wrote:
| SC2 patches to an older version if you load an old
| version. Then patches back to current version when you
| want to play again. It requires the game to restart --
| and while SC2 has a feature where you can watch a live
| replay with your friends (by all joining the same game
| which is hosted as type "replay") -- that "watch replay
| with others" feature only works with replays which were
| created in the latest version because only the latest
| version of the client can connect to the multiplayer
| servers.
|
| I'm not sure if the older patches are hot-downloaded on-
| demand when you load an old replay, I think they actually
| might be -- but I don't know.
|
| You can also "resume" a replay from any point and you and
| a friend can take over the game from that point. I think
| Harstem (both a professional player and a big streamer/
| YouTuber) has a few YouTube videos where he explores a
| famous game where one pro player "had clearly lost and
| had absolutely no chance to win" but the losing player
| stayed in the game for a really, really long time and
| people were talking about whether and to what degree that
| was "unsportsmanlike" (similar to not resigning in a
| clearly-lost chess game). Harstem would take the losing
| side at the point in time that was generally agreed to be
| "lost" and some of his viewers of various ability levels
| and IIRC he beat all of them.
| sebosp wrote:
| Wow that is interesting that it patches back and forth,
| super cool! How did you find out about this? I wonder now
| and then if one can "consume" the events on a live game
| somehow, like connect as a bot or sniff the events before
| they are encrypted through the network... Also curious
| about the drone kill counter mod/overlay that Harstem
| uses, do you know of the name of that? I can't figure it
| out but would love to enrich it with stats from the data
| analysis
| reaperman wrote:
| > How did you find out about this?
|
| I studied SC2 altogether far, far, too much from
| 2019-2022. Had a goal to get into masters, played tens of
| thousands of games, watched multiple YouTube videos every
| day, memorized and practiced dozens of build orders, but
| still got stuck at my peak in low diamond.
|
| When you're playing that much and reviewing replays, you
| start wondering why you can't watch old replays with
| people coaching you and someone eventually ends up
| explaining the system to you.
| reaperman wrote:
| > sniff the events before they are encrypted through the
| network
|
| Uhh, this is confusing as written. But there are cheating
| hacks which allow you to see extra data that is sent to
| your client in-game, such as the entire map and your's
| and your opponents production queues. And while I'm not
| sure if this is what you were asking: "sniffing" events
| from other clients which haven't been sent to the
| blizzard server is not possible, because they're
| delivered to your client via the blizzard server (it's
| not P2P).
|
| > Also curious about the drone kill counter mod/overlay
| that Harstem uses, do you know of the name of that?
|
| It's a combination of two things: 1) A customized UI
| layout for replays called "GameHeart" and 2) a custom SC2
| game mod called "GameHeart" -- you can use the UI layout
| for ladder games which weren't created as custom games
| with the GameHeart mod, but for that replay you'll miss
| out on features like the drone kill counter and income
| graph vs. time for which the data is generated by the
| game mod.
| beebeepka wrote:
| Exactly. Divergence is immediate and you wasted your high
| waiting for something to happen, only to discover that
| replay was generated by another game version. There was no
| youtube at the time and downloading replays was the only
| option and the good players weren't exactly thrilled to
| share them.
| rcoveson wrote:
| > imagine the cost of Probe getting increased from 50
| minerals to 60
|
| Yes, please.
|
| - T, Z
| jsphweid wrote:
| Artosis alt account?! :P
| marginalia_nu wrote:
| 60 mins, 15 gas and a forge.
| sebosp wrote:
| Btw such move instructions I visualize here with the amazing
| rerun.io https://sebosp.github.io/swarmy/public/0.5.1/2023-GS
| L-S1-RO1...
| sebosp wrote:
| I fantasize if training an AI to watch all the replays of a
| map on YouTube can create a repro of the state machine logic
| that moves the units through the game, or to find the parts
| of a map that cannot be walked upon. In the replay it emits
| an event when the user wants a unit to move to x, y but then
| it has to path its way there, and I cannot predict (without
| their logic) how it would move there
| sebosp wrote:
| Is there a general name to the way the data is
| compressed/serialized in SC2 Replays? Yesterday somebody was
| talking about other ways to serialize/pack data with borsh but
| not sure it's at all similar... Is the SC1 broodwar replay
| format similar with byte-alignment and bit "packing"? BTW I
| call the parsers "bitpacked" or "byte-aligned", but I'm not
| sure what the right name for them is...
| icza wrote:
| SC2 replays are MPQ files, which is a proprietary format
| created and used by Blizzard. It's an archive that may
| contain multiple files stored with different compression and
| optionally encrypted. I wrote a lib to parse MPQ files that
| embodies SC2Replays: https://github.com/icza/mpq. I also
| wrote an SC2 replay parser that is more or less a port of the
| official s2protocol: https://github.com/icza/s2prot
| sebosp wrote:
| I mean for example https://github.com/sebosp/s2protocol-
| rs/blob/755098fb86ab6b1... I hacked my way around the json
| protocol specification, an enum has n types, using log I
| can find the number of bits I need to read to uniquely
| identify each variant, that kind of serialization I wonder
| the name of.
| icza wrote:
| I'm not sure about an official name, I also just call it
| "bitpacked".
| benmanns wrote:
| I may be the only one not familiar, but nom refers to
| https://github.com/rust-bakery/nom which looks like a pretty
| handy way to parse binary data in Rust.
| brundolf wrote:
| It's a pretty handy way to parse any sequence of things in
| Rust! I'm using it to write a compiler for a toy language and
| it's been a delight
| IshKebab wrote:
| I would consider Chumsky for a programming language. It has
| recoverable parsing which is important for IDE support.
| flaghacker wrote:
| Could you share the repo? I've been curious what nontrivial
| parsers look like woth nom, I always lose the overview pretty
| quickly.
| TimWolla wrote:
| I'm using nom to parse a simple access control DSL in
| https://github.com/wbbaddons/Tims-
| PackageServer/blob/master/.... You can see how it looks in
| the tests of the linked file.
|
| The DSL supports comparison operators, '&&', '||' and
| nested expressions, while preventing mixing of '&&' and
| '||' without making precedence clear using parentheses.
| This DSL should fit 'non-trivial', but it still should be
| simple enough to easily understand it.
| epage wrote:
| For a very rough comparison of parsers, see
| https://github.com/rosetta-rs/parse-rosetta-rs
|
| (which both includes a full json parser for each parser)
| [deleted]
___________________________________________________________________
(page generated 2023-10-04 23:01 UTC)