[HN Gopher] Preparing your game for deterministic netcode
___________________________________________________________________
Preparing your game for deterministic netcode
Author : manjana
Score : 109 points
Date : 2021-05-23 10:19 UTC (12 hours ago)
(HTM) web link (yal.cc)
(TXT) w3m dump (yal.cc)
| [deleted]
| wheybags wrote:
| Another really good article on the subject of deterministic
| lockstep multiplayer here:
| https://www.gafferongames.com/post/deterministic_lockstep/
|
| In particular, he talks a lot more about problems in floating
| point determinism, which was missing from the posted article IMO.
| It is possible to make it work without resorting to fixed point,
| however it is a big pain point.
| pastrami_panda wrote:
| Just wanted to add that Unity can be a good candidate for
| deterministic netcode, but you'd have to look toward their new
| packages to implement it. Photon Quantum is a high quality
| solution for deterministic netcode as an example.
| hesdeadjim wrote:
| Determinism isn't implemented (yet, maybe never) in DOTS
| NetCode. They have a lot bigger fish to fry right now.
|
| I built a small game with Quantum a couple years back when I
| took a "sabbatical" (shipped an indie game didn't know what to
| do next). I found it extremely limiting at the time, and
| despite determinism being the promise, I ran into constant
| desync issues.
|
| Things have probably changed in the last few years, but Photon
| isn't a large company and they have limited resources to
| improve it. YMMV
| golergka wrote:
| I've built a proof-of-concept Unity lockstep once:
| https://github.com/golergka/KingOfCastles
|
| But in real game development, there's too few situations where
| lockstep fits, as it makes the game pretty slow to react to
| player's actions. And rollback brings so much complexity to the
| whole game that it significantly impacts development costs.
| hesdeadjim wrote:
| If I hear the words "make the game multiplayer", I
| immediately 3-5x the development cost and difficulty.
| golergka wrote:
| Oh yes. I was just comparing this to different multiplayer
| architectures: 3-5x for client-server, easily x10 for
| rollback.
| [deleted]
| londons_explore wrote:
| I wrote all this stuff from scratch for a toy game engine I was
| making.
|
| The 'desync' stuff was surprisingly easy to resolve. Instead of
| trying to make all game state deterministic (a near impossible
| task in some languages, where for example even basic math
| functions don't have fully defined rounding charactistics), I
| went for another approach...
|
| Detect desyncs by periodically taking a ram snapshot of the whole
| game and calculating a CRC over it. It means your game state
| needs to be serializable, but you probably need that anyway so a
| new player can join the game mid game.
|
| If the CRC doesn't match, do a majority vote to find out which
| player is wrong, then do the rsync algorithm to sync differences
| between the players. In nearly all cases there will only be a few
| bytes of differences.
|
| There is no need for the game to halt or lag during any of this -
| the game can continue as normal. Players are likely seeing the
| exact same thing on the screen, because the differences between
| game states are probably an error of 0.00001 in some player
| position.
|
| When the corrected state has arrived, replay frames to bring it
| up to the current timestamp and switch out states. Job done. User
| sees no glitchyness. Developers don't have to worry about tiny
| bits of nondeterministic behaviour in libraries out of their
| control. Everyone's happy.
|
| Make sure it doesn't happen _every_ frame though, or you 'll
| waste a lot of CPU.
| nightpool wrote:
| Wait, so you're saying that a majority of players can override
| any other player's _entire RAM_? That sounds like an amazing
| paradise for hackers and griefers :D
| londons_explore wrote:
| Well in the case of my game engine, all the game logic was
| untrusted anyway - in fact, in some games the actual game
| mechanic involved live-editing the code of the running game
| to try and give yourself an advantage - without ever causing
| a syntax error and while others are also live-editing the
| same code. Any keystroke causing a syntax error or infinite
| loop would be an insta-kick for the player who made that
| keystroke.
| vlovich123 wrote:
| If you're frightened by that, you may be shocked to realize
| your browser is running unprotected code from random people.
| x4e wrote:
| Running untrusted code in a sandboxed virtual machine is
| quite different to on a native CPU
| jayd16 wrote:
| It's actually very protected, no?
| vlovich123 wrote:
| Still basically an infinitely larger attack surface than
| what OP describes which was fancy wording for some basic
| IPC primitives to synchronize some game state.
| jayd16 wrote:
| Depending on the game the error is often much larger.
| Especially when you have things like free player movement and
| obstructions. A little error can become a big error if one
| client sees a player move around a wall and the other see's
| them hit the wall and stop.
| aliswe wrote:
| I was a contributor to the old Playstation 1 title, 3D RTS game
| Warzone 2100, now FOSS.
|
| I was well on the way to implement a "Drive Mode" when I came
| across several hurdles relating to the net code.
|
| First I was on the path of implementing a pure FPS like game mode
| where any unit could be controlled, be it tanks, cyborgs,
| turrets, towers, bunkers, artillery, airplanes, boats, even
| walls! (Although they obviously couldnt move)
|
| The first prototypes were awesome, really inspiring and the
| community gave lots of brainstorming.
|
| We would be making battle royales, one on ones, elite single
| missions, awesome multiplayer coops, you name it.
|
| Then, I found out that the netcode only allows for move orders,
| go forward, turn left, speed 100, etc. Seemed manageable. So far
| so good.
|
| But wait. What about aircraft? Ouch, thats important though. Now
| they just move with WSAD, and you just stop midair.
|
| After that, I found out we couldnt simply click somewhere to
| shoot. Netcode only allowed to fire with intent to hit a specific
| unit. Hmmm ... Ok .. so turret cant rotate except when attacking.
|
| And, dont click to shoot, but "click to attack".
|
| I was initially really enthusiastic over reinstating this awesome
| game mode, but these restrictions totally killed it
| unfortunately.
|
| I guess the bottom line is that yes robust deterministic netcode
| is good and all, but it kinda locks in your game to what it
| currently is designed to do and fundamental changes to that will
| sometimes be overwhelmingly difficult if not impossible.
|
| PR on Github: https://github.com/Warzone2100/warzone2100/pull/829
|
| My initial prototype, no relation to the netcode, point to shoot,
| camera follows turret: https://youtu.be/itsd29NebGE
|
| My second prototype, no camera following aim, click to attack
| (iirc): https://youtu.be/ODBmOakkqWk
| golergka wrote:
| It's not only about "whats is currently designed". If you
| choose lockstep, then after you get the player's input, you
| have to wait at least a couple of roundtrips before she's going
| to get a reaction, which is easily in 100-200ms. For RTS it's
| OK, but I don't think you could make a convincing action game
| with this limitation.
| LeftTriangle wrote:
| > This has to serialize/deserialize the entire game state
| (everything that affects gameplay) into some format that can be
| later read from - conventionally, binary serialization, but you
| can technically do whatever you want so long as it's fast enough
| (can execute in <10% of your game frame time).
|
| This is only true if you use impure data structures. If you use
| persistent data structures you get this "for free" (the cost, of
| course, is worse asymptotics on mutable arrays and also
| potentially more GC pressure).
|
| You just keep a copy of your old game states around as long as
| you want, which doesn't cost anything except a few extra bytes
| that can't be reclaimed by the GC for a bit.
| hesdeadjim wrote:
| If you want to not hate your life building a real-time
| multiplayer game, fully embrace data oriented design.
|
| I took the gamble and picked Unity DOTS and NetCode for our
| current MOBA-esque game (not other packages, we bolt regular
| Unity visuals on much like the V in MVVM). Outside the headaches
| of using preview tech, it has been amazing what we have been able
| to do compared to regular OOP hell.
|
| By not having hidden game state lurking in classes, lambdas or
| coroutines, we have been able to hide much of the networking
| layer as an encapsulated set of glue systems that shuttle _just_
| enough data across the wire that the view-layer can present the
| game to the user. Movement and other prediction has been much
| easier too, because the game state is just data and not
| intertwined with behavior, I just run most of the same algorithms
| locally as I do on the server -- AI, steering, etc.
|
| This technique still requires a lot of foresight and challenging
| architecture decisions of course. I forced our core gameplay
| logic to not know anything about the visuals, which has made some
| things harder. However when it came time to make a headless build
| with no graphics, it was as simple as flipping a switch. No
| changes in our gameplay code.
|
| I have a principal engineer from Sea of Thieves on my team, and
| he has also breathed huge sighs of relief working on this
| project. Unreal is the complete antithesis to this design
| pattern, and is full of hidden state, magic behavior, and
| heisenbugs.
| karpierz wrote:
| Tangentially related, do you have any advice for moving into
| indie game development from a regular software engineering job
| at a big tech company?
| hesdeadjim wrote:
| I'd suggest searching for this question on reddit's
| /r/gamedev to hear a lot of perspectives.
|
| Mine? Don't. By that I mean, and assuming you haven't
| already, spend a few years making small games from start to
| finish. Pick games to copy, don't even hold yourself to
| having a unique spin on things. The goal being to 1) practice
| for the real thing and 2) decide if you actually want to make
| games or just fantasize about doing so. The latter isn't a
| judgement, I've almost left games twice because of the
| emotional burnout.
|
| For a point of reference, I spent two years getting up at 6am
| twice a week with a friend to gamejam and practice. All in
| prep to go indie. This was even after 6 years making games
| and leading two.
|
| The result? We almost failed completely. If I hadn't decided
| to say fuck it and stick the game on PS4 for fun, we would
| have grossed a grand total of $100k for a year of work. The
| more depressing fact? $100k is considered a significant
| success for an indie game on Steam.
|
| Happy to provide more anecdotal experience around this if you
| like.
| karpierz wrote:
| Thanks for the grounding take; I'd appreciate hearing more
| about your experience if you wouldn't mind. Either here or
| offline if you'd rather not share publicly.
| QuantumYeti wrote:
| Take my cynical pov with a pinch of salt... I left (indie?)
| gamedev to get a normal job, and have been much better off
| both mentally and financially.
|
| In addition to what hesdeadjim said, I'd also like to add
| don't blindly accept the common gamedev maxims. For example,
| "Make the game YOU want to play" is repeated as good advice,
| but I haven't found it to be very useful.
|
| First off, if I'm working as the sole developer and have
| total control over design decisions, it's easier to follow
| that advice than if I'm working with multiple people who all
| play different genres of games. A lot of the time people will
| only use gamedev-jargon to back up their design decisions
| (immersion, positive feedback, vertical slices, etc), which
| are hard to argue about because they're so subjective and
| ill-defined.
|
| Also, keep in mind how many times Level 1 gets tested. At a
| certain point, no matter how excited I am about the game, I'm
| going to get sick of playing it. From that point on, I'm no
| longer developing a game I want to play. I'm developing a
| game I should want to play but don't (I think this adds a lot
| to burnout). Maybe start the project as a game you want to
| play, but make peace with the fact that it's ultimately not
| for you to enjoy playing. It's for the players. You're very
| likely going to be sick of the game by the time it ships.
|
| "Make it juicy" and "avoid premature optimizations" are more
| maxims with limited use, and are even contradictory in a lot
| of instances. For juice/polish: Focus on the core gameplay
| mechanics. Once those are set in stone, make them juicy if
| you want/have the time. A juicy boring game is still a boring
| game. For performance/optimization: If a core part of your
| game is having 10,000 enemies on the screen at once, you're
| going to have to make design decisions based on performance
| at the very beginning of development. Kinda related: I've
| shipped a game where at the beginning of development, I
| suggested using Multiple Render Targets (we were doing
| deferred rendering with the XNA Framework) to speed up the
| rendering (instead of rendering everything, switching render
| targets, rendering everything again, switching render
| targets, repeat...) but the lead dev took the "avoid
| premature optimizations" maxim to heart, and we ended up with
| a significantly hobbled final product on PS4s and lower-end
| PCs. I guess I could have gone into "crunch mode" and rewrote
| the entire rendering pipeline to make up for poor decisions
| at the start of the project, but I wasn't willing to do that.
|
| It was my dream to make games from the age of probably 13 to
| ~28. I'm 35 now, and am not all that interested in games any
| more. I enjoy doing "gamedev" as a hobby, because I've
| realized my preferred method of "playing" is to program
| little toys I can tinker with for a few hours and then
| abandon. I also have MDD, so YMMV. Best of luck, though.
| bob1029 wrote:
| > If you want to not hate your life building a real-time
| multiplayer game, fully embrace data oriented design.
|
| I'd say if you want to not hate you life building _any_
| software, start with the data /schema/et. al. Before you open
| Visual Studio, open Excel and conference in your business
| stakeholders for a few design sessions.
|
| We recently started down a "data-driven" path with our B2B
| application. We went from ~90% of the application being code-
| based to ~95% of the application being configuration-driven in
| a matter of months. We can bootstrap an entire customer's setup
| by doing some simple JSON copypasta now.
|
| The best general model I have found so far would be to declare
| a global Domain type, and then put a List<T>/T of each
| constituent domain type within it. No nesting of complex types
| (reference types) should be permitted. All properties in the
| domain model types can map cleanly to/from SQL tables. The
| overall schema should be in 3NF/BCNF/DKNF. Relation types will
| be required. You can trivially serialize instances of the
| Domain type for whatever purpose. Each serialized copy of the
| Domain type can be thought of as a deterministic snapshot of
| all state, assuming you did everything correctly elsewhere.
|
| Use of language features such as LINQ can make this approach
| almost fun to work with. A clean, well-managed schema is the
| most powerful tool in your box when dealing with complex
| problem domains.
| syspec wrote:
| Here's a great video on rollback in a recent MK fighting game
|
| > 8 frames in 16ms
|
| https://m.youtube.com/watch?v=7jb0FOcImdg
| peter_l_downs wrote:
| A friend and I (mostly him, I just helped out a little) recently
| used netplayjs to create a p2p version of slime volleyball. It
| was surprisingly fun, not that hard, and this article is a pretty
| fair description of what to keep an eye out for. If you're
| looking to make an online p2p game, highly recommend this
| library, it's quite easy to work with.
|
| netplayjs: https://github.com/rameshvarun/netplayjs
| indentit wrote:
| A link to your Slime Volleyball implementation would be welcome
| too - haven't played it since about 2005 when all the slime
| games were java applets ;)
| Jyaif wrote:
| Regarding rollback, there's an interesting technique that I
| stumbled upon:
|
| Instead of serializing individual objects, you serialize all the
| memory relevant to the gameplay by using a custom memory
| allocator whenever the gameplay related code runs. This custom
| memory allocator is designed to allow taking fast memory
| snapshots, and supports restoring memory snapshots.
|
| This allows rollbacking _any_ code, including in my case
| rollbacking user-submitted Lua scripts.
|
| I wrote about it at https://www.jfgeyelin.com/2021/02/a-general-
| state-rollback-t...
| happyweasel wrote:
| client/server maybe?
| GuB-42 wrote:
| It is not the subject of this article but the advantages and
| disadvantages of deterministic netcode over client/server are
| talked about.
|
| You have other articles discussing client/server netcode. The
| one from Quake 3 is a good start (and it is open source).
| davedx wrote:
| Nice article.
|
| I implemented netcode for a canvas game I made a while ago over
| websockets. It used client-server with various RPCs from/to the
| clients and worked pretty well. I'm just starting on a kind of
| follow up of that game that will have more discrete movement so
| it was useful to read this overview. I'd maybe like to add that
| often games make a distinction between "authorized" actions
| players take (eg spell casts that damage other players), that
| often use the client-server RPC mode, and the "p2p rollback"
| actions that are less strict but generate more traffic like
| player movement. I believe World of Warcraft uses this model, and
| I also worked on a FPS that IIRC also worked this way: a gun fire
| was RPC over TCP; movement was done via UDP broadcast.
___________________________________________________________________
(page generated 2021-05-23 23:02 UTC)