[HN Gopher] Creating a matchmaker for your multiplayer game
___________________________________________________________________
Creating a matchmaker for your multiplayer game
Author : gafferongames
Score : 101 points
Date : 2024-06-08 19:38 UTC (1 days ago)
(HTM) web link (mas-bandwidth.com)
(TXT) w3m dump (mas-bandwidth.com)
| nightowl_games wrote:
| Glenn Fiedler is the best blogger on game networking. We used
| your stuff to guide our netcode development ~6 years ago. We now
| have a great stack on top of Godot for multiplayer games on the
| web over webrtc or websocket and enet on mobile and native. We're
| a small time company (3 people) and we have several multiplayer
| games..our netcode is a serious competitive advantage largely due
| to unity's failure to provide a system to our competitors.
|
| Thanks for helping us, Glenn.
|
| Our matchmaker needs a rewrite, and these posts are nice.
| gafferongames wrote:
| > We now have a great stack on top of Godot for multiplayer
| games on the web over webrtc or websocket and enet on mobile
| and native. We're a small time company (3 people) and we have
| several multiplayer games..our netcode is a serious competitive
| advantage
|
| That's awesome! Well done
| lostmsu wrote:
| What library do you use for WebRTC? The native libwebrtc?
| Sean-Der wrote:
| Godot uses libdatachannel. I remember them asking for mbedtls
| support
| nightowl_games wrote:
| Yeah we use libdatachannel. tbh we didnt see much (if any)
| network improvement using WebRTC over WebSockets, but our
| testing/analysis is pretty shallow (3 person company - whos
| got time for that, lets make products)
| intelVISA wrote:
| > our netcode is a serious competitive advantage
|
| What do you sell that isn't already solved by XDP?
| nightowl_games wrote:
| Googling XDP tells me it's extreme diesel performance. I'm
| sure theyre engines are great but we make videogames.
| remram wrote:
| https://en.wikipedia.org/wiki/Express_Data_Path is a
| feature of recent Linux kernels allowing you to
| bypass/replace most of the normal Linux networking code,
| allowing for higher performance. Some network cards will
| even run your network code directly on their hardware.
|
| (not sure what GP meant, just providing the reference)
| zarzavat wrote:
| I think they were confused by the term "netcode", which
| in game developer parlance means "code which implements
| networked gameplay in a multiplayer game", rather than
| "code that implements networking in an OS".
| nightowl_games wrote:
| Yeah, our netcode forward predicts from the server state
| to the local time to allow the local player to see their
| own character without network artifacts, and see other
| players in approximately their true location. It's the
| 'server authoritative, with client side prediction and
| rollback' technique. Consult Gaffer On Games' blog posts
| to learn more.
| roes9627 wrote:
| Great stuff!
| tithe wrote:
| Is matchmaking the process of players "always [being] sent to the
| datacenters with the best chance of having low latency"?
|
| EDIT: No, it seems that it's literally grouping players together
| in multiplayer games (but doing so in a way where latency between
| players is minimized.)
| gafferongames wrote:
| Yes, but much more too.
|
| The example here is really simplified to focus mostly on the
| finding datacenters with low latency problem, but it could also
| include things like matching players of similar skill together,
| finding a set of players that would make balanced teams, making
| sure that players who are partied up together play in the same
| match and so on.
|
| Basically, just like matchmaking in real life. It's the thing
| that works out groups of players who should play together in a
| match.
| Aachen wrote:
| My understanding of the post:
|
| 1. Let the game client apply to servers in datacenters where they
| will get low latency if that's relevant for your game
|
| 2. Wait 1 second on the server, then shuffle the list of clients
| and group every set of 4 into a match.
|
| 3. If no match is found after 10 seconds, add the client to
| servers in suboptimal datacenters
|
| 4. If still no match after another 10 seconds, put them in all
| the queues and join them to the first game it finds so they can
| at least play at all, even if they'll have lag no matter where
| they go
|
| 5. Determine latency either using the same IP:port as the game
| server will have because the route might differ from another port
| (citation needed imo. The author links another blog post for
| further info which is mainly a statistics lecture about averages
| hiding outliers and doesn't back up this claim that the port
| number makes a significant difference for latency in any but the
| most exceptional of situations where you're sent eastward instead
| of westward around the globe or so) or by collecting statistics
| to find which server was best for a given location and ISP over
| the past 30 days (seems over-engineered with extra downsides,
| like that you will have gaps in the data and that it might have
| changed)
|
| Did I miss anything? The whole thing seems somewhere between "do
| the most naive implication" (1-4) and doubtful (5). It's useful
| for devs to confirm the intuition they might have for the
| simplest implementation is correct but other comments saying this
| is a great blog surprise me when looking at these two posts
|
| The hardest part about matchmaking seems to me finding players
| well-matched in skill without making them wait long, especially
| during off-peak hours, or having off-putting loss streaks because
| they (50% chance for each game) got matched against a slightly
| stronger or equal team and lost. The article doesn't even mention
| these aspects exist, let alone offer advice on tackling them
| gafferongames wrote:
| It's a shame that you missed the key point of the article in
| #5. Many games today have suboptimal inputs to their
| matchmaker, because like you, they refuse to believe that the
| internet can route a significant percentage of players
| incorrectly.
|
| Source: Ran a network accelerator with more than 50 million
| unique players for 5 years. Bad network performance is much
| more pervasive than you think. https://networknext.com
| remram wrote:
| If it can route incorrectly, that seems like a strong signal
| not to rely on your maps. It doesn't matter if you know what
| the correct latency is, if the users only experience the
| incorrect one.
| gafferongames wrote:
| The solution is to pair the matchmaker using maps with a
| network acceleration product that fixes the bad routes,
| then the latency map and the actual latency experienced are
| very close.
|
| The reason you use the maps in the matchmaker is so you can
| look up the latency from client to datacenter in constant
| time and you don't have to waste any time on the client
| pinging ping servers to get the results.
|
| Plus, it costs money to run pings through the network
| accelerator, and they always (from experience) converge to
| the latency map anyway.
| remram wrote:
| How much time is wasted? 50 milliseconds at most?
|
| And you're about to send 50 packets per second for an
| hour during the game. Is sending one packet per
| datacenter really a cost worth mentioning?
| gafferongames wrote:
| Ship a multiplayer game and find out yourself? I've
| patiently explained how it is out there to you enough
| already.
| remram wrote:
| You know, I really loved your previous blog [1]. So many
| great tips and tidbits in that series. With your new blog
| you seem like a completely different man, posting those
| surface-level overviews full of gaps that seem to only
| push your product. I can see what happened.
|
| Sure, you don't owe me anything, don't reply if you don't
| have time. Don't pretend that you've answered my
| questions already though, everyone can see you didn't.
| Honestly why reply at all, if you are going to lie and
| use the "I know better noob" line?
|
| Never meet your heroes.
|
| [1]: https://www.gafferongames.com/categories/game-
| physics/
| gafferongames wrote:
| > Don't pretend that you've answered my questions already
| though, everyone can see you didn't.
|
| Jesus Christ dude. I'm just a guy who posted an article
| with a nice dataset and source code to help people out.
|
| While at network next, I dug in to matchmakers with
| multiple customers who were having weird problems like
| "why are my ping server pings good, but the player gets
| high latency when they connect to the server?" and "why
| are my players in Peru all getting high latency?".
|
| If you don't want to accept what I've shared about these
| experiences being true, that's fine. Write a matchmaker
| that ignores my advice, ship a game with it and see what
| happens when you hit 100k CCU.
| cmdrk wrote:
| Love the articles but I think you should disclose Network
| Next is not merely "a" network accelerator. It's the network
| accelerator that you're the CEO/founder of, which makes this
| read a bit like an advertisement.
| remram wrote:
| Players are coming in and out all the time, you can't "wait 1
| second and shuffle the list". 1 second after what?
| gafferongames wrote:
| The algorithm just has a tick rate at 1 second. Assume
| requests come in continuously, but only do work at some rate,
| eg. once every 1 second for a matchmaker that is meant to
| very quickly find games, and maybe once every 10-15 seconds
| for a higher quality matchmaker that is going to do a lot of
| work to match players together by skill etc (longer tick
| times means you'll have more players to consider in each
| iteration)
| toast0 wrote:
| > 5. Determine latency either using the same IP:port as the
| game server will have because the route might differ from
| another port (citation needed imo.
|
| No citation, but here's some things to think about.
|
| You can easily have two servers in the same facility, in the
| same rack, with very different ip addresses. Most of the time
| you'll get the same general route for all destinations within a
| /24 (v4) or /48 (v6), but not always. Two /24s at the same
| facility might get different routes, especially if there are
| capacity issues on the better route at any point.
|
| From same IP to same IP, anywhere there's a redundant link,
| which link to use is generally selected with a hash on the
| 5-tuple {protocol, destination ip, sender ip, destination port,
| sender port} (sometimes on a smaller tuple). If traffic is
| unbalanced or if one link is lossy, you can get drastically
| different experiences with a change in port number.
|
| If you've done traceroutes and seen multiple similar ips at a
| given hop number, it's probably the second effect; this is
| pretty common. Typical traceroute sends udp packets to the
| destination and modulates the sending port number. Sometimes,
| rarely, you can see drastically different routes within a
| traceroute. It's easier to do traffic engineering on a /24
| level though.
|
| Traffic on redundant links is balanced by hashing flows rather
| than on a per packet basis so that packets within a flow tend
| to arrive in order; out of order arrival is very expensive for
| endpoints, so it's better to ocassionaly have unbalanced usage
| than to regularly have out of order arrival.
| bit_flipper wrote:
| I've written game matchmaking before for games and found it far
| more challenging to do the portions related to matchmaking groups
| by skill. If you're hyper latency sensitive then perhaps this
| blog post is really useful? But for the games I worked on we
| would trade 10ms worse average ping for 10% better skill pairings
| without question. If you have any advice on improving skill
| matchmaking I would be quite interested.
| gafferongames wrote:
| > If you have any advice on improving skill matchmaking I would
| be quite interested.
|
| Samy Duc wrote a really good deep dive into Apex Legends
| matchmaking that covers skill and team balancing:
| https://www.ea.com/games/apex-legends/news/matchmaking-2023
| bit_flipper wrote:
| Thanks for the link, I hadn't read this one yet. Very
| interesting. I feel like as an industry we're still in the
| stone ages on how we do skill matching systems. A lot of the
| current and even future system they describe is really not
| great, and these flaws are definitely not unique to them (the
| systems I worked on are even more flawed)! Trying to
| matchmake dozens of players at a time is such a cool,
| challenging problem.
| bob1029 wrote:
| I really think it's a cursed mission to chase perfectly
| balanced matches.
|
| In terms of overall experience, the best approach in general is
| a purely random one. This is how you avoid the experience of
| getting trapped in a 20 game losing streak, even if you are a
| really good player.
|
| If you allow the natural balance of good players winning more
| often and bad players losing more often, you will find things
| get a lot less messy in between. This also provides your bad
| players an opportunity to occasionally witness what they could
| become. When you only play someone 1% stronger than yourself,
| you probably don't have a great idea of what the upper bound
| actually feels like. This can become a serious trap for players
| who are seeking to grow their skills.
| pynappo wrote:
| There's different levels of SBMM (skill-based matchmaking)
| though, most games nowadays have a choice between ranked
| (where the appeal _is_ the well-matched games and the
| possibility of increasing your rank) and quick play (where
| the appeal is the low queue time and more casual play, and
| the matchmaker basically is making sure that the lobby is
| filled with players who are _very_ roughly in the same skill
| range)
|
| While I also have fond memories of pre-SBMM Halo/CoD from the
| 2000s, they're more focused on pure shooter mechanics rather
| forcing players to work together to win an objective, so
| doing well but losing still feels fine. I find SBMM is needed
| in more objective focused esport games like overwatch,
| because the game is designed such that it's harder to do well
| individually (and thus have any fun) if your teammates aren't
| doing well.
| bob1029 wrote:
| > it's harder to do well individually (and thus have any
| fun) if your teammates aren't doing well.
|
| Perhaps this is the actual cursed aspect. In a free-for-all
| context (i.e. a team size of one), you would never have
| this kind of a problem. As you increase the mandatory team
| size to _six_ , you are creating an entirely new universe
| of effects to compensate for.
|
| In my experience, 99.99%+ of the frustration in Overwatch
| and League of Legends emerges from dealing with your own
| teammates, not your opponents.
| bit_flipper wrote:
| A lot of what you're saying depends greatly on the exact game
| you're talking about and what winning and losing means, for
| instance if the game has discrete places instead of a bimodal
| winner/loser.
|
| For a few of the games I worked on, random matchmaking like
| you describe is a non-starter. If you're a 90th percentile
| player in one of our games, you effectively never lose to a
| 70th percentile player or below. Your rating will be so high
| that we can't give you any rating system points for the win.
| So the person who won got no reward other than the feeling of
| winning, and the person who lost played a match they had no
| hope of winning. It ends up feeling pointless to play as a
| top player because only 1 or 2 matches in 10 on average have
| any meaning for you. Needless to say, it also feels worse as
| a low rated player because you simply lose more often.
| smarkov wrote:
| > the matchmaker shuffles the queue to avoid the same players
| being matched together repeatedly
|
| This is actually something I hate about multiplayer games with
| matchmaking nowadays. I made the majority of my childhood friends
| only because we stayed on the same server and played together for
| hours on end. I don't think it's a stretch to say that a key
| reason for why we play multiplayer rather than single player
| games is to socialize. This has become increasingly more
| difficult when you just get a new set of people every 10-30
| minutes.
| hlfshell wrote:
| This is why smaller AA games still keep this alive - to build
| community. For instance: Squad, a 80-100 player multiplayer
| game which primarily depends upon privately hosted servers ran
| by communities that actually care about the community and game.
| You'll still run into jerks like any online game, but it's a
| world of difference to the one-match/single-serving strangers
| you usually play with in other games.
| tomr75 wrote:
| can't you queue with people
| gryn wrote:
| Can't you make a group with people you like ? There a feature
| like that in rocket league.
|
| Getting grouped with the same bad team mates repeatedly would
| just make me quit the game.
| rcxdude wrote:
| You can queue with people you already know, and some games do
| have a 'stay as a team in the next queue' option, but there's
| no real opportunity to actually get familiar with anyone, let
| alone a community, and play with them more in the same way
| that you get with community servers
| halfmatthalfcat wrote:
| Group with the same team until the magic runs out in 3 games
| then everyone quits lol
| cableshaft wrote:
| Yeah, way back when I used to play Counter-Strike, I eventually
| landed on one server I really liked and kept going to. It would
| be interesting to see who was on it at different times of the
| day, and eventually I got to be friendly with most of the
| regulars that showed up, and they knew me.
|
| Turns out several lived near the server's location, in Texas,
| and at one point my friends and I just happened to be going
| there to visit a friend who was stationed at the nearby
| military base, and so I ended up meeting up with them for
| lunch. Nice guys.
| CaptainOfCoit wrote:
| > Turns out several lived near the server's location, in
| Texas, and at one point my friends
|
| Also ended up in a similar situation multiple times (bunch of
| randoms found some server we liked, sticked around for
| matches across weeks, eventually became regulars and
| eventually figured out we lived nearby). Sometimes we'd bump
| into each other on other servers too.
|
| After a couple of times of hanging out we've found out why
| (probably at least) we came across each other, we all default
| to sorting the server list based on ping (latency), and since
| we were all geographically close, we tended to end up on the
| same servers.
| szundi wrote:
| An option for not shuffling would be nice
| gafferongames wrote:
| The full source code is here: https://github.com/mas-
| bandwidth/matchmaker
|
| You can turn off shuffling if you want.
___________________________________________________________________
(page generated 2024-06-09 23:02 UTC)