https://prdeving.wordpress.com/2023/09/29/mmo-architecture-source-of-truth-dataflows-i-o-bottlenecks-and-how-to-solve-them/ Skip to content PRDeving typeof FUN Menu * Home * About * Contact * Twitter * RSS Feed PRDeving MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve them [62ced21e57da79004fbe5094_bottleneck-in-project-management-p-1080] By certain absurdities of life that only someone whose hobby perfectly aligns with his job can understand, lately I've been involved in the design and architecture of an MMO game. As much as it may seem that such an application fits perfectly in what we would unconsciously consider "distributed architectures", the specific details (small and large) of this kind of solutions turn what, for any capable engineer would be a simple design process into a headache of biblical proportions. --------------------------------------------------------------------- Latencies, race conditions, synchrony and availability is something that every architect faces, practically every day, however, in practically all scenarios, the solution is a renegotiation of the functional and technical requirements (is it enough to have latency less than 10 seconds?) and rarely do we design complete solutions due to their complexity. The world of multiplayer video games is kinda different, the spatial complexity goes to infinity, and we try to keep the temporal one to a minimum. We work with multiple strictly deterministic modules that we can easily maintain and deploy in parallel. And sooner or later, we will hit hell, an evil that lurks behind every MMO, the I/O bottleneck in the database. The main purpose of this post is to explore the limitations that we find at the data I/O level in the design of MMO systems, the interactions that the system makes with the data, its derived problems and how to solve them. In addition, we will suffer the cognitive dissonance in its purest state, since the solutions that are adopted in MMO type systems are a certain death in an enterprise environment (e-commerce?). It is interesting to see how different the solutions are depending on the problem and the system requirements. The beauty of software architecture the source of truth of your game world IS NOT the database. This is a complicated concept to accept for those of us who come from enterprise systems, but it is so. In online games, the source of truth of the state of the world is the in-memory world state, not the database. In these cases, we consider the database a persistence medium, not a source of truth. Whereas the real source of truth of the virtual world we are concerned with always resides in memory. Heresy! well, a little bit. let's go deeper into this concept with MATH. let's say we have a MMORPG game like world of warcraft with 1000 players, its world is of course divided into zones, so that we can do sharding. what we are interested in here, however, is that all players live together in the same environment. They have to be able to see each other, talk to each other, pass items from one to another, see the level of other players in the world, and so on. By pure data logic, for this to work in normal terms the state of the world has to be unique. Let's assume that these players just go around, happily killing boars. Let's also assume that our game client, in order to make life easier for our infrastructure, sends the players' actions to the server once per second. Knowing how to multiply is enough to realize that we would be talking about N player position update messages sent per second, where N is the number of players. And that's only taking into consideration the position of the players in the world! Now you have to add experience gained, sword strokes, chat messages, etc. Considering our database as the source of truth, it forces us to persist all this information, which, to put it mildly, is N writings per second. [image] Best of luck. The solution to such a problem is relatively simple once we get rid of the idea of using the database as the source of truth. Cache, lots of cache When I said earlier that the source of truth of the game world state resides in memory I meant it, though not in the simplest sense. In these cases our requirements are not too many, though complicated: * we need direct and low latency connection between game services and world state. * we need the world state to be persisted in order to replicate or recover from errors or outages * we need it to be scalable for the future * we need to avoid race conditions To meet these specifications, we usually use a data broker pattern. We create a service with direct connection to the database that will keep the full state in memory and connect our game world services to it via RPC. So it acts as a kind of cache of the database. [image-1] But how does this meet the requirements? Let's go in parts We need direct and low latency connection between game services and world state. This is perhaps the simplest part, our game world services connect by RPC to the game state service and execute actions. Here comes an important point that, although perhaps slightly out of the scope of this post, is worth mentioning. These RPC commands must be specific to our game logic, no SQL or obscure requests linked to persistence or the concept of "data"! grantPlayerExperience, playerChangingZone, etc. It is the data service that is responsible for implementing the API so that, when it receives a "grantPlayerExperience" command, it adds N experience points to the player X. This way, we keep our layers separate, we decouple the data API from the implementation and keep our game logic isolated. We need the world state to be persisted in order to replicate or recover from errors or outages Persist yes, but What and When? this is more of a philosophical and product design job than software architecture per se, but let's see how we can dig into it a bit. The first question is, how much is acceptable? sometimes we tend to think that even the smallest movement is likely to be saved for posterity, but often it is not. Let's take for example the game League of Legends, what would we save and when in this case? If you ask me, I would save the final state of the game, once it is finished (we ignore replays, obeservability, etc for this example). The reasons are simple, in a system of this type, we look for persistence to be able to recover from problems, not as a business definition. Let's suppose that we are saving EVERY change throughout the game and our instance explodes, goes offline or is sucked into the infinite void of space. Can we recover from that error? Can we restore the last saved state? No, we can't. It is likely that, by the time we restore the state, half of the players will no longer be connected. So what are we going to persist all the changes in the database for? The tricky part here, as I said above, is what to save and when to save it. In the case of the example, what do we save? well, the player's stats at the end of the game and little else. When? at the end of the game. In a more complex case like World of warcraft, we could save, for example: the inventory when an item changes hands or is used, the player's position every certain time (30 seconds maybe?), the zone every time the player changes, etc. The trick is to minimize as much as possible the writes to persistence by writing only those things that are essential for recovery. everything else, we keep it in memory in the game state service. We need it to be scalable for the future Here perhaps we touch on one of the tricky points. In terms of scalability, having a service that has the entire world state in memory is perhaps not the best option a priori, since, although it is easy to scale it vertically, horizontal scaling runs into several problems. Apart from being a single point of failure, an important issue that I will not cover in this post. A relatively simple solution may be to take advantage of the Redis pub/sub system to synchronize the state services. Also, let's think about it this way, if you have 100 game world services acting against a single state service, we are talking about 100 open sockets, it is not comparable with the amount of thousands of database commands that we would have if we ignore the broker, right? We need to avoid race conditions This is perhaps the most trivial point, since, as long as we maintain a single-threaded architecture in our data service, it is practically guaranteed that we will not have significant race conditions. However, when working with distributed data or in multi-threaded environments we have a great friend. CAS (Compare-and-swap) every write operation performed on our data services should be done with a CAS instruction, so that we make sure that our writes are synchronous. The thing is simple, you take a hash of the state (or a part of it) before starting the operation which we will call, version hash. You prepare the new state, generate a hash for the new state and ONLY persist the two things if the version hash of the state matches the one you took at the beginning. [image-2] A simple version control system and what happens if the CAS fails? retry, N times, as many as you think, and if it is not possible, return error. --------------------------------------------------------------------- Understanding the state of our game world as ephemeral, and knowing how to identify the exact data amenable to persistence is perhaps more of a learned art than a teachable science. In summary, when working in high traffic distributed systems, with many agents constantly changing dynamic data, considering the database as the source of truth leads to bottlenecks and availability problems sooner rather than later. Strategies, perhaps counter-intuitive or heretical to those of us who come from enterprise architectures, are usually the standard solution accepted by the industry. Not without their own problems and weaknesses. A good planning of the update and persistence windows of our data and a thorough analysis of our system's "use cases" and scenarios, together with a data broker pattern allows us to free the database from unnecessary writes and our wallets from bulging invoices. Understanding the state of our game world as ephemeral, and knowing how to identify the exact data amenable to persistence is perhaps more of a learned art than a teachable science. The beauty of software architecture. Share this: * Twitter * Facebook * Like this: Like Loading... Related [b16a6]Author PRDevingPosted on September 29, 2023September 29, 2023 Categories architecture, game development, philosophyTags availability, cache, CAS (Compare-and-swap), data broker pattern, data I/O, data persistence, database bottleneck, distributed architectures, error recovery, game architecture, game design, in-memory world state, knowledge, latency, MMO, race conditions, Redis pub/sub system, RPC (Remote Procedure Call), scalability, sharding, single-threaded architecture, software architecture, synchrony, use cases, version control One thought on "MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve them" 1. Pingback: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve - Veritas Reporters Leave a Reply Cancel reply [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Post navigation Previous Previous post: How to write a game engine in pure C: Part 3 - The Engine Entity Stalk me on: GitHub MDN NPM Twitter profile for RDeving on Stack Exchange, a network of free, community-driven Q&A sites Categories * architecture (1) * C/C++ (15) * engineering (6) * game development (10) * hacking (6) * javascript (6) * nodejs (2) * Optimization (6) * philosophy (3) * security (5) * talks (1) * technique (23) Archives * September 2023 (1) * June 2019 (2) * May 2019 (1) * September 2018 (1) * June 2018 (1) * July 2017 (1) * April 2017 (3) * March 2017 (2) * February 2017 (3) * January 2017 (1) * October 2016 (1) * September 2016 (1) * August 2016 (1) * May 2016 (10) * Home * About * Contact * Twitter * RSS Feed PRDeving Create a free website or blog at WordPress.com. [Close and accept] Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use. To find out more, including how to control cookies, see here: Cookie Policy * Follow Following + [wpcom-] PRDeving [ ] Sign me up + Already have a WordPress.com account? Log in now. * + [wpcom-] PRDeving + Customize + Follow Following + Sign up + Log in + Copy shortlink + Report this content + View post in Reader + Manage subscriptions + Collapse this bar Loading Comments... Write a Comment... [ ] Email (Required) [ ] Name (Required) [ ] Website [ ] [Post Comment] %d bloggers like this: [b]