[HN Gopher] An analysis of recent multithreading improvements fo...
___________________________________________________________________
An analysis of recent multithreading improvements for a smoother
game
Author : diggan
Score : 41 points
Date : 2025-06-17 19:56 UTC (3 days ago)
(HTM) web link (dev.arma3.com)
(TXT) w3m dump (dev.arma3.com)
| hinkley wrote:
| > The design of the job system decides how we are able to utilize
| it, and the old design (in use since Arma 2) worked, but it was
| quite primitive. It only allowed submitting jobs as one block,
| and there could only be one block active at a time. This means
| all jobs had to be submitted at once, after which we needed to
| wait for the jobs to be finished, before we could continue doing
| other things.
|
| > This is also called "Fork-Join".
|
| I removed one of these from a batch processing system and got
| about 3x the throughput in the process.
|
| As it was written, the system was trying not to be a noisy
| neighbor for online (consumer-facing) traffic and was not
| succeeding. It could still cause brownouts if too many other
| things were running at the same time. So the authors thought to
| have the users run Splunk queries, looking for traffic from these
| other jobs, to see if it was safe to run the CI/CD pipeline.
| That's ridiculous. It's the sort of thing someone Ops-facing
| forgets that nobody else on the team gives a shit about most
| days.
|
| If we wanted people to be able to run it without consequence, it
| needed to go quite a bit slower, but it was already more than
| half of the wall clock time being spent in a runbook as it was.
| So I replaced it with a queue instead, that would have n tasks
| running all the time, instead of up to n and as few as 1. I ended
| up being able to tune it to about 75-80% of the original number
| with little problem. Well, one problem. There was one group of
| customers whose assets generated about 10x the load of any other
| customer, and the way we queued the work clustered by group. Once
| I sorted by customer instead of group we stopped getting a
| clustering of excess workload, and it also made it a hell of a
| lot easier to eyeball the status messages and estimate how far
| along you were.
|
| Spiky processes cause problems for both load-shedding and
| autoscaling schemes. And starting too many tasks at once causes
| memory pressure on your end, since most data processing tasks
| take more memory in the middle than at the beginning or the end.
| You are better off self-throttling your traffic so you make the
| memory load more sequential and you allow compensatory systems
| the time they need to adapt appropriately to your sudden peak of
| traffic.
| o11c wrote:
| Unrelated to the specific game:
|
| Note that the "can't parallelize AI/scripting" is a consequence
| for a design choice that many people make without thinking -
| namely, that all actors should, internally, have perfectly
| accurate and up-to-date knowledge of the world, using the same
| in-game global objects.
|
| If each actor makes a _copy_ of the world for what they know,
| there 's nothing preventing parallelism. This does imply
| quadratic memory, but you can just cap this - if there's a lot
| going on, it makes _sense_ for an actor to lose track of some of
| it. And once you 're working with imperfect knowledge, you can
| just ... throttle the AI's think time if it's doing too much.
|
| Another thing you can do, assuming you already have suspendable
| scripts, is "merge" the thread affinity of two scriptable objects
| when you know they need to interact (only really needed for
| transaction-like things; often you can just emit some async state
| to be resolved later). Actually, you don't need to suspend if you
| have enough static analysis, but suspending is probably actually
| the easier thing to do.
|
| Related to this, IMO it's a mistake to expose _functions_ to the
| scripting API that access state implicitly. It 's a better design
| to expose _objects_ (that can only be accessed /mutated if passed
| as an argument) - and to expose a different set of objects
| depending on what context the script is called in. A type-checker
| is really useful here.
| dustbunny wrote:
| Another model may be that all actors read from frame N-1 and
| write to frame N. Then there's not many copies, just one.
| egypturnash wrote:
| "From the feedback on Profiling branch we have seen several times
| that players had issues caused by setting wrong command-line
| parameters. That was typically because an old optimization guide
| told them it would make their game run better. But now that the
| game is more multithreaded than it used to be, bad settings have
| a more noticeable and potentially negative impact."
|
| I wonder if they also added a popup that _detects_ some of these
| pessimal command-line parameters being used and says "hey we did
| a lot of optimizing work that's made these settings iffy, please
| go check out this blog post for more details".
___________________________________________________________________
(page generated 2025-06-20 23:00 UTC)