[HN Gopher] Benchmarking Malloc with Doom 3
___________________________________________________________________
Benchmarking Malloc with Doom 3
Author : zdw
Score : 61 points
Date : 2022-06-05 14:36 UTC (2 days ago)
(HTM) web link (www.forrestthewoods.com)
(TXT) w3m dump (www.forrestthewoods.com)
| nemetroid wrote:
| Eleven million mallocs over seven minutes of gameplay at 60 fps
| comes out at an average of 437 mallocs per frame (this includes
| "load level" and "die and reload", so the average during gameplay
| is likely even lower).
|
| The author writes "Doom 3 hits 60fps in debug and calls malloc
| all over the place.", but to me those stats suggest malloc-averse
| code.
| abainbridge wrote:
| "My first attempt at a benchmark involved allocating and freeing
| blocks of random size. Twitter friends correctly scolded me and
| said that's not good enough. I need real data with real
| allocation patterns and sizes."
|
| "The goal is to create a "journal" of memory operations. It
| should record malloc and free operations with their inputs and
| outputs. Then the journal can be replayed with different
| allocators to compare performance."
|
| I don't think that is sufficient either. You need to mix the
| malloc/free with the other work, because the other work knocks
| your allocator data structures out of cache and pollutes your TLB
| and branch predictor. And loads up the dram interface (eg by the
| gpu fetching data from it). Etc etc.
| sitkack wrote:
| I would have measured the time directly inside of the Doom
| allocator and the OP could produce the same charts directly
| from the measured data without replaying the allocation log.
|
| And IF I were going to collect that data, I'd grab the call
| stack as well to see where the allocations were coming from.
| There might be a chance to retro fit an arena allocator, but
| would think Carmack code is already _probably_ optimal? Yes it
| is, for the reason we can even have this discussion about it.
|
| The other change I would make to the experiment, is to have the
| traces be taken from a timedemo and not via regular gameplay so
| that hypothesis can be tested. At the very least, one could
| precisely time the impact of instrumentation.
| moomin wrote:
| I suspect you'd find Carmack's code was "good enough that
| there was something more important to work on" rather than
| necessarily "optimal". So it's entirely possible there's
| still some fruit to find, just no low-hanging ones.
| thxg wrote:
| > I would have measured the time directly inside of the Doom
| allocator
|
| Exactly! The author already has all the infra necessary
| (rdtsc for each call). Measuring time inside the game would
| be more accurate and simpler. Why did the author do things
| this way? I must be missing something.
|
| (By the way, the author even seems to know about this issue
| since they added code that simulates using the allocated
| blocks (touching one byte for every allocated 4k), but that
| does not feel like it's nearly enough.)
|
| > grab the call stack as well
|
| Maybe this part could be done with no code with VTune / Linux
| perf? Sure, those only gather stochastic measurements (so not
| ideal for the original latency measurement). But to get a
| rough idea of where the costly allocations come from, it
| could be an easy way.
| imron wrote:
| > Why did the author do things this way? I must be missing
| something.
|
| Likely because it's easier to test against different
| allocators with a small replay tool than it is to try and
| get Doom3 to compile against dlmalloc, jemalloc, mimalloc,
| rpmalloc and tlsf.
| mcronce wrote:
| I would also bet that getting the game to perform the
| exact same series of allocations would be an intractable
| problem to solve. I don't think Doom 3 has a benchmark
| mode; the author just recorded themselves loading the
| game, loading a level, doing a bit of gameplay, etc.
| sitkack wrote:
| Carmack is a great empiricist, meaning he pays a lot of
| attention to how things work in a running system. All of
| his engines provide rich facilities for record and replay
| as well as running benchmarks.
|
| timedemo <demo file> uscache
|
| The DOOM3 source is here https://github.com/id-
| Software/DOOM-3
|
| There are lots of facilities for recording and playing
| back demos. The author could record their own gameplay
| and play it back in realtime vs running timedemo (which
| plays back as fast as possible).
|
| https://github.com/id-
| Software/DOOM-3/blob/master/neo/framew...
| 2865: cmdSystem->AddCommand( "recordDemo", "records a
| demo" ); 2866: cmdSystem->AddCommand(
| "stopRecording", "stops demo recording" ); 2867:
| cmdSystem->AddCommand( "playDemo", "plays back a demo",
| 2868: cmdSystem->AddCommand( "timeDemo", "times a demo",
| 2869: cmdSystem->AddCommand( "timeDemoQuit", "times a
| demo and quits",
|
| https://www.youtube.com/watch?v=CLA42q3myCg
|
| https://www.youtube.com/watch?v=9KIhRmdz2cI
| nym375 wrote:
| > I don't think Doom 3 has a benchmark mode
|
| id Tech games have "timedemo" mode, where they replay a
| pre-recorded demo as fast as possible, and report how
| many FPS it was able to process. These were very popular
| benchmarks long ago.
| mcronce wrote:
| I stand corrected then, this would probably be a great
| fit
| abainbridge wrote:
| > I would have measured the time directly inside of the Doom
| allocator
|
| Yeah, that'd be much better. It'd still only tell us how the
| allocator performed when having to share cache, memory B/W
| etc with Doom 3 though. Performance of similar allocations in
| a different application could be different. Even changing the
| screen resolution of Doom 3 might be enough to change the
| performance of the allocator. To get anywhere we need to
| understand what all the hardware components that alter
| performance are, the limits of their various resources and
| the penalties for running out of those resources. I normally
| find it is easier to remove the allocs from my program's main
| loop than to do this kind of analysis. In fact, I think this
| analysis is impossible. What happens when some background
| process like a virus checker decides to run? Doing less work
| in your program is almost always better. Moving the
| allocations out of the main loop normally makes the ownership
| easier to understand too, so helps to minimise code
| complexity.
| [deleted]
| mhh__ wrote:
| For some programs allocation can be a lot slower than actually
| accessing the memory unfortunately.
| trasz wrote:
| >The goal is to create a "journal" of memory operations.
|
| Hm, I wrote something like this once - https://github.com/CTSRD-
| CHERI/cmt/blob/master/README.md
| Shadonototra wrote:
| the biggest enemy is non-predictable performance, otherwise you
| get visible stutters
|
| but nowadays nobody gives a shit anymore since most games are
| made using unreal/unity wich ship with a shitty GC
|
| generation of developers who think pumping more hardware will fix
| all their problems, little do they know
|
| wich is why most games using these engines have troubles porting
| their game to resource constrained devices like
| smartphones/switch
|
| frame budget for 120fps is just 8ms
| AtlasBarfed wrote:
| How do people log hyperfast data like this without impacting the
| thing you're measuring?
|
| ... preallocate storage? (is this actual, real irony?)
| tomovo wrote:
| See also - Arma 3 benchmarks comparing different malloc
| implementations here: https://www.youtube.com/watch?v=kNmeMxEel4M
| ultrahax wrote:
| Game dev here ( with significant experience in engines with
| significant idTech ancestry ) - the goal is, generally, to avoid
| mallocs _during_ a frame - you alloc everything you need during
| startup. I'm surprised there's that much ongoing allocation.
| Syzygies wrote:
| Back in the 1980's I co-wrote a computer algebra system, that ran
| for example on many colleagues' first computer: The first
| Macintosh. Resources were scarce, everything mattered.
|
| Malloc attempts to work acceptably well under diverse scenarios.
| Specific programs such as ours can adopt simpler memory
| disciplines, supported by an interface with malloc.
|
| Almost all of our memory needs could be served by pools of fixed-
| size slugs of memory. Each pool experienced considerable churn
| and reuse, but the total memory needed by each pool was unimodal,
| growing to a peak then subsiding until the pool was no longer
| needed. We chose a universal fixed-size block to be served by
| malloc. We kept these blocks on a master stack, to assign to
| pools as needed. We kept the slugs in each pool on stacks, to
| reuse as needed. Closing a pool returned its blocks to the master
| stack.
|
| There's nothing novel here; this is about as dumb a protocol as
| one can imagine, and correspondingly faster than anything else we
| tried.
|
| I still believe the lesson today: One's core algorithms need to
| be crafted to cooperate with the fastest memory protocols. The
| doctor can't do everything without the patient's cooperation.
|
| For example, I believe that the approach taken by Rust will
| ascend as we better learn how to work with it.
| throwawaymaths wrote:
| You mean zig, right? Zig is allocator-agnostic. It's hard (but
| not impossible) to implement type-specific allocators in Rust,
| and if you want the same type to take context-dependent
| allocator types, or context-dependent allocator instances, it's
| much , much harder, if I'm not mistaken. And the default way of
| programming Rust doesn't really tolerate allocation failures
| (which you might run up against with fixed memory slabs)
| fisherjeff wrote:
| I wonder how many of the longer execution times are just thread
| pre-emptions?
| jeffbee wrote:
| The fact that all the allocators have the same tail performance
| suggests the measurement is polluted by something like thread
| preemption or processor power state transitions.
| smat wrote:
| Not sure what this empirical analysis adds to answer the initial
| question. Sure, modern computers are fast and modern operating
| systems are designed to reduce latency.
|
| At the same time the "best practices" of not using mutexes and
| malloc on real-time threads are there for a reason: They
| potentially trigger a system call, which will add quite some
| latency (as the measurements show). Because real-time is all
| about deterministic latency, this is undesired.
| sitkack wrote:
| > Not sure what this empirical analysis adds to answer the
| initial question.
|
| Why would you discourage someone from doing this? You thinking
| you already knew the answer doesn't help the OP at all.
| smat wrote:
| I did not mean to discourage people from making their own
| evaluations. Maybe I did not word it correctly.
|
| The article starts with questioning best practices for
| implementing real-time processing threads. The statistical
| analysis presented in the article is based on a single
| workload and a single machine, which in my opinion does not
| help to answer this question.
| sitkack wrote:
| Oh, ok, I understand.
|
| I agree that the empirical technique could be better, but
| the experiment is neat even if slightly flawed.
|
| If the author wanted to show how much memory allocation
| could be done in an audio thread, they should construct an
| audio workload where they can tune the allocation and
| memory access patterns and then determine exactly how much
| is too much. But really that was the motivation, then as
| they admit in the start, they got nerd sniped and started
| measuring DOOM memory allocations (also fun). We should
| ignore the original motivation unless one uses a completely
| different experiment to justify something that wasn't
| tested.
| [deleted]
| dwdz wrote:
| Please don't use Jet color scheme. Matplotlib uses excellent
| viridis by default. There's no reason not to use it. [0][1]
|
| [0] https://stats.stackexchange.com/questions/223315/why-use-
| col...
|
| [1] https://www.youtube.com/watch?v=xAoljeRJ3lU
___________________________________________________________________
(page generated 2022-06-07 23:02 UTC)