[HN Gopher] When allocators are hoarding your precious memory
___________________________________________________________________
When allocators are hoarding your precious memory
Author : marcodiego
Score : 64 points
Date : 2023-10-06 15:15 UTC (7 hours ago)
(HTM) web link (www.algolia.com)
(TXT) w3m dump (www.algolia.com)
| vlovich123 wrote:
| TLDR glibc's malloc doesn't work well for server workloads. Nice
| deep dive into the available options to trim it, but the FUD
| around switching to something like mimalloc or the most recent
| tcmalloc is unwarranted considering both have seen large scale
| deployments and have same defaults out of the gate that don't
| need tuning (and any tuning you do for glibc would need similar
| validation requirements).
| jeffbee wrote:
| glibc's malloc doesn't work well for any workload. If your
| workload is anything other than trivial you should be using
| another one.
|
| > FUD around switching to something like mimalloc or the most
| recent tcmalloc
|
| Certainly, and the specific FUD about libc++ is unwarranted for
| tcmalloc, considering the developer of tcmalloc also uses
| libc++.
| spockz wrote:
| So which Malloc should we be using? How about setting that as
| a default? If apparently the trade off is so cookie cutter
| clear.
| jeffbee wrote:
| If you have a non-trivial workload (i.e. one that costs
| money) you should shop around and use the one that is the
| most efficient in your application.
| hinkley wrote:
| In early days Java's allocator was quite a bit faster than C++
| in part because memory allocation was not fully concurrent,
| making allocation part of the sequential element of Amdahl's
| law.
|
| Where we are now is better, but has its own problems.
| masklinn wrote:
| > Where we are now is better
|
| Not really. Lots of rust users come from high level languages
| so when people come in and complain about rust being
| unexpectedly slow allocation issues are in the top 5 at
| least. It's incredible how shit the standard allocators are
| in almost all systems.
| athanagor2 wrote:
| Isn't this inevitable if the user allocates each time a
| struct/array is created, rather than allocating upfront?
| masklinn wrote:
| Meh. Users not used to tracking allocations keep to the
| same regime (lots of allocations), but because the
| platform allocators are utterly terrible the allocation
| overhead is orders of magnitude higher than in even a
| relatively basic runtime, thus the program is dog slow.
|
| This is not inevitable, platforms could provide
| allocators which are less awful. Obviously they can't be
| as fast as specialised runtime facilities but when you
| see the gains many applications get by just swapping in
| jemalloc or some such...
| PH95VuimJjqBqy wrote:
| I would simplify it and say it's an issue with developers
| who don't understand manual memory management (allocating
| memory pools up front is just 1 such strategy for manual
| memory management).
| perlgeek wrote:
| This vaguely reminded of the time where our self-hosted
| installation of JVM-based app worked fine in DEV and QA (on
| vmware-based VMs), but in PROD it would error out the time with
| "out of memory", even though the machines (bare metal) had far
| more RAM.
|
| It turned out they also had lots more CPU cores (they were
| repurposed former DB machines), and by default the memory manager
| (of the JVM, I suppose?) scaled the allocation block size the
| number of CPUs, so it tried to allocate far more memory than
| necessary, and since this was still a 32-bit JVM it quickly ran
| out of space for its allocations.
|
| Once we found out what the issue was, it was pretty easy to tune
| with an environment variable.
| jon-wood wrote:
| Off topic, so apologies in advance. Why do environments so
| often get capitalised like that? Prod isn't an acronym, it's an
| abbreviation of Production. Similarly Dev is an abbreviation of
| Development.
|
| I know in the grand scheme of things it's meaningless, this is
| just one of those things that's like someone scratching their
| fingers down a blackboard when I see it day to day.
| alexstore06 wrote:
| My first guess for that particular case is that PROD and DEV
| are often either parts of hostnames for those particular
| machines or are referring to hostnames, which are often
| written in all caps.
| mikebenfield wrote:
| I don't know, but I see unnecessary all caps in other words
| too - it's not infrequent that I see JAVA and sometimes RUST
| (for the programming languages). Kind of irks me, but
| obviously doesn't really matter. Ultimately I think it's just
| a weird niche cultural holdover from the very early days of
| computing when lots of random stuff was arbitrarily
| capitalized, like FORTRAN was originally all caps, maybe
| related to early character sets that only had capital
| letters.
| mike_hock wrote:
| And some all caps languages are still around. C, D, R, C++,
| C#, F#.
| burnished wrote:
| Those follow normal punctuation rules in that the first
| letter of the proper noun is capitalized.
| tom_ wrote:
| This sort of thing has long annoyed me too when I see it, so
| thank you for stepping into the firing line for at least the
| both of us. See also, "FED" for Federal Reserve.
|
| I'm trying to train myself out of this nonsense, but some
| habits die hard.
| atq2119 wrote:
| The worst example of this is VISA instead of visa when
| talking about international travel as opposed to
| credit/debit cards. (Hats off though to whoever decided on
| that marketing name...)
| Terr_ wrote:
| > Prod isn't an acronym, it's an abbreviation of Production.
|
| Tangentially, "Prod" is justified since it's being treated as
| a proper name, like "Paris", "Roderick", or "Mount Rushmore."
|
| But yes, "PROD" is harder to explain, and I suspect it might
| be bleed-over from looking at code or config files where it
| gets capitalized.
| otikik wrote:
| It might be a leftover from the time where we had text-only
| docs (no courier) and the use of `backticks` was not
| commonplace. With those restrictions, ALL_CAPS would be a way
| to signal "this is a name that has meaning for machines",
| simply because humans don't use that convention.
| bregma wrote:
| "PROD" is an acronym (a shortened form of a more formal name
| for something). What it isn't is an initialism (a word coined
| by taking the first initial of a descriptive phrase).
| Traditionally, both initialisms (eg. CPU) and acronyms (eg.
| NORAD) use all majescule typography. Sometimes the coined
| words become common usages and are commonly rendered in
| capital or miniscule case as appropriate (eg. radar, scuba).
|
| This is, of course, descriptive typography. Prescriptive
| typography will vary from style manual to style manual.
| Calavar wrote:
| I think you're getting the definition of acronym mixed up
| with that of abbreviation. An acronym is an initialism that
| is pronounced as if it is a natural word. For example, NASA
| is both an initialism and an acronym. FBI is an initialism,
| but not an acronym.
|
| Well that's the dictionary definition anyway. In practical
| terms, I think most people use acronym as a synonym of
| initialism, at least in the US. This is one of the places
| where the dictionary definition doesn't match with real
| world usage.
| PH95VuimJjqBqy wrote:
| I don't have a solid explanation but I sometimes have a
| tendency to capitalize things that are "important".
|
| Think about book titles. "Exploration of Paris", the
| important words are capitalized.
|
| I can imagine many people capitalize the environment names
| without thinking too hard about _why_ they do it.
| kevingadd wrote:
| 32 bit and high core count is definitely a nasty combo. Many
| applications spawn 2-4 threads per core which is a great way to
| eat up that address space.
| nine_k wrote:
| But don't all threads of a process share mostly the same
| address space? I could imagine that starting many processes
| would exhaust address space, or allocating a lot of virtual
| memory / mmap without committing.
| simcop2387 wrote:
| That's one of the fun bits, other processes don't share the
| memory space. That's one reason you can have 16GiB of ram
| on a 32bit x86 server without much issue. This was done on
| x86 through a tech called PAE (Physical Address Extension)
| that let the OS know there was more memory and allocate it
| around, but any one process was limited to 4GiB (usually
| lower due to kernel/userspace barriers) because of 32 bit
| pointers. So if those threads all allocated memory it was
| easier to hit the limit but if you used separate processes
| then they could each reach 4GiB (or whatever the limit is)
| independently without causing problems.
| dilyevsky wrote:
| They better hurry up with that - i think 16.04 only getting
| secuirty patches until the end of this year without enterprise
| license
| com2kid wrote:
| The number of arguments I've gotten into with other engineers
| that "manual memory allocation is deterministic, GCs are not!" is
| too high.
|
| First off, GCs are quite deterministic, unless they are calling
| rand() someplace. I've walked through GC code under controlled
| conditions with identical repeated allocations, GCs do the exact
| same thing under the exact same circumstances every time.
|
| Second off, you have to pay the price of allocation and deal with
| memory fragmentation somewhere. GCs typically pay the price at
| the tail end (deallocation, compaction) but have absurdly fast
| allocators (a handful of instructions), where as manual memory
| allocators pay the price for allocation (finding a slab of
| memory) up front, and unless you know how to work around the
| details of your allocator, you end up dealing with fragmentation
| yourself.
|
| FWIW this also means many, many, types of applications can gain a
| boost from intelligent memory allocation strategies. There is the
| famous example of the cruise missile that never worried about
| deallocation because it would explode before it ran out of
| memory, but scenarios involving bespoke memory management being
| better than "just use libc!" are actually quite common.
|
| Have a stateless microservice? If you know the max size of memory
| it'll use, for each connection that comes in just allocate a slab
| of memory and when the connection closes free the entire slab,
| far more efficient than using a GC or any standard allocator.
| Also having to calculate the max amount of memory a single
| connection can use is a great way to figure out how much load an
| instance can handle (depending if you are CPU bound or not).
|
| For certain stages of compilation, compilers can get away with
| not freeing memory and just exiting and letting the OS reclaim
| everything. (Of course less doable now days with compilers
| running as background services recompiling code as it is typed,
| and also whole program optimization has made it so compilers use
| a lot more memory, so I don't know if this strategy is still in
| use in these modern times!)
|
| The tl;dr is that freeing memory is complicated no matter if you
| have a GC or not. It is something every developer should be
| thinking about from time to time.
| Karellen wrote:
| I thought that GCs are often run in their own thread, and
| because scheduling at the OS layer is seen as non-deterministic
| from the application's viewpoint (you never know when your own
| threads are going to get stopped/started), you can't control
| exactly when the GC is run?
|
| Is that wrong? Or obsolete? Or GC-dependent?
|
| But also, just because something is theoretically deterministic
| (e.g. it might technically be possible to work out when a GC
| sweep will run, and multiple runs of the exact same program
| will cause the exact same set of GC sweeps), actually working
| out when a GC sweep will run is so non-trivial the only way to
| actually work it out is to run the program and watch it happen,
| and also it becomes impossible to predict how any changes at
| all to a program will affect how the GC runs as a result (other
| than making the change and running it).
|
| Or are there ways to deduce when GC sweeps will run for a given
| program, without actually running it?
| theolivenbaum wrote:
| Interesting, we had very similar issues with malloc and RocksDB
| on a production instance, ended up switching to jemalloc and the
| problem went away. Cloudflare had some similar issues with it as
| well: https://blog.cloudflare.com/the-effect-of-switching-to-
| tcmal...
| ceeam wrote:
| Well, Linus told you ages ago that the "fundamental design
| principle of Linux is to have fun". So they change stuff like
| this for fun. Isn't it fun for you too? And I will stick to
| FreeBSD, thank you.
|
| And yes, I understand that technically glibc is not a part of
| "Linux", but...
| SilasHaslam wrote:
| I read the headline as: When alligators are boarding your
| precious memories. Sigh..
| pwagland wrote:
| We once ran into a very similar issue, but using the old glibc
| allocator in bionic. For us the difference was between JVM
| versions/vendors. The arenas in glibc all allocated memory, based
| on what thread they are in. Because threads in Java are somewhat
| ubiquitous, then this caused massive "memory leaks", read
| allocated memory that wasn't being used, as every allocator was
| getting used, which caused them to all build up their buffers.
| This in turn caused a large increase in AWS bills, as we had to
| increase the memory allocated to avoid the OOMK.
|
| We ended up moving to tcmalloc, mostly because the Debian
| packages worked the best for us. we did testing between the
| different malloc replacements, and honestly, there wasn't a huge
| difference between them for our workloads, although they _did_
| have differences, with minor variations in peak/avg memory usage
| and CPU usage.
___________________________________________________________________
(page generated 2023-10-06 23:01 UTC)