[HN Gopher] There is an OOM kill count in Linux
___________________________________________________________________
There is an OOM kill count in Linux
Author : kiyanwang
Score : 53 points
Date : 2023-12-31 12:49 UTC (10 hours ago)
(HTM) web link (medium.com)
(TXT) w3m dump (medium.com)
| Gys wrote:
| Not mentioned in the post but OOM = Out Of Memory
| Elias-Braun wrote:
| Thank you, this was my question after reading this : )
| amelius wrote:
| The reader should probably also know about the OOM-killer:
|
| https://neo4j.com/developer/kb/linux-out-of-memory-killer/
| dumbfounder wrote:
| Than never?
| stavros wrote:
| In the order of preference, it goes "now", then "never".
| Presumably "later" is even later, so if you don't find it now,
| you'd better never find it.
| contrarian1234 wrote:
| Out of curiosity, what do people recommend for OOM these days on
| Linux?
|
| My number one recurring issue on Ubuntu is me not noticing the
| memory usage, running out of memory and locking up my system
| (forcing a hard reboot and loss of unsaved work)
|
| EDIT: I'd preferably like an OOM handler that would freeze my
| system and pop up a little menu from which I could select which
| process to nuke
| stathibus wrote:
| early-oom
| manymatter wrote:
| I've never had that problem. Maybe you don't setup much swap
| space as I would expect the system to run well enough to save
| your work and shutdown for most common leaks.
| gkbrk wrote:
| Swap usually makes it worse, without swap there is some
| chance that the Linux OOM killer does something useful and
| saves the system. With swap, it becomes a frozen system that
| never manages to kill anything due to all the swapping. You
| can wait 5 minutes, 10 minutes, or 15 minutes, but the system
| never recovers without a reboot.
| manymatter wrote:
| That would normally mean you ignore the system getting
| slower for a long time.
| gkbrk wrote:
| Not a long time. With swap enabled, when a process
| consumes too much memory your system goes from perfect
| performance to cursor lagging to everything is frozen and
| you can't even switch to a TTY within 5-10 seconds.
|
| Without swap, the system lags for a couple seconds, OOM
| killer frees up memory and you're good to go again. The
| only slowdown is any pages that were kicked out from the
| file cache. But those quickly come back after the OOM
| killer does its thing.
| manymatter wrote:
| What if the thing you kill is in the critical stack to
| saving your work? If it isn't I don't really understand
| why you would be swapping it in a lot
|
| I would view the OOm solution as a compute as cattle
| thinb, but here we are talking about a user desktop where
| the user can take the best action for themselves once
| they realize there's a problem.
| gkbrk wrote:
| > If it isn't I don't really understand why you would be
| swapping it in a lot.
|
| The user doesn't decide which processes are swapped in.
| If the process gets CPU time and tries to access its
| data, that data will get swapped in.
|
| > We are talking about a user desktop where the user can
| take the best action for themselves once they realize
| there's a problem.
|
| You can't do that with swap, because once you realize
| there's a problem, you cannot even move your cursor or
| run commands to take any actions.
| manymatter wrote:
| This sounds nothing like the gradual leak problem
| described.. An OOM killer is great once you actually run
| out of resources, removing virtual resources to always
| run out and play roulette is using it as a fad hammer.
| grumbel wrote:
| You have only about 10sec between the system getting
| slower and the system locking up completely. If you
| manage to hit the Magic SysRq key combination to trigger
| OOM manually, that can save the system, but you have to
| be quick.
|
| [1] https://en.wikipedia.org/wiki/Magic_SysRq_key
| ynik wrote:
| That only works if the system is accessing mostly anonymous
| pages. If the load on the system is accessing plenty of
| mmapped code/data pages, it can still trash those even if
| swap is disabled. I've still seen systems hanging for 10+
| minutes without recovering even though swap was already
| disabled.
|
| The Linux kernel OOM killer only acts if there's nothing
| left that can be discarded, which often happens way too
| late to save the system. You need a user-mode OOM-killer
| like earlyoom if you want to keep the system responsive.
| gkbrk wrote:
| Yes, I don't disagree that user-mode OOM-killers are
| helpful, or that the system can still hang without swap.
|
| I'm just saying turning on swap, or increasing the swap
| capacity does not fix the problem, and it usually makes
| it worse.
| lokar wrote:
| This is why on production systems important binaries
| should mlock all of their code pages into ram at startup.
| chrismorgan wrote:
| My previous Linux laptop with 8GB of RAM (2014-2017 or so, I
| think) I never even got round to setting up swap, and I only
| ran into problems two or three times total (when running two
| or three Firefox instances _and_ Chromium _and_ a VM taking
| 2GB _and_ a bunch of other things running--some of the
| biggest consumers, namely browsers, actually notice if
| they're using too much of your RAM and adjust how they work
| so they don't ask for so much RAM, at what I believe is a
| fairly slight performance cost).
|
| On my current laptop and with my current habits, I'm
| consuming a lot more memory, and so the technique I use to
| avoid OOM is simply having 40GB of RAM. (As it happens, I do
| actually have swap set up at present, because certain
| circumstances meant I wanted to hibernate it occasionally;
| should disable swap again now I'm back to not needing to
| hibernate.)
| TheDong wrote:
| systemd-oomd [0].
|
| You can also use facebook's oomd, but the systemd one is
| basically the same and slightly better.
|
| If you want, you can also write your own custom memory pressure
| handler using the recent PSI features:
| https://docs.kernel.org/accounting/psi.html
|
| The other, easier, answer is to buy more ram, if you simply
| have more memory than you will ever use (such as having 128GiB
| of memory when you only regularly use 30GiB), you'll rarely run
| into OOM issues.
|
| [0]:
| https://www.freedesktop.org/software/systemd/man/latest/syst...
| melvyn2 wrote:
| For desktop use, nohang does what the name says.
|
| https://github.com/hakavlad/nohang
| magicalhippo wrote:
| This is one area where Windows is far superior to Linux as a
| desktop experience.
|
| By accident, I once wrote an infinite loop that just allocated
| a bunch of memory. As I ran the program my Windows system
| quickly became laggy and unresponsive. However, not completely.
| With a bit of patience I got Task Manager up and managed to
| kill the process and the system was back to normal within a
| minute.
|
| On Linux, I experience the same as you. Everything is fine
| until suddenly it's completely unresponsive. Most of the time I
| can't even manage to shut down the system cleanly and have to
| hard reset.
| gnramires wrote:
| Indeed! I wonder why the system can't just stop giving the
| process memory when it's starting to become critical? (say,
| you're over 95% memory used)
|
| I really think a more general solution is treating memory
| (and CPU time as welL!) as a scarce resource, and programs
| should either: (1) Deal with having severely denied resources
| (default behavior); (2) Use a communication protocol to
| negotiate memory with the system.
|
| Negotiation could mean (a) The process freeing memory
| spontaneously when there's pressure (another process requires
| it) and it's inactive, (b) Requesting the system free memory
| from other processes when user requires.
|
| This would impose more memory management burden on developers
| (trying to fulfill OS requests), but in turn it would make
| for a far better experience when there's memory contention.
|
| Disclaimer: I have no idea how memory allocation works in
| detail. Maybe this is reinventing the wheel?
| willcipriano wrote:
| > I wonder why the system can't just stop giving the
| process memory when it's starting to become critical
|
| If you turn swap off the allocator will error out when you
| are out of memory. I don't think most software handles that
| error but all the wiring is there to do that.
| roblabla wrote:
| > Maybe this is reinventing the wheel?
|
| I believe something like that exists on mobile platforms
| for memory, at least on iOS you get a message
| (applicationDidReceiveMemoryWarning:) when the system is
| memory starved and wants your app to free memory. If you
| don't release enough memory and the memory pressure doesn't
| go down, the system will start killing apps.
| alpaca128 wrote:
| And on Mac OS you'll get a popup dialog that lets you
| select programs you want to force quit to free memory.
| Which is a nice idea as everything else stays responsive,
| unfortunately it doesn't list any system processes so
| it's useless if one of them has a bug - which is the
| reason I even learned of the existence of this oom
| dialog.
| kaba0 wrote:
| Aren't it simply more common for apps to go into a "save
| state and basically exit" mode, from which they are
| expected to recover without any fuss? Like, ios is very
| aggressive in "killing" background apps, and since this
| is the general consensus, apps just behave nicely the
| majority of time.
| samus wrote:
| Also on Android there are explicit guidelines that
| processes have to be prepared to be paused, hibernated,
| or killed at any time. Data in memory is _not_ safe in
| any way.
| hnfong wrote:
| Most software doesn't really handle memory allocation
| failures too well.
|
| malloc() can fail, everyone knows this in theory, but
| assume you're the programmer handling the malloc() error --
| you know the system is likely to have already run out of
| free memory -- how do you make the program fail gracefully
| without doing _anything_ at all to allocate memory?
|
| It's not even safe to call printf() since it might have to
| allocate a new string. There are very few things in modern
| code that doesn't "incidentally" allocate a couple bytes
| here and there.
|
| Not to mention languages higher level than C where you
| don't directly call malloc() but have the language handle
| it for you (GC etc.). Handling OOM for those programs is
| even more impossible because you really have no idea
| whatever class/method/function you're calling is going to
| trigger an allocation or not.
|
| The best you can do in those cases is try to commit
| critical data (if any) to disk and have the program die
| ASAP. Which is not _that_ different from having the kernel
| kill it for you.
| weebull wrote:
| The "attempt to save critical data" is a pretty major
| difference.
| samus wrote:
| This is already the case. Processes have to ask politely
| for memory.
|
| The OS could really just say no. But that would commit
| memory to a process even though it might not actually end
| up using it, making it unavailable for processes that need
| it _right now_. Also, memory allocation latency is higher
| since the OS has to really do all the bookkeeping up front.
| Therefore, most Linux distributions by default
| optimistically grant allocation requests even though there
| might not be enough memory available yet.
|
| The big disadvantage is that the OS might be unable to
| actually grant memory. The process then gets OOM killed.
| This is sort of fine on servers since users usually get
| assigned limits. But there is something left to be desired
| for single-user systems.
|
| Processes can at any time return memory to the OS when they
| don't need it. Usually they do that, but this has the
| disadvantage that they have to ask the OS again for it,
| which is quite slow compared to keeping it around.
|
| Few applications are actually able to just release memory
| on request. It's critical user and internal application
| data after all. Databases and applications with garbage
| collectors come closest.
| GTP wrote:
| On the other hand, at least in my experience, Windows becomes
| laggy much sooner than Linux. IMO which of the two behaviors
| is better really depends on the use case.
| kaba0 wrote:
| I don't think you can make an apples to oranges comparison
| here, there are too many differences. But sure, linux is
| mostly server-workload oriented.
| samus wrote:
| Since Linux has Unix-legacy and is usually intended to be
| used on servers, there is one well-intended assumption at
| play here: unless different settings are applied, all
| processes are in principle created equal. This also applies
| to GUI applications, and means that a single out-of-control
| application can lock up the whole system.
|
| That's actually fine on multi-user systems. A competent admin
| would set up tight limits and prevent one user taking over
| all resources, and intervenes if they mess up. On single-user
| machines, this limitation is not there because there is
| effectively only one user. And since desktop Linux is not a
| priority for most Linux vendors, this scenario sees
| comparatively little consideration.
|
| In comparison, on Windows some applications (at least
| explorer.exe and the task manager) seem to have way higher
| priority. No matter what else is going on, the user must be
| able to use these application to rein in other applications.
| The drawback is that there is no recourse should explorer.exe
| ever lock up.
| Dalewyn wrote:
| Explorer and Task Manager are both shells (along with their
| CLI sibling shells CMD and Powershell), but Task Manager is
| special in that Windows Logon is always actively monitoring
| for CTRL+ALT+DEL.
|
| Windows Logon is superior to all the shells as a core
| Windows subsystem in that it is the one that executes the
| shell (usually Explorer) on boot/login and can also execute
| Task Manager, in addition to its primary duties of tracking
| and securing user logins (hence the name).
|
| This means even if Explorer has completely froze or has
| crashed, you can always get to Task Manager via Windows
| Logon and then kill or execute another instance of Explorer
| (or CMD or Powershell!) or otherwise force reboot the
| system from Windows Logon.
| kaba0 wrote:
| Linux has something similar to ctrl+alt+del monitoring in
| the form of REISUB and alia. But it is much more crude
| (and nowadays mostly disabled by default).
| psyclobe wrote:
| I've noticed this too, make -j and boom hard lock only option
| is to reboot. Really annoying l! Windows kinda deals with it
| but it has other issues; I have also noticed this behavior on
| MacOS.
| samus wrote:
| It should be possible to avoid it by using ulimit to set
| memory and subprocess limits.
| Kwpolska wrote:
| Buying more RAM?
| danieldk wrote:
| At the risk of this being tongue-in-cheeck: if you have a
| rogue process, any amount of RAM is going to be eaten up.
|
| (It sometimes happens in software development and it's nice
| if it doesn't take your machine in swap hell as a result.)
| Toorkit wrote:
| I recently installed Linux on my main desktop again. After
| freezing up twice in a week, I looked around.
|
| I enabled systemd-oom, but it hasn't been necessary, because I
| also enabled zram as a swap device. I've seen it compress
| multiple gb of ram into a few hundred mb and haven't frozen my
| desktop since.
| seba_dos1 wrote:
| You want to enable thrashing prevention:
| https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm...
|
| Make sure you use zram (or zswap) as well. May also consider
| enabling userspace oom handler like others already suggested,
| but it's less important than the rest.
| mgdlbp wrote:
| Enable Alt+SysRq+F and do it manually. Works in seconds every
| time; the kernel can take days because it's not eager enough
| when thrashing.
|
| Can have security implications, but the target is chosen in the
| typical way and screen locks should be setting a score_adj to
| avoid being picked.
| wakeupcall wrote:
| I'm using earlyoom on desktop, systemd-oomd on servers.
|
| Due to cgroupv2, systemd-oomd tends to kill entire process
| groups instead of a single run-away process [1]. It's perfect
| for servers where almost every process is part of a system
| service already.
|
| For desktop usage I found this behavior unacceptable. I would
| sometimes get my entire X session killed when trying something
| dumb (which is the reason I run such a daemon in the first
| place), or my editor when spawning a helper subprocess...
|
| earlyoom was the simplest daemon of the bunch I tried, and
| worked out of the box without special configuration.
|
| [1] Or at least this was the state ~1yr ago.
| grumbel wrote:
| More RAM. It's the only thing that can reliably prevent it.
| Magic SysRq key[1] can trigger the OOM manually, which can
| sometimes rescue the system if you notice it soon enough. But
| if you run into the situation regularly under normal use, not
| programs running amok, more RAM is a must.
|
| If you can't upgrade the RAM, zram[2] is also worth a try.
|
| [1] https://en.wikipedia.org/wiki/Magic_SysRq_key
|
| [2] https://en.wikipedia.org/wiki/Zram
| marcodiego wrote:
| Is this still the case on a modern distro using MGLRU and
| systemd-oomd?
| seba_dos1 wrote:
| Not really, but not all distros have caught up yet.
| DefineOutside wrote:
| Look into oom_kill_allocating_task. There's likely a better
| solution such as an earlier OOM, but killing the wrong process
| seems better than locking the system.
| larve wrote:
| In a previous embedded job, I built a daemon watching a couple
| of "at risk of eating too much memory" processes that were not
| critical for the actual function of the machine, and killed
| them proactively when memory started getting low. This
| prevented the system from going into starvation and spending 10
| seconds doing whatever each time an Ethernet packet came in.
|
| As for my pragmatic solution at home: max out the physical
| RAM...
| mgdlbp wrote:
| Tangentially, an issue with the desktop use case is that the
| main process of a long-running browser is often the first to be
| killed, instead of one of its content processes, which it can
| survive losing. This is the case since 2019 as the kernel no
| longer prefers killing children.[0]
|
| As far as I know, unless a browser manages its oom_score_adj,
| which none do, a wrapper is insufficient as a workaround and a
| daemon would have to do so on its behalf, because the value is
| inherited when forking.
|
| [0]
| https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
| mattpallissard wrote:
| Wrap everything in systemd-run. I documented some of it here on
| my shitty site. Some of the memory limits have changed over
| time, notably the browsers unit
|
| https://pallissard.net/2022/06/27/limiting_application_resou...
|
| Edit: I like your user space oomd idea though, that would be
| killer
| unixhero wrote:
| Kool, but what are OOMs?
| ChatGTP wrote:
| https://en.wikipedia.org/wiki/Out_of_memory
| yonatan8070 wrote:
| OOM stands for Out Of Memory, it's a daemon that kills
| processes when the system runs out of RAM
| dang wrote:
| Related: _I wish Linux exposed a 'OOM kills due to cgroup limits'
| kernel statistic_ -
| https://utcc.utoronto.ca/~cks/space/blog/linux/OOMFromCgroup...
|
| (via https://news.ycombinator.com/item?id=37640389, but no
| comments there)
___________________________________________________________________
(page generated 2023-12-31 23:03 UTC)