[HN Gopher] Linux Memory Overcommit (2007)
___________________________________________________________________
Linux Memory Overcommit (2007)
Author : signa11
Score : 13 points
Date : 2024-08-17 01:10 UTC (1 days ago)
(HTM) web link (opsmonkey.blogspot.com)
(TXT) w3m dump (opsmonkey.blogspot.com)
| pcwalton wrote:
| Reminder: if overcommit as a concept seems distasteful, the real
| ire should be directed at the Unix fork syscall, an API that will
| always fail on a process using over 50% of the available memory
| without overcommit. Ideally apps would use vfork or posix_spawn
| instead, but that isn't the world we live in. Overcommit is a
| sensible way to help apps that use a lot of memory "just work".
| You can always turn it off if you don't need apps using a lot of
| memory to be able to fork.
| senderista wrote:
| There are also legitimate uses of overcommit for reserving huge
| areas of virtual memory and relying on demand paging to
| allocate physical memory, although strictly speaking enabling
| overcommit isn't necessary if you separate the "reserve"
| (mmap(PROT_NONE)) and "commit" (mmap(MAP_FIXED, PROT_READ |
| PROT_WRITE)) stages (which is what Windows forces you to do).
| dap wrote:
| I'm not sure how this works on Linux exactly but in principle
| you don't need to support overcommit to avoid that problem. On
| illumos for example anonymous memory allocations come from
| swap, which is a virtual resource that includes both disk and
| memory. You do still have this problem if something is using
| more than half of swap, but (1) that's much larger, and (2) you
| can augment it with more disk space, which won't actually be
| used unless stuff actually uses all the physical memory. This
| has its own tradeoffs but it's useful to keep in mind that
| there are other ways to do things.
| wmf wrote:
| If available memory means physical + swap then you're saying
| the same thing as pcwalton.
| thanatos519 wrote:
| I thought fork was copy on write nowadays.
| layer8 wrote:
| It is, but "write" doesn't have an error code it can return
| on out-of-memory.
| wmf wrote:
| Fork is copy-on-write. The problem is that there's no way for
| fork() to know how much memory the child process needs. It
| might call exec() immediately, using very little memory. It
| might keep running, mostly sharing pages with the parent. It
| might touch every memory page, requiring a lot of memory.
| __turbobrew__ wrote:
| If you want to optimize for throughout one option is to
| overcommit resources (CPU/MEM/IO/NET) but utilize backpressure
| mechanisms to reduce load during times of saturation.
|
| Kubernetes does this through node pressure eviction but it is
| pretty easy to hook into the pressure stall information and have
| the application handle this as well (for example, start returning
| 429 HTTP responses when PSI goes over a certain level).
|
| At the end of the day the optimal overcommit is workload
| dependent -- good metrics and an iterative approach is needed.
| johnea wrote:
| Is any of this still applicable in 2024?
|
| One has to doubt...
| nabla9 wrote:
| As then as now, you can manage it.
|
| vm.overcommit_memory == 0 (default) is heuristic overcommit, with
| 'vm.overcommit_ratio' you can set up the limits.
|
| vm.overcommit_memory == 1 (full overcommit) allows allocating
| more memory than there is ram + swap.
|
| vm.overcommit_memory == 2 (disable) Memory allocation works only
| if there is enough physical ram or virtual swap to allocate the
| memory.
| layer8 wrote:
| Can this be set for a specific process (or process group/tree),
| or only globally?
___________________________________________________________________
(page generated 2024-08-18 23:00 UTC)