https://lemire.me/blog/2021/10/27/in-c-how-do-you-know-if-the-dynamic-allocation-succeeded/ Skip to content Daniel Lemire's blog Daniel Lemire is a computer science professor at the University of Quebec (TELUQ) in Montreal. His research is focused on software performance and data engineering. He is a techno-optimist and a free-speech advocate. Menu and widgets * My home page * My papers * My software Subscribe Email Address [ ] [ ] [Subscribe by email] You can also follow this blog on telegram. Where to find me? I am on Twitter and GitHub: Follow @lemire You can also find Daniel Lemire on * on Google Scholar with 4k citations and over 75 peer-reviewed publications, * on Facebook, * and on LinkedIn. Before the pandemic of 2020, you could meet Daniel in person, as he was organizing regular talks open to the public in Montreal: tribalab and technolab . Search for: [ ] [Search] Support my work! I do not accept any advertisement. However, you can support the blog with donations through paypal. Please consider getting in touch if you are a supporter so that I can thank you. Recent Posts * Stop spending so much time being trolled by billionaire corporations! * Science and Technology (October 31st 2021) * In C, how do you know if the dynamic allocation succeeded? * In C++, is empty() faster than comparing the size with zero? * Science and Technology links (October 23rd 2021) Recent Comments * M. on In C, how do you know if the dynamic allocation succeeded? * tdiff on In C, how do you know if the dynamic allocation succeeded? * Daniel Lemire on A fast 16-bit random number generator? * Erik van Oosten on Stop spending so much time being trolled by billionaire corporations! * James Houx on A fast 16-bit random number generator? Pages * A short history of technology * About me * Book recommendations * Cognitive biases * Interviews and talks * My bets * My favorite articles * My favorite quotes * My readers * My sayings * Predictions * Recommended video games * Terms of use * Write good papers Archives Archives [Select Month ] Boring stuff * Log in * Entries feed * Comments feed * WordPress.org In C, how do you know if the dynamic allocation succeeded? In the C programming language, we allocate memory dynamically (on the heap) using the malloc function. You pass malloc a size parameter corresponding to the number of bytes you need. The function returns either a pointer to the allocated memory or the NULL pointer if the memory could not be allocated. Or so you may think. Let us write a program that allocates 1 terabytes of memory and then tries to write to this newly allocated memory: #include #include int main() { size_t large = 1099511627776; char *buffer = (char *)malloc(large); if (buffer == NULL) { printf("error!\n"); return EXIT_FAILURE; } printf("Memory allocated\n"); for (size_t i = 0; i < large; i += 4096) { buffer[i] = 0; } free(buffer); return EXIT_SUCCESS; } After running and compiling this program, you would expect to either get the message "error!", in which case the program terminates immediately... or else you might expect to see "Memory allocated" (if 1 terabyte of memory is available) in which case the program should terminate successfully. Under both macOS/clang and Linux/GCC, I find that the program prints "Memory allocated" and then crashes. What is happening? The malloc call does allocate memory, but it will almost surely allocate "virtual memory". At the margin, it could be that no physical memory is allocated at all. The system merely sets aside the address space for the memory allocation. It is when you try to use the memory that the physical allocation happens. It may then fail. It means that checking the memory allocation was a success is probably much less useful than you might have imagined when calling malloc. It also leads to confusion when people ask how much memory a program uses. It is wrong to add up the calls to malloc because you get the virtual memory usage. If you use a system-level tool that reports the memory usage of your processes, you should look at the real memory usage. Here is the current memory usage of a couple of processes on my laptop currently: process memory (virtual) memory (real) qemu 3.94 GB 32 MB safari 3.7 GB 180 MB Dynamic memory is allocated in pages (e.g., 4 kB) and it is often much smaller than the virtual memory. Doing "malloc(x)" is not the same as taking x bytes of physical memory. In general, it is therefore a difficult question to know whether the allocation succeeded when relying on malloc. You may only find out when you write and read to the newly allocated memory. Published by [2ca999] Daniel Lemire A computer science professor at the University of Quebec (TELUQ). View all posts by Daniel Lemire Posted on October 27, 2021October 27, 2021Author Daniel Lemire Categories 10 thoughts on "In C, how do you know if the dynamic allocation succeeded?" 1. [fe4f33] roystgnr says: October 27, 2021 at 4:36 pm And shared libraries might be counted in both real and virtual memory for every process that uses them even though they're taking up the same pages of read-only or copy-on-write memory, and memory mapped files might be counted in their entirety in virtual memory even if only a small section of the file ever gets read, and on Linux the Out-Of-Memory killer might choose to kill a *different* process when your process tries to get real access to overallocated virtual memory, and the C shared library might not *really* release the memory you free() because it's faster to keep it around to avoid hitting up the kernel when you next try to malloc(), and none of this stuff is set in stone in a standard so for all I know anything or everything I've just recalled might be years out of date... I hate this stuff. At least heap allocation all ends up going through a few bottleneck APIs, so you can get a vague handle on memory usage optimization with an LD_PRELOAD to intercept and tally those calls. I was originally going to comment that it was disappointing when your blog post didn't really answer the question posed in your blog title, especially compared to your usual detailed answers to these sorts of questions. But I guess that's not at all your fault; the best answer we can get might really be: if nothing crashes later then the allocation must have succeeded in some sense? I assume this is why some embedded systems people end up trying to just keep everything on the stack or in static globals. Reply 1. [2ca999] Daniel Lemire says: October 27, 2021 at 4:42 pm Thanks for the comment. I do not answer my own question because it is, as you remarked, non-trivial, at least if you rely on purely standard C. Reply 2. [2a52be] Georg Nikodym says: November 3, 2021 at 4:29 pm In embedded: - we rarely have a lot of memory and not all of it has the same properties - standard C doesn't have a way to interrogate your heap (or stack) utilization... - other platform specific stuff (heck, you might not even have an allocator) - debugging can be quite challenging all contributing to the "odd" ways we write C code. Reply 1. [5e9d03] M. says: November 5, 2021 at 10:03 pm Also in embedded we aren't sharing the memory with other processes so it's all ours from the get go. Reply 2. [4f6c84] Alex says: October 27, 2021 at 4:54 pm This actually has more to do with how "overcommit" is set up on the system, than with the difference between virtual and physical memory. You may try the same on a Linux system with overcommit turned off. This is done with /proc/sys/vm/overcommit_memory unless I remember wrong. Or, try on a Windows system - Windows does not allow overcommit (but still uses the same virtual/physical memory design). The VM subsystem knows perfectly well that you're allocating more than it can deliver, despite the memory being virtual. Overcommit was originally allowed to ease porting of some older software to Linux. Today it may make sense if a process uses vast address space for IO - that is not backed up by actual memory pages - but in general, turning overcommit off is a good thing for development. Makes finding memory related issues a lot quicker. Reply 3. [1e188f] Raivokas Ripuli says: October 27, 2021 at 7:59 pm This is operating system issue. Linux has 3 overcommit modes, heuristic (default), always, and never. https://www.kernel.org/doc/Documentation/vm/overcommit-accounting Reply 1. [78de6d] tdiff says: November 5, 2021 at 8:26 pm Interestingly, it says "Obvious overcommits of address space are refused." Does it mean we can observe malloc failure in this mode? Reply 4. [57cded] Alexander Adler says: October 28, 2021 at 1:57 pm I tried on my box: If /proc/sys/vm/overcommit_memory is zero, the process exits cleanly with "error!"; if it is one, the process is terminated by the OOM killer after some time (my Laptop does not have 1T Normally, I have /proc/sys/vm/overcommit_memory set to zero. As the other Alex said, it is convenient for developing. Reply 5. [30de1f] Jakub says: October 28, 2021 at 2:56 pm Recommend taking a look here for anyone who is interested in some more details about overcommit and OOM killer. Reply 1. [30de1f] Jakub says: October 29, 2021 at 9:38 am Link got removed, pasting here again: https://www.win.tue.nl/ ~aeb/linux/lk/lk-9.html#ss9.6 Reply Leave a Reply Cancel reply Your email address will not be published. The comment form expects plain text. If you need to format your text, you can use HTML elements such strong, blockquote, cite, code and em. For formatting code as HTML automatically, I recommend tohtml.com. [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment [ ] Name * [ ] Email * [ ] Website [ ] [ ] Save my name, email, and website in this browser for the next time I comment. Receive Email Notifications? [no, do not subscribe ] [instantly ] Or, you can subscribe without commenting. [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] You may subscribe to this blog by email. Post navigation Previous Previous post: In C++, is empty() faster than comparing the size with zero? Next Next post: Science and Technology (October 31st 2021) Proudly powered by WordPress