[HN Gopher] Why MMAP in llama.cpp hides true memory usage
___________________________________________________________________
Why MMAP in llama.cpp hides true memory usage
Author : flurly
Score : 114 points
Date : 2023-04-03 16:27 UTC (6 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| btown wrote:
| I mean, if we're being pedantic, I think it is _possible_ to run
| llama.cpp with real memory savings this way... in a contrived
| situation where you were only generating a single token with a
| single forward pass, and you were limited in RAM.
|
| The new MMAP system would not require you to load the whole model
| from disk up front, but rather would allow you to load and
| immediately forget each layer of parameters as you do a forward
| pass through the GPT architecture. And if you had just tried to
| use normal swap paging to do this without MMAP, you'd essentially
| be doing 2x the loading work, since the page for the first layer
| likely got swapped out as you loaded the rest of the model into
| "memory."
|
| Of course, as soon as you want to generate a new token, you'd
| need to reload all the pages of the model parameters. Every token
| generated would take as long as it would if you were loading the
| model from scratch.
|
| But for a classification problem where you only need to generate
| one token, and you need or want to do this in a serverless
| environment where you'd need to load from disk anyways? I think
| you'd be able to do that workflow with significantly less RAM in
| the same amount of clock time, now.
| alchemist1e9 wrote:
| Is it possible to do multi-GPU inference the way you describe
| and have say 4 GPUs but each doing different layers on a large
| model?
|
| If that is the case then perhaps a high end consumer system
| with max pcie 5 nvme bandwidth and multi gpus could do
| inference on large LLM models.
| rfoo wrote:
| Congratulations on rediscovering how the original researchers
| run these models!
| alchemist1e9 wrote:
| Do they do it in a way that each GPU can have smaller
| memory than the model weights?
|
| I keep seeing references to these absolutely massive 48GB+
| gpu memory sizes and those are extremely expensive. If more
| but smaller memory gpus can work that could open up a much
| larger range of hardware and individuals to run inference,
| especially if efficient load/unload of weight layers from
| raid pcie5 nvme devices is feasible.
| rfoo wrote:
| Yes?
|
| The original LLaMA-65B on fp16 weight is 130G. There are
| no NVIDIA GPU with 130G memory.
|
| As for why they don't use int8/int4: would you try to
| optimize your code before you confirmed it can actually
| run?
| alchemist1e9 wrote:
| Got it, so looking ahead to building hardware that is
| useful for inference would you agree with these
| considerations:
|
| - max PCI lanes for more GPUs. AMD cpus and boards
| probably better
|
| - PCIe 5 nvme. ideally enough to saturate bus
|
| - max GPUs with max memory. memory more important than
| FLOPs probably.
|
| for a given budget one tries to get as much as possible
| of above.
|
| How long before a consumer component machine for less
| than $10K can do inference for a gpt 3.5 turbo quality
| model?
| visarga wrote:
| > Every token generated would take as long as it would if you
| were loading the model from scratch.
|
| Wouldn't make sense to keep most of the model in GPU (as much
| as it fits) and only load the remaining layers on each pass?
| Animats wrote:
| Well, yeah.
|
| Short version: the claim that llama could use much less memory by
| memory-mapping the model data file was wrong. It's just that the
| Linux utilities for memory use don't count memory-mapped file
| area.
| pavon wrote:
| I'm not convinced that there isn't more to the story than that.
| People (including Justine who implemented MMAPing the model and
| knows how to monitor memory use properly) are seeing that much
| of the file isn't being paged into memory, and are not quite
| sure why[1]. Is it a bug? Is the distribution of weights used
| highly dependent on the prompts? Are some weights somehow
| unused altogether? This twitter thread rules out some of those,
| but I don't think it closes the door.
|
| [1] https://news.ycombinator.com/item?id=35393615
|
| edit: softened some of my claims after reading more updates
| from over the weekend.
| Animats wrote:
| Now that's interesting. Entire memory pages of the model
| aren't being referenced?
| sitkack wrote:
| eBPF
|
| echo 3 > /proc/sys/vm/drop_caches
| wmf wrote:
| Other people are saying that the whole file does get paged in
| and transformers access all their weights by design.
| peterfirefly wrote:
| The kernel will read more than just the page that faulted.
| Jart counts just the page faults and multiplies by the page
| size.
| astrange wrote:
| File backed memory is "less" than heap memory, because it can
| be thrown away when needed instead of being swapped out to
| disk.
|
| And because reading the file probably allocated file backed
| pages for it anyway unless you used O_DIRECT (and profiled it
| first.)
|
| There's also wired/mlock() memory which is "worse" than regular
| memory because it can't be swapped/evicted at all.
| thatcherc wrote:
| Good thread!
|
| https://threadreaderapp.com/thread/1642726595436883969.html
| garganzol wrote:
| mmap does not hide the real RAM usage, it just maps a part of
| virtual memory address space to an I/O-bound storage medium (e.g.
| file). That trick does not consume RAM in any significant
| amounts, so there is nothing to hide.
|
| However, such image mapping technique is not always suitable for
| data-intensive workloads due to page thrashing. If it works for
| llama.cpp then it can be considered a huge success.
| ImprobableTruth wrote:
| >mmap is a really nifty feature of modern operating systems
|
| ... for some definition of "modern".
| [deleted]
| koito17 wrote:
| I wonder how the author would label Windows' MapViewOfFile :)
| lionkor wrote:
| you dont get the devs who spend their day on twitter instead of
| writing code to read your thread if it doesnt mention "modern"
| at least once. /s
|
| Its quite obvious that the author "dumbs down" all the
| information for a very non-developer audience, so I assume that
| gives some leniency. If you tell people that we have had things
| like mmap() for decades, they may start asking too many
| questions about why every piece of software underperforms so
| horribly below what was possible decades ago.
|
| Bit of a rant, but I feel that we lose a little bit of
| potential every time a developer calls operator<< on a
| std::istream in a loop.
| CoolGuySteve wrote:
| To be fair I find it completely baffling that R, Pandas, and
| all sorts of other tools make it difficult or impossible to
| mmap a column to a dataframe.
|
| Instead you have to do backflips to load data that doesn't
| fit in RAM.
| verall wrote:
| What's wrong with operator<< on std::istream?
| chaxor wrote:
| Devs of 2023: "why does this wheel have rubber on it? It
| needs to be more modern. Let's take away the rubber and add
| firecrackers in its place as a nice feature, so really pops
| and makes the experience more exciting"
| munk-a wrote:
| The same can be said of devs of 2010 and devs of 2000.
| There are constant innovations and trends in technology and
| there are also some very core, generally very old and
| extremely critical components in the technology stacks we
| all use - people will see that these components don't use
| "teh hotness" and assume it would work better with the
| paradigm that's in favor at the time. Sometimes these
| people are right, sometimes they're wrong - we shouldn't
| discourage questioning the status quo as, critically,
| _sometimes these people are right_ - but it is important to
| be cautious with such changes since generally there are a
| lot of very precise decisions that have been carefully
| considered. However, one need not look any further than
| OpenSSH to see how very serious bugs can exist in
| "unimpeachable" software that everyone just assumes it's
| safe to use.
|
| This is, as an aside, the reason why I don't comment my
| code often (I personally dislike docblocks that do nothing
| but obvious explanations) - but whenever I think hard about
| a thing I always record it in a comment with the problem,
| my reasoning, and my rational for choosing the solution
| used... i.e. don't comment on the what, comment on the why.
| malux85 wrote:
| Chesterton's fence,
| https://en.wiktionary.org/wiki/Chesterton%27s_fence
|
| You may consider removing the rubber only when you can
| adqueately explain to me why it is there
| antonvs wrote:
| Does this mean DEC's TOPS-20 from 1969 is a modern OS?
| gumby wrote:
| Mmap was the only way to do disk I/O in Multics -- we're
| talking about a design a decade earlier than that.
|
| (Also you're thinking of TOPS-10 -- TOPS-20 and TWENEX were
| developed in the 70s. But your heart is in the right place!
| riedel wrote:
| Best part: >AI-generated summary:
|
| >"This thread explains how the mmap feature of modern operating
| systems can be used to reduce memory usage when running large
| deep learning models. It also explains how...
|
| Yet I wonder, if this was just another of those humans imitating
| a trustworthy AI...
| bragadiru_mafia wrote:
| Justine has done more to move the image of the trans community to
| "top tier human person and who cares about your identity this
| human being can code like a living God I don't care about fluid
| gender anymore you got me", than any amount of trans human
| billionaires who hate poor people and their pr campaigns.
|
| You go they/them. Ignore haters
| CyberDildonics wrote:
| This isn't why mmap in llama.cpp hides true memory usage, it is
| why mmap hides true memory usage.
| mhh__ wrote:
| Soft page faults considered genius
| flurly wrote:
| TL;DR
|
| > The way mmap handles memory makes it a bit tricky for the OS to
| report on a per-process level how that memory is being used. So
| it generally will just show it as "cache" use. That's why
| (AFAICT) there was some initial misunderstanding regarding the
| memory savings
| vadansky wrote:
| I lost track since things move so quickly. Was there still
| memory savings just not as drastic? Or no memory savings, just
| a speed-up?
| eigenvalue wrote:
| It did somewhat reduce the total memory used. Now you can
| load the 30B model while only using ~20gb of RAM, which is
| about the aggregate size of the 4bit quantized weight files
| for that model. The real win is that you can kill the main
| inference binary and try another prompt, and it will start
| doing inference basically immediately instead of spending
| 10-15 seconds loading up all the weights into RAM each time.
| chpatrick wrote:
| It's neither memory savings or a speed-up really. The
| advantage of mmap is that you can treat a file on a disk a
| block of memory, so pages from it can be loaded (or unloaded)
| as necessary instead of one big upfront load into RAM. The
| benefit is that you can work with data that's bigger than
| your physical RAM because the kernel can swap it back out to
| disk if needed. Another benefit could be that if only a small
| part of the data is needed to compute something then the OS
| will automatically only load those, but it's unclear to me
| whether this is the case with LLaMA.
| toxik wrote:
| Regarding your last point: No, you need all of the weights
| all of the time.
|
| Edit: except embedding weights but those are not the
| problem.
| simion314 wrote:
| It is a speedup for me. When I run llama.cpp from CLI first
| tiem it takes a very long tiem to load the model in memory.
| If the program exits or I stop it with Ctrl+C and start it
| again it will start almost instant.
| chaboud wrote:
| That's down to caching. If you used your system to do
| something else for a while, you'd find those pages
| evicted and the performance back down to Earth. That's
| one of the things that makes mmap so useful, though. The
| system can take advantage of access patterns to
| dramatically improve performance.
| simion314 wrote:
| Yes, makes sense. And is great. Though honestly not sure
| why it takes minutes to load a 23Gb model in RAM, I feel
| is not proportional with the smaller models.
| CyberDildonics wrote:
| They weren't asking about mmap, they were asking about the
| program itself.
| [deleted]
| iforgotpassword wrote:
| It's still somewhat faster if you benchmark it. I assume
| the os is doing good enough prefetching in the mmap case to
| hide the loads from disk mostly. So it's not just hiding
| the initial load of 30gb from disk.
|
| Obviously if you're swapping because you don't have enough
| memory to hold the model in RAM, the mmap version is going
| to be much faster, since you don't need to swap anything
| out to disk but just discard the page and re-read from disk
| if you need it again later.
| antonvs wrote:
| > So it's not just hiding the initial load of 30gb from
| disk.
|
| The issue is typically that that initial load involves
| some sort of transformation - parsing, instantiating
| structures, etc. If you can arrange it so that the data
| is stored in the format you actually need it in memory,
| then you can skip that entire transformation phase.
|
| I don't know if that's what's been done with llama.cop
| though.
| rovr138 wrote:
| It's basically paging to disk.
|
| Not necessarily memory savings, but the improvements here are
| that it will run on computers with less ram because it can
| page to disk.
|
| Not necessarily the number reported (since you do need to
| load chunks into ram), but still lower.
| detrites wrote:
| If that's the case then part of this may be the different
| interpretations of "memory".
|
| One persons "paging the same memory requirement from disk
| to RAM" can be someone elses "requiring less memory/RAM".
| chaboud wrote:
| Sort of, but without the duplication _and_ initial wait to
| load. A traditional fat in-memory app would do this:
|
| file (DISK) to process active pages (RAM) to paged out
| virtual memory if saturated (elsewhere on DISK)
|
| Using mmap typically goes something like:
|
| file (DISK) to process active pages (RAM) to released and
| cached (RAM) to uncached if evicted (back to same place on
| DISK) OR back to process active pages from cache (RAM)
|
| For the cost of fixing up some process page tables, the
| physical memory pages necessary can be brought back from
| the cache rather than read from disk. It's an orders-of-
| magnitude performance savings.
| hnav wrote:
| more like memory mis-reporting, since when you mmap a file
| in, IIRC that counts against page cache rather than memory
| usage (you can evict the page without causing write IO so the
| memory isn't "used")
| astrange wrote:
| It being safe to evict is why it's correct to not report it
| as "memory usage".
|
| It's part of the program's working set but measuring that
| is a completely different story.
| rnnr wrote:
| Memory mapped files also known as sections in VMS/NT have just
| two advantages: * Fewer context switches among
| user space / kernel syscalls * No need to copy
| _modified_ data into the swap. The behavior for read only data
| doesn't change
|
| That's it, nothing miraculous about it.
| astrange wrote:
| No, it also shares them between multiple runs of the same
| process, and it reuses the pages that were in your file cache
| anyway.
|
| > * Fewer context switches among user space / kernel syscalls
|
| This is not necessarily true. There's lots of cases where it's
| actually slower.
| Thaxll wrote:
| People are discovering mmap, it has been used for a very long
| time especially in databases.
| rajnathani wrote:
| Databases usually avoid mmap in favor of their own mmap-like
| system: https://db.cs.cmu.edu/mmap-cidr2022/
| dfox wrote:
| That depends on exactly what kind of DBMS and underlying OS
| one is talking about. For DBMS that runs as one daemon or set
| of related daemons that has some kind of mechanism to share
| the afore mentioned buffer pool and synchronize acces to it
| then sure, it makes more sense, especially as you can do
| fine-grained copy on write in that bufferpool as an
| transaction isolation mechanism.
|
| For embedded databases that operate on shared file somewhere
| just mmap()ing that file allows you to use part of the same
| file as shared memory segment and for POSIX IPC and thus
| synchronization between otherwise unrelated processes that
| access the same data. One big caveat of this approach is that
| it depends on optional parts of POSIX that are realistically
| usable on Linux and Solaris and maybe some obscure systems
| (you can probably do the same thing on Windows NT), notably
| this does not work on essentially anything BSD-derived.
| xiphias2 wrote:
| It's true for databases, but this usage is closer to how shared
| libraries are loaded, just with extenal data.
|
| Also just to show how buggy mmap is, it's disabled in SQLite by
| default for example:
|
| https://www.sqlite.org/mmap.html
|
| Even now the windows version is buggy, but the great thing is
| that after it's fixed in llama.cpp, the open source community
| can just copy the solution.
|
| I don't know of many interactive utilities that were known for
| fast startup time _because_ of the use of mmap, but now we have
| one, which means many more are coming :)
| quotemstr wrote:
| Buggy? In what way?
| masklinn wrote:
| The "But there are also disadvantages" of the linked pages
| provides some of the issues with mmap. This also matches
| burntsushi's experience with mmap in ripgrep:
|
| - depending on concurrent accesses mmap can just sigbus on
| you (e.g. if the mapped file is being truncated)
|
| - mmap simply does not work with virtual filesystems, and
| will blow up on large files on 32b systems (windows also
| has further limitations on mmaps)
|
| - depending on workload mmap may not be faster than regular
| reads and memory buffers, ripgrep will mmap when working on
| just a few files, but will use normal buffers for large
| file counts, because when you start reusing buffer you
| amortise allocation costs which you can't amortise when
| creating and destroying mappings
|
| - not only that but mmap/munmap are also globally blocking
| on the process, so in multithreaded processes it's very bad
| to map/unmap a lot, you can stall your own application
|
| - the semantics of mmap on crash are also somewhat risky,
| in that the OS will try very hard to sync, but that may not
| be desirable if the application crashes in the middle of a
| write
| quotemstr wrote:
| I wouldn't call any of these things a "bug", in the sense
| of a behavior that departs from an explicit or implicit
| specification. Some of them are just advantages and
| disadvantages of that overall approach.
|
| Also, I'm disappointed that VM languages don't turn
| SIGBUS accessing a memory mapped region into a VM
| exception. They could, and it wouldn't even be that hard.
| smeenai wrote:
| https://reviews.llvm.org/D69294 is an interesting case we
| ran into with LLD (the LLVM linker), where the the memory
| pressure from mmap'ing a large output file combined with
| filesystem compression resulted in particularly bad
| performance.
| morelisp wrote:
| It has been used for so long in databases it should no longer
| be used in databases.
|
| http://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
| [deleted]
| rajnathani wrote:
| Just saw your reply, we replied with the same CMU link :)
| mftb wrote:
| They must use it in games too, no? For level loading, assets,
| etc...
| jbverschoor wrote:
| In general game engines are almost full operating systems and
| reimplement many things such as memory management
| eachro wrote:
| What does mmap do exactly? Why was the transition to using it a
| big improvement in llama.cpp?
| programmarchy wrote:
| My understanding is that it maps a file directly to memory to
| reduce disk usage.
| debatem1 wrote:
| Doesn't change disk usage. The file is still on disk.
|
| The difference is between reading a file and memory mapping
| (mmap'ing) it.
|
| If you read a 1TB file into memory you use 1TB of disk and
| 1TB of physical memory. If you then access that data it's as
| fast as RAM because that's where it is.
|
| If you mmap a 1TB file you use 1TB of disk and 1TB of
| _virtual_ memory. If you then access that data it may be
| mapped into virtual memory but not actually be in RAM. This
| triggers a page fault, at which point the correct page is
| loaded from disk to physical memory, and handed back to you.
|
| The key observation is that the amount of physical memory
| occupied by the mmap'd file is much smaller than the entirety
| of the file unless you access almost all of it.
|
| If your filesystem supports holes, this can also be useful
| for writes: it's possible to map files vastly larger than
| physical disk space, but so long as the actual number of
| places written to is quite small you won't run out.
|
| The combination is very useful for datastructures because you
| basically don't have to care about data being extremely
| sparse until that data is also getting quite large, which
| means you can use cheap/fast approaches to indexing, etc.
| Salgat wrote:
| For example, you create a 100MB file. You tell the operating
| system to map that file to memory, and it gives you a pointer
| to 100MB of memory. Whatever you read from that 100MB of memory
| is what is actually in the file. You can also write to that
| memory and commit it back to the file. The parts you read from
| the pointer are the only parts from the file that are loaded
| into memory. So if you memory map a 100GB file, the operating
| system won't actually load all 100GB into memory, only what is
| accessed (and this is all handled for you automatically).
|
| The operating system is free to load and cache the file into
| memory in whatever way it wants, so for large memory mapped
| files it'll often try to use all available memory to cache as
| much as possible. If another program needs memory, the
| operating system will simply lower the amount of memory
| available to the memory mapped file for caching. This is
| extremely useful for databases, since it greatly simplifies
| both how to persist the data along with how to load and cache
| the persisted data.
|
| This all comes with a big caveat however. The less memory you
| have, the more file accesses occur (similar to your pagefile
| when you're thrashing), which can dramatically slow down your
| memory operations.
|
| tldr; it lets you designate a file to use as a region of
| memory.
| chasd00 wrote:
| so basically an MRU cache for a file's contents? Is there
| enough information to know what bytes from the file are
| cache'd and which are not? It would interesting to see like a
| heatmap of a large model showing what portions of the neural
| network are being used the most. ..like a brain MRI more or
| less.
| AceJohnny2 wrote:
| Quoting another user, jcranmer [1]:
|
| > _" The fundamental operation of mmap is to add new entries to
| the page table of a process, and the precise properties of
| those entries are heavily dependent on what the arguments to
| mmap are._
|
| > _When you mmap a regular file, you 're essentially adding an
| entry to the page table that shares the data with the kernel's
| filesystem cache."_
|
| Now, understanding this requires some understanding of an OS's
| virtual memory function, and what "page tables" are. Those are
| what the OS use to track a process' memory, whose granularity
| is in "pages" (historically 4kB on Linux, though others use
| larger granularity such as 16kB).
|
| mmap() has a a _lot_ of flags [2] that affect the properties of
| those mapped pages. It is the swiss-army knife of memory
| management on Linux.
|
| Some of those properties allow you to share memory with other
| processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory
| (MAP_ANONYMOUS) or, by default, map an (open) file specified by
| the `fd` argument.
|
| (fun fact! On linux, when you malloc(), you don't actually get
| memory, just an IOU from the kernel. Only when you _access_
| that memory, ie accessing those memory pages, does the kernel
| actually make the effort of allocating you that memory.)
|
| [1] https://news.ycombinator.com/item?id=35412842
|
| [2] https://linux.die.net/man/2/mmap
| xiphias2 wrote:
| One thing I don't understand is that if I read a gigabyte
| from a file with the read call, why the kernel can't see that
| it's unused allocated memory, create copy on write pages,
| which could just be shared with the disk cache as long as the
| data is aligned (which it should be after a malloc + read
| call).
|
| It seems the intuitive way to implement read if there's a
| complex memory subsystem that does all these great things
| anyways.
| andrewf wrote:
| That's what mmap is for :)
|
| When a read call completes, you know you've successfully
| read the data from the underlying device. If you have swap
| disabled you know you can read and write from that memory
| quickly.
|
| With mmap'd data, a memory read/write can trigger a page
| fault and I/O, and if you encounter an error like a network
| filesystem not responding in time, the program sees a
| segfault.
| Joker_vD wrote:
| > it's unused allocated memory
|
| Well, first of all, it's not: malloc may have previously
| stored its internal structures in that memory so it may be
| dirty. Second, AFAIK if you ask malloc() for a gigabyte,
| it'll internally call mmap() on an anonymous file and will
| give you its result.
| xiphias2 wrote:
| I see, that's how it's implemented.
|
| But from what I understand there may be nothing in POSIX
| forbidding the kernel to do this optimization, just
| kernel authors decided against it (which is
| understandable).
| comex wrote:
| That's right. Indeed, if the optimization is implemented
| properly it should have no observable behavioral
| difference except for performance (better in some cases,
| worse in others) which is not really something POSIX
| concerns itself with. So POSIX doesn't forbid it.
|
| I can think of one edge case not mentioned by others,
| though.
|
| One of the main benefits of mmap is that the data can be
| paged out while not used. But when you try to page the
| data back in, the attempt may fail, or it may return
| different data than was originally there. This is
| particularly likely with network filesystems or external
| storage. With mmap, these situations result in,
| respectively, the process receiving SIGBUS, or the
| contents of memory changing out from under it. The former
| is usually a crash (unless the program installs a signal
| handler), and the latter can be worse than a crash, if
| either program logic or compiler optimizations make
| assumptions about memory staying consistent when not
| written to. A program using mmap is opting in to that
| risk. A program using malloc and read is not.
|
| That said, it may be reasonable to trust that the
| system's main disk(s) (however you define that) won't
| misbehave this way, and limit the optimization to files
| mapped from them. After all, if swap is enabled, the same
| potential risks exist for any memory that's swapped out.
| Even without swap, if the root partition starts failing,
| the system is not going to stay up for long.
|
| Or, the optimization could be enabled for other disks,
| but with the caveat that the pages would be swapped to
| the swap file/partition rather than paged out normally.
| xiphias2 wrote:
| I see, the main disks should be probably defined when
| mounting (allow memory mapping), but it needs a lot of
| kernel work to know whether it all is worth it or not.
|
| Still, the current problem with mmap is that it's buggy
| on some operating systems according to
| https://www.sqlite.org/mmap.html, so I guess it will
| cause more frustrations until AGI takes over the world :)
| ElectricalUnion wrote:
| > if I read a gigabyte from a file with the read call
|
| `mmap` signals the kernel that you want to use the data
| someday. You `mmap` it, it automatically handles fetching
| and throwing unused bytes away.
|
| `read` signals the kernel that you want to do something
| with the data _now_.
|
| You `read` it, you need it in the memory _now_.
|
| > why the kernel can't see that it's unused allocated
| memory
|
| Only a in-process garbage collector would really know what
| part of your memory is unused. The kernel can't really know
| for sure.
| remexre wrote:
| malloc() and the kernel would need to conspire to know this
| before you call read, when malloc() gives you back the
| memory in the first place
| xiphias2 wrote:
| I guess you haven't read my parent: malloc doesn't give
| you memory, just reserves pages to be used in the future.
| dfox wrote:
| malloc(3) does not know or care about whether the OS
| actually commits the VM allocation that it got from
| somewhere (be it mmap(2) or sbrk(2)), apart from the fact
| that most implementations of malloc would write some kind
| of book-keeping header before the returned object
| (observe that on amd64 glibc, malloc(3) for large
| allocations returns pointers that are offset +0x10 from
| page alignment, that is where the two word allocated
| block header lives). The fact that the memory might not
| really exist is mostly invisible to userspace (as in,
| there is no portable way to find out that it does not
| exist).
|
| The OS somehow detecting that argument to read(2) is page
| aligned block of uncommited meory and transparently
| creating MAP_PRIVATE mapping from that seems like nifty
| idea, but userspace tends to not have page-aligned
| buffers that often (see the aforementioned 0x10 offset
| for one reason why) and applications that intentionally
| allocates IO buffers with some particular alignment do so
| in order to bypass the block cache for particular IO
| patterns and are exactly the kinds of applications that
| tend to use mmap(2) for IO where that does not matter.
| comex wrote:
| > most implementations of malloc would write some kind of
| book-keeping header before the returned object (observe
| that on amd64 glibc, malloc(3) for large allocations
| returns pointers that are offset +0x10 from page
| alignment, that is where the two word allocated block
| header lives).
|
| Hmm. You're right about glibc, but macOS for instance
| reliably returns page-aligned pointers for large
| allocations. I wonder how Windows behaves.
| pkaye wrote:
| mmap maps files into virtual memory. When a program read the
| mapped are of memory, that portion of the file is read.
| akiselev wrote:
| The transition to mmap offloaded memory management to the
| kernel, which can lazily load in parts of the file from disk as
| its memory mapped pages are accessed. The original version
| eagerly read the files into memory before running inference.
| deckard1 wrote:
| this is probably the best source to actually understand how
| memory works in Linux:
|
| https://manybutfinite.com/post/anatomy-of-a-program-in-memor...
| [1]
|
| You can't really understand mmap/sbrk without understanding
| virtual memory and process space layout.
|
| [1] Images are broken. Just open
| https://static.duartes.org/img/blogPosts/kernelUserMemorySpl...
| and go to advanced and "Proceed to static.duartes.org" to
| workaround their https issues. The duartes.org host is owned by
| the blog author. Refresh the blog article and images should
| load now.
| mrbonner wrote:
| mmap is modern? Lol. I used mmap in Java back in 2013 to offheap
| large matrices in our regression engine. Mmap probably exists
| long before that. Edit: and yes, the JVM only reported a few
| hundreds MBs used in the heap. In reality, the memory mapped file
| is several GBs in size.
| antonvs wrote:
| > modern? Lol. 2013
|
| How to tell if a dev is in their 20s or thereabouts
| Kiro wrote:
| 2013 is modern.
| queuebert wrote:
| I have kids older than your usage of mmap.
| loeg wrote:
| The syscall was described in 4.2-4.3BSD in the 80s and shipped
| in 4.3BSD-Reno in 1990. The underlying concept of memory-mapped
| files dates back to at least the 70s. So all of this to say --
| I agree, it's not especially new.
|
| https://en.wikipedia.org/wiki/Mmap
| peterfirefly wrote:
| https://en.wikipedia.org/wiki/Multics#Novel_ideas
___________________________________________________________________
(page generated 2023-04-03 23:02 UTC)