[HN Gopher] Why MMAP in llama.cpp hides true memory usage
       ___________________________________________________________________
        
       Why MMAP in llama.cpp hides true memory usage
        
       Author : flurly
       Score  : 114 points
       Date   : 2023-04-03 16:27 UTC (6 hours ago)
        
 (HTM) web link (twitter.com)
 (TXT) w3m dump (twitter.com)
        
       | btown wrote:
       | I mean, if we're being pedantic, I think it is _possible_ to run
       | llama.cpp with real memory savings this way... in a contrived
       | situation where you were only generating a single token with a
       | single forward pass, and you were limited in RAM.
       | 
       | The new MMAP system would not require you to load the whole model
       | from disk up front, but rather would allow you to load and
       | immediately forget each layer of parameters as you do a forward
       | pass through the GPT architecture. And if you had just tried to
       | use normal swap paging to do this without MMAP, you'd essentially
       | be doing 2x the loading work, since the page for the first layer
       | likely got swapped out as you loaded the rest of the model into
       | "memory."
       | 
       | Of course, as soon as you want to generate a new token, you'd
       | need to reload all the pages of the model parameters. Every token
       | generated would take as long as it would if you were loading the
       | model from scratch.
       | 
       | But for a classification problem where you only need to generate
       | one token, and you need or want to do this in a serverless
       | environment where you'd need to load from disk anyways? I think
       | you'd be able to do that workflow with significantly less RAM in
       | the same amount of clock time, now.
        
         | alchemist1e9 wrote:
         | Is it possible to do multi-GPU inference the way you describe
         | and have say 4 GPUs but each doing different layers on a large
         | model?
         | 
         | If that is the case then perhaps a high end consumer system
         | with max pcie 5 nvme bandwidth and multi gpus could do
         | inference on large LLM models.
        
           | rfoo wrote:
           | Congratulations on rediscovering how the original researchers
           | run these models!
        
             | alchemist1e9 wrote:
             | Do they do it in a way that each GPU can have smaller
             | memory than the model weights?
             | 
             | I keep seeing references to these absolutely massive 48GB+
             | gpu memory sizes and those are extremely expensive. If more
             | but smaller memory gpus can work that could open up a much
             | larger range of hardware and individuals to run inference,
             | especially if efficient load/unload of weight layers from
             | raid pcie5 nvme devices is feasible.
        
               | rfoo wrote:
               | Yes?
               | 
               | The original LLaMA-65B on fp16 weight is 130G. There are
               | no NVIDIA GPU with 130G memory.
               | 
               | As for why they don't use int8/int4: would you try to
               | optimize your code before you confirmed it can actually
               | run?
        
               | alchemist1e9 wrote:
               | Got it, so looking ahead to building hardware that is
               | useful for inference would you agree with these
               | considerations:
               | 
               | - max PCI lanes for more GPUs. AMD cpus and boards
               | probably better
               | 
               | - PCIe 5 nvme. ideally enough to saturate bus
               | 
               | - max GPUs with max memory. memory more important than
               | FLOPs probably.
               | 
               | for a given budget one tries to get as much as possible
               | of above.
               | 
               | How long before a consumer component machine for less
               | than $10K can do inference for a gpt 3.5 turbo quality
               | model?
        
         | visarga wrote:
         | > Every token generated would take as long as it would if you
         | were loading the model from scratch.
         | 
         | Wouldn't make sense to keep most of the model in GPU (as much
         | as it fits) and only load the remaining layers on each pass?
        
       | Animats wrote:
       | Well, yeah.
       | 
       | Short version: the claim that llama could use much less memory by
       | memory-mapping the model data file was wrong. It's just that the
       | Linux utilities for memory use don't count memory-mapped file
       | area.
        
         | pavon wrote:
         | I'm not convinced that there isn't more to the story than that.
         | People (including Justine who implemented MMAPing the model and
         | knows how to monitor memory use properly) are seeing that much
         | of the file isn't being paged into memory, and are not quite
         | sure why[1]. Is it a bug? Is the distribution of weights used
         | highly dependent on the prompts? Are some weights somehow
         | unused altogether? This twitter thread rules out some of those,
         | but I don't think it closes the door.
         | 
         | [1] https://news.ycombinator.com/item?id=35393615
         | 
         | edit: softened some of my claims after reading more updates
         | from over the weekend.
        
           | Animats wrote:
           | Now that's interesting. Entire memory pages of the model
           | aren't being referenced?
        
             | sitkack wrote:
             | eBPF
             | 
             | echo 3 > /proc/sys/vm/drop_caches
        
           | wmf wrote:
           | Other people are saying that the whole file does get paged in
           | and transformers access all their weights by design.
        
           | peterfirefly wrote:
           | The kernel will read more than just the page that faulted.
           | Jart counts just the page faults and multiplies by the page
           | size.
        
         | astrange wrote:
         | File backed memory is "less" than heap memory, because it can
         | be thrown away when needed instead of being swapped out to
         | disk.
         | 
         | And because reading the file probably allocated file backed
         | pages for it anyway unless you used O_DIRECT (and profiled it
         | first.)
         | 
         | There's also wired/mlock() memory which is "worse" than regular
         | memory because it can't be swapped/evicted at all.
        
       | thatcherc wrote:
       | Good thread!
       | 
       | https://threadreaderapp.com/thread/1642726595436883969.html
        
       | garganzol wrote:
       | mmap does not hide the real RAM usage, it just maps a part of
       | virtual memory address space to an I/O-bound storage medium (e.g.
       | file). That trick does not consume RAM in any significant
       | amounts, so there is nothing to hide.
       | 
       | However, such image mapping technique is not always suitable for
       | data-intensive workloads due to page thrashing. If it works for
       | llama.cpp then it can be considered a huge success.
        
       | ImprobableTruth wrote:
       | >mmap is a really nifty feature of modern operating systems
       | 
       | ... for some definition of "modern".
        
         | [deleted]
        
         | koito17 wrote:
         | I wonder how the author would label Windows' MapViewOfFile :)
        
         | lionkor wrote:
         | you dont get the devs who spend their day on twitter instead of
         | writing code to read your thread if it doesnt mention "modern"
         | at least once. /s
         | 
         | Its quite obvious that the author "dumbs down" all the
         | information for a very non-developer audience, so I assume that
         | gives some leniency. If you tell people that we have had things
         | like mmap() for decades, they may start asking too many
         | questions about why every piece of software underperforms so
         | horribly below what was possible decades ago.
         | 
         | Bit of a rant, but I feel that we lose a little bit of
         | potential every time a developer calls operator<< on a
         | std::istream in a loop.
        
           | CoolGuySteve wrote:
           | To be fair I find it completely baffling that R, Pandas, and
           | all sorts of other tools make it difficult or impossible to
           | mmap a column to a dataframe.
           | 
           | Instead you have to do backflips to load data that doesn't
           | fit in RAM.
        
           | verall wrote:
           | What's wrong with operator<< on std::istream?
        
           | chaxor wrote:
           | Devs of 2023: "why does this wheel have rubber on it? It
           | needs to be more modern. Let's take away the rubber and add
           | firecrackers in its place as a nice feature, so really pops
           | and makes the experience more exciting"
        
             | munk-a wrote:
             | The same can be said of devs of 2010 and devs of 2000.
             | There are constant innovations and trends in technology and
             | there are also some very core, generally very old and
             | extremely critical components in the technology stacks we
             | all use - people will see that these components don't use
             | "teh hotness" and assume it would work better with the
             | paradigm that's in favor at the time. Sometimes these
             | people are right, sometimes they're wrong - we shouldn't
             | discourage questioning the status quo as, critically,
             | _sometimes these people are right_ - but it is important to
             | be cautious with such changes since generally there are a
             | lot of very precise decisions that have been carefully
             | considered. However, one need not look any further than
             | OpenSSH to see how very serious bugs can exist in
             | "unimpeachable" software that everyone just assumes it's
             | safe to use.
             | 
             | This is, as an aside, the reason why I don't comment my
             | code often (I personally dislike docblocks that do nothing
             | but obvious explanations) - but whenever I think hard about
             | a thing I always record it in a comment with the problem,
             | my reasoning, and my rational for choosing the solution
             | used... i.e. don't comment on the what, comment on the why.
        
             | malux85 wrote:
             | Chesterton's fence,
             | https://en.wiktionary.org/wiki/Chesterton%27s_fence
             | 
             | You may consider removing the rubber only when you can
             | adqueately explain to me why it is there
        
         | antonvs wrote:
         | Does this mean DEC's TOPS-20 from 1969 is a modern OS?
        
           | gumby wrote:
           | Mmap was the only way to do disk I/O in Multics -- we're
           | talking about a design a decade earlier than that.
           | 
           | (Also you're thinking of TOPS-10 -- TOPS-20 and TWENEX were
           | developed in the 70s. But your heart is in the right place!
        
       | riedel wrote:
       | Best part: >AI-generated summary:
       | 
       | >"This thread explains how the mmap feature of modern operating
       | systems can be used to reduce memory usage when running large
       | deep learning models. It also explains how...
       | 
       | Yet I wonder, if this was just another of those humans imitating
       | a trustworthy AI...
        
       | bragadiru_mafia wrote:
       | Justine has done more to move the image of the trans community to
       | "top tier human person and who cares about your identity this
       | human being can code like a living God I don't care about fluid
       | gender anymore you got me", than any amount of trans human
       | billionaires who hate poor people and their pr campaigns.
       | 
       | You go they/them. Ignore haters
        
       | CyberDildonics wrote:
       | This isn't why mmap in llama.cpp hides true memory usage, it is
       | why mmap hides true memory usage.
        
       | mhh__ wrote:
       | Soft page faults considered genius
        
       | flurly wrote:
       | TL;DR
       | 
       | > The way mmap handles memory makes it a bit tricky for the OS to
       | report on a per-process level how that memory is being used. So
       | it generally will just show it as "cache" use. That's why
       | (AFAICT) there was some initial misunderstanding regarding the
       | memory savings
        
         | vadansky wrote:
         | I lost track since things move so quickly. Was there still
         | memory savings just not as drastic? Or no memory savings, just
         | a speed-up?
        
           | eigenvalue wrote:
           | It did somewhat reduce the total memory used. Now you can
           | load the 30B model while only using ~20gb of RAM, which is
           | about the aggregate size of the 4bit quantized weight files
           | for that model. The real win is that you can kill the main
           | inference binary and try another prompt, and it will start
           | doing inference basically immediately instead of spending
           | 10-15 seconds loading up all the weights into RAM each time.
        
           | chpatrick wrote:
           | It's neither memory savings or a speed-up really. The
           | advantage of mmap is that you can treat a file on a disk a
           | block of memory, so pages from it can be loaded (or unloaded)
           | as necessary instead of one big upfront load into RAM. The
           | benefit is that you can work with data that's bigger than
           | your physical RAM because the kernel can swap it back out to
           | disk if needed. Another benefit could be that if only a small
           | part of the data is needed to compute something then the OS
           | will automatically only load those, but it's unclear to me
           | whether this is the case with LLaMA.
        
             | toxik wrote:
             | Regarding your last point: No, you need all of the weights
             | all of the time.
             | 
             | Edit: except embedding weights but those are not the
             | problem.
        
             | simion314 wrote:
             | It is a speedup for me. When I run llama.cpp from CLI first
             | tiem it takes a very long tiem to load the model in memory.
             | If the program exits or I stop it with Ctrl+C and start it
             | again it will start almost instant.
        
               | chaboud wrote:
               | That's down to caching. If you used your system to do
               | something else for a while, you'd find those pages
               | evicted and the performance back down to Earth. That's
               | one of the things that makes mmap so useful, though. The
               | system can take advantage of access patterns to
               | dramatically improve performance.
        
               | simion314 wrote:
               | Yes, makes sense. And is great. Though honestly not sure
               | why it takes minutes to load a 23Gb model in RAM, I feel
               | is not proportional with the smaller models.
        
             | CyberDildonics wrote:
             | They weren't asking about mmap, they were asking about the
             | program itself.
        
             | [deleted]
        
             | iforgotpassword wrote:
             | It's still somewhat faster if you benchmark it. I assume
             | the os is doing good enough prefetching in the mmap case to
             | hide the loads from disk mostly. So it's not just hiding
             | the initial load of 30gb from disk.
             | 
             | Obviously if you're swapping because you don't have enough
             | memory to hold the model in RAM, the mmap version is going
             | to be much faster, since you don't need to swap anything
             | out to disk but just discard the page and re-read from disk
             | if you need it again later.
        
               | antonvs wrote:
               | > So it's not just hiding the initial load of 30gb from
               | disk.
               | 
               | The issue is typically that that initial load involves
               | some sort of transformation - parsing, instantiating
               | structures, etc. If you can arrange it so that the data
               | is stored in the format you actually need it in memory,
               | then you can skip that entire transformation phase.
               | 
               | I don't know if that's what's been done with llama.cop
               | though.
        
           | rovr138 wrote:
           | It's basically paging to disk.
           | 
           | Not necessarily memory savings, but the improvements here are
           | that it will run on computers with less ram because it can
           | page to disk.
           | 
           | Not necessarily the number reported (since you do need to
           | load chunks into ram), but still lower.
        
             | detrites wrote:
             | If that's the case then part of this may be the different
             | interpretations of "memory".
             | 
             | One persons "paging the same memory requirement from disk
             | to RAM" can be someone elses "requiring less memory/RAM".
        
             | chaboud wrote:
             | Sort of, but without the duplication _and_ initial wait to
             | load. A traditional fat in-memory app would do this:
             | 
             | file (DISK) to process active pages (RAM) to paged out
             | virtual memory if saturated (elsewhere on DISK)
             | 
             | Using mmap typically goes something like:
             | 
             | file (DISK) to process active pages (RAM) to released and
             | cached (RAM) to uncached if evicted (back to same place on
             | DISK) OR back to process active pages from cache (RAM)
             | 
             | For the cost of fixing up some process page tables, the
             | physical memory pages necessary can be brought back from
             | the cache rather than read from disk. It's an orders-of-
             | magnitude performance savings.
        
           | hnav wrote:
           | more like memory mis-reporting, since when you mmap a file
           | in, IIRC that counts against page cache rather than memory
           | usage (you can evict the page without causing write IO so the
           | memory isn't "used")
        
             | astrange wrote:
             | It being safe to evict is why it's correct to not report it
             | as "memory usage".
             | 
             | It's part of the program's working set but measuring that
             | is a completely different story.
        
       | rnnr wrote:
       | Memory mapped files also known as sections in VMS/NT have just
       | two advantages:                 * Fewer context switches among
       | user space / kernel syscalls            * No need to copy
       | _modified_ data into the swap. The behavior for read only data
       | doesn't change
       | 
       | That's it, nothing miraculous about it.
        
         | astrange wrote:
         | No, it also shares them between multiple runs of the same
         | process, and it reuses the pages that were in your file cache
         | anyway.
         | 
         | > * Fewer context switches among user space / kernel syscalls
         | 
         | This is not necessarily true. There's lots of cases where it's
         | actually slower.
        
       | Thaxll wrote:
       | People are discovering mmap, it has been used for a very long
       | time especially in databases.
        
         | rajnathani wrote:
         | Databases usually avoid mmap in favor of their own mmap-like
         | system: https://db.cs.cmu.edu/mmap-cidr2022/
        
           | dfox wrote:
           | That depends on exactly what kind of DBMS and underlying OS
           | one is talking about. For DBMS that runs as one daemon or set
           | of related daemons that has some kind of mechanism to share
           | the afore mentioned buffer pool and synchronize acces to it
           | then sure, it makes more sense, especially as you can do
           | fine-grained copy on write in that bufferpool as an
           | transaction isolation mechanism.
           | 
           | For embedded databases that operate on shared file somewhere
           | just mmap()ing that file allows you to use part of the same
           | file as shared memory segment and for POSIX IPC and thus
           | synchronization between otherwise unrelated processes that
           | access the same data. One big caveat of this approach is that
           | it depends on optional parts of POSIX that are realistically
           | usable on Linux and Solaris and maybe some obscure systems
           | (you can probably do the same thing on Windows NT), notably
           | this does not work on essentially anything BSD-derived.
        
         | xiphias2 wrote:
         | It's true for databases, but this usage is closer to how shared
         | libraries are loaded, just with extenal data.
         | 
         | Also just to show how buggy mmap is, it's disabled in SQLite by
         | default for example:
         | 
         | https://www.sqlite.org/mmap.html
         | 
         | Even now the windows version is buggy, but the great thing is
         | that after it's fixed in llama.cpp, the open source community
         | can just copy the solution.
         | 
         | I don't know of many interactive utilities that were known for
         | fast startup time _because_ of the use of mmap, but now we have
         | one, which means many more are coming :)
        
           | quotemstr wrote:
           | Buggy? In what way?
        
             | masklinn wrote:
             | The "But there are also disadvantages" of the linked pages
             | provides some of the issues with mmap. This also matches
             | burntsushi's experience with mmap in ripgrep:
             | 
             | - depending on concurrent accesses mmap can just sigbus on
             | you (e.g. if the mapped file is being truncated)
             | 
             | - mmap simply does not work with virtual filesystems, and
             | will blow up on large files on 32b systems (windows also
             | has further limitations on mmaps)
             | 
             | - depending on workload mmap may not be faster than regular
             | reads and memory buffers, ripgrep will mmap when working on
             | just a few files, but will use normal buffers for large
             | file counts, because when you start reusing buffer you
             | amortise allocation costs which you can't amortise when
             | creating and destroying mappings
             | 
             | - not only that but mmap/munmap are also globally blocking
             | on the process, so in multithreaded processes it's very bad
             | to map/unmap a lot, you can stall your own application
             | 
             | - the semantics of mmap on crash are also somewhat risky,
             | in that the OS will try very hard to sync, but that may not
             | be desirable if the application crashes in the middle of a
             | write
        
               | quotemstr wrote:
               | I wouldn't call any of these things a "bug", in the sense
               | of a behavior that departs from an explicit or implicit
               | specification. Some of them are just advantages and
               | disadvantages of that overall approach.
               | 
               | Also, I'm disappointed that VM languages don't turn
               | SIGBUS accessing a memory mapped region into a VM
               | exception. They could, and it wouldn't even be that hard.
        
               | smeenai wrote:
               | https://reviews.llvm.org/D69294 is an interesting case we
               | ran into with LLD (the LLVM linker), where the the memory
               | pressure from mmap'ing a large output file combined with
               | filesystem compression resulted in particularly bad
               | performance.
        
         | morelisp wrote:
         | It has been used for so long in databases it should no longer
         | be used in databases.
         | 
         | http://www.cidrdb.org/cidr2022/papers/p13-crotty.pdf
        
           | [deleted]
        
           | rajnathani wrote:
           | Just saw your reply, we replied with the same CMU link :)
        
         | mftb wrote:
         | They must use it in games too, no? For level loading, assets,
         | etc...
        
           | jbverschoor wrote:
           | In general game engines are almost full operating systems and
           | reimplement many things such as memory management
        
       | eachro wrote:
       | What does mmap do exactly? Why was the transition to using it a
       | big improvement in llama.cpp?
        
         | programmarchy wrote:
         | My understanding is that it maps a file directly to memory to
         | reduce disk usage.
        
           | debatem1 wrote:
           | Doesn't change disk usage. The file is still on disk.
           | 
           | The difference is between reading a file and memory mapping
           | (mmap'ing) it.
           | 
           | If you read a 1TB file into memory you use 1TB of disk and
           | 1TB of physical memory. If you then access that data it's as
           | fast as RAM because that's where it is.
           | 
           | If you mmap a 1TB file you use 1TB of disk and 1TB of
           | _virtual_ memory. If you then access that data it may be
           | mapped into virtual memory but not actually be in RAM. This
           | triggers a page fault, at which point the correct page is
           | loaded from disk to physical memory, and handed back to you.
           | 
           | The key observation is that the amount of physical memory
           | occupied by the mmap'd file is much smaller than the entirety
           | of the file unless you access almost all of it.
           | 
           | If your filesystem supports holes, this can also be useful
           | for writes: it's possible to map files vastly larger than
           | physical disk space, but so long as the actual number of
           | places written to is quite small you won't run out.
           | 
           | The combination is very useful for datastructures because you
           | basically don't have to care about data being extremely
           | sparse until that data is also getting quite large, which
           | means you can use cheap/fast approaches to indexing, etc.
        
         | Salgat wrote:
         | For example, you create a 100MB file. You tell the operating
         | system to map that file to memory, and it gives you a pointer
         | to 100MB of memory. Whatever you read from that 100MB of memory
         | is what is actually in the file. You can also write to that
         | memory and commit it back to the file. The parts you read from
         | the pointer are the only parts from the file that are loaded
         | into memory. So if you memory map a 100GB file, the operating
         | system won't actually load all 100GB into memory, only what is
         | accessed (and this is all handled for you automatically).
         | 
         | The operating system is free to load and cache the file into
         | memory in whatever way it wants, so for large memory mapped
         | files it'll often try to use all available memory to cache as
         | much as possible. If another program needs memory, the
         | operating system will simply lower the amount of memory
         | available to the memory mapped file for caching. This is
         | extremely useful for databases, since it greatly simplifies
         | both how to persist the data along with how to load and cache
         | the persisted data.
         | 
         | This all comes with a big caveat however. The less memory you
         | have, the more file accesses occur (similar to your pagefile
         | when you're thrashing), which can dramatically slow down your
         | memory operations.
         | 
         | tldr; it lets you designate a file to use as a region of
         | memory.
        
           | chasd00 wrote:
           | so basically an MRU cache for a file's contents? Is there
           | enough information to know what bytes from the file are
           | cache'd and which are not? It would interesting to see like a
           | heatmap of a large model showing what portions of the neural
           | network are being used the most. ..like a brain MRI more or
           | less.
        
         | AceJohnny2 wrote:
         | Quoting another user, jcranmer [1]:
         | 
         | > _" The fundamental operation of mmap is to add new entries to
         | the page table of a process, and the precise properties of
         | those entries are heavily dependent on what the arguments to
         | mmap are._
         | 
         | > _When you mmap a regular file, you 're essentially adding an
         | entry to the page table that shares the data with the kernel's
         | filesystem cache."_
         | 
         | Now, understanding this requires some understanding of an OS's
         | virtual memory function, and what "page tables" are. Those are
         | what the OS use to track a process' memory, whose granularity
         | is in "pages" (historically 4kB on Linux, though others use
         | larger granularity such as 16kB).
         | 
         | mmap() has a a _lot_ of flags [2] that affect the properties of
         | those mapped pages. It is the swiss-army knife of memory
         | management on Linux.
         | 
         | Some of those properties allow you to share memory with other
         | processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory
         | (MAP_ANONYMOUS) or, by default, map an (open) file specified by
         | the `fd` argument.
         | 
         | (fun fact! On linux, when you malloc(), you don't actually get
         | memory, just an IOU from the kernel. Only when you _access_
         | that memory, ie accessing those memory pages, does the kernel
         | actually make the effort of allocating you that memory.)
         | 
         | [1] https://news.ycombinator.com/item?id=35412842
         | 
         | [2] https://linux.die.net/man/2/mmap
        
           | xiphias2 wrote:
           | One thing I don't understand is that if I read a gigabyte
           | from a file with the read call, why the kernel can't see that
           | it's unused allocated memory, create copy on write pages,
           | which could just be shared with the disk cache as long as the
           | data is aligned (which it should be after a malloc + read
           | call).
           | 
           | It seems the intuitive way to implement read if there's a
           | complex memory subsystem that does all these great things
           | anyways.
        
             | andrewf wrote:
             | That's what mmap is for :)
             | 
             | When a read call completes, you know you've successfully
             | read the data from the underlying device. If you have swap
             | disabled you know you can read and write from that memory
             | quickly.
             | 
             | With mmap'd data, a memory read/write can trigger a page
             | fault and I/O, and if you encounter an error like a network
             | filesystem not responding in time, the program sees a
             | segfault.
        
             | Joker_vD wrote:
             | > it's unused allocated memory
             | 
             | Well, first of all, it's not: malloc may have previously
             | stored its internal structures in that memory so it may be
             | dirty. Second, AFAIK if you ask malloc() for a gigabyte,
             | it'll internally call mmap() on an anonymous file and will
             | give you its result.
        
               | xiphias2 wrote:
               | I see, that's how it's implemented.
               | 
               | But from what I understand there may be nothing in POSIX
               | forbidding the kernel to do this optimization, just
               | kernel authors decided against it (which is
               | understandable).
        
               | comex wrote:
               | That's right. Indeed, if the optimization is implemented
               | properly it should have no observable behavioral
               | difference except for performance (better in some cases,
               | worse in others) which is not really something POSIX
               | concerns itself with. So POSIX doesn't forbid it.
               | 
               | I can think of one edge case not mentioned by others,
               | though.
               | 
               | One of the main benefits of mmap is that the data can be
               | paged out while not used. But when you try to page the
               | data back in, the attempt may fail, or it may return
               | different data than was originally there. This is
               | particularly likely with network filesystems or external
               | storage. With mmap, these situations result in,
               | respectively, the process receiving SIGBUS, or the
               | contents of memory changing out from under it. The former
               | is usually a crash (unless the program installs a signal
               | handler), and the latter can be worse than a crash, if
               | either program logic or compiler optimizations make
               | assumptions about memory staying consistent when not
               | written to. A program using mmap is opting in to that
               | risk. A program using malloc and read is not.
               | 
               | That said, it may be reasonable to trust that the
               | system's main disk(s) (however you define that) won't
               | misbehave this way, and limit the optimization to files
               | mapped from them. After all, if swap is enabled, the same
               | potential risks exist for any memory that's swapped out.
               | Even without swap, if the root partition starts failing,
               | the system is not going to stay up for long.
               | 
               | Or, the optimization could be enabled for other disks,
               | but with the caveat that the pages would be swapped to
               | the swap file/partition rather than paged out normally.
        
               | xiphias2 wrote:
               | I see, the main disks should be probably defined when
               | mounting (allow memory mapping), but it needs a lot of
               | kernel work to know whether it all is worth it or not.
               | 
               | Still, the current problem with mmap is that it's buggy
               | on some operating systems according to
               | https://www.sqlite.org/mmap.html, so I guess it will
               | cause more frustrations until AGI takes over the world :)
        
             | ElectricalUnion wrote:
             | > if I read a gigabyte from a file with the read call
             | 
             | `mmap` signals the kernel that you want to use the data
             | someday. You `mmap` it, it automatically handles fetching
             | and throwing unused bytes away.
             | 
             | `read` signals the kernel that you want to do something
             | with the data _now_.
             | 
             | You `read` it, you need it in the memory _now_.
             | 
             | > why the kernel can't see that it's unused allocated
             | memory
             | 
             | Only a in-process garbage collector would really know what
             | part of your memory is unused. The kernel can't really know
             | for sure.
        
             | remexre wrote:
             | malloc() and the kernel would need to conspire to know this
             | before you call read, when malloc() gives you back the
             | memory in the first place
        
               | xiphias2 wrote:
               | I guess you haven't read my parent: malloc doesn't give
               | you memory, just reserves pages to be used in the future.
        
               | dfox wrote:
               | malloc(3) does not know or care about whether the OS
               | actually commits the VM allocation that it got from
               | somewhere (be it mmap(2) or sbrk(2)), apart from the fact
               | that most implementations of malloc would write some kind
               | of book-keeping header before the returned object
               | (observe that on amd64 glibc, malloc(3) for large
               | allocations returns pointers that are offset +0x10 from
               | page alignment, that is where the two word allocated
               | block header lives). The fact that the memory might not
               | really exist is mostly invisible to userspace (as in,
               | there is no portable way to find out that it does not
               | exist).
               | 
               | The OS somehow detecting that argument to read(2) is page
               | aligned block of uncommited meory and transparently
               | creating MAP_PRIVATE mapping from that seems like nifty
               | idea, but userspace tends to not have page-aligned
               | buffers that often (see the aforementioned 0x10 offset
               | for one reason why) and applications that intentionally
               | allocates IO buffers with some particular alignment do so
               | in order to bypass the block cache for particular IO
               | patterns and are exactly the kinds of applications that
               | tend to use mmap(2) for IO where that does not matter.
        
               | comex wrote:
               | > most implementations of malloc would write some kind of
               | book-keeping header before the returned object (observe
               | that on amd64 glibc, malloc(3) for large allocations
               | returns pointers that are offset +0x10 from page
               | alignment, that is where the two word allocated block
               | header lives).
               | 
               | Hmm. You're right about glibc, but macOS for instance
               | reliably returns page-aligned pointers for large
               | allocations. I wonder how Windows behaves.
        
         | pkaye wrote:
         | mmap maps files into virtual memory. When a program read the
         | mapped are of memory, that portion of the file is read.
        
         | akiselev wrote:
         | The transition to mmap offloaded memory management to the
         | kernel, which can lazily load in parts of the file from disk as
         | its memory mapped pages are accessed. The original version
         | eagerly read the files into memory before running inference.
        
         | deckard1 wrote:
         | this is probably the best source to actually understand how
         | memory works in Linux:
         | 
         | https://manybutfinite.com/post/anatomy-of-a-program-in-memor...
         | [1]
         | 
         | You can't really understand mmap/sbrk without understanding
         | virtual memory and process space layout.
         | 
         | [1] Images are broken. Just open
         | https://static.duartes.org/img/blogPosts/kernelUserMemorySpl...
         | and go to advanced and "Proceed to static.duartes.org" to
         | workaround their https issues. The duartes.org host is owned by
         | the blog author. Refresh the blog article and images should
         | load now.
        
       | mrbonner wrote:
       | mmap is modern? Lol. I used mmap in Java back in 2013 to offheap
       | large matrices in our regression engine. Mmap probably exists
       | long before that. Edit: and yes, the JVM only reported a few
       | hundreds MBs used in the heap. In reality, the memory mapped file
       | is several GBs in size.
        
         | antonvs wrote:
         | > modern? Lol. 2013
         | 
         | How to tell if a dev is in their 20s or thereabouts
        
         | Kiro wrote:
         | 2013 is modern.
        
         | queuebert wrote:
         | I have kids older than your usage of mmap.
        
         | loeg wrote:
         | The syscall was described in 4.2-4.3BSD in the 80s and shipped
         | in 4.3BSD-Reno in 1990. The underlying concept of memory-mapped
         | files dates back to at least the 70s. So all of this to say --
         | I agree, it's not especially new.
         | 
         | https://en.wikipedia.org/wiki/Mmap
        
           | peterfirefly wrote:
           | https://en.wikipedia.org/wiki/Multics#Novel_ideas
        
       ___________________________________________________________________
       (page generated 2023-04-03 23:02 UTC)