[HN Gopher] Are You Sure You Want to Use MMAP in Your Database M...
___________________________________________________________________
Are You Sure You Want to Use MMAP in Your Database Management
System? (2022)
Author : nethunters
Score : 101 points
Date : 2023-07-02 16:48 UTC (6 hours ago)
(HTM) web link (db.cs.cmu.edu)
(TXT) w3m dump (db.cs.cmu.edu)
| AnotherGoodName wrote:
| A well written bespoke function can beat a generalized function
| at a specific task.
|
| If you have the resources to write and maintain the bespoke
| method great. The large database developers probably have this.
| For others please don't take this link and go around claiming
| mmap is bad though. That gets tiresome and is misguided. Mmap is
| a shortcut to access large files in a non linear fashion. It's
| good at that too. Just not as good as a bespoke function.
| formerly_proven wrote:
| mmap can be handy but usually is not a good idea when you care
| about ACID properties. So it tends to be most useful outside
| databases.
| josephg wrote:
| Can you give some examples where mmap is useful?
| fatboy wrote:
| One place I've seen it used was a lib by a guy called
| DHoerl for reading images that are too big to fit in memory
| (this was years ago on iOS).
|
| A very over-simplified and probably a bit incorrect
| description of what it did was to create a smaller version
| of the image - one that could fit in memory - by sub
| sampling every nth pixel, which was addressed via mmap.
|
| It actually dealt with jpegs so I have no idea how that bit
| worked as they are not bitmaps.
| whartung wrote:
| It's useful in a world of several processes sharing things.
| This is much less common today in a world of "single
| process" containers and VMs as well as monolithic processes
| using threads or async techniques.
|
| However, Java can build a special library file of the core
| JRE classes that it can mmap into memory with the intent to
| speed up startup times, mostly for small Java programs.
|
| Guile scheme will mmap files that have been compiled to
| byte code. You can visualize a contrived (especially today)
| scenario where Guile is used for CGI handlers, having the
| bulk of their code mapped, the overall memory impact of
| simultaneous handlers is much lower, as well as start up
| times.
|
| The process model is less common today so the value of this
| goes down, but it can still have its place.
| duped wrote:
| I once improved a parser's performance a huge amount (iirc,
| something like 500x) when parsing large (>1GB) text files
| by mmap'ing the files instead of reading them into a byte
| array. It's not a magic bullet but it was alright for that
| application.
|
| Another technique that can only be done with mmap is to map
| two contiguous regions of virtual memory to the same
| underlying buffer. This allows you to use a ring buffer but
| only read from/write to what looks like a contiguous region
| of memory.
| [deleted]
| dataflow wrote:
| If your data is likely to already be in the system cache,
| memory mapping can achieve zero copying of the data,
| whereas reading will perform at least one memcpy. So there
| can be a performance advantage depending on the usage
| pattern.
|
| Also, I've never tested this, but I believe mapped files
| will get flushed as long as the system stays running. So if
| you only need resilience against abnormal termination
| rather than system crashes, it seems like a good option?
| amluto wrote:
| > Also, I've never tested this, but I believe mapped
| files will get flushed as long as the system stays
| running. So if you only need resilience against abnormal
| termination rather than system crashes, it seems like a
| good option?
|
| Linux will not lose data written to a MAP_SHARED mapping
| when the process crashes.
|
| But! Linux _will_ synchronously update mtime when
| starting to write to a currently write protected mapping
| (e.g. one which was just written out). This means (a)
| POSIX is violated (IMO) and (b) what should be a minor
| fault to enable writes turns into an actual metadata
| write, which can cause actual synchronous IO.
|
| I have an ancient patch set to fix this, but I never got
| it all the way into upstream Linux.
|
| What you _can_ do is mmap a file on a tmpfs as long as
| you trust yourself to have some other reliable process
| handle the data even if your application terminates
| abnormally. This is awkward with a container solution if
| you need to survive termination of the entire container.
| dist1ll wrote:
| This paper isn't aimed at random developers, and it's not a
| criticism of mmap in general.
|
| This is an appeal to core database engineers to stop using the
| wrong tool for the job.
| pjdesno wrote:
| Not just databases - we ran into the same issues when we needed a
| high-performance caching HTTP reverse proxy for a research
| project. We were just going to drop in Varnish, which is mmap-
| based, but performance sucked and we had to write our own.
|
| Note that Varnish dates to 2006, in the days of hard disk drives,
| SCSI, and 2-core server CPUs. Mmap might well have been as good
| or even better than I/O back then - a lot of the issues discussed
| in this paper (TLB shootdown overhead, single flush thread) get
| much worse as the core count increases.
| Sesse__ wrote:
| Varnish' design wasn't very fast even for 2006-era hardware. It
| _was_ fast compared to Squid, though (which was the only real
| competitor at the time), and most importantly, much more
| flexible for the origin server case. But it came from a culture
| of "the FreeBSD kernel is so awesome that the best thing
| userspace can do is to offload as many decisions as humanly
| possible to the kernel", which caused, well, suboptimal
| performance.
|
| AFAIK the persistent backend was dropped pretty early on
| (eventually replaced with a more traditional
| read()/write()-based one as part of Varnish Plus), and the
| general recommendation became just to use malloc and hope you
| didn't swap.
| tayo42 wrote:
| Varnish has a file system backed cache that depends on the page
| cache to keep it fast.
|
| What did you differently in your custom one that was faster
| then varnish?
| SoftTalker wrote:
| This reads more like "don't write your own DBMS" than "don't use
| mmap."
| hyc_symas wrote:
| This is a pretty old argument and IMO it's far out of
| date/obsolete.
|
| Taking full control of your I/O and buffer management is great if
| (a) your developers are all smart and experienced enough to be
| kernel programmers and (b) your DBMS is the only process running
| on a machine. In practice, (a) is never true, and (b) is no
| longer true because everyone is running apps inside containers
| inside shared VMs. In the modern application/server environment,
| no user level process has accurate information about the total
| state of the machine, only the kernel (or hypervisor) does and
| it's an exercise in futility to try to manage paging etc at the
| user level.
|
| As Dr. Michael Stonebraker put it: The Traditional RDBMS Wisdom
| is (Almost Certainly) All Wrong.
| https://slideshot.epfl.ch/play/suri_stonebraker (See the slide at
| 21:25 into the video). Modern DBMSs spend 96% of their time
| managing buffers and locks, and only 4% doing actual useful work
| for the caller.
|
| Granted, even using mmap you still need to know wtf you're doing.
| MongoDB's original mmap backing store was a poster child for
| Doing It Wrong, getting all of the reliability problems and none
| of the performance benefits. LMDB is an example of doing it
| right: perfect crash-proof reliability, and perfect linear read
| scalability across arbitrarily many CPUs with zero-copy reads and
| no wasted effort, and a hot code path that fits into a CPU's 32KB
| L1 instruction cache.
| gavinray wrote:
| Out of curiosity, how many databases have you written?
|
| This is co-authored by Pavlo, Viktor Leiss, with feedback from
| Neumann. I'm sorry, but if someone on the internet claims to
| know better than those 3, you're going to need some monumental
| evidence of your credibility.
|
| Additionally, what you link here: > ... (See
| the slide at 21:25 into the video). Modern DBMSs spend 96% of
| their time managing buffers and locks, and only 4% doing actual
| useful work for the caller.
|
| Is discussing "Main Memory" databases. These databases do no
| I/O outside of potential initial reads, because all of the data
| fits in-memory!
|
| These databases represent a small portion of contemporary DBMS
| usage when compared to traditional RDBMS.
|
| All you have to do is look at the bandwidth and reads/sec from
| the paper when using O_DIRECT "pread()"s versus mmap'ed IO.
| LAC-Tech wrote:
| This is a classic appeal to authority. Let's play the
| argument, not the man.
|
| (My understanding is that the GP wrote LMDB, works on
| openLDAP, and was a maintainer for BerkelyDB for a number of
| years. But even if he'd only written 'hello, world!' I'm much
| more interested in the specific arguments).
| jemfinch wrote:
| "Taking full control of your I/O and buffer management is
| great if (a) your developers are all smart and experienced
| enough to be kernel programmers" is already an appeal to
| authority in itself.
|
| We shouldn't apply a higher bar to the counterargument than
| we applied to the argument in the first place.
| hyc_symas wrote:
| Correct, and thank you. I wrote LMDB, wrote a lot of
| OpenLDAP, and worked on BerkeleyDB for many years. And
| actually Andy Pavlo invited me to CMU to give a lecture on
| LMDB a few years back.
| https://www.youtube.com/watch?v=tEa5sAh-kVk
|
| Andy and I have had this debate going for a long time
| already.
| gavinray wrote:
| Well, I eat my shorts.
|
| Isn't LMBD closer to an embedded key-value store than an
| RDBMS, though? Also there's a section in the paper that
| mentions it's single-writer.
| hyc_symas wrote:
| Yes, LMDB is an embedded key/value store but it can be
| used as the backing store of any other DB model you care
| for. E.g. as a backend to MySQL, or SQLite, or OpenLDAP,
| or whatever.
| gavinray wrote:
| The argument is that:
|
| - Queries can trigger blocking page faults when accessing
| (transparently) evicted pages, causing unexpected I/O
| stalls
|
| - mmap() complicates transactionality and error-handling
|
| - Page table contention, single-threaded page eviction, and
| TLB shootdowns become bottlenecks
| hyc_symas wrote:
| 1 - for reading any uncached data, the I/O stalls are
| unavoidable. Whatever client requested that data is going
| to have to wait regardless.
|
| 2 - complexity? this is simply false. LMDB's ACID txns
| using MVCC are much simpler than any "traditional"
| approach.
|
| 3 - contention is a red herring since this approach is
| already single-writer, as is common for most embedded k/v
| stores these days. You lose more perf by trying to make
| the write path multi-threaded, in lock contention and
| cache thrashing.
| jandrewrogers wrote:
| I think the real argument is more nuanced. Where you see
| mmap() fail badly on Linux, even for read-only workloads,
| is under a few specific conditions: very large storage
| volumes, highly concurrent access, non-trivial access
| patterns (e.g. high-dimensionality access methods). Most
| people do not operate data models under these conditions,
| but if you do then you can achieve large integer factor
| gains in throughput by not using mmap().
|
| Interestingly, most of the reason for these problems has to
| do with theoretical limitations of cache replacement
| algorithms as drivers of I/O scheduling. There are
| alternative approaches to scheduling I/O that work much
| better in these cases but mmap() can't express them, so in
| those cases bypassing mmap() offers large gains.
| danappelxx wrote:
| Who is deploying databases in containers?
| morelisp wrote:
| I'm running prod databases in containers so the server infra
| team doesn't have to know anything about how that specific
| database works or how to upgrade it, they just need to know
| how to issue generic container start/stop commands if they
| want to do some maintenance.
|
| (But just in containers, not in Kubernetes. I'm not crazy.)
| orbz wrote:
| A disturbingly large number of deployments I've seen using
| Kubernetes or docker compose have databases deployed as such.
| spockz wrote:
| Given the ability to deploy pods to dedicated nodes based
| on label selectors, what is the actual performance impact
| of running a database in a container on a bare metal host
| with mounted volume versus running that same process with
| say systemd on that same node? Basically, shouldn't the
| overhead of running a container be minimal?
| crabbone wrote:
| The problem is kubelet likes to spike in memory / CPU /
| network usage. It's not a well-behaved program to put
| alongside a database. It's not written with an eye for
| resource utilization.
|
| Also, it brings nothing of value to the table, but
| requires a lot of dance around it to keep it going. I.e.
| if you are a decent DBA, you don't have a problem setting
| up a node to run your database of choice, you would be
| probably opposed to using pre-packaged Docker images
| anyways.
|
| Also, Kubernetes sucks at managing storage... basically,
| it doesn't offer anything that'd be useful to a DBA.
| Things that _might_ be useful come as CSI... and,
| obviously, it 's better / easier to not use a CSI, but to
| interface directly with the storage you want instead.
|
| That's not to say that storage products don't offer these
| CSI... so, a legitimate question would be why would
| anyone do that? -- and the answer is -- not because it's
| useful, but because a lot of people think they need /
| want it. Instead of fighting stupidity, why not make an
| extra buck?
| FridgeSeal wrote:
| I run DB's on K8s, not because I don't know what I'm
| doing, but because most of the trade offs are worth it.
|
| If I run a db workload in K8s, it's a tiny fraction of
| the operational overhead, and not a massively noticeable
| performance loss.
|
| I would absolutely _love_ a way to deploy and manage db's
| as easily as K8s with fewer of the quite significant
| issues that have mentioned, so if you know of something
| that is better behaved around singular workloads, but
| keeps the simple deploys, the resiliency, the ease of
| networking and config deployments, the ease of
| monitoring, etc, I am all ears.
| crabbone wrote:
| If you think that deploying anything with Kubernetes is
| simple... well, I have bad news for you.
|
| It's simple, until you hit a problem. And then it becomes
| a lot worse than if you had never touched it. You are now
| in the stage of a person who'd never made backups and
| never had a failure that required them to restore from
| backups, and you are wondering why would anyone do it.
| Adverse events are rare, and you may go like this for
| years, or, perhaps the rest of your life...
| unfortunately, your experience will not translate into a
| general advice.
|
| But, again, you just might be in the camp where
| performance doesn't matter. Nor does uptime matter, nor
| does your data have very high value... and in that case
| it's OK to use tools that don't offer any of that, and
| save you some time. But, you cannot advise others based
| on that perspective. Or, at least, not w/o mentioning the
| downsides.
| danappelxx wrote:
| IMO if you're concerned about performance and yet are
| deploying databases this way -- mmap should not even be on
| the radar.
| charcircuit wrote:
| How would containers even hurt performance? How does the
| database no longer having the ability to see other
| processes on the machine somehow make it slower?
| danappelxx wrote:
| I'll assume the worst case:
|
| - lots of containers running on a single host
|
| - containers are each isolated in a VM (aka virtualized)
|
| - workloads are not homogenous and change often (your
| neighbor today may not be your neighbor tomorrow)
|
| I believe these are fair assumptions if you're running on
| generic infrastructure with kubernetes.
|
| In this setup, my concerns are pretty much noisy
| neighbors + throttling. You may get latency spikes out of
| nowhere and the cause could be any of:
|
| - your neighbor is hogging IO (disk or network)
|
| - your database spawned too many threads and got
| throttled by CFS
|
| - CFS scheduled your DBs threads on a different CPU and
| you lost your cache lines
|
| In short, the DB does not have stable, predictable
| performance, which are exactly the characteristics you
| want it to have. If you ran the DB on a dedicated host
| you avoid this whole suite of issues.
|
| You can alleviate most of this if you make sure the DB's
| container gets the entire host's resources and doesn't
| have neighbors.
| gcoakes wrote:
| > - containers are each isolated in a VM (aka
| virtualized)
|
| Why are you assuming containers are virtualized? Is there
| some container runtime that does that as an added
| security measure? I thought they all use namespaces on
| Linux.
| danappelxx wrote:
| It's becoming standard as a security measure. See: Kata
| containers, Firecracker VM
| crabbone wrote:
| There are many "holes" in these containers.
|
| 1. fsync. You cannot "divide" it between containers.
| Whoever does it, stalls I/O for everyone else.
|
| 2. Context switches. Unless you do a lot of
| configurations _outside_ of container runtime, you cannot
| ensure exclusive access to the number of CPU cores you
| need.
|
| 3. Networking has the same problem. You would either have
| to dedicate a whole NIC or SRI-OV-style virtual NIC to
| your database server. Otherwise just the amount of
| chatter that goes on through the control plane of
| something like Kubernetes will be a noticeable
| disadvantage. Again, containers don't help here, they
| only get in the way as to get that kind of exclusive
| network access you need _more_ configuration on the host,
| and, possible an CNI to deal with it.
|
| 4. kubelet is not optimized to get out of your way. It
| needs a lot of resources and may spike, hindering or
| outright stalling database process.
|
| 5. Kubernetes sucks at managing memory-intensive
| processes. It doesn't work (well or at all) with swap
| (which, again, cannot be properly divided between
| containers). It doesn't integrate well with OOM killer
| (it cannot replace it, so any configurations you make
| inside Kubernetes are kind of irrelevant, because
| system's OOM killer will do how it pleases, ignoring
| Kubernetes).
|
| ---
|
| Bottom line... Kubernetes is lame from infrastructure
| perspective. It's written for Web developers. To make
| things appear simpler for them, while sacrificing a lot
| of resources and hiding a lot of actual complexity...
| which is impossible to hide, and which, in an even of
| failure will come to bite you. You don't want that kind
| of program near your database.
| FridgeSeal wrote:
| As these are obviously very real issues, and Kubernetes
| also isn't going away imminently, how many of these can
| be fixed/improved with different design on the
| application front?
|
| Would using direct-Io API's fix most of the fsync issues?
| If workloads pin their stuff to specific cores can we
| incite some of the overhead here? (Assuming we're only
| running a single dedicated workload + kubelet on the
| node).
|
| > You would either have to dedicate a whole NIC or SRI-
| OV-style virtual NIC to your database server
|
| Tbh I've no idea we could do this with commodity cloud
| servers, nor do I know how, but I'm terribly interested
| in knowing how, do you know if there's like a "dummy's
| guide to better networking"? Haha
|
| > kubelet is not optimized to get out of your
| way...Kubernetes sucks at managing memory-intensive
| processes
|
| Definitely agree on both these issues, I've blown up the
| kubelet by overallocating memory before, which basically
| borked the node until some watchdog process kicked in.
| Sounds like the better solution here is a kubelet rebuilt
| to operate more efficiently and more predictably? Is the
| solution a db-optimised kubelet/K8s?
| huahaiy wrote:
| Embedded DB
| crabbone wrote:
| Nobody who matters.
|
| Those who do that don't know what they are doing (even if
| they outnumber the other side hundred to one, they "don't
| count" because they aren't aiming for good performance
| anyways).
|
| Well, maybe not quite... of course it's possible that someone
| would want to deploy a database in a container because of the
| convenience of assembling all dependencies in a single
| "package", however, they would never run database on the same
| node as applications -- that's insanity.
|
| But, even the idea of deploying a database alongside
| something like kubelet service is cringe... This service is
| very "fat" and can spike in memory / CPU usage. I would be
| very strongly opposed to an idea of running a database on the
| same VM that runs Kubernetes or any container runtime that
| requires a service to run it.
|
| Obviously, it says nothing about the number of processes that
| will run on the database node. At the minimum, you'd want to
| run some stuff for monitoring, that's beside all the system
| services... but I don't think GP meant "one process"
| literally. Neither that is realistic nor is it necessary.
| hyc_symas wrote:
| >but I don't think GP meant "one process" literally.
| Neither that is realistic nor is it necessary.
|
| The point was simply about other processes that could be
| competing for resources - CPU, memory, or I/O. It is
| expensive for a user-level process to perform accounting
| for all of these resources, and without such accounting you
| can't optimally allocate them.
|
| If there are other apps that can suddenly spike memory
| usage then any careful buffer tuning you've done goes out
| the window. Likewise for any I/O scheduling you've done,
| etc.
| tadfisher wrote:
| Maybe someone should pull LMDB's mmap/paging system into a
| usable library. I'd love to use the k/v store part of course,
| but I keep hitting the default key size limitation and would
| prefer not to link statically.
| hyc_symas wrote:
| It wouldn't be much use without the B+tree as well; it's the
| B+tree's cache friendliness that allows applications to run
| so efficiently without the OS knowing any specifics of the
| app's usage patterns.
| crabbone wrote:
| > your DBMS is the only process running on a machine. In
| practice, (a) is never true, and (b) is no longer true because
| everyone is running apps inside containers inside shared VMs.
|
| There's nothing special about kernel programmers. In fact, if I
| had to compare, I'd go with storage people being the more
| experienced / knowledgeable ones. They have a highly
| competitive environment, which requires a lot more
| understanding and inventiveness to succeed, whereas kernel
| programmers proper don't compete -- Linux won many years ago.
| Kernel programmers who deal with stuff like drivers or various
| "extensions" are, largely, in the same group as storage (often
| time literally the same people).
|
| As for "single process" argument... well, if you run a database
| inside an OS, then, obviously, that will never happen as OS has
| its own processes to run. But, if you ignore that -- no DBA
| worth their salt would put database in the environment where it
| has to share resources with applications. People who do that
| are, probably, Web developers who don't have high expectations
| from their database anyways and would have no idea how to
| configure / tune it for high performance, so, it doesn't matter
| how they run it, they aren't the target audience -- they are
| light years behind on what's possible to achieve with their
| resources.
|
| This has nothing to do with mmap though. mmap shouldn't be used
| for storage applications for other reasons. mmap doesn't allow
| their users to precisely control the persistence aspect...
| which is kind of the central point of databases. So, it's a
| mostly worthless tool in that context. Maybe fine for some
| throw-away work, but definitely not for storing users' data or
| database's own data.
| hyc_symas wrote:
| > There's nothing special about kernel programmers.
|
| Yes, that was a shorthand generalization for "people who've
| studied computer architecture" - which most application
| developers never have.
|
| > no DBA worth their salt would put database in the
| environment where it has to share resources with
| applications.
|
| Most applications today are running on smartphones/mobile
| devices. That means they're running with local embedded
| databases - it's all about "edge computing". There's far more
| DBs in use in the world than there are DBAs managing them.
|
| > mmap shouldn't be used for storage applications for other
| reasons. mmap doesn't allow their users to precisely control
| the persistence aspect... which is kind of the central point
| of databases. So, it's a mostly worthless tool in that
| context. Maybe fine for some throw-away work, but definitely
| not for storing users' data or database's own data.
|
| Well, you're half right. That's why by default LMDB uses a
| read-only mmap and uses regular (p)write syscalls for writes.
| But the central point of databases is to be able to persist
| data _such that it can be retrieved again in the future,
| efficiently_. And that 's where the read characteristics of
| using mmap are superior.
| crabbone wrote:
| > Most applications today are running on smartphones/mobile
| devices.
|
| That's patently false. There are about 8 bn. people. Even
| if everyone has a smartphone or two, it's nothing compared
| to the total of all devices that can be called "computer".
| I think that "smart TV" alone will beat the number of
| smartphones. But even that is a drop in a bucket when it
| comes to the total of running programs on Earth / its
| orbit.
|
| But, that's beside the point. Smartphones aren't designed
| to run database servers. Even if they indeed were the
| majority, they'd still be irrelevant for this conversation
| because they are a wrong platform for deploying databases.
| In other words, it doesn't matter how people deploy
| databases to smartphones -- they have no hopes of achieving
| good performance, and whether they use mmap or not is of no
| consequences -- they've lost the race before they even
| qualified for it.
|
| > LMDB
|
| Are we talking about this?
| https://en.wikipedia.org/wiki/Lightning_Memory-
| Mapped_Databa... If so, this is irrelevant for databases in
| general.
|
| > LMDB databases may have only one writer at a time
|
| (Taken from the page above) -- this isn't a serious
| contender for database server space. It's a toy database.
| You shouldn't give general advice based on whatever this
| system does or doesn't.
| benlivengood wrote:
| For all of its usefulness in the good old days of rusty disks I
| wonder if virtual memory is worth having for dedicated databases,
| caches, and storage heads. Avoiding TLB flushes entirely sounds
| like a huge win for massively multithreaded software and memory
| management in a large shared flat address space doesn't sound
| impossibly hard.
| jasonhansel wrote:
| I've become convinced that there are very few, if any, reasons to
| MMAP a file on disk. It seems to simplify things in the common
| case, but in the end it adds a massive amount of unnecessary
| complexity.
| AnotherGoodName wrote:
| Complexity? You mmap it in and then read the multi terrabyte
| file as if it was an array.
|
| The opposite with actual file io sucks in terms of complexity.
| I get that you can write bespoke code that performs better but
| mmap is a one liner to turn a file into an array.
| Dwedit wrote:
| Need to handle the exceptions/signals every time a disk read
| fails. With classic IO, you know when the read will happen.
| But with memory-mapped files, the exception can happen at any
| time you are reading from the memory range.
|
| As for why disk reads fail, yes that's a thing. Less common
| on internal storage (bad sectors), but more common on
| removable USB devices or Network drives (especially on wifi).
| Sesse__ wrote:
| Multi-terabyte? Better hope you have lots of spare RAM for
| all those page structures the kernel has to keep.
| gavinray wrote:
| "mmap" in the general case is incredibly useful.
|
| There's so much you get "for free" and the UX/DX of
| reads/writes to it, especially if you're primarily operating on
| structs instead of raw byte/string data.
|
| (Example, reading a file and "reinterpret_cast<>"'ing it from
| bytes to in-memory struct representations)
|
| It's just that for the _particular_ case of a DBMS that relies
| on optimal I/O and transactionality, the general-purpose kernel
| implementation of mmap falls short of what you can implement by
| hand.
| vvanders wrote:
| It's incredibly useful in read-only, memory constrained
| scenarios. I.E. we used to mmap all of our animation data on
| many rendering engines I worked on where having ~20-50mb of
| animation data and only "paying" a couple 10s of kb based on
| usage patterns was very handy. It becomes even more powerful
| when you have multiple processes sharing that data and the
| kernel is able to re-use clean pages across processes.
|
| From reading the paper most of the concerns are around the
| write side. LMDB is the primary implementation that I know
| which leans heavily into mmap but it also comes with a number
| of constraints there(single writer, read locks can lead to
| unbounded appending to the WAL, etc). As with any tech choice
| it's about knowing constraints/trade-offs and making
| appropriate choices for your domain.
| jjtheblunt wrote:
| if you truss starting up a binary, the OS normally mmaps the
| binary, at least in tests i ran.
| mpweiher wrote:
| Yes, I definitely would _want_ to use mmap() in my storage
| system. And would love to see the limitations that make this
| tricky addressed.
| jandrewrogers wrote:
| Another interesting limitation of mmap() is that real-world
| storage volumes can exceed the virtual address space a CPU can
| address. A 64-bit CPU may have 64-bit pointers but typically
| cannot address anywhere close to 64 bits of memory, virtually or
| physically. A normal buffer pool does not have this limitation.
| You can get EC2 instances on AWS with more direct-attached
| storage than addressable virtual address space on the local
| microarchitecture.
| jFriedensreich wrote:
| maybe a stupid question but what is wrong with coffee and spicy
| food?
| toxik wrote:
| Just doesn't taste good together I think
| orf wrote:
| For the majority of the world, nothing. But if your diet
| consists of fairly bland food then it can result in unpleasant
| trips to the toilet.
| mattnewton wrote:
| Acid reflux I thought
| wood_spirit wrote:
| Old timers will recall when using mmap was a prominently promoted
| selling point for the "no sql" dbms.
| nemo44x wrote:
| For documents it made access fast since there's no joins, etc.
| that require paging from all over. The problem ended up being
| updates and compaction issues.
| wood_spirit wrote:
| My memory is that the problem was ACID. The document stores
| didn't promise to be reliable because apparently that didn't
| scale.
|
| And there was a very well known cartoon video discussion
| about it with "web scale" and "just write to dev null" and
| other classics that became memes :)
| cratermoon wrote:
| Did you ever read Pat Helland's article, "Life Beyond
| Distributed Transactions: An apostate's opinion"
| https://dl.acm.org/doi/10.1145/3012426.3025012? "This
| article explores and names some of the practical approaches
| used in the implementation of large-scale mission-critical
| applications in a world that rejects distributed
| transactions."
| wood_spirit wrote:
| No I haven't. Thanks for the interesting link :)
|
| Admittedly I live in a world where big distributed
| transactions are a given and work fine and sql speeds us
| up not slows us down. I'm guessing sql and acid scaled
| after all?
| cratermoon wrote:
| > I'm guessing sql and acid scaled after all?
|
| Yes and no. Distributed transactions and two-phase commit
| have been superseded by things like Paxos and Raft, with
| a variety of consistency models, so the implementation is
| drastically different.
| ren_engineer wrote:
| seems like all databases are moving towards the middle.
| Postgres has JSON support, MongoDB has transactions and also a
| columnar extension for OLAP type data. NoSQL seems almost
| meaningless as a term now. Feels like a move towards a winner
| takes all multi-modal database that can work with most types of
| data fairly well. Postgres with all of it's specialized
| extensions seems like it will be the most popular choice. The
| convenience of not having to manage multiple databases is hard
| to beat unless performance is exponentially better, Postgres
| with these extensions can probably be "good enough" for a lot
| of companies
|
| reminds me of how industries typically start out dominated by
| vertically integrated companies, move to specialized horizontal
| companies, then generally move back to vertical integration due
| to efficiency. Car industry started this way with Ford, went
| away from it, and now Tesla is doing it again. Lots of other
| examples in other industries
| TheGeminon wrote:
| The pendulum swing is common in any system, and is a really
| effective mechanism for evaluation.
|
| You almost always want somewhere in the middle, but it's
| often much easier to move back after a large jump in one
| direction than to push towards the middle.
| [deleted]
| Dwedit wrote:
| Memory-Mapped Files = access violations when a disk read fails.
| If you're not prepared to handle those, don't use memory-mapped
| files. (Access violation exceptions are the same thing that
| happens when you attempt to read a null pointer)
|
| Then there's the part with writes being delayed. Be prepared to
| deal with blocks not necessarily updating to disk in the order
| they were written to, and 10 seconds after the fact. This can
| make power failures cause inconsistencies.
| wmf wrote:
| I wonder how many apps don't handle errors from read() anyway.
| afr0ck wrote:
| Linux throws a SIGBUS. A process should anticipate such I/O
| failures by implementing a SIGBUS handler, especially a
| database server.
|
| For the second part of your comment, on Linux systems, there is
| the msync() system call that can be used to flush the page
| cache on demand.
| crabbone wrote:
| > msync() system call that can be used to flush the page
| cache on demand.
|
| _for everyone_ , not just the file you mapped to memory.
| I.e. the guarantee is that your file will be written, but
| there's no way to do that w/o affecting others. This is not
| such a hot idea in an environment where multiple threads /
| processes are doing I/O.
| kentonv wrote:
| > Be prepared to deal with blocks not necessarily updating to
| disk in the order they were written to, and 10 seconds after
| the fact. This can make power failures cause inconsistencies.
|
| This is not specific to mmap -- regular old write() calls have
| the same behavior. You need to fsync() (or, with mmap, msync())
| to guarantee data is on disk.
| crabbone wrote:
| > This is not specific to mmap -- regular old write() calls
| have the same behavior.
|
| This is not true. This depends on how the file was opened.
| You may request DIRECT | SYNC when opening and the writes are
| acknowledged when they are actually written. This is
| obviously a lot slower than writing to cache, but this is the
| way for "simple" user-space applications to implement their
| own cache.
|
| In the world of today, you are very rarely writing to
| something that's not network attached, and depending on your
| appliance, the meaning of acknowledgement from write()
| differs. Sometimes it's even configurable. This is why
| databases also offer various modes of synchronization -- you
| need to know how your appliance works and configure the
| database accordingly.
| sidewndr46 wrote:
| does that get delivered as SIGSEGV to the process or something
| else?
| afr0ck wrote:
| On Linux, it's a SIGBUS.
| zffr wrote:
| The TLDR is that MMAP sorta does what you want, but DBMSes need
| more control over how/when data is paged in/out of memory.
| Without this extra control, there can be issues with
| transactional safety, and performance.
| kwohlfahrt wrote:
| It sounds like a lot of the performance issues are TLB-related.
| Am I right in thinking huge-pages would help here? If so, it's a
| bit unfortunate they didn't test this in the paper.
|
| Edit: Hm, it might not be possible to mmap files with huge-pages.
| This LWN article[1] from 5 years ago talks about the work that
| would be required, but I haven't seen any follow-ups.
|
| [1]: https://lwn.net/Articles/718102/
| dist1ll wrote:
| Many general-purpose OS abstractions start leaking when you're
| working on systems-like software.
|
| You notice it when web servers are doing kernel bypass to for
| zero-copy, low-latency networking, or database engines throw away
| the kernel's page cache to implement their own file buffer.
| kentonv wrote:
| Yes. I think mmap() is misunderstood as being an advanced tool
| for systems hackers, but it's actually the opposite: it's a
| tool to make application code simpler by leaving the systems
| stuff to the kernel.
|
| With mmap, you get to avoid thinking about how much data to
| buffer at once, caching data to speed up repeated access, or
| shedding that cache when memory pressure is high. The kernel
| does all that. It may not do it in the absolute ideal way for
| your program but the benefit is you don't have to think about
| these logistics.
|
| But if you're already writing intense systems code then you can
| probably do a better job than the kernel by optimizing for your
| use case.
| arter4 wrote:
| Web servers doing kernel bypass for zero-copy networking? Do
| you have a specific example in mind? I'm curious.
| dist1ll wrote:
| The most common example is DPDK [1]. It's a framework for
| building bespoke networking stacks that are usable from
| userspace, without involving the kernel.
|
| You'll find DPDK mentioned a lot in the networking/HPC/data
| center literature. An example of a backend framework that
| uses DPDK is the seastar framework [2]. Also, I recently
| stumbled upon a paper for efficient RPC networks in data
| centers [3].
|
| If you want to learn more, the p99 conference has tons of
| speakers talking about some interesting challenges in that
| space.
|
| [1] https://www.dpdk.org/.
|
| [2] https://github.com/scylladb/seastar
|
| [3] https://github.com/erpc-io/eRPC
| arter4 wrote:
| Interesting. I hear a lot more about sendfile(), kTLS and
| general kernel space tricks than I do about DPDK and
| userspace networking, but maybe it's just me.
|
| I do wonder what trend is going to win: bypass the kernel
| or embrace the kernel for everything?
|
| The way I see it, latency decreases either way (as long as
| you don't have to switch back and forth between kernel and
| user space), but userspace seems better from a security
| standpoint.
|
| Then again, everyone is doing eBPF, so probably the
| "embrace the kernel" approach is going to win. Who knows.
| kentonv wrote:
| Probably the most common example is sendfile() for writing
| file contents out to a socket without reading them into
| userspace:
|
| https://man7.org/linux/man-pages/man2/sendfile.2.html
| loeg wrote:
| Sendfile isn't kernel bypass.
| arter4 wrote:
| Yes, I knew about sendfile() but I wasnt't aware of any web
| server using that (though I know Kafka uses it).
|
| Then I found out Apache supports it via the EnableSendfile
| directive. Nice.
|
| >This directive controls whether httpd may use the sendfile
| support from the kernel to transmit file contents to the
| client. By default, when the handling of a request requires
| no access to the data within a file -- for example, when
| delivering a static file -- Apache httpd uses sendfile to
| deliver the file contents without ever reading the file if
| the OS supports it.
| kentonv wrote:
| I'd expect most serious web servers support it. I've
| written one that does (workerd), it's not too hard.
|
| That said, it's tricky to use if the server also does TLS
| termination... then you need kTLS, which is a much bigger
| can of worms.
| nh2 wrote:
| Pretty much all modern Linux web servers support
| sendfile(). Examples:
|
| * nginx: [1] * Haskell webserver module: [2] * caddy: [3]
|
| [1]: https://nginx.org/en/docs/http/ngx_http_core_module.
| html#sen... [2]: https://hackage.haskell.org/package/warp
| -3.3.28/docs/Network... [3]:
| https://github.com/caddyserver/caddy/pull/5022
| mrfox321 wrote:
| Isn't that the opposite? That is, bypassing user space, not
| kernel space?
| kentonv wrote:
| Oh, hmm, yeah, perhaps OP meant something more like using
| raw sockets to get packets directly into userspace
| without relying on the kernel to arrange them into
| streams?
|
| I'm not very familiar with that though.
___________________________________________________________________
(page generated 2023-07-02 23:01 UTC)