[HN Gopher] Is sequential IO dead in the era of the NVMe drive?
___________________________________________________________________
Is sequential IO dead in the era of the NVMe drive?
Author : eatonphil
Score : 87 points
Date : 2023-05-09 19:05 UTC (3 hours ago)
(HTM) web link (jack-vanlightly.com)
(TXT) w3m dump (jack-vanlightly.com)
| issafram wrote:
| I'm still of the opinion that even with SSD/NVMe, the random
| reads/writes are still very slow.
| 95014_refugee wrote:
| This obviously depends substantially on the implementation, but
| (especially for standalone devices) there tends to be metadata
| access associated with translating a given LBA to a physical
| page. This brings cache locality of reference into the picture,
| and any workload that is "random enough" such that the metadata
| working set exceeds the cache will suffer...
| exabrial wrote:
| In my experience, depends on the SSD. Consumer grade ones
| appear really fast because they have a DRAM cache, and as soon
| as you start having a bunch of cache-misses you realize how
| slow they actually are.
|
| We got a bunch of enterprise grade Samsung SSDs and there'a
| large difference in sustained I/O. It's not "instant" by any
| means, but it there are other things slowing I/O.
| zamadatix wrote:
| Only the very best drives hit >200 MB/s in 4k write (even
| with DRAM as cache) and for 4k mixed not even Optane drives
| get that high.
| willis936 wrote:
| That's why all of my minecraft servers are on RAM drives.
| marginalia_nu wrote:
| With SSDs it's more like a discount on sequential/adjacent
| reads within a physical block that is of a size that varies
| with model and manufacturer.
| thelastparadise wrote:
| I don't understand why so much emphasis is put on optimizing
| for sequential reads and writes. Sequential IO is not something
| that happens in the real world except in rare circumstances. In
| most applications IO is random, even if it's a log file.
| dmitrygr wrote:
| > In most applications IO is random, even if it's a log file.
|
| I am all ears... how are log writes random?
| wmf wrote:
| Relative to what?
| antx wrote:
| "Writes to the WAL are purely sequential and writes to long-term
| storage are purely sequential."
|
| Is that simply a typo or am I missing something? Shouldn't it be
| mostly random for long-term storage?
| jasonwatkinspdx wrote:
| No. In WAL based systems new writes are appended to the write
| ahead log, and the dirty database pages held in ram.
| Periodically, a checkpoint process writes out all those dirty
| pages to long term storage, then truncates the write ahead log.
| So both these writes can be sequential. This is one of the
| reasons databases have the performance they do vs just using
| lots of little files like a database naively.
| eatonphil wrote:
| > So both these writes can be sequential.
|
| In the worst case though, the long-term storage can still
| have random writes though no (for btrees anyway)? I figure
| LSM tree writes can always be sequential, even during
| compaction. But btrees can only do solely sequential writes
| if the entire tree needs to be rewritten. And you don't want
| to always do that if only parts of the tree need to be
| updated?
| saltcured wrote:
| I would take it as a simplification to really mean
| "sequential enough" when writing out the reorganized copy
| to the long-term storage.
|
| With all the different layers in modern storage, I think
| you can get asymptotically close to the sequential rate
| once you start hitting some of these other multi-block
| granularities, i.e. around the block size for flash
| erasure, encryption, redundancy coding, etc. These don't
| have to be that big, i.e. several megabytes.
| cperciva wrote:
| Moving from spinning rust to solid-state storage dramatically
| improved the performance of random _reads_ , but random _writes_
| still carry a penalty compared to sequential writes. So
| sequential I /O is perhaps half-dead; there's no need to optimize
| layouts for sequential reads any more.
| xenadu02 wrote:
| Flash controllers provide an abstraction layer that should turn
| any write workload into a sequential one, freeing software
| developers from having to care about it.
| wtallis wrote:
| That abstraction layer cannot always avoid creating write
| amplification when you send it writes that are too small.
| GauntletWizard wrote:
| There's still a ton of performance benefit that can be gained
| by keeping your read pipeline full, and you can most easily
| keep your read pipeline full by making it guessable by doing
| readahead. It doesn't need to be strictly disk-layout
| sequential, file-layout sequential will usually do.
| Proven wrote:
| Far from it, sequential IO optimization isn't dead.
|
| Sequential reads need to be optimized to produce optimal
| performance for selected workload. That usually means applying
| a lower level of inline compression.
|
| In some cases deduplication works better before, in other
| after, compression. Sometimes post-process dedupe is more
| suitable than inline.
|
| Then there's erasure coding and data protection methods that
| are still being optimized for NVMe with sequential workloads,
| including random workloads which are being sequentialized to
| work better with latest flash media.
|
| I would even say developments in sequential IO are becoming
| more important than random IO.
| loeg wrote:
| > So sequential I/O is perhaps half-dead; there's no need to
| optimize layouts for sequential reads any more.
|
| Yes and no. Large contiguous IOs are still faster to read than
| a bunch of random small sectors. You generally want your
| frequently read files/blobs to be split into as few operations
| as possible.
|
| https://dl.acm.org/doi/abs/10.1145/3477132.3483593
| [deleted]
| almog wrote:
| To add to that, the penalty for random writes is paid not only
| with performance but also in durability of the drive.
| ilyt wrote:
| Sequential reads will still generally be faster at least for
| small ones, just that NVMe are fast enough that this rarely
| matters
| [deleted]
| jasonwatkinspdx wrote:
| Yes. And the Flash Translation Layer inside modern drives are
| quite sophisticated. This means you could start out with a
| write pattern that's causing a lot of copying and
| reorganization of pages in the back ground, typically because
| it's doing a lot of unaligned random writes smaller than the
| erase block size, and initially the performance will be fine.
| But as the drive wears in, and the capacity fills up, the FTL
| has less extra space to work with, forcing this garbage
| collection activity to become increasingly common. So you get
| performance degradation and premature wearing of the drive in a
| way that's opaque.
| bcrl wrote:
| Short stroking SSDs is a commonly used trick in the toolbag
| of wise sysadmins for dealing with this kind of workload.
| oakwhiz wrote:
| I wish that it was easier to trim unused unpartitioned
| space. I find myself fumbling on BSD and Linux trying to
| remember how to get the exact offsets for that. You could
| write zeroes but there isn't a guarantee that the FTL will
| interpret that as unused.
| Waterluvian wrote:
| I'm guessing that means pretending they're smaller than
| they really are?
| wtallis wrote:
| Yes. The term dates back to hard drives, where using only
| a fraction of their capacity would minimize the worst-
| case distance the heads needed to travel, reducing the
| portion of seek latency due to head movements (but not
| helping with rotational latency).
|
| On most hard drives, the beginning of the logical block
| address space corresponds to the outer edges of the
| platters where the bits are going past heads with the
| highest linear velocity, so sequential throughput is
| higher than elsewhere on the disk.
| colordrops wrote:
| Looks like that's the case based on a cursory google
| search. Seems analogous to only charging batteries up to
| 80%. I wish companies would factor these issues into
| their products and leave invisible buffers, but I guess
| making them cheaper is more important than being
| consistent and reliable.
| Waterluvian wrote:
| Sales and marketing will never leave anything on the
| table that might make their job easier.
| wtallis wrote:
| They _do_ leave invisible buffers, commonly referred to
| as over provisioning. The discrepancy between GB and GiB
| is ~7% hiding in plain sight, and then more expensive
| enterprise drives are commonly sold in (usable)
| capacities like 960GB and 7.68TB rather than 1TB and 8TB.
| jrockway wrote:
| I always worry that hardware "locks in" software. SSDs showed up
| after everyone wrote their databases to be optimal on spinning
| rust, so SSDs were built to look at the flow of requests, assume
| that the application was written for spinning rust, and optimize
| the data layout accordingly. Whether or not this is as good as
| brand-new software that managed the layout itself is debatable,
| and nobody will write that software anyway because they won't
| have users that have SSDs that let the database manage the disk
| in full. So the software has to tune it for one manufacturer's
| reorganization algorithm and hope for the best. The path to a
| "global maximum" remains unclear because of this.
|
| CPUs are similar. People started writing programs in C, so CPUs
| started optimizing for the outputs of C compilers. If someone
| were to invent CPUs and compilers today, the list of
| optimizations would probably be different, and the performance
| characteristics of real-world software would probably be
| different. (It's not just C; people wanted virtual address
| spaces, and that was slow in software, so now there is a TLB,
| etc.)
|
| Actually, it goes even deeper. Half the startups I see on HN
| related to software operations have a quickstart like "don't
| worry, you don't have to rewrite your code, we'll magically do
| everything for you". Why not just refactor the code? It would
| take about an hour, and then you don't need a crazy Rube Goldberg
| machine to achieve your desired results. (If you want specifics,
| think service meshes, and how they now detect what language your
| app is written in so they can rewrite your HTTP handling
| functions to pass around a trace ID. Back in the day, you just
| set outgoing.Headers["x-trace-id"] to incoming.Headers["x-trace-
| id"]. Now we have layers and layers of OS-level machinery that
| still don't do as good a job as spending half a day adjusting
| your codebase. Billions of dollars invested in saving half a
| developer day! Wow!)
| JoeAltmaier wrote:
| Deeper than that! Such fundamentals as 'the stack is accessible
| by code' goes back to Fortran. C doesn't need that. The call
| stack and the 'display' stack could be different things and no
| C programmer need care.
|
| This is the root of attacks that rewrite the return address of
| a kernel call. If the return address were managed/protected
| outside normal data operation this would not exist.
| paulmd wrote:
| Probably the biggest ground-up rethink of SSDs that I've seen
| is the Samsung SSDs with a key-value store mode.
|
| Like, think about it, a filesystem is really a database, right?
| Copy-on-write even makes the "transactionality" explicit. And a
| lot of high-performance databases will go farther and skip the
| filesystem and treat the device as block storage... which it
| is. The filesystem is a leaky abstraction with filesystem
| blocks and flash pages and flash block erasure, etc.
|
| Well, what if the SSD was just an object store? In that model
| you slice away a bunch of layers of abstraction and just let
| the SSD worry about all that. SSD says it's committed? Alright
| then, guess it is.
|
| Obviously you are very much at the mercy of the SSD to
| implement ACID correctly however...
| wtallis wrote:
| Typical SSDs all assume your IO will be in 4kB-aligned chunks
| even though the storage is denominated in 512-byte sectors for
| compatibility reasons. So we have managed to move past the
| legacy of early hard drives to some degree, and are now being
| held back by assumptions that derive more from x86 page sizes.
| [deleted]
| layer8 wrote:
| Tldr: No - "the benefits of sequential IO are alive and well,
| even in this new era of NAND flash."
|
| (Betteridge's law of headlines strikes again.)
| rektide wrote:
| It's weird to me that there's so many promising reliefs that seem
| near at hand, but which have simply never been delivered. Zoned
| Storage has existed for a while, for Shingled Magnetic Recording
| & tape, and there was much hubbub in 2018-2020 about NVMe getting
| Zoned Namespaces, which happened. https://zonedstorage.io/
|
| And there's now a bunch of fs implementations. f2fs and btfs both
| have some support.
|
| But there's still no products actually available to buy. We could
| be getting so much better at NVMe, so systematically making
| things better. But we've kind of been stalled out for a while,
| after a bunch of ceremony figuring out what we wanted to do.
| bcrl wrote:
| SMR and zoned storage offer compete and total utter crap
| performance in the real world. They have all the glass jaws of
| early flash firmware (garbage collection that took seconds to
| complete during which time no additional I/Os completed) while
| running on a medium that has many orders of magnitude higher
| latency than flash. Take 2008 era USB flash drives and that's
| about what you'll get out of an SMR HDD when using a write
| workload that isn't 100% large sequential writes.
|
| SMR might have become more relevant if it had provided a useful
| increase in density, but that never materialized as was
| originally promised. Retail prices for SMR drives were never
| much better than CMR. The largest generally available HDDs are
| available in CMR flavours because that's what's needed in real
| world servers. There's no 10x density improvement (or even 2x).
| Meanwhile flash is marching relentlessly down the cost
| reduction path afford by Moore's law + 3D layer stacking.
| Really fast 1TB NVMe SSDs are under $100 now, and it doesn't
| look like flash cost reduction is going to slow down any time
| soon.
| rektide wrote:
| I wasn't really talking about SMR though, just some protocols
| they had that are now being re-used. I'm talking about Zoned
| Namespace SSDs (ZNS).
|
| Let's go back in time almost a decade. A bunch of smart
| people had figured out that the cumbersome Flash Translation
| Layer (FTL) on SSDs made it really hard to get expectable &
| consistent performance. They were building possible specs to
| try to directly access each flash block, to be able to fill
| them up as they saw fit & clear them out as they saw fit.
| They wanted direct access, with far less work juggling
| complex data-mappings done on the SSD itself. Two examples of
| Open Channel SSD works:
| https://openchannelssd.readthedocs.io/en/latest/
| http://lightnvm.io/
|
| Note the huge banner on the second: " _Zoned Namespace (ZNS)
| SSDs have replaced the work on OCSSDs, and is now a
| standardized interface._ " Everyone realized that the
| protocols we had for Zoned Storage were basically adequate to
| get us what we wanted. We'd just make a lot of 128kB or
| whatever sized zones on the SSD, and let people manage them
| themselves.
|
| Currently this means opening blocks & then appending to them.
| There's been outstanding hope, in the future, that we might
| go further and allow more random write patterns, but still
| with the same contract of no over-writes (just clears).
|
| That work seemed like it was ready to go in 2020 & 2021. SSDs
| were supposedly sampling/becoming available.
| https://blog.westerndigital.com/zns-ssd-ultrastar-dc-
| zn540-s...
| https://semiconductor.samsung.com/newsroom/news/samsung-
| intr... And that was reaffirmed again a year latter.
| https://www.tomshardware.com/news/samsung-and-western-
| digita...
|
| But here we are in 2023 & there's still no Zoned Namespace
| SSDs (ZNS) one can purchase.
| b33j0r wrote:
| The thing that still bothers me is that the read-write cycles
| and lifetime are sort of unpredictable to me personally.
|
| For an analogy, I only buy LED lightbulbs, love that tech. When
| they first became widely-available, my house had a few
| incandescent bulbs on the porch that I probably just left on
| for 5 years.
|
| The supposedly superior led bulbs in that adoption period often
| died surprisingly fast, compared to the claims.
|
| To bring it back to storage. I use as much NVMe as I can, but
| I'm still a bit uncertain on how much a risk I'm taking with my
| filesystem choice and access patterns.
|
| My answer: buy more SSDs when they go on sale! I don't have
| intuition about lifetime even now.
|
| I know, HDDs have equivalent classes of worse problems. I just
| learned to expect them to reliably fail, like a few days after
| a build ;)
| macjohnmcc wrote:
| The recent issues with Samsung's SSD's don't help with those
| feelings I'm sure. I have a WD-Black 1TB SSD in my system
| that I built in 2018 and I used it heavily all day long and I
| have not suffered any issues from the SSD. I also used a
| MacBook Pro from early 2013 and only recently sold it still
| going strong with it's SSD (proprietary). I read recently
| that some HDD were starting to have reliability problems
| after a few years.
| kijiki wrote:
| Google, Facebook, Microsoft, Amazon and so forth buy most of
| the spinning rust, and have for years. Each of them have their
| own extremely proprietary storage layer that knows how to
| manage Shingled Magnetic Recording.
|
| So when SMR got released into the open market, it was
| exclusively drive managed, and mainly on the low-end, as a cost
| reduction strategy. With all the performance downsides you'd
| expect.
|
| It is a real shame that host managed SMR isn't available
| outside the mega-scale corps. It would be nice to have access
| to an intelligent storage stack mixing flash and SMR drives
| without having to go work for one of the big N companies.
| numpad0 wrote:
| Completely tangential, but Yangtze Memory/YMTC 232-layer NAND
| flash that was supposed to be used in iPhone 14 is on fire sale
| due to US import restrictions, to the point that some speculates
| that YMTC will be gone soon.
|
| The point is, DRAM-less 2TB M.2 NVMe disks are at $80 right now
| in select non-US markets. This is mere ~5x HDD, and _slow_ as
| 4Gbps on writes at worst conditions.
| zamadatix wrote:
| DRAM-less 2 TB m.2 NVMe drives are <$80 in the US market as
| well https://www.amazon.com/dp/B08CDM2HSS. There are a couple
| such drives in this price range, even from companies like Intel
| with their 670p.
| rektide wrote:
| Minor detail, Intel sold their SSD business to SK Hynix for
| $7b. SK Hynix created a new company Solidigm out of the
| acquisition. So it is now a Solidigm 670p.
| https://www.solidigm.com/products/client/d6/670p.html
|
| Personally it really felt a bit rotten to me that Intel had
| created drives like the 670p that would simply stop working
| after a designated amount of writes. On the one hand,
| predictable life-span is kind of of a good thing, but sending
| drives to the landfill before they're actually done seems
| monstrous. Anyways though, yeah, DRAM-less drives are
| available for incredible prices.
| jonstewart wrote:
| The interesting question --- I'd appreciate any links --- is how
| to change our programming patterns to get the most performance
| out of NVMe SSDs. I've seen before that it takes a lot of
| parallel requests to saturate a fast NVMe drive --- but how to
| change our application architectures to generate such a workload?
| COGlory wrote:
| Not to harp too much on the title (I read the article and found
| it very interesting), but are there any industries storing their
| data on all-flash? My intuition is that most institutions still
| have a giant array of spinning rust somewhere underpinning
| everything they do.
| tatersolid wrote:
| Our datacenter at $dayjob retired our last spinning rust drive
| way back in 2017. Backups are done over the wire to dedupe-
| enabled targets 600km away; dedupe restore times are shit
| unless you use all-SSD there as well.
| rch wrote:
| I see a fair number of all SSD clusters, but most have a mix of
| drive types. Interestingly, the ratio of drive types is aligned
| with optimal performance rather than cost.
| birdman3131 wrote:
| Outside of backups I would expect many business to be all SSD.
| Maybe not the majority but for many use cases I see no good
| reason to go HDD for most small businesses.
|
| I know I went pretty much all SSD ages ago. I do run backups to
| a few 14TB externals but nothing outside of that is spinning
| rust.
| [deleted]
| kccqzy wrote:
| I do not think it is feasible to store all of YouTube videos
| on SSD. Google must be using spinning rust for the long tail
| of videos.
| girvo wrote:
| I ditched my final spinning rust (a WD Black 2.5" 7200RPM
| drive I had jammed into my home server) last night actually.
| It's all SSDs in my server, and it's all NVMe in my gaming
| desktop. My Macs and my partner iPad are all flash too
| obviously.
| loeg wrote:
| Gobs. This is Pure Storage's entire market.
| xmonkee wrote:
| What a great and clear write-up. Thanks to the author.
| jscipione wrote:
| COW file systems are pretty amazing, especially on flash memory-
| based storage.
| loeg wrote:
| Even setting aside internal GC, there is read-side benefit to
| having fewer contiguous sections: your large read request needs
| to be split into fewer IOs. This matters if you're reading the
| output of this workload.
| https://dl.acm.org/doi/abs/10.1145/3477132.3483593 discusses it
| in a sciencey way.
| gfody wrote:
| you'd need something byte addressable for access pattern not to
| matter.
| sylware wrote:
| Huh?
|
| PCIE is serial.
| wmf wrote:
| That's true(ish) but it has nothing to do with sequential
| access patterns in LBA space.
| dboreham wrote:
| Everything is serial.
| sylware wrote:
| At least you could have said they are several lanes, a bit of
| //.
| hot_gril wrote:
| No, because flash memory (and so-called RAM) still do sequential
| reads and writes faster, but for sure I'll tune a default
| Postgres config to give _less_ of a penalty to random reads.
| revelio wrote:
| Amazing how much faster f2fs is than btrfs. Now I have some
| buyer's remorse over using btrfs...
| infamouscow wrote:
| If you like f2fs, you might also be excited about ssdfs:
| https://news.ycombinator.com/item?id=34939248
| bcrl wrote:
| Or bcachefs...
| rektide wrote:
| What are you citing or talking about? Neither f2fs nor btrfs
| show up in the article.
| kardos wrote:
| See the reproduced Figure 9
| ilyt wrote:
| It was always slow-ish compared to traditional ones, same with
| ZFS in some workloads
| thewataccount wrote:
| It's because of the COW I think right?
| cainxinth wrote:
| GPT-4 summary:
|
| > Despite the high performance of SSDs on random IO workloads,
| sequential IO remains relevant due to the housekeeping processes
| like wear leveling and garbage collection required by SSDs,
| including NVMe drives. Sequential IO reduces garbage collection
| overhead and write-amplification, while random IO leads to more
| fragmentation. Over-provisioning can mitigate garbage collection
| costs for random IO at the expense of storage density and cost.
| NVMe drive technology advancements have improved performance, but
| fragmentation and housekeeping challenges persist. Address
| remapping algorithms offer potential solutions to random IO
| limitations.
| hawk_ wrote:
| > There are different techniques for implementing OP (Over
| provisioning) yourself. You can simply leave a portion of the
| drive unpartitioned
|
| This doesn't sound right. Does this mean that the drive decides
| to arbitrarily move blocks between your partition and unpartioned
| spaces to manage GC?
| [deleted]
| Dalewyn wrote:
| Unlike hard drives of yore where ordering the drive to write a
| bit at a given location results in the drive writing a bit at a
| given location, SSDs abstract it away as a matter of their
| design.
|
| An SSD will still provide a virtual bitmap so file systems can
| still order it to write a bit at a given location like ye olde
| days, but the SSD controller ultimately decides where to
| actually write that bit which more than likely has no relation
| to the specified location.
|
| Incidentally, yes: This also means defragmenting an SSD is not
| only harmful, it is outright misleading because the state of
| the file system is not related to the state of the bits on the
| drive.
___________________________________________________________________
(page generated 2023-05-09 23:00 UTC)