[HN Gopher] Apple's custom NVMes are amazingly fast - if you don...
___________________________________________________________________
Apple's custom NVMes are amazingly fast - if you don't care about
data integrity
Author : omnibrain
Score : 598 points
Date : 2022-02-17 07:54 UTC (15 hours ago)
(HTM) web link (twitter.com)
(TXT) w3m dump (twitter.com)
| joshspankit wrote:
| Apple _doesn't_ care about on-disk data integrity.
|
| They just direct people to use Time Machine or iCloud, then look
| quizzically at you when have issue with writing off lost hours of
| work as cost of doing business.
| lowbloodsugar wrote:
| I'm not here to defend apple, but if you have a desktop and you
| don't want to lose data then get a UPS. Proper write handling on
| the disk wont help if you haven't saved your doc in ten minutes.
| throwawaylinux wrote:
| The dirty secret about today's high density NAND is that tPROG is
| not fast. It's an order of magnitude slower than the heyday of
| SLC. Now that doesn't really matter for enterprise drives, they
| complete writes into very fast storage that is made durable one
| way or another (e.g., flush on power fail), and this small store
| gets streamed out to the NAND log asynchronously. This is why
| random single queue depth _durable_ writes can actually be faster
| than reads on enterprise drives, because random reads have to
| come from NAND (tREAD is still very fast, just not as fast as
| writing to DRAM).
|
| Apple may not implement such a durable cache, that's fine it's
| not an enterprise device and it's a cost tradeoff. So they might
| have to flush to NAND on any FUA, and that's slow as we've said,
| but not 25ms slow. Modern QLC NAND tPROG latency is more like
| 2.5ms-5ms, which could just about explain the EVO results when
| you include the OS and SATA stack and drive controller.
|
| There's pretty close to 0% chance Apple would have messed this up
| accidentally though, in my opinion. It would have been a
| deliberate design choice for some reason. One possible reason
| that comes to mind is that some drives gang a bunch of chips in
| parallel end you end up with pretty big "logical" pages. Flushing
| a big logical page on a 4kB write is going to cause a lot of
| write amp and drive wear, so you might delay for a short period
| (20ms) to try pick up other writes and reduce your inefficiency.
| marcan_42 wrote:
| Nope, it's not a deliberate optimization / delay. Doing the
| flushes creates an extra ~10MB/s of DRAM memory traffic from
| the NVMe controller vs. not doing them while creating the same
| write rate. The firmware is doing something dumb when issued a
| flush command, it's not just sitting around and waiting.
|
| > There's pretty close to 0% chance Apple would have messed
| this up accidentally though, in my opinion
|
| There's pretty close to 100% chance Apple would not have
| cared/optimized for this when designing this SSD controller,
| because it was designed for iOS devices which always have a
| battery, and where next to no software would be issuing
| flushes.
|
| And then they put this hardware into desktops. Oops :-)
|
| Lots of things about the M1 were rushed and have been fixed
| along the way. I wouldn't be in the least bit surprised if this
| were one more of them that gets fixed a couple macOS versions
| down the line, now that I've made some noise about it.
| throwawaylinux wrote:
| > Nope, it's not a deliberate optimization / delay. Doing the
| flushes creates an extra ~10MB/s of DRAM memory traffic from
| the NVMe controller vs. not doing them while creating the
| same write rate.
|
| How are you measuring that and how do you figure it means the
| NAND writes are not being held off? Clearly they are by one
| means or another.
|
| > The firmware is doing something dumb when issued a flush
| command, it's not just sitting around and waiting.
|
| > There's pretty close to 100% chance Apple would not have
| cared/optimized for this when designing this SSD controller,
| because it was designed for iOS devices which always have a
| battery, and where next to no software would be issuing
| flushes.
|
| Yes. It is clear the _hardware_ was never optimized for it.
| Because it is so slow. I 'm almost certain that is a
| deliberate choice, and delaying the update is a possible
| reason for that choice. It's pretty clear the hardware can
| run this much faster, because it does when it's streaming
| data out.
|
| NAND and the controller and FTL just isn't rocket science
| that you'd have hardware that can sustain the rates that
| Apple's can and then through some crazy unforeseen problem
| this would suddenly go slow. Flushing data out of your cache
| into the log is the FTL's bread and butter. It doesn't
| suddenly become much more complicated when it's a synchronous
| flush rather than a capacity flush, it's the same hardware
| data and control paths, the same data structures in the FTL
| firmware and would use most of the same code paths even.
|
| Pull blocks from the buffer in order and build pages,
| allocate pages in NAND to send them, update forward map,
| repeat.
| marcan_42 wrote:
| > How are you measuring that
|
| powermetrics gives you DRAM bandwidth per SoC block, before
| and after the system level caches.
|
| > how do you figure it means the NAND writes are not being
| held off? Clearly they are by one means or another.
|
| I mean they're not _just_ being held off. It 's doing
| something, not waiting.
|
| > Yes. It is clear the _hardware_ was never optimized for
| it.
|
| This is a firmware issue. The controller runs on firmware.
| I can even tell you where to get it and you can throw it in
| a decompiler and see if you can find the issue, if you're
| so inclined :-)
|
| > I'm almost certain that is a deliberate choice, and
| delaying the update is a possible reason for that choice.
|
| Delaying the update does not explain 10MB/s of memory
| traffic. That means it's doing something, not waiting.
|
| > It's pretty clear the hardware can run this much faster,
| because it does when it's streaming data out.
|
| Indeed, thus it's highly likely this is a dumb firmware
| bug, like the FLUSH implementation being really naive and
| nobody having cared until now because it wasn't a problem
| on devices where nothing flushes anyway.
|
| > NAND and the controller and FTL just isn't rocket science
| that you'd have hardware that can sustain the rates that
| Apple's can and then through some crazy unforeseen problem
| this would suddenly go slow.
|
| Yup, it's not rocket science, it's humans writing code. And
| humans write bad code. Apple engineers write bad code too,
| just take a look at some parts of XNU ;-)
|
| > Flushing data out of your cache into the log is the FTL's
| bread and butter.
|
| Full flushes are rare on devices where the cache can be
| considered persistent anyway because there's a battery and
| the kernel is set up to flush on panics/emergency
| situations (which it is). Thus nobody ever ran into the
| performance problem, thus it never got fixed.
|
| > It doesn't suddenly become much more complicated when
| it's a synchronous flush rather than a capacity flush, it's
| the same hardware data and control paths, the same data
| structures in the FTL firmware and would use most of the
| same code paths even.
|
| The dumbest cache implementation is a big fixed size hash
| table. That's easy to background flush incrementally on
| capacity, but then if you want to do a full flush you end
| up having to do a linear scan even if the cache is mostly
| empty. And Apple have _big_ SSD caches - on the M1 Max the
| NVMe carveout is almost 1 gigabyte. Wouldn 't surprise me
| at all if there is some pathological linear scan going on
| in the case of host flush requests, or some other data
| structure issue. Or just an outright bug, a cache locality
| issue, or any other number of things that can kill
| performance. It's code. Code has bugs and performance
| issues.
| mhh__ wrote:
| I don't know what tPROG is (or anything else), is there a "What
| every programmer should know about storage" a la Drepper's work
| on memory?
| throwawaylinux wrote:
| tPROG is time it takes to program a NAND page from when you
| put the "program page" command on the pins to when you read
| off a successful status.
|
| Some of the basic NAND guides they put out are simple enough
| to understand the basics of operation
|
| https://www.micron.com/-/media/client/global/documents/produ.
| ..
|
| The details get very complicated and proprietary. NAND wears
| out as you use it. But it also has a retention time. It
| gradually loses charge and won't read back if you leave it
| unpowered for long enough. This is actually where enterprise
| drives can be speced worse than consumer. So durability /
| lifetime is specified as meeting specified uncorrected error
| rates at the given retention period. The physics of NAND are
| pretty interesting too and how it translates into how a
| controller optimizes these parameters. Temperature at various
| stages of operation and retention changes properties, time
| between erase and program does too. You can adjust voltages
| on read, program, erase, and those can help you read data out
| or change the profile of the data. Reading can disturb parts
| of other pages (similar to rowhammer). Multilevel cells are
| actually interesting some of them you program in passes so
| that's a whole other spanner in the works.
|
| I don't know of a good place that covers all that, but much
| beyond "read/program/erase + wear + retention" is probably
| beyond "what every programmer should know".
|
| The way you turn a bunch of NAND chips that have a
| "read/program/erase" programming model into something that
| has a read/write model (the flash translation layer or FTL)
| is a whole other thing again though. And all the endurance
| management and optimization, error correction... Pretty
| fascinating details really. The basic details though is that
| they use the same concepts as the "log structured
| filesystem", turns out a log structure with garbage
| collection is about a perfect it for turning the
| program/erase model into a random write model. That's
| probably what every programmer should know about that
| (assuming you know something about LSFs -- garbage
| collection, write amplification, forward and reverse mapping
| schemes, etc).
| mhh__ wrote:
| What every programmer should know in this context is a
| euphemism for how Drepper views that set of things to know
| i.e. Yes it's hard and yes really should know, you're a
| professional programmer. Storage is a little bit further
| away than memory, but it's still very important in certain
| lines of work
| SmellTheGlove wrote:
| > Apple may not implement such a durable cache, that's fine
| it's not an enterprise device and it's a cost tradeoff.
|
| I disagree with this - my Apple is an enterprise device. It's a
| Macbook Pro, issued by my employer, to do real work. I wouldn't
| give Apple a pass on this dimension. I get that the "Pro" label
| doesn't mean what it used to, but these aren't toys either.
| etaioinshrdlu wrote:
| As a laptop user I would probably opt to make the same choice as
| Apple here. I like the idea mentioned to allow a tunable
| parameter to only allow ever losing 1 second of data.
|
| Although, I also have the seemingly rare opinion here that ECC
| ram doesn't really matter on a laptop or desktop.
| mordae wrote:
| It's not only losing a couple seconds of data. Write ordering
| does not work, meaning journals don't. You get a possibility of
| silent data corruption.
| londons_explore wrote:
| But apple could quite easily fix write ordering
| DooMMasteR wrote:
| NVME even allows to make queues write through, so e.g. the
| kernel/fs driver could have/access the drive via a safe
| queue that always gets written. You can also prioritize
| queues to lower the chances of important data to be lost,
| though Apple seems to be super aggressive on caching and
| the drives tend to keep some written data in cache for
| quite long intervals.
| rowanG077 wrote:
| You think it's oke that restarting your PC leads to data loss
| or corruption? That's basically a product killer for me. I
| reboot my laptop everyday.
| marcan_42 wrote:
| You presumably don't reboot your laptop by connecting a USB-
| PD gadget that issues a hard reset. A normal OS reboot is
| fine, that will flush the cache.
|
| The most common situation where this would affect laptops, in
| my experience so far, would be a broken driver causing a
| kernel lockup (not a panic) which triggers a watchdog reboot.
| That situation wouldn't allow for an NVMe flush.
| rowanG077 wrote:
| Depends what exactly is a hard reboot. I don't reboot my
| laptop by issueing USB-PD command. But I do by holding the
| power button.
| Cthulhu_ wrote:
| See, that's a forced shutdown, a last resort measure;
| it's using a sledgehammer to tap in a nail. You shouldn't
| do that as a habit, even if this particular optimization
| issue wasn't a thing.
|
| I mean I grew up diligently turning off my PC by parking
| the disk and using the various operating system level
| shutdown procedures. Nowadays I smack the off button, but
| that still just triggers the OS shutdown procedure. I
| don't turn my Mac off as a rule, its sleep mode actually
| works. ish.
| Thomaschaaf wrote:
| Care to explain why?
| rowanG077 wrote:
| Systemd takes 2 minutes to shutdown and I never got any
| way to resolve that.
| vgel wrote:
| Linux Magic SysRq + R S E I V B key chord will
| immediately shut down while still properly flushing disk
| cache and such. A bit annoying to enter, but a handy tool
| to have in your toolbox.
| tashbarg wrote:
| That's not the right keys and not the right order to do
| that. You should not flush caches before you terminated
| as much processes as possible correctly. And you are
| rebooting at the end.
|
| REISUB for a somewhat safe EMERGENCY reboot and O instead
| of B at the end for shutdown.
| vgel wrote:
| Oh hmm, you're right. I've always done it with the other
| order and never had problems. Forget where I learned it
| that way...
| olliej wrote:
| If they're like me: outside of a software update I only
| reboot when the machine is not responding, at which point
| hard reboot is faster and more robust. I recognize it's
| not ideal, but I also don't think it's reasonable for the
| system to ever get to a point where I should be wanting
| to restart to "fix" it - and I would think it is a
| serious bug if doing so ever corrupted the system or lost
| any "saved" data.
| marcan_42 wrote:
| I just tested that. Holding down the power button invokes
| a (somewhat special) btn_rst kernel panic before it has a
| chance to invoke a true hardware reset, and kernel panics
| involve an NVMe driver hook which I'm pretty sure issues
| a flush. Should be safe.
|
| At least re: this issue; it's still a bad idea because
| it's only safe if all software is written following data
| integrity and flush rules to the letter, and most
| software isn't. You're eventually going to run into
| issues on any OS by doing that, because most software
| doesn't get this right unless it's a database. And you're
| still going to lose data that's in buffer cache, I'm
| pretty sure that won't get flushed.
| signal11 wrote:
| For products like the Mac Mini, which don't have a battery,
| does this mean that a loss of mains power will cause data
| loss? Because brownouts do happen occasionally...
| marcan_42 wrote:
| Yes. I've tested yanking the power and can easily see 5
| seconds of data loss for data that was fsync()ed (but not
| full synced). I'm not sure yet if corruption due to
| reordering is also possible, but it seems likely.
| vondro wrote:
| Shouldn't Mac OS issue flush on restart, as it does on sleep?
| Bud wrote:
| 1) A normal restart doesn't have this issue, at all.
|
| 2) Why are you rebooting a laptop daily? My uptime on my
| MacBook Pro averages 30-60 days. There's zero reason to
| reboot any modern OS daily.
| lobocinza wrote:
| > There's zero reason to reboot any modern OS daily.
|
| - I use Arch, I like to avoid accumulating too much major
| updates between reboots. - For a time I was facing a bug
| that resulted in a black screen of death after resuming
| sleep.
| koala_man wrote:
| > only allow ever losing 1 second of data
|
| For a database this means that every transaction will take a
| minimum of 1 second, otherwise you can't guarantee durability.
| yumraj wrote:
| This sounds like laptops are fine, but iMacs and Minis are effed.
|
| Curious, what's the real world risk of full OS level corruption
| and not just data loss?
| marcan_42 wrote:
| Good question. I just started up a loop doing USB-PD hard
| reboots on my MBA every 18 seconds (that's about one second
| into the desktop with autologin on, where it should still be
| doing stuff in the background). Let's see if it eats itself.
| joshspankit wrote:
| How can we get notified about your results?
| bestouff wrote:
| Famous last words
| marcan_42 wrote:
| This is just a test machine I also sometimes use as a dumb
| terminal around the house; I'm not going to cry if the OS
| eats itself :P
| masklinn wrote:
| Hopefully the ssd doesn't either though, bricking it
| would be hilarious but not ideal.
| marcan_42 wrote:
| Finding out if a DFU restore can recover a corrupted SSD
| storage would be an interesting test in and of itself!
|
| But to be honest, if I end up really bricking a machine
| for science, that will be worth it for the information it
| gives us. Obviously I'm not _trying_ to destroy my
| hardware, but I 'm very grateful that I can afford it if
| it happens thanks to all the support I'm getting from
| folks for the project.
| CrLf wrote:
| Laptops are fine unless your battery has issues and you get
| occasional power losses, which seems to be not too uncommon for
| third-party batteries (which themselves are not too uncommon
| since Apple will charge you an arm and a leg to replace half
| your laptop if you have a defective battery).
| marcan_42 wrote:
| Bad batteries generally allow for last-gasp handling, and
| I've definitely seen the SMC throw a fit on some properties a
| few seconds before shutdown due to the battery being really
| dead. Not sure if macOS handles this properly, but I'd hope
| it does, and if it doesn't they could certainly add the
| feature. It would be quite an extreme case to have a battery
| failure be so sudden the voltage doesn't drop slowly enough
| to invoke this.
| joshspankit wrote:
| iOS doesn't. A bad battery makes it think it has more time
| than it does, and cleanup tasks can get killed just as they
| start.
| chrismorgan wrote:
| A fair fraction of the bad batteries I have seen have not
| behaved like this. Things like immediate power failure on
| disconnecting AC power, or claiming to be at 30% and then
| dying, or denying the existence of the battery altogether
| (two of these have happened to me personally--one at the
| ripe age of four months rather than due to age--and three
| or four to other family members). It's certainly more
| common for them to just fade fairly rapidly to zero and die
| there, but it's by no means rare for them to spontaneously
| fall over.
| marcan_42 wrote:
| We're talking different timescales here. All you need is
| one second or so to command the NVMe controller to flush,
| and killing other power consumers in the mean time would
| buy you more time by reducing load, possibly even giving
| you several minutes the way batteries work (they tend to
| fall over under load when defective/dead). What may
| visually appear as power suddenly failing isn't
| necessarily so at the scale of voltage threshold
| interrupts and PMICs.
|
| What usually happens is battery internal resistance is
| too high to sustain a given power load, so once load
| crosses a threshold the system goes into a spiral of doom
| increasing current as battery voltage decreases and you
| end up in a shutdown. That's the "30% and suddenly 0% or
| a shutdown" scenario. But if you catch it before it's too
| late, you can just stop consuming power and let the NVMe
| controller flush.
| chrismorgan wrote:
| The case I have in mind where it would suddenly die
| around 30% would happen around that point regardless of
| load, even asleep, after following a sufficiently
| typically linear discharge curve up to that point. Maybe
| the power management system gets a fraction of a second's
| notice, I don't know; but it wasn't a 30% plummeting to
| zero over the course of ten or thirty seconds, or even a
| "30%; no--0%; no--dead" case, which seem to be the much
| more common failure modes. As for the "pull the AC power
| and it instantly dies" cases, I'm a layman in battery
| matters, with no more than high school electronics, but
| I'd be surprised if there's enough in there for it to do
| anything--those are cases where either it literally has
| no battery to draw on (because it's electronically dead),
| or thinks it has a battery but discovers as soon as it
| tries to draw on it that it effectively doesn't actually.
| joshspankit wrote:
| My personal experience with 3 Apple devices:
|
| "30% to 0" and "Pull AC and it instantly dies" are
| typically a combination of load and device temperature.
| High CPU/GPU usage, high brightness, 3G/LTE usage, and
| cold temps and the device doesn't have a chance.
|
| It's been somewhat fascinating to monitor power usage in
| this really crude way. TikTok on iOS, for example, uses
| so much power that it's the most likely to cause the
| device to shut off. FB Messenger is in the top 5. Some of
| Apple's background processes will also cause it, as will
| paging memory to disk.
|
| There's another bit of information that will not surprise
| many people on HN: high-amperage charging will cause the
| battery percentage to be "more wrong". Devices will
| report 45% or higher and still die as if they were
| reporting 30%. Charging at 500mA will not only make it
| "more correct", but will typically mean that a device
| will not suddenly die until it's in the single digits.
|
| This is still n=1 of course.
| marcan_42 wrote:
| If it's literally dying at 30% with no warning, it's
| either the battery polling being too slow (keep in mind
| the UI will usually only refresh once a minute or so for
| these things; the power management system has faster
| stats), or the charge estimation being way off. There's
| very little reason for a battery to drop from true 30%
| SoC to completely dead, without first going into a power
| draw spiral of doom which you can revert if you stop
| consuming as much power.
| winter_blue wrote:
| Wait, why are iMacs and Minis affected more? (I read the
| twitter thread; I'm not seeing why.)
| Tarq0n wrote:
| They're dependent on external power, which can acutely fail.
| cerved wrote:
| not battery powered
| Gaelan wrote:
| Laptops have batteries, so an AC power failure doesn't mean
| they immediately crash: they just keep running on battery
| until the battery gets low, at which point the system cleanly
| hibernates.
| coldcode wrote:
| Does anyone here run a desktop Mac without a battery backup
| device?
|
| All of my Macs are either laptops or have a hardware backup
| device, so unlikely a write would be lost due to power failure
| (unless backup device failed which could happen).
| wila wrote:
| Sure.. last power failure was like 4 years ago and the one
| before that was also measured in multiple years.
|
| Back when I still used a UPS down here, it was usually the
| UPS that died and triggered the power failure. So I stopped
| investing in a UPS.
| joshspankit wrote:
| Where I live the power is quite dirty, so even when power
| _losses_ are measured in years I invest in line-filtering
| UPS' to extend the life of my systems.
|
| I even lost a MBP to a light flickering event with 0 power
| loss. Fried the charging circuit straight through the
| original power brick.
| fmajid wrote:
| There's a reason why Apple uses Linux for its server
| infrastructure.
| BirAdam wrote:
| Sure, but I do not think it's due to their feeling that their
| own software is inferior. I think much more of that is cost.
| They needn't pay to develop yet another OS variant, and instead
| benefit off of the open source community and their past
| contributions to said community.
| sudosysgen wrote:
| They wouldn't really need to develop a variant. Plenty of
| people used to run servers on macOS just configured to be
| headless. It just doesn't meet the standard anymore.
| cutthegrass2 wrote:
| Wonder whether the AWS Mac EC2 instance types are affected too,
| anyone know?
| my123 wrote:
| You get an OS drive backed by EBS on those, through the AWS
| Nitro System.
|
| As such, they share the same storage infrastructure as other
| EC2 instances.
| aaaaaaaaaaab wrote:
| Do you really think those machines are just plugged into the
| mains socket in Amazon's data centers?
| rvense wrote:
| Yes. And the cheap instances use shorter leads that
| technicians might trip over at any given moment.
| voxadam wrote:
| https://xkcd.com/908/
| duxup wrote:
| Is it possible to quantify how likely you are to hit a data
| integrity issue because of this?
| StillBored wrote:
| Hmm as slow as that is, does the controller support VERIFY?
| Because there is FUA in verify which forces the range to flush as
| well, and it could be used as a range flush. Depending on how
| they implement the disk cache its possible that is faster than a
| full cache walk (which is likely what they are doing).
|
| This is one of those things that SCSI was much better at, SYNC
| CACHE had a range option which could be used to flush say
| particular files/database tables/objects/whatever to nonvolatile
| storage. Of course out of the box Linux (and most other OSs)
| don't track their page/buffer caches closely enough to pull this
| off, so that fsync(fileno) is closer to sync(). So, few storage
| systems implemented it properly anyway.
|
| The choice of ignoring flushes vaguely makes sense if you assume
| the mac's SSD is in a laptop with a battery. In theory then the
| disk cache is non-volatile (and this assumption is made on
| various enterprise storage arrays with battery backup as well,
| although frequently its a controller setting). But i'm guessing
| someone just ignored the case of the mac mini without a battery.
| supermatt wrote:
| I assumed the barrier was doing something like that, but marcan
| was able to inspect the actual nvme commands issued and has
| confirmed thats not the case.
|
| But that would be awesome, especially with these ever growing
| cache capacities.
| vbezhenar wrote:
| It's still not clear why Apple SSD so slow. Surely there's more
| to it. May be other SSDs are cheating in firmware? Or may be it's
| just bug in Apple firmware? I'm really interested if there will
| be follow ups on Apple side.
| marcan_42 wrote:
| Since this design is inherited from iDevices, my guess is they
| never bothered to optimize this command since software on a
| battery-powered device would almost never need to issue it. It
| should be something they can improve in firmware.
| joshspankit wrote:
| From my understanding, the thing that's slow is writing data to
| "permanent storage" (aka the layer under all the caching).
|
| Some storage tech is just slow at that, and manufacturers muddy
| the water by rating some (SSDs|Micro SDs|whatever) in GB/s
| overall when much of those big numbers are a combination of
| caches and trickery.
|
| I would not be surprised if Apple is using a tech that just has
| slow write speeds in trade for fast read speeds since most
| Apple users will be happy with faster read speeds.
| claudex wrote:
| Maybe it allow to design a device with less power consumption.
| ladyanita22 wrote:
| This is such an ugly hack...
| headmelted wrote:
| Honestly I don't know. The order-of-magnitude performance
| difference in deferring the flush feels worth it to me if the
| risk is mitigated to sudden power loss.
|
| I would think when the last of Apple's hardware moves to ARM
| they'll ensure there's enough onboard battery to ensure the
| flushes happen reliably across form factors even if there's a
| power cut.
|
| If anything, now that the reason for the performance difference
| has been identified, I'd hope to see numbers for Linux and
| Windows storage access come up to par with these numbers as
| they go down this road too (e.g. via the NVME flush toggle
| mentioned in the article).
| rfoo wrote:
| Yeah. If the same thing happens to a brand-less garbage SSD
| you purchased from Aliexpress it's clearly cheat and plainly
| malicious and incompetence, but the Apple tag certainly made
| us believe there is a second reason.
|
| Trading correctness for performance without shouting at the
| users "YOUR DATA IS NOT SAFE WHEN YOU DO THIS" multiple times
| a day in a storage is benchmark-snake-oil. Period.
| headmelted wrote:
| I'm not arguing for or against, I'm just pointing out that
| trading the possibility of data loss in the few seconds
| after a power cut for a difference of this magnitude
| actually makes sense in a lot of use cases.
|
| My point above was that the same "cheat" (to use your word)
| could be applied to the unbranded SSD too, with similar
| performance gains.
|
| I'm not giving Apple a pass for low flush performance, I'm
| saying there's nothing I can see here that's uniquely
| available to Apple that would prevent others from deferring
| flushes in the same way for similar performance gains -
| which would make sense in many cases.
| coldtea wrote:
| The same thing happens in POSIX, most unices, including
| Linux itself until recently.
|
| So there's that.
|
| It's rather Apple's slow drive firmware checks though, that
| is problematic.
| coldtea wrote:
| > _If anything, now that the reason for the performance
| difference has been identified_
|
| That's not the reason for the M1 performance differences (as
| a CPU).
|
| Just for the disk writing (which isn't the fastest around to
| begin with anyway).
| marcan_42 wrote:
| There's no good reason for this to be a performance
| tradeoff. Flushes taking this long on Apple SSDs has to be
| a dumb firmware performance bug.
| coldtea wrote:
| Agree, I mean that this fsync behavior is not what makes
| M1 fast as a platform (which the parent seemed to imply)
| - it just speeds up the disk part. The CPU is fast on its
| own.
| rowanG077 wrote:
| I think most SSD have dram cache on board. Could they design
| issue here be Apple doesn't have that and instead using system
| RAM as SSD dram cache?
| marcan_42 wrote:
| They do use system RAM as cache, but that has no effect on
| performance. If anything it should be way faster than the puny
| RAM cache chips on typical SSDs. It doesn't explain the slow
| flush perf.
| rowanG077 wrote:
| Afaik if you fsync an SSD with dram cache it won't hit the
| NAND cells. Those SSD do have some way to flush before they
| lose juice though.
|
| This was for enterprise SSD though a few years back.
| marcan_42 wrote:
| That is only for enterprise SSDs. Consumer SSDs do not have
| capacitor banks to do a full DRAM flush. Some have
| capacitor banks to ensure data at rest integrity and some
| use another mechanism for that, but I'm not aware of any
| that will guarantee full cache stability.
| supermatt wrote:
| This F_FULLFSYNC behaviour has been like this on OSX for as long
| as I can remember. It is a hint to ensures that the data in the
| write buffer has been flushed to stable storage - this is
| historically a limitation of fsync that is being accounted for -
| are you 1000% sure it does as you expect on other OSes?
|
| POSIX spec says no:
| https://pubs.opengroup.org/onlinepubs/9699919799/functions/f...
|
| Maybe unrealistic expectation for all OSes to behave like linux.
|
| Maybe linux fsync is more like F_BARRIERFSYNC than F_FULLFSYNC.
| You can retry with those for your benchmarks.
|
| Also note that 3rd party drives are known to ignore F_FULLFSYNC,
| which is why there is an approved list of drives for mac pros.
| This could explain why you are seeing different figures if you
| are supplying F_FULLFSYNC in your benchmarks using those 3rd
| party drives.
| otterley wrote:
| It's important to read the entire document including the notes,
| which informs the reader of a pretty clear intent (emphasis
| mine):
|
| > The fsync() function is intended to force a _physical write_
| of data from the buffer cache, and _to assure that after a
| system crash or other failure that all data up to the time of
| the fsync() call is recorded on the disk._
|
| This seems consistent with user expectations - fsync()
| completion should mean data is fully recorded and therefore
| power-cycle- or crash-safe.
| formerly_proven wrote:
| You are quoting the non-normative informative part. If
| _POSIX_SYNCHRONIZED_IO is not defined, your fsync can
| literally be this and still be compliant:
| int fsync(int) {}
|
| Quick Google search (maybe someone with a MBP can confirm)
| says that macOS doesn't purport to implement SIO.
| otterley wrote:
| That particular implementation seems inconsistent with the
| following requirement:
|
| > The fsync() function shall request that all data for the
| open file descriptor named by fildes is to be transferred
| to the storage device associated with the file described by
| fildes.
|
| If I wrote that requirement in a classroom programming
| assignment and you presented me with that code, you'd get a
| failing grade. Similarly, if I were a product manager and
| put that in the spec and you submitted the above code, it
| wouldn't be merged.
|
| > You are quoting the non-normative informative part
|
| Indeed, I am! It is important. Context matters, both in law
| and in programming. As a legal analogy, if you study
| Supreme Court rulings, you will find that in addition to
| examining the text of legislation or regulatory rules, the
| court frequently looks to legislative history, including
| Congressional findings and statements by regulators and
| legislators in order to figure out how to best interpret
| the law - especially when the text is ambiguous.
| danudey wrote:
| > If I wrote that requirement in a classroom programming
| assignment and you presented me with that code, you'd get
| a failing grade.
|
| It's a good thing operating systems aren't made up
| entirely of classroom programming assignments.
|
| Picture an OS which always runs on fully-synchronized
| storage (perhaps a custom Linux or BSD or QNX kernel). If
| there's no write cache and all writes are synchronous,
| then fsync() doesn't need to do anything at all;
| therefore `int fsync(int) {return 0}` is valid because
| fsync()'s method is implementation-specific.
|
| This allows you to have no software or hardware write
| cache and not implement fsync() and still be POSIX-
| compliant.
|
| > Context matters, both in law and in programming. As a
| legal analogy, if you study Supreme Court rulings, you
| will find that in addition to examining the text of
| legislation or regulatory rules, the court frequently
| looks to legislative history, including Congressional
| findings and statements by regulators and legislators in
| order to figure out how to best interpret the law -
| especially when the text is ambiguous.
|
| The POSIX specification is not a court of law, and the
| context is pretty clear: fsync() should do whatever it
| needs to do to request that pending writes are written to
| the storage device. In some valid cases, that could be
| nothing.
| otterley wrote:
| > Picture an OS which always runs on fully-synchronized
| storage (perhaps a custom Linux or BSD or QNX kernel). If
| there's no write cache and all writes are synchronous,
| then fsync() doesn't need to do anything at all;
| therefore `int fsync(int) {return 0}` is valid because
| fsync()'s method is implementation-specific.
|
| Sure, I'll give you that, in a corner case where all
| writes are synchronized to storage before completing.
| However, most modern computers cache writes for
| performance, and the speed/security tradeoff is the
| context of this discussion. We wouldn't be having this
| debate in the first place if computers and storage
| devices didn't cache writes.
|
| > The POSIX specification is not a court of law
|
| Indeed, it isn't; nor is legislative text (the closest
| analogy in law). Hence the need for interpretation.
|
| > fsync() should do whatever it needs to do to request
| that pending writes are written to the storage device
|
| We are in violent agreement about this :-)
| rbanffy wrote:
| There's also the very likely possibility that the storage
| is lying to the OS, that the data that was accepted and
| which is in the buffer has been written somewhere durable
| while it's actually waiting for an erase to finish or a
| head to get wherever it needs to be. There are disk
| controllers with batteries precisely for those
| situations.
|
| And, if cheating will give better numbers on benchmarks,
| I'm willing to bet money most manufacturers will cheat.
| formerly_proven wrote:
| The wording here is quite subtle. Without SIO, fsync is
| merely a request, returning an error if one occurred. As
| the informative section points out, this means that the
| request may be ignored, which is not an error.
|
| > If _POSIX_SYNCHRONIZED_IO is not defined, the wording
| relies heavily on the conformance document to tell the
| user what can be expected from the system. It is
| explicitly intended that a null implementation is
| permitted.
|
| Compare this to e.g. the wording for write(2):
|
| > The write() function shall attempt to write nbyte bytes
| from the buffer pointed to by buf to the file associated
| with the open file descriptor, fildes. [yadadada]
|
| This actually specifies that an action needs to be
| performed. fsync(2) sans SIO is merely a request form
| that the OS can respond to or not. And because macOS does
| not define SIO, you have to go out and find out what that
| particular implementation is actually doing and the
| answer is: essentially nothing for fsync.
| marcan_42 wrote:
| Yes. fsync() on Linux pushes down to stable storage, not just
| drive cache.
|
| OpenBSD, though, apparently behaves like macOS. I'm not sure I
| like that.
| olliej wrote:
| Linux does that now. It didn't in the past (something like
| 2008), and I recall many people arguing about performance or
| similar at that time :D
| supermatt wrote:
| So its basically implementation specific, and macOS has its
| own way of handling it.
|
| That doesnt make it worse - in fact it permits the
| flexibility you are now struggling with.
|
| edit: downvotes for truth? nice. go read the posix spec then
| come back and remove your downvotes...
| marcan_42 wrote:
| What is worse is their NVMe controller having 50x worse
| flush performance than the competition.
| supermatt wrote:
| The competitions controller may be ignoring the
| F_FULLFSYNC. This is a known issue which is why apple
| have approved vendors for mac pro drives.
| marcan_42 wrote:
| It isn't, because otherwise it would be showing the ~same
| performance with and without sync commands, as I showed
| in the thread. There is a significant performance loss
| for every drive, but Apple's is way worse.
|
| There is no real excuse for a single sector write to take
| ~20ms to flush to NAND, all the while the NAND controller
| is generating some 10MB/s of DRAM traffic. This is a dumb
| firmware design issue.
| supermatt wrote:
| It may be interpreting it differently. You arent
| comparing apples to apples, quite literally.
|
| Why not compare macOS and linux on approved x86 mac
| hardware. i.e. fusion drive or whatever.
|
| Also, as suggested - try F_BARRIERFSYNC, which flushes
| anything before the barrier (used for WAL IIRC).
| throwawaylinux wrote:
| It seems to be pretty apples to apples, they're running
| the same benchmark using equivalent data storage APIs on
| both systems. What are you thinking might be different?
| The Linux+WD drive isn't making the data durable? Or that
| OSX does something stupid which could be the cause of the
| slowdown rather than the drive? Both seem implausible.
| marcan_42 wrote:
| This affects T2 Macs too, which use the same NVMe
| controller design as M1 Macs.
|
| We've looked at NVMe command traces from running macOS
| under a transparent hypervisor. We've issued NVMe
| commands outside of Linux from a bare-metal environment.
| The 20ms flush penalty is there for Apple's NVMe
| implementation. It's not some OS thing. And other drives
| don't have it. And I checked and Apple's NVMe controller
| is doing 10MB/s of DRAM memory traffic when issued
| flushes, for some reason (yes, we can get those stats).
| And we know macOS does not properly flush with just
| fsync() because it actively loses data on hard shutdowns.
| We've been fighting this issue for a while now, it's just
| that it only just hit us yesterday/today that there is no
| magic in macOS - it just doesn't flush, and doesn't
| guarantee data persistence, on fsync().
| supermatt wrote:
| Ive just been scanning through linux kernel code (inc
| ext4). Are you sure that its not issuing a PREFLUSH? What
| are your barrier options on the mount? I think you will
| find these are going to be more like F_BARRIERFSYNC.
|
| I couldnt find much info about it - but the official docs
| are here: https://kernel.org/doc/html/v5.17-rc3/block/wri
| teback_cache_...
| marcan_42 wrote:
| Those are Linux concepts. What you're looking for is the
| actual NVMe commands. There's two things: FLUSH (which
| flushes the whole cache), and a WRITE with the FUA bit
| set (which basically turns that write into write-through,
| but does not guarantee anything about other commands).
| The latter isn't very useful for most cases, since you
| usually want at least barrier semantics if not a full
| flush for previously completed writes. And that leaves
| you with FLUSH. Which is the one that takes 20ms on these
| drives.
| supermatt wrote:
| > Those are Linux concepts. What you're looking for is
| the actual NVMe commands.
|
| Im not sure what commands are being sent to the NVMe
| drive. But what you are describing as a flush would be
| F_BARRIERFSYNC - NOT the F_FULLFSYNC which youve been
| benchmarking.
| marcan_42 wrote:
| Sigh, no. A barrier is not a full flush. A barrier does
| not guarantee data persistence, it guarantees write
| ordering. A barrier will not make sure the data hits disk
| and is not lost on power failure. It just makes sure that
| subsequent data won't show up and not the prior data, on
| power failure. NVMe doesn't even _have_ a concept of
| barriers in this sense. An OS-level barrier can be faster
| than a full sync only because it doesn 't need to wait
| for the FLUSH to actually complete, it can just maintain
| a concept of ordering within the OS and make sure it is
| maintained with interleaved FLUSH calls.
|
| I don't know why you keep pressing on this issue. macOS
| has the same performance with F_FULLFSYNC as Linux does
| with fsync(). Why would they be different things? We're
| getting the same numbers. This entire thing started
| because fsync() on these Macs on _Linux_ was dog slow and
| we couldn 't figure out why macOS was fast. Then we found
| F_FULLFSYNC which has the same semantics as fsync() on
| Linux. And now both OSes perform equally slowly on this
| hardware. They're obviously doing the same thing. And the
| same thing on Linux on non-Apple SSDs is faster. I'm sure
| I could install macOS on this x86 iMac again and show you
| how F_FULLFSYNC on macOS also gives better performance on
| this WD drive than on the M1, but honestly, I don't have
| the time for that, the isssue has been thoroughly proved
| already.
|
| Actually, I have a better one that won't waste as much of
| my time.
|
| _Plugs in a shitty USB3 flash drive into the M1_.
|
| 224 IOPS with F_FULLFSYNC. On a shitty flash drive. 58
| IOPS with F_FULLFSYNC. On internal NVMe.
|
| Both FAT32.
|
| Are you convinced there's a problem yet?
|
| (I'm pretty sure the USB flash drive has no write cache,
| so of course it is equally fast/slow with just fsync(),
| but my point still stands - committing writes to
| persistent storage is slower on this NVMe controller than
| on a random USB drive)
| supermatt wrote:
| OK - thanks for humouring me marcan. Sorry to waste your
| time. Clearly something is not right here.
| jcranberry wrote:
| Thank you, you've made this very clear for me.
| rsync wrote:
| I downvoted you because you complained about your
| downvotes.
| dathinab wrote:
| Probably more like downvoted because missing the point.
|
| Sure fsync allows that behavior, but also it's so widely
| misunderstood that a lot of programs which should do a
| "full" flush only do a fsync, including Benchmarks. In
| which case they are not comparable and doing so is
| cheating.
|
| But that's not the point!
|
| The point is that with the M1 Macs SSDs the performance
| with fully flushing to disk is abysmal bad.
|
| And as such any application with cares for data integrity
| and does a full flush can expect noticable performance
| degradation.
|
| The fact that Apple neither forces frequent full syncs or
| at least full syncs when a Application is closed doesn't
| make it better.
|
| Though it is also not surprising as it's not the first time
| Apple set things up under the assumption their hardware is
| unfailable.
|
| And maybe for a desktop focused high end designs where most
| devices sold are battery powered that is a reasonable
| design choice.
| ClumsyPilot wrote:
| "And maybe for a desktop focused high end designs where
| most devices sold are battery powered that is a
| reasonable design choice"
|
| Does the battery last forever? Do they never shut down
| from overheating, shut down from being too cold, freeze
| up, they are water and coffee proof?
|
| Talk to anyone that repairs mac about how high-end and
| reliable their designs trully are - they are better than
| bottomn of the barrel craptops, sure, but not
| particularly amazing and have some astounding design
| flaws.
| Ryder123 wrote:
| As the article points out, a lot of those cases can be
| detected with advanced notice (dying battery, and
| overheating - probably even being too cold). In those
| cases the OS makes sure all the caches are flushed.
|
| Spilled drinks are a viable cause for concern, but if
| they do enough damage to cause an unexpected shutdown,
| you've probably got bigger issues than unflushed cache.
| dathinab wrote:
| I think that's misleading.
|
| On many laptops even with water damage you can recover
| your local data fully, not do for Macs (for more reasons
| then just data loss/corruption due to non flushing).
|
| Especially if you are already in a bad situation you
| don't want your OS to make it worse.
| BenjiWiebe wrote:
| How cold is too cold for a computer?
| dathinab wrote:
| Many phones, laptops cameras and similar are only
| guaranteeing functionally by above 0 degree....
|
| Luckily they often operate in lower temperatures too, but
| not seldomly by hoping they don't get cooled that much
| themself (because they are e.g. in your pocket).
| michaelmrose wrote:
| The CPU can't possibly get too cold. See for example
| overclocking performed by cooling the CPU with liquid
| nitrogen. Condensation is a factor as is lost of
| ductility of plastic at low temp making it brittle.
| Expansion and contraction of materials especially when
| different materials expand to different degrees.
| Fatnino wrote:
| I've had my phone shut off on me from being out in the
| Chicago cold for a couple hours. Battery over 50% when I
| brought it back inside and warmed it up.
| ClumsyPilot wrote:
| If i go ousode in winter, the bsttery dies around zero
| degrees. Keep in mind that you laptop could be in a bag
| in sleep mode or idle
| thebigman433 wrote:
| The biggest thing is the battery. The CPU doesnt get too
| cold, but batteries degrade or stop performing when they
| get too cold.
|
| Edit: For actual temperatures, in my experience its when
| the device is in use for a sustained amount of time in
| under 10f weather
| gpm wrote:
| Incidentally CPUs do get too cold, not at a reasonable
| temperature, but sufficiently low temperatures do change
| the characteristics of semi conductors. Not something to
| worry about if you're not using liquid nitrogen (or
| colder).
| rbanffy wrote:
| > The point is that with the M1 Macs SSDs the performance
| with fully flushing to disk is abysmal bad.
|
| How sure are we the drives that flush caches more quickly
| are actually flushing the caches?
| dathinab wrote:
| Good Point.
|
| A simple test can be to see the degree of dataloss you
| can occur with a hard power off.
|
| I think the author did that test for M1 Mac but idk. if
| they did the test with the other laptops.
|
| But then the M1 Mac is slower when flushing then most
| SSDs out there and even some HDDs. I think if most SSDs
| wouldn't flush data at all we would know of that and I
| should have run into problems with the few docent hard
| resets I ran into in the last few years. (And sure there
| are probably some SSDs which cheap out on cache flushing
| in a dangerous way, but most shouldn't as far as I can
| tell).
| [deleted]
| spookthesunset wrote:
| I mean the Apple hardware in question is usually a
| laptop, which has its own very well instrumented battery
| backup. In most cases the hardware knows well in advance
| if the battery is gonna run dry.
|
| And yes the hardware is failable. But the kind if failure
| that would cause the device to completely lose power is
| extremely rare. The OS has many chances to take the hint
| and flush the cache before powering down.
|
| Note: this is pure conjecture.
| monocasa wrote:
| Last time I checked (which is a while at this point, pre SSD)
| nearly all consumer drives and even most enterprise drives
| would lie in response to commands to flush the drive cache.
| Working on a storage appliance at the time, the specifics of
| a major drive manufacturer's secret SCSI vendor page knock to
| actually flush their cache was one of the things on their
| deepest NDAs. Apparently ignoring cache flushing was so
| ubiquitous that any drive manufacturer looking to have
| correct semantics would take a beating in benchmarks and lose
| marketshare. : \
|
| So, as of about 2014, any difference here not being backed by
| per manufacturer secret knocks or NDAed, one-off drive
| firmware was just a magic show, with perhaps Linux at least
| being able to say "hey, at least the kernel tried and it's
| not our fault". The cynic in me thinks that the BSDs
| continuing to define fsync() as only hitting the drive cache
| is to keep a semantically clean pathway for "actually flush"
| for storage appliance vendors to stick on the side of their
| kernels that they can't upstream because of the NDAs. A sort
| of dotted line around missing functionality that is obvious
| 'if you know to look for it'.
|
| It wouldn't surprise me at all if Apple's NVME controller is
| the only drive you can easily put your hands on that actually
| does the correct things on flush, since they're pretty much
| the only ones without the perverse market pressure to
| intentionally not implement it correctly.
|
| Since this is getting updoots: Sort of in defense of the
| drive manufacturers (or at least stating one of the defenses
| I heard), they try to spec out the capacitance on the drive
| so that when the controller gets a power loss NMI, they
| generally have enough time to flush then. That always seemed
| like a stretch for spinning rust (the drive motor itself was
| quite a chonker in the watt/ms range being talked about
| particularly considering seeks are in the 100ms range to
| start with, but also they have pretty big electrolytic caps
| on spinning rust so maybe they can go longer?), but this
| might be less of a white lie for SSDs. If they can stay up
| for 200ms after power loss, I can maybe see them being able
| to flush cache. Gods help those HMB drives though, I don't
| know how you'd guarantee access to the host memory used for
| cache on power loss without a full system approach to what
| power loss looks like.
| garaetjjte wrote:
| Flush with other vendors at least does _something_ as they
| block for some time too, just not as long as Apple.
|
| Apple implementation is weird because actual amount of data
| written doesn't seem to affect flush time.
| monocasa wrote:
| On at least one drive I saw, the flush command was
| instead interpreted as a barrier to commands being
| committed to the log in controller DRAM, which could cut
| into parallelization, and therefore throughput, looking
| like a latency spike but not a flush out of the cache.
| CyberRabbi wrote:
| I like that. Fsync() was designed with the block cache in
| mind. IMO how the underlying hardware handles durability is
| its own business. I think a hack to issue a "full fsync" when
| battery is below some threshold is a good compromise.
| ribit wrote:
| Something that is not quite clear to me yet (I did read the
| discussion below, thank you Hector for indulging us, very
| informative): isn't the end behaviour up to the drive
| controller? That is, how can we be sure that Linux actually
| does push to the storage or is it possible that the
| controller cheats? For example, you mention the USB drive
| test on a Mac -- how can we know that the USB stick
| controller actually does the full flush?
|
| Regardless, I certainly agree that the performance hit seems
| excessive. Hopefully it's just an algorithm, issue and Apple
| can fix this with a software update.
| shellac wrote:
| *BSDs mostly followed this semantic, as I recall. Probably
| inherited from a common ancestor.
| mrjin wrote:
| MacOS was really just FreeBSD with a fancier UI. Not sure
| what is the behavior now, but I'm pretty sure FreeBSD
| behaved almost exactly the same as a power loss rendered my
| system unbootable over 10 years ago.
| xoa wrote:
| > _MacOS was really just FreeBSD with a fancier UI._
|
| I'm sorry but this is incorrect. NeXTSTEP was the primary
| foundation for Mac OS X, and the XNU kernel was derived
| from Mach and IIRC 4.4BSD. FreeBSD source was certainly
| an important sync jumping off point for a number of Unix
| components of the kernel and CLI userland, there was some
| code sharing going on for a while (still?), but large
| components of the kernel and core frameworks were unique
| (for better or worse).
| masklinn wrote:
| > and IIRC 4.4BSD
|
| 4.3, only Rhapsody incorporated elements from 4.4, but
| that was the tail end of nextstep, essentially the
| initial preview of macos (it was released as osx server
| 1.0, then forked to darwin from which the actual OSX 10.0
| would be built, two major pieces missing from rhapody
| were Classic and Carbon, so it really was nextstep with
| an OS9 skin).
| xoa wrote:
| Thanks for the correction, man has it been a long, long
| time. I had the Public Beta and than got on the OS X
| train pretty fast on a good old B&W G3. Even with the
| slowness the multitasking still allowed getting around it
| and having all Unix right there with a big rush to
| initial porting was really interesting, good times. I
| remember calling Apple for help getting Apache compiled
| and got forwarded right out of the regular call system to
| some dev whose name I sadly forget and we worked through
| it.
|
| Everything is a million times more refined and overall
| better now but I do have a bit of nostalgia for the
| community and really getting your hands dirty back then
| while still having a fairly decent fallback. I haven't
| actually needed to mess with kernel stuff since 10.5 or
| so but thinking back makes me wonder about paths not
| taken.
| Maursault wrote:
| > so it [Rhapsody] really was nextstep with an OS9 skin
|
| Sorry to be pedantic, but Rhapsody's user interface is
| modeled after the _Mac OS 8_ "Platinum" design language.
| Though 9 also was modeled on Platinum, Rhapsody's
| interface appears nearly identical to Mac OS 8's except
| for the Workspace Manager which doesn't exist in 8.
| KerrAvon wrote:
| Rhapsody was a fairly ugly and distorted copy of the
| Platinum theme if we're honest.
| mmis1000 wrote:
| At least it is also implemented by windows, which cause apt-get
| in hyperv vm slower
|
| And also unbearable slow for loopback device backed docker
| container in the vm due to double layer of cache. I just add
| eat-my-data happily because you can't save a half finished
| docker image anyway.
| ecf wrote:
| How many hundreds of millions of people use OSX over the years
| and never encountered any problems whatsoever?
|
| This article is a non-issue, people just like to upvote Apple
| bashing.
| ClumsyPilot wrote:
| "never encountered any problems whatsoever?"
|
| And how do you know they didn't, did you do a poll?
|
| How many people had random files dissapear or get corrupted
| or settings get reset and probzbly thought they must have
| done something wrong?
| meibo wrote:
| If you need to run software/servers with any kind of data
| consistency/reliability on OS X this is definitely something
| you should be aware of and will be a footgun if you're used
| to Linux.
|
| Macs in datacentres are becoming increasingly common for CI,
| MDM, etc.
| parkingrift wrote:
| I'd rather solve for redundant power than worry about this.
| It's really only critical if you're running a database. Who
| runs a database on macOS?
| acchow wrote:
| Every single iOS app using Core Data (which runs SQLite
| under the hood)
| tinus_hn wrote:
| The OS itself contains hundreds of databases.
| withinboredom wrote:
| People doing CI? Or MDM?
| parkingrift wrote:
| Not sure how either of those would be critically impacted
| by a 1-2 second data loss in a power failure.
| withinboredom wrote:
| I believe it's at least 5s. Marcan didn't specify how
| long it was, but gave an example of at least 5s. That
| could cause a device to think it's allowed to do
| something via MDM but not actually have a record in the
| database allowing it to do so.
| throwawaylinux wrote:
| OSX defines _POSIX_SYNCHRONIZED_IO though, doesn't it? I don't
| have one at hand but IIRC it did.
|
| At least the OSX man page admits to the detail.
|
| The rationale in the POSIX document for a null implementation
| seems reasonable (or at least plausible), but it does not
| really seem to apply to general OSX systems at all. So even if
| they didn't define _POSIX_SYNCHRONIZED_IO it would be against
| the spirit of the specification.
|
| I'm actually curious why they made fsync do anything at all
| though.
| supermatt wrote:
| > OSX defines _POSIX_SYNCHRONIZED_IO though, doesn't it?
|
| Nope: https://opensource.apple.com/source/Libc/Libc-1439.40.1
| 1/inc...
| throwawaylinux wrote:
| > #define _POSIX_SYNCHRONIZED_IO (-1) /* [SIO] */
| supermatt wrote:
| ok - its "defined" as not supported. Im not sure i
| understand your point.
| throwawaylinux wrote:
| Oh sorry you're right... Too much C, not enough POSIX.
|
| Okay, so OSX is right by the letter of the standard. Not
| by the spirit though, when you look at the rationale for
| allowing the exception.
| supermatt wrote:
| No problem - sorry if i came off harsh, i thought you
| were being pedantic :D
|
| TBH, im not so sure its that different. Scanning through
| the linux docs it seems that this behaviour can be
| configured as part of mount options (e.g. barrier on
| ext4). At least its explicit on macOS (with compliant
| hardware).
| throwawaylinux wrote:
| > No problem - sorry if i came off harsh, i thought you
| were being pedantic :D
|
| No just I did a ctrl+F ctrl+C ctrl+V without thinking
| enough. No need to apologize though, my reply was
| actually flippant I should have been more respectful of
| your (correct) point.
|
| > TBH, im not so sure its that different. Scanning
| through the linux docs it seems that this behaviour can
| be configured as part of mount options (e.g. barrier on
| ext4). At least its explicit on macOS (with compliant
| hardware).
|
| I disagree (unless Linux short-cuts this by default). The
| reason is in the POSIX rationale:
|
| *RATIONALE*
|
| > _The fsync() function is intended to force a physical
| write of data from the buffer cache, and to assure that
| after a system crash or other failure that all data up to
| the time of the fsync() call is recorded on the disk.
| Since the concepts of "buffer cache", "system crash",
| "physical write", and "non-volatile storage" are not
| defined here, the wording has to be more abstract._
|
| The first paragraph gives the intention of the interface.
| It's clearly to persist data.
|
| > _If _POSIX_SYNCHRONIZED_IO is not defined, the wording
| relies heavily on the conformance document to tell the
| user what can be expected from the system. It is
| explicitly intended that a null implementation is
| permitted. This could be valid in the case where the
| system cannot assure non-volatile storage under any
| circumstances or when the system is highly fault-tolerant
| and the functionality is not required. In the middle
| ground between these extremes, fsync() might or might not
| actually cause data to be written where it is safe from a
| power failure. The conformance document should identify
| at least that one configuration exists (and how to obtain
| that configuration) where this can be assured for at
| least some files that the user can select to use for
| critical data. It is not intended that an exhaustive list
| is required, but rather sufficient information is
| provided so that if critical data needs to be saved, the
| user can determine how the system is to be configured to
| allow the data to be written to non-volatile storage._
|
| Now this gives a rationale for why you might not include
| it. And lists three examples of where it _could be valid_
| to water down the intended semantics. The system can not
| support it; the functionality is not required because
| data durability is guaranteed in other ways; the
| functionality is traded off in cases where major risks
| have been reduced.
|
| OSX on a consumer Mac doesn't fit those cases.
|
| Linux with the option is violating POSIX even by the
| letter because presumably mounting the drive with
| -onobarrier does not cause all your applications to be
| recompiled with the property undefined. But it's not that
| unreasonable an option, it's clearly not feasible to have
| two sets of all your software compiled and select one or
| the other depending on whether your UPS is operational or
| not.
| supermatt wrote:
| Oh yeah, I definitely agree with you on this. If anything
| you should be able to pass in flags to reduce resiliency
| - not have the default be that way. Maybe thats how the
| actual SIO spec reads (i havent read it).
| acchow wrote:
| Do you mean on Linux that calling fsync might not _actually_
| flush to the drive?
| cryptonector wrote:
| How are you reading POSIX as "saying no"??
|
| From that page: The fsync() function shall
| request that all data for the open file descriptor named
| by fildes is to be transferred to the storage device
| associated with the file described by fildes. The nature
| of the transfer is implementation-defined. The fsync()
| function shall not return until the system has completed
| that action or until an error is detected.
|
| then: The fsync() function is intended to force
| a physical write of data from the buffer cache, and to
| assure that after a system crash or other failure that
| all data up to the time of the fsync() call is recorded
| on the disk. Since the concepts of "buffer cache",
| "system crash", "physical write", and "non-volatile
| storage" are not defined here, the wording has to be more
| abstract.
|
| The only reason to doubt the clarity of the above is that POSIX
| does not consider crashes and power failures to be in scope. It
| says so right in the quoted text.
|
| Crashes and power failures are just not part of the POSIX
| worldview, so in POSIX there can be no need for sync(2) or
| fsync(2), or fcntl(2) w/ F_FULLFSYNC! Why even bother having
| those system calls? Why even bother having the spec refer to
| the concept at all?
|
| Well, the reality is that _some_ allowance must be made for
| crashes and power failures, and that includes some mechanism
| for flushing caches all the way to persistent storage. POSIX is
| a standard that some real-life operating systems aim to meet,
| but those operating systems have to deal with crashes and power
| failures because those things happen in real life, and because
| their _users_ want the operating systems to handle those events
| as gracefully as possible. Some data loss is always
| inescapable, but data corruption would be very bad, which is
| why filesystems and applications try to do things like write-
| ahead logging and so on.
|
| That is why sync(2), fsync(2), fdatasync(2), and F_FULLFSYNC
| exist. It's why they [well, some of them] existed in _Unix_ ,
| it's why they still exist in Unix derivatives, it's why they
| exist in Unix-alike systems, it's why they exist in Windows and
| other not-remotely-POSIX operating systems, and it's why they
| exist in POSIX.
|
| If they must exist in POSIX, then we should read the quoted and
| linked page, and it is pretty clear: "transferred to the
| storage device" and "intended to force a physical write" can
| only mean... what that says.
|
| It would be fairly outrageous for an operating system to say
| that since crashes and power failures are outside the scope of
| POSIX, the operating system will not provide any way to save
| data persistently other than to shut down!
| mannykannot wrote:
| OP appears to be giving useful information about OSX,
| regardless of what other OSes do.
| simonh wrote:
| The implication (in fact no, it's explicitly stated) is that
| this fsync() behaviour on OSX will be a surprise for
| developers working on cross platform code or coming from
| other OS's and will catch them out.
|
| However if in fact it's quite common for other OS's to
| exhibit the same or similar behaviour (BSD for example does
| this too, which makes sense as OSX has a lot of BSD lineage),
| that argument of least surprise falls a bit flat.
|
| That's not to say this is good behaviour, I think Linux does
| this right, the real issue is the appalling performance for
| flushing writes.
| jorangreef wrote:
| > Also note that 3rd party drives are known to ignore
| F_FULLFSYNC
|
| SQLite, MySQL et al. [1] fall back to `fsync()` if F_FULLFSYNC
| fails, in order to cover this case of 3rd party or external
| drives.
|
| [1]
| https://twitter.com/TigerBeetleDB/status/1422855270716293123
| trasz wrote:
| tl;dr OSX handles fsync() the way Linux used to, by not flushing
| to hardware
| lazerl0rd wrote:
| The funny thing here is that battery-backed enterprise systems
| are worse off in that manner, because you're much more likely to
| notice a dying battery that your entire device relies on than the
| little battery pack hooked up to your RAID array.
|
| Sure, you could write a program that periodically checks the
| battery rate (you'd have to poll since there's no ACPI
| notification like with a "device battery") and sends an email to
| the admin or something. However that's a tool that doesn't
| "exist" (as in, there isn't notable program that does so) which
| possibly hints that this isn't something system admins often do.
|
| The above also requires there to be an interface available from
| userland, not only in the management firmware or BIOS/UEFI. That
| exists for HP, but I'm not sure _all_ other OEMs do so.
| claudex wrote:
| Enterprise system have monitoring through the BIOS which will
| send an email, expose the status via SNMP and other method of
| monitoring (same as having a faulty fan).
| lazerl0rd wrote:
| Correct me if I'm wrong, but I wouldn't call the management
| engine (eg. HP iLO) the BIOS. Whilst those may support such
| warnings:
|
| 1) Not everyone wants to use iLO or whatever equivalent
| another OEM provides.
|
| 2) Whilst such systems do support sending warnings about
| system components via email, dashboards, etc. that doesn't
| mean they'll necessarily warn about a RAID controller's
| battery being depleted. If I remember correctly, iLO4
| doesn't.
|
| 3) What about RAID cards like the P420 (*not* the P420i) that
| either aren't hooked up to a management engine or are from an
| entirely separate OEM?
| selectodude wrote:
| >1) Not everyone wants to use iLO or whatever equivalent
| another OEM provides.
|
| Then you aren't an enterprise because they're absurdly
| useful for managing dozens/hundreds/thousands of systems.
|
| >3) What about RAID cards like the P420 ( _not_ the P420i)
| that either aren 't hooked up to a management engine or are
| from an entirely separate OEM?
|
| There's a reason enterprises standardize on a common
| infrastructure from an OEM that supports everything in the
| box even though you could go on Newegg and build your own
| systems for thousands of dollars less.
| maccolgan wrote:
| External batteries can often be connected to via serial (most
| common), via USB or via IP, so that is definitely one.
| lazerl0rd wrote:
| That's the first time I've heard of batteries [for RAID
| controllers] having an entirely separate port than that which
| hooks them up to the controller. Is this a "there are some of
| X" or have I just been out of the loop?
| londons_explore wrote:
| To emulate a flushing SSD, the signal really needs to go
| directly to the SSD firmware so it can decide which is the last
| OS write it can accept while still having enough power to
| persist all write and flush requests it has already accepted.
|
| Getting all that right sounds so hard it is probably better to
| just have enterprise SSD's have a built in supercap to give 5
| seconds or so of power to do all the necessary flushing, and
| for laptop/desktop grade SSD's they only need to offer barriers
| for data consistency. Laptop and desktop users don't care if
| they lose the last 1 second of data before a crash as long as
| what is on the drive is self consistent.
| lazerl0rd wrote:
| I should've been a little clearer; by "enterprise systems" I
| was referring to RAID controllers and the like. Though yes, I
| believe enterprise SSDs/NVMes likely have a capacitor or, as
| one friend put it, an "overkill battery" to use for flushing
| data.
|
| To be fair though, I sidetracked from the discussion at hand.
| The issue Marcan described was regarding the OS -> Disk
| rather than a "power loss situation". The latter does play in
| with the former, but solving the latter doesn't necessarily
| solve the former.
| steeve wrote:
| As the article mentions, on laptops, this is pretty clever. On
| desktops though...
|
| Perhaps real macs should be equipped with internal batteries to
| flush to disk in the case of power loss?
|
| I think I heard some enterprise
| motherboards/controllers/computers did just that, given the
| upside in normal operation.
| FooBarWidget wrote:
| Even on laptops I feel uncomfortable. My macOS freezes or
| kernel panics on me from time to time.
| marcan_42 wrote:
| I believe the NVMe driver has a kernel panic hook; I would
| hope it is used to issue a flush.
|
| OTOH, if you have watchdog timeouts (I've seen this from bad
| drivers), those would certainly not give the kernel a chance
| to do that.
| throwawaylinux wrote:
| Oh geez, _deliberately_ issuing commands to storage after
| your kernel panics? It just keeps getting better :(
| ladyanita22 wrote:
| What would you implement in Asahi? Would you follow Apple's
| approach and defer flushes, implementing a kernel panic
| hook and having some kind of F_FULLFSYNC or just keep
| Linux' current implementation?
| marcan_42 wrote:
| We're probably going to have a knob to defer flushes (but
| still do them, unlike Apple, after a max timeout) that
| will be on by default on laptops, and make sure panics
| flush the cache if we can. Also apparently we need to do
| something for the power button too, as I just tested how
| macOS handles that. There is a warning before the system
| shuts down but we need to listen to it. Same with
| critical battery states.
| ladyanita22 wrote:
| Then I misunderstood. Do you mean that Apple doesn't
| implement ANY timeout? So they only flush when the cache
| is full or when a shutdown routine has started?
| marcan_42 wrote:
| They flush the cache when something requests the cache be
| flushed; I don't know if there is a timeout, because
| presumably it's not difficult for some random process to
| issue a FULLFSYNC and flush everything prior as a side-
| effect (the flush is global). But I've seen at least 5-10
| seconds of data loss from drive cache loss on the Mac
| Mini, so if they do do deferred flushes the timeout is
| longer than that.
| ladyanita22 wrote:
| WTF, that is worse than I thought then. That's the
| dirtiest hack I've read, it's of very low quality for a
| company like Apple. That I'd expect for a OnePlus device,
| not for a full fledged Macbook.
| Dagonfly wrote:
| When do off-the-shelf NVMe controllers flush their
| internal DRAM buffer? I presume that happened after a
| timeout, even if the OS does not issue a NVMe flush
| command.
|
| Does Apple implement the NVMe spec on their controller,
| i.e. do they indicate "Volatile Write Cache"?
| [deleted]
| dzhiurgis wrote:
| Does disk gets flushed in case of kernel panic?
| coldtea wrote:
| > _Perhaps real macs should be equipped with internal batteries
| to flush to disk in the case of power loss?_
|
| Or just add a UPS?
| throw3838 wrote:
| How exactly this is clever? Maybe on some toy, not on
| workstation!
| marcan_42 wrote:
| These machines are actually low-power enough that you could
| implement a last-gasp flush mechanism. The Mac Mini already
| survives 1-2 seconds without AC power (at least if idle). You
| could plausibly detect AC power being yanked and immediately
| power down all downstream USB/TB3 devices and the display (on
| iMacs), freeze all CPUs into idle, and have plenty enough
| reservoir cap to let NVMe issue a flush.
|
| But they aren't doing that. I tested it on the Mac Mini. It
| loses several seconds of fsync()ed data on hard shutdown.
|
| This does require a last-gasp indication from the PSU to the
| rest of the system, so if they don't have that, it's not
| something they could add in a firmware update.
| fauigerzigerk wrote:
| _> But they aren't doing that. I tested it on the Mac Mini.
| It loses several seconds of fsync()ed data on hard shutdown._
|
| That's unfortunate. My Mac Mini crashes every other night
| during sleep. I guess I'm going to have to shut it down to
| avoid any data corruption.
| marcan_42 wrote:
| It should be flushing the drive cache on sleep. This is
| mostly an issue for sudden AC power loss.
| fauigerzigerk wrote:
| Ah, thanks! That's good to know.
| sgt wrote:
| Why does it crash? Mac Minis are some of the most reliable
| machines on the market, in my experience. Maybe a faulty
| unit?
| fauigerzigerk wrote:
| It has started crashing the night after I upgraded to
| macOS 12.2.0. The latest update to 12.2.1 hasn't fixed
| it. I'm pretty sure it's not hardware related as I had no
| issues before the OS upgrade.
|
| Edit: Here's the first line of the crash log (which I'm
| sending to Apple every time): panic(cpu 3
| caller 0xfffffe0023be8be0): [data.kalloc.16]:
| element modified after free (off:0,
| val:0x0000000000000030, sz:16, ptr:0xfffffe2fffc9bb00)
|
| Looks like a use after free bug.
| yborg wrote:
| This is why I'm still on Catalina. I used to be in the
| "trail by one point release" mode on macOS, now I'm in
| the "trail by 2 major releases" camp.
| david_allison wrote:
| I get regular crashes restoring from sleep on my 2014 Mac
| Mini (running Monterey)
| salamandersauce wrote:
| Shitty software? My 2018 Mac Mini would crash every
| single time going to sleep on the last version of Mojave.
| I'm not alone in this as there's huge threads on
| MacRumors and Apple's support forum about it. Apple's
| "fix" was to just update to Catalina which indeed fixes
| it but doesn't really help if you want to run 32 bit
| software. Wouldn't surprise me if they did something
| similar again.
| rubatuga wrote:
| I mean the ATX standard has this signal built in, so Apple
| could just copy it:
|
| https://en.wikipedia.org/wiki/Power_good_signal
| dale_glass wrote:
| > The ATX specification requires that the power-good signal
| ("PWR_OK") go high no sooner than 100 ms after the power
| rails have stabilized, and remain high for 16 ms after loss
| of AC power, and fall (to less than 0.4 V) at least 1 ms
| before the power rails fall out of specification (to 95% of
| their nominal value).
|
| I don't think that quite works for the purpose. What you'd
| want is a second signal that goes low as soon as possible
| after loss of AC power.
|
| My reading here is that PWR_OK going low is an indication
| that the PSU has stopped providing good power, and the CPU
| must shut down immediately, or it might miscompute
| something due to low voltage. At this point you absolutely
| don't want to do any last-minute writing, you'd be risking
| corruption.
|
| What you need here is an early warning signal that you can
| react to while the PSU is still coasting on the internal
| capacitors.
| fps-hero wrote:
| Power OK signals are used to prevent latch ups in silicon
| due to power glitches. The signals will route to power
| management ICs to ensure a full reset with proper
| bringing up of the power rails on any power glitch.
| jaidan wrote:
| 16ms is just longer than one AC cycle at 60Hz and less
| than one AC cycle at 50Hz.
|
| I would has a guess that 16ms is the physical limit for
| most consumer hardware (and maybe commercial computing)
| to detect mains loss.
|
| Of course there is industrial hardware that can detect
| quicker than this but it would add a LOT of cost for
| arguably little gain, or something that could be solved
| in another manner.
| dale_glass wrote:
| > I would has a guess that 16ms is the physical limit for
| most consumer hardware (and maybe commercial computing)
| to detect mains loss.
|
| Doubtful. 16ms is an awfully long time these days.
| There's no reason why you couldn't detect power loss much
| sooner, given a good input signal. The concept also gets
| used quite often, in the form of SSRs with zero crossing
| detection. Those are used for dimmers.
|
| The reason is likely related to the awful waveforms
| produced by some UPSes and inverters:
|
| https://www.christidis.info/images/blog/scope_20.png
|
| Unlike a nice sine wave, those spend a good while
| hovering near zero volts, so the PSU has to be able to
| tolerate that. Detecting loss of power sooner in this
| case isn't a question of cost, it's a question of that
| you don't have a good signal to do the detection on to
| start with.
| mschuster91 wrote:
| > Unlike a nice sine wave, those spend a good while
| hovering near zero volts, so the PSU has to be able to
| tolerate that.
|
| That wave chart was _atrocious_. I wonder if the extra
| load on the DC-side caps leads to them having lower life
| expectancy than the ones in a PSU attached to a proper
| power grid?
| jb1991 wrote:
| > Of course, in normal usage, this is basically never an issue on
| laptops; given the right software hooks, they should never run
| out of power before the OS has a chance to issue a disk flush
| command
| simonh wrote:
| But since I only use Mac desktops.....
| jb1991 wrote:
| I guess a UPS powerbackup would be useful. Laptops basically
| have built-in UPS which is perhaps why Apple has gone in that
| direction. I wonder if their high-end desktops with Apple
| Silicon will do something different there.
| freemint wrote:
| An UPS won't help though unless you program your Mac to
| check the UPS status and flush accordingly.
| jb1991 wrote:
| I'm not entirely sure how a UPS works with a computer,
| nor exactly how this flushing works, but doesn't a UPS
| run for awhile in the event of a power disruption,
| certainly more than the several seconds delay with this
| flushing?
| acdha wrote:
| That's been built in to macOS since the 2000s - there's a
| standard USB protocol which most UPS vendors support.
| kevincox wrote:
| Unless of course your kernel panics. (Although there may still
| be a best-effort flush here, so it probably depends on how
| exactly it dies)
| belter wrote:
| For some here comparing and contrasting both documentation
| ambiguity and fsync behaviour between OSX and Linux, these two
| are probably useful:
|
| "Linux Fsync Issue for Buffered IO and Its Preliminary Fix for
| PostgreSQL"
|
| https://news.ycombinator.com/item?id=19238121
|
| "PostgreSQL used fsync incorrectly for 20 years (2019) [video]
| (fosdem.org)"
|
| https://news.ycombinator.com/item?id=30131165
| vgel wrote:
| What confuses me about this is why are they so slow with
| F_FULLSYNC? Since that's the equivalent of what non-Apple NVMEs
| do under, say, Linux, and they manage to be much faster.
| marcan_42 wrote:
| The OS does not matter; it's strictly about the drive. macOS on
| a non-Apple SSD should be equally fast with F_FULLSYNC.
|
| Indeed, I would very much like to know what on earth the ANS
| firmware is doing on flushes to make them so hideously slow. We
| do have the firmware blobs (for both the NVMe/ANS side and the
| downstream S5C NAND device controllers), so if someone is bored
| enough they could try to reverse engineer it... it also seems
| there's a bunch of debug mode options, so maybe we can even get
| some logs at some point.
| supermatt wrote:
| Drives are known to ignore that hint... Thats why you should
| use vendor approved hardware if such things matter to you.
| yalogin wrote:
| > fsync() will both flush writes to the drive, and ask it to
| flush its write cache to stable storage.
|
| Can someone explain what "flushing write cache to stable storage"
| means? Isn't that the same as "writes to the drive". I am
| obviously not well versed in this area. Also what is stable
| storage? Never heard that term before.
| [deleted]
| fred123 wrote:
| SSDs and other storage drives have two layers (or more). The
| last layer is stable storage (= when you disconnect power no
| data is lost or corrupted). When you write to such a device
| your writes are first made in an earlier layer that is more
| like your computer's main memory than actual storage (when you
| lose power your data is gone or corrupted). Only after time or
| when the cache is full an actual persistent write is made.
| serguzest wrote:
| maybe apple Nvmes have some sort of short time battery/capacitor
| gives it a time to finish the write once the power lost?
| shellac wrote:
| Variants of the FSYNC story have been going on for decades now.
| The framing varies, but typically somebody is benchmarking IO
| (often in the context of database benchmarking) and discovers a
| curious variance by OS.
|
| On NVMes I wonder whether this really matters, but it's a serious
| issue on spinning disks: do you really need to flush everything
| to the disk (and interrupt more efficient access patterns)?
| marcan_42 wrote:
| On this NVMe, flushing is slower than on some spinning disks,
| so it apparently matters.
| shellac wrote:
| Yes, I would have skipped the fsync thing, which carries a
| lot of baggage, and concentrate on this.
|
| Btw, are you sure those spinning disks are actually flushing
| to rust? Caches all the way down... ;-)
| marcan_42 wrote:
| I mean, typical seek time on rust is O(10ms) and these
| controllers are spending 20ms flushing a few sectors.
| Obviously rust would do worse if you have the cache full of
| random writes, though. The problem here is the huge base
| cost.
| tlamponi wrote:
| > On NVMes I wonder whether this really matters, but it's a
| serious issue on spinning disks: do you really need to flush
| everything to the disk (and interrupt more efficient access
| patterns)?
|
| That depends on the drive having power loss protection, which
| comes most of the time in the form of a capacitor that powers
| the drive long enough to guarantee that its buffers are flushed
| to persistent storage.
|
| Consumer SSDs often do not have that, so flushing is really
| important there, at least if your data, or no FS corruption is
| important to you.
|
| Enterprise SSDs almost always have power loss protection, so
| there it isn't required for consistency's sake, albeit in-
| flight data that didn't hit the block device yet is naturally
| not protected by that, most FS handle that fine by default
| though.
|
| Note that Linux, for example, does by default a periodic flush
| every 30s independent of caching/flush settings, so that's
| normally the upper limit you'd lose, depending on the workload
| it can be still a relatively long time frame.
|
| https://sysctl-explorer.net/vm/dirty_expire_centisecs/
| marcan_42 wrote:
| Those VM tunables are about dirty OS cache, not dirty drive
| cache. If you fsync() a file on Linux it will be pushed to
| the drive and (if the drive does not have battery/capacitor-
| backed cache) flushed from drive cache to stable storage. If
| you don't fsync() then AIUI all bets are off, but in practice
| the drive will eventually get around to flushing your data
| anyway. The OS has one timeout for cache flushes and the
| drive should have another one, one would hope.
| shivak wrote:
| As you noted, Apple's fsync() behavior is defensible if PLP
| is assumed. Committing through the PLP cache isn't how
| these drives are meant to operate - hence the poor behavior
| of F_FULLSYNC.
|
| But this isn't specific to Macs and iDevices. Some non-PLP
| drives also struggle with sync writes on FreeBSD [1]. Most
| enterprises running RDBMS mandate PLP for both performance
| and reliability. I understand why this is frustrating for
| porting Linux, but Apple is allowed to make strong
| assumptions about how their hardware interoperates.
|
| [1] https://www.truenas.com/community/threads/slog-and-
| power-los...
| DooMMasteR wrote:
| On my Linux (at least to my SATA drive) fsync() issues a
| "FLUSH_CACHE" to the drive too.
| rossmohax wrote:
| Docs [1] suggests that even F_FULLFSYNC might not be enough.
| Quote:
|
| > Note that F_FULLFSYNC represents a best-effort guarantee that
| iOS writes data to the disk, but data can still be lost in the
| case of sudden power loss.
|
| [1] https://developer.apple.com/documentation/xcode/reducing-
| dis...
| ayende wrote:
| When building databases, we care about durability, so database
| authors are usually well aware that you _have_ to use
| `F_FULLSYNC` for safety. The fact that `F_FULLSYNC` isn't safe
| means that you cannot write a transactional database on Mac, it
| is also a surprise to me.
|
| Note that the man page for `F_FULLSYNC` itself doesn't mention
| that it is not reliable:
| https://developer.apple.com/library/archive/documentation/Sy...
|
| Having a separate syscall is annoying, but workable. Having a
| scenario where we call flush and cannot ensure that this is the
| case is _BAD_. Note that handling flush failures is expected,
| but all databases require that flushing successfully will make
| the data durable.
|
| Without that, there are no way to ensure durable writes and you
| might get data loss or data corruption.
| rbanffy wrote:
| > Without that, there are no way to ensure durable writes and
| you might get data loss or data corruption.
|
| The best the OS can do is to trust the device that the data
| was, indeed, written to durable storage. Unfortunately, many
| devices lie about that. If you do a `F_FULLSYNC`, you can say
| you did your best, but the data is out of your hands now.
| geocar wrote:
| > When building databases, we care about durability, so
| database authors are usually well aware that you _have_ to
| use `F_FULLSYNC` for safety. The fact that `F_FULLSYNC` isn't
| safe means that you cannot write a transactional database on
| Mac, it is also a surprise to me.
|
| > Without that, there are no way to ensure durable writes and
| you might get data loss or data corruption.
|
| No, not without that. Even with that, you can't have durable
| writes; Not on a mac, or linux or anywhere else, if you are
| worried about fsync()/fcntl+F_FULLSYNC because they do
| nothing to protect against hardware failure: The only thing
| that does is shipping the data someplace else (and depending
| on the criticality of the data, possibly quite far).
|
| As soon as you have _two_ database servers, you 're in a much
| better shape, and many databases like to try and use fsync()
| as a barrier to that replication, but this is a _waste of
| time_ because your chances of a single hardware failure
| _remain the same_ -- the only thing that really matters is
| that 1 /2 is smaller than 1/1.
|
| So okay, maybe you're not trying to protect against _all_
| hardware failure, or even just the flash failure (it will
| fail when it fails! better to have two nvme boards than one!)
| but maybe just _some_ failure -- like a power failure, but
| guess what: We just need to put a big beefy capacitor on the
| board, or a battery someplace to protect against that. We
| _don 't_ need to write the flash blocks and read them back
| before returning from fsync() to get reliability because
| _that 's not the failure you're trying to protect against_.
|
| What does fsync() actually protect against? Well, sometimes
| that battery fails, or that capacitor blows: The hardware
| needed to write data to a spinning platter of metal and rust
| used to have a lot more failure points than today's solid
| state, and in _those_ days, maybe it made some sense to add a
| system call instead of adding more hardware, but _modern_
| systems aren 't like that: It is almost always cheaper in the
| long run to just buy two than to try and squeeze a little
| more edge out of one, but maybe, if there's a case where
| fsync() helps today, it's a situation where that isn't true
| -- but even that is a long way from you need fsync() to have
| durable writes and avoid data loss or corruption.
| throwawaylinux wrote:
| > No, not without that. Even with that, you can't have
| durable writes; Not on a mac, or linux or anywhere else, if
| you are worried about fsync()/fcntl+F_FULLSYNC because they
| do nothing to protect against hardware failure: The only
| thing that does is shipping the data someplace else (and
| depending on the criticality of the data, possibly quite
| far).
|
| "The sun might explode so nothing guarantees integrity",
| come on, get real. This is pointless nitpicking.
|
| Of course fsync ensures durable writes on systems like
| Linux with drives that honor FUA. The reliability of the
| device and stack in question is implied in this and anybody
| who talks about data integrity understands that. This is
| how you can calculate and manage error rates of your
| system.
| geocar wrote:
| > "The sun might explode so nothing guarantees
| integrity", come on, get real. This is pointless
| nitpicking.
|
| I think most people understand that there is a huge
| difference between the sun exploding and a single
| hardware failure.
|
| If you _really_ don 't understand that, I have no idea
| what to say.
|
| > Of course fsync ensures durable writes on systems like
| Linux with drives that honor FUA
|
| No it does not. The drive can still fail after you
| write() and nobody will care how often you called
| fsync(). The _only_ thing that can help is writing it
| more than once.
| throwawaylinux wrote:
| What is the difference in the context of your comment?
| The likelihood of the risk, and nothing else. So what is
| the exact magic amount of risk that makes one thing
| durable and another not, and who made you the arbiter of
| this?
|
| > No it does not. The drive can still fail after you
| write() and nobody will care how often you called
| fsync(). The only thing that can help is writing it more
| than once.
|
| It does to anybody who actually understands these
| definitions. It is durable according to the design (i.e.,
| UBER rates) of your system. That's what it means, that's
| always what it meant. If you _really_ don 't understand
| that, I have no idea what to say.
|
| > The only thing that can help is writing it more than
| once.
|
| This just shows a fundamental misunderstanding. You
| achieve a desired uncorrected error rate by looking at
| the risks and designing parts and redundancy and error
| correction appropriately. The reliability of one
| drive/system might be greater than two less reliable
| ones, so "writing it more than once" is not only not the
| only thing that can help, it doesn't necessarily achieve
| the required durability.
| geocar wrote:
| > What is the difference in the context of your comment?
| The likelihood of the risk, and nothing else. So what is
| the exact magic amount of risk that makes one thing
| durable and another not, and who made you the arbiter of
| this?
|
| What's the difference between the sun exploding and a
| single machine failing?
|
| I have no idea how to answer that. Maybe it's because
| many people have seen a single machine fail, but nobody
| has seen the sun explode? I guess I've never had a need
| to give it more thought than that.
|
| > It does to anybody who actually understands these
| definitions. It is durable according to the design (i.e.,
| UBER rates) of your system.
|
| You are wrong about that: Nobody cares if something is
| "designed to be durable according to the definition in
| the design". That's just more weasel words. They care
| what are the risks, how you actually protect against
| them, and what it costs to do. That's it.
| throwawaylinux wrote:
| I was asking about the _context of the conversation_. And
| I answered it for you. It 's the likelihood of the risk.
| Two computers in two different locations can and do fail.
|
| > You are wrong about that: Nobody cares if something is
| "designed to be durable according to the definition in
| the design".
|
| No I'm not, that's what the word means and that's how
| it's used. That's how it's defined in operating systems,
| that's how it's defined by disk manufacturers, that's how
| it's used by people who write databases.
|
| > That's just more weasel words.
|
| No it's not, its the only sane definition because all
| hardware and software is different, and so is everybody's
| appetite for risk and cost. And you don't know what any
| of those things are in any situation.
|
| > They care what are the risks, how you actually protect
| against them, and what it costs to do. That's it.
|
| You seem to be arguing against yourself here. Lots of
| people (e.g., personal users) store a lot of their data
| on a single device for significant periods of time,
| because that's reasonably durable for their use.
| akrymski wrote:
| There is a point at which a redundant array of
| inexpensive and unreliable replicas is more durable than
| a single drive. Even N in-memory databases spread across
| the world is more durable than a single one with fsync.
|
| Unfortunately few databases besides maybe blockchains
| have been engineered with that in mind.
| throwawaylinux wrote:
| > There is a point at which a redundant array of
| inexpensive and unreliable replicas is more durable than
| a single drive. Even N in-memory databases spread across
| the world is more durable than a single one with fsync.
|
| Unless a failure mode you are concerned about include
| being cut off from the internet, or your system isn't
| network connected in the first place, in which case maybe
| not eh?
|
| Anyway surely the point is clear. "Durable" doesn't mean
| "durable according to the whims of some anonymous denizen
| of the other side of the internet who is imagining a
| scenario which is completely irrelevant to what I'm
| actually doing with my data".
|
| It means that the data is flushed to what your system
| considers to be durable storage.
|
| Also hardware failures and software bugs can exist. You
| can talk about durable storage without being some kind of
| cosmic-ray-denier or anti-backup cultist.
| kikimora wrote:
| This is not about hardware failure but OS crashes and
| bugs that much more frequent.
| ClumsyPilot wrote:
| "but guess what: We just need to put a big beefy capacitor
| on the board, or a battery someplace to protect against
| that. We don't need to write the flash blocks and read them
| back before returning from fsync() to get reliability"
|
| I believe drives that do have capacitors are aware of it
| and return immediately from fsync() without writing to
| flash. Thats the point of this API
|
| Since neither Macs nor any other laptops have SSDs with
| capacitors, this point is kind of moot.
| IshKebab wrote:
| > The fact that `F_FULLSYNC` isn't safe means that you cannot
| write a transactional database on Mac, it is also a surprise
| to me.
|
| Yeah you can definitely write a transactional database
| without having to rely on knowing you've flushed data to
| disk. Not only can you, but you surely _have to_ otherwise
| you risk data corruption e.g. when there 's a power-cut mid-
| write.
| dboreham wrote:
| The whole point of transactional flush to disk is that you
| get confirmation that data is now safe from power loss. You
| don't get any guarantee because you _called_ flush. The
| guarantee comes from flush returning.
| stefan_ wrote:
| I checked a few and they seem to do F_FULLFSYNC (sic), except
| MySQL, they deleted it to make it run faster:
|
| https://github.com/mysql/mysql-
| server/commit/3cb16e9c3879d17...
| nickpeterson wrote:
| Oh MySQL, I'm a world turned upside down you are my North
| Star.
| rat9988 wrote:
| "the possible durability gain is slim to none. This also
| makes OS X behave similar to other platforms."
|
| You didn't report the full reasoning.
| [deleted]
| midrus wrote:
| I've been using Macs (both desktop and laptops) since I have
| memory. I've had the M1 since launch day, and I use it all day,
| both for work and personal use.
|
| Why this never happened to me? Why I don't know anyone which had
| this problem? Why nobody is complaining as it happened with the
| previous gen keyboards?
|
| I think we might be missing something in this analysis. I don't
| think Apple engineers are idiots.
| marcan_42 wrote:
| Most people don't unplug their Mac Mini in the middle of
| working, and most users who do lose data after that happens
| would just think it's normal and not realize there is an
| underlying problem and modern OSes aren't supposed to do that.
|
| I've seen APFS filesystems eat themselves in production (and
| had to do data recovery), twice. Apple don't have a perfect
| data integrity track record.
| wolf550e wrote:
| On laptop, you would get data loss / corruption on sudden power
| loss. This is rare. With "flush to storage device's RAM", even
| a kernel panic would not lose data if you let the storage
| device flush to flash without power loss.
| perryizgr8 wrote:
| For their problem, they can easily solve it by replacing the SSD.
| Then fsync will be normal speed.
| jorangreef wrote:
| Fantastic thread.
|
| The history is also interesting. It's not that "macOS cheats",
| but that it sincerely inherited the status quo of many years,
| then tried to go further by adding F_FULLFSYNC. However, Linux
| since got better, leaving macOS stuck in the past and everybody
| surprised. It's a big problem.
|
| Here's Dominic Giampaolo from Apple discussing this back in 2005,
| before Linux fixed fsync() to flush past the disk cache:
| https://lists.apple.com/archives/darwin-dev/2005/Feb/msg0008...
|
| And here's TigerBeetle's Twitter thread with more of the history
| and how projects like LevelDB, SQLite and various language std
| libs were also affected:
| https://twitter.com/TigerBeetleDB/status/1422854779009654785
| gok wrote:
| Amusingly XNU adopted this behavior because it's what Linux did
| in the early 2000s and people complained that fsync was too slow
| without it.
| borodi wrote:
| I wonder if you hit the drive hard enough, so that the cache gets
| filled, does the performance degrade by that same magnitude?
| BirAdam wrote:
| In my use. Yes. I didn't realize this was the reason until I
| saw this thread, and now I've tested it. Luckily, I don't do
| massive data transfers nor do I do any large data work. When I
| got my M1 Mac Mini, however, I did and had immediate buyer's
| remorse. I thought that I/O must be terrible on this thing, and
| I felt cheated. After the initial stand-up, I wasn't so angry.
| For most tasks, it's faster than my old TR4 1950X.
| rossmohax wrote:
| In Apple defense, sloppy fsync behaviour is clearly documented:
| https://developer.apple.com/library/archive/documentation/Sy...
| lbriner wrote:
| That's not defence. It fails the principle of least-surprise.
| If everyone's experience is that fsync is flushing then why
| would somebody think to look up the docs for Mac in case they
| do it differently?
| olliej wrote:
| Linux only stopped doing the clearly wrong thing in 2008 or
| so iirc.
|
| It is still dumb that there's a definition of fsync() that
| does not sync :-/
| threeseed wrote:
| > why would somebody think to look up the docs
|
| It seems reckless to me to not do this when you're
| interacting with the filesystem using low-level APIs (i.e not
| via Swift/Obj-C).
| coldtea wrote:
| > _That 's not defence. It fails the principle of least-
| surprise._
|
| Only if the standard where anything else is a "surprise" is
| 2022 Linux.
|
| Many (all?) other unices and macOS itself since forever work
| like that. Including Linux itself in the past [1]
|
| [1] https://lwn.net/Articles/270891/
| fulafel wrote:
| Any references for Unices traditionally skipping FUA or
| synchronize cache to the storage stack? Sounds surprising
| to me. Here's Solaris for example: https://utcc.utoronto.ca
| /~cks/space/blog/solaris/ZFSDiskWrit...
|
| Also re Linux here's eg PostgreSQL 9.0 documentation saying
| ext4/zfs + scsi used the "SYNCHRONIZE CACHE" command with
| fsync even back then, and a equivalent SATA command being
| used by the storage stack with SATA-6 and later drives:
| https://www.postgresql.org/docs/9.0/wal-reliability.html
| marcan_42 wrote:
| Drive caches also used to not exist in the past. At that
| point, behavior was the same as it is on Linux today. It
| then _regressed_ when drive caches became a thing.
|
| Maybe it not being added to OSes when drive caches came
| into the picture was arguably a bug, and Linux has been the
| first OS to fix it properly. macOS instead introduced new,
| non-buggy behavior, and left the buggy one behind :-)
| acdha wrote:
| > Drive caches also used to not exist in the past. At
| that point, behavior was the same as it is on Linux
| today. It then regressed when drive caches became a
| thing.
|
| You mean in the 1980s? Linux wasn't used before this
| wasn't a concern for sysadmins and DBAs. This concern has
| been raised for years - back in the PowerPC era the
| numbers were lower but you had the same arguments about
| whether Apple had made the right trade-offs, or Linux or
| Solaris, etc.
|
| Given the extreme rarity of filesystem corruption being a
| problem these days, one might conclude that the engineers
| who made the assumption that batteries covered laptop
| users and anyone who cares about this will be using
| clustering / UPS were correct.
| CyberRabbi wrote:
| The minute storage manufacturers introduced drive caches
| is the minute this bug became the responsibility of
| storage manufacturers. IMO it's not the kernel's
| responsibility.
| fomine3 wrote:
| Now Apple is the primary storage manufacturer for Mac.
| ecf wrote:
| I'd argue maybe .5% of people are working on something where
| this is even close to being a concern. Those people probably
| know what they need to use.
|
| Apple doesn't need to defend anything.
| ClumsyPilot wrote:
| I am sick of this callous and capricious disrespect for
| users and their data, rampant throughtout this wanky
| industry.
|
| Do lawyers use Apple computers? Do they work on important
| documents relating to life and death?
|
| Some people have literally been executed because developers
| couldn't do their job properly. People have been sent to
| jail for decades because developers fucked up in the
| british postmaster scandal.
|
| Average people life in a dangerous world- work with
| documents about their financial wellbeing. They live in
| opressive countries where being gay is punishable by death.
| They drive 2 ton death machines. And now that we have put
| computers in places where life and limb depends on them, we
| are responsible for doing the job properly, that's why we
| get paid.
| masklinn wrote:
| > That's not defence. It fails the principle of least-
| surprise.
|
| Welcome to C APIs in general, and POSIX in particular.
| supermatt wrote:
| They do it according to POSIX spec. Linux is the oddball
| here.
| marcan_42 wrote:
| The POSIX spec is deliberately ambiguous about this.
| coldtea wrote:
| So? Did Linux do it like that before?
|
| https://lwn.net/Articles/270891/
| marcan_42 wrote:
| It does it like that now, which is what I'd expect if I'm
| writing software.
| wil421 wrote:
| I'd argue more people develop on non-Linux devices such
| as Windows and MacOS on linux itself.
| marcan_42 wrote:
| F_FULLFSYNC is nonstandard. As far as I know there is no
| standard-complicant way to get data on to stable storage
| on macOS. That's a bit of a problem. It makes a lot more
| sense to make the standard-compliant way actually sane.
| supermatt wrote:
| I have said a few times already - F_BARRIERFSYNC. This is
| likely equivalent to what linux is doing.
|
| edit: sorry - not 'standards compliant' (whatever that is
| - does linux declare support for SIO?), but probably what
| you are looking for.
| marcan_42 wrote:
| It isn't. I already replied to you above. A barrier does
| not guarantee data durability and we already know Linux
| fsync() == macOS F_FULLFSYNC because _they have the same
| (lack of) performance on the same hardware_.
| supermatt wrote:
| Thanks marcan. Apologies for wasting your time.
| viraptor wrote:
| Slightly related: if a drive runs with a properly journaled,
| fully checksummed filesystem, for example zfs or btrfs - does the
| write-through mode guarantee that you can only lose new data and
| not corrupt the old?
| viraptor wrote:
| Found it kind of answered in the side thread:
| https://mobile.twitter.com/marcan42/status/14942278033275985...
|
| In short - no, you'll still see corruption.
| floatboth wrote:
| No, you won't see corruption on ZFS. Cutting power to the
| drive is always safe, you can slice a SATA cable with a
| guillotine if you want, you'll always see a consistent state
| of the filesystem. ZFS transactions are entirely atomic.
|
| ZFS (and btrfs) is not "journaled", it's copy-on-write.
| viraptor wrote:
| You won't see corruption of the filesystem itself, but
| you'll see data corruption as described in the thread. If
| the writes are delayed, the write ordering can get messed
| up. + Zfs has ZIL, which is basically journal equivalent.
| EugeneOZ wrote:
| Click-bait title again.
|
| "You can lose some of your file changes in case of hard-reboot"
| is more correct.
|
| It was a given truth for me all the time and I can tolerate some
| data losses if power was accidentally turned off for my desktop,
| or if OS panicked (it happens ~ once per year to me).
|
| If this is a price for a 1000x speed increase - I'm more than
| happy they have implemented it this way.
| marcan_42 wrote:
| You can lose some file changes _even after asking the OS to
| make sure they don 't get lost, the normal way_.
|
| That's a problem. It means e.g. transactional databases (which
| cannot afford to lose data like that) have a huge performance
| hit on these machines, since they have to use F_FULLFSYNC. And
| since that "no really, save my data" feature is not the
| standard fsync(), it means any portable software compiled for
| Linux will be safe, but will be unsafe on macOS, by default.
| That is a significant gotcha.
|
| The question is why do other NVMe manufacturers _not_ have such
| a performance penalty? 10x is fine; 1000x is not. This is
| something Apple should fix. It 's a firmware problem.
| EugeneOZ wrote:
| _No, it's not a problem, it is expected. If you are running a
| transactional database on your desktop - at least add a UPS
| to your system._
| olliej wrote:
| The whole point of a transactional database is that even in
| the case of a power loss you do not lose data. If you UPS
| blows up, and so you lose power, you should not lose data.
|
| The point here is that on the apple systems if you do the
| correct thing your performance drops to that of spinning
| disks.
| EugeneOZ wrote:
| Add a secondary UPS. What will be the next excuse?
|
| It's ridiculous to think that in case of power loss you
| expect 100% data integrity - it might happen in the
| middle of the command execution. If the system should be
| unkillable, it should have an unkillable power source in
| the first place.
| coldtea wrote:
| > _Add a secondary UPS. What will be the next excuse?_
|
| You shouldn't have to add a secondary UPS at all, period,
| and still get that.
|
| Databases are designed that way (for integrity under
| sudden power loss) - the OS just needs to provide a
| standard call for the sync that they can use.
|
| Now, fsync not guaranteeing a write is one thing -- and
| it's common in other OSes, even Linux behaved like that.
|
| The non-commital fullsync on the other hand (and the slow
| speed) are problematic, and that's not an excuse for the
| user having such a bizarro case as wanting to run a DB on
| their Mac Mini without 2 UPS, that's you excusing Apple.
|
| Not to mention that 2 UPS wont solve the problem if
| you're not there to shut down the computer gracefully as
| they, themselves, are depleted (e.g. at night) when
| there's a powerloss.
| marcan_42 wrote:
| The entire point of modern journaling filesystems and
| properly designed transactional databases is to ensure
| 100% data integrity in case of power loss, every time, no
| matter what. The thinking you have is from the 1990s. We
| can (and do) do better today.
|
| A properly designed transactional database will only ever
| "fail ahead". If power fails a transaction that was in
| the process of committing _might_ commit without an ack,
| but will never return an ack and then be lost on the next
| startup. The ack means the data is safe, regardless of
| what happened afterwards.
| EugeneOZ wrote:
| After reading this comment I'm not so sure about that:
| https://news.ycombinator.com/item?id=30371181
| marcan_42 wrote:
| That comment is about the semantics of OS APIs;
| filesystems are designed not to corrupt themselves in
| case of hard shutdown, and this is true as long as the
| underlying storage is well-behaved (e.g. honors flush
| requests). Databases on macOS already use F_FULLFSYNC (if
| they noticed this issue) to provide those guarantees. On
| Linux they just use fsync().
| freemint wrote:
| Since no Mac device has two power support adding a second
| UPS means chaining which will only increase the risk of
| something going wrong in the chain.
|
| Nobody expects 100% data integrity on power of. What is
| expected is that data that what was fsynced has 100% data
| integrity once that system call returns. This information
| is also used when moving files across the network, the
| file gets deleted on the sender when the receiver said
| fsync is completed. This means you could loose entire
| files of data when moving things over the network onto a
| Mac
| marcan_42 wrote:
| For what it's worth, adding an external UPS to a Mac
| laptop counts as two, and in fact you can add one per
| Type C port + MagSafe, so you can have up to 5 battery
| backups for the 2021 Macbook Pro line (internal + MagSafe
| + 3 x Type C).
|
| It's silly, but you _can_ :-)
| freemint wrote:
| I never considered Mac Laptops as they are not vulnerable
| to data loss in case of a power outage. How is the
| situation with Mac Minis which don't have a battery? Are
| there multiple redundant power input the Mac Mini can
| switch in-between without tuning of?
| EugeneOZ wrote:
| This means the world would be full of complaints from
| macOS users, but for some reason, we only know about this
| detail because of that "shocking" Twitter thread.
| layer8 wrote:
| Looking at the Apple support forums, the world _is_ full
| of complaints about all sorts of malfunctions (which may
| or may not be related).
| EugeneOZ wrote:
| Not sure if intentional or not, but it was a good joke,
| thanks :)
| marcan_42 wrote:
| #!/usr/bin/python import os, sys, time, datetime
| t = datetime.datetime.now().isoformat()
| print(t) for i in range(5):
| time.sleep(1) print(4 - i) fd =
| os.open(sys.argv[1], os.O_RDWR|os.O_CREAT)
| os.lseek(fd, 0, 0) os.write(fd, b"test: " +
| t.encode("ascii") + b"\n"); os.fsync(fd)
| print("done!") time.sleep(100)
|
| Run that on a Mac Mini. Do it a couple times. Remember
| the timestamp of the last one. Let it count down, then
| pull the plug within a few seconds after "done!" shows
| up. Boot up again. The file contents will have reverted
| to a prior point in time.
|
| This isn't some hypothetical thing, this is a trivial
| test you can do. fsync() on macOS does not guarantee data
| is on stable storage. And this is actually well
| documented.
|
| Then if you want to see the performance problem, make it
| a loop instead and use `fcntl.fcntl(fd,
| fcntl.F_FULLFSYNC, 1)`. You'll get 40-odd IOPS, but at
| least your data won't disappear after power loss.
| EugeneOZ wrote:
| Does it somehow prove that people are losing files when
| transferring them over the network? People would
| definitely notice it.
| marcan_42 wrote:
| You're very persistent, aren't you?
| marcan@raider:~/tmp -$ echo "very important data" >
| file.txt marcan@raider:~/tmp -$ rsync --remove-
| source-files file.txt macmini.lan:
|
| Yanked power to macmini.lan after the rsync completed,
| then turned it on again
| marcan@raider:~/tmp -$ ls file.txt ls: cannot
| access 'file.txt': No such file or directory
| marcan@raider:~/tmp 2$ ssh macmini.lan Last login:
| Thu Feb 17 23:45:26 2022 from 192.168.3.10
| marcan@Mini-M1-2020 ~ % ls file.txt ls: file.txt:
| No such file or directory
|
| Data's gone.
|
| This is real, please stop pretending it isn't.
| EugeneOZ wrote:
| At least I'm not as rude as you. You'll need to be
| persistent to accomplish your results with the Asahi
| Linux project - good luck with that.
|
| I had no doubts that you can lose your file if you are
| moving it and some very lucky power outage hits. I was
| not "pretending" it's not real.
|
| What I have doubts about, is that it's a real concern for
| 99.93% of the users. As we've found here, it's not even a
| rare case in other kinds of OS, so users would definitely
| notice it.
|
| It is theoretically possible, of course. In practice,
| it's just too rare to consider.
|
| But still, I hope this topic will be noticed by Apple and
| they fix that low performance issue for fullsync. Also, I
| hope they will not make fullsync as the default behavior
| - it doesn't worth the risks (and those for whom it does
| - should use some Linux for sure).
| camgunz wrote:
| You're missing the point. Anything wanting data integrity
| will now have to use F_FULLFSYNC, which is slow on this
| firmware. You might say "you shouldn't care about data
| integrity to this degree", but databases do, and lots of
| developers run databases on their machines, and now
| they'll be real slow. Maybe they'll add some config
| options like `i_promise_i_dont_care_about_data_integrity
| = 1`, but come on.
| EugeneOZ wrote:
| I've mentioned this issue.
| camgunz wrote:
| What you wrote is:
|
| > No, it's not a problem, it is expected. If you are
| running a transactional database on your desktop - at
| least add a UPS to your system.
|
| This misses the point--PostgreSQL on my machine is either
| lossy or slow. A UPS (or battery, in my case) doesn't fix
| that.
| Maxion wrote:
| I guess I am old but the assumption I live by is that if power
| is suddenly cut from a computer - no matter desktop or laptop -
| it can damage the FS and/or cause data loss.
|
| For any mission critical stuff, I have it behind a UPS.
| olliej wrote:
| No, modern filesystems aren't expected to be corrupted by
| sudden power loss, and "put it behind a ups" assumes that
| it's impossible for a UPS to fail.
| sudosysgen wrote:
| Modern filesystems are designed so that you can unplug the
| hard drive and it will not be in a corrupted state.
|
| UPSes can and do fail.
| marcan_42 wrote:
| At least your thinking is old. Modern filesystems and
| databases are designed to prevent data loss in that scenario.
|
| The last time I saw a modern filesystem eat itself on sudden
| power loss was when I was evaluating btrfs in a datacenter
| setting, and that absolutely told me it was not a reliable FS
| and we went with something else. I've never seen it happen
| with ext4 or XFS (configured properly) in over a decade,
| assuming the underlying storage is well-behaved.
|
| OTOH, I've seen cases of e.g. data in files being replaced by
| zeroes and applications crashing due to that (it's pretty
| common that zsh complains about .zsh_history being corrupted
| after a crash due to a trailing block of zeroes). This
| happens when filesystems are mounted with metadata journaling
| but no data journaling. If you use data journaling (or a
| filesystem designed to inherently avoid this, e.g. COW
| cases), that situation can't happen either. Most databases
| would be designed to gracefully handle this kind of situation
| without requiring systemwide data journaling though. That's a
| tradeoff that is available to the user depending on their
| specific use case and whether the applications are designed
| with that in mind or not.
| freedomben wrote:
| how long ago did you last test btrfs?
| newaccount74 wrote:
| The problem isn't that the default case is unsafe -- the
| problem is that the safe case is so extremely slow.
| formerly_proven wrote:
| Deferring flushes on the NVMe level could also corrupt a
| journaling FS itself, not just the contents of files written with
| proper fsync incantations.
| marcan_42 wrote:
| Indeed, though that is somewhat rare. For our distro, I would
| opt to enable it by default on laptops (which is quite safe)
| and disable it on desktops.
| zeotroph wrote:
| APFS at least has metadata checksums to prevent that. However
| it does not do data checksums (weird decision...), despite
| being a CoW fs with snapshotting, similar to ZFS and btrfs.
| vbezhenar wrote:
| They rely on hardware storing checksums and on protocol using
| checksums to prevent data corruption on all levels.
| formerly_proven wrote:
| Everyone else does that as well and it's not a substitute
| for end-to-end data integrity.
| matthewmacleod wrote:
| That's interesting! I'd be fascinated to know what the underlying
| cause is - these full-sync numbers are _amazingly_ low.
| marcan_42 wrote:
| I dug a bit further and the NVMe controller is doing about
| 6.2MB/s of DRAM reads and 10MB/s of DRAM writes while doing a
| flush loop like this (which it isn't doing with the same
| traffic sans the flushes). I wonder if it's doing something
| dumb like linear scanning a cache hash table to find things to
| flush... or maybe something with bad cache locality?
|
| I'm pretty sure, whatever it is, Apple could fix it in a
| firmware update.
___________________________________________________________________
(page generated 2022-02-17 23:02 UTC)