[HN Gopher] SquirrelFS: Using the Rust compiler to check file-sy...
___________________________________________________________________
SquirrelFS: Using the Rust compiler to check file-system crash
consistency
Author : belter
Score : 98 points
Date : 2024-06-23 14:02 UTC (1 days ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| metadat wrote:
| Does this have a practical use? It's definitely a novel
| application of a property of Rust. It's also been my impression
| filesystem consistency is largely a solved problem thanks to
| write-ahead logs (WAL) and the like.
|
| It's nice the authors included a link to the underlying source
| code in the last paragraph:
|
| https://github.com/utsaslab/squirrelfs
| vlovich123 wrote:
| There's a pretty big difference between leveraging the compiler
| to do a formal proof of the codebase on your behalf vs runtime
| tests to make sure your WAL is maintaining the invariants you
| expect it to have.
| IshKebab wrote:
| Does what have a practical use? The actual filesystem, or using
| the typestate pattern to prove the absence of bugs?
|
| The latter is clearly a practical use - no bugs (or at least
| vastly less likely bugs).
|
| The former is maybe a no since it seems like the authors
| couldn't achieve their results for "normal" filesystems because
| they are asynchronous - only for "persistent memory" which is
| synchronous. I dunno exactly who has persistent memory though.
| I guess it's stuff like Intel's Optane (which was
| discontinued)?
| jmartin2683 wrote:
| The idea of typestate programming and how it allows you to
| model all of the valid states and transitions between them is
| definitely useful in a lot of contexts. if you've ever had to
| call some init() method on an object in an OO language before
| you could call something else, this pattern would throw a
| compile time error if you violate that.
| klysm wrote:
| This is definitely part of the future for better storage systems.
| Too much responsibility is currently in the hands of programmers
| (like me) to not make any mistakes. The storage layer can be a
| high consequence place to make a mistake!
| Shoop wrote:
| I'm guessing that the synchronous update architecture they're
| using only really only makes sense for persistent memory and that
| this couldn't easily be adapted to conventional hard drives or
| SSDs?
| 01HNNWZ0MV43FF wrote:
| If the drive controllers don't lie about fsync, then maybe?
| KennyBlanken wrote:
| This is great but may ultimately be useless, because the storage
| industry has a long history of not obeying SCSI/IDE/SATA/NVMe
| commands around flushes and lying to the OS about when data has
| been committed to disk. You can't trust a drive to write stuff
| when you tell it to, even when you tell it REALLY NO, SERIOUSLY,
| WRITE THAT SHIT DOWN NOW...and certainly not in the order you
| tell it to.
|
| Long ago an Apple engineer told me that one of the reasons Apple
| sold 68k Macs with Apple-branded SCSI hard drives (and disk
| utilities that wouldn't work with non-Apple drives) and continued
| to use Apple-branded drives well into the PowerPC and IDE era was
| because Apple had been burned by writing operating system /
| filesystem code that took what drives reported at face value and
| assumed they were doing what they were supposed to according to
| the specs for the interface standard they'd supposedly been
| certified to meet.
|
| Sure, it also enabled markups, but it wasn't just about having a
| big margin on storage, and there are additional costs involved.
| It's also why often Apple/Quantum drives were slower - they were
| actually doing what they were supposed to be doing, whereas
| everyone else as doing what was fastest to juice reviews, and
| magazine reviewers were either too cozy with manufacturers, or
| too lazy/stupid, to actually test to see if a drive was doing
| what it was supposed to be doing.
|
| The groups behind the interface standards? Trademark money
| printer go brrrrrrrrrrrrrr. Those golf balls and yachts and
| supercars aren't gonna drive themselves.
|
| And yes, this has persisted into the age of NVMe flash. Power
| failures with SATA and NVMe flash drives can be a real risky
| business, because the controller is shifting data every which
| way. From OS memory (some NVMe drives use host memory for caching
| at the hardware level, which is real snake-oil salesman shit
| Barnum would be proud of) or drive RAM, SLC mode short term
| storage, and higher level cells (MLC, QLC, etc.) And the
| controller has to keep track of all this, in addition to its
| tables providing the mapping between what the OS considers
| physical blocks and where those blocks are actually written, due
| to wear leveling.
|
| A power failure with an SSD can potentially brick the drive,
| (effectively) permanently.
|
| Do reviewers test for this? Nope. Most of them didn't even
| realize manufacturers were sending higher-spec NVMe drives out
| for review and in initial distribution, and then very quickly
| shifting to much cheaper controllers and flash, or they were in
| on it and kept their mouths shut so they didn't lose access to
| review sample hardware.
|
| Do not trust your storage, at all. Fault tolerance, backups, and
| power protection suitable for the importance of the data...
| johnisgood wrote:
| How does it compare to Nova-Fortis, PMFS, Strata, Ziggurat
| (Rust), SplitFS, and Aerie?
___________________________________________________________________
(page generated 2024-06-24 23:02 UTC)