[HN Gopher] Fast Commits for Ext4
       ___________________________________________________________________
        
       Fast Commits for Ext4
        
       Author : joseluisq
       Score  : 80 points
       Date   : 2021-01-30 10:22 UTC (12 hours ago)
        
 (HTM) web link (lwn.net)
 (TXT) w3m dump (lwn.net)
        
       | syoc wrote:
       | Previous discussion (51 comments)
       | https://news.ycombinator.com/item?id=25794659
        
       | Badfood wrote:
       | I didn't realise fsync syncs the whole fs. This explains some
       | write cache issues I've been having writing real-time data
       | streams to sometimes slow SD cards. This change would solve my
       | problems rather than create problems. But hey now I can fix it
       | anyway
        
         | sbierwagen wrote:
         | fsync is _supposed_ to sync the entire filesystem, yes. On a SD
         | card, I would expect it to do what it says it does.
         | 
         | However, on regular hard drives, filesystem devs and hardware
         | vendors are all incentivized to cheat, since io benchmarks are
         | big and obvious, while dropping data on the floor with power
         | loss or failing hardware is subtle and hard to notice.
         | 
         | In practice, reliably syncing a file can be surprisingly hard:
         | https://danluu.com/file-consistency/
        
       | tobias3 wrote:
       | Great development! I'm certainly all for this becoming the
       | standard/default. Unfortunately for those relying on fsync()
       | flushing the whole file system (which one shouldn't) syncfs()
       | isn't a great replacement. Last time I tested that it didn't
       | return errors on failure nor was it making certain all changes to
       | the file system before syncfs() made it to disk.
       | 
       | Btrfs had this "fast fsync" for a while already btw. (out of
       | necessity -- otherwise fsync would be even more unbearably slow
       | on btrfs). This seemed to be difficult to implement (lots of
       | corner cases). Every release there were "fast fsync" bug fixes
       | such as e.g. https://lore.kernel.org/linux-
       | btrfs/20200220132949.20571-1-f... https://lore.kernel.org/linux-
       | btrfs/20200722112837.15516-1-f... . Let's hope the ext4 version
       | of this has less initial issues.
        
       | breck wrote:
       | Anyone have links to the best resources to master the core
       | paradigms of file systems and stay up to date with future R&D?
       | 
       | Best books? Websites? Journals? People on Twitter to follow?
       | GitHub projects to watch? YouTube videos?
        
         | digikata wrote:
         | If you read a couple of key papers, you'd be well introduced to
         | the fundamentals of linux in-kernel filesystems.
         | 
         | File systems like ext4 are descended from this approach:
         | https://docs.freebsd.org/44doc/smm/05.fastfs/paper.pdf
         | 
         | File systems like btrfs, f2fs, etc are descended from this
         | approach (log structured file systems):
         | https://web.stanford.edu/~ouster/cgi-bin/papers/lfs.pdf
         | 
         | For cloud based storage, the LSM paper (w/ leveldb), and HDFS
         | are good foundational references imho.
         | 
         | Obviously there are a lot of details beyond those areas, but
         | understanding these are fairly short but will give you a basis
         | to get much further to more cutting edge R&D discussions.
        
         | jorangreef wrote:
         | I would recommend reading all the fantastic storage papers
         | coming out of WISC under Remzi and Andrea Arpaci-Dusseau:
         | 
         | http://pages.cs.wisc.edu/~remzi/research/
         | 
         | And since you're looking for a GitHub project to watch, I'll go
         | ahead and share TigerBeetle since I'm working on that :) and
         | since we're implementing many of these new storage ideas
         | (disentangling journal corruption from system crashes,
         | protocol-aware recovery for consensus-based storage, hash-
         | chaining to detect misdirected writes and disk corruption, and
         | Direct I/O to recover from the kernel page cache going out of
         | sync with the disk):
         | 
         | https://github.com/coilhq/tigerbeetle
         | 
         | And then finally since you also asked for a YouTube video :)
         | here's a talk we did on Thursday for the Mojaloop Foundation,
         | discussing many of these ideas around storage safety and also
         | around optimizing I/O performance (io_uring, the cost of
         | syscalls and context switches, cache line aligned structures,
         | Direct I/O) and putting this all together into a replicated
         | state machine with strict serializability:
         | 
         | https://www.youtube.com/watch?v=6D7B6v06mBo
        
       ___________________________________________________________________
       (page generated 2021-01-30 23:01 UTC)