[HN Gopher] Bcachefs Merged into the Linux 6.7 Kernel
___________________________________________________________________
Bcachefs Merged into the Linux 6.7 Kernel
Author : LinuxBender
Score : 248 points
Date : 2023-10-31 11:19 UTC (11 hours ago)
(HTM) web link (www.phoronix.com)
(TXT) w3m dump (www.phoronix.com)
| the_duke wrote:
| Exciting!
|
| I've played around with it a few times, since it's been easily
| available in NixOS for a while. Didn't run into any issues with a
| few disks and a few hundred GB of data.
|
| Some very interesting properties, including (actually efficient)
| snapshots, spreading data over multiple disks, using a fast SSD
| as a cache layer for HDDs, built-in encryption (not audited yet
| though!), automatic deduplication, compression, ...
|
| A lot of that is already available through other file systems
| (btfs, zfs) and/or by layering different solutions (LVM, dm-
| crypt, ...) , but getting all of it out of the box with a single
| FS that's in the mainline kernel is quite appealing.
| mike_hock wrote:
| I don't find this lack of separation of concerns appealing,
| especially for crypto since it dilutes auditing resources. And
| blockdev-level crypto is simpler and harder to fuck up.
|
| Snapshots and compression OTOH are better done on the FS level.
| kevincox wrote:
| > blockdev-level crypto is simpler and harder to fuck up.
|
| Simpler yes, but since generally no extra metadata is
| allocated it is vulnerable to some cryptanalysis notably
| comparing different snapshots of the drive. Doing this
| properly requires storing unique keys for different versions
| of data. Doing this with typical blockdev-level encryption is
| very expensive (you either need to reduce the effective block
| size which disrupts lots of software that assumes things
| about block size or store the data out-of-line (typically at
| the end of the disk) which requires up to 2x writes. Doing
| this in the filesystem allows strong encryption with minimal
| performance impact (as the IV write is co-located with data
| that is changing anyways).
| Hello71 wrote:
| > or store the data out-of-line [...] which requires up to
| 2x writes
|
| and because writes to separate sectors aren't atomic, you
| probably want to add journaling or some kind of CoW for
| crash safety, and oh look now you're actually just writing
| a filesystem and it's not simpler anymore.
| mike_hock wrote:
| No, you're writing something that happens to resemble a
| small subset of the functionality of a filesystem.
|
| Most importantly, you're not duplicating the effort for
| every filesystem that you want to support encryption for,
| and the code can largely remain fixed once mature.
| Tobu wrote:
| kernel developers are free to factor in common
| functionality, calling into it library style, without
| making it into an externally visible layer. IIRC fs
| encryption and case insensitivity are often done that
| way, and I think I'd count the page cache and bios as
| larger library-style components as well (as opposed to
| the VFS layer which is more in a framework style).
| jhoechtl wrote:
| It should be highlighted that all these features are supported
| by btrfs too which has certainly seen much more test in the
| last years.
|
| Yes there have been issues in the past but since quite a a
| while its stable and feature-rich. Easily the most advanced
| free file system
| the_duke wrote:
| btrfs does not have the fast-disk cache feature I think, and
| definitely not the built-in encryption.
| MrDrMcCoy wrote:
| It also doesn't do block volumes or active dedupe. Btrfs is
| great, but I'm very excited about having all these features
| in one place!
| Ecco wrote:
| Having just heard of bcachefs when reading this article, I tried
| to understand what makes it better than other existing FS but
| couldn't quite find a clear answer. It feels like it's feature
| set is equivalent to ZFS.
|
| Do you guys know why someone should get excited by bcachefs?
| pantalaimon wrote:
| ZFS will never be integrated into the Linux kernel due to it's
| licence. btrfs is complicated to use and has many pitfalls that
| can lead to it eating your data.
| ndsipa_pomu wrote:
| > btrfs is complicated to use and has many pitfalls that can
| lead to it eating your data
|
| I use btrfs in preference over ext4 for Linux filesystems and
| turn on zstd compression for performance and a bit of space
| saving. It seems simple enough for my use case, though I'm
| not doing any snapshots etc.
|
| What are some of the potential pitfalls?
| pantalaimon wrote:
| I was very excited about btrfs' advanced features, but that
| meant that btrfs would bite me multiple times when I
| expected it to 'just work(tm)':
|
| - RAID5/6 are still not stable
|
| - it will not mount a RAID in degraded mode automatically,
| failing the high availability promise that might tempt you
| towards RAID.
|
| - swapfile support exists, but it breaks snapshots (and I
| don't want to snapshot the swapfile)
|
| - Just an Ubuntu/Debian thing, but snapshots are not
| integrated into the update process unless you install `apt-
| btrfs-snapshot` (and know that package exists)
| ndsipa_pomu wrote:
| > Just an Ubuntu/Debian thing, but snapshots are not
| integrated into the update process unless you install
| `apt-btrfs-snapshot` (and know that package exists)
|
| Thanks - I did not know about that.
|
| I agree with the stance of not mounting a degraded RAID
| automatically as then the danger is that someone might
| not notice it and be subjected to total data loss later
| on. The best option would be to allow over-riding that
| choice if the RAID is otherwise monitored.
| webstrand wrote:
| Can you not just place the swapfile inside of another
| subvolume? Since subvolumes are not included in
| snapshots. There's generally a bunch of stuff in /var
| that you don't want to include in snapshots, too, so it's
| not like putting the swapfile inside of a subvolume is an
| exotic task.
| pantalaimon wrote:
| I guess so, I just didn't know when I created the
| swapfile and was surprised later when I tried to create a
| snapshot.
| xorcist wrote:
| Be careful, just like disk images, you probably don't
| want to place swapfiles on a CoW filesystem.
| rini17 wrote:
| - RAID5/6 are still not stable
|
| Why does everyone insist on RAID5? I am completely fine
| with btrfs RAID1, survived few disk crashes as
| advertised.
|
| - it will not mount a RAID in degraded mode automatically
|
| it will if using -o degraded, it's the tooling which
| generally sucks and won't support this. I even had
| problems booting at all with / on multi-device btrfs
| using recommended tools (dracut, grub-mkconfig), the bugs
| are known and unfixed, ended up rolling out own
| initramfs.
|
| - swapfile support exists, but it breaks snapshots
|
| you are supposed to have all your stuff in a subvolume,
| and snapshot that, not whole toplevel root
| filesystem...and yes it should be documented better that
| it interferes with snapshots
|
| - but snapshots are not integrated..
|
| yep tooling sucks
| dinosaurdynasty wrote:
| RAID1 is a lot more expensive and/or stores a lot less
| data than RAID5/6.
| rini17 wrote:
| How much is the "a lot"?
| pantalaimon wrote:
| You can try yourself: https://carfax.org.uk/btrfs-usage/
|
| But generally, RAID1 gets you 50% of storage space,
| whereas RAID5 gets you 66% (and any odd combination of
| disks).
| bmicraft wrote:
| RAID5 should actually get you n-1 disks of space (if
| they're equally large), while raid1 only gets you n/2
| __david__ wrote:
| Raid1 actually gets you less than that. If you mirror 5
| drives that only gives you 1 drive worth of data (with 4
| redundancies).
|
| But... you have to weigh it against losing all your data.
| With raid5 if any 2 of your disks die at the same time
| then you've lost your volume and most likely 100% of your
| data (no matter how many disks you have).
|
| With raid1 you have to lose _all_ your disks before that
| happens. Typically that's also just 2 but you can mirror
| 3 or more drives if you need some data to be _really_
| resistant to disk failures and you don't care about
| "wasting" n-1 times the space.
|
| So you end up trading off efficiency of storage space and
| resiliency to data loss.
|
| Myself, disks got cheap enough that I always just buy 2
| disks and mirror them. I find it easier to reason about
| overall, especially in the face of a degraded array.
| keep_reading wrote:
| > it will not mount a RAID in degraded mode
| automatically,
|
| I remember when btrfs was very young and they announced
| the ability to create mirrors. I tested this out and was
| pleased, and then I tested the failure scenario: pull a
| drive, try to boot.
|
| It wouldn't boot!
|
| I jump into IRC and ask if it's expected that you can't
| boot from a degraded mirror and the answer was "not
| supported" which means the mirror is pointless.
|
| Obviously it has improved since then as there's a way to
| force it to work, but I returned to ZFS on FreeBSD and
| never looked back
| nwmcsween wrote:
| Yeah this is the probably the worst default for a fs that
| supports raid.
| throw0101c wrote:
| > _What are some of the potential pitfalls?_
|
| RAID-5/6:
|
| * https://btrfs.readthedocs.io/en/latest/btrfs-
| man5.html#raid5...
| viraptor wrote:
| Wouldn't describe that as a pitfall. As far as I can
| tell, every place in the docs/help which talks about
| those raid modes tell you they're experimental and
| shouldn't be used. At that point it's "you've done a
| stupid thing and discovered the consequences you were
| told about".
| throw0101c wrote:
| It's a feature that other file systems have: why should I
| choose Btrfs when (e.g.) ZFS can do everything it can
| _and more_ on top of that?
|
| I remember when Btrfs was announced in 2007 as I was
| already running Solaris 10 with ZFS in production (and
| ZFS had non-"experimental" RAID-5-like RAID-Z from day
| one). Here we are 15+ years later and Btrfs _still_ doesn
| 't have it?
| viraptor wrote:
| I'm just saying it's not a pitfall, because it's clearly
| documented up front and warned about. That's orthogonal
| to whether you should choose this filesystem and why a
| feature is implemented or not.
| pantalaimon wrote:
| The nice thing about btrfs is that you can add/remove
| drives at will. That allows for easy expansion of an
| existing pool.
| chasil wrote:
| The one thing that ZFS cannot do is defragment a pool.
|
| There are abuse patterns that are toxic for ZFS pools
| (and all other filesystems). Btrfs appears to be able to
| repair this damage.
|
| https://tim.cexx.org/?p=1236
|
| https://www.usenix.org/system/files/login/articles/login_
| sum...
| pantalaimon wrote:
| eh 10 years ago RAID56 was already declared 'pretty much
| ready', that hint was added later when major data loss
| bugs were discovered.
| viraptor wrote:
| One that seems to catch out many people is that you need to
| explicitly add the "degraded" option to allow mounting raid
| volumes which contain a broken drive. This is opposite to
| almost all other filesystems. I've seen people confused and
| thinking they lost the whole volume, even though they just
| need to replace the bad drive and rebuild as usual.
| ndsipa_pomu wrote:
| Personally, I agree with that. It's good practise for
| systems to fail quickly if they find themselves in an
| uncertain state and with RAID, it's important for the
| operator to know if they've suddenly lost redundancy so
| that they can resolve the issue.
| viraptor wrote:
| It really depends on what's your situation. At home I
| probably want the disks to stay idle until replacement.
| In a bigger production system I want to keep the
| availability and ping the on-call person to replace the
| drive in the background.
| ndsipa_pomu wrote:
| Exactly - the default should be to get someone to pay
| attention to it when something breaks, and if you're
| planning on high availability, then you can choose that,
| assuming you keep an eye on the health of the RAID.
| keep_reading wrote:
| If you can't even boot because of a mystery error how are
| you supposed to resolve the issue?
|
| It should instead be giving the user error messages
| written to their terminal, in logs, etc instead of
| breaking the entire system until the user finds the
| manual
| wtallis wrote:
| If your NAS has a hot spare drive installed then it
| should probably include the degraded mount option by
| default and automatically add the hot spare drive to the
| filesystem in the event of a failure. Alternatively, if
| there's enough free space on the surviving drives to
| rebalance the array and restore redundancy without
| replacing the failed drive, that operation could be
| kicked off automatically. Or the filesystem can be
| (hopefully temporarily) set to _not_ store new data
| redundantly, if that is an acceptable risk for the user.
| But the filesystem cannot know which method the user
| would prefer; automatically rebuilding the array involves
| policy decisions that are outside the scope of the
| filesystem and requires userspace tooling.
|
| If the system _doesn 't_ have spare capacity ready, the
| only sane response is to _not_ boot /mount normally.
| "giving the user error messages written to their
| terminal, in logs, etc" isn't a real solution for
| something like a NAS with no terminal connected and
| nobody looking at the logs as long as they can still
| establish a SMB connection; it's too likely to be a
| silent failure in practice. Mounting the filesystem
| degraded but read-only makes sense if it's necessary to
| boot the system so that the user (or their pre-configured
| userspace tooling) can decide how to deal with the
| problem, but a lot of Linux distros aren't happy with the
| root filesystem being read-only.
|
| In summary: there's no single right answer to the problem
| of a failed drive, and btrfs defaults to what is the
| safest behavior based on the information available to the
| filesystem itself. Userspace tooling with more
| information can make other, less universal choices. A
| distro that tries to simply adopt btrfs as a drop-in
| replacement for ext4 probably doesn't have all the
| tooling necessary to make good use of the unique features
| of btrfs.
| keep_reading wrote:
| > If the system doesn't have spare capacity ready, the
| only sane response is to not boot/mount normally.
|
| It doesn't need the spare to "boot normally" and the
| system can turn on a scary LED, ring bells, call you,
| text you, hit you up on WhatsApp, DM you on Instagram, or
| whatever method you want your NAS to use to notify you
| there's a degradation. (You're monitoring it right??)
|
| This explanation of "it's dangerous to boot off a
| degraded array" is lunacy. I will not take this terrible
| advice from armchair experts when I've been doing this
| for over 25 years
| wtallis wrote:
| I didn't say it's dangerous to boot off a degraded array.
| I said it's dangerous to boot off a degraded array
| _normally_. Mounting it degraded but _read-only_ is
| reasonable, because that prevents silently writing new
| data without the level of redundancy the user previously
| requested.
|
| There's nothing terrible about advice against responding
| to a drive failure by putting the system into an even
| more precarious state without user interaction.
| nwmcsween wrote:
| Just wondering have you worked on any large DCs or large
| NAS or SAN systems? Drive failures are a daily occurrence
| in places with a lot of spinning metal, having things
| fail to boot by default would be a nightmare.
| curt15 wrote:
| It performs poorly for certain workloads -- notably DBs --
| unless you disable copy-on-write, compression, and
| checksumming. But then why use it over ext4 in the first
| place?
| ndsipa_pomu wrote:
| That's not surprising though, as DBs usually work best
| when given raw storage - the features of the filesystem
| are being duplicated by the DB and thus doing almost
| twice the work.
|
| What seems particularly interesting about Bcachefs is how
| much it seems to be using database concepts to implement
| a general filesystem. Ultimately, it seems inevitable
| that filesystems and databases will converge as they're
| both supposed to manage data.
| PlutoIsAPlanet wrote:
| The "fix" to databases and VMs on btrfs by disabling CoW
| unfortunately disables nearly all the useful features.
| Ridj48dhsnsh wrote:
| Just an anecdote, but when I used gocryptfs on a btrfs
| partition, I'd always end up with a few corrupted files on
| power failure. After switching to gocryptfs on ext4, I
| never have any corruption.
| lproven wrote:
| Aside from the issues described further downthread:
|
| * On Btrfs the `df` command lies. You can't get an accurate
| count of free space.
|
| * There is no working `fsck` and the existing repair tools
| come with dire warnings. Take these very _very_ seriously.
| I have tested them. They do not work and _will_ destroy
| data.
|
| * The main point of Btrfs is snapshots. [open]SUSE, Spiral
| Linux, Garuda Linux and siduction all use these heavily for
| transactional updates.
|
| _But_ the snapshot tool cannot test that there 's enough
| free space for the snapshot, because `df` lies. So, it will
| fill up your disk.
|
| Writing to a full Btrfs volume _will_ corrupt it. In my
| testing it destroyed my root partition roughly once per
| year. It was the most unstable fs I have tried since the
| era of ext2 in the mid-1990s. (Yes I am that old.)
| VancouverMan wrote:
| I've experienced corruption and data loss with Btrfs each
| of the times I've tried using it, too, after only about a
| week of use at most.
|
| Thankfully, all of those incidents were with some non-
| critical, throw-away VMs where the data loss wasn't
| really an issue.
|
| I've also used ext4 under the same circumstances for
| years, and I can't think of a single time that I've lost
| data, nor have I experienced corruption that fsck
| couldn't easily deal with.
|
| I, too, would have to go back to the 1990s to think of a
| filesystem I used that was that unreliable.
|
| After what I experienced, I don't trust Btrfs at all, and
| I have no plans to ever use it again.
| lproven wrote:
| I am glad to hear it's not just me!
|
| I worked at SUSE for 4Y and used it every day. The
| company is in deep denial about its problems, or that
| there _are_ any problems, and when I pointed at ZFS as a
| more mature tool, this was actually mocked.
| jhoechtl wrote:
| When did you run in any of these issues recently? These
| issues were a fact in the past but have been sorted out.
|
| df is not lying because the fs layer reports corrupt data
| but because of dedup and fs layering.
| creatonez wrote:
| > There is no working `fsck` and the existing repair
| tools come with dire warnings
|
| This stems from a misunderstanding. Fsck and fsck-
| adjacent tools have three purposes:
|
| 1. Replay journal entries in a journalled filesystem, so
| that the filesystem is repaired to a good state for
| mounting
|
| 2. Scrub through checksums and recover any data/metadata
| that has a redundant copy
|
| 3. In rare cases, a fsck tool encountering invalid data
| can make guesses as to how the filesystems should be
| structured -- basically, shot in the dark attempts at
| recovery.
|
| Btrfs does not need #1 because it is not journalled.
| Assuming write barriers are working, any partially
| written copy of the filesystem is valid and will simply
| appear as if the pending writes had been rolled back.
| This aspect alone greatly diminishes the need for a fsck
| tool that filesystems like ext4 have.
|
| As for #2, Btrfs already has scrub support. No issues
| there.
|
| As for #3, it's questionable whether you should ever rely
| on such functionality, and fsck tools that do implement
| such functionality tend to have little maneuverability in
| the first place.
| chasil wrote:
| With ZFS, if a redundant member of a pool goes offline for
| a time but is then returned, only the updated blocks are
| written to bring it up to date, and this happens
| automatically.
|
| Unfortunately, btrfs is not that smart, and you must
| trigger a rebalance event to rewrite every block in the
| filesystem to return to full redundancy.
|
| This rebalance behavior is a deal-killer for many uses.
| fodkodrasz wrote:
| It is (soon) officially in kernel, as opposed to zfs.
|
| It has a more limited feature set and said to have simpler
| codebase than zfs/btrfs. It has a single outstanding non-stable
| feature.
|
| It seems to be in active development, while btrfs seems to have
| become stagnant/abandonware before it was finished/stabilised
| completely. I have read several horror stories about data loss,
| so I have avoided it so far.
|
| On the other hand it is not widely deployed yet, there is less
| accumulated knowledge than in case of zfs.
|
| I'm looking forward to trying it in my NAS when buying new
| disks next year. The COW snapshots would fit my needs
| (automatic daily snapshots, weekly backups).
|
| (Now using LUKS+LVM+ext4, this would give a better, more
| integrated, deduplicated solution, I have lots of duplicated
| data right now)
| viraptor wrote:
| > while btrfs seems to have become stagnant/abandonware
| before it was finished/stabilised completely
|
| Why would you think so? I can't remember the last time a
| kernel was released without something at least a bit exciting
| about btrfs
|
| https://kernelnewbies.org/LinuxChanges#Linux_6.5.File_system.
| ..
| Geezus_42 wrote:
| Because they still haven't fixed the write hole issues in
| certain RAID configurations,which have been known for a
| decade or more.
| viraptor wrote:
| A project not finishing some feature is not the same as
| being abandoned. It seems lots of people are happy to use
| btrfs in production without that raid mode. In other
| words, for all the complaining about raid5 that happens
| every time btrfs is mentioned, you'd think there would be
| at least one person who cares enough to implement it. Yet
| people use it in production and keep improving the other
| parts of that project instead.
| Geezus_42 wrote:
| It's fine for single a single disk or something that
| presents as a single disk, like a SAN. RAID1 seems fine
| also. I really wanted to love it, but after it ate my
| data a couple of times, I gave up trying to use it for
| mass storage. At that time they didn't have warnings in
| the documentation and had stated that RAID5/6 were
| _basically_ complete.
|
| I found using mirrored vdevs in ZFS much easier to manage
| and much more stable.
| wtallis wrote:
| > I found using mirrored vdevs in ZFS much easier to
| manage and much more stable.
|
| That's not exactly a fair comparison. If you restricted
| your usage of btrfs to a similarly narrow range of
| features, you would probably have had a much better
| experience.
| creatonez wrote:
| RAID1 in Btrfs is not entirely fine. It won't nuke your
| data and there is no write-hole issue, but if a disk
| fails you'll have to go into a read-only mode during the
| rebuild and deal with various hurdles in getting it
| rebuilt.
| creatonez wrote:
| > I can't remember the last time a kernel was released
| without something at least a bit exciting about btrfs
|
| They are fixing the fixable issues, but the on-disk format
| still makes some gotchas inevitable. It sounds like there's
| never going to be a great solution to live rebuilding of
| redundancy.
| c0balt wrote:
| The main advantage, if I understood it correctly, is supposed
| to be performance. The promise is to have similar speeds to
| ext4/xfs with the feature set of btrfs/ZFS. While that sounds
| nice it took a lot of time to get it stable and upstreamed.
| Like any FS you might not want to go with the latest shiny
| thing but there are some that are willing to risk it, similar
| to debates around Btrfs vs ZFS.
|
| The last benchmarks from Phoenix are a few years old but look
| promising: https://www.phoronix.com/review/bcachefs-linux-2019
| ndsipa_pomu wrote:
| From that benchmark article
|
| > The design features of this file-system are similar to
| ZFS/Btrfs and include native encryption, snapshots,
| compression, caching, multi-device/RAID support, and more.
| But even with all of its features, it aims to offer
| XFS/EXT4-like performance, which is something that can't
| generally be said for Btrfs.
|
| I was surprised at that as I believed that btrfs is generally
| faster than ext4. Looking ahead to the last page, the
| geometric mean or the benchmarks supports that view too.
| BenjiWiebe wrote:
| The geometric mean on the last page shows ext4 to be faster
| than btrfs.
| ndsipa_pomu wrote:
| Oops - I was thinking that smaller was better.
| 2OEH8eoCRo0 wrote:
| > It feels like it's feature set is equivalent to ZFS.
|
| It does something that ZFS can't- be merged into the kernel.
| linsomniac wrote:
| Working deduplication would be amazing! ZFS has deduplication,
| but every time I've tried it has ended in a world of pain.
| Maybe they've fixed it in a more recent release, but the amount
| of RAM required for deduplication always outstripped the amount
| of RAM I had available to give it (the deduplication tables
| have to reside in RAM).
| ptman wrote:
| ZFS now has reflink support, which doesn't require lots of
| RAM, but isn't done automatically while writing. You need to
| run something like https://github.com/markfasheh/duperemove
| mastax wrote:
| The deduplication tables can be put in a special vdev now, I
| think.
| kevincox wrote:
| IMHO bcachefs has important advantages over ZFS. It is far more
| flexible. ZFS is really similar to traditional block-based
| RAID. You can get pretty flexible configurations but 1. They
| are largely fixed after creation and 2. They only operate at
| "dataset" level granularity.
|
| bachefs has a really flexible design here where you basically
| add all of your disks to the storage pool and then you can pick
| redundancy and performance settings per folder (arbitrary
| subtrees, not just datasets decided at setup time) or even
| file. For example you can configure a default of 2 replicas for
| all data, but for your cache directory set it to 1 replica. If
| you have an important documents folder you can set that to 3
| replicas, or 4.2 erasure coding.
|
| Similarly you can tell it to put your cache folder on devices
| labeled "ssd" but your documents folder should write to "ssd"
| but then be migrated to "hdd" when they are cold.
|
| And again, all of this can be set at any time on any subtree.
| Not just when you initially set up your disks or create the
| directories.
| soupdiver wrote:
| that actually sounds quite neat
| FullyFunctional wrote:
| The #1 point of ZFS is protections again bitrot, ie.
| checksums on all data. Does bcachefs do this?
| lizknope wrote:
| https://bcachefs.org/
|
| It's literally the second item listed on the main web site
|
| Full data and metadata checksumming
| Tuna-Fish wrote:
| bcachefs currently has full metadata and data checksumming,
| but there is no scrub implementation. This will likely be
| in the works soon now that it has been merged.
| jl6 wrote:
| If you mix a small SSD and a large HDD into a single bcachefs
| pool, and set it to 2 replicas, do writes have to succeed on
| both devices before returning success? I.e. is performance
| constrained by the slowest device? And what happens when the
| small SSD fills up? Does it carry on writing to the HDD but
| with only one replica, or does it report no space left?
| wtallis wrote:
| Are you trying to ask if bcachefs lets you set up stupid
| configurations? If you only have two devices and you
| configure it use replication to store two copies of all
| your data, then you are unavoidably constrained by the
| capacity of the smaller device. Whether the smaller device
| is a hard drive or SSD is irrelevant, because neither copy
| can be regarded as a discardable cache when they're both
| necessary to maintain the requested level of redundancy.
| jl6 wrote:
| I am trying to understand bcachefs by asking about an
| edge case that might be illustrative.
|
| > you are unavoidably constrained by the capacity of the
| smaller device
|
| Sure, so what does bcachefs actually do about it? ENOSPC?
|
| Answers to the question on synchronous write behavior
| also welcome.
| thomastjeffery wrote:
| You explicitly configure the cache, as laid out here:
| https://bcachefs.org/Caching/
|
| In the common case that you mentioned, data present on the
| full SSD would be overwritten "in standard LRU fashion";
| meaning the "Least Recently Used" data would no longer be
| cached. New data would be written to the SSD while a
| background "rebalance thread" would copy that data to the
| HDD. I assume that the "sync" command would wait for the
| "rebalance thread" to finish, though I will admit my own
| ignorance on that front.
| lproven wrote:
| I tried to explain in this piece:
|
| https://www.theregister.com/2022/03/18/bcachefs/
| _joel wrote:
| Brilliant, I played with this for Ceph OSD's waaaaay back and it
| worked quite well, albeit a little fragile to deploy.
| throw0101c wrote:
| One pet peeve is with some of nomenclature/UI that they are using
| (SS2.5):
|
| > _Snapshots are writeable and may be snapshotted again, creating
| a tree of snapshots._
|
| * https://bcachefs.org/bcachefs-principles-of-operation.pdf
|
| > _bcachefs provides btrfs style writeable snapshots, at
| subvolume granularity._
|
| * https://bcachefs.org/Snapshots/
|
| Every other implementation of the concept has the implicit idea
| that snapshots are read-only:
|
| * https://en.wikipedia.org/wiki/Snapshot_(computer_storage)
|
| The word "clone" seems to have been settled on for a read-write
| copy of things.
| pm215 wrote:
| I'm not sure that's completely standard terminology. For
| example LVM has "snapshots" that are read-write.
| tw04 wrote:
| It is basically universal in the enterprise storage world.
| throw0101c wrote:
| From _lvcreate(1)_ on an Ubuntu 22.04 LTS system I have CLI
| on: -s|--snapshot
| Create a snapshot. Snapshots provide a "frozen image" of an
| ori- gin LV. The snapshot LV can be used,
| e.g. for backups, while the origin LV
| continues to be used. This option can create a
| [...]
|
| * Also: https://manpages.ubuntu.com/manpages/lunar/en/man8/lv
| create....
|
| The word 'frozen' to me means unmoving / fixed.
|
| As mentioned in the _Wikipedia_ article, the analogy comes
| from photography where a picture / snap(shot) is a moment
| frozen in time.
|
| I've admined NetApps in the past, used Veritas VxFS back in
| the day, and currently run a lot of ZFS (first using it on
| Solaris 10), and "snapshot" has meant read-only for the past
| few decades whenever I've run across it.
| pm215 wrote:
| The LVM howto https://tldp.org/HOWTO/LVM-
| HOWTO/snapshotintro.html says "In LVM2, snapshots are
| read/write by default". And RedHat's docs
| https://access.redhat.com/documentation/en-
| us/red_hat_enterp... include text "Since the snapshot is
| read/write".
|
| So I think LVM2 snapshots are indeed read/write. Perhaps
| that manpage sentence was not updated since LVM1 read-only
| snapshots ?
|
| (I agree with you that 'snapshot' to me strongly suggests
| read/write; I'm just saying that you can't actually rely on
| that assumption because it's not just bcachefs that doesn't
| use that meaning.)
| yrro wrote:
| The man page continues:
|
| > This option can create a COW (copy on write) snapshot,
| or a thin snapshot (in a thin pool.) [...] COW snapshots
| are created when a size is specified. The size is
| allocated from space in the VG, and is the amount of
| space that can be used for saving COW blocks as writes
| occur to the origin or snapshot.
|
| Likely snapshots _were_ originally read-only, and the
| description of creating thin and COW snapshots was added
| later, but the man page text was not re-written
| completely; rather the description of thin and COW
| snapshots were added to the end of the existing text.
| brnt wrote:
| Id call a RW snapshot a "fork".
| treesciencebot wrote:
| Previous discussion:
| https://news.ycombinator.com/item?id=38071842
| sekao wrote:
| As I said in the last thread, the author could use support:
| https://www.patreon.com/join/bcachefs
|
| This has been a solitary and largely self-financed effort by Kent
| over many years. He must feel pretty great to finally see this
| happen!
| KMag wrote:
| When I became a supporter yesterday, the average was under $10
| per supporter per month. There's no shame in being a small
| supporter.
|
| Edit: I currently see 279 supporters for a total of $2,328 per
| month, so $8.34 average per month per supporter.
| KennyBlanken wrote:
| It's currently 428 @ $2382, $5.5/per.
|
| The pre-populated options are $20 and $100 a month, which is
| a _lot_. I think he 'd get a lot more supporters if he
| dropped those asks to more like $1 and $5.
| KMag wrote:
| I should have said "paid members" instead of "supporters"
| to be more precise. He's currently at 432 members, but of
| those only 283 are paid members.
|
| Either way, $5 is probably both the median and the mode for
| the payment distribution, whether you include non-paying
| members or not.
| tpetry wrote:
| Its not like that more people will pay with $1. most
| probably the same amount will pay but he'll just get only
| 10% of the funding.
| jbverschoor wrote:
| From the FAQ https://bcachefs.org/FAQ/ :
|
| Bcachefs is safer to use than btrfs and is also shown to
| outperform zfs in terms of speed and reliability
|
| So what makes it more reliable? I can't find a simple overview of
| the design / reasoning behind the whole thing and what makes it
| 'better' than the rest.
| dralley wrote:
| Well, keep in mind that language has remained unchanged for a
| few years now. Certainly BcacheFS has some theoretical
| advantages (lack of write hole etc.) but BTRFS has nonetheless
| improved since then and BcacheFS has gotten more complex.
|
| It's still a very exciting filesystem so I'm sure we'll be
| seeing third parties test it rigorously very soon.
| kevincox wrote:
| > Bcachefs is safer to use than btrfs
|
| Citation needed. With a sample size of 1 it ate my data and
| BTRFS has been running perfectly fine on that system (after I
| bailed off of bcachefs) and other systems. I think it is great
| that they consider data safety very important but it will take
| lots of testing and real-world experience to validate that
| claim.
| __turbobrew__ wrote:
| >Bcachefs is safer to use than btrfs
|
| That is a pretty bold claim given that Facebook runs btrfs in
| prod across the majority of their fleet and almost nobody uses
| bcachefs.
| MrDrMcCoy wrote:
| Could just mean that RAID5/6 isn't broken...
| KennyBlanken wrote:
| Facebook et al generally structure their systems to tolerate
| node failures through redundancy at higher levels. In short:
| they can not care, or work around, design problems most
| others can't - or use it for a specific purpose, such as
| logging, etc.
|
| Btrfs was terrible in the early days, took ages to "git gud",
| and given a filesystem is supposed to be among the most
| stable code in the OS, that burned a lot of bridges. It
| wasn't until fairly recently that btrfs could tolerate being
| completely filled.
|
| I have no idea how valid the claims are, but bcache's
| developer claims that btrfs suffers from a lot of terrible
| early design decisions that can't be undone.
|
| The show-stopper for me is that bcachefs lacks a scrub:
|
| > We're still missing scrub support. Scrub's job will be to
| walk all data in the filesystem and verify checksums,
| recovery bad data from a good copy if it exists or notifying
| the user if data is unrecoverable.
|
| The only argument I see for btrfs is that it supports
| throwing random drives into a pool and btrfs magically
| handles redundancy across them.
|
| ...but it doesn't support tiered storage like bcachefs does.
| We're well past "I want to have a redundant filesystem I can
| randomly add a drive to and magically my shit is mirrored."
| These days people want to have a large pile of spinning rust
| with some SSD in front of it, and doing that in all but ZFS
| is kind of a pain.
| devit wrote:
| Can't you do most of what scrub would do with something
| like: find . -xdev -print0|xargs -0 cat >
| /dev/null
|
| (ideally replacing cat with something that continues after
| errors)
| stonogo wrote:
| Citing the origin of "move fast and break things" as the
| exemplar of safety is itself pretty bold.
| Tuna-Fish wrote:
| The bcachefs architecture overview is here:
| https://bcachefs.org/Architecture/
|
| The claim for reliability comes from the idea that bcache has
| been in heavy production use for a decade, and considered rock
| solid with plenty of testing of corner cases, and that bcachefs
| builds a filesystem over the bcache block store so that most of
| the hard things (like locking and such) are managed by the
| underlying block store, not the bcachefs layer. This way, the
| filesystem code itself is simple and easy to understand.
| contr-error wrote:
| So not Bca chefs!
| withinboredom wrote:
| I can't unsee this. Thanks!
| seanw444 wrote:
| Finally! I've been waiting to give this a try. I thought about
| adding it manually, but didn't realize it required patching and
| recompiling my kernel, which wasn't a worthwhile endeavour to me.
| ksec wrote:
| Am I correct to assume this is GPL and not BSD or MIT? If this is
| GPL I guess there is no chance this will ship with BSD. And ZFS
| is sadly ( but not deal breaking ) CDDL.
| rollcat wrote:
| Which BSD? ;) DragonFly has HAMMER2, and the CDDL doesn't seem
| to be negatively impacting the development and integration of
| ZFS in FreeBSD.
|
| I too am not a big fan of the GPL, especially if the text of
| the license is longer than the program itself. But any
| filesystem (let alone a modern one) is very much a non-trivial
| feat of engineering; the author should have the full right to
| protect their (and their users') interests.
| ksec wrote:
| I am just thinking if we could have one decent FS across both
| BSD and Linux. And it would still be better than CDDL.
| rollcat wrote:
| If you just want to exchange data on removable media, use
| ExFAT. What problem are you trying to solve?
|
| Different OS's prefer different filesystems, because
| filesystems tend to be both complicated _and_ heavily
| opinionated in design and implementation - just like
| different OS 's. Linux is the odd one by supporting several
| dozen, all the other OS's stick to 1 or 2 (usually "old"
| and "new" like HFS+/APFS, FAT/NTFS, etc), plus UDF&FAT as
| the lowest common denominator for data interchange. There
| is very little precedent / use cases for sharing volumes
| like you suggest: non-removable disks tend to stay in one
| machine for their lifetime; dual-booting is extremely niche
| (where Linux/BSD themselves are all niche) and mostly a
| domain of enthusiasts.
| nolist_policy wrote:
| BSD and GPL is compatible thought, you just have to ship the
| resulting work under GPL ;)
| yjftsjthsd-h wrote:
| Thus making it just as bad as CDDL+GPL; you can do it out of
| tree, but it'll never get mainlined.
| olavgg wrote:
| I love storage and filesystems, and I am really looking forward
| to play with bcachefs. Now bcachefs can be tested easily by
| millions and build a solid reputation as a true next generation
| Linux filesystem.
| shmerl wrote:
| Nice and congrats!
|
| How stable it to use for day to day desktop tasks?
| carlhjerpe wrote:
| It's surprising no one has mentioned the fact that ZFS doesn't
| support suspend/resume. So it's a fat no-go for laptops, whereas
| btrfs and hopefully bcachefs can shine bright supporting all cool
| features on my laptop so I can learn by playing with them.
|
| (LVM + LUKS + BTRFS does it for me right now)
___________________________________________________________________
(page generated 2023-10-31 23:02 UTC)