[HN Gopher] XFS metadata corruption after upgrade to 6.3.3 kernel
___________________________________________________________________
XFS metadata corruption after upgrade to 6.3.3 kernel
Author : itvision
Score : 115 points
Date : 2023-05-26 11:37 UTC (11 hours ago)
(HTM) web link (bugzilla.redhat.com)
(TXT) w3m dump (bugzilla.redhat.com)
| fbhabbed wrote:
| And this is why I was never a fan of rolling release distros
| mholt wrote:
| Apparently the 6.3 kernel made significant changes to some
| relevant APIs -- I noticed OpenZFS is not released for 6.3 yet.
| https://github.com/openzfs/zfs/issues/14622
| speed_spread wrote:
| I keep having this kinky fantasy of a GPL'd Rust ZFS re-
| implementation. Yeah, I know.
| dralley wrote:
| You'd be more likely to get one for bcachefs. Kent has
| expressed some interest in Rust although he is (correctly)
| just focusing on finishing the filesystem first.
| koverstreet wrote:
| yep, just waiting on gcc-rust support to land and then
| we'll be moving our Rust interfaces into the kernel
| dralley wrote:
| gcc-rs will take a few years, rustc_codegen_gcc is pretty
| close to being ready to go.
| speed_spread wrote:
| I'm curious about which would stabilize more quickly, an
| all new filesystem or the reimplementation of an existing
| one in a different language? The reimpl has the advantage
| of having the original to compare and test with, but the
| fresh design can use a staged approach. Yet more
| hypothetical Friday-style questions.
| koverstreet wrote:
| Incremental file by file rewrite, for sure.
| Someone wrote:
| There's a lot of unexplored stuff w.r.t. the GPL license,
| but I doubt that would legally work for a rewrite of
| CDDL-licensed ZFS to a GPL license.
| whoisthis4chan wrote:
| > So far, it looks like this will be a major undertaking.
| Whoever was responsible for the namespace support is going to
| have to rebase the entire dang thing with alternate code paths
| for everything, because as of Linux 6.3, they replaced a whole
| mess of APIs so they either newly accept struct mnt_idmap *
| instead of struct user_namespace *, or accept the former in
| addition to the previous parameters. This has far reaching
| implications, I think. Good luck to whoever attempts the task.
|
| ouch
| Volundr wrote:
| Ouch indeed. I've been wondering why it seems to be taking
| longer than usual for ZFS to build against the new kernel.
| Guess I'll be on LTS kernels for awhile yet.
|
| For once at least I feel like these changes aren't just the
| Linux devs breaking ZFS for funsies.
| rincebrain wrote:
| Support is already merged in git tip, a stable release just
| hasn't been cut with those changes backported yet.
|
| I don't think it'd be that problematic to backport, from
| what I saw, just nobody's done it.
| koverstreet wrote:
| the struct mnt_idmap stuff was badly needed, before struct
| user_namespace was being passed around for two different
| purposes - it was just asking for confusion and bugs.
| e12e wrote:
| I think the title is somewhat unnecessarily editorialized from:
|
| > Bug 2208553 - xfs metadata corruption after upgrade to 6.3.3
| kernel [NEEDINFO]
|
| There are reported problems, but also:
|
| > I note that all these reported failures seem to be using
| hardware raid and with stripe (whether either is relevant is
| unknown).
|
| > I am also using XFS (without issue) on a number of other
| systems with a 6.3 kernel, but none of them are using hardware
| raid or stripes.
|
| Ed: That said - it's quite worrying to see corruption on xfs as
| the filesystem should be really rock solid by now.
| parasense wrote:
| There is also LVM2 in the picture, but I'm not pointing any
| fingers. My gut feeling is this smells like some cache type
| thing waking up to sync/flush buffers and the pointer they
| follow lands them in the middle of some unwanted region.
|
| https://en.wikipedia.org/wiki/Dm-cache
|
| But I didn't see any evidence of caching in the BZ, so it's a
| potential redherring.
|
| The data spanning the corrupted regions was an RSA key, which
| is fine... some folks might get their kackles up, but meh...
| I'm not sure why a key file would be cached, unless it's a read
| cache, and then the normal vfs cache would be where.
| itvision wrote:
| > I note that all these reported failures seem to be using
| hardware raid and with stripe (whether either is relevant is
| unknown).
|
| That's not true. Later in the comments there is a person
| without HW raid who also has been affected.
|
| There's nothing indicating that's limited to 6.3.3 exactly. In
| fact in the same discussion kernel 6.3.1 was reported to having
| had this issue.
|
| Also, a developer chimed it and said it's probably specific to
| the entire 6.3.x series because patches for 6.3.1-6.3.4 did not
| introduce any serious XFS specific changes.
| jeffbee wrote:
| So it only affects users who care about their data? Heck of a
| take.
| e12e wrote:
| It might not be xfs - it could be the hw raid drivers.
|
| I get that the poster wanted to get the word out - but this
| (for now) only affects opt-in testers; some due diligence is
| expected.
|
| Additionally, that doesn't change the fact that hn frowns on
| editorializing titles.
| ericbarrett wrote:
| > It might not be xfs - it could be the hw raid drivers.
|
| It could be that XFS users see the corruption first because
| of its CRC-verified metadata; you'd expect ZFS to see this
| quickly too, but another commenter mentioned that OpenZFS
| isn't even out yet for 6.3. Or perhaps because XFS happens
| to put its metadata where the corruption occurs. Unclear at
| this point.
| rincebrain wrote:
| At least one person in there reported it with no HW RAID
| controller in the mix.
| itvision wrote:
| > only affects opt-in testers
|
| That's a falsehood. Kernel releases on kernel.org are _not_
| considered "unstable". That's a myth open source
| aficionados have to drop ASAP.
| ilyt wrote:
| Waht "myth" ? Most people just dont compile kernel
| directly from there but instead takes it from distro
| e12e wrote:
| I'm sorry - _in the context of rhel (ed: Fedora)_ - this
| affects opt-in testers.
|
| Tbh, I'd say anyone compiling their kernel off kernel.org
| releases, _also_ do opt-in testing in a certain sense
| (you 're more likely to end up with a concrete "sum" of
| compiled/configured kernel/user space that is unique to
| _your_ setup). Contrast this with a (stable) distro
| release, where you can at least hope what you get has
| been tested (like by the people reporting this bug).
|
| I certainly agree that any stable kernel org release is
| expected to work - but also somewhat expected to receive
| fixes for as-yet unknown bugs (hence point releases).
|
| TFA is a bug report against rhel (Ed: Fedora )- i can't
| find anything (recent) on lkml?
| mroche wrote:
| The "opt-in testers" is referring to the context of this
| Bugzilla report against Fedora. The only Fedora users who
| will have the 6.3 kernel series on their systems are
| those who willingly and intentionally enable the updates-
| testing repository. It is not a reference to the Linux
| ecosystem at large.
|
| Putting that aside... this probably should have been
| caught by upstream before releasing, though. I find it a
| bit hard to believe no kernel developer, particularly
| those working on the storage subsystems, don't have some
| kind of system using XFS within a hardware or software
| RAID configuration.
| itvision wrote:
| > The only Fedora users who will have the 6.3 kernel
| series on their systems are those who willingly and
| intentionally enable the updates-testing repository.
|
| 1. People compile released kernels found on kernel.org. I
| do. 2. Multiple distros release kernel.org kernel
| releases quite fast including Arch and Gentoo. There's
| also a Ubuntu PPA with the mainline kernel many people,
| who I personally know, use. 3. Where's the line between
| "stable" and unstable/beta/whatever? Who draws it? What
| about "stable" LTS kernel regressions which have been
| aplenty so far?
|
| Also please refer to this comment
| https://www.phoronix.com/forums/forum/phoronix/latest-
| phoron...
|
| because this argument that "LTS or distro kernels are
| stable" has never been true. It's been nothing but a very
| bad myth.
|
| > a hardware or software RAID configuration.
|
| Again, there's a person without raid in the comments.
| Could you please drop it and make it look like those
| people are somehow extremely unlucky and everyone else is
| safe?
| e12e wrote:
| > this argument that "LTS or distro kernels are stable"
| has never been true. It's been nothing but a very bad
| myth.
|
| It might have some mythical aspects, but if 10 000 people
| have run a certain kernel/FS combination without issue
| for a few months - you can reasonably expect that
| particular set of binary artifacts to work well if you
| install/run them in the same way.
|
| Now, if you change the sources, adjust the configs and
| compile flags - you can be less certain everything will
| work. That doesn't mean kernel.org kernels are unstable -
| it just means change drives risk (often justifiable
| risk).
| mroche wrote:
| > 1. People compile released kernels found on kernel.org.
| I do. 2. Multiple distros release kernel.org kernel
| releases quite fast including Arch and Gentoo. There's
| also a Ubuntu PPA with the mainline kernel many people,
| who I personally know, use.
|
| To repeat myself, the is the _Red Hat Bugzilla bug
| tracker for Fedora_. While what you say is true, that's
| outside the context of what the OP and I are saying, and
| the statement in the ticket. As seen in the ticket, users
| of other distros and kernel configs have made comments
| reporting their experience with the 6.3 series, but that
| is besides the point I was making above.
|
| Anyone performing their own builds (even on Fedora) are
| more than welcome to contribute and test, but that's
| outside the bounds of the Fedora QA process.
|
| > Again, there's a person without raid in the comments.
| Could you please drop it and make it look like those
| people are somehow extremely unlucky and everyone else is
| safe?
|
| I'm not the original thread starter, but I read through
| the ticket earlier and unless I misinterpreted one of the
| comments or missed something in the wall of text that is
| RHBZ, I didn't see a RAID-less reporter.
| ofrzeta wrote:
| Do you imply that hardware RAID is safer than software RAID?
| mschuster91 wrote:
| If it's battery-backed, yes.
| mnw21cam wrote:
| It shouldn't be _safer_. However, using a battery-backed
| write cache in hardware RAID allows the OS to just dump
| the writes into the cache, get the message that they 're
| safe very quickly, and then move on to the next thing.
| So, a battery-backed write cache allows it to be _faster_
| while still being safe. You can still get the same level
| of safety without it, and you can get very close to the
| same speed by switching off all the protections (don 't
| do that), or by logging writes to a separate fast storage
| device (like a SSD or a battery-backed RAM).
| dralley wrote:
| I was under the impression that hardware RAID is generally
| considered unreliable.
| jeffbee wrote:
| Hardware RAID is now almost universal, we just call it
| "SSDs". With an SSD the relationship between the kernel's
| block device abstraction and the reality of how the blocks
| are stored has been interdicted by an opaque controller
| that decides what blocks to write, how to exploit device
| parallelism, and what mathematical codes to use for
| redundancy.
| ilyt wrote:
| Uh, no. Main purpose for RAID is device failure
| prevention, SSDs do nothing to address that as controller
| managing storage chips can still fail, or go "well, one
| of chips is dead, you're not getting your data back".
| oneplane wrote:
| Hardware RAID is pretty much a niche at this stage. And
| unreliable and unsafe in general. Using it is like having
| an open fire pit inside your home because it's nice and
| cozy, and assuming that your house not having burned down
| yet means it's a good idea.
|
| So far the only deployments where I've still seen it
| somewhat relevant is single-node Windows machines without
| virtualisation (so the balance between trusting windows
| with your block storage vs. trusting a RAID controller
| swings towards the controller), and large proprietary SANs
| where you don't really have a choice.
|
| Everything else is either local software RAID (or ZFS), or
| a cluster filesystem with block storage support for SAN
| consumption.
|
| And then there is the worst of all worlds: firmware-enabled
| software RAID like the ones you get with consumer
| mainboards. As far as I know, that only exists because of
| windows and the legacy inability to boot from anything
| except a single-disk magic method on MBR or single ESP
| partition based booting in UEFI. It's all just crutches for
| bad systems to not have to do it correctly.
| Volundr wrote:
| It's complicated. For consumer level users I'd recommend
| software RAID because knowing what your getting with
| hardware RAID is tricky. If your motherboard offers a
| built-in RAID it's almost certainly crap and your better
| off disabling it first thing. In general these aren't
| actually hardware RAIDs anyway, it's actually implemented
| in the drivers.
|
| Low end RAID cards are sometimes better, but mdraid or
| windows RAID will still probably be more reliable.
|
| It's really only expensive enterprise-grade RAID cards that
| meaningfully compete with software RAID and you'll want to
| look at the tradeoffs of each.
| ilyt wrote:
| It's generally PITA, as you need to use a piece of
| proprietary software to manage, and often have more
| limitations than mdadm when it comes to what is possible
| and migration between RAID levels.
| chasil wrote:
| Interesting that ElRepo is on 6.3.4.
|
| http://elrepo.org/tiki/kernel-ml
| LinuxBender wrote:
| "-ml" is always the latest upstream. "-lt" [1] will have the
| latest LTS kernel if that is what you were looking for.
|
| _Using the kernel-ml may expose your system to security,
| performance and /or data corruption issues._
|
| [1] - https://elrepo.org/tiki/kernel-lt
___________________________________________________________________
(page generated 2023-05-26 23:02 UTC)