Post B5VGkbAalNK1AmOJA8 by mwl@io.mwl.io
(DIR) More posts by mwl@io.mwl.io
(DIR) Post #B5SdYaMnddIqg8SiGG by wollman@mastodon.social
0 likes, 1 repeats
There's something that #ZFS #OpenZFS does at mount time (I'm told it's processing deferred frees) that sometimes takes an incredibly long time (hours), and I'm still struggling to understand: why must this be done at mount? (And why mount and not pool import?) Why can't it continue to be deferred like it was during operations?Still waiting for `zfs mount -a` after an unplanned power outage, 90 minutes after restore.
(DIR) Post #B5SdYajUHHS1oV4pcm by wollman@mastodon.social
0 likes, 0 repeats
The server eventually finished booting, after about three hours.
(DIR) Post #B5SdhU8iA01RyI7FwG by mwl@io.mwl.io
0 likes, 0 repeats
@wollman that is odd, I've never seen that. Does boot -v offer any hints?
(DIR) Post #B5SjYykVaIkm1Z7edM by wollman@mastodon.social
0 likes, 0 repeats
@mwl There's never a situation in which that would be an option. The server has already rebooted at the point I'm being paged and the users want to know why their NFS isn't working.
(DIR) Post #B5So8qpgtWVjiIuJCi by mwl@io.mwl.io
0 likes, 0 repeats
@wollman ugh. you could set -v in boot.conf and capture the next dmesg?
(DIR) Post #B5SsZ1j3Pui6qLXXPc by wollman@mastodon.social
0 likes, 0 repeats
@mwl Maybe. These servers have 60+ drives in them; I'm not sure how big I would have to make the message buffer to be able to collect all of that. And I'd have to do it across all 22 servers, since I have no way of knowing a priori which one is going to have the problem, which can't happen until our June outage window anyway.
(DIR) Post #B5StIcX5PrtK1sgOkC by mwl@io.mwl.io
0 likes, 0 repeats
@wollman oh wow.tell me you don't have single VDEVs with 60 drives.
(DIR) Post #B5Suino8DcvCTthhp2 by wollman@mastodon.social
0 likes, 0 repeats
@mwl Oh god no. Our typical configuration is 28 mirrors, 4 spares. Some older machines have 7x8 RAID-Z2 but we've gotten away from that as the resilver times are too high (and some users need more than 7 virtual spindles). But most of our older machines have multipath in the mix as well, so there's 2 daN devices per physical disk. Biggest server is 102 drives: 2 enclosures with 51 drives each (51 mirrors and 6 spares total with each mirror split between the enclosures).
(DIR) Post #B5SwDCpHreuwiVJ1VY by mwl@io.mwl.io
0 likes, 0 repeats
@wollman perfectly sensible.I wonder if the multipath is confusing things? you might check "zpool history" for clues
(DIR) Post #B5SxEVNgSo0IVVZtuS by wollman@mastodon.social
0 likes, 0 repeats
@mwl If the multipath were confusing things, that would show up during pool import and not mount.
(DIR) Post #B5VGkZvJOhVNJ6Eahs by dexter@bsd.network
0 likes, 0 repeats
@mwl @wollman Or add ‘set -x’ for *really* verbose output.
(DIR) Post #B5VGkaAuShyw5TX31E by wollman@mastodon.social
0 likes, 0 repeats
@dexter @mwl I *know* what /etc/rc.d/zfs is doing: it's running `zfs mount -va`. What I don't know is what ZFS itself is doing, why it takes so long, or why it has to do it right then and there.
(DIR) Post #B5VGkaMFmX3Weeq6hU by dexter@bsd.network
0 likes, 0 repeats
@wollman @mwl Are you using any user properties?
(DIR) Post #B5VGkaUlGtrT52otxg by wollman@mastodon.social
0 likes, 0 repeats
@dexter @mwl Five or fewer per filesystem.
(DIR) Post #B5VGkaf2eg5Javd6zA by wollman@mastodon.social
0 likes, 0 repeats
@dexter @mwl (Not seeing how that would cause the same server with the same filesystems to sometimes take a minute to mount 90 filesystems and other times take three hours, but OK, if someone can tell me a plausible story I can consider a different way to store that information.)
(DIR) Post #B5VGkar5vrj4CJGjlw by dexter@bsd.network
0 likes, 0 repeats
@wollman @mwl I ask because I see few people use them. Nothing you have described so far should be a problem, but you have a problem. So consider the less-obvious things that, truly “shouldn’t be a problem”.
(DIR) Post #B5VGkb1NJdwuiC4wnQ by wollman@mastodon.social
0 likes, 0 repeats
@dexter @mwl Without some sort plausible mechanism it's hard to consider, especially for something that only shows up during recovery from an outage.
(DIR) Post #B5VGkbAalNK1AmOJA8 by mwl@io.mwl.io
0 likes, 0 repeats
@wollman @dexter The voices in my head are muttering something about "insisting on a scrub before mounting." The time seems about right?I don't listen to the voices in my head. They usually have no basis for their mutterings.
(DIR) Post #B5VGkbKWATGHfZ2EdM by wollman@mastodon.social
0 likes, 0 repeats
@mwl @dexter A scrub on one of our pools takes more than a day; it's hard to imagine any of Sun's customers BITD finding that acceptable either. We certainly do notice interrupted scrubs resuming immediately on boot, but not waiting for completion.