Post AOQulEtnElmxVKAddY by atoponce@fosstodon.org
(DIR) More posts by atoponce@fosstodon.org
(DIR) Post #AOQosxcsk1eKWdYjC4 by atoponce@fosstodon.org
2022-10-10T17:45:44Z
0 likes, 0 repeats
Anyone notice #ZFS errors over the weekend? I had pool errors pop up on Sunday on *all* pools I manage. One of which had a hard crash and needed physical intervention to bring it back online.Curious if we had a solar burst, extra cosmic radiation, or *something* flipping bits.
(DIR) Post #AOQp2bK6W9wyFZNAwq by atoponce@fosstodon.org
2022-10-10T17:47:20Z
0 likes, 0 repeats
No data loss, because #ZFS kicks ass, but errors across arbitrary disks early Sunday morning that initiated resilvering.
(DIR) Post #AOQpiJ6idNnbuv006y by funnylookinhat@fosstodon.org
2022-10-10T17:55:02Z
0 likes, 0 repeats
@atoponce any chance all of the sets had at least one HDD from the same batch? Reminds me of that firmware bug that killed a bunch of WD SSDs at 40k hours of uptime
(DIR) Post #AOQpqNbVndWQBDFeWe by atoponce@fosstodon.org
2022-10-10T17:56:36Z
0 likes, 0 repeats
@funnylookinhat Plausible, as most of our disks are Western Digital Blacks, so I could look. But I know they're not all out of the same batch, and there are also a good mix of Toshiba and Seagate drives across these pools. Worth checking, regardless.
(DIR) Post #AOQrA8y81SWYXFon20 by schenklklopfer@chaos.social
2022-10-10T18:10:00Z
0 likes, 0 repeats
@atoponce nope, everything good here
(DIR) Post #AOQtpbfkPMXVL57TBg by bsod@soc.zom.bi
2022-10-10T18:41:05Z
0 likes, 0 repeats
@atoponce I didn't see anything round here in germany (no ecc errors reported and no data errors during the last scrub which ran this sunday)
(DIR) Post #AOQulEtnElmxVKAddY by atoponce@fosstodon.org
2022-10-10T18:51:40Z
0 likes, 0 repeats
@bsod Thanks. I'm beginning to think we had a localized occurrence, like EMF interference or a small earthquake enough to create jitter. Either way, weird.
(DIR) Post #AOQw0bBKlvWEuZGMpU by fink@chaos.social
2022-10-10T19:05:24Z
0 likes, 0 repeats
@atoponce nothing here, but one two boxes...
(DIR) Post #AOR4o324oOB56z1UW0 by adlerweb@social.adlerweb.info
2022-10-10T20:43:20Z
0 likes, 0 repeats
@atoponce Uhhh. Yes, lost two disks. Luckily enough redundancy.
(DIR) Post #AOR52TDKGaNKMlA6AS by atoponce@fosstodon.org
2022-10-10T20:46:44Z
0 likes, 0 repeats
@adlerweb Generally speaking, where are you located? United State west coast by chance?
(DIR) Post #AOR53MQgyakNc8q38a by atoponce@fosstodon.org
2022-10-10T20:46:47Z
0 likes, 0 repeats
@fink Generally speaking, where are you located? United State west coast by chance?
(DIR) Post #AOR56bPZjKHgwsrUno by adlerweb@social.adlerweb.info
2022-10-10T20:47:05Z
0 likes, 0 repeats
@atoponce Nah, Germany.
(DIR) Post #AOR6h3qvZ13NXKbWsq by fink@chaos.social
2022-10-10T21:05:04Z
0 likes, 0 repeats
@atoponce Germany ^^
(DIR) Post #AORBD6i4sLlVTv08B6 by Zzim@mastodon.social
2022-10-10T21:55:58Z
0 likes, 0 repeats
@atoponce just seen yer postYesterday there was a large gamma-ray burstCharacterised as gamma-ray burst GRB 221009A after initially named as new x-ray source Swift J1913.1+1946My pools ok, others had weird happenings
(DIR) Post #AORBVi9EQ9Zuo1Zb4y by atoponce@fosstodon.org
2022-10-10T21:59:08Z
0 likes, 0 repeats
@Zzim Do you have a link?
(DIR) Post #AORC83gOnpTsbEN9Hs by Zzim@mastodon.social
2022-10-10T22:06:16Z
0 likes, 0 repeats
@atoponce sorry, meant to have included https://www.swift.ac.uk/search/summary.php?obsid=01126853#xrt; (tell me if this doesn’t work)
(DIR) Post #AORDCNxFTUEmaFsfRI by atoponce@fosstodon.org
2022-10-10T22:18:16Z
0 likes, 0 repeats
@Zzim Works. Thanks!
(DIR) Post #AORDebK5MgjK9Z16MC by Zzim@mastodon.social
2022-10-10T22:23:21Z
0 likes, 0 repeats
@atoponce good, sorry I missed from first post. Happy you recovered
(DIR) Post #AORDjbWC4Ac0Ct49Gi by atoponce@fosstodon.org
2022-10-10T22:24:15Z
0 likes, 0 repeats
@Zzim Yeah. No worries about data consistency. I have full trust in ZFS. Just curious that all my pools reported errors early Sunday morning.
(DIR) Post #AORE64o5qtctPn9c48 by Zzim@mastodon.social
2022-10-10T22:28:16Z
0 likes, 0 repeats
@atoponce ZFS is cool, do the timings from the report coincide with yours (allowing for time zone differences) ?
(DIR) Post #AORErR74jhKWCscEGu by atoponce@fosstodon.org
2022-10-10T22:36:53Z
0 likes, 0 repeats
@Zzim I don't see date timestamps on this data. Where can I find it?
(DIR) Post #AORFnWy57qNhA46uKu by funnylookinhat@fosstodon.org
2022-10-10T22:47:23Z
0 likes, 0 repeats
@atoponce yeah, I guess I'm way quicker to suspect faulty firmware than a bug of that caliber sneaking into a project like ZFS :-)
(DIR) Post #AORG0qFIlLM8iNVoDQ by Zzim@mastodon.social
2022-10-10T22:49:48Z
0 likes, 0 repeats
@atoponce arrrr, and my battery’s about outI think it was originally publicly notified at about 13:10 UTC yesterdayI’ll see if I have a record of actual time and get back when (if) I’ve found the office charger
(DIR) Post #AORH0pESdB7vWdftwm by wholesomedonut@fosstodon.org
2022-10-10T23:00:57Z
0 likes, 0 repeats
@atoponce no angry blips on my dashboards today or this weekend; we're pretty much fine.