Post B5RqXUfGSRpUwxwsEa by azonenberg@ioc.exchange
(DIR) More posts by azonenberg@ioc.exchange
(DIR) Post #B5RpaHxQ3k6kSDD2My by azonenberg@ioc.exchange
0 likes, 0 repeats
Progress on my latest round of craziness in the lab having problems that are Very Computer.To recap, the sequence of events was:1) Swap 100G NICs to different PCIe slots in storage cluster nodes 3 and 2 without incident2) Add 100G NIC to storage cluster node 1 (the current active metadata server)3) Office, soldering bench, and microscope bench workstations hang with a kernel stack trace from a null deref in the Ceph client driver as soon as node 1 drops offline.4) After hard power cycling, office workstation (2x Xeon 6144 in Supermicro X11DAi-N) comes up without issue.Soldering bench (Xeon 8362 in Supermicro X12SPi-TF) and microscope bench (Xeon 5320 in Supermicro X12SPi-TF) fail to boot. Soldering bench machine hangs during POST initializing the USB controller while microscope bench machine POSTs and can get into setup or boot menu, but is unable to launch a bootloader either via removable media or internal SSD.5) Lots of head scratching and confusion as to how a network filesystem driver segfault could make a machine not POSTAfter trying everything from removing DIMMs to re-torquing the CPU I pretty much gave up on it being a hardware problem and, at a friend's suggestion, reflashed the BIOS on the microscope bench machine via the BMC (updating from v2.3 to 2.5)It recovered and now boots, I'm posting from it. Fingers crossed the other machine can be saved the same way.
(DIR) Post #B5RpeJJgwosJpOXq9g by azonenberg@ioc.exchange
0 likes, 0 repeats
No root cause at this point; working hypothesis is that maybe before it hit the null, whatever went wrong in the Ceph driver might have also scribbled over EFI variables or SMM memory or something that put the firmware into a bad state but that the reflash cleared to a clean slateThe X12SPi-TF mainboard seems to be the common thread here, no machine running a different mobo even with the same kernel driver had the boot failure (although the X11DAi-N machine did hit the kernel crash, it came back up fine after)
(DIR) Post #B5RqXUfGSRpUwxwsEa by azonenberg@ioc.exchange
0 likes, 0 repeats
The machine hanging during POST is on BIOS 1.1 which is even older (from 2021). I had never had a reason to reflash it and while I do usually run fwupd from time to time, maybe I forgot on this machine or supermicro didnt push updates?Here's hoping that's the problem.
(DIR) Post #B5Rrh6ieROi1XDgGwK by azonenberg@ioc.exchange
0 likes, 0 repeats
And the second one is now POSTing after the BIOS reflash too.Just gotta put the GPU and rest of the RAM back in that I took out while troubleshooting.Seriously though, *how*???
(DIR) Post #B5RskWOhWgkrztXbxQ by Thunderbolt93@chaos.social
0 likes, 0 repeats
@azonenberg while you have the machines apart: did you check the CMOS battery?I've had a weird "boot device not found" issue with my Dell optiplex SFF NAS caused by a dead CMOS battery, aparrently GRUB ist held in volatile memory backed by the CMOS battery?
(DIR) Post #B5Rsz3FIC5aTvd4Jsm by azonenberg@ioc.exchange
0 likes, 0 repeats
@Thunderbolt93 That's probably the UEFI device selector not the GRUB itself.I wiped CMOS settings as step one of troubleshooting and it came up clean but still woudlnt POST
(DIR) Post #B5Rx0YgRC5i5XNjvKS by webhat@infosec.exchange
0 likes, 0 repeats
@azonenberg when I was refurbishing lots of dumpster find and secondhand computers I would often run into these issues. I discovered that there are a number of PCI(e)/ISA boards that will snoop and give me better reasons why the POST failed to help me quickly debug the issue and determine whether I should spend more time on salvage, or just recyclinghttps://en.wikipedia.org/wiki/POST_card
(DIR) Post #B5RyY4S0CmubZWX10a by dianea@lgbtqia.space
0 likes, 0 repeats
@azonenberg One of the most replaced chips on my bench is the boot flash. I guess that chip is often programmed in haste and barely made the verify when programmed, if verified at all.
(DIR) Post #B5RyY4cdZFQ26VVVaK by azonenberg@ioc.exchange
0 likes, 0 repeats
@dianea There's no way bad programming would make two machines that both reboot decently often, one with a mobo bought a month or two ago and one from 2021 but both the same model, fail simultaneously.This was definitely some kind of software/firmware issue causing a write command to corrupt... something. Code or EFI variable is anybody's guess
(DIR) Post #B5S04I7R1RfrBTq51U by azonenberg@ioc.exchange
0 likes, 0 repeats
Ok all reassembled and patched and took care of some other housekeeping plus reopened most of the wip stuff i wanted to keep active.Now it's almost 5am and I'm about where I wanted to be at 2200 last night (pcie all shuffled, 100g nic installed in last cluster node, desktops functional)
(DIR) Post #B5S0AoSfgci9EKskDI by jmax@mastodon.social
0 likes, 0 repeats
@azonenberg So, overall, win. Well done.
(DIR) Post #B5S0MkBDQmRXQUPFk8 by azonenberg@ioc.exchange
0 likes, 0 repeats
@jmax well win, except that I had planned to wrap this up around the time my wife went to bed then be up another couple hours tying up some loose ends on a project I wanted to have finished in the lab this weekend.Which only now am I ready to *start*.
(DIR) Post #B5SLILmoHY5jeLjlEu by redsakana@infosec.exchange
0 likes, 0 repeats
@azonenbergSomehow rolling a boulder over EFI vars would have been my guess too, either by somehow interacting with efivarfs or the variables being memory-mapped somewhere that got scribbled over. Super weird tho