Post B5df6qag3u5BXDPhsO by whitequark@social.treehouse.systems
(DIR) More posts by whitequark@social.treehouse.systems
(DIR) Post #B5dPfym61ikHmoU8A4 by whitequark@social.treehouse.systems
0 likes, 0 repeats
can someone explain to me why server motherboards are so fucking awful (it could also just be supermicro. i've had two server motherboards and both were from supermicro)the cheapest, most cost-reduced, aliexpress tier desktop motherboard easily outperforms both of those supermicro boards, combined, when it comes to things like "fucking booting like a normal PC"
(DIR) Post #B5dPfz6enHByoa6YD2 by grumpybozo@toad.social
0 likes, 0 repeats
@whitequark I know why *my* Supermicro servers suck, but I don’t know that I would blame the motherboards. Mine take forever to POST because they have lots of stuff to test. Especially memory. The IPMIs rely on bad antique Java and seem impossible to get usable SOL configured.
(DIR) Post #B5dPfzTLQvL9wwifZY by whitequark@social.treehouse.systems
0 likes, 0 repeats
@grumpybozo long POST times are just DRAM being DRAM. i don't mind that too much. every motherboard is gonna have that problem to some extent
(DIR) Post #B5dPfzhAbWOodpBi7c by IngaLovinde@embracing.space
0 likes, 0 repeats
@whitequark @grumpybozo idk my laptop (P14s G4 AMD) takes a few seconds to boot, with 64GB RAM. While my supermicro motherboard (A2SDi-16C-HLN4F) used to take several minutes even back when it only had 16GB installed.
(DIR) Post #B5dPg0FCYzcaLN6tAO by whitequark@social.treehouse.systems
0 likes, 0 repeats
i was going to set this up to relax and i'm genuinely considering going back to doing my taxes
(DIR) Post #B5dPg1YNhAYcP95ihU by whitequark@social.treehouse.systems
0 likes, 0 repeats
at least the tax code is rational and easy to comprehend. comparatively
(DIR) Post #B5dPg2xEUG1wkVj54i by whitequark@social.treehouse.systems
0 likes, 0 repeats
i like how half the responses to this are like "you haven't seen even half the ways in which these motherboards suck. you are like a little baby" and honestly i'm both impressed and scared
(DIR) Post #B5dPg4JxPFnmzHMk8O by whitequark@social.treehouse.systems
0 likes, 0 repeats
i feel like my next step is, once again, foregoing any remote administration capacity the manufacturer may have allegedly put into the board and just set an IP camera pointing at the screen
(DIR) Post #B5dPg5lI37GBSLA5NQ by whitequark@social.treehouse.systems
0 likes, 0 repeats
the BMC doesn't respond over the IPMI Ethernet connector at all. no traffic, not even DHCPis the fucking thing just deaddo i have to start treating it like "average dogshit embedded platform i fished out of a trash can with no documentation where i have to probe random testpoints until i see dmesg on uart"
(DIR) Post #B5dPg79mrWRvmbdACO by whitequark@social.treehouse.systems
0 likes, 0 repeats
did thatbmc has two SPI flashes. both have evidence of firmware doing a thing. didn't look furthercpu has one SPI flash. no traffic on it whatsoever.
(DIR) Post #B5dPg8YdebvG7yGWZc by whitequark@social.treehouse.systems
0 likes, 0 repeats
here's the firmware from the first flashraw SPI captures: https://upload.whitequark.org/1777074184-bmc_w25q16.txtbinary: https://upload.whitequark.org/1777074188-bmc_w25q16.binthis is the bootloader for the AST2400. you can see in strings that it initializes PCIe and NIC but, oddly, seemingly not DRAM?
(DIR) Post #B5dPgA4w01LmqQNpYG by whitequark@social.treehouse.systems
0 likes, 0 repeats
i'll get the firmware from the second flash because i want a dump of the BMC code but i don't want to desolder the flash and ruin my warrantyhowever it'll be a hot minute because i forgot that the pinout for the 16 pin flash package was made by a crackhead and i'll have to wire something up. stand by
(DIR) Post #B5dSV6QDYQsWBFkyWm by whitequark@social.treehouse.systems
0 likes, 0 repeats
okay i grabbed a capture. it seems somewhat corrupted (because there's like 30 cm of spaghetti in the way) but let's see if anything can be extracted out of it. if it turns out to be important i can always make a better capturearchive with raw reads, data, mask: https://upload.whitequark.org/1777076196-bmc_mx25l25635.tar.zst
(DIR) Post #B5dSV6iITDL95KDPhw by whitequark@social.treehouse.systems
0 likes, 0 repeats
well there's definitely u-boot therethis also explains why i have, once, seen tcpdump tell me that something knocked at it from 192.168.10.235
(DIR) Post #B5dSV6xXYXX7qbLaT2 by whitequark@social.treehouse.systems
0 likes, 0 repeats
yaaaaaaaaaaaaaaaaayyyy
(DIR) Post #B5dSV76l0GuEJBewpk by whitequark@social.treehouse.systems
0 likes, 1 repeats
despite the mild corruption the trace is good enough to recover a filesystem fromremember, i didn't read the flash. i clamped glasgow onto the SPI bus to watch traffic as it flies by, and recording everything that Linux touches, then reconstructing a file out of it
(DIR) Post #B5dSV7ISImGOtT8I4G by alina@girldick.gay
0 likes, 0 repeats
@whitequark waow,,
(DIR) Post #B5dSVB91zl6MpOkovw by whitequark@social.treehouse.systems
0 likes, 0 repeats
squints suspiciously
(DIR) Post #B5dSYENmPgCRjs6lf6 by alina@girldick.gay
0 likes, 0 repeats
@whitequark dumb question, how many layers below listening to syscalls with strace is that
(DIR) Post #B5dSztnGYWbpzYDpzM by azonenberg@ioc.exchange
0 likes, 0 repeats
@whitequark that pinout looks like "we rotated the die 90 degrees then stretched it to add more capacity" to me. entirely reasonable if you are trying to jam too much capacity into the package because your bits/mm^2 on the process is insufficient
(DIR) Post #B5dT8JNnIeOhfiJrl2 by azonenberg@ioc.exchange
0 likes, 0 repeats
@whitequark ngscopeclient's decode can do this too i've found it surprisingly useful.I really want to build a better fixture for in-system spi flash sniffing with good signal integrity though. years ago a coworker bricked a laptop by probing the spi flash and creating enough reflections it flipped an address bit during a write/erase command and corrupted the firmware partition instead of writing to the uefi log region
(DIR) Post #B5dTGLdGosgCxvOG1I by whitequark@social.treehouse.systems
0 likes, 0 repeats
@azonenberg no it's not the dieit's so you can do PCB layout where the SO8 is rotated 90 degrees and placed between the pads of the SOP16
(DIR) Post #B5dTGLs9vWabi6M9E8 by azonenberg@ioc.exchange
0 likes, 0 repeats
@whitequark i mean, the die is almost certainly rotated within the package as welli.e. the die stays put and they rotate the leadframe 90 deg around itbut the reason for the so16 footprint existing in the first place is to enable larger capacities that would not fit in a so8 leadframe
(DIR) Post #B5dTNhZ7fgJF16xNrc by whitequark@social.treehouse.systems
0 likes, 0 repeats
@azonenberg agree. Maya wanted to do this as well, an active solder-in probe on flex
(DIR) Post #B5dTNhmwqHMthzQQPg by azonenberg@ioc.exchange
0 likes, 0 repeats
@whitequark I have a passive interposer built now but it's targeting flash emulation type applications so you connect the flash to an fpga board and the fpga to the dut, dump the flash, read into on board ram, then serve spi read/write requests out of the ram tampering as you see fit.I have all the hardware just havent had time to write the RTL to make it work
(DIR) Post #B5decbJUIXExSvx0Ge by whitequark@social.treehouse.systems
0 likes, 0 repeats
there's a file called "bios_storage" which i'm not sure what it contains. EFI variables?
(DIR) Post #B5decbXfRoaCAuaKMy by whitequark@social.treehouse.systems
0 likes, 0 repeats
this could prove useful!!who needs UART on the BMC if you can just spy on its SPI bus lmfao
(DIR) Post #B5decbgstXxIdUtgjg by whitequark@social.treehouse.systems
0 likes, 0 repeats
it looks like resetting these second-hand motherboards to a default configuration doesn't totally erase all activity
(DIR) Post #B5decbp2PETf2miCRc by whitequark@social.treehouse.systems
0 likes, 0 repeats
oh, found the EFI NVM storagecan... can you mount efivarfs on a loop device? :D
(DIR) Post #B5decbxXtbHbTAgzho by whitequark@social.treehouse.systems
0 likes, 0 repeats
it always cracks me up when the OEM ships an SMBIOS full of "To Be Filled By O.E.M."like you had literally one job
(DIR) Post #B5deccCR0FC0DLesue by whitequark@social.treehouse.systems
0 likes, 0 repeats
i think i can figure out which exact parts were installed on this motherboard before it was sold, like down to the asset tag?
(DIR) Post #B5deccLITIHWepnxj6 by whitequark@social.treehouse.systems
0 likes, 0 repeats
the firmware was compiled by "ivan"
(DIR) Post #B5deccY3hqURIPm9cO by whitequark@social.treehouse.systems
0 likes, 0 repeats
i looked through the bmc firmware and it's surprisingly not atrocious. usually i immediately find something revolting, this one is just kind of aggressively mediocre
(DIR) Post #B5deccjl0LqbshFUqu by whitequark@social.treehouse.systems
0 likes, 0 repeats
fucking finallyi needed to give it DHCPv6, or it refused to talk to me. sudo dnsmasq -dq --log-dhcp --dhcp-range=::2,::100,constructor:enp0s20f0u5 did the trick ... almostdnsmasq complained to me that dnsmasq-dhcp: 9874580 cannot determine client MAC address, a message that only one person on the internet has encountered in any way before and their issue was automatically closed after 30 days of inactivityto fix it i had to do sudo sysctl -w net.ipv6.conf.enp0s20f0u5.addr_gen_mode=0
(DIR) Post #B5deccwsDaL6XNNyIS by whitequark@social.treehouse.systems
0 likes, 0 repeats
so what happens now is that it stays up for like 5 seconds and then falls off ethernet again. for a reason i do not understand at all
(DIR) Post #B5decd6Rdzzn13rcDQ by whitequark@social.treehouse.systems
0 likes, 0 repeats
ok, here's the situationfor about 3 seconds per power cycle i can interact with the BMC via IPi need to permanently reset the settings of the thing somehow. using that window. and without having to stand up and walk over to the motherboard every single time i do this, ideally
(DIR) Post #B5decdGj1mDdWwfpEu by azonenberg@ioc.exchange
0 likes, 0 repeats
@whitequark how do you get in such cursed situations lol?I have a whole fleet of supermicro boards (at least 8 off the top of my head) and almost never find it necessary to use the BMC at all. I used the web interface for the first time in 10ish years last week when i had that bios corruption issue and had to reflash through the BMC to save the board.The BMC was the only *not* broken part of the setup.
(DIR) Post #B5dech80g7clV4cvD6 by whitequark@social.treehouse.systems
0 likes, 0 repeats
anyway this is all good fun but i still don't know why the damn thing refuses to boot. let's start swapping CPUs
(DIR) Post #B5df6qag3u5BXDPhsO by whitequark@social.treehouse.systems
0 likes, 0 repeats
@azonenberg I literally just bought a motherboard
(DIR) Post #B5df6qpZAXzaHONb5E by azonenberg@ioc.exchange
0 likes, 0 repeats
@whitequark is there a name for the hacker's Midas touch because sometimes I feel like i have it too
(DIR) Post #B5dllnCUAeaOcO0PFw by whitequark@social.treehouse.systems
0 likes, 0 repeats
anyway i asked myself "why does the godforsaken thing fall off ethernet, anyway" and the obvious problem is power. it probably consumes way too much 5Vsb for the cheap Amazon desktop PSU that i got (which was never meant to power a hungry BMC)so i made this here contraption. "how do you call it?" "the aristocra... I mean the standby power supply"a brave Molex Mini-Fit Jr cable connector has been generously told to sacrifice itself. this will be forgotten in a few days unless i step on some of the plastic
(DIR) Post #B5dllnfYQZq24XbcZ6 by whitequark@social.treehouse.systems
0 likes, 0 repeats
did it work? yesdid it confirm that the BMC draws 1.2 A peak from 5Vsb? yesis that well within the standard-specified value of 3 A? yeswas there any change in behavior? no, still crashessupermicro is definitely going on my shitlist. no way i'm going to deal with this for the third time, ever again in my life
(DIR) Post #B5dllo0p9Uqt8VYbia by whitequark@social.treehouse.systems
0 likes, 0 repeats
there should be a way to reset the admin password by writing the BMC flash. so i decided to try that, backing it up firstbecause nothing is sacred today, this is an ancient Macronix device that advertises support for switching between 3 and 4 byte addressing modes... but doesn't tell you the mechanism (it implements two. at least two, i think?)you know, the usual https://github.com/GlasgowEmbedded/glasgow/pull/1156
(DIR) Post #B5dlloF0ImC7qUBvou by whitequark@social.treehouse.systems
0 likes, 1 repeats
the estimated remaining time to reprogram this 32MB flash is 20 minutes. i hope i don't have to do this more than once(instead of rewriting just the modified region, erased the entire thing then wrote the modified dump back. i was not being smart at all)
(DIR) Post #B5dua8Ti8M6mirZxM8 by RoganDawes@infosec.exchange
0 likes, 0 repeats
@azonenberg @whitequark basically the spispy approach, but using hardware to reduce latencies?
(DIR) Post #B5dx4U92LHtiW3VZmS by whitequark@social.treehouse.systems
0 likes, 0 repeats
this is a penance for my sins
(DIR) Post #B5dx4UKjdnFt6Kyv0y by whitequark@social.treehouse.systems
0 likes, 0 repeats
people under 25: this is approximately the experience that UV-eraseable PROMs gifted you with back in the day
(DIR) Post #B5dx4UVN0FlJdJxPai by whitequark@social.treehouse.systems
0 likes, 0 repeats
imagine watching one full-length episode of anime and the flash still isn't done programming
(DIR) Post #B5dx4UhmG7geFnlJvk by whitequark@social.treehouse.systems
0 likes, 0 repeats
i really should've finished my taxes instead. filling out a long list of invoices in a spreadsheet sounds so calm and peaceful in comparison to this...
(DIR) Post #B5dx4UuXUftYtNjVp2 by whitequark@social.treehouse.systems
0 likes, 0 repeats
2 minutes left. chat, will it fail verification
(DIR) Post #B5dx4V32z2hVJliJ5E by whitequark@social.treehouse.systems
0 likes, 0 repeats
oh thank heavensI: g.applet.memory.25q: verify PASS
(DIR) Post #B5dx4VBCUjDrj3WonA by whitequark@social.treehouse.systems
0 likes, 0 repeats
Freya smiles on me today. the BMC stopped crashingnow let's see if this makes the CPU start booting or if it's still not feeling like being awake at (checks clock) 5:33 in the morning
(DIR) Post #B5dx4VOJhxiMNjfIEi by whitequark@social.treehouse.systems
0 likes, 0 repeats
i don't think it's gonna bootso we're sort of back where we started, after spending eight hours tracking down a mysterious IPMI crash that had no bearing on the actual boot failure
(DIR) Post #B5dx4VZf1mmwwuyLuy by whitequark@social.treehouse.systems
0 likes, 0 repeats
also i should give a shout out to this blog post https://watchmysys.com/blog/2019/09/resetting-supermicro-x10-series-bmc-to-factory-defaults/ which was instrumental in getting the BMC to stop falling off ethernet for no discernible reason
(DIR) Post #B5dx4VnUCNqbdnROT2 by whitequark@social.treehouse.systems
0 likes, 1 repeats
i decided to check out the "Maintenance Log" sectioni take it back. the firmware was clearly written by crackheads
(DIR) Post #B5dx4cUbIztsPFeuVE by whitequark@social.treehouse.systems
0 likes, 0 repeats
i can't decide if i love or hate it that i bought one of the perhaps least embedded things that i could physically fit in my living space, and the winning strategy was still "to treat it like it's an awful embedded device put together in a manner not welcome in polite society" (i.e. with extreme disrespect) https://social.treehouse.systems/@whitequark/116461671951678722RT: https://social.treehouse.systems/users/whitequark/statuses/116461671951678722
(DIR) Post #B5dx4e8hBPZBVtQRdY by whitequark@social.treehouse.systems
0 likes, 0 repeats
well, i have no idea what to do nowi've tried three different CPUs (two models) in socket 1 with nothing in socket 2; the outcome is exactly the same: the CPU doesn't start reading code from flashmy one remaining guess is "power, again": I've measured the +12V rail at +11.8V, which isn't that low in general (the ATX 2.2 specification, contemporaneous for this motherboard, allows for +11.4V at a minimum) but also for a server platform that can draw a few dozen amps, it does seem like it might be low enough that it will complaini'll go hook it up to a car battery procure a better power supply somewhere and see if that changes anything. i guess. after that i just return the motherboard
(DIR) Post #B5dx4jO7fTsTnBgKsS by whitequark@social.treehouse.systems
0 likes, 0 repeats
to recap, the fact that there is no activity at the platform flash means:what's in the second socket basically does not matter (it should be booting starting from the first socket, far as i know)which DIMMs are plugged and where doesn't matter (the memory training blob lives in the platform flash)what the BMC is doing doesn't matter (we clearly established that by reviving the BMC into a fully functional state with no change in behavior)so what gives? is it just faulty? logs extracted from the BMC tell me it has booted in the past at least once. but maybe it stopped at one point. and i guess i should just get a refund...
(DIR) Post #B5eESD8gv1IlweFIXo by Rairii@labyrinth.zone
0 likes, 0 repeats
@whitequark "wait, it's all just awful embedded devices? always has been"
(DIR) Post #B5eKvblT2Z92RTmf1k by jernej__s@infosec.exchange
1 likes, 0 repeats
@Rairii @whitequark A server is just a lot of embedded devices in a trench coat.
(DIR) Post #B5eRJUJSzpxM0SSNaC by mcc@mastodon.social
0 likes, 0 repeats
@whitequark I want to make some kind of post like "geez, you didn't check the enp0s20f0u5 device? like that's what i would have done first" but i can't find a way to phrase it it's sufficiently obvious it's a joke
(DIR) Post #B5eRJUYM6TrkkdQGn2 by IngaLovinde@embracing.space
0 likes, 0 repeats
@mcc @whitequark I also like that these modern device names look like keysmashes or cat walking across keyboard
(DIR) Post #B5jBxNW73XWJ2IAeKu by vjon@mastodon.online
0 likes, 0 repeats
@azonenberg @whitequark I think it's called "WHY?!" (As opposed to what the normal person would say: "The board I bought was dead on arrival, I request a refund.")In my limited experience with server boards, I suspect a main CPU power issue, but that's just wild unsubstantiated guessing.