Post B6J7YeWzFvxeRG5jYO by linear@nya.social
(DIR) More posts by linear@nya.social
(DIR) Post #B6ICQlhp0AvuzFG8Ei by whitequark@social.treehouse.systems
0 likes, 0 repeats
oh yay, new motherboard day! my h12dsi-n6 has just been deliveredtime to work on that CI service i want to eventually offer to general public :)
(DIR) Post #B6ICQltAK00VYQZBuy by whitequark@social.treehouse.systems
0 likes, 0 repeats
many thanks to @janl for covering 50% of the motherboard price!
(DIR) Post #B6ICQm83QduuIbX57o by whitequark@social.treehouse.systems
0 likes, 1 repeats
HELL YEAH
(DIR) Post #B6ICr3iYem2WNY8jgW by whitequark@social.treehouse.systems
0 likes, 1 repeats
no BMC fuckery this time, i think the motherboard was legitimately shipped new in box to me? three different unbroken seals in various places would mean a very serious refurb
(DIR) Post #B6ICu0wNREpSdP0r8C by whitequark@social.treehouse.systems
0 likes, 1 repeats
yeah it contains a BMC firmware
(DIR) Post #B6IPIXPxXW6FlDJHRQ by whitequark@social.treehouse.systems
0 likes, 0 repeats
i like how this link in a PDF is not only not clickable, but you can't even select it (unlike the surrounding text)
(DIR) Post #B6IPIXYp0ZBmChSMFs by whitequark@social.treehouse.systems
0 likes, 0 repeats
nevermind i can't log into it with the credentials on the sticker :(
(DIR) Post #B6IPIXi2SIYsfHlica by whitequark@social.treehouse.systems
0 likes, 0 repeats
thanks to @jernej__s for letting me skip some annoying steps in getting an IPMICFG.EFI applet that lets you reset the password from the EFI shell that the board boots into by default
(DIR) Post #B6IPIXt1nRLtDMuUka by whitequark@social.treehouse.systems
0 likes, 0 repeats
success :D
(DIR) Post #B6IPIY1XHo9pdktI0m by whitequark@social.treehouse.systems
0 likes, 0 repeats
this looks so funny
(DIR) Post #B6IPIYA2mAxm48s5Gy by whitequark@social.treehouse.systems
0 likes, 0 repeats
one quarter of the cores this machine will have
(DIR) Post #B6IPLEWSFRWY1idKFc by linear@nya.social
0 likes, 0 repeats
@whitequark@social.treehouse.systems welcome to the club :)
(DIR) Post #B6ISjutodxQGGi1Q7k by jernej__s@infosec.exchange
0 likes, 0 repeats
@linear @whitequark … is that 256 threads and just 80 GB RAM?
(DIR) Post #B6ISjv59xmUqptKTo0 by linear@nya.social
0 likes, 0 repeats
@jernej__s@infosec.exchange @whitequark@social.treehouse.systems it is, and it makes sense for how i am using this machine. they are Xeon Phi 72x0 systems. they are cheap and weird and terrible unless you are doing specific types of high performance computing. the single-thread performance is that of a 10 year old intel atom, unless your workload is AVX512-heavy.i actually have four of these, they're blades in a single 2u chassis, they have 16GiB of HBM on-package. most of them are off most of the time and do not have any additional RAM. i also have one more in a workstation form factor, with 272 threads, which also has 64G added (using the onboard 16G of HBM as L4 cache instead of extra RAM, since these systems let you do that).the workloads i run (mathematical; factoring big numbers, proving their primality/compositeness,etc) i generally split it into either 1, 2, or 4 jobs per machine (matching their sub-package NUMA configuration), and the cores all share that RAM.
(DIR) Post #B6ITRiC7F0GZbgSs5I by linear@nya.social
0 likes, 0 repeats
@jernej__s@infosec.exchange @whitequark@social.treehouse.systems if i need fairly high core count and high RAM i use my PowerEdge R640 compute server instead, which has 1TiB of RAMi'm not exactly rich, so it's not 1TiB of actual DDR4, it's Intel Optane DCPMMs. at the time i got this, before the prices spiked, it was roughly a third to a quarter of the cost of using DDR4 for the whole thing. 96G or 192G of real DDR4 is in it (depending on if i need to steal it for other machines) acting as L4 cache for the Optane
(DIR) Post #B6IxPDgCiiMV0rGYOO by whitequark@social.treehouse.systems
0 likes, 0 repeats
@linear @jernej__s how good is optane? i got a pcie one to act as swap but never properly tested it
(DIR) Post #B6IxPDu1tJQ9hjjawS by linear@nya.social
0 likes, 0 repeats
@whitequark@social.treehouse.systems @jernej__s@infosec.exchange the PCIe ones i have work pretty good for swap, as well as for a ZFS ZIL device, but (at least under FreeBSD) when i had one fail, it ended up sending a type of failure message on the bus that took the kernel down with it, which really shouldn't have been necessary in that setup. i still use them for swap (if it dies while i'm using it for swap, it's fucked anyway). of... i think six total? that i've been using for this, only one died, and it did so within 72 hours of me sticking it in.the DCPMMs are a different beast entirely. as far as my server is concerned, it is just RAM. the latency is about 2 to 3 times worse than DDR4, and the throughput about 12 times worse, in the worst case i've managed to see (which isn't far from what white papers about it say). but the DDR4 being used as L4 cache means it's largely invisible to any workload that fits in the DDR4, even when running multiple workloads in parallel. it's great for being able to put a silly number of VMs on one machine and have them actually have a reasonable amount of RAM. perfect homelab material, basically.the DCPMMs can be used as weird memory-mapped permanent storage on Linux, but there's caveats because of how this interacts with Linux's page cache, and special filesystem support exists for devices like these. aiui this is incompatible with journaling and simultaneously makes unsafe poweroffs even more risky. i do not use them in this mode
(DIR) Post #B6J6WAqEZPobpCLeUa by whitequark@social.treehouse.systems
0 likes, 0 repeats
@linear @jernej__s right yeah i've read up on this before; i didn't get the DCPMMs because they were kinda pricey all in combo and i didn't want to be locked into a very specific intel platform without a clear cost advantage. i figured i could get much of the advantage of being able to have a big resident set in system memory proper without the downsides of the lock-in. maybe i've underestimated it
(DIR) Post #B6J6WBRSL1abgdlNVg by linear@nya.social
0 likes, 0 repeats
@whitequark@social.treehouse.systems @jernej__s@infosec.exchange that generation of server hardware is an awfully cheap option aftermarket right now, i just helped someone else put together a very capable base system with only 16GiB of RAM for just under $300 that could've been expanded to have 250GiB of RAM as two DCPMMs for a bit under $200 on top of that at current prices (but their use case is pretty light on RAM needs). the equivalent in DDR4 would be much more than $200 right now
(DIR) Post #B6J7YeWzFvxeRG5jYO by linear@nya.social
0 likes, 0 repeats
@whitequark@social.treehouse.systems @jernej__s@infosec.exchange specifically my server and also the one i helped someone else choose is a Dell Poweredge R640 server, Xeon Gold 62xx series processor for this. the 61xx processors will not support it, i did try (my understanding is that this is a firmware limitation mostly, they had a early and slightly buggy version of the controller that could work and supposedly some folks have gotten some activation blob to make it work, but it is nigh-mythical at this point)when i started putting this together last year i managed to get it for $22 per ~125GiB stick, which is absolutely mind-boggling when considering what RAM costs have been lately