Post B5RPmNfy1goCrA9SaW by azonenberg@ioc.exchange
(DIR) More posts by azonenberg@ioc.exchange
(DIR) Post #B5RPmNfy1goCrA9SaW by azonenberg@ioc.exchange
0 likes, 0 repeats
Poking around a bit more at the PCIe configuration on my storage cluster nodes and between the UEFI config and the manual I think I fully understand the topology now.(do consumer mainboards ever publish this kind of info? I'm gonna guess no... but I havent bought one in ~10 years)The CPU itself has 48 PCIe lanes, which was the most available when I deployed these (icelake-SP bumped this up to 64, and sapphire rapids to 80).They're divided into three x16 root complexes.Group 1 goes into a mux (Br3) which either connects to slot 1 as x16, or bifurcates to x4 to each of the four M.2 ports.Group 2 goes into a Broadcom PCIe switch which splits off to four slots 5:2. These slots are paired, and each pair (2/3 and 4/5) can be either dual x8s or a single x16, but all four share the single x16 to the root.Group 3 goes to a mux (Br2) which either connects to slot 7 as x16, or slots 6/7 as dual x8. It can also bifurcate to quad x4+x4+x4+x4 in slot 7 or dual x4+x4 in each of 6/7.
(DIR) Post #B5RQWDZVvQJUq8zR68 by azonenberg@ioc.exchange
0 likes, 0 repeats
If I want to have a 100G port with full performance eventually, plus as many NVMe as I can pack in there, it seems the optimum configuration is:* Group 1: 4x M.2 22110, one for boot drive, 3 for OSDs. Leave slot 1 blank* Group 2: 4x M.2 or E1.S drives on PCIe AIC adapter boards in slots 5:2. (I *may* be able to bifurcate these slots into x4 + x4 but it's unclear as I didn't see settings for this in the UEFI and I don't know if the PEX8747 will allow it. Currently assuming no.)* Group 3: 100G NIC in slot 7, leave slot 6 blankThis will let me fit a total of eight NVMe drives, boot volume + 7 OSDs, and have full non-oversubscribed gen3 x4 bandwidth from each drive to the root complex. The NIC at gen3 x16 has 128 Gbps of PCIe bandwidth available, enough to max a single 100G link but there will be little point to using the second port.
(DIR) Post #B5RQku3mkfEYfY9Sca by azonenberg@ioc.exchange
0 likes, 0 repeats
Right now each node has a single 1.92T M.2 22110 boot/mon drive, two 3.84T M.2 22110 OSD drives, and a single 7.68T E1.S OSD drive.So (once the NAND price bubble pops, that is) I will be able to fit up to another four NVMe drives into each node. Of these drives, one has to be a M.2 (to fit in the mobo M.2 socket) while the rest can be either M.2 or E1.S as the situation dictates.
(DIR) Post #B5RTazatcZPSncbYOG by gsuberland@chaos.social
0 likes, 0 repeats
@azonenberg yes, they're usually at the end of the manual.
(DIR) Post #B5SWhL7H0cGIJ7QXBI by johnlogic@sfba.social
0 likes, 0 repeats
@azonenberg Does the PCIe now attach directly to the CPU?I recall microprocessors (about 25 years ago) using one side of their square (-ish) package to interface with RAM and the other 3 sides used as HT (Hyper Transport or the other HT brand); these would link to other microprocessors or to a north bridge, which I thought handled the interfaces with PCI/PCIe and a south bridge.
(DIR) Post #B5SXP4owxN1mVdiC8m by azonenberg@ioc.exchange
0 likes, 0 repeats
@johnlogic the northbridge is now part of the CPU die (or sometimes a separate chiplet in the package) and contains the RAM controllers and some or all of the PCIe.The southbridge is a separate chip package still and has all of the slow IOs and sometimes some additional lower gen PCIe lanes like the gen2 x4 to the NIC at the very bottom of this screenshot
(DIR) Post #B5SavvxHEv9QQoxwTA by jenesuispersonne@piaille.fr
0 likes, 0 repeats
@azonenberg I have this kind of schematic for my Gigabyte motherboard, but as it's for consumers, the schematic is much simpler.
(DIR) Post #B5Sb5j7XoR25WRRDFI by azonenberg@ioc.exchange
0 likes, 0 repeats
@jenesuispersonne this is also nowhere near the entire board (and this is a several gens old platform - the CPU is from 2019 - I'm sure current is even crazier)Check out page 16 of https://www.supermicro.com/manuals/motherboard/C620/MNL-2173.pdf for the whole thing.
(DIR) Post #B5SbGzzvP7Gt3EFUum by jenesuispersonne@piaille.fr
0 likes, 0 repeats
@azonenberg Mine: https://download.gigabyte.com/FileList/Manual/mb_manual_x870e-aorus-elite-wifi7-ice_1301_e.pdf?v=ef6f52a7ab2d3d202eead1561eb2584fPage 5
(DIR) Post #B5SramW8pe6sIhlzEm by gsuberland@chaos.social
0 likes, 0 repeats
@azonenberg I actually haven't seen a motherboard manual without a block diagram in years now, consumer or otherwise.
(DIR) Post #B5Sramkfxbjh1mZatM by choofa@infosec.exchange
0 likes, 0 repeats
@gsuberland @azonenberg block diagrams are common but to what detail is sadly all over the map. things like PCIe lane configurability for consumer even HEDT platforms are left for the user and community to suss out. sites like this have spun up in response(only AM4/5 atm) https://mobomaps.com/
(DIR) Post #B5SramyV8CnLif2dRQ by azonenberg@ioc.exchange
0 likes, 0 repeats
@choofa @gsuberland yeah details of which pcie slots share bandwidth or which slots degrade other slots when active is the kind of thing that really matters for use cases like "hang as many NVMe as you can off a 100G pipe"
(DIR) Post #B5SvTcwKRVOGNyrk8W by erincandescent@erincandescent.net
0 likes, 0 repeats
@azonenberg gotta say i’m very happy right now our server is an EPYC model where the block diagram is just the CPU with dozens of lines going to it, basically none of this switch nonsense
(DIR) Post #B5SvTdA9c6Rv4rKmga by azonenberg@ioc.exchange
0 likes, 0 repeats
@erincandescent newer Intel stuff has a lot more pcie lanes too so there's less muxing etc, they went from 48 to 80+ lanes since this generation.But these cluster nodes are only like halfway through their planned ten year service life
(DIR) Post #B5Svp14fuDx7Jqy8Ey by azonenberg@ioc.exchange
0 likes, 0 repeats
@erincandescent over their lifespan so far they're gone from 24 to 48 GB of RAM, 1x 10G to 2x 10G and now to 40G (100G capable nic but i need a better switch) networking.They started with 2x 1.92T NVMe and a single... 4T? HDD and now are 2x 3.84 + 1x 7.68T NVMe and no HDD, with capacity to scale to another four SSDs per nodd
(DIR) Post #B5T6dyv8fkJgwwRuu8 by erincandescent@erincandescent.net
0 likes, 0 repeats
@azonenberg this is also about 3y into service life, not sure how long that will be. But it's a model from ~2019, bought on clearance
(DIR) Post #B5T6dzC9eTvZniPVQW by azonenberg@ioc.exchange
0 likes, 0 repeats
@erincandescent this was a 2019 -2020 vintage new build replacing my old single storage server with nfs and mdraid.Plan is to keep them in service at least until 2030