Post B6JiYvoeggaGLpLs9I by yannsionneau@mastodon.online
(DIR) More posts by yannsionneau@mastodon.online
(DIR) Post #B6IjqFa2MaklMrNOvw by azonenberg@ioc.exchange
0 likes, 0 repeats
Progress! I found that FIREFLY, one of my other lab machines, had the same issue but less severe.I pulled and reseated the NIC, moved to another slot, cleaned the contacts with IPA, no luck.But using setpci commands to forcibly downgrade it to gen3 x16 instead of gen4 fixed it. So it seems like due to some combination of card/CPU aging, dirt, loss, or damage somewhere in the system it can't reliably maintain a link at gen4 speeds but works fine at gen3 (and is able to comfortably saturate 40GbE).Gen3 x16 should be enough to do a single 100G link too, the only thing I really need gen4 for is *dual* 100G which i don't expect to need any time soon.Now to see if the VM server can be fixed the same way... only problem is the pcie_set_speed.sh script I'm using doesn't seem to work on xcp-ng's old kernel/bash/something.
(DIR) Post #B6IlBjMAwz9bkkhPw8 by azonenberg@ioc.exchange
0 likes, 0 repeats
Yep, VM server is now saturating 40G with iperf traffic.So the problem was the PCIe link training up to gen4 x16 then failing after a short time., flapping, and barely having any time to get any packets through.At gen3 x16 it's stable.
(DIR) Post #B6IlCbEEeit8plOq3c by penguin42@mastodon.org.uk
0 likes, 0 repeats
@azonenberg That's weird isn't it to effect multiple machines; I wonder if that was flagging AER retries, you should be able to read them under /sys/bus/pci/devices/..devicenumber/aer_dev*
(DIR) Post #B6IlVDBIHuDpkotmgi by azonenberg@ioc.exchange
0 likes, 0 repeats
I can now even saturate 40G to iperf *in a VM*. And I bet the only thing holding me back from saturating 100G is the switch port not being 100G capable.Problem solved. I'll still shut the box down tomorrow to swap in the cooling fan so I can remove this janky external NIC cooler (I also got another of the single slot GPUs since they seem to be performing well).But the immediate "VNC to my virtual desktops is a slideshow and my CI builds take >2x as long as they should" issue is fixed.Why it cropped up after this reboot is anybody's guess.
(DIR) Post #B6Ild0ieVMtbQ9DkEy by azonenberg@ioc.exchange
0 likes, 0 repeats
@penguin42 My entire fleet of... five, I think, connectx6 NICs came from the same $bigtech fleet of decommissioned servers so likely same age, subjected to similar abuse, etc.The connectx5 in the core router and the Intel X520/XL710 cards were purchased separately.
(DIR) Post #B6IlllHTH2nK95gClE by azonenberg@ioc.exchange
0 likes, 0 repeats
@penguin42 I see nothing in AER counters on Firefly (lab machine not VM server). All zero.
(DIR) Post #B6ImTIcjBKUq0bNn6m by azonenberg@ioc.exchange
0 likes, 0 repeats
also to be clear, i'm unsure from the perf counters I was looking at if the link was flapping and retraining, or if it was just having a lot of data link layer retries with CRC errors while remaining linked up stably at gen4 speeds.I wasn't about to throw a protocol analyzer on the thing, and gen4 is borderline for my 16 GHz scope anyway (I could decode it, but not see SI issues).
(DIR) Post #B6In9ATbrXhUu55AJM by azonenberg@ioc.exchange
0 likes, 0 repeats
I wonder if this also explains the 5+ minute post times with dxe pcie enumeration taking forever.Maybe there's a bios setting to forcibly downgrade the link at startup and not try to train to gen4 at all? I'll have to look
(DIR) Post #B6InIT427l5L9qYO3M by azonenberg@ioc.exchange
0 likes, 0 repeats
But hey, these nics were >$1k new and I got them for free. If i need to run a shell script every boot to downgrade the pcie speed, that seems like a fair trade.As long as they stay stable at gen3 and don't degrade any further. Time will tell.
(DIR) Post #B6InxTT5WhXSHfpI1Y by CliffsEsport@mastodon.social
0 likes, 0 repeats
@azonenberg Nice name.Tangent machine names: assuming that is ref to the TV show/movie. I like to use ship names from Red Dwarf and other shows/stories.
(DIR) Post #B6Io70zoExsGsIgGZs by azonenberg@ioc.exchange
0 likes, 0 repeats
@CliffsEsport it's not named after the show.https://en.wikipedia.org/wiki/Firefly_(key_exchange_protocol)I've run out of unclassified crypto systems to use as namesakes. My office workstation is HAVE QUICK.
(DIR) Post #B6IoE1A61PyhPspbrk by azonenberg@ioc.exchange
0 likes, 0 repeats
@CliffsEsport and the microscope bench has SKIPJACK
(DIR) Post #B6IoHJW8EWolDIi5XE by penguin42@mastodon.org.uk
0 likes, 0 repeats
@azonenberg I wonder about flashing the latest, standard Connectx6 firmware on them?
(DIR) Post #B6Iqiid7AmzipHAiA4 by azonenberg@ioc.exchange
0 likes, 0 repeats
@sen It's a beauty all right, although expensive to keep happy.Of course, I'm always wishing I had better. Know of anybody getting rid of a cheap UXR?
(DIR) Post #B6Iqr2fmXNj5IyO7xQ by azonenberg@ioc.exchange
0 likes, 0 repeats
Update: looks like the workaround will be temporary, the friend I got them from offered to replace them with new ones.He wants one of the failed ones to experiment with, the other is going to be microscope food.So in a couple weeks I should be doing a thread/blog with a ConnectX6 board/silicon level teardown.
(DIR) Post #B6JiYvoeggaGLpLs9I by yannsionneau@mastodon.online
0 likes, 0 repeats
@azonenberg If you have issues or questions related to xcp-ng don't hesitate to ask, I can relay the question internally :)As you may already know, you can also ask on the Forum which is quite active and lots of people are helping (including Vates employees).Can you share the content of the script for instance and the error message that you get?Do you run this on dom0?Can you also share a lspci -vvv ?
(DIR) Post #B6K1rxS4jCN23VZmiW by azonenberg@ioc.exchange
0 likes, 0 repeats
@yannsionneau I figured out the issues with the script, i just had to use a newer version of it. It's all good now.But yes, it runs on dom0 since it needs to manipulate the raw pcie interface of the main NIC for the host.