Post B4BXyokqomcXcP0F2u by ldcd@social.treehouse.systems
 (DIR) More posts by ldcd@social.treehouse.systems
 (DIR) Post #B4BXynVDTQWJjcgF2O by tnt@chaos.social
       0 likes, 0 repeats
       
       How does one debug a PCIe gen 3 link SI issue (probably) without any test gear 😅   (Well without any gear that has nearly enough BW at least, I'm about 10x too slow ...)
       
 (DIR) Post #B4BXynkSYkiIUtoPnU by yannsionneau@mastodon.online
       0 likes, 0 repeats
       
       @tnt I'm guessing you already instrumented the Link Training and Status State Machine (LTSSM) at each side...
       
 (DIR) Post #B4BXynuNxqeYzgSLGi by tnt@chaos.social
       0 likes, 0 repeats
       
       @yannsionneau On one side yes, ( Xilinx FPGA ), the other side, I have more limited visibility.Basically AFAICT it's just not happy with link quality / error rate after the EQ phases and retries and eventually gives up.The path is very weird since I'm trying to get things to work over an optical link. ( Works fine at Gen2 FWIW ) and there are plenty of opportunity for signal to get screwed up.
       
 (DIR) Post #B4BXyo65GM0jZxvgVE by tnt@chaos.social
       0 likes, 0 repeats
       
       @yannsionneau I have several places where I can configure EQ/DeEmphasis and stuff like that but with no real visibility of what the signal look like at the various stages, it's kind of hard to know what to do.One of the side is a PEX8747 PCIe switch and I'm trying to get their java software to run to see if it says anything 😅
       
 (DIR) Post #B4BXyoEwjP6G1S4lJg by yannsionneau@mastodon.online
       0 likes, 0 repeats
       
       @tnt I think some piece of hw can even give you eye diagram data by poking their registers (pcie IP or DDR IP)But I can't help much I'm sorry.I guess you're gonna have to find some lab somewhere with a big scope ^^
       
 (DIR) Post #B4BXyoN6F5ccQjtH1c by tnt@chaos.social
       0 likes, 0 repeats
       
       @yannsionneau Yes, that's why I'm trying to get that software running, it can collect statistics about the eye opening. It's not exactly the same as an eye diagram but can give a clue.
       
 (DIR) Post #B4BXyoYRYuhCzvCKhs by tnt@chaos.social
       0 likes, 0 repeats
       
       @yannsionneau Well ... not very useful. And this is _NOT_ the link I'm debugging, this is the eye for the upstream port which works fine .... so doesn't seem to be working :/
       
 (DIR) Post #B4BXyokqomcXcP0F2u by ldcd@social.treehouse.systems
       0 likes, 0 repeats
       
       @tnt @yannsionneau clock recovery issue?@azonenberg
       
 (DIR) Post #B4BXyouQFCHE65Tsxs by azonenberg@ioc.exchange
       0 likes, 0 repeats
       
       @ldcd @tnt @yannsionneau Its been a while since i've dug into gen3 training but my recollection is that there is some bidirectional negotiation of equalizer parameters or something?
       
 (DIR) Post #B4BYQkAZSWixVRLixU by tnt@chaos.social
       0 likes, 0 repeats
       
       @azonenberg @ldcd @yannsionneau Yes. Although, I actually disable that 😅 Given the goal is PCIe over optics, the feedback loop of that training doesn't work, it's not meant for it.Instead of optics, I also tried a DAC cable and leave the auto-EQ on, but doesn't work either.
       
 (DIR) Post #B4BYVYgRahHnKaYO0m by azonenberg@ioc.exchange
       0 likes, 0 repeats
       
       @tnt @ldcd @yannsionneau yeah I was about to wonder if that loop was your problem. There goes that idea
       
 (DIR) Post #B4C5ZREqVie9t06QQS by tnt@chaos.social
       0 likes, 0 repeats
       
       🎉  Got the link working.Still quite many bad DLLP / TLP looking at the couters, but good enough I can actually run a DMA test and can do 4.5 ~ 5 Gbps in both direction at the same time over a 2m fiber link.Definitely more work and tweaking needed but it's good enough to convince me it's doable and invest more time/money in making a better setup. And also trying QSFP transceiver. ( My goal is 4x Gen3 over a 40G-LR4 using a single fiber pair ).
       
 (DIR) Post #B4C5ZRfQurujDSXero by azonenberg@ioc.exchange
       0 likes, 0 repeats
       
       @tnt what was the root cause so far / what did you change?
       
 (DIR) Post #B4C5pjRAcEd4xhdWrI by tnt@chaos.social
       0 likes, 0 repeats
       
       @azonenberg Tweaking various parameters.  I had some redriver in the path (mostly to trick the receiver detection since SFP don't provide a proper load) and they were set for -2dB de-emphasis, I changed that to 0 dB. And then changed the de-emphasis of the PCIe transmitter from -6 dB to -3.5 dB.