[HN Gopher] PCIe for Hackers: The Diffpair Prelude
___________________________________________________________________
PCIe for Hackers: The Diffpair Prelude
Author : zdw
Score : 111 points
Date : 2023-03-14 15:28 UTC (2 days ago)
(HTM) web link (hackaday.com)
(TXT) w3m dump (hackaday.com)
| nanumbat wrote:
| From the comments: "Protip: don't try to be fancy and provide a
| local REFCLK from an independent oscillator. Always use the
| host's REFCLK as it may be spread spectrum modulated for EMI
| compliance in most PC systems."
|
| Someone handed me a non-functional prototype system a few years
| ago (the first time I'd encountered PCIe), and it had a local
| REFCLK and an upstream Android-style host which, indeed, had
| enabled spread-spectrum clocking. Took me two miserable weeks to
| figure that one out.
| jhallenworld wrote:
| Also it's a good point to understand just how these SERDES
| blocks work. There is a CDR circuit, but it is fully dependent
| on the refclk, which can only be off a tiny amount from the one
| used on the transmit side- it's not like there's a wide-
| bandwidth PLL that recovers the embedded clock from the encoded
| data.
|
| This is unfortunate, because for transporting video you would
| like to also transport the pixel clock, but you don't get it
| for free from the SERDES CDR.
|
| DisplayPort use SERDES and have to transport the pixel clock.
| They do it by sending a message with fractional relationship
| between the pixel clock and the SERDES clock to the receive
| side, which derives the pixel clock from the recovered SERDES
| CDR clock using a fractional-N PLL.
|
| I had the idea of transporting video over PCIe at one point, so
| I was interested in this. The reason is that PCIe is sometimes
| available for free, so why not use it. I wanted to use a local
| reference clock, so would have been forced to use the
| DisplayPort scheme (but no fractional-N PLL available).
|
| SERDES with actual PLL CDRs in them for video do exist, but
| they are different from the generic SERDES used for PCIe and
| networking.
|
| [edit: removed HDMI, only DisplayPort works as I said above]
| formerly_proven wrote:
| iirc that's why only very few GPUs supported more than two
| HDMI/DVI outputs, you need a complicated clock generator for
| them as the bit clock is a multiple of the pixel clock.
| Meanwhile DisplayPort only has like four possible link
| clocks.
| dfox wrote:
| HDMI/DVI has dedicated pixel clock pair as do the various
| LVDS LCD panel interfaces (and CameraLink, which is a related
| PHY). These interfaces are more or less "dumb" and simply
| serialize the display data onto three pairs with the same
| timing as VGA/DPI (discrete serdeses for these interfaces
| tend to have internal x10 PLL for generating the bit-clock).
| jhallenworld wrote:
| The newer versions of Camera Link don't need the clock
| lane, FPD Link-II and above or Channel Link-II and above.
| samtho wrote:
| One of the most fascinating thing about working with hardware vs
| software is your designs are influenced by the laws of physics,
| but more importantly, how we choose to implement our
| understanding of these laws. I've gotten away with quick and
| dirty prototypes with no length matching but always keeping the
| two traces as close to each others' path as possible with as few
| vias as possible (even avoiding if able), but other times you
| need to get it right as I suspect that some chips are more robust
| than others with their signal processing. For CAN bus and PCIe,
| you can use garbage routing and tin foil to connect your diff
| pairs, but for HDMI, you better get that right or suffer random,
| intermittent problems forever.
| DHaldane wrote:
| From the article:
|
| > any interference affects the [diff pair] signals equally - as
| signals are compared to each other to receive information, this
| means that the information received is not affected by noise
| overlaid onto both of the signals.
|
| This is not true - a common myth about differential pairs. Don't
| rely on it for your designs. More info - see page 15
| https://www.speedingedge.com/PDF-Files/DiffSigDesign.pdf
| DHaldane wrote:
| To expand - the way most relevant noise propagates on a board
| is often assumed to couple equally into both sides of the
| differential pair. It doesn't.
| twawaaay wrote:
| For anyone interested in the topic, I can highly recommend
| "Signal and Power Integrity Simplified" by Eric Bogatin.
|
| I found this to be absolutely best resource on the topic for
| somebody with no formal electronics education like me.
| DHaldane wrote:
| I like this book a lot. Has a heavy focus on physics and an
| intuitive understanding of physics, rather than just giving a
| list of heuristics to abide by.
| twawaaay wrote:
| Yes, that's exactly the point. Heuristics is good until it is
| not. And any heuristics will eventually fail at high enough
| frequency...
|
| With intuitive understanding you start seeing how various
| design choices affect the performance and you can focus on
| checking what might be the cause of the problem and fixing
| things that have biggest potential for improvement. It allows
| you to learn from your mistakes whereas with heuristics you
| feel like a dog that is being beaten by the owner but without
| any idea why.
| gaudat wrote:
| How do I get into the layers between differential pairs and
| drivers running on my computer? I think we need at least a mid-
| end FPGA board in ordor to play with all the high speed goodness.
| The crypto crash gave us a bunch of those boards but now supply
| is drying up.
| pjc50 wrote:
| Judging from https://www.fpgadeveloper.com/list-of-fpga-dev-
| boards-for-pc..., your best bet is one that plugs into an M.2
| slot.
| zokier wrote:
| Sounds like this might be a start of a series of articles on
| HaD, hopefully they'll cover making a custom endpoint. Because
| that is also my impression that you need beefy fpga and
| possibly even proprietary ip cores to play with pcie, but it'd
| be cool to be proven wrong there
| marktangotango wrote:
| This is my understanding too. Is it possible to use an fpga
| without built in diff pairs and serdes, and use the io's that
| are built with a suitable high speed clock domain for a "diy"
| differential pair? Maybe not as fast, but still faster than
| any other io on the fpga?
| WJW wrote:
| Get yourself a mid-end FPGA before it is too late then. :)
|
| Tinkering with anything physical is always going to need
| appropriate tools, so there is probably no better way than to
| find some of the tools as cheaply as possible.
| jhallenworld wrote:
| Here is one thing I've learned about PCIe- the .1 uF series
| capacitors must be .1 uF, .01 uF will not work. The reason is
| that the link detection depends on the RC charging rate of these
| capacitors. Can you guess why I know this?
| westmeal wrote:
| I can and I am so sorry.
| izacus wrote:
| Isn't Thunderbolt also "just" PCIe over USB-C cabling? :)
|
| PCIe seems to be a well designed standard all things considered -
| very fast and very robust.
| jjoonathan wrote:
| In college I read (parts of) the big computer standards -- USB,
| PCIe, SATA (rip), and Bluetooth. PCIe was consistently head and
| shoulders above the rest.
|
| This was true on an editorial level and on a structural level.
| They defined terms before they used them, they didn't use the
| same term to mean two different things, they foresaw and
| sidestepped problems that other standards blundered into, they
| didn't go on architecture astronaut tangents, they weren't
| stingy with bits when they could buy
| simplicity/clarity/separation of concerns but they spared no
| effort when performance was on the line. These were all
| frequent sins in the other standards.
|
| I've always wondered what human factors led to this. Good
| leadership? Fewer cooks in the kitchen? More time? There is
| probably a lesson in there, but I didn't stay in computer
| engineering long enough to gain industry contacts and figure it
| out.
| AdamH12113 wrote:
| I suspect it was a combination of technical limitations and
| the nature of the developers. The USB 1.0 standard was
| finalized in 1996, seven years before PCIe 1.0 (2003), and
| was one of the first (maybe the first?) consumer protocols to
| fully support automatically-enumerated hot-swappable devices.
| (Having worked on USB drivers a bit, I still kind of hate the
| protocol, though.) Keep in mind too that there is a _massive_
| difference in computing power, memory, etc. between 1996 and
| 2003. Similarly, I get the impression that "low-power
| wireless" was a herculean effort when Bluetooth was developed
| in ~1999.
|
| It looks like PCIe 1.0 was designed solely by Intel, which
| probably accounts for a lot of the coherence and editing.
| SATA (2003) was designed by committee.
| Arrath wrote:
| This is something I would be very interested in reading in
| more detail, with examples from multiple specs pointing out
| pitfalls or successes. Dammit I'm supposed to be working not
| diving down a rabbit hole.
| jeffbee wrote:
| Thunderbolt is confusing because of the many roles it can
| adopt. When Thunderbolt is used to connect a PCIe peripheral to
| a host, it is tunneling PCIe inside packets on Thunderbolt
| lower layers. The reason this is confusing is because
| Thunderbolt can also encapsulate DisplayPort traffic, and it
| can encapsulate DisplayPort and PCIe traffic at the same time,
| but Thunderbolt ports can also just _be_ DisplayPort ports,
| electrically, but they cannot _be_ PCIe ports. If you plug a
| Thunderbolt ports into a DisplayPort display, the Thunderbolt
| interface will switch modes and simply become a DisplayPort
| interface. It has the same ability to become a USB port, if it
| thinks it is connected to a USB peer. But there is no such way
| for Thunderbolt to be connected to a PCIe device and act like
| it is just pure PCIe.
| wtallis wrote:
| Thunderbolt is quite different at the lower layers. It's more
| accurate to think of Thunderbolt as its own connection
| technology that can carry both PCIe and DisplayPort data
| packets multiplexed on one cable, with the DisplayPort packets
| getting priority.
| jjoonathan wrote:
| I've heard that Thunderbolt and USB4 "merged" -- does this
| mean it's Thunderbolt at the lowest layers and USB4 is
| encapsulated? Or are there two different signaling standards
| sharing the same PHY?
| rektide wrote:
| Don't let Usb4 hear you saying that, it'll be mad. USB4 is
| usb4. Which... very closely resembles thunderbolt.
| Thunderbolt 4 is PCIe tunneling over USB4.
|
| Note also Display Port now works similarly: instead of
| switching to an alt-mode (dedicating half the lanes to a
| video signal), in USB4 Display Port packets are
| encapsulated in usb4 packets. So now you can attach a
| variety of displays, to a tree of hubs, provided you have
| bandwidth, whereas this was a cluster fuck for a while...
| it's part of the reason usb-c output hubs were so rare, no
| one wanted to explain to consumers why a couple monitors
| with usb-c cables wouldn't work with their hub. Now, they
| will (provided there's sufficient bandwidth).
| wtallis wrote:
| When USB is involved, you can always assume they chose the
| more complicated option.
|
| USB4 is a competing, incompatible method of encapsulating
| and multiplexing PCIe, DisplayPort and USB data packets
| over one cable. But the USB4 spec incorporates Thunderbolt
| as an alternate mode. Thunderbolt support is required for
| USB4 hubs but optional for USB4 hosts and devices. So you
| will often (maybe even always, in practice) be able to use
| a USB4 host port as a Thunderbolt port.
| izacus wrote:
| It's worth mentioning that Microsoft seems to demand
| support for Thunderbolt mode for Win11 certification.
| [deleted]
| datenwolf wrote:
| > First off, you want to keep both of the pair's signals close to
| each other throughout their length. The closer the two signals
| are, the better external interference cancellation works, and the
| less noise they radiate (...)
|
| Wrong! This is a myth that's stubbornly difficult to weed out.
| The coupling between the wires of a pair is only ~12% the amount
| of coupling to the reference plane.
|
| On a PCB it's best to treat each signal on its own and route it
| as a single ended, coplanar waveguide surrounded by reference
| planes. This allows to use far thicker widths, which reduces
| inductance, which also reduces the coupling to radiating EM
| fields.
|
| If you want a longer talk on the topic:
| https://www.youtube.com/watch?v=QG0Apol-oj0
| asguy wrote:
| Rick Hartley cracks me up. He's like a PCB design cross between
| R. Lee Ermey and Tony Robbins.
| r0l1 wrote:
| The talk was great! Thanks for sharing.
| cushychicken wrote:
| Their point about diff pairs being more resistant to noise is
| correct, but it's not the primary reason for using diff pairs.
|
| Differential signals, at a physical layer level, are _faster_
| than single ended IO. It's because you have double the current
| drive /sinking capability with two drivers. You're also getting
| capacitive coupling between each leg of the pair working in your
| favor, which keeps your edge transition nice and fast.
|
| Important stuff when your unit interval is under 500ps!
| k0stas wrote:
| > Their point about diff pairs being more resistant to noise is
| correct, but it's not the primary reason for using diff pairs.
|
| The article is correct. This comment is wrong.
|
| > Differential signals, at a physical layer level, are faster
| than single ended IO. It's because you have double the current
| drive/sinking capability with two drivers.
|
| The load is also double in a diff pair compared to a single
| wire, so the net effect is a wash compared to a single wire.
|
| > You're also getting capacitive coupling between each leg of
| the pair working in your favor, which keeps your edge
| transition nice and fast.
|
| The opposite is true. Differential capacitance effectively
| appears 2X higher than the nominal capacitance to differential
| signals, making it a drawback of differential signaling rather
| than a benefit.
| CamperBob2 wrote:
| When you see a differential pair, simply imagine two
| independent lines that are each referenced to ground. (That's
| close to what happens on the PCB anyway, if a ground plane is
| present. Most of the return current ends up ground-
| referenced.) Two lines that would have exhibited 50-ohm
| characteristic impedance in a single-ended circuit will form
| a ~100-ohm diff pair.
|
| In other words, the capacitance isn't doubled, since the
| capacitance is split by the imaginary ground between the two
| lines. It looks like two caps in series, not in parallel.
| Same is true for the load resistance.
| DHaldane wrote:
| I think they're a bit off on the noise explanation, as they
| invoke a common misunderstanding of differential pairs on
| circuit boards (free common mode rejection).
|
| You're right that differential signals work much better at high
| speed than single-ended! However one major reason is that you
| can get a much higher data rate using _slower_ edges by
| comparing two signals. Diff pair signals don 't look more like
| square waves due to more drive power - they look more like
| sinusoids. Which uses way less power!
| exabrial wrote:
| Someday, I wish I could understand electrical impedance.
|
| I understand DC resistance. After designing and building my own
| Tube based electric guitar amplifier, I head to learn the
| relation between DC resistance and current _intimately_.
|
| What I still didn't have a grasp on is the concept of impedance.
| While I was able to calculate correct values for my guitar amp, I
| didn't know _why_ I was doing it which is a bit frustrating. For
| instance, I still don't understand the output transformer that
| "matches the impedance of the speaker to the output tube". Le
| sigh.
| edrxty wrote:
| I'm no expert but FWIW, I find thinking about it as a
| resistance is unhelpful. It's much closer to a sort of
| "resonance coefficient". Sure it's measured in ohms but a 50
| ohm cable will not "resist" the flow of signal like a 50 ohm
| resistor because a resistor is a load (can simulate an antenna)
| while the cable is just resonant with the load. What makes a
| given line/cable "50 ohm" is largely capacitance driven because
| it needs to be resonant with a 50 ohm load at one end and a 50
| ohm source at the other.
|
| I find amateur radio literature to be pretty good for this kind
| of stuff. It's a very central issue to them.
| meatmanek wrote:
| I think the OP is talking about input/output impedances of
| amplifiers/guitar pickups, while you're talking about
| transmission line characteristic impedance.
|
| Impedance of a transmission line is weird: in some ways, it
| acts exactly like a resistor, and in some ways it doesn't.
|
| In contrast to a resistor, an ideal transmission line doesn't
| convert electrical energy to heat. Ideally, 100% of
| electrical energy put in one end of the line will make it to
| the other end of the line intact.
|
| However, just like an ideal resistor, an ideal transmission
| line will have a real-valued impedance, not a capacitive
| (negative imaginary values) or inductive (positive imaginary
| values) impedance; nor will an ideal transmission line have
| any frequency dependence in its impedance: 50 ohms is 50
| ohms.
|
| One way I like to think of characteristic impedance is that
| it's the temporary impedance a change in signal will see
| until current/voltage wave reflections make it back from the
| other end:
|
| For a 50-ohm ideal transmission line one light-second long
| (pretending we have velocity factor 1.0 to make the math
| easy), your ohmmeter would read 50 ohms for two seconds, and
| after that it would read whatever resistance is connected to
| the other end. If the other end is an open circuit, the
| ohmmeter would measure 50 ohms then infinite resistance; if
| the other end is a short circuit, the ohmmeter would see 50
| ohms then 0 ohms; if the other end is terminated with a
| 50-ohm resistor, the ohmmeter would measure 50 ohms
| indefinitely.
|
| If you have an infinitely long ideal transmission line with
| 50 ohm impedance, and hooked up an ohmmeter to it, it would
| measure 50 ohms. If you hooked up an LCR meter, it would show
| 0 impedance and 0 capacitance.
|
| This indistinguishability between an infinitely long
| transmission line and a resistor is why a matched termination
| resistor prevents signal reflections: If you have some finite
| length of transmission line, and you attach either a matched
| resistor or an infinitely long transmission line with the
| same impedance to the end, the first length of transmission
| line cannot tell which you have attached - its behavior will
| be the same in either case: all the energy is passed into the
| next section.
| duped wrote:
| Do you mean impedance the concept (it's just resistance, in the
| frequency domain), the fact that amplifiers have an "output"
| impedance, maximum power transfer, or how a transformer works
| to convert output impedance?
|
| There's a lot of inter-layered things behind "why do tube amps
| need an output transformer." The concept of impedance is pretty
| simple and doesn't really deal with that, other than being the
| 0th layer.
| myself248 wrote:
| "Impedance" as used by audio people is completely different
| from "impedance" as used by RF people.
|
| In audio, it's about the equivalent resistance of an input or
| load. If you put a dummy resistor in place of a load, how would
| the output/driver treat it? How do you choose what value of
| resistor to use, to stand in for a given load/input? That's the
| impedance.
|
| In RF, it's about the equivalent resistance of the transmission
| line itself, because at high enough frequencies the speed-of-
| light prevents you from even seeing the far end. That
| phenomenon is well treated here:
|
| https://www.ibiblio.org/kuphaldt/electricCircuits/AC/AC_14.h...
| edrxty wrote:
| I don't believe this is true, AF and RF have significant
| overlap on the frequency domain, AF is just baseband RF when
| it's on the wire. RF very much can work with a resistive
| load, a 50ohm dummy load will ohm out as 50 ohm DC on any
| cheap multimeter.
| lightedman wrote:
| Impedance is basically the resistance of the change in current
| direction. In DC systems, you almost never deal with this. You
| deal with this constantly in AC systems and LC circuits.
|
| https://pediaa.com/difference-between-impedance-and-resistan...
| DarthNebo wrote:
| Someone figure out eGPUs for Apple Silicon, maybe a kickstarter?
| prettyStandard wrote:
| I hope to see more about this. One of the things I would like to
| see are adapters that lets you bond multiple thunderbolt cables
| to get a full 16x PCIe interface. The reverse of a bifurcator.
|
| I'd like an external GPU for a high-end laptop with no
| compromises.
___________________________________________________________________
(page generated 2023-03-16 23:01 UTC)