[HN Gopher] I built a special PCIe card to test GPUs on the Pi
___________________________________________________________________
I built a special PCIe card to test GPUs on the Pi
Author : gravitate
Score : 115 points
Date : 2023-04-24 15:32 UTC (7 hours ago)
(HTM) web link (www.jeffgeerling.com)
(TXT) w3m dump (www.jeffgeerling.com)
| dvdbloc wrote:
| Sooo where can I buy one?
| geerlingguy wrote:
| Right now we aren't selling any prebuilt cards--but the
| hardware is under an OSHW license and the designs are available
| here: https://github.com/geerlingguy/pi4gpu
| b33j0r wrote:
| You can know that someone has built a brand when I didn't have to
| check that this was Jeff! However, I also saw his youtube video
| about it over a week ago.
|
| Given his progress and dedication, I fully expect a stack of
| A100's "graciously sent by _someone_" by this time in 2024 ;) To
| compare with a 32xRP2040 UHD VGA cluster in a separate post. Or
| something! This Dude is killing it in this niche.
| dcow wrote:
| Often overlooked value of open source: free labor. I bet you'd
| see a patched driver within a week if the sources were available
| to enthusiasts.
| faefox wrote:
| Overlooked? It's the foundation of the modern tech industry. :)
| Hell, we've all been brainwashed to the point where OSS
| maintainers having the audacity to ask for a modicum of support
| for their efforts is met with scorn and ridicule.
| koito17 wrote:
| > Often overlooked
|
| Why do you think corporations and everyone here push the MIT
| license so hard and in general dislike copyleft licenses, even
| weak ones like the EPL? It always boils down to "I want to use
| and modify your work for free at my workplace", though people
| will usually try to say this _very_ indirectly, usually with
| some rant on "commercialization".
| dcow wrote:
| I was referring to the reverse but wasn't clear: Nvidia would
| get better driver compatibility for free if they open-sourced
| it fully. Point was not that #corporation gets free labor by
| using open source, it's that their product would be better
| for free if open sourced.
| rektide wrote:
| I'll give $20 to the first person who gets one of these cards &
| sets up a host with Non-Transparent Bridging (NTB, effectively
| making that PCIe slot look like a device), and connects & talks
| to the Pi.
| jacquesm wrote:
| You're cheap ;)
| geerlingguy wrote:
| Supposedly the folks at Timebeat are working on a similar
| card[1], but one that appears as a PCIe device with two serial
| ports--one through to a Raspberry Pi, and one through to a
| U-blox module for timekeeping.
|
| [1] https://store.timebeat.app/products/ocp-tap-timecard-mini-
| pl...
| rektide wrote:
| Woof $1200! But for a very neat application, certainly, that
| certainly vendors would have been charging a lot more for in
| the past.
|
| And it includes the $100 LEA-M8F GPS module... And the $150
| SiTIME SiT5721 oxco over controlled oscillator. Excellent
| parts.
|
| I was going to ask if the Pi had decent ptp support to
| actually be useful speaking ethernet, but you've already
| answered that (yes),
| https://www.jeffgeerling.com/blog/2022/ptp-and-
| ieee-1588-har...
|
| I do kind of wish it had a PCIe ethernet controller onboard
| for the host too, also with ptp tied to the host. Sending
| that fine timing data across serial & PCIe bused to the host
| & to the hosts card feels unideal, like a source of some
| jitter, but maybe it's not really an issue.
| codedokode wrote:
| Interesting, I thought that to initialize a PCI card one needs to
| run Video BIOS located on the card in a x86 emulator. Is that
| wrong?
| geerlingguy wrote:
| It might be the case on some of the older cards, but most
| modern(ish) cards that don't have VGA anymore don't require
| that.
| monocasa wrote:
| There's also some cute hacks to emulate in different parts of
| the software stack for the cases where there's still value in
| running the onboard boot time firmware. For instance
| https://github.com/openSUSE/X86EmulatorPkg and https://cgit.f
| reedesktop.org/xorg/xserver/tree/hw/xfree86/in...
| [deleted]
| vxNsr wrote:
| I'm always impressed when YouTubers who mostly to me appear to be
| reviewers go down a technical rabbit hole and make something that
| requires a lot of research and technical expertise.
| krabizzwainch wrote:
| He does also run a company (some kind of web solutions
| business?) and is heavily involved in Ansible (having written
| multiple books). The youtube channel came after that stuff.
| dcow wrote:
| Did this guy start as a reviewer, or is it more that YT
| generally feels product review heavy on the software tech side
| so you assumed this guy was cut from the same stuff? (Honestly
| don't know I've only seen a few of his videos and they were all
| deeply technical.)
| michaelt wrote:
| The thumbnail is triggering people's youtube bullshit
| detectors.
|
| That big shot of the reviewer's face. The giant red arrow.
| The bright background. The breathless title with ALL CAPS.
| That's like 4/5 for clickbait videos. All it's missing is the
| face showing some ridiculously exaggerated emotion.
| iso1631 wrote:
| Trouble is clickbait works :(
|
| Veratisium did a video on it
| https://www.youtube.com/watch?v=S2xHZPH5Sng
| loudmax wrote:
| Jeff Geerling is working within the YouTube algorithm and
| he does post some clickbaity thumbnails.
|
| Don't let those thumbnails mislead you. geerlingguy is an
| active member of the open source community and he produces
| some absolute top rate instructional and informative
| videos.
| geerlingguy wrote:
| He's also here on HN ;)
|
| And regarding clickbait, I wrote a bit on that topic over
| in the Ampere Dev Community site[1], but in summary:
|
| > I decided to focus the intro bit on 'Could this be the
| next Mac Pro'? (needs to be slightly clickbaity to get
| people to actually click. All of us engineering types SAY
| that we don't like clickbait, but for 99% of us, if you
| gave us the clickbait version and the non-clickbait
| version in your YouTube feed... we'd click the clickbait
| version 100% of the time
|
| [1] https://community.amperecomputing.com/t/review-of-
| the-ampere...
| ddalex wrote:
| Dude, keep up the good work, you're an inspiration !
| zamadatix wrote:
| Clickbait is absolutely effective, I don't think anyone
| is doubting so when they say they don't like clickbait.
| It's in the same boat as ads. Few like ads but even fewer
| actually avoid ad funded content (particularly when it
| comes to sponsors) so ads are effective monetization.
| traverseda wrote:
| I've used some of his ansible stuff and was very
| surprised when he showed up in my youtube feed.
| rsync wrote:
| He has addressed this in the past ... apparently this motif
| is the only thing that works in the youtube
| environment/interface and other more sensible, or
| reasonable video thumbnails produce orders of magnitude
| less engagement.
|
| I, also, strongly dislike this (thumbnail) presentation but
| I suspect it must be necessary as it is _very_ widely
| adopted.
| r2_pilot wrote:
| Jeff Geerling is majorly involved with Ansible, so he's not
| really reviewing for the sake of reviewing. It's more that
| he's sharing his knowledge about his off-work pursuits, I
| believe.
| sureglymop wrote:
| Iirc he was also heavily involved with Linux phones like
| the Pinephone and maybe even wrote firmware for it. I think
| it's more that the guy is a professional who happens to
| also have a YouTube account, not the other way around.
| vxNsr wrote:
| Ah ok. Yea I only know him from when he appears in the
| sidebar next to other technical videos, there he's mostly
| reviewing docker setups or various self hosted options. I
| didn't realize how technical he is.
| villgax wrote:
| Apple Silicon really dropped the ball on Thunderbolt 4 with no
| support for eGPUs of any kind....
| rustybolt wrote:
| > there's no desire to either maintain a fork of their drivers or
| spend time hacking their driver to work on the buggy PCIe bus on
| the Pi
|
| That seems like the right choice. Specs exist for a reason. If
| you don't follow them, there is no reason your device should
| work. To adapt the world to work around your mistakes is
| insanity.
| cesarb wrote:
| Search for "quirk" on the Linux kernel sources, and you'll see
| plenty of working around other's mistakes. One could even argue
| that being tolerant of shitty hardware is one of the reasons
| that allowed Linux to become popular; it could run on the
| hardware people _had_ , instead of on the hardware people
| _wished they had_.
| joshspankit wrote:
| And widely known as one of the reasons Windows is so
| ubiquitous
| ChuckMcM wrote:
| Implicit assumption that the Pi's PCIe doesn't follow the spec?
| There is a lot of room between "it works in our target market"
| and "it follows the spec". I only mention that because I have
| come across a number of situations where the specifications
| related to "downsizing" the interface when insufficient
| resources are detected are often ignored when the target
| platform is invested in providing only the maximum amount of
| resources. Think USB2.0 high speed devices failing to down
| throttle in a USB 1.1 port and still work as an example.
|
| That said, there are often latent bugs in these situations that
| this sort of research/experimentation can uncover.
| Manufacturers would do well to invest the time in understanding
| _why_ it doesn 't work because those reasons can often lead to
| problems that are hard to catch when it _is_ working.
| manv1 wrote:
| FTA there seem to be at least 2 problems:
|
| "the Pi has problems with cache coherency on the PCI Express
| bus beyond 32-bits. And many (well, all nowadays) of the
| drivers expect that to function."
|
| Yeah, that's a problem.
|
| "there were issues with the BAR (Base Address Register) space
| allocated on the Pi's OS."
|
| From the GitHub link:
|
| # The default BAR address space available on the CM4 may be
| too small to allow # some devices to initialize correctly. To
| avoid 'failed to assign memory'
|
| These don't look like implementation quirks, they are just
| flat-out errors.
| simcop2387 wrote:
| I believe from previous articles/videos from Jeff that the
| BAR one is at least addressable via a custom kernel or
| device tree that sets up a large enough BAR space for those
| devices. That said I would be surprised if there's a way to
| easily fix the first problem in software aside from forcing
| constant cache flushes when dealing with MMIO registers
| which would be a huge undertaking and probably tank
| performance in ways that nobody would want since this
| should be handled by the controller on an as needed basis
| instead.
___________________________________________________________________
(page generated 2023-04-24 23:02 UTC)