[HN Gopher] XDP for game programmers
___________________________________________________________________
XDP for game programmers
Author : gafferongames
Score : 32 points
Date : 2024-04-04 17:46 UTC (5 hours ago)
(HTM) web link (mas-bandwidth.com)
(TXT) w3m dump (mas-bandwidth.com)
| nicebyte wrote:
| I'm calling eBPF "kernel shaders" to confuse graphics, os and
| gpgpu people all at the same time.
| meindnoch wrote:
| I'm stealing this.
| sakras wrote:
| I've been exactly thinking about it this way for a long time.
| Actually once we're able to push computation down into our disk
| drives, I wouldn't be surprised if these "Disk Shaders" will be
| written in eBPF.
| reactordev wrote:
| not "koroutines"? I like "kernel shaders" though.
| jauntywundrkind wrote:
| Decade and a half ago there was PacketShader, which used
| usermode networking+GPUs to do packet routing. It was a
| thrilling time. I thought this kind of effort to integrate off-
| the-shelf hardware and open source software-defined-networking
| (SDN) was only going to build and amplify. We do have some
| great SDN but remains a fairly niche world that stays largely
| behind the scenes.
| https://shader.kaist.edu/packetshader/index.html
|
| I wish someone would take up this effort again. It'd be awesome
| to see VPP or someone target offload to GPUs again. It feels
| like there's a ton of optimization we could do today based
| around PCI-P2P, where the network card could DMA direct to the
| GPU and back out without having to transit main-memory/the CPU
| at all; lower latency & very efficient. It's a long leap & long
| hope, but I very much dream that CXL eventually brings us
| closer to that "disaggregated rack" model where a less host-
| based fabric starts disrupting architecture, create more deeply
| connected systems.
|
| That said, just dropping down an fpga right on the nic is
| probably/definitely a smarter move. Seems like a bunch of
| hyperscaler do this. Unclear how much traction Marvell/Nvidia
| get from BlueField being on their boxes but it's there.
| Actually using the fpga is hard of course. Xilinx/AMD have a
| track record of kicking out some open source projects that seem
| interesting but don't seem to have any follow through. Nanotube
| being an XDP offload engine seemed brilliant, like a sure win.
| https://github.com/Xilinx/nanotube and
| https://github.com/Xilinx/open-nic .
| hobs wrote:
| > Within 10 years, everybody is going to have 10gbps internet
|
| LOL, no way in America is that going to be true.
| gafferongames wrote:
| https://www.cablelabs.com/10g
| hobs wrote:
| Is this supposed to be a joke? I cant even reliably get 1G in
| most places.
| hansvm wrote:
| I can't get 0.05G reliably in most suburbs, even when I do
| the worst-case ping in a typical 10-minute window is
| 500+ms, and there are still many places in the US where the
| best internet I can find is under 0.001G.
| Animats wrote:
| I've actually written a high-performance metaverse client, one
| that can usefully pull half a gigabit per second and more from
| the Internet. So I get to see this happening. I'm looking at a
| highly detailed area right now, from the air, and traffic peaked
| around 200Mb/s. This XDP thing seems to address the wrong
| problem.
|
| Actual traffic for a metaverse is mostly bulk content download.
| Highly interactive traffic over UDP is maybe 1MB/second,
| including voice. You're mostly sending positions and orientations
| for moving objects. Latency matters for that, but an extra few
| hundred microseconds won't hurt. The rest is large file
| transfers. Those may be from totally different servers than the
| ones that talk interactive UDP. There's probably a CDN involved,
| and you're talking to caches. Latency doesn't matter that much,
| but big-block bandwidth does.
|
| Practical problems include data caps. If you go driving around a
| big metaverse, you can easily pull 200GB/hour from the asset
| servers. Don't try this on "AT&T Unlimited Extra(r) EL". Check
| your data plan.
|
| The last thing you want is game-specific code in the kernel. That
| creates a whole new attack surface.
| lisplist wrote:
| What is a metaverse client? Do you just mean a cross platform
| VR app?
| gafferongames wrote:
| > Actual traffic for a metaverse is mostly bulk content
| download. Highly interactive traffic over UDP is maybe
| 1MB/second, including voice.
|
| Typical bandwidth for multiplayer games like FPS
| (Counterstrike, Apex Legends) are around 512kbps-1mbit per-
| second down per-client, and this is old information, newer
| games almost certainly use more.
|
| It's easy to see a more high fidelity gaming experience taking
| 10mbit - 100mbit traffic from server to client, just increase
| the size and fidelity of the world. next, increase player
| counts and you can easily fill 10gbit/sec for a future FPS/MMO
| hybrid.
|
| God save us from the egress BW costs though :)
| Animats wrote:
| There are only so many pixels on the screen, as Epic points
| out. The need for bandwidth is finite.
|
| It will be interesting to see if Epic makes a streamed
| version of the Matrix Awakens demo. You can download that and
| build it. It's about 1TB after decompression. If they can
| make that work with their asset streaming system, that will
| settle the question of what you really need from the network.
| gafferongames wrote:
| You can't client side predict pixels.
| Galanwe wrote:
| This article is plain ridiculous.
|
| > Why? Because otherwise, the overhead of processing each packet
| in the kernel and passing it down to user space and back up to
| the kernel and out to the NIC limits the throughput you can
| achieve. We're talking 10gbps and above here.
|
| _Throughpout_ is not problematic at all for the Linux network
| stack, even at 100gbps. What is problematic is >10gbps line rate.
| In other words, unless you're receiving 10gbps unshaped UDP
| datagrams with no payloads at line rate, the problem is non
| existant. Considering internet is 99% fat TCP packets, this
| sentence is completely absurd.
|
| > With other kernel bypass technologies like DPDK you needed to
| install a second NIC to run your program or basically implement
| (or license) an entire TCP/IP network stack to make sure that
| everything works correctly under the hood
|
| That is just wrong on so many levels.
|
| First, DPDK allows reinjecting packets in the Linux network
| stack. That is called queue splitting,is done by the NIC, and can
| be trivially achieved using e.g. the bifurcated driver.
|
| Second, there are plenty of available performant network stacks
| out there, especially considering high end NICs implement 80% of
| the performance sensitive parts of the stack on chip.
|
| Last, kernel bypassing is made on _trusted private networks_, you
| would have to be crazy or damn well know what you're doing to
| bypass on publicly addressable networks, otherwise you will have
| a bad reality check. There are decades of security checks and
| counter measures baked in the Linux network stack that a game
| would be irresponsible to ask his players to skip.
|
| I'm not even mentioning the ridiculous latency gains to be
| achieved here. Wire tapping the packet "NIC in" to userspace
| buffer should be in the ballpark of 3us. If you think you can do
| better and this latency is too much for your application, you're
| either day dreaming or you're not working in the video game
| industry.
| gafferongames wrote:
| > _Throughpout_ is not problematic at all for the Linux network
| stack, even at 100gbps. What is problematic is >10gbps line
| rate. In other words, unless you're receiving 10gbps unshaped
| UDP small datagrams at line rate, the problem is non existant.
| Considering internet is 99% fat TCP packets, this sentence is
| completely absurd.
|
| But games are not 99% fat TCP packets.
|
| Games are typically networked with UDP small datagrams sent at
| high rates for most recent state or inputs, with custom
| protocols built on top of UDP to avoid TCP head of line
| blocking. Packet send rates per-client can often exceed 60HZ,
| especially when games tie client packet send rate to the
| display frequency, eg. Valve and Apex Legends network models.
|
| Now imagine you have many thousands of players and you can see
| that the problem does indeed exist. If not for current games,
| for future games and metaverse applications when we start to
| scale up the player counts from the typical 16, 32 or 64
| players per-server instance, and try to merge something like
| FPS techniques with the scale of MMOs, which is actively
| something I'm actually doing.
|
| XDP/eBPF is a useful set of technologies for people who develop
| multiplayer games with custom UDP based protocols. You'll see a
| lot more usage of this in the future moving forward, as player
| counts increase for multiplayer games, and other metaverse-like
| experiences.
|
| Best wishes
|
| - Glenn
| gafferongames wrote:
| > First, DPDK allows reinjecting packets in the Linux network
| stack. That is called queue splitting,is done by the NIC, and
| can be trivially achieved using e.g. the bifurcated driver.
|
| My apologies, the last time I looked at DPDK was in 2016, and
| I don't believe this was true back then. Either way, it seems
| that XDP/eBPF is much easier to use. YMMV!
| mondobe wrote:
| > Within 10 years, everybody is going to have 10gbps internet.
| How will this change how games are made? How are we going to use
| all this bandwidth? 5v5 shooters just aren't going to cut it
| anymore.
|
| I have no doubt that more bandwidth will change how games are
| made, but 5v5 shooters (and pretty much all existing multiplayer
| styles) are here to stay for a lot longer than that, in some form
| or another.
___________________________________________________________________
(page generated 2024-04-04 23:00 UTC)