[HN Gopher] The Case for a High-Level Kernel-Bypass I/O Abstract...
___________________________________________________________________
The Case for a High-Level Kernel-Bypass I/O Abstraction (2019)
Author : eventhelix
Score : 50 points
Date : 2024-11-22 14:32 UTC (3 days ago)
(HTM) web link (irenezhang.net)
(TXT) w3m dump (irenezhang.net)
| FridgeSeal wrote:
| This is a super cool idea, and it's something that sounds fun to
| play with/try out.
|
| Therefore, I eagerly await the inevitable influx of:
|
| - "you don't need it"
|
| - "you're not FAANG enough to justify it",
|
| -"seems overly complicated my Python-on-Ubuntu-is-good-enough and
| who needs more"
|
| Style comments telling us why we shouldn't have fun things like
| this.
|
| Anyone got anymore comments to add to the bingo-card?
| wmf wrote:
| Preemptive cynicism is even worse than regular cynicism.
| dijksterhuis wrote:
| if you personally want to play with it, go ahead.
|
| i think my personal feeling is that those sorts of comments you
| listed come out of the woodwork more when the comments section
| starts turning into an "oh man, this _should be the standard
| for everyone_ " kind of discussion, which is never the case and
| is usually the point of those kinds of replies.
|
| at least they are when i reply with those kinds of comments
| anyway
| Gollapalli wrote:
| This is great! I think that there are a lot of latency sensitive
| applications which really do need to spare the kernel latency.
| blibble wrote:
| 7-10us for what is a hashtable set/get is really, really bad
|
| I can get a packet out to a switch and back to another machine
| and in 1-2us
| gtirloni wrote:
| Do you mean 1-2ms?
| eqvinox wrote:
| No, 1-2us is correct for that -- in a datacenter, with cut-
| through switching.
| gtirloni wrote:
| That's really impressive. I need to update myself on this
| topic. Thanks.
| mickg10 wrote:
| In reality - with decent switches at 25g - and no fec -
| node to node is reliably under 300ns (0.3 us)
| davekeck wrote:
| Out of curiosity, how is that measured across machines?
|
| (The first thing that comes to my mind would be to use an
| oscilloscope with two probes, one to each machine, but
| I'm guessing that's not it.)
| toast0 wrote:
| Measure the round trip and divide by two for the
| approximate one way time. It'd be really neat to measure
| the time it takes for a packet to travel in one
| direction, but it's somewhere between hard and
| impossible[1]; a very short path has less room to be
| asymetric though.
|
| [1] If the clocks are synchronized, you can measure send
| time on one end, and receive time on the other. But
| synchronizing clocks involves estimating the time it
| takes for signals to pass im each direction, typically
| assuming each direction takes half the round trip.
| pkhuong wrote:
| You can use something like White Rabbit
| (https://en.wikipedia.org/wiki/White_Rabbit_Project) to
| keep clocks in sync. That still involves estimates, but a
| dedicated time sync network can do things like make sure
| all the cables are the same length.
| publicmail wrote:
| What is the hardware involved in this test?
|
| Is this is with PCIe based Ethernet NICs? A single PCIe
| read probably takes at least a hundred nanos, right?
|
| Maybe I could see it with something like a busy poll
| pingpong over RDMA/Infiniband, but that seems really low
| for "traditional" networking. It's probably not possible
| to even send and receive a packet via loopback device
| that fast.
| znyboy wrote:
| Considering that 300 light-nanoseconds is about 90m,
| getting a response (or even just one-way) in that time is
| essentially running right at the limits of
| physics/causality.
| jiggawatts wrote:
| Meanwhile the best network I've ever benchmarked was AWS
| and measured about 55us for a round trip!
|
| What on earth are you using that gets you down to single
| digits!?
| crest wrote:
| I assume 1-3 hops of modern switches without congestion.
| Given 100Gb/s lanes these numbers are possible if you get
| all the bottlenecks out of the way. The moment you hit a
| deep queue the latency explodes.
| jiggawatts wrote:
| So, are you talking about _theoretical_ latencies here
| based on bandwidths and cable lengths, or actual measured
| latencies end-to-end between hosts?
|
| I know that "in principle" the physics of the cabling
| allows single digit microseconds, but I've never seen it
| anywhere near that low even with cross-over cables with
| zero switches in-path!
| eqvinox wrote:
| You need high bandwidth links (time to get the entire
| packet across starts to matter), run on bare metal (or
| have _very_ well working HW virtualisation support), and
| tune NIC parameters and OS processing appropriately. But
| it 's practically achievable.
|
| Switches in these scenarios (e.g. 25GE DC targeted) are
| pretty predictable and add <1ms (unless misconfigured)
| blibble wrote:
| that's because cloud networks are complete shit
|
| this is xilinux/mellanox cards with kernel bypass and
| cut-through switches with busy-waiting
|
| in reality, in a prod system
| Galanwe wrote:
| > the best network I've ever benchmarked was AWS and
| measured about 55us for a round trip
|
| What is "a network" here?
|
| Few infrastructures are optimised for latency, most are
| geared toward providing high throughput instead.
|
| In fact, apart from HFT, I don't think most businesses
| are all that latency sensitive. Most infrastructure
| providers will give you SLAs of high single or low double
| digits microseconds from Mahwa/Carteret to NY4, but these
| are private/dedicated links. There's little point to
| optimising latency when your network ends up on internet
| where the smallest hops are milliseconds away.
| dahfizz wrote:
| The key is that blibbe is talking about switches. Modern
| switches can process packets at line rate.
|
| If you're working in AWS, you almost certainly are
| hitting a router, which is comparably slower. Not to
| mention you are dealing with virtualized hardware, and
| you are probably sharing all the switches & routers along
| your path (if someone else's packet is ahead of yours in
| the queue, you have to wait).
| joeblubaugh wrote:
| It's really frustrating that the HotOS paper itself has no
| details about the benchmarking, and the blog post just says
| "redis benchmark". What was the system setup? Persistence
| options? What was ported to demikernel? The client writing, the
| server reading from the NIC? Based on the problem specified in
| the paper, I assume its reading from the NIC that was implemented
| in DemiOS
| r00tbeer wrote:
| See https://irenezhang.net/papers/demikernel-sosp21.pdf for a
| more thorough paper on the Demikernel from 2021. There are some
| great ideas for improving the kernel interface while still
| allowing efficient DPDK-style pipelines.
| crest wrote:
| For a such an interface to be feasible to support in common open
| source infrastructure it needs a pure software implementation for
| testing and development purposes. Even better something along the
| lines of coz to even model performance by throttling down
| everything else proportionally.
| kd913 wrote:
| What is being asked for already exists? It is called Onload.
|
| https://github.com/Xilinx-CNS/onload
| a-dub wrote:
| it is my understanding that io_uring is the generalized open
| source implementation of this, although i do not think it
| bypasses the kernel fib trie like openonload does...
| gpderetta wrote:
| Aside for onload being open source, not really. AF_XDP is the
| generalized, hardware agnostic, version of kernel bypass.
|
| In addition to bypass onload also provides a full IP/TCP user
| space stack and non-intrusive support for existing binaries
| using the standard BSD socket interface (incidentally onload
| also supports XDP now).
|
| io_uring is really for asynchronous communication with the
| kernel.
| secondcoming wrote:
| I looked at using DPDK on some of our GCP instances but it
| requires setting up a second VPC, which was one hurdle too much.
|
| I'm hoping that io_uring makes all of this unnecessary anyway.
|
| I recall reading a paper where someone noticed that for every
| packet the Linux kernel receives it has to check if any
| application has opened a raw socket. Raw sockets are initially
| needed to allow DHCP to work, so once your machine has been
| assigned an IP address you can (probably) turn this service off
| and so give the kernel less work to do. (My memory of the exact
| details may be sketchy).
| Matthias247 wrote:
| io_uring reduces the overhead of system calls - but it doesn't
| do anything to reduce the overhead of the actual networking
| stack.
|
| If your send/receive calls spend most CPU time in going through
| routing/fragmentation/filter/BPF/etc path in the networking
| stack, then uring (or other APIs which just reduce the system
| call overhead, like SendMmsg/Recvmmsg for UDP) might only make
| a small difference. Source: Lots of profiling while
| implementing QUIC libraries.
|
| An alternative to DPDK that allows to bypass the kernel
| networking stack would be AF_XDP.
| Polizeiposaune wrote:
| DHCP issues address leases, not permanent assignments; leases
| have an expiration time (and earlier suggested renewal/rebind
| times). So the DHCP client must periodically renew -- if the
| tenant doesn't renew (perhaps because the DHCP client has been
| disabled), the DHCP service may lease the address to another
| tenant.
|
| If the DHCP server hasn't moved to a new address this renewal
| can be done over unicast using the leased address - however, if
| the client doesn't receive a response from the server the
| client state machine will eventually discard the leased address
| and fall back to broadcast with an all-zeros source address
| (which is presumably what requires a raw socket).
|
| The DHCP client implementation in question likely keeps the raw
| socket open for potential future use in this case. A client
| might be able to close the raw socket and reopen it later (but
| security folks might also want it to drop the privilege
| required to reopen the raw socket, and it might be hard to have
| an ironclad guarantee that the raw socket can be reopened later
| on a machine that's short on free kernel memory..).
| secondcoming wrote:
| Not on GCP's GCE at least
___________________________________________________________________
(page generated 2024-11-25 23:01 UTC)