[HN Gopher] Case Study: ByteDance Uses eBPF to Enhance Networkin...
___________________________________________________________________
Case Study: ByteDance Uses eBPF to Enhance Networking Performance
Author : ChrisArchitect
Score : 166 points
Date : 2025-01-29 15:58 UTC (1 days ago)
(HTM) web link (ebpf.foundation)
(TXT) w3m dump (ebpf.foundation)
| erulabs wrote:
| I'd love to see a more complete picture of ByteDance's TikTok
| infra. They released "KubeAdmiral" (1) so I'm assuming they're
| using eBPF via a Kubernetes CNI, and I see ByteDance listed on
| Cilium's github (2). They're also using KubeRay (3) to
| orchestrate huge inference tasks. It's annoying that a company I
| definitely do not want to work for has such an incredibly
| interesting infrastructure!
|
| 1. https://github.com/kubewharf/kubeadmiral
|
| 2. https://github.com/cilium/cilium/blob/main/USERS.md
|
| 3. https://www.anyscale.com/blog/how-bytedance-scales-
| offline-i...
| koakuma-chan wrote:
| They also made monoio, an io-uring based async runtime for
| Rust: https://github.com/bytedance/monoio
| ddxv wrote:
| Here's my list of the decompiled apps tools and business SDKs
| they are using:
|
| https://appgoblin.info/apps/com.zhiliaoapp.musically/sdks
| dilyevsky wrote:
| I also heard they replace k8s etcd with a shim [0] similar to
| kine because their clusters are so large.
|
| [0] - https://github.com/kubewharf/kubebrain
| tptacek wrote:
| Netkit, which is what this is built on, is pretty neat. For
| transmitting packets from one container/VM to another, the
| conventional solution is to give each its own veth device. When
| you do that, the kernel network stack, at like the broad logic
| level, is sort of oblivious to the fact that the devices aren't
| real ethernet devices and don't have to go through the ethernet
| motions to transact.
|
| Netkit replaces that logic with a simple pairing of sending and
| receiving eBPF programs; it's an eBPF cut-through for packet-
| level networking between networks that share a host kernel. It's
| faster, and it's simpler to reason about; the netkit.c code is
| pretty easy to read straight through.
| akamaka wrote:
| Thanks for the clear explanation!
| charleslmunger wrote:
| >When you do that, the kernel network stack, at like the broad
| logic level, is sort of oblivious to the fact that the devices
| aren't real ethernet devices and don't have to go through the
| ethernet motions to transact.
|
| Is that true even for virtio-net? I guess I just assumed all
| these virtual devices worked like virtiofs and had low overhead
| fast paths for host and guest communication.
| XorNot wrote:
| Yeah this is a surprise to me too - my impression was things
| like loopback and virtio devices were used explicitly because
| they don't pretend to ever be real devices, and thus bypass
| all the real device handling.
|
| What additional overhead is cut out by the netkit approach?
| tptacek wrote:
| Are you using virtual machines? They're not.
|
| The big win here as I understand it is that it gives you
| roughly the same efficient inter-device forwarding path
| that XDP gives you: you can bounce from one interface to
| another in eBPF without converting buffers back into
| skbuffs and snaking them through the stack again.
| XorNot wrote:
| But in containers we use the "veth" devices, which aren't
| even virtio and are only ever routed routed locally on
| the Linux kernel. So my question is, if this sort of
| optimization is possible, what does it sacrifice compared
| to veth to do it, given the constraints are (apparently)
| the same?
| tptacek wrote:
| I assume the thing here is that veth simply doesn't do
| it? We're talking about a programmable fast path that
| bypasses the stack to get from interface A to interface
| B. For an ethernet interface, that's what XDP does.
| kapilvt wrote:
| This article from isovalent introducing netkit walks
| through the benefits and tradeoffs
|
| https://isovalent.com/blog/post/cilium-netkit-a-new-
| containe...
| lsnd-95 wrote:
| It would be nice to see an implementation of TCP fusion (on
| Solaris) or SIO_LOOPBACK_FASTPATH (on Windows) for Linux.
| jiveturkey wrote:
| Came here to say the same. I'm glad linux is finally catching
| up to Solaris.
| sirjaz wrote:
| Someone on HN giving kudos to Windows for once. Has hell
| frozen over.
| jigneshdarji91 wrote:
| netkit.c:
| https://codebrowser.dev/linux/linux/drivers/net/netkit.c.htm...
| preisschild wrote:
| Cilium (a Kubernetes CNI) can use netkit instead of veth
| bridges since netkit was introduced in the kernel
|
| https://isovalent.com/blog/post/cilium-netkit-a-new-containe...
| ignoramous wrote:
| > _Netkit, which is what this is built on, is pretty neat. For
| transmitting packets from one container /VM to another ..._
|
| Sounds like virtio but intra-host?
| tptacek wrote:
| No, virtio presents to the network stack the same way other
| devices do.
| nighthawk454 wrote:
| > eBPF is a technology that can run programs in a privileged
| context such as the operating system kernel. It is the successor
| to the Berkeley Packet Filter (BPF, with the "e" originally
| meaning "extended") filtering mechanism in Linux and is also used
| in non-networking parts of the Linux kernel as well.
|
| > It is used to safely and efficiently extend the capabilities of
| the kernel at runtime without requiring changes to kernel source
| code or loading kernel modules. Safety is provided through an in-
| kernel verifier which performs static code analysis and rejects
| programs which crash, hang or otherwise interfere with the kernel
| negatively.
|
| https://en.wikipedia.org/wiki/EBPF?useskin=vector
| throw78311 wrote:
| I guess this is why everything is under Federation/default now,
| the old mess was annoying to work with.
| tomohawk wrote:
| pretty cool, but basically solves a problem caused by one too
| many layers of abstraction.
| udev4096 wrote:
| There's https://github.com/eunomia-bpf/bpf-developer-tutorial if
| anyone wanted to get started with eBPF
| bogantech wrote:
| Semi related: is there some way to check what eBPF programs are
| installed on a system and explore what they're attached to /
| doing etc?
|
| Whenever I see a problem solved with eBPFs I feel like it's also
| making things more opaque and difficult to troubleshoot but I'm
| guessing that's just because I don't know enough about it
| AlotOfReading wrote:
| That's what bpftool is for. It follows the grand Linux
| tradition of making everything possible, but not necessarily
| easy.
___________________________________________________________________
(page generated 2025-01-30 23:02 UTC)