[HN Gopher] Surpassing 10Gb/S over Tailscale
___________________________________________________________________
Surpassing 10Gb/S over Tailscale
Author : mssdvd
Score : 121 points
Date : 2023-04-13 18:11 UTC (4 hours ago)
(HTM) web link (tailscale.com)
(TXT) w3m dump (tailscale.com)
| withinboredom wrote:
| I've really wanted to try tailscale. I fear I'll like it, and I
| don't want another company to have a monopoly on simple things so
| everyone forgets how to do them.
| beambot wrote:
| I don't think I'd classify their zero-config p2p-style VPN as
| "simple" -- or at least, certainly not simple to replicate...
|
| More to the point, I hope their technology becomes commonplace
| & gratis a la LetsEncrypt for SSL Certificates.
| withinboredom wrote:
| I mean, setting up a WireGuard vpn is pretty darn simple,
| even into a k8s cluster. It's not rocket science or anything;
| which is kinda my point. They make it too easy, and that
| worries me.
| dcomp wrote:
| I've switched to tailscale because their nat busting is
| actually hard to do "by hand"
| withinboredom wrote:
| That's sort of the problem, right. Joining two networks
| is pretty simple, once you do it a few times. I remember
| when it was mandatory to know how to set up an email
| server (for more than one user), configure a secure FTP
| (+ WebDAV for a little while), and probably other things
| I've totally forgetting about. These things were passed
| down from senior to junior like we pass down how to write
| Docker images, and set up our ide while those very simple
| services of yesterday have been eaten up by monopolies.
| I'm not saying we shouldn't have services to make our
| lives easier... I'm saying we should have more of them.
| I'm not interested in this space, but someone who is
| should see this company and go "damn, these guys have
| validated an idea for me. Maybe I can take some of their
| pie." Instead, we just give them more money ...
|
| Look at email. It's basically a "lost technology" in that
| it is nearly impossible to self-host (though there are
| people out there doing it, there are very few modern
| guides from zero to production). Same with file sharing
| and IRC servers.
|
| Maybe I'm just rambling in my "old" age...
| bananapub wrote:
| none of those things are analogous to Tailscale having
| done loads of hard work to automate NAT busting.
| wmf wrote:
| Tailscale has several competitors such as ZeroTier and
| Nebula. There does appear to be a winner-take-all dynamic
| where being slightly better lets Tailscale take 10x more
| mindshare than competitors, but I don't see any way
| around that.
| 2bluesc wrote:
| Setting up a few p2p wg VPNs is manageable.
|
| However, when you have 10 nodes and need to add one more
| node, you now need to update all other nodes so they can
| speak p2p. Management with scale is the struggle.
| yjftsjthsd-h wrote:
| If you have 10 nodes, you should already be automating
| with ansible/chef/puppet/whatever, at which point adding
| another link config is easy.
| 2bluesc wrote:
| For servers sure, but things like `tailscale` exist to
| save every laptop and cell phone from looking like a
| devops project.
|
| Furthermore you could extend this argument almost every
| other cloud service with a primary feature of
| "convenience" and/or "management". Just build everything
| yourself.
| necubi wrote:
| Tailscale is amazing. I was able to set up our AWS VPN with it
| in <30 minutes, and it's just worked ever since. Getting new
| users set up is similarly seamless.
|
| If this means I continue to forget how to run OpenVPN I
| consider that well worth it.
| thenipper wrote:
| It's made putting internal apps in a private subnet on a VPC
| a very trivial process. Like took me an afternoon and works
| well for my small 40 person company.
| 2bluesc wrote:
| Had similar feelings and did like it more then I thought I
| could.
|
| My escape hatch from the monopoly is headscale[0] which I can
| self host.
|
| [0] https://github.com/juanfont/headscale
| xena wrote:
| You can even host Headscale over Tailscale, amusingly:
| https://tailscale.dev/blog/headscale-funnel
| [deleted]
| infogulch wrote:
| Love to hear optimization stories, great work!
| red0point wrote:
| What's missing from all these figure is the resulting latency.
| It's often the case that vendors show impressive throughput
| numbers, but then the latency is terrible at that throughput.
|
| Do you have those numbers as well?
| raggi wrote:
| We do look at them to check on how we're doing, and I want to
| dig into this area more over time. In particular we don't do
| classful prioritization right now, which if you look at the
| typical tests for this they're often focused on multi-flow
| classifications. We also don't set specific congestion
| algorithms on our interfaces right now - availability is
| variable, as is the cost of them. You can see in the post here
| that Jordan documents that the tests in the blog were all
| explicitly over cubic.
|
| We increased the sizes of the UDP buffers in the prior round of
| optimizations. The kernel defaults for UDP buffers are too
| small to approach the throughput discussed here - and the
| default sizings were the primary source of lots of dropped
| packets. I raised those to 7mb, which seems like an odd number,
| but it's the largest you can set on macOS before the kernel
| rejects it - likely we'll eventually head for a per-platform
| split. At these speeds a 7mb buffer represents up to 5ms of
| flow data, though this does not imply that it creates 5ms of
| bufferbloat - it just means that this increased buffer could
| itself account for 5ms in the worst non-lossy case. On the
| userspace side Tailscale also has some more buffer space now
| (we're reading and writing lists of packets at a time, not
| single packets), but the sizing there is more complex.
|
| This topic in general is much more complex - in the first
| throughput post I originally started to dig into it, and we cut
| that in editing because it was making the post too dense and
| there wasn't space to give the topic the attention it deserves.
| One day we'll talk about this too. Typically right now we add
| very little latency, low millis or lower - we actually add more
| jitter than latency, as any userspace program would. It's still
| orders of magnitude lower than the levels which even concern a
| typical realtime application such as gaming or communications -
| for example someone was recently talking about using Tailscale
| on their Steamdeck while on vacation to play Hogwarts streaming
| from their PC.
|
| In the meantime, a real world example for you. I have a border
| router that I built using a relatively cheap piece of hardware
| (Intel(R) Celeron(R) J4105 CPU @ 1.50GHz). It has NICs that
| support GRO/GSO, but the CPU is the bottleneck for throughput.
| The box does 563MBits/sec inbound to the LAN over Tailscale
| (949 Mbits/sec raw). I run this as an exit-node for my
| workstation all the time, even though that's in the same
| building - and do so for the sake of diagnosing bugs and
| experiencing the product full time. In my initial test today,
| under peak load the exit node adds 35ms of latency each way. I
| was surprised by this, so I checked when going direct rather
| than via the exit node, I see 15ms down and 30ms up of latency
| increase under peak load. It seems Comcast dropped some
| capacity since I last tuned my uplink!
|
| I then re-tuned CAKE on the router uplink to be more aggressive
| resulting in a raw bloat of 0ms/0ms, and then retested with the
| Tailscale exit node. With these more aggressive CAKE tunings,
| Tailscale also stayed at 0ms/0ms. This CAKE tuning ate a chunk
| of throughput capacity, as expected. The specific tuning here
| being for a Comcast 1000/40 link, and the system CPU bound at
| 500mbps for forwarding: + tc qdisc add dev
| internet root handle 1: cake docsis ack-filter-aggressive nat
| bandwidth 40mbit lan + ip link add name ifbinternet type
| ifb + tc qdisc add dev internet handle ffff: ingress
| + tc qdisc add dev ifbinternet root cake bandwidth 500mbit lan
| + ip link set ifbinternet up + tc filter add dev internet
| parent ffff: matchall action mirred egress redirect dev
| ifbinternet
|
| On the LAN side, between the same machines (fq_codel only,
| default settings), running iperf3 alongside ping:
|
| Under max load ([ 5] 0.00-57.73 sec 3.72 GBytes 554 Mbits/sec
| receiver): 10 packets transmitted, 10 received,
| 0% packet loss, time 9013ms rtt min/avg/max/mdev =
| 2.625/3.620/4.536/0.646 ms
|
| Zero load: 10 packets transmitted, 10 received,
| 0% packet loss, time 9014ms rtt min/avg/max/mdev =
| 0.648/0.954/1.713/0.306 ms
|
| What do these numbers mean? In practice they mean you'll notice
| WiFi more than you'll notice Tailscale, but we can and will
| still do better over time. Here's WiFi from a MacBook to the
| border router on the same LAN segment (no WireGuard/Tailscale):
| 10 packets transmitted, 10 packets received, 0.0% packet loss
| round-trip min/avg/max/stddev = 3.845/11.363/34.152/8.940 ms
|
| This is already long for an HN response, and so much more to
| say, but I hope it helps!
| nikisweeting wrote:
| Very curious to learn more about CAKE tuning with tailscale,
| would love to see a post someday about how the two interact
| and when/why it might be needed?
| timando wrote:
| I have no idea what a gigabit per siemens is supposed to mean.
| bradfitz wrote:
| The blog post's actual title doesn't use that case.
| 1vuio0pswjnm7 wrote:
| https://github.com/juanfont/headscale/releases/expanded_asse...
| kristofferR wrote:
| Tailscale is awesome, so damn recommended. Taildrop (AirDrop for
| everything, included in Tailscale) is especially recommended, it
| makes it so damn easy to send files between all your devices.
| raybb wrote:
| https://tailscale.com/kb/1106/taildrop/ seems to be the docs.
|
| It's the first I hear of this. I wonder if there's any big
| advantage of this for someone who is already using syncthing
| for the same purpose? Biggest thing I could hope for is that
| it's faster. But I generally don't keep Tailscale running on
| mobile because I don't need it to and don't like the persistent
| notification.
| klabb3 wrote:
| Sync, continuous backup and transfer are all quite different
| use-cases.
|
| Most backup/sync products are designed to work in the
| background and often require upload before download. I don't
| know if syncthing does streaming syncs though.
|
| Another difference is transfers can easily be untrusted, as
| in sender and receiver don't need access to each others file
| systems. Take magic wormhole (or email attachments for that
| matter) as an example.
|
| Taildrop is somewhere in between - I think you have to be on
| same tail net, but no need for awareness of the other
| device's file system.
| 0xQSL wrote:
| Nice improvements! I'd be interested to see how much overhead
| tailscales magicsock adds and what a flamegraph after the change
| looks like. Mostly crypto or still a lot of networking syscall
| time?
| raggi wrote:
| magicsock definitely does a bunch more work, and we do look at
| both profiles. The magicsock profile is harder to read as a
| consequence of being a more complex path, adding packet
| filters, the indirection for DERP and other NAT busting
| details, etc. Jordan did do some optimizations in the magicsock
| path alongside this wireguard-go work to get us over the 10gbps
| line.
|
| Overall the summary of time spent is still a similar story at
| the coarse scale - our recent optimizations mean that we're
| getting ever closer to the point where we need to start working
| on the next layer, such as optimizing the queues (visible here
| in the chanrecv and scheduler times - Go runtime stuff), and
| once we get that out of the way things like crypto and copying
| will become targets. The work goes on, we have lots of plans
| and ideas!
| ignoramous wrote:
| Super neat.
|
| Have these optimizations (TCP GRO/GSO) been applied to _non-
| root_ tailscale? I imagine the changes needed are wildly
| different as the TUN device itself is gvisor /netstack. I
| believe the UDP GRO/GSO part (discussed in today's blog post)
| may work as-is.
| raggi wrote:
| Good question, it's bits and pieces. I know there's more we
| can do with the userspace stack - netstack has some support
| for GRO/GSO, but unless I'm forgetting a detail we haven't
| fully plumbed that yet. It would definitely be interesting
| to do so - avoiding TUN turnaround while still utilizing
| mmsg and so on should provide excellent performance for
| something like a tsnet/libtailscale based server. We did
| recently improve performance in that configuration by
| enabling SACK, which is very significant.
| jeffbee wrote:
| Half-way through the article it just says UDP receive coalescing,
| once, and never mentions it again. Do they mean interrupt
| mitigation? If so, using what parameters?
| ignoramous wrote:
| My guess is UDP receive coalescing is UDP GRO (generic recv
| offload) + recvmmsg(2)
| perryh2 wrote:
| I see a new blog post from Tailscale, I upvote.
| ThePhysicist wrote:
| Pretty amazing that you can achieve such a throughput in a Golang
| userspace program. I wonder if other UDP based protocols like
| QUIC can attain those numbers as well.
| kodah wrote:
| Slack has a system called Nebula that's pretty adjacent to
| userspace WireGuard.
| unixhero wrote:
| Nebula is a Tailscale clone
| psanford wrote:
| Nebula predates tailscale.
| preseinger wrote:
| go is pretty fast
|
| in fact, i have a standing bet with some of my rustacean
| friends that they can't show me a typical HTTP service in rust,
| which has performance numbers (rps, latency, throughput) that i
| can't meet or beat in go
|
| of course lots of caveats there, what does normal-ish mean,
| well probably most of the work is gonna be i/o bound, it should
| run on normal server-class hardware, et cetera et cetera
|
| but nothing yet
| adam_arthur wrote:
| Seems Rust places well in some composite benchmarks. Go is
| further down the list. Of course this depends on the quality
| of the implementation and doesn't account for UX/usability
|
| https://www.techempower.com/benchmarks/#section=data-r21&tes.
| ..
| fpoling wrote:
| Caddy (a web server written in Go) is like two times slower
| than Nginx on many benchmarks.
| hnav wrote:
| caddy is written in terms of net/http, nginx is written
| right on top of epoll/kqueue with bespoke HTTP/1x parser in
| a manually memory managed language. I think the point was
| not that go is "faster" than anything, it's that it makes
| it easy to write hard-to-beat network software once you get
| out of the realm of toy or highly specialized problems.
| api wrote:
| For most compiled languages or languages with _very good_ VMs
| like Java benchmarks are really testing the quality of the
| implementation and the depth of the implementor 's
| understanding.
|
| I'd bet that very good Go and Rust programmers could probably
| converge to almost identical performance.
|
| What I wouldn't be on is that Go could equal Rust in the area
| of small memory footprint or on small devices.
| VWWHFSfQ wrote:
| is the go userspace program actually shoveling this data or are
| they in-kernel buffer copies a la sendfile and the like
| wmf wrote:
| Yes, userspace has to touch the data to encrypt/decrypt it.
| ffk wrote:
| Interestingly, the fastest CPU based network switches tend to
| do full kernel bypass. The kernel is generally slow compared to
| OVS and VPP, especially when they traverse over something like
| DPDK.
| 2bluesc wrote:
| The missing feature from Tailscale for me is the ability to host
| a Tailscale only DNS zone.
|
| They have Magic DNS, but that only works for individual Tailscale
| nodes. I want multiple DNA records pointing to a single Tailscale
| node. Would be even better if I could use my own domain
| (subdomain even better) instead of their long `foo-bar.ts.net`
| domain.
|
| Currently need to do this manually, but seems overly redundant
| since Tailscale already does 90% of this with MagicDNS and is
| fast because it's in their client vs a remote server.
| zrail wrote:
| Step 1: install Tailscale and Docker on a VM or whatever
|
| Step 2: set up a Technitium container in host networking mode
|
| Step 3: configure Technitium with a stub zone pointing your
| ts.net name at 100.100.100.100
|
| Step 4: set up a zone for whatever.tld
|
| Step 5: set up a DNAME record for ts.whatever.tld pointing at
| your ts.net domain
|
| Result: querying this new DNS server with
| machine.ts.whatever.tld resolves to machine.blah-foo.ts.net
| resolves to that machine's 100.64.0.0 address.
|
| https://technitium.com/dns/
___________________________________________________________________
(page generated 2023-04-13 23:01 UTC)