[HN Gopher] Going Big with TCP Packets
___________________________________________________________________
Going Big with TCP Packets
Author : pabs3
Score : 123 points
Date : 2022-02-15 04:51 UTC (1 days ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| [deleted]
| mprovost wrote:
| I've played around with jumbo frames in the past. One issue is
| that none of the checksums at any layer were designed for packets
| this big. Both the Ethernet and TCP checksums don't have enough
| bits to reliably detect errors, even on 9k frames. This is ok if
| you're doing checksumming at the application layer, but often
| you're not, maybe it's something like NFS that assumes lower
| levels of the stack take care of message integrity. As
| transmission speeds have gone up, the bit error rate has stayed
| the same, so you can just do the math on how many errors you're
| going to have per day.
| brohee wrote:
| The NFS WG was adding Integrity Measurement Architecture that
| would address that and more, but the effort seems to have
| fizzled out, i assume COVID didn't help.
| https://datatracker.ietf.org/doc/html/draft-ietf-nfsv4-integ...
| Sesse__ wrote:
| You need to have application-level checksums anyway, since
| pretty much every router or switch these days will recalculate
| checksums on all layers (they need to if routing the packet,
| and since pretty much all COTS chipsets can do L3, they do so
| all the time even if the switch's software won't let you
| configure it), so if a bit flips in any router's processing,
| your checksums won't detect it.
| mprovost wrote:
| Sure but that means rewriting every application. There was a
| mind blowing Defcon presentation a few years ago "Bit-
| squatting: DNS Hijacking Without Exploitation" by Artem
| Dinaburg where they registered a bunch of domains that were
| one bit off from well known cloud providers, and recorded all
| of the requests that leaked out of their internal networks
| when the DNS response had a bit flipped. That's not to say
| that you're going to have a huge DNS response, but there is
| already some amount of this going on at every level of the
| stack and unless you rewrite every application from DNS on up
| to check then you're still going to encounter errors.
| Sesse__ wrote:
| Well, use TLS and every TCP-using application is already
| secured.
| touisteur wrote:
| That's actually a very interesting observation. How much bits
| should a checksum take for such payloads? At least the 9000
| jumbo things.
| mprovost wrote:
| iSCSI uses a 32 bit CRC. (TCP is 16 bits). There was a draft
| to use 32 bit TCP checksums but it never got traction:
| https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-
| anumit...
| rjsw wrote:
| SCTP uses a 32 bit checksum.
| bogomipz wrote:
| Can someone say what the benefit of this is over just using
| existing Ethernet jumbo frames?
| jmyeet wrote:
| I'm a big believer in Postel's Law (also known as the robustness
| principle) [1]. Basically be liberal in what you accept and
| conservative in what you send.
|
| The problem with TCP is that the actual Layer 2 and 3
| infrastructure doesn't obey this principle. It's taken the
| opposite stance to simply reject anything that's weird or
| unexpected. There's a reason and history for why this is. The
| most defensible is security. Less defensible is things like deep
| packet inspection.
|
| This is a well-knwon problem and falls under the term (which I
| love) "ossification" [2].
|
| As an example, much of the Internet stops working with MTUs above
| ~1500 (which this article mentions). Large packets (eg MTU ~9000)
| are almost a necessity for 10+ GbE but that often won't work
| outside of the LAN.
|
| So I guess my point is changes like this will take a long time
| (if ever) to get widespread support. This could still impact,
| say, data center and other "local" deployments but I guess you
| have to start somewhere.
|
| [1]: https://en.wikipedia.org/wiki/Robustness_principle
|
| [2]: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7738442
| sly010 wrote:
| In this case I think the robustness principle is actually
| counter productive, here me out:
|
| MTU discovery is "robust". Most nodes already forward any
| packet they can handle. The problem is that _some_ nodes filter
| out ICMP packets (hence breaking MTU discovery) but the link
| still "kind-of-works" (tm). Nodes that break MTU discovery
| should be considered 100% broken, but instead of fixing them we
| "accommodate" them by statically (manually) lowering our
| outgoing MTU, hence proliferating the problem.
|
| Robustness works against us. If half-broken firewalls were
| considered fully-broken, they would be replaced instead of
| accommodated.
|
| Aside: the `Fragmentation Needed` packet should have been part
| of the IP header, not a separate protocol.
| sly010 wrote:
| > Most nodes already forward any packet they can handle On
| second thought, I guess I am wrong here, since the default
| MTU in Linux is a very low number (something like 1500).
| Robustness would be to have limit by default and fully rely
| on MTU discovery.
|
| Although the low default was probably established because of
| the aforementioned ICMP filtering issue.
| treesknees wrote:
| Part of this problem is there really isn't a layer 2 protocol
| for discovering the max MTU of a path. You have to use a higher
| protocol like ICMP to find it and then adjust your endpoint.
|
| The Internet will work with large L3/L4 packets assuming the
| path is able to create fragments. The article talked about the
| kernel's ability to hold fragments but it wasn't clear to me
| how Internet fragmentation comes into play or whether the
| performance gain is lost when using this on the Internet versus
| a LAN.
|
| Side tangent, this is one issue I have with the major cloud
| providers. Azure, AWS, et al all advertise high-bandwidth
| interfaces upwards of 100Gbps, but you can't saturate it
| anywhere near those rates with 1500 Byte packets from the
| Internet.
| jeffbee wrote:
| Sure, but the amount of traffic _inside_ AWS is way bigger
| than the amount of traffic that crosses its boundary. This is
| true for a lot of operators. Netflix and YouTube and similar
| loads that are virtually all egress are special cases.
| drewg123 wrote:
| If you read the actual LKML message, and not just the LWN
| article, it becomes clear this is really about bumping TSO/LRO
| sizes in linux, not about using link layer MTUs in excess of
| 64k. This is more akin to Microsoft's "LSOv2" (which is even
| mentioned in Erick's commit message for the mlx5 driver).
|
| I'm not sure if the LWN author just doesn't understand the
| distinction, or if they were trying to simplify things for
| their readers..
| Sesse__ wrote:
| The article is clear that it's about TSO:
|
| > Modern network interfaces perform segmentation offloading,
| meaning that much of the work of creating individual packets
| is done within the interface itself. Making segmentation
| offloading work with jumbo packets tends to involve a small
| number of tweaks; a few drivers are updated in the patch set.
| drewg123 wrote:
| I was keying off of this " _The BIG TCP patch set adds the
| logic necessary to generate and accept jumbo packets when
| the maximum transmission unit (MTU) of a connection is set
| sufficiently high._ ", which made it sound to me like the
| author believed that the MTU was 185k
| jbverschoor wrote:
| > https://en.wikipedia.org/wiki/Robustness_principle
|
| And I am totally against this. I agree for human / interactive
| input. But for automated things, be as strict as you can be,
| and hard fail on anything else. Specs are there for a reason,
| and not adhering them creates compatibility issues.
|
| > RFC 1122 (1989) expanded on Postel's principle by
| recommending that programmers "assume that the network is
| filled with malevolent entities that will send in packets
| designed to have the worst possible effect
|
| Well, that's gonna be a tough one if you try to make sense of
| garbage data (or data which doesn't adhere to the specs)
|
| > Protocols should allow for the addition of new codes for
| existing fields in future versions of protocols by accepting
| messages with unknown codes (possibly logging them).
|
| I agree with that. That does not mean "be liberal in what you
| accept". It means the protocol is designed in a better way.
|
| > Basically be liberal in what you accept and conservative in
| what you send
|
| That's not the same as "be conservative in what you do, be
| liberal in what you accept from others". "Doing" is a lot more
| than "sending".
|
| You know.. many programmers sometimes say that when things go
| wrong it's "a user error", or "a human error". But most of the
| time it is either a programming error, or a lazy programmer.
| It's so easy for a programmer to send the correct data. But we
| live in a world where usually the least correct/secure software
| is chosen, because it's slightly easier to get started, or
| allows for input errors (mysql data truncation, mongodb, etc,
| etc)
| david422 wrote:
| > And I am totally against this. I agree for human /
| interactive input. But for automated things, be as strict as
| you can be, and hard fail on anything else. Specs are there
| for a reason, and not adhering them creates compatibility
| issues.
|
| Totally agreed on this. When a human inputs a name with a
| trailing space, trim it for them. Or when they input a phone
| number with funny dashes, remove them.
|
| But when an API does this, it means the API client is simply
| incorrect. Incorrect requests should be rejected. Don't leave
| it up to the server to try to interpret it, this just leads
| to incorrect behavior.
| tomxor wrote:
| > As an example, much of the Internet stops working with MTUs
| above ~1500 (which this article mentions).
|
| I experienced this first hand recently with a lower MTU limit
| and it's pretty confusing. The annoying thing is that when it's
| only wrong at one end, most of the internet and web pages will
| actually carry on working, i.e it's only some HTTP requests
| with large enough request header that push past the MTU
| fragmentation threshold and disappear. Which results in
| websites half loading, or sometimes loading fine, but not other
| times.
|
| I have to use LTE to get decent internet in the city I live,
| and have been experimenting with different providers. Turns out
| most LTE networks use quite low MTUs, and so PMTUD (MTU
| discovery) _must_ work correctly otherwise you get the
| behaviour I described above... For the network with the best
| reception and backhaul, PMTUD just didn't seem to work, so it
| defaults to 1500, and all the wifi connections to the LTE
| router will be told to use 1500, so my wiregaurd sets itself to
| 1420 (less 80 bytes) and even that wont save itself when the
| UDP packets vanish.
|
| Now when the internet seems fishy my first response after a
| simple ping is to try ping at the network interface supposed
| MTU, e.g `ping -M do -s 1472 1.1.1.1` (-28 bytes for the
| header).
|
| Even more confusing is that for these LTE networks the
| effective MTU seems to change, possible as different
| combinations of cells are used for carrier aggregation... so if
| you are trying to use this for home internet you need to keep
| testing for the MTU over time to discover the lowest, then
| change your router wifi MTU (not even possible for most LTE
| routers) to the minimum for reliable internet.
| Sesse__ wrote:
| PMTUD on the Internet, sadly, does not work anymore. It was
| wonky for a few years, then totally died when AWS and the
| other cloud providers started blocking all ICMP in the
| default configuration. (It is also notoriously hard to get to
| work through packet-based load balancers, since most routers
| won't include enough of the original packet in the "too big"
| ICMP to reliably send it the same way as all the data packets
| that form the flow.)
|
| The reason why you can browse the web at all with PMTU <1500
| is due to the awful hack that is TCP MSS clamping, where the
| router will rewrite your TCP SYN packets to advertise a
| different MSS (which is roughly the same as MTU, only on
| layer 3) based on what it thinks the PMTU to the host is.
|
| Seriously, try browsing the Internet with MTU 1500, PMTU
| <1500 and a router in place that does _not_ do TCP MSS
| clamping. It's an incredibly frustrating experience.
| [deleted]
| valyagolev wrote:
| arguments against this principle were discussed here recently:
| https://news.ycombinator.com/item?id=30328446
|
| I honestly don't think an implementation of TCP that would try
| and follow this principle would bring anything but misery. I
| lived on the internet since IE5... imagine including "hacks for
| Cisco" in your packets
| sriram_sun wrote:
| Do virtual adapters on VMs for communication between host and
| guest stand to benefit from this? For e.g. scp between host and
| guest. Current max mtu size on VirtualBox is 9000 right?
| drewg123 wrote:
| The article says " _Enabling a packet size of 185,000 bytes
| increased network throughput by nearly 50%_ " So they essentially
| tripled the max TSO/GRO size from ~64k to 185k and that resulted
| in more throughput, presumably due to less overhead.
|
| That's a LOT of per-packet overheads, and seems a bit surprising
| to me. In tests I've done, I have not seen much difference
| between a 16k max tso and a 64k max TSO. Then again, my benchmark
| is generally a TCP_STREAM (throughput), and not a TCP_RR (ping
| pong) test.
|
| In general, outside a datacenter, the last thing you want to do
| is dump 185KB for the same connection on the wire in a giant
| burst, as you'll blow out router buffers when transiting the
| internet from 100GbE in your datacenter to some crappy DSL router
| at your users' houses. So I wonder how much the benefit is
| reduced when using some kind of software based packet pacing? I
| think linux has some kind of packet scheduler designed to avoid
| large bursts, etc. Does it break apart these giant TSOs? Or does
| using it just cap the max TSO size?
|
| At least in FreeBSD. the TCP stack controls pacing and one thing
| it does is to send down smaller TSOs to NICs that don't support
| hardware pacing. The goal is to reduce the burst size on the
| wire, so TCP will send down a few KB, wait several milliseconds,
| dribbled down more, etc. This reduces our average TSO size, and
| increases overhead.
| touisteur wrote:
| One of the things datacenter people want is to avoid dpdk-
| everything just because they put 4 200GbE in their server. I
| sure would like to just zmq everything... Especially for
| internal streaming applications, that would be a boon.
|
| I haven't read the patchset yet but I'm wondering whether it
| would work with UDP. TCP can be too much 'kitchensink' if you
| don't need reordering or many-resends.
| bogomipz wrote:
| >"Imagine, for a second, that you are trying to keep up with a
| 100Gb/s network adapter. As networking developer Jesper Brouer
| described back in 2015, if one is using the longstanding maximum
| packet size of 1,538 bytes, running the interface at full speed
| means coping with over eight-million packets per second."
|
| Should this not be "frame size" instead of "packet size"? Unless
| I am reading this incorrectly the maximum TCP packet size is
| 1500(20 bytes for IP header, 20 bytes for TCP header and 1460
| bytes for the actual payload.) I'm not trying to be pedantic but
| the other 38 bytes would all be considered part of the Ethernet
| and not TCP no?
| codechad wrote:
| Interesting development. However, I do wonder at some point if
| those desiring fast packet processing shouldn't just bite the
| bullet and adopt something DPDK-based (eg, FDIO/VPP).
|
| I think pure-interrupt packet processing is challenged (for the
| reasons laid out in the article). But getting people to dedicate
| a core/thread [to RX/TX polling] is also a hard sell.
|
| Either way this is good news - more pps/bps!
| jeffbee wrote:
| The organization in question already bypasses the kernel with
| their "SNAP" scheme. Perhaps they are motivated to keep
| improving kernel IP stack simply because practically all GCP
| customers still use it.
|
| https://blog.acolyer.org/2019/11/11/snap-networking/
| codechad wrote:
| touisteur wrote:
| I wanted to pay someone to work on a dpdk layer for zeromq, I'm
| not sure anyone would be interested but I feel it'd help
| alleviate part of the dpdk pain.
| gjulianm wrote:
| > But getting people to dedicate a core/thread [to RX/TX
| polling] is also a hard sell.
|
| I think that if you're at the level where the kernel network
| stack is not enough, core pinning is a no brainer. You're
| probably spending more than one core in receiving data, moving
| to a polling thread pinned to a single core will not only
| improve performance due to the polling, but also due to the
| scheduler not pushing your process out and to the possibility
| of putting the threads on the appropriate NUMA nodes.
| baruch wrote:
| Part of the pain with kernel bypass is memory management. For
| real performance gains you need zero copy and for that you need
| to work inside the packets. Perfectly doable for a green field
| project not so east I think for a large existing system.
| trasz wrote:
| Or simply use network cards with full TCP offload
| (https://en.wikipedia.org/wiki/TCP_offload_engine).
| [deleted]
| jsnell wrote:
| Kernel bypass is not a panacea. Even with a totally userspace
| networking stack, there will be a lot of per-packet rather than
| per-byte processing, and often packets from different sources
| will be sufficiently interleaved that you can't even do any
| kind of useful batching fastpath .
| signa11 wrote:
| taking cpu-cores out of normal scheduling, and dedicating
| them for just packet-forwarding etc. is the first step. not
| sure how popular ddio is outside of intel ecosystem, but
| shunting packets between nic and cpu-cache (to and from) is
| very useful for such workloads imho.
| cyberge99 wrote:
| Aren't cat videos delivered with UDP?
| zamadatix wrote:
| Depends, if the client connects via HTTP/3 or QUIC yes, HTTP/2
| or older no. There are also the few odd live cat video sites
| that'll use WebRTC (UDP).
| donkarma wrote:
| shame we couldn't have used SCTP due to middleboxes, although
| I assume corporate networks will kill QUIC anyways
| bragr wrote:
| Seems pretty useful once all the drivers, utilities, network
| devices catch up. This will be all the rage in SAN and other
| performance network environments in a couple years. Would be
| great to see a more thorough performance evaluation. For example
| I wonder if the throughput and latency hold in different network
| conditions - e.g. network conditions with high latency, packet
| loss, and both. I presume after a certain point of degradation
| the larger packet size becomes of penalty because your
| retransmissions become very wasteful.
| joshbaptiste wrote:
| What are the current MTU sizes of these performance backbone
| environments?
___________________________________________________________________
(page generated 2022-02-16 23:01 UTC)