https://blog.ipspace.net/2022/12/quick-look-aws-srd.html ipSpace Logo * subscriptions + individual + team + free + frequent questions * training + online courses + webinars + workshops + technology areas and roadmaps * consulting + ExpertExpress + case studies + contact * blog * resources + ipSpace.net blog + articles + technology resources + Software Gone Wild podcast + books ------------------------------------------------------------- + events and conferences + interviews * about + ipSpace.net + Ivan Pepelnjak + webinar and course authors + ExpertExpress team ------------------------------------------------------------- + company information + contact [ ] get started log in Building network automation solutions 9 module online course Start now! Wednesday, December 14, 2022 07:17 UTC A Quick Look at AWS Scalable Reliable Datagram Protocol One of the most exciting announcements from the last AWS re:Invent was the Elastic Network Adapter (ENA) Express functionality that uses the Scalable Reliable Datagram (SRD) protocol as the transport protocol for the overlay virtual networks. AWS claims ENA Express can push 25 Gbps over a single TCP flow and that SRD improves the tail latency (99.9 percentile) for high-throughput workloads by 85%. Ignoring the "DPUs could change the network forever" blogosphere reactions (hint: they won't), let's see what could be happening behind the scenes and why SRD improves TCP throughput and tail latency. AWS developed SRD as a transport protocol for Elastic Fabric Adapters (the networking part of their HPC implementation). There's no magic behind SRD; it's just another data point in the transport protocol solution space: * Like UDP, it provides a datagram transport. * Unlike UDP, the datagram delivery is reliable. * Unlike TCP (another reliable delivery protocol), SRD can reorder packets in transit and deliver them out-of-order. SRD consumers are supposed to deal with packet reordering (unlike some UDP consumers that drop reordered packets). Dropping the "in-order delivery" requirement allows AWS to send SRD packets in parallel over all alternate paths. That approach also diminishes link congestion - instead of a long burst of packets landing on a single link, the packet burst is spread across many parallel links. Next step: ENA Express uses SRD instead of GRE (or VXLAN or GENEVE) to transport Ethernet frames between hypervisor hosts, resulting in faster, reliable delivery of potentially reordered packets. It's obvious why that increases the throughput of a single TCP session - packets from a single session can be sent over multiple links^1 - but why does it decrease the tail latency? I know just enough about TCP to have incorrect opinions, but (based on what people who should know better told me) there are several reasons for variability in TCP throughput and latency: * Packet drops could be a Really Bad Thing if your TCP stack uses drop-sensitive congestion avoidance algorithm. Having reliable underlay transport solves this one. * Early TCP implementations could interpret reordered packets as a packet drop of the intermediate packets (see above). I was told this was a solved problem and should have disappeared in recent TCP implementations after the TCP Selective ACK was implemented. * Packet reordering also kills hardware-based Receive Side Coalescing, but it looks like AWS was more than willing to sacrifice the CPU cycles needed to sort the packets in software to get better performance. * Losing the last packet of a packet burst is a killer, even if your TCP stack uses Selective ACK. The receiver can't acknowledge the lost packet or send Selective ACK because there's no subsequent packet, forcing the sender to wait for the timeout. The real SNAFU: the minimum TCP timeout on most operating systems is a few milliseconds, while the fabric transit times are measured in microseconds. No wonder the tail latency is through the roof and can be fixed with reliable transport of IP datagrams. Now that you understand some of the reasons why ENA Express improves performance and tail latency, let's quickly deal with the inevitable hype: * Is this a new idea? Of course not. We've used reliable frame transport for ages on media that were too noisy (or drop-prone) for TCP, starting with X.25 and analog modems using V.42 error correction and continuing with radio networks. * Do you need a DPU to implement something like ENA Express? Absolutely not; you could implement it in the virtual switch like GRE, VXLAN, or GENEVE. It's just the question of which CPU cycles you like to burn. * Could someone else do something similar? Of course, but it would require a focus on customer performance, a deep understanding of transport protocols, and engineering prowess. Charging the customers for services they consume also helps to focus your thinking. Is it fair to expect any of that from a company that needed years to add LACP to its virtual switch? Want to know more about networking in AWS? Watch the Amazon Web Services Networking webinar ;) --------------------------------------------------------------------- 1. Brocade did something similar ages ago in the VCS Fabric. -[?] * AWS * switching Recent posts in the same categories AWS * AWS Automatic EC2 Instance Recovery * New Content in AWS Networking Webinar * Intricate AWS IPv6 Direct Connect Challenges * Intermittent Terraform Authentication Failure Using AWS Provider in a Vagrant VM * Relative Speed of Public Cloud Orchestration Systems * Deploying Advanced AWS Networking Features switching * DPU Hype Considered Harmful * Running Routing Protocols over MLAG Links * Are DPUs Any Good? * Combining MLAG Clusters with VXLAN Fabric * VLAN Interfaces and Subinterfaces * How Routers Became Bridges 2 comments: 1. Daniel S 14 December 2022 01:02 Wouldn't it make sense to implement SRD into, for example, the linux kernel and get rid of TCP for a lot of applications? I don't see it replacing UDP. But SRD for all of webtraffic? Wouldn't be that bad. I read somewhere a few days ago that we should replace TCP in the datacenter; HA.. I found it again: https://arxiv.org/pdf/2210.00714.pdf They're talking about "Homa", I don't know enough application and transport stuff to know if the Homa idea is good or bad. And how it relates to SRD. Replies 1. Ivan Pepelnjak 14 December 2022 01:43 In the AWS case, SRD is used instead of VXLAN/GENEVE to transport customer data, so it's still Ethernet/IP/TCP above it. Replacing TCP with SRD is hard because the original designers of the socket API wrote QDS code before thinking it through... https://blog.ipspace.net/2009/08/ what-went-wrong-socket-api.html As for "replacing TCP in data center", that article is so full of bullshit that I postponed writing a reply till next year ;) 2. Daniel S 14 December 2022 04:45 I thought it was your idea for a next step to use SRD as a replacement for overlays because you wrote ,,Next step". Sorry. But could it replace TCP for real? I'm not talking overlays, just plain TCP. 3. Ivan Pepelnjak 14 December 2022 04:51 Could SRD replace TCP? Of course not. Applications consuming TCP streams expect in-order delivery. You could add in-order delivery to SRD, but then you just reinvented TCP ;) 2. jsicuran 16 December 2022 01:38 When I first read about AWS ENA and SRD I was thinking, did AWS "roll their own" in a manner like RTP. EIGRP's own protocol to reliably send data to all "nodes", with piggyback acks, etc that uses multicast and unicast methods. Add comment Newer Post Home Older Post Sidebar Follow us SUBSCRIBE TO SDN MAILING LIST More information Recently updated * ICMP Redirects and Suboptimal Routing Featured * Multi-Chassis Link Aggregation * Build Virtual Labs with netlab * High Availability Switching Software-Defined Networking * The OpenFlow/SDN Hype * OpenFlow Basics * What Is SDN? Popular posts * Why Would You Need an Overlay Network? * Multihoming Cannot Be Solved within a Network * Network Automation: a Service Provider Perspective * Running Routing Protocols over MLAG Links * Are DPUs Any Good? * Integrated Routing and Bridging (IRB) Design Models * Running a Ubuntu VM on a Mac M1 * BGP Unnumbered Duct Tape * A Quick Look at AWS Scalable Reliable Datagram Protocol * Arista EOS Configuration Automation see more Blog Archive * 2022 (246) + December 2022 (14) + November 2022 (27) + October 2022 (25) + September 2022 (27) + August 2022 (1) + July 2022 (4) + June 2022 (20) + May 2022 (26) + April 2022 (26) + March 2022 (28) + February 2022 (26) + January 2022 (22) * 2021 (267) + December 2021 (15) + November 2021 (29) + October 2021 (29) + September 2021 (23) + August 2021 (4) + July 2021 (3) + June 2021 (29) + May 2021 (29) + April 2021 (28) + March 2021 (29) + February 2021 (25) + January 2021 (24) * 2020 (268) + December 2020 (19) + November 2020 (25) + October 2020 (30) + September 2020 (29) + August 2020 (14) + July 2020 (6) + June 2020 (22) + May 2020 (25) + April 2020 (26) + March 2020 (23) + February 2020 (23) + January 2020 (26) * 2019 (225) + December 2019 (14) + November 2019 (24) + October 2019 (26) + September 2019 (24) + August 2019 (12) + July 2019 (2) + June 2019 (18) + May 2019 (24) + April 2019 (22) + March 2019 (22) + February 2019 (20) + January 2019 (17) * 2018 (235) + December 2018 (15) + November 2018 (20) + October 2018 (23) + September 2018 (20) + August 2018 (16) + July 2018 (8) + June 2018 (16) + May 2018 (24) + April 2018 (25) + March 2018 (25) + February 2018 (23) + January 2018 (20) * 2017 (233) + December 2017 (21) + November 2017 (22) + October 2017 (24) + September 2017 (22) + August 2017 (16) + July 2017 (10) + June 2017 (18) + May 2017 (20) + April 2017 (18) + March 2017 (24) + February 2017 (20) + January 2017 (18) * 2016 (227) + December 2016 (19) + November 2016 (21) + October 2016 (22) + September 2016 (23) + August 2016 (16) + July 2016 (12) + June 2016 (18) + May 2016 (21) + April 2016 (18) + March 2016 (21) + February 2016 (21) + January 2016 (15) * 2015 (245) + December 2015 (13) + November 2015 (20) + October 2015 (21) + September 2015 (18) + August 2015 (19) + July 2015 (19) + June 2015 (24) + May 2015 (21) + April 2015 (22) + March 2015 (23) + February 2015 (24) + January 2015 (21) * 2014 (247) + December 2014 (13) + November 2014 (22) + October 2014 (24) + September 2014 (26) + August 2014 (20) + July 2014 (16) + June 2014 (24) + May 2014 (21) + April 2014 (18) + March 2014 (21) + February 2014 (20) + January 2014 (22) * 2013 (244) + December 2013 (15) + November 2013 (19) + October 2013 (24) + September 2013 (25) + August 2013 (22) + July 2013 (17) + June 2013 (19) + May 2013 (24) + April 2013 (24) + March 2013 (17) + February 2013 (20) + January 2013 (18) * 2012 (190) + December 2012 (10) + November 2012 (17) + October 2012 (17) + September 2012 (13) + August 2012 (18) + July 2012 (15) + June 2012 (10) + May 2012 (15) + April 2012 (16) + March 2012 (19) + February 2012 (20) + January 2012 (20) * 2011 (244) + December 2011 (17) + November 2011 (24) + October 2011 (22) + September 2011 (22) + August 2011 (21) + July 2011 (13) + June 2011 (21) + May 2011 (21) + April 2011 (21) + March 2011 (19) + February 2011 (21) + January 2011 (22) * 2010 (229) + December 2010 (23) + November 2010 (19) + October 2010 (19) + September 2010 (23) + August 2010 (24) + July 2010 (22) + June 2010 (25) + May 2010 (14) + April 2010 (21) + March 2010 (21) + February 2010 (9) + January 2010 (9) * 2009 (177) + December 2009 (13) + November 2009 (18) + October 2009 (14) + September 2009 (10) + August 2009 (10) + July 2009 (18) + June 2009 (18) + May 2009 (17) + April 2009 (16) + March 2009 (22) + February 2009 (11) + January 2009 (10) * 2008 (228) + December 2008 (14) + November 2008 (21) + October 2008 (19) + September 2008 (19) + August 2008 (20) + July 2008 (16) + June 2008 (20) + May 2008 (20) + April 2008 (15) + March 2008 (20) + February 2008 (21) + January 2008 (23) * 2007 (205) + December 2007 (11) + November 2007 (22) + October 2007 (18) + September 2007 (20) + August 2007 (19) + July 2007 (10) + June 2007 (16) + May 2007 (17) + April 2007 (17) + March 2007 (19) + February 2007 (19) + January 2007 (17) * 2006 (35) + December 2006 (9) + November 2006 (7) + October 2006 (16) + September 2006 (1) + August 2006 (2) Recent tags * automation (402) * worth reading (117) * switching (205) * IP routing (216) * cloud (180) * EVPN (50) * virtualization (263) * BGP (260) * VXLAN (79) * networking fundamentals (102) Other tags * data center (628) * SDN (356) * security (246) * IPv6 (245) * design (235) * fabric (180) * OpenFlow (147) * WAN (141) * high availability (128) * overlay networks (128) * Software Gone Wild (116) * OSPF (113) * Internet (109) * network management (104) * firewall (103) * MPLS (102) * bridging (102) * MPLS VPN (87) * Ansible (74) * QoS (73) * load balancing (69) * EEM (59) * command line interface (59) * LAN (55) * DMVPN (55) * Tcl (50) * certifications (47) * DHCP (44) * SD-WAN (43) * configuration (36) * vMotion (36) * access control (34) * web (33) * service providers (32) * SAN (30) * NSX (30) * HTTP (28) * AWS (28) * EIGRP (26) * PPP (26) * scalability (26) * traffic engineering (25) * humor (23) * Azure (23) * DNS (22) * NAT (22) * IS-IS (22) * FCoE (22) * link aggregation (22) * ACI (21) * show filters (20) * IPsec (20) * segment routing (19) * logging (18) * TRILL (18) * Cumulus Linux (18) * TCP (18) * CEF (17) * VPN (16) * ARP (15) * containers (15) * syslog (14) * what went wrong (13) * CLNP (12) * DCB (12) * intent-based networking (12) * performance (11) * history (6) * MLAG (3) * BGP (2) * vxlan (2) * OSPF (1) * evpn (1) * networking fundamentals (1) * overlay networks (1) see more tags The author Ivan Pepelnjak Photo Ivan Pepelnjak (CCIE#1354 Emeritus), Independent Network Architect at ipSpace.net, has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced internetworking technologies since 1990. More content * Webinars * Podcasts * Books * Articles * Presentations Shortcuts * Home * Blog * Even more content * Contact Subscribe to * Blog posts * Software Gone Wild podcast * Videos * SDN mailing list * Follow @ioshints Disclaimer The opinions expressed in individual articles, blog posts, videos or webinars are entirely the author's opinions. Read more Copyright 2006 - 2022 ipSpace.net AGcounter stats