https://adriano.fyi/post/2023/2023-04-16-att-traffic-shaping-makes-websites-unusable/ Toggle navigation Adriano Caloiaro * About Me AT&T Wireless traffic shaping apparently making some websites unusable Posted on April 16, 2023 | Adriano When I'm living in my RV, wireless service providers are my primary source of connectivity. So when either AT&T or Verizon make major changes, I take notice. I recently noticed that multiple websites are quite slow when browsing with my AT&T business plan, listed in AT&T Premier (business account management UI) as "Wireless Broadband Ultra for Router or Hotspt (sic)". This is an "unlimited" 100Mbit plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. To be honest, I've never noticed a discernible difference between Fast Track and non-Fast Track data rates. This is all to say that I have no reason to believe that I'm being deprioritized due to usage. Naturally, the first hing I did was conduct a speed test. I already knew from previous experience that for some reason, AT&T traffic to fast.com is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress. Linode.com has speed tests that AT&T has no special treatment for, and the nearest one to me was in Fremont, CA. The speedtest revealed 21Mbps down and 4.5Mbps up - pretty reasonable in a relatively rural area like Durango, CO. Latency was ~130ms. That speed certainly wouldn't explain why it took anywhere between 15 seconds and 2 minutes to load strava.com. So I opened up the "Network" tab in Firefox and could clearly see that dozens of resources from cloudfront.com were taking multiple seconds to load. The problem clearly has something to do with Cloudfront. [ ] Firefox network tab loading strava.com Is Cloudfront having problems? That's easy enough to verify; my Sierra Wireless RV55 CAT-12 LTE-A router also contains an unlimited Verizon Business SIM card that I can use to conduct tests on Cloudfront, independent of AT&T. I noticed that one of Strava's javascript resources clocked in at 1.68MB, making it a nice test subject for speed tests (https:// web-assets.strava.com/assets/federated/find-and-invite-friends/827.js ). At the time of writing web-assets.strava.com resolves to dgpcy4fyk1eox.cloudfront.net, so rest assured, we are dealing with Cloudfront. After switching to Verizon, I could see that Cloudfront was having no problems. Our friend 827.js downloaded in just over 1 second at 1.4MB /s. I clearly saw earlier in the Firefox network tab that this resource took nearly 1 minute to load on AT&T. While wget is not my goto for command line HTTP fetching, it displays transfer rates in a human friendly format by default, so I used the following as my test case: wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/ find-and-invite-friends/827.js So the problem isn't Cloudfront, because Verizon was fast enough. It wasn't blazing fast by any means, but I also didn't have to wait 2 minutes to learn whether Strava awarded me King of the Moutain on a local trail (I wasn't). Maybe this is a global AT&T problem. That's easy to test as well - my iPhone is also on the same AT&T business account as my data-only plan, so I turned on the iPhone hotspot and made it the router's WAN device to make sure we're changing a single variable at a time. I conducted another speed test, revealing 23Mbps down and 3Mbps up. Nothing surprising there - normal bandwidth fluctuations for a wireless device. How about our wget test? 1.7MB/s. The problem is clearly not all of AT&T wireless. Let's go back to the original configuration: connect directly to my AT&T data-only plan with my router and re-run the wget test. Maybe I was imagining things. I'm somewhat surprised to see the wget test with a transfer rate of ~30KB/s. I believe this rules out AT&T with a souring peering agreement somewhere between me and Cloudfront. My phone traffic to Cloudfront is unaffected; it's only my data-only plan that is affected. I now have a pretty clear picture of what is likely happening. Good old-fashion traffic shaping. Now that the router is connected directly to AT&T, the true test of traffic shaping is transfer rates while connected to a VPN. I'll let the image speak for itself. [ ] Traffic shaping in progress --------------------------------------------------------------------- As of the time of writing, I'm unsure what is causing such a significant slowdown. It has rendered some websites effectively useless. Everything in this writeup indicates, to me, that AT&T is engaged in extremely aggressive traffic shaping for some plans, rendering many websites nearly unusable. Do you have any ideas how to diagnose this problem further? Do you know the best way engage AT&T's technical folks to take this seriously? Write me at att-traffic-shaping @ this domain. I'll add updates here if anything changes, or I get a response on https:// bizcommunity.att.com. Updates --------------------------------------------------------------------- Update #1 (19:03:09 UTC) Here is a pcap file captured with sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w capture.pcap: capture.pcap I've also attempted to disable/enable IPv6 on my local machine and my VPN connection to check for any differences. No differences were observed. Update #2 (19:43:00 UTC) Traceroutes Connected to AT&T iPhone via Wifi hotspot traceroute web-assets.strava.com 130 | traceroute to web-assets.strava.com (99.84.208.10), 30 hops max, 60 byte packets 1 _gateway (172.20.10.1) 4.386 ms 4.280 ms 5.292 ms 2 107.243.82.3 (107.243.82.3) 139.792 ms * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 12.117.216.210 (12.117.216.210) 244.538 ms 369.608 ms 933.487 ms 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 server-99-84-208-10.iad79.r.cloudfront.net (99.84.208.10) 206.664 ms 206.643 ms 206.623 ms Connected to router on AT&T business plan traceroute web-assets.strava.com traceroute to web-assets.strava.com (99.84.208.115), 30 hops max, 60 byte packets 1 _gateway (192.168.13.31) 61.896 ms 61.950 ms 62.064 ms 2 172.26.96.161 (172.26.96.161) 132.156 ms 132.775 ms 132.749 ms 3 107.72.231.188 (107.72.231.188) 132.743 ms 132.718 ms 132.199 ms 4 * * * 5 12.83.179.49 (12.83.179.49) 148.138 ms 148.187 ms 148.162 ms 6 slkut21crs.ip.att.net (12.122.1.186) 148.361 ms 137.643 ms 142.401 ms 7 dvmco22crs.ip.att.net (12.122.28.45) 146.657 ms 146.627 ms 149.389 ms 8 cgcil21crs.ip.att.net (12.122.28.78) 149.369 ms 149.345 ms 216.643 ms 9 12.122.28.206 (12.122.28.206) 149.302 ms 146.467 ms 149.256 ms 10 wshdc84crs.ip.att.net (12.122.135.230) 146.421 ms 149.211 ms 149.161 ms 11 wshdc406me9.ip.att.net (12.123.10.125) 143.819 ms 139.466 ms 139.378 ms 12 12.117.216.210 (12.117.216.210) 139.342 ms 139.314 ms 187.019 ms 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 server-99-84-208-115.iad79.r.cloudfront.net (99.84.208.115) 152.577 ms 140.190 ms 141.460 ms Update #3 (21:54:03 UTC) Here is a pcap file captured with sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w iphone-capture.pcap while tethered to my AT&T iPhone and fetching 827.js with wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/ find-and-invite-friends/827.js : iphone-capture.pcap wireless at&t * - Previous Post * * Adriano * (c) 2023 * Adriano Caloiaro Follow on Mostodon Hugo v0.107.0 powered * Theme Beautiful Hugo adapted from Beautiful Jekyll