[HN Gopher] Ask HN: How do carriers throttle streaming to 480p r...
___________________________________________________________________
Ask HN: How do carriers throttle streaming to 480p resolution?
I have seen that most carriers, by default, have a "stream saver"
turned on, which, I am assuming detects when you're on YouTube or
Netflix and automatically throttles your bandwidth to these sites.
Assuming these connections are happening over SSL, how are the
carriers able to modify the sites to disable users from selecting
HD or 4k resolutions?
Author : akouri
Score : 69 points
Date : 2022-02-03 14:29 UTC (8 hours ago)
| dhuertas wrote:
| MNOs really don't need to know the traffic source address to
| apply throttling.
|
| As per the throttling algorithm, most of the times it's a Leaky
| Bucket variant (https://en.wikipedia.org/wiki/Leaky_bucket). The
| variant usually allows short bursts of packets, then throttles
| down the downstream traffic for long connections to match the
| configured rate.
|
| A trick to know if the operator is using DPI to extract the SNI
| in HTTPS/encrypted traffic: play a YouTube video, then do a speed
| test (e.g. iperf) while it is playing. Two things could happen:
| either both apps are throttled (no DPI) or only Youtube is
| (there's some level of DPI).
| fumar wrote:
| I assume its domain based. When I use a VPN for video on T-mobile
| there is no resolution or speed decrease. I've tested this
| several times with 4g and 5g areas and browser vs. apps like
| YouTube.
| s800 wrote:
| I don't think this can be the case across the board. I control
| the reverse DNS for my cable modem IPs and still see the
| downgrades during peak-neighborhood hours.
| deepsun wrote:
| But I have another question -- are ISPs allowed to do that,
| considering net neutrality? At least in countries/states where
| it's still alive?
| ars wrote:
| At least with T-Mobile it's a checkbox, you can turn it on or
| off as you choose, and they give you data benefits (such as
| zero rating or extra data) if you turn it on.
| Bedon292 wrote:
| I think some of it is optional, they pitch it as a data saver
| so you don't run out of data. Or tell you can get unlimited
| streaming data as long as its low resolution. And for QoS
| reasons, to keep bandwidth available for everyone. I think they
| are attempting to do it across the board for all video, so its
| not prioritizing one specific service over another. And remains
| semi neutral, I guess.
| HWR_14 wrote:
| What's really interesting about TMobiles plan is(was) that they
| would zero rate any video service that allowed them to set the
| steam to 480p. I don't know if small players ever made it
| through the process, but that universal option may have been
| legally required to avoid that same question.
| scarface74 wrote:
| Someone posted on HN that they just submitted an application
| to T-mobile to have their non profit video stream zero rated
| HWR_14 wrote:
| In this Ask HN? I'd like to find that post.
| scarface74 wrote:
| It was a long time ago.
| AtNightWeCode wrote:
| Yes, in some countries there is even free traffic for some
| services. Carriers have free Spotify traffic and so on.
| zokier wrote:
| They could be just cooperating with the major streaming services.
| AtNightWeCode wrote:
| I think the most honest reply is that the open Internet does not
| exist in the western world. Read that again. All major traffic
| from MS, Netflix and so on is already very segregated. There are
| several locations were any provider can do QoS in layer 7.
|
| And do not underestimate the power of carriers. They are the
| reason why you can not use mobiles on a plane.
| sgt101 wrote:
| Contention happens in many places in the pipe, it can happen
| because of the path or because of a fault. It can happen inside
| the streaming providers infrastructure or in the egress. The
| delivery system adapts to the bandwidth that is available and
| chooses all sorts of ways to deal with this.
| phkahler wrote:
| Sometimes my Amazon firestick/prime video goes to crap as if my
| network connection was really bad. I hop on my google wifi gizmo
| and run a speed test and get 100-200mbps down. Then sometimes the
| movie starts playing just fine. It's like they want to throttle
| my connection unless I'm looking.
| dbrueck wrote:
| It is somewhat rare but not entirely unheard of for carriers to
| have agreements with the major streaming providers, and so in
| some cases the streaming provider will actually aid in the
| process. Most of the time, however, this isn't needed because
| what's actually happening is limiting bandwidth, not resolution,
| but there is a very strong correlation between the two, so by
| limiting bandwidth you end up with the desired effect of limiting
| resolution.
|
| These days most streaming providers use some form of adaptive
| streaming in which client-side logic decides to get bigger or
| smaller "chunks" of video based on how quickly prior chunks
| downloaded. A rudimentary solution for a carrier would be to
| simply implement logic like, "if throughput to device X >
| someLimit, add a delay in delivering packets to device X". From
| the client's perspective, getting the bigger (and higher quality)
| video chunks will take too long, so it will naturally shift to
| the smaller (and lower quality) chunks.
| exikyut wrote:
| Two clarifying questions:
|
| 1) What does the feature look like, screenshot-wise?
|
| 2) Can you confirm the HD/4K option actually disappears or is
| disabled, or if the site(s) in question just trend toward
| autoselecting 480p/720p over time?
|
| Like most other comments here I suspect IP-based bandwidth
| limiting. Given the unbounded complexity scale of keeping the
| internet actually working :) I can totally see infrastructure
| being able to single out the activity of a single connection and
| track what it's doing over time. The chances are the
| implementation is eyebrow-raisingly impressive but still compact
| and approachable at the end of the day.
| jiripospisil wrote:
| How do mobile carriers know video resolution over HTTPS
| connections?
|
| https://security.stackexchange.com/questions/172212/how-do-m...
| exikyut wrote:
| The top-rated answer there is IMHO an excellent response to
| OP's question.
| bluedino wrote:
| > Verizon doesn't LITERALLY recognize and block encrypted HD
| video... but it has agreements with services like Netflix and
| Youtube that require THEM to recognize Verizon IP address
| ranges & respect any limits imposed by Verizon.
|
| This is an interesting 'answer', but how would 'tiered'
| unlimited plans work if it was handled by the streaming service
| and not the ISP? Some allow unlimited 480 while the higher
| priced plans allow unlimited HD.
| exikyut wrote:
| I can only echo the sole comment under that answer:
|
| > _Do you have a reference /proof for the first paragraph
| about agreements?_
|
| ---
|
| IMHO, I find this hard to believe myself. The team(s) in
| charge of implementing this functionality likely don't have
| the requisite high level of lateral coordination necessary to
| facilitate agreements with all the major streaming services.
|
| Practically speaking, I see this sort of thing as balancing
|
| 1) total network saturation/hard capacity limits
|
| 2) encouraging user base to use more data up to thresholds in
| (1)
|
| 3) customer experience (wrt data overages)
|
| likely in that order.
|
| This functionality isn't a first-class user-facing "generate
| piles of money"-center; it describes limitations and
| conservatism and risk balance. It isn't a "pedal to the
| floor" sort of subject, but more of a continuous optimization
| concern.
|
| I don't really see telcos going to all the streaming services
| and working out agreements given this sort of status
| quo/sentiment. It's more the sort of thing that is kept
| internal.
|
| And then of course there's the fact that this would be
| playing out for every telco relative to every streaming
| service.
|
| At that point everyone would just throw their hands up in the
| air and make a global standard for cooperative ratelimiting,
| and we'd all know about it. For the first few years it would
| be obscure and you'd have to google the terms a few times to
| correctly identify it, and then someone would put a 43%
| quality JPEG screenshot of the documentation on facebook and
| it would go viral and get linked to 5G causing cancer or
| whatnot.
|
| It's just IP ratelimiting.
| Bedon292 wrote:
| Many ISPs do have peering agreements with Netflix[1] and
| the like. So they may be they are able to shape the traffic
| that way as well. All your Netflix traffic is going through
| that connection, so even if its not on Netflix side your
| ISP knows to throttle to the bandwidth that will get the
| resolution they want you to have.
|
| [1] https://openconnect.netflix.com/en/
| wongarsu wrote:
| The ISP could use different IP ranges for the different
| tiers, and just communicate those ranges to the streaming
| providers they have agreements with.
| JoshTriplett wrote:
| They could also handle this by routing packets from
| different customer tiers to different hosts that all handle
| the same TLS hostname. Or, even, to different interfaces on
| the same Netflix CDN box they have sitting in a local rack
| caching everything. That wouldn't break TLS, because all of
| those hosts work as a TLS endpoint for the same host.
| lelandfe wrote:
| Tl;dr streaming video protocols often have unique traffic
| shapes, and you can also make reasonable guesses as to the
| resolution of the served video. After that, the provider simply
| throttles bandwidth of the connection.
|
| What a great post.
| unfocussed_mike wrote:
| You learn something new every day... ace.
| betterunix2 wrote:
| They do not know, but they also do not need to know.
| Technically they are not limiting resolution, they are limiting
| throughput on their network; the streaming service will reduce
| the resolution to avoid long lag/loading times for users.
| Bedon292 wrote:
| The urls that are used for the streams are pretty well known, and
| can limit all traffic on those URLS to specific speeds. Here are
| a few for example:
|
| Twitch: *.ttvnw.net
|
| Netflix: *.nflxvideo.net
|
| Hulu: *.hulustream.com
|
| YouTube: *.googlevideo.com
|
| Amazon Prime: *.aiv-cdn.net
|
| Edit: This is by no means the only way to do it, just a potential
| way to do it.
| joecool1029 wrote:
| This is mostly the correct answer, some creative Plex users
| with shitty cable ISP's (lookin at you Optimum) will run their
| connections through Cloudflare to avoid upstream throttling of
| video. It all looks like TLS traffic at the end of the day and
| since Cloudflare is probably the most common CDN, they can't
| apply rules to this range.
|
| For mobile ISP's like T-Mobile, they cannot tell that video
| content is streamed through a VPN protocol like wireguard (but
| they could add large public providers to the list).
|
| As a good rule of thumb I tell people to run speedtests from
| both speedtest.net (which is usually whitelisted by every
| provider) and fast.com (which usually shows up as Netflix video
| traffic) to see if 'video shaping' rules are in effect.
| watermelon0 wrote:
| FYI Cloudflare terms of service (at least for free accounts)
| doesn't allow using it for video streaming purposes such as
| Plex:
|
| https://community.cloudflare.com/t/is-plex-allowed/172158/5
| joecool1029 wrote:
| Thanks for the update, previously (before 2020) it was
| believed that if caching was turned off and it was purely
| routing over Cloudflare, they wouldn't get pissy over the
| extra traffic. I guess this shows they will anyway.
| dylan604 wrote:
| Don't forget that Netflix has it's own CDN, so you could just
| throttle anything from that peering connection knowing that
| only video content is coming from it.
|
| Oh, wait, they've already been doing that for <insert Gary
| Oldman> EVERYONE!</insert>
| 3pt14159 wrote:
| But isn't this over HTTPS? I know the DNS lookup may leak the
| domain, but that's heavily cached.
|
| I think its just network analysis or possibly even those
| caching servers that are run by ISPs which hold onto content
| network's most heavily used files.
| spapas82 wrote:
| But the provider knows where you are connecting, no matter if
| https is used or not. Https will encrypt the payload/data
| (layer 7) not where you are connecting (layers 3/4)
| nitwit005 wrote:
| They likely have a peering setup with each of these
| companies, so they know the IP address ranges they're using
| for this video traffic. They can tag the traffic and shape it
| (drop packets in excess of some rate).
| desdiv wrote:
| I see two possible ways:
|
| 1. during the TLS handshake, the domain name itself might be
| sent in the clear if the SNI extension is used, and if the
| SNI extension isn't encrypted[1]
|
| 2. if the carrier knows the IP, then they can do a reverse
| DNS lookup to find the domain name
|
| [1] https://stackoverflow.com/questions/499591/are-https-
| urls-en...
| sumtechguy wrote:
| The packet header would leak that info too with src/dst IP
| addresses.
| brewmarche wrote:
| SNI leaks the host name over HTTPS. Also, if the IP addresses
| are distinct enough they could be used to detect the
| destination as well.
| dtech wrote:
| Note that isn't the case any more with modern browser +
| server, TLS 1.3 Encrypted Client Hello (ECH) fixes this.
| caskstrength wrote:
| Last time I checked TLS Encrypted Client Hello standard
| was still under heavy development and implementation in
| Firefox was disabled by default. Is this not the case
| anymore?
| silon42 wrote:
| IMO, it's a bad design... SNI encryption should be done
| by firewall, not the client.
| hammock wrote:
| Would a vpn work around this?
| betterunix2 wrote:
| Yes in my experience.
| rwc wrote:
| Yes, the challenge is some of those sites don't allow
| streaming on a VPN.
| EveYoung wrote:
| If you have enough upload bandwidth, you could set up a VPN
| server at home.
| apple4ever wrote:
| Which is what I do. I run pfSense (will switch over
| someday to opnSense but its what I have now) and just use
| the built in OpenVPN server. Works very well.
| flerchin wrote:
| I don't think we can assume that the video is transferred over
| SSL.
| dbrueck wrote:
| These days it is nearly 100% of the time because browsers will
| put up lots of scary warnings for non-HTTPS requests (and most
| video these days is delivered in chunks over HTTPS instead of
| custom or lower-level protocols).
| enduser wrote:
| Most large ISPs are running CDN nodes for the streaming providers
| out of their own NOCs. Those nodes are provided by the streamers
| and work with the ISP's QoS policies.
| tgsovlerkhgsel wrote:
| I thought at least some of the services offered a DNS-based
| approach (similar to Google's "forcedsafesearch" cname -
| https://support.google.com/websearch/answer/186669?hl=en), but I
| couldn't find any documentation.
|
| Possible otheroptions:
|
| - agreements with the service providers to throttle users from
| certain netblocks (the carriers partner with the service
| providers to some extent e.g. to deploy CDN nodes, so such
| agreements would be plausible)
|
| - throttling bandwidth (potentially selectively to/from streaming
| providers) and letting the service figure it out
|
| - separate host names for high res content that can be DNS-
| blocked
| znpy wrote:
| I'd try two things:
|
| 1. try and see if accessing youtube through a vpn improves the
| bandwidth (in that case, your ISP is probably looking at both dns
| requests and connection endpoint ownership)
|
| 2. preload and buffer whole videos (es:
| https://www.technorms.com/35122/preload-buffer-entire-youtub...)
| this aims to get your traffic usage pattern not having the
| "usual" shape of a typical youtube session (that is: brief burst
| of full-speed downloads)
| PaulHoule wrote:
| Even if they can't look into an SSL connection they can measure
| the amount of data going through it and rate limit it.
| mnw21cam wrote:
| And they can look at the IP address and see if that matches
| their list of streaming servers.
| unfocussed_mike wrote:
| I guess this is the $5 hammer.
| jerf wrote:
| A streaming video connection will also have a fairly
| characteristic bandwidth usage pattern. Tracking the requisite
| state is a bit tricky, but doable. This would allow them to
| throttle anything that looks like video streaming.
|
| Whether or not they do this, I don't know. But it is
| technically feasible.
|
| Now, _technically_ , it is probably pretty obvious that what
| they ought to do is just limit bandwidth or not, regardless of
| what it is used for. But the rest of the business, and for that
| matter the rest of the _world_ , conceives of video streaming
| as something different than "just using bandwidth", so they'll
| sign contracts about how video streaming will be treated better
| or worse depending on this or that condition ("Free netflix as
| long as we can throttle it", "we'll prioritize Disney+ because
| they paid us and everyone else gets degraded", etc.), so even
| if it makes no "logical" sense, network hardware will just have
| to figure out what "video streaming" is to fulfill the
| contracts whether the engineers like it or not.
| PaulHoule wrote:
| Most of the common protocols for streaming video involve
| making a series of http requests to fetch short segments of
| video from a server. If there wasn't pipelining or QUIC
| requests going on it would be pretty obvious what was going
| on, but maybe less so if the requests were spread out across
| a number of servers.
|
| With pipelining or some other protocols the signature is
| going to be a sustained download that runs at a rate less
| than the maximum possible.
|
| If the servers are being found through DNS that's another
| tool to figure out what is going on. If not, the carrier is
| going to want a list of IP blocks associated with video
| streaming.
|
| What I don't get is customers caring a lot about whether they
| are getting 480p vs better. The old NTSC video was outright
| awful, and the difference between that and 480p is greater
| than that between a clean 480p and 720p, 1080p or even 4k. In
| the best circumstances the returns are diminishing but if you
| are viewing on a little phone screen in bright light what's
| the point?
| jerf wrote:
| I wasn't even thinking of looking at where the traffic is
| going. You end up with a very characteristic pattern of
| bandwidth usage that can be detected, on that basis alone.
|
| It'll false positive and false negative sometimes but on
| average it'll be good enough. We're not talking cutting
| connections off, just throttling. If something other than
| video streaming briefly slows down, who's going to know to
| blame a false positive on the streaming throttling?
| Piskvorrr wrote:
| Well there's your answer, right in the question: " _if_ you
| are viewing on a little phone screen in bright light ".
| OTOH, if you are viewing on a big screen (such as, IDK, you
| might find on a TV), the resolution and compression
| artifacts become not just visible, but magnified.
| qwerty456127 wrote:
| I hope this doesn't happen to videos in which people teach to
| code or teach to use particular apps - 480p can be insufficient
| to read display text reliably if recorded at full HD.
| WesSouza wrote:
| I use Visible and noticed this.
|
| As soon as I enable a VPN (I use PIA), speeds go back to normal.
| [deleted]
| jrrrr wrote:
| I assume carriers limit only the _bandwidth_, and count on
| streaming apps to react by automatically degrading the
| resolution.
| 3pt14159 wrote:
| I don't use Apple TV much, but one of the things I like about
| it is how it lets me download the video file locally. The
| quality is so, so crisp. I pay for the more expensive version
| of Netflix, but it's not quite the same start to finish.
| tomc1985 wrote:
| Piracy is also great for this
| gattopalla wrote:
| Agreed
| jrockway wrote:
| Yeah, I hate watching 15 minutes of something at low quality
| because one TCP packet was dropped. Paranoia leads to the
| most user satisfaction ("at least the show didn't freeze")
| but I would rather buffer the thing locally and watch in
| uninterrupted 4k @ 50Mbps or whatever. (Also, what's the
| point of gigabit internet if I can't watch a measly 50Mbps
| stream?)
| beardedetim wrote:
| Most players that use adaptive bit rate allow you to set it
| at a constant one instead, with the buffering and all.
|
| I feel the same way and often click a specific rendition
| and then let it buffer locally.
| OJFord wrote:
| But Netflix (which GP was talking about implicitly by the
| comment they replied to) doesn't.
| douglasisshiny wrote:
| Sure, but Netflix uses HLS, I assume, and doesn't have
| segments that are 15 minutes in length. So if you did
| drop a tcp packet, you're more likely to have lower
| quality for 10-20 seconds, not 15 minutes.
| Mindwipe wrote:
| HLS to Safari/iOS and Dash to everything else.
| belthesar wrote:
| Sure, if you wanna pay the DASH tax to a hundred patent
| holders. DASH doesn't even really hold much of a
| technological benefit anymore now that their's finally a
| standard for HLS-LL.
| drclau wrote:
| Netflix mobile apps allow you to download most content, too,
| so you can compare the quality to streamed content. However,
| then you are limited to phone/tablet screen size for
| comparison.
| jhasse wrote:
| The Windows app allows downloads, too.
| wongarsu wrote:
| Many Android phones work with USB-C monitors. Just plug it
| in and your phone screen is mirrored on the monitor.
| Mindwipe wrote:
| Netflix disables HDMI out on it's Android and iOS apps
| with downloaded content, this won't work.
| closewith wrote:
| I only wish their Apple TV hardware allowed you to do this.
| dhosek wrote:
| I thought it does--we rented a movie with Apple TV at one
| point a couple years ago and waited for it do download to
| local storage before it was played. Unfortunately, I had
| opted for highest quality at a time when our network
| connection was being really flaky and it took a day to do
| so.
| Mindwipe wrote:
| You used to be able to for HD, you've never been able to
| for UHD.
| Pulcinella wrote:
| You cannot unfortunately: https://support.apple.com/en-
| us/HT211825
|
| Maybe certain streaming services allow you to download as
| well though their app on the Apple TV, but most of the
| big ones don't (Apple, Netflix, HBO, etc.)
|
| I really don't like it because, besides wanting to pre-
| download something for higher quality, there are often
| certain shows and movies my kids want to watch over and
| over and they might as well just live on the hard drive.
|
| So I have a 64GB Apple TV and most of the storage is
| empty. Theoretically it's for games, but ATV games are
| limited to 100MB and then anything else has to be
| downloaded after the game has installed. (In practice
| there aren't any games near that big). I suppose device
| might also progressively download the video in the
| background and that would use storage, but that's all an
| implementation detail and entirely up to the heuristics
| of the device.
| dhosek wrote:
| Ooh, I hadn't thought about the streaming services. I was
| really annoyed to discover that a movie I'd downloaded to
| my iPad from Disney+ would not play if the iPad was
| connected to a TV which was damned obnoxious. This was a
| rental from the Apple TV store and not an Apple+ stream,
| so maybe that's where the difference comes in.
| bluedino wrote:
| Not likely as a common workaround was streaming over a VPN
| service or SSH tunnel.
| rahimnathwani wrote:
| They could throttle bandwidth selectively, based on
| destination IP address range.
| Mindwipe wrote:
| In Netflix's case they don't even need that, the content
| will be coming from an OpenConnect cache in their premises.
| jtbayly wrote:
| No, that still makes sense. They limit the bandwidth to/from
| a particular domain. The VPN hides that. SSL doesn't.
| franga2000 wrote:
| That actually makes it more likely. They're limiting speed to
| certain IP ranges, corresponding to streaming provider
| servers. If you connect via a tunnel, the destination address
| is different and therefore your traffic isn't throtteled.
| jakub_g wrote:
| This; "adaptive streaming" is the keyword. A video on the web
| is actually a series of mp4 files which are a few seconds
| length each. They are generally assembled at runtime via
| JavaScript or native technology into continuous playback. The
| browser/app monitors whether chunks are downloaded fast enough
| to assure continuous playback. That's also why often streams
| start in low quality (to minimize buffering time) and once the
| app knows it can go to higher resolution, it switches at the
| next boundary between the chunks.
| asow92 wrote:
| HLS handles this well.
| redisman wrote:
| HLS automatically will download a smaller res chunk if it can't
| keep up. Video players (and some formats) react to strangled
| bandwidth
___________________________________________________________________
(page generated 2022-02-03 23:01 UTC)