[HN Gopher] Netflix buffering issues: Boxing fans complain about...
___________________________________________________________________
Netflix buffering issues: Boxing fans complain about Jake Paul vs.
Mike Tyson
Author : storf45
Score : 348 points
Date : 2024-11-16 03:42 UTC (19 hours ago)
(HTM) web link (www.sportingnews.com)
(TXT) w3m dump (www.sportingnews.com)
| suzzer99 wrote:
| On a few forum sites I'm on, people are just giving up. Looking
| forward to the post-mortem on how they weren't ready for this
| (with just a tiny bit of schadenfreude because they've
| interviewed and rejected me twice).
| itsthecourier wrote:
| They sabotaging OP just for a reverse schadenfreude play
| Amfy wrote:
| Can you share which forums
| suzzer99 wrote:
| Chiefsplanet.com, unstuckpolitics.com, my buddies on group
| text :)
| boppo1 wrote:
| /r/netflix and /sp/
| dyauspitr wrote:
| It's not everyone. Works fine for me though I did have to
| reload the page when I skipped past the woman match to the
| Barrios Ramos fight and it was stuck buffering at 99%.
| icameron wrote:
| You skipped the best part.
| moralestapia wrote:
| The post-mortem will be interesting indeed.
| scruple wrote:
| AB84 streamed it live from a box at the arena to ~5M viewers on
| Twitter. I was watching it on Netflix, I didn't have any
| problems, but I also put his live stream up for the hell of it.
| He didn't have any issues that I saw.
| almost_usual wrote:
| > He didn't have any issues that I saw.
|
| He's definitely got issues..
| HDThoreaun wrote:
| were calling antonio brown ab84 now? What happened to Mr. BC?
| tiffanyh wrote:
| Dupe: https://news.ycombinator.com/item?id=42153906
| josh2600 wrote:
| FWIW, works fine for me.
| iancmceachern wrote:
| Us too
| roamerz wrote:
| Been working great for me as well. Starlink in Oregon.
| whoknowsidont wrote:
| Please don't make these types of comments, they mean nothing
| and they serve no purpose.
| pentagrama wrote:
| It means, it is different if the service goes down to 100,
| 50, 10 percent of users. I watched the show with no issues.
| whoknowsidont wrote:
| Comments on forums do not provide that data. And if you
| want to extrapolate self-reports, it's obviously fine (to
| varying degrees) for the vast majority of people, but
| that's not the "issue."
|
| These kind of reports are the equivalent of saying "I have
| power" when you're hundreds of miles away from where a
| hurricane landed. It's uninteresting, it's likely you have
| power, and it does literally nothing for people that do not
| have power.
|
| It doesn't advance the topic anywhere. There are other
| places to report these issues directly and in aggregate
| with other people -- HN (and many other forums) are not
| that place.
| kortilla wrote:
| You're in a fucking thread about people commenting in a
| forum about an outage that may or may not have been
| caused by Netflix
| whoknowsidont wrote:
| Which has many interesting facets worthy of discussion!
| No need to be extremely aggressive in your tone.
| kortilla wrote:
| You are the one telling people their comments provide no
| value. That's far more aggressive than using a word not
| even directed at you
| rascul wrote:
| Why is it acceptable to share that it doesn't work but not
| acceptable to share that it does?
| addicted wrote:
| For the same reason that pointing out the sun rose in the
| east today would be ridiculous but if it happened to rise
| in the west, or you perceived it to rise in the west, that
| would be worth sharing.
|
| Being able to livestream a sporting event is the default
| now and has been for at least over a decade since HBO's
| original effort to stream a Game of Thrones season opener
| failed because of the MSFT guy they hired, and they fixed
| it by handing it over to MLBAM.
|
| Maybe that's what Netflix should do. Buy Disney so they can
| get access to the MLBAM people (now Disney Streaming
| because they bought it from MLB).
| gbil wrote:
| The stream never buffered on my side but quality was for the
| whole duration of the stream pretty basic I doubt it was even
| 720p
| Justin_K wrote:
| I love how I can come to HN to instantly find out if it's Netflix
| or my WiFi.
| jfdi wrote:
| Right?!
| slicktux wrote:
| This! I was checking my WiFi and then I instinctively checked
| HN and what do you know!
| Jiahang wrote:
| metoo!
| dghughes wrote:
| Wifi wifi or wifi as in your ISP Internet connection? Sp many
| people now call an Internet connection "wifi".
|
| Anyway, network cable is the only way to go!
| anakaine wrote:
| "Many people", those who call their ISP connection WiFi, are
| technology potato's.
| fred_is_fred wrote:
| To be fair, a lot of people pay their ISP for a
| modem/router combo and connect to something like "Xfinity"
| at their house. So to them, there is no difference.
| rascul wrote:
| I have the wifis with the geebees.
| bluSCALE4 wrote:
| Looks like shit for me. Buffered a bit as well.
| freditup wrote:
| I wonder if there will be any long term reputational
| repercussions for Netflix because of this. Amongst SWEs, Netflix
| is known for hiring the best people and their streaming service
| normally seems very solid. Other streaming services have
| definitely caught up a bit and are much more reliable then in the
| early days, but my impression still has always been that Netflix
| is a step above the rest technically.
|
| This sure doesn't help with that impression, and it hasn't just
| been a momentary glitch but hours of instability. And the Netflix
| status page saying "Netflix is up! We are not currently
| experiencing an interruption to our streaming service." doesn't
| help either...
| ocdtrekkie wrote:
| So the issue is that Netflix gets its performance from
| colocating caches of movies in ISP datacenters, and a live
| broadcast doesn't work with that. It's not just about the sheer
| numbers of viewers, it's that a live model totally undermines
| their entire infrastructure advantage.
|
| See: https://openconnect.netflix.com/en/
| pinkmuffinere wrote:
| Damn that sucks. I wonder if they could have intentionally
| streamed it 5 min late? I don't have all the context around
| the fight though -- maybe a competing service would win if
| Netflix intentionally induced delay?
| adrr wrote:
| they were introducing 5 minute delays on some of the
| clients. I noticed my ipad was always live and the smart tv
| had a 5 minute delay but you could fast forward to live.
| stingraycharles wrote:
| Correct, this is not Netflix' regular cup of tea, and it's a
| very different problem to solve. They can probably use their
| edge caches, but it's challenging.
| nicce wrote:
| How YouTube does this? Netflix is like drop in the ocean
| when compared to.
| Bilal_io wrote:
| Not sure how Netflix does it. But this is not very time
| sensitive, and I would have delayed the stream by 15 to
| 30 seconds to cache it and then deliver to everyone.
| unsnap_biceps wrote:
| My wild assed guess is the differences in the edge nodes.
|
| Netflix's edge nodes are optimized for streaming already
| encoded videos to end users. They have to transcode some
| number of formats from the source and send them all to
| the edge nodes to flow out. It's harder to manage a ton
| of different streams flowing out to the edge nodes
| cleanly.
|
| I would guess YouTube, being built on google's
| infrastructure , has powerful enough edge nodes that they
| stream one video stream to each edge location and the
| edges transcode for the clients. Only one stream from
| source to edge to worry about and is much simpler to
| support and reason about.
|
| But that's just my wild assed guess.
| vitus wrote:
| > I would guess YouTube, being built on google's
| infrastructure , has powerful enough edge nodes that they
| stream one video stream to each edge location and the
| edges transcode for the clients.
|
| Ha, no, our edge nodes don't have anywhere near enough
| spare CPU to do transcoding on the fly.
|
| We have our own issues with livestreaming, but our
| system's developed differently over the past 15 years
| compared to Netflix's. While they've historically focused
| on intelligent pre-placement of data (which of course
| doesn't work for livestreaming), such an approach was
| never feasible for YT with the sheer size of our catalog
| (thanks to user-generated content).
|
| Netflix is still new to the space, and there isn't a good
| substitute for real-world experience for understanding
| how your systems behave under wildly different traffic
| patterns. Give them some time.
| kortilla wrote:
| It also helps that youtube serves shit tier quality
| videos more gracefully. Everyone is used to the step down
| to pixel-world on youtube to the point where they don't
| complain much.
| Ekaros wrote:
| And decent part of these users are on free tier, so they
| are not paying for it. That alone gives you some level of
| forgiveness. At least I am not paying anything for this
| experience.
| spike021 wrote:
| In my experience even YouTubeTV has problems sometimes.
| I'll have the 1080p (and enhanced mode also I think)
| quality set and still deal with a lot of compression
| artifacts.
| sgarland wrote:
| I wonder how effective it would be to cache live events with
| a delay. Write to the tail, read from the head.
| thefreeman wrote:
| that's totally unacceptable for live sports which people
| are able to bet on
| dullcrisp wrote:
| Live sports require microwave relays for high frequency
| sports bets
| rk06 wrote:
| Why should they catering to such an audience in first
| place?
|
| I think this could be one of upsells that Netflix could
| use.
|
| Premium: get no delay
|
| Normal users: get cache and delay
| ctvo wrote:
| Or, hear me out here, it's a wild concept, just work.
|
| You know, like every other broadcaster, streaming
| platform, and company that does live content has been
| able to do.
|
| Acting like this is a novel, hard problem that needs to
| be solved and we need to "upsell" it in tiers because
| Netflix is incompetent and live broadcasting hasn't been
| around for 80+ years is so fucking stupid.
| kortilla wrote:
| Every other live platform has a delay of multiple seconds
| Brybry wrote:
| This is kind of silly because the delay between actual
| event happening to showing up on OTA TV or cable TV to
| showing up on satellite TV can already be tens of
| seconds.
| JCharante wrote:
| isn't this why people would listen via radio?
| kortilla wrote:
| I have bad news for you. This is how it works already for
| "live" sports
| YZF wrote:
| Yep. Having actually worked on this sort of stuff I can
| confirm.
|
| Your ISP doesn't have enough bandwidth to the Internet
| (generally speaking) for all users to get their feed
| directly from a central location. And that central
| location doesn't have enough bandwidth to serve all users
| even if the ISP could. That said, the delay can be pretty
| small, e.g. the first user to hit the cache goes
| upstream, the others basically get the stream as it comes
| in to the cache. This doesn't make things worse, it makes
| them better.
| jrpelkonen wrote:
| Correct. Here are some latency numbers from the last
| SuperBowl: https://www.phenixrts.com/resource/super-
| bowl-2024
|
| Even the best latency is dozens of seconds behind live
| action.
| squeaky-clean wrote:
| I don't bet so I have no clue, but why is that? Are
| people able to place bets in the middle of the match or
| something? I would have assumed bets get locked in when
| the fight starts
| umanwizard wrote:
| Idk about traditional sports books but on Polymarket you
| can certainly continue betting at any time until the
| market resolves.
| pests wrote:
| They end betting some minutes before the fight ends.
|
| I last saw Tyson at +500 while Jake was around -800 on
| DraftKings somewhere in the 6th round.
| JCharante wrote:
| a match has multiple rounds doesn't it? Seems logical to
| bet on individual rounds or events that can occur
| throughout the match.
| andreimackenzie wrote:
| I would be surprised if they don't already do this. The
| question is how big a buffer to trade off for delay...
| bushbaba wrote:
| Not sure I fully buy that. The "live" stream is rarely
| "live". It's often a highly cached buffer that's a few mins
| from latest. Those in isp caches can still help here.
| h4l wrote:
| That model still works for streaming. You have a central
| source stream only to the distributed edge locations, then
| have clients only stream from their local edge location. Even
| if one region is overwhelmed, the rest can still work. Load
| on the central source is bounded.
| _dark_matter_ wrote:
| I don't think that live doesn't work with caches. No one
| watching live would care about a O(s) delay, which is highly
| amenable to caching at ISPs and streaming changes from there
| to downstream clients. Offhand I'd say that would support
| O(ms) delay but no less.
| ethbr1 wrote:
| I'm curious if the root cause is more variable than usual
| latency.
|
| Sample size 1, but...
|
| I saw a ton of buffering and failure on an embedded Netflix
| app on a TV, including some infinite freezes.
|
| Moved over to laptop, zero buffering.
|
| I assume the web app runs with a lot bigger buffer than
| whatever is squeezed into the underpowered TV.
| YZF wrote:
| Likely these devices use different media formats and/or
| quality levels. And yes, it's possible one device buffers
| more than the other. Infinite freezes sounds like some
| routing issues or bugs.
| ethbr1 wrote:
| When I was watching the behavior on the tv, was wondering
| if buffering sends some separate, non-business-as-usual
| requests, and that part of Netflix's delivery
| architecture was being overloaded.
|
| E.g. "give me this previous chunk" vs "send me the
| current stream"
| YZF wrote:
| Buffering typically just consumes the same live stream
| until there's enough in the buffer. No difference other
| than the request rate being potentially higher. At least
| I can confidently say that for the standard players/video
| platforms. NetFlix could be doing something different.
| I'm not sure if they have their own protocols. But I'd be
| very surprised if the buffering used a completely
| different mechanism.
| jimbob45 wrote:
| Yeah, the funny part is that Hulu, Amazon Prime, and Peacock
| have all demonstrated the ability to handle an event of this
| caliber with no issue. Netflix now may never get another
| opportunity like this again.
| dyauspitr wrote:
| Sure they will. They'll just set up the next event and
| outside of some tech folks no one will remember this.
| CitrusFruits wrote:
| I mean, I guarantee you every boxing fan is never going to
| trust Netflix again for an event like this.
| dyauspitr wrote:
| What are they going to do? Just not watch the next fight?
| kortilla wrote:
| 0 people give a shit about boxing fans. It's not up to
| them.
| silisili wrote:
| AFAIK those three farm out to CDNs with tons of edges who
| know what they're doing.
|
| I have a feeling Netflix said 'how hard could this be?' and
| is finding out right now.
| anshumankmr wrote:
| Chances are Jake might fight Connor McGregor. Sure, Connor is
| not as famous as Tyson but that will also invite a large
| number of people to stream it.
| bsimpson wrote:
| I don't spend much time streaming, but I got a glimpse of the
| Amazon Prime catalog yesterday, and was surprised at how many
| titles on the front page were movies I'd actually watch.
| Reminded me of Netflix a dozen years ago.
| nicce wrote:
| Amazon Prime front page includes currently a lot of ads for
| movies that you can rent or "buy". Are you sure these movies
| weren't them?
| joshuakcockrell wrote:
| This is my biggest issue with Prime Video. You never know
| what's included and what costs extra.
| squeaky-clean wrote:
| There's also the "FreeVee" items, which have ads
| regardless of whether you're a prime subscriber or not.
| And it feels like a lot of their catalog has been
| transferred over to FreeVee.
| HDThoreaun wrote:
| You can toggle to only show free content. Still get ads
| but theyre obvious.
| LollipopYakuza wrote:
| Prime Video has to be the worst of all major streaming
| services. The video quality is horrible, its crippled with
| ads (3 not skippable ads for an episode of 45 minutes,
| lastly), and a lot of interesting titles are behind a
| "partner paywall".
| unsnap_biceps wrote:
| I was watching the rings of power and it started with a
| "Commercial free experience provided by so and so" with a
| long ad at the start of the episode, and then a third of
| the way into the episode, at a critical action part, it
| broke in the middle of the actor's sentence to a 6 minute
| ad block.
|
| I exited playback and haven't gone back to finish it. I'll
| wait for it eventually to make it to a Blu-ray release
| someday.
| m463 wrote:
| I have prime and my _shopping_ experience is crippled with
| ads too.
|
| I think it got worse for sellers recently too. If I search
| for something, like a specific item using its description,
| sometimes the only result for it shows "sponsored".
|
| It used to show up as sponsored and also unsponsored below.
|
| If this changed, I assume it is bad for the seller. Either
| they pay for all search results, or their metrics are
| skewed because all searches were helped by "sponsorship"
| (and there are no longer unsponsored hits)
| RaftPeople wrote:
| > _Prime Video has to be the worst of all major streaming
| services_
|
| I would put Prime Video at 2nd worst. Absolute worst IME is
| Paramount+.
|
| Edit: worst for streaming quality
| hughesjj wrote:
| It's also super annoying to try to watch on a computer
| compared to Netflix or YouTube
| stackghost wrote:
| > Reminded me of Netflix a dozen years ago.
|
| It's been pretty rough the last few years. So many great
| films and series, not to mention kids programming, removed to
| make way for mediocre NeTfLiX oRiGiNaLs and Bollywood trash.
| JCharante wrote:
| is this specifically in India? I never see bollywood stuff
| in the US but half the catalogue is dubbed/subbed korean
| dramas
| Spooky23 wrote:
| Netflix pivoted to be a platform to waste as much of your
| time as possible vs entertain.
|
| The Amazon originals are way better imo. They do the dark
| pattern crap with paid content, as one would expect from
| Amazon.
| djbusby wrote:
| Every Amazon show looks the same, yellow washed or
| something; and they should spend more money on costumes -
| they get beat by low budget cosplay.
| nicce wrote:
| Fallout was pretty good. Very loyal to the game.
| spike021 wrote:
| Amazon Prime isn't so great. Lots of for rent/purchase
| content or content with ads these days. And they end up
| repeating slots of content in all the rows in their UI, so I
| end up seeing the same suggestions everywhere rather than
| much that's new (other than first party productions).
|
| To me they're basically padding their front page.
|
| But honestly that's most of the major streaming platforms
| these days. I recently cancelled Disney Plus for similar
| reasons. The only reasons I don't cancel prime or Netflix are
| because I have family members I split the memberships with to
| share.
| chamomeal wrote:
| I recently found a lil dvd rental place in my city. It's a
| non-profit, they also do archivals and stuff.
|
| It's pretty much a two-story townhouse packed head to toe
| with DVDs (lots of blu rays!)
|
| You don't realize how limited the streaming collection is
| until you're back in a movie store, looking through
| thousands and thousands of movies you would never find
| otherwise.
|
| Since I found it, I've started doing movie night every week
| with my friends. It's such an absolute blast. We go to the
| store, each pick out a random movie that looks good (or
| bad, in a good way) or just different.
|
| All of a sudden, I love movies again!!
| spike021 wrote:
| That's an excellent option. I think it'd be remiss not to
| mention local libraries. Of course, your mileage may
| vary, but the ones I've gone to do seem to have adequate
| selections. I just don't often make time to go there and
| browse like I would have at traditional video rental
| places back in the day.
|
| Heck, mine even have some video games; though from when
| I've checked they're usually pretty back-reserved.
| bsimpson wrote:
| I was in high school in the early 00s, and going to the
| movies was such a big part of my life. Now, I never even
| know what's out.
|
| I suspect life stage is a factor, but it does feel like
| there are many classes of entertainment (cinema and
| standup come to mind) that don't resonate like they used
| to.
| Yeul wrote:
| Back in the day everyone was watching the same thing. The
| choices for entertainment were limited to whatever was
| showing in movie theatres, whatever was on TV and
| whatever record stores were selling.
| justinclift wrote:
| Heh Heh Heh. Maybe there will be a resurgence of
| Blockbuster style retail stores... ;)
| bsimpson wrote:
| I've given Netflix a lot more money than I've gotten value
| out of. I've had an account for ~15y and only really use it
| for airplanes unless there's a specific thing I'm excited
| to watch.
|
| I'm in the same boat where as soon as they make it too hard
| to share, I'll probably cancel it. I think the main reason
| their sharing crackdown hasn't been a problem so far is
| that I use it so seldomly, it thinks the "main" address is
| my parents, which makes it easy for me to pass the "are you
| traveling" 2FA on my own phone when I do want to watch
| something.
| 0x0000000 wrote:
| > And they end up repeating slots of content in all the
| rows in their UI, so I end up seeing the same suggestions
| everywhere rather than much that's new
|
| All of the streaming services do this and I _hate_ it.
| Netflix is the worst of the bunch, in my experience. I
| already scrolled past a movie, I don 't want to watch it,
| don't show it to me six more times.
|
| Imagine walking through a Blockbuster where every aisle was
| the same movies over and over again.
| ilaksh wrote:
| I think they have to refund the fees for a month to anyone who
| streamed this fight. That's the only thing that seems fair.
|
| It has been pretty useless. At the moment seems to be working
| only when running in non-live mode several minutes behind.
|
| So if there are 1 million trying to stream it, that means they
| would lose $15 million. So.. they might only give a partial
| refund.
|
| But people should push for an automatic refund instead of a
| class action.
| SoftTalker wrote:
| Is it really that big a deal if you are watching a few
| minutes behind?
|
| I've watched ball games on streaming networks where I can
| also hear a local radio broadcast, and the stream is always
| delayed compared to the radio, sometimes by quite a lot. But
| you'd never know it if you were just watching the stream.
| notatoad wrote:
| >Is it really that big a deal if you are watching a few
| minutes behind?
|
| i don't bet on sports. but from friends who do: yes, it's a
| really really big deal.
| defrost wrote:
| Potentially more so in this brave new world of increased
| sports betting.
|
| https://en.wikipedia.org/wiki/The_Sting
|
| Offshore combined streaming and betting houses will be
| cleaning up the rubes.
| SoftTalker wrote:
| Internet streams are not real-time even in the best case.
| There is always a few seconds of delay, often quite a bit
| more than that depending on number of hops and link
| speeds, congestion, etc.
| ilaksh wrote:
| The issue is that most people are trying to watch live
| which is what it's advertised as. And until they figure out
| that they need to watch X minutes behind, it is
| unwatchable. Many will not figure that out.
|
| So for the first hour it was just total frustration until I
| stopped trying to go back to live mode.
| JCharante wrote:
| well it's live sports, watching live is the big deal. Also
| people are gambling on the outcomes so watching a few
| minutes behind is a big deal
| jeromegv wrote:
| Not the same demographic but their last large attempt at live
| was through a Love is blind reunion. It was the same thing,
| millions of people logging in, epic failure, nothing worked.
|
| They never tried to do a live reunion again. I suppose they
| should have to get the experience. Because they are hitting the
| same problems with a much bigger stake event.
| barkingcat wrote:
| yup wanted to say that live stream stuttering has happened
| before on Netflix - I don't think the reputation is deserved.
|
| From a livestreaming standpoint, netflix is 0/x - for many
| large events such as love is blind, etc.
|
| From a livestreaming standpoint, look to broadcast news,
| sports / Olympics broadcasters, etc and you'll see
| technology, equipment, bandwidth, planning, and
| professionalism at 1000x of netflix.
|
| Heck, for publicly traded quarterly earnings livestream
| meetings, they book direct satellite time in addition to
| fiber to make sure they don't rely only on terrestrial
| networks which can fail. From a business standpoint, failure
| during a quarterly meeting stream can mean the destruction of
| a company (by making shareholders mad that they can't see and
| vote during the meeting making them push for internal change)
| - so the stakes are much higher than live entertainment
| streaming.
|
| Netflix is good at many things, livestreaming is not one of
| those things.
| glimshe wrote:
| Even some of the old guard can do this. The Olympics worked
| pretty well (despite the awkward UI), and that was
| Peacock/NBC.
|
| Perhaps Netflix still needs a dozen more microservices to
| get this right...
| emeril wrote:
| All valid points though each of those examples seemingly
| only has a fraction of the viewers of the netflix events,
| right?
| barkingcat wrote:
| for livestreams, individual events like the Olympics
| probably has a surge audience of 10x of netflix events.
|
| Netflix events is small potatoes compared to other
| livestream stalwarts.
|
| Imagine having to stream a cricket match internationally
| to UK / India / Australia with combined audience that
| crushes the Superbowl or a football match to all of
| Europe, or even something like livestreaming F1 racing
| that has multiple magnitudes of audience than a boxing
| match and also has 10x the number of cameras (at 8K+
| resolution) across a large physical staging arena (the
| size of the track/course) in realtime, in addition to
| streaming directly from the cockpit of cars that are
| racing 200mph++.
|
| Livestream focused outfits do this all day, everyday.
|
| Netflix doesn't even come close to scratching the
| "beginner" level of these kinds of live events.
|
| It's a matter of competencies. We wouldn't expect Netflix
| to be able to serve burgers like McDonald's does -
| Livestreaming is a completely different discipline and
| it's hubris on Netflix's part to assume just because
| they're good at sending video across the internet they
| can competently do livestreaming.
| KennyBlanken wrote:
| Is anyone surprised? I don't see how their infrastructure can
| handle this when it was designed for non-realtime precaching
| of prerecorded content.
| tylerchilds wrote:
| this is false, the tom brady roast was live streamed
|
| yes, love is blind failed, but was definitely not the most
| recent attempt. they did some other golf thing too, iirc
| anshumankmr wrote:
| tom brady is largely a guy popular in the USA whereas Mike
| Tyson is globally famous. It follows that this fight would
| attract a larger audience.
| tylerchilds wrote:
| the point i'm making is that the netflix live streaming
| timeline didn't go
|
| chris rock -> love is blind -> mike tyson
|
| they have had other, successful executions in between.
| the comment i was replying to had cherry picked failures
| and i'm trying to git rebase them onto main.
| Spooky23 wrote:
| It may vary by ISP. It's been fine for me.
| graton wrote:
| I think you are correct. Ziply Fiber said they were seeing
| 2.1 times their normal peak [1].
|
| But also people were saying they weren't having any issues
| streaming on Ziply.
|
| [1] https://www.reddit.com/r/ZiplyFiber/comments/1gsenik/netf
| lix...
| rvz wrote:
| I don't think Netflix is even designed to handle very extreme
| multi-region live-streaming at scale as evidenced in this event
| with hundreds of millions simultaneously watching.
|
| YouTube, Twitch, Amazon Prime, Hulu, etc have all demonstrated
| to stream simultaneously to hundreds of millions live without
| any issues. This was Netflix's chance to do this and they have
| largely failed at this.
|
| There are no excuses or juniors to blame this time. Quite the
| inexperience from the 'senior' engineers at Netflix not being
| able to handle the scale of live-streaming which they may lose
| contracts for this given the downtime across the world over
| this high impact event.
|
| Very embarrassing for a multi-billion dollar publicly traded
| company.
| gonzo41 wrote:
| They must have try to do this on the cheap, thinking they
| could dynamically scale on the fly for this. Big mistake.
| anakaine wrote:
| This is a total supposition without any proof.
| colesantiago wrote:
| What more proof do you need other than the fact that
| streams went down worldwide on a highly anticipated event
| from a public company?
|
| I wouldn't be surprised if lots of engineers at Netflix
| are currently now writing up a length post mortem of
| this.
|
| And this is from the company that created the discipline
| of chaos engineering for resilience.
|
| It is clear they under invested and took the eye of the
| ball with this.
|
| This is bad, like very very bad.
| jhugo wrote:
| The assumption that it was related to insufficient
| investment isn't supported by any evidence. Flawed
| technical decisions can be made by the most expensive
| engineers too.
| colesantiago wrote:
| The evidence is that the stream went down.
|
| We will see why it went down and to what extent they
| underinvested in their post mortem.
| fragmede wrote:
| Their CDN is colo and doesn't run on AWS.
| 2OEH8eoCRo0 wrote:
| Is cable and broadcast better for live TV? No scaling issues.
| Doesn't matter how many people tune in.
| Valord wrote:
| Seems like it
| betaby wrote:
| Interestingly that TV nowadays is delivered through IP.
| vitus wrote:
| It's fundamentally different, for sure.
|
| Most third-party internet-based streaming solutions are
| overlaid on top of a point-to-point network, while broadcast
| is one-to-many, and even cable tends to use multicast within
| the cable provider's network.
|
| You have potentially different problems, e.g. limited
| bandwidth / spectrum. If, say, there are multiple games going
| on at the same time, you can only watch whichever feed the
| broadcaster decides to air. And, of course, regardless of the
| technology in use, there are matters of acquiring rights for
| various events. One benefit of internet-based streaming is
| that one service can acquire the rights and be able to reach
| everyone, whereas an individual cable provider might only
| reach its direct subscribers.
| Ekaros wrote:
| On cable(terrestrial is entirely different) even the
| bandwidth or spectrum is less limiting for broadcasting
| multiple games. Hard thing is the other parts of
| production, like cameras, live-directing and live
| commentary. Adding new channels is less challenging than
| actual producing content at expected level there.
| oaththrowaway wrote:
| I used to work for a live streaming platform once. We always
| joked that VOD (Netflix) was "easy" compared to live.
| taeric wrote:
| Not really a joke, though? VOD has obvious methods to cheat a
| bit. Redundancy abounds and you can even network shape for
| costs. Could probably get even better compression for clear
| reasons.
|
| Live, not so much. One source that you have to fanout from
| and absolutely no way to get cheap redundancy. Right?
| oaththrowaway wrote:
| Yes, of course but it was still a cope because we never saw
| more than maybe 3M concurrent viewers at a time
| noncoml wrote:
| > their streaming service normally seems very solid
|
| Not trying to downplay their complexity, but last I heard
| Netflix is splitting the shows in small data chunks and just
| serves them as static files.
|
| Live streaming is a different beast
| treflop wrote:
| Static files have been pretty much the standard streaming
| protocols for both VOD and live for the last 15 years ago.
| Before, it was Adobe Flash (RTMP).
|
| With the way that they are designed, you can even use a
| regular CDN.
| adrr wrote:
| You can push these files to all the edges before you
| release the content which will protect your origin.
| Livestream all your edge servers are grabbing content from
| the origin unless you have another tier of regional servers
| to alleviate load.
| treflop wrote:
| Sure but that's why your edge servers do request
| collapsing. And there are full blown CDN companies that
| will write an enterprise contract with you that can do
| this stuff with ease. Akamai is like 25 years old now.
|
| Scale has increased but the techniques were figured out
| 20 years ago. There is not much left to invent in this
| space at the current moment so screwing up more than once
| is a bit unacceptable.
| astrange wrote:
| That's how streaming (usually) works. The main URL is a
| playlist of transport stream files and it just downloads them
| in the background as you go.
| a_random_name wrote:
| If Netflix still interviews on hacker rank puzzles I think this
| should be a wake up call. Interviewing on irrelevant logic
| puzzles is no match for systems engineering.
| rvz wrote:
| For systems design and engineering, absolutely this. I
| expected the very highest standards and upmost uptime from
| Netflix, similar to Google and Amazon.
|
| Tells you the uselessness of their engineering blogs.
| VirusNewbie wrote:
| I did a round of netflix interviews, didn't get an offer (but
| passed the technical coding rounds) they absolutely had the
| best interview process of any company I've interviewed at my
| entire career.
|
| They do make you code but the questions were 1. Not on hacker
| rank or leetcode 2. Pratical coding questions that didn't
| require anything more than basic
| hashmaps/lists/loops/recursion if you want. Some string
| parsing, etc.
|
| They were still hard, you had to code a fast, but no tricky
| algorithms required. It also felt very collaborative, it felt
| like you were driving pair programming. Highly recommended
| even though didn't get an offer!
| that_guy_iain wrote:
| I don't think it'll be long-term. Most people will forget about
| this really quickly. It's not like there will be many people
| saying "Oh, you don't want to sign up for Netflix, the Tyson
| fight wasn't well streamed" in even 6 months nevermind 10
| years.
| lawgimenez wrote:
| There's an upcoming NFL game on Netflix next month. They need
| to get their shit together.
| caseyohara wrote:
| Two games actually, both on Christmas Day. A day when most
| people are at home or the home of family or friends, and they
| are both pretty good late-season matchups (Chiefs-Steelers
| and Ravens-Texans) so I imagine viewership will be high.
|
| If they botch the NFL games, it will surely hurt their
| reputation.
| jpalawaga wrote:
| From what I've heard, Netflix has really diluted the culture
| that people know of from the Patty McCord days.
|
| In particular, they have been revising their compensation
| structure to issue RSUs, add in a bunch of annoying review
| process, add in a bunch of leveling and titles, begin hiring
| down market (e.g. non-sr employees), etc.
|
| In addition to doing this, shuffling headcount, budgets, and
| title quotas around has in general made the company a lot more
| bureaucratic.
|
| I think, as streaming matured as a solution space, this (what
| is equivalent to cost-cutting) was inevitable.
|
| If Netflix was running the same team/culture as it was 10 years
| ago, I'd like to say that they would have been able to pull of
| streaming.
| neilv wrote:
| Were they not able to hire enough top-skilled people? If not,
| why not?
|
| Or did they have a lot of needs that they decided didn't
| require top-skilled people?
|
| Or was this a beancounter thing, of someone deciding that the
| company was paying more money on staffing than they needed
| to, without understanding it?
| jpalawaga wrote:
| Combination of 2 and 3. The business changed. Streaming was
| more or less a solved problem for Netflix. They needed
| money for content, not expensive engineers. Ted is co-
| ceo... you can see where the priority is.
| AtlasBarfed wrote:
| My observation is that Netflix is one of those places that
| brags about how they do so much with so little employees.
| dboreham wrote:
| Few. Little employees would be...small.
| squeaky-clean wrote:
| Has Netflix ever live streamed something before? People on
| reddit are reporting that if you back up the play marker by
| about 3 minutes the lag goes away. They've got a handle on
| streaming things when they have a day in advance to encode it
| into different formats and push it to regional CDNs. But I
| can't recall them ever live streaming something. Definitely
| nothing this hyped.
| ipaddr wrote:
| Love is blind reunion which had major problems. Chris Rock
| comedy which only had a few issues. The Netflix Cup which had
| issues.
| StevePerkins wrote:
| Chris Rock comedy special, and the Tom Brady roast. Nothing
| on this scale, though.
| Fire-Dragon-DoL wrote:
| For me, netflix constantly forget the last episode/spot I was
| in a TV show. Beyond frustrating
| geodel wrote:
| I think Netflix will have even more sw engineers looking to
| work there once they notice even for average quality of work
| they can get paid 3 times more than their current pay.
| test6554 wrote:
| Yea, it's a bad look. But I switched to watching some other
| Netflix video and it seemed fine. Just this event had some
| early issues. Looks fine now though.
| rapind wrote:
| > ut my impression still has always been that Netflix is a step
| above the rest technically.
|
| I always assumed youtube was top dog for performance and
| stability. I can't remember the last time I had issues with
| them and don't they handle basically more traffic than any
| other video service?
| normie3000 wrote:
| Maybe a client issue, but i've got a low-end smart tv which
| handles netflix fine, but youtube is unwatchable due to
| buffering and failed cuts to adverts
| rapind wrote:
| Maybe that's it. I pay from premium though so don't have
| the advert issue (apples to apples).
| kenjackson wrote:
| Streamed glitch free for me both on my phone and Xbox. The
| fight wasn't so great though, but still a fun event. Jake Paul
| is a money machine right now.
| ikt wrote:
| Based on this I'm wondering whether it was straight up they did
| not expect it to be this popular?
|
| > Some Cricket graphs of our #Netflix cache for the
| #PaulVsTyson fight. It has a 40 Gbps connection and it held
| steady almost 100% saturated the entire time.
|
| https://fosstodon.org/@atoponce/113491103342509883
| dh2022 wrote:
| I think why I will remember about this fight is not the (small)
| streaming issue I encountered as much as the poor quality of
| the fight itself. For me that was the reputational loss.
| Netflix was touting "NFL is coming to Netflix". This fight did
| not really make me want to watch that.
| BLKNSLVR wrote:
| I don't care about boxing or UFC or the grade-A douchebags
| that are the Paul brothers, but I tuned in just because I had
| the time and a Netflix subscription.
|
| It was actually great that the fight itself was so boring
| because it justifies never having to spend time / money on
| that kind of bullshit. It was a farce. A very bright, loud,
| sparkly, and expensive (for some people) farce.
|
| The value I got from it was the knowledge that missing out on
| that kind of thing isn't really missing out on anything at
| all.
| tass wrote:
| Was live streaming much of a use case for them before this?
|
| They stream plenty of pre recorded video, often collocated.
| Live streaming seems like something they aren't yet good at.
| someothherguyy wrote:
| https://www.cnn.com/2023/03/04/entertainment/chris-rock-
| netf... (2023-03-06)
| fred_is_fred wrote:
| If places like Paramount+ can figure it out, Netflix, given
| their 10+ year head start on streaming and strong engineering
| culture, should also have been able to. And if you don't like
| my example, literally every other streaming service has
| streamed live sports without issue. YT TV, Hulu, Paramount+,
| Amazon Prime, Peacock, even Apple TV streams live sports.
|
| It may be "new" to them, but they should have been ready.
| tass wrote:
| I won't argue that they shouldn't have done better, I'm
| only pointing out that this is fairly different from their
| usual product. Amazon, YouTube, and Hulu all have a ton of
| experience with live streaming by now. Apple has live
| streamed wwdc for several years.
|
| I did expect that Netflix would have appropriately
| accounted for demand and scale, though, especially given
| the hype for this particular event.
| apexalpha wrote:
| >but my impression still has always been that Netflix is a step
| above the rest technically.
|
| Maybe if we're not counting Youtube as 'streaming', but in my
| mind no one holds a candle to YT quality in (live)streaming.
| Yeul wrote:
| There's a difference between live broadcasts and serving up
| content that's sitting on a server I guess?
|
| In my country every time there's a big football match the
| people who try to watch it on the internet face issues.
| MyFedora wrote:
| Netflix won't take a hit here.
|
| Most people pay Netflix to watch movies and tv shows, not
| sports. If I hadn't checked Hacker News today, I wouldn't even
| know they streamed sports, let alone that they had issues with
| it. Even now that I do, it doesn't affect how I see their core
| offering, which is their library of on-demand content.
|
| Netflix's infrastructure is clearly built for static content,
| not live events, so it's no shock they aren't as polished in
| this area. Streaming anything live over the internet is a tough
| technical challenge compared to traditional cable.
| oplav wrote:
| Netflix is trying to expand into live sports. This event
| wasn't a one off thing. There is an NFL game they are
| streaming at the end of the year.
| Willingham wrote:
| I can feel the pressure on the network engineers from here XD
| nightowl_games wrote:
| From my experience, it works if your not watching it 'live'. But
| the moment I put my devices to 'live' it perma-breaks. 504
| gateway timed out in web developer tools hitting my local CDN.
| probably works on some CDNs, doesnt on others. Probably works if
| your not 'live'
|
| edit: literally a nginx gateway timed out screen if you view the
| response from the cdn... wow
| jeffbee wrote:
| Why do they want to get into the live business? It doesn't seem
| to synergize with their infrastructure. Sending the same stream
| in real time to numerous people just isn't the same task as
| letting people stream optimized artifacts that are prepositioned
| at the edge of the network.
| dyauspitr wrote:
| They want to break into sports because it's such a big business
| and if you do sports you need to be able to stream live.
| smegger001 wrote:
| Because its the one area traditional networks have the
| advantage in
| jeffbee wrote:
| The marginal cost to add a viewer to broadcast sports is
| zero! That's what I am getting at. I know why someone would
| want this business, I just don't see what aspect of
| Netflix's existing business made them think they'd be good
| at it.
| VirusNewbie wrote:
| Most PPV is what, $50-$70? So subscribing to Netflix for $20 or
| whatever per month sounds like a bargain for anyone who is
| interested and not already a customer. Then assume some large
| percentage doesn't cancel either because they forgot, or
| because they started watching a show and then decided to keep
| paying.
| dilyevsky wrote:
| Not sure why this is being downvoted. I can see your point -
| it's much harder to this live but a lot of their cdn infra can
| be reused.
| SkyPuncher wrote:
| Live is the only thing that won't be commodified entirely.
| "Anyone" can pump out stream-when-you-want TV shows. Live
| events are generally exclusive, unpredictable, and cultural
| moments .
| jairuhme wrote:
| Every time it buffers for me, Netflix does an internet test only
| for it to come back and say its fast...
| cranberryturkey wrote:
| yeah i'm using iptv which is just a rip of NF and its stuck
| buffering.
| rvz wrote:
| All these engineering blog posts, distributed systems and these
| complex micro-services clearly didn't help with this issue.
|
| Netflix is clearly not designed nor prepared for scalable multi-
| region live-streaming, no matter the amount of 'senior' engineers
| they throw at the problem.
| ilrwbwrkhv wrote:
| This. Overly complex nonsense that they have built up is crazy
| and the fact that we on a tech forum agree to all of this is
| crazier.
|
| It's almost like this platform has been taken over by
| JavaScripters completely.
| sgarland wrote:
| > taken over by JavaScripters
|
| That is an incredible way to phrase the sentiment, thank you.
| robertlagrant wrote:
| > Netflix is clearly not designed nor prepared for scalable
| multi-region live-streaming
|
| Well, yes. Who would think Netflix was designed for that? They
| do VOD. They're only trying to move into this now.
| yoshamano wrote:
| I did some VPN hopping and connecting to an endpoint in Dallas
| has allowed me to start watching again. Not live though, that
| throws me back into buffering hell.
| runjake wrote:
| It's like watching a Minecraft cosplay of the event.
| yapyap wrote:
| Hell, I'd complaing about Jake Paul vs Mike Tyson as well if I
| was a boxing fan. Even without buffering issues
| koolba wrote:
| They better get have some better judges and refs too. The co-
| headline title fight was a joke.
| smolder wrote:
| Yes, it was utterly boring, but they made their money. I don't
| like either Paul brother, so I only watched in hopes shorter,
| much-older Tyson would make Jake look as foolish as he is.
| ackbar03 wrote:
| This reminds me of that scene in Silicon Valley
| neilv wrote:
| https://www.youtube.com/watch?v=9IGvzb-KCpY&t=50s
| abixb wrote:
| I clicked on this thread to type that exact thing, holy smokes.
|
| You're referring to Hooli's streaming of UFC fight that goes
| awry and Gavin Belson totally loses it, lol. Great scene and
| totally relevant to what's happening with Netflix rn.
| a_random_name wrote:
| Ota broadcasts are clearer
| causality0 wrote:
| Weird that an organization like Netflix is having problems with
| this considering their depth of both experience and pockets. I
| wonder if they didn't expect the number of people who were
| interested in finding out what the pay-per-view experience is
| like without spending any extra money. Still, I suppose we can
| all be thankful Netflix is getting to cut their live event teeth
| on "alleged rapist vs convicted rapist" instead of something more
| important.
| marxisttemp wrote:
| > alleged rapist vs convicted rapist
|
| And you'll never guess which Presidential candidate they both
| support!
| benreesman wrote:
| This Serrano fight is just an insane display of excellence.
|
| If anyone was waiting for the main card to tune in, I recommend
| tuning in now.
| Waterluvian wrote:
| I know nothing about boxing and this fight was just
| ridiculously impressive. I kept tuning out of the earlier
| fights. They felt like some sort of filler. I didn't get the
| allure. But Taylor v Serrano was just obvious talent that even
| I could appreciate it.
| vFunct wrote:
| Serrano was robbed!
| slicktux wrote:
| That was a savage fight!
| anakaine wrote:
| Absolutely excellent fight. 10 full rounds with full effort
| until the end. Fantastic.
|
| Also, no buffering issues on my end. Have to wonder if it's a
| regional issue.
| 29athrowaway wrote:
| Serrano should have won.
| agrippanux wrote:
| What was an amazing fight - that Serrano won. I have no idea
| how Taylor was scored the winner.
| cco wrote:
| naw, taylor head butting the whole fight was dirty and really
| took the wind out of it
| sgarland wrote:
| A friend and I, in separate states, found that it wouldn't stream
| from TVs, Roku, etc. but would stream from mobile. And for me,
| using a mobile hotspot to a laptop; though that implies checking
| IP address range instead of just user-agent, so that seems
| unlikely.
|
| Anyway, I wouldn't be surprised if they were prioritizing mobile
| traffic because it's more forgiving of shitty bitrate.
| arcbyte wrote:
| I just left a bar streaming it on a smart TV and back in my
| home it's streaming on the Roku just fine.
| sgarland wrote:
| Guess I was looking for explanations too hard.
| anakaine wrote:
| I wonder if this points to network peering and edge nodes.
| Mobile network vs cabled network likely being routed to
| different places.
| wonderwonder wrote:
| Is this potentially an aws issue?
| spencerchubb wrote:
| I would assume not because twitch runs on aws. I think netflix
| engineers haven't optimized as much for livestreaming like
| twitch has
| anshumankmr wrote:
| I do not think twitch has the amount of concurrent users
| netflix might have had today morning for the fight.
| chevman wrote:
| Guess they should have livestreamed it on X to be safe!
| kevinventullo wrote:
| Or Facebook. Or YouTube. Or Vimeo. Or LiveLeak.
| spyda56 wrote:
| Working okay for me
| voyagerfoil wrote:
| Over promised and under delivered. That's a bad look
| boppo1 wrote:
| I'm watching on a 'pirate' stream because my netflix stream is
| absolutely frozen.
| walrushunter wrote:
| I'm an engineering manager at a Fortune 500 company. The dumbest
| engineer on our team left for Netflix. He got a pay raise too.
|
| Our engineers are fucking morons. And this guy was the dumbest of
| the bunch. If you think Netflix hires top tier talent, you don't
| know Netflix.
| moralestapia wrote:
| I can +1 with a similar anecdote.
|
| They obviously have some really good engineers, but many low-
| tier ones as well. No idea how long they stay there, though.
|
| I'm watching the fight now and have experienced the buffering
| issues. Bit embarrassing for a company that fundamentally only
| does a single thing, which is this. Also, yeah, 900k TC and
| whatnot but irl you get this. Mediocre.
| spencerchubb wrote:
| livestream is quite different from streaming pre-processed
| video, so I'm not surprised by the scaling issues
| moralestapia wrote:
| I know.
|
| But given how much they spend on engineering, how much time
| they had and how important this event is ... mediocre
| performance.
| jhugo wrote:
| All true, but this part of your GP comment:
|
| > a company that fundamentally only does a single thing,
| which is this
|
| ... isn't true. From the couch, watching Suits and
| watching a live sports match may seem similar but they're
| very different technical problems to solve.
| Moru wrote:
| Or in other words: In one case the "stream" is stored on
| a harddrive not far away from you, only competing for
| bandwidth in the last section to you. In the other case
| the "stream" is comming over the Internet to you and
| everyone else at the same time.
| katbyte wrote:
| i was mildly interested and managed to find a pirate
| livestream, it didn't have buffering issues lol
| dh2022 wrote:
| Well the pirate site was not live-streaming to 100
| million users...
| anakaine wrote:
| I have to wonder if its a regional thing. I'm watching from
| the southern pacific in HD, and its been excellent.
| diab0lic wrote:
| I'd imagine it is fairly dependent on which cache you're
| connected to.
| reagan83 wrote:
| Curb your enthusiasm had a good segment of episodes that were a
| parody on Netflix and how they shifted hiring from merit to
| other criteria.
| indigodaddy wrote:
| Curb did? What season/episodes?
| ldjkfkdsjnv wrote:
| Its all ego when these companies think they hire the best.
| 0xpgm wrote:
| An engineering manager who thinks his engineers are morons and
| dumb?
|
| I have questions..
| wordofx wrote:
| None of the Fortune 500 companies hire top talent. They have a
| few good people but 98% of the engineers are average at best.
| Over paid.
| qingcharles wrote:
| This is every dev house I've worked at. For most people
| (mostly not the ones on HN), coding is a 9-5 job. No
| ambition. Just lines of code. Go home. I don't know there is
| anything particularly wrong with that.
|
| You just have to accept most staff at any corporation are
| simply just average. There has to be an average for there to
| be better and worse.
| AlotOfReading wrote:
| I've never seen a team that has somehow managed to hire
| exclusively morons. Even the shittiest of call center jobs and
| construction crews have a few people with enough brain cells to
| tie their shoelaces.
|
| Have you considered that maybe you're being overly harsh about
| your co-workers? Maybe take the fact that one of them was hired
| by a top paying employer as a sign that you should improve your
| own ability to judge skill?
| theendisney wrote:
| I've seen tons of them! The formula is to create conditions
| that will make even slightly competent people leave. They
| hire their morron nephew, he is always 30 min late then they
| moan when you are 5 min late because the parking lot was
| blocked by their car. He always leaves 2 hours early while
| you do overtime that they regularly forget to pay for. Your
| day is filled with your own work PLUS that of your retarded
| coworkers who only drink coffee while joking about you doing
| their work. You are not as fast as the last guy! haha! If
| something goes wrong the morrons collectively blame you, just
| like the last time. You get a formal warning. etc etc The
| other normal person they hire is let go after 2 days because
| they complaint which means they didnt fit the team.
|
| And so on
|
| If he still works there the morron who left was less of a.
| TylerE wrote:
| At least half of that is on you. NEVER work unpaid/unlogged
| OT.
| scott_w wrote:
| Can't speak for every place but that's not always an
| option. As a teenager, I worked at Sports Direct where
| the management would regularly work us after our allotted
| hours and bar us putting the extra time onto our
| timesheet. If I recall correctly, the company eventually
| got pulled for it but the money they'd have saved over
| years would have outweighed the fine.
|
| The timesheets were on paper so good luck putting your
| real hours on without your manager, who files it, finding
| out.
|
| I'd be amazed if they ever cleaned up their act.
| TylerE wrote:
| Report that shit to your local Department of Labor
| equivalent. They would have gotten you, and everyone else
| in that store, their owed money.
| scott_w wrote:
| You're asking children to have full understanding of
| their rights and how to enforce them. Also,
| investigations into this started in 2020: over a decade
| after I left. Do you think nobody had reported this in
| all that time? Looks like the system wasn't working as
| well as you think it does.
| specialist wrote:
| Yes and: IIRC, the USA has at least $8b of wage theft per
| year.
| djbusby wrote:
| Why is your team morons? Kind of disparaging maybe? Fish rots
| from the head situation?
| that_guy_iain wrote:
| Having worked with a bunch of guys who have gone on to "top
| teams", I no longer believe they have top teams. My fav was the
| guy who said the system could scale indefinitely after it
| literally fell on its ass from too much traffic. He couldn't
| understand that just because Lambdas my themselves can scale,
| they are limited by the resources they use, so just ignored it
| and insisted that it could. The same guy also kept on saying we
| should change the TPEG standard because he didn't like how it
| worked. And these companies are seriously pretending they've
| got the best and brightest. If that's really true, I really
| need to find another profession.
| YZF wrote:
| I've worked for many companies that said they hired the best.
| And to be honest when I hire I also try to hire good people.
| I think I could hire better if a) I had an open cheque, b) I
| was running coolest project in the universe. I did hire for
| some interesting projects but nothing close to an open
| cheque. Even under these conditions it's tough to find great
| people. You can go after people with a proven track record
| but even that doesn't always guarantee their next project
| will be as successful.
|
| The reality though is that large companies with thousands of
| people generally end up having _average_ people. Some company
| may hire more PhD 's. But on average those aren't better
| software engineers than non-PhD's. Some might hire people who
| are strong competitive coders, but that also on average isn't
| really that strong of a signal for strong engineers.
|
| Once you have a mix of average people, on a curve, which is
| the norm, the question becomes do you have an environment
| where the better people can be successful. In many corporate
| environments this doesn't happen. Better engineers may have
| obstacles put in front of them or they can forced out of the
| organization. This is natural because for most organizations
| can be more of a political question than a technical
| question.
|
| Smaller organizations, that are very successful (so can meet
| my two criterias) and can be highly selective or are highly
| desirable, can have better teams. By their nature as smaller
| organizations those teams can also be effective. As
| organizations grow the talent will spread out towards average
| and the politics/processes/debt/legacy will make those teams
| less effective.
| that_guy_iain wrote:
| To be fair, when you need to hire hundreds or thousands of
| people, you gotta hire average people. The best is a finite
| resource and not all of the best want to work for FAANG or
| any megacorp.
|
| I used to want to work at a FAANG-like company when I was
| just starting out thinking they were going to be full of
| amazing devs. But over the years, I've seen some of the
| worst devs go to these companies so that just destroyed
| that illusion. And the more you hear about the sort of work
| they do, it just sounds boring compared to startups.
| kkielhofner wrote:
| > I'm an engineering manager
|
| How are you involved in the hiring process?
|
| > Our engineers are fucking morons. And this guy was the
| dumbest of the bunch.
|
| Very indicative of a toxic culture you seem to have been pulled
| in to and likely have contributed to by this point given your
| language and broad generalizations.
|
| Describing a wide group of people you're also responsible for
| as "fucking morons" says more about you than them.
| pfannkuchen wrote:
| Reputation usually lags reality by 5+ years. See: Google.
| renewiltord wrote:
| Absolutely right. Netflix was once all about the sports team
| mentality. Now they're Man Utd.
| scott_w wrote:
| Given Man Utd under Ferguson used to be THE football team,
| you could say they were always Man Utd and still are ;-)
| renewiltord wrote:
| Haha indeed, but I'm a gooner. You'd never see me admit
| that.
| Lammy wrote:
| I hope to never have a manager who is mentally stack ranking me
| and my coworkers in terms of perceived dumbness instead of in
| terms of any positive trait.
| geodel wrote:
| Btw how do you know your current manager is not doing that.
| Lammy wrote:
| I don't. That's why I said hope :)
| ohyes wrote:
| You'll know when he ends every meeting with "dummies, get
| back to work"
| bryanrasmussen wrote:
| dumbness is ranking intelligence, which is a positive trait,
| dumbness is just a metric for how often intelligence fails.
|
| Example - the manager who started this sub-thread may be a
| pretty smart guy and able to accurately rate the intelligence
| of the engineers at his organization - but he had a minor
| momentary failing of intelligence to post on HN calling those
| engineers fucking morons.
|
| You've got to rank how often the intelligence fails in
| someone to be able to figure out how reliable their
| intelligence is.
| JamesBarney wrote:
| Almost everyone I know manager or not is usually ranking
| everyone they work with on various attributes.
|
| In fact it would be incredibly weird to ask a close friend
| who at their work kicks ass and who sucks and have them
| respond back, "I've never really thought about how good any
| of my coworkers were at their jobs"
| strken wrote:
| I'm not a manager and I don't stack rank people, but I am
| 100% capable of knowing when one of my co-workers or
| predecessors is a fucking moron.
|
| The trick is to use my massive brain to root cause several
| significant outages, discover that most of them originate in
| code written by the same employee, and notice that said
| employee liked to write things like // @ts-
| nocheck // obligatory disabling of typescript: static
| typing is hard, so why bother with it? async function
| upsertWidget() { try { // await
| api.doSomeOtherThings({ ... }) // 20 line block
| of commented-out useless code // pretend this
| went on much longer let result = await
| api.createWidget({ a, b, c }) if (!result.ok) {
| result = await api.createWidget({ a, b }) // retries for
| days! except with different args, how fun if
| (!result.ok) { result = await
| api.updateWidget({ a, b, c }) // oh wait, this time we're
| updating } } // notice
| that api.updateWidget() can fail silently //
| also, the three function calls can each return different
| data, I sure am glad we disabled typescript
| return result } catch (error) { return
| error // I sure do love this pattern of returning errors and
| then not checking whether the result was an error or the
| intended object } } function
| doSomething() { const widget = await upsertWidget()
| }
|
| ...except even worse, because instead of createWidget the
| name was something far less descriptive, the nesting was
| deeper and involved loops, there were random assignments that
| made no goddamn sense, and the API calls just went to an
| unnecessary microservice that was _only_ called from here and
| which literally just passed the data through to a third party
| with minor changes. Those minor changes resulted in an
| internal API that was actually worse than the unmodified
| third party API.
|
| I am so tired of these people. I am not a 10x rockstar
| engineer and not without flaws, but they are just so awful
| and draining, and they never seem to get caught in time to
| stop them ruining perfectly fine companies. Every
| try>catch>return is like an icy cat hand from the grave
| reaching up to knock my coffee off my desk.
| spike021 wrote:
| Isn't that a problem with your code review process? Why is
| that person's code making it to production?
|
| So again, maybe they're a bad employee but it seems like
| nothing's done to even try and minimize the risks they
| present.
| krysp wrote:
| There's a disincentive to actively block PRs if you don't
| want your coworkers to think you are a bad colleague /
| not on their side. So you often see suboptimal code
| making its way to production. This has a worse effect the
| more terrible engineers there are.
| spike021 wrote:
| Except in this case it's clearly affecting at minimum the
| rest of OP's team.
|
| At that point it's not one person being obnoxious and
| never approving their team members diffs and more of a
| team effort to do so.
|
| But at minimum if you have a culture of trying to improve
| your codebase you'll inevitably set up tests, ci/cd with
| checks, etc. before any code can be deployed. Which
| should really take any weight of responsibility out of
| any one member of the team. Whether trying to put out
| code or reject said code.
| mathgeek wrote:
| Turning this into an incentive that everyone values is a
| signal that a team has a great culture.
| strken wrote:
| In this specific case, the fucking moron in question was
| the one who designed the code review process and hired
| the other engineers, and it took place a significant
| length of time before my involvement.
|
| Which, yes, does raise interesting questions about how
| someone who can't be trusted to handle errors in an API
| call ended up in a senior enough position to do that.
| fragmede wrote:
| I dunno, I've gone and done a "git blame" to find out who
| the fucking moron that wrote the code was, only to find out
| it was me three years ago.
|
| Sure, there's such a thing as stupid code, but without
| knowing the whole context under which a bit of code was
| produced, unless it's utterly moronic, (which is entirely
| possible, dailywtf has some shiners), it's hard to really
| judge it. Hindsight, as applied to code, is 2020.
| strken wrote:
| I agree with the general sentiment ("one instance of bad
| code might have been me") but not the specific sentiment
| ("I could easily decide to catch and ignore errors
| through every bit of code I worked on without knowing why
| that was bad, and commit other, similar crimes against
| good taste in the same way").
|
| The difference for me is that this is _pervasive_. Yes,
| sometimes I might write code with a bug in error handling
| at 3am when I 'm trying to bring a service back up, but I
| don't do it consistently across all the code that I
| touch.
|
| I accept that the scope is hard to understand without
| having worked on a codebase which a genuine fucking moron
| has also touched. "Oh strken," you might say, "surely it
| can't be that bad." Yes, it can. I have never seen
| anything like this before. It's like the difference
| between a house that hasn't been cleaned in a week and a
| hoarder's house. If I tried to explain what hoarding is,
| well, maybe you'd reply that sometimes you don't do the
| dishes every day or mop the floor every week, and then
| I'd have to explain that the kitchen I'm talking about is
| filled from floor to roof with dirty dishes and discarded
| wrappers, including meat trays, and smells like a dead
| possum.
| fragmede wrote:
| Hey, that possum's name was Roger and I'm really sad that
| it died. I've been feeding it for weeks! There are
| definitely bad programmers out there who's code is only
| suitable for public shaming via the daily wtf.
| spike021 wrote:
| Hmm. Engineering managers should be setting the team culture
| and determining best criteria for extending an offer to a
| candidate. If theres a problem with the hiring process I'd look
| for the closest source that could or should be fixing it.
|
| I don't think I'd want to work for you.
| geodel wrote:
| It mostly makes sense to me. From their bombastic blogs to
| github projects full of overwrought Enterprise java design
| patterns. The only thing great about Netflix is it pays a lot
| more.
| globalnode wrote:
| this is why managers get a bad rap. what proportion think like
| this? hopefully not a large one but i do worry. ultimately if
| the team sucks its because of the management. theyre the ones
| with the greatest power to change things for the better.
| AdieuToLogic wrote:
| > I'm an engineering manager at a Fortune 500 company. The
| dumbest engineer on our team left for Netflix. He got a pay
| raise too.
|
| Apparently he was smart enough to get away from the Fortune 500
| company he worked at, reporting to yourself, and "got a pay
| raise too."
|
| > Our engineers are fucking morons. And this guy was the
| dumbest of the bunch.
|
| See above.
|
| > If you think Netflix hires top tier talent, you don't know
| Netflix.
|
| Maybe you don't know the talent within your own organization.
| Which is entirely understandable given your proclamation:
| Our engineers are fucking morons.
|
| Then again, maybe this person who left your organization is
| accurately described as such, which really says more about the
| Fortune 500 company employing him and presumably continues to
| employ yourself.
|
| IOW, either the guy left to get out from under an EM who says
| he is a "fucking moron" _or_ he actually is a "fucking moron"
| and you failed as a manager to elevate his skills/performance
| to a satisfactory level.
| JCharante wrote:
| > or he actually is a "fucking moron" and you failed as a
| manager to elevate his skills/performance to a satisfactory
| level.
|
| sometimes managers don't have the authority to fire somebody
| and are forced to keep their subordinates. Yes good managers
| can polish gold, but polishing poop still results in poop.
| bryanrasmussen wrote:
| I was consulting at a place, there was a very bad
| programmer whose code looked sort of like this
|
| cost arrayIneed = [];
|
| const arrayIdontNeed = firstArray.map(item => {
|
| if(item.hasProp) { arrayIneed.push(item); }
|
| });
|
| return arrayIneed;
|
| the above is very much a cleaned up and elegant version of
| what he would actually push into the repo.
|
| he left for a competitor in the same industry, this was at
| the second biggest company for the industry in Denmark and
| he left for the biggest company - presumably he got a pay
| raise.
|
| I asked the manager after he was gone, one time when I was
| refactoring some code of his - which in the end just meant
| throwing it all out and rewriting from scratch - why he had
| been kept on so long, and the manager said there were some
| layoffs coming up and he would have been out with those but
| because of the way things worked it didn't make sense to
| let him go earlier.
| thrwaway1985882 wrote:
| > the manager said there were some layoffs coming up and
| he would have been out with those but because of the way
| things worked it didn't make sense to let him go earlier.
|
| Incentives are fucked across the board right now.
|
| Move on a low performer today and you'll have an uphill
| battle for a backfill at all. If you get one, many
| companies are "level-normalizing" (read: level-- for all
| backfills). Or perhaps your management thinks the job
| could be done overseas cheaper, or you get pushed to turn
| it into a set of tasks so you can farm it out to
| contractors.
|
| So you keep at least some shitty devs to hold their
| positions, and as ballast to throw overboard when your
| bosses say "5% flat cut, give me your names". We all do
| it. If we get back to ZIRP I'll get rid of the actively
| bad devs when I won't risk losing their position
| entirely. Until then, it's all about squeezing what
| limited value they have out and keeping them away from
| anything important.
| bryanrasmussen wrote:
| this however was back when incentives were not so messed
| up, but sure.
| briansm wrote:
| White-Knighting for 'fucking morons' is not a good look
| though. You'll end up in a world where packets of peanuts
| have a label on saying 'may contain nuts'.
| rjh29 wrote:
| Which would be doubly silly as peanuts aren't actually
| nuts.
| jhugo wrote:
| ... which is why the label makes sense. They may have
| been contaminated with nuts during production.
| bryanrasmussen wrote:
| I think acting as if peanuts are actually nuts for
| purposes of communication is much more defensible than
| acting as if tomatoes are vegetables, in short you are
| dying on a hill that was paved over long ago.
| rjh29 wrote:
| I agree most people will conflate them, but someone who's
| allergic to peanuts but not tree nuts (or vice versa),
| i.e. the people the labels are intended for, are going to
| care about the difference.
| briansm wrote:
| ... or a world where grown adults pay millions of dollars
| to watch grown adults fighting like school children.
|
| In fact, what am I even doing in this thread? - close-tab.
| horns4lyfe wrote:
| And you think white knighting for managers who call their
| directs all "fucking morons" is a good look?
| ripped_britches wrote:
| This is the funniest thing I've read today
| IshKebab wrote:
| > failed as a manager to elevate...
|
| Managers aren't teachers. They can spend _some_ time
| mentoring and teaching but there 's a limit to that. I've
| worked with someone who could not write good code and no
| manager could change that.
|
| Most people I've worked with aren't like that of course
| (there's really only one that stands out), so maybe you've
| just been lucky enough to avoid them.
|
| I do find it unlikely that _all_ of his engineers are morons,
| but on the other hand I haven 't worked for a typical fortune
| 500 company - maybe that's where all the mediocre programmers
| end up.
| midtake wrote:
| Top troll bro
| 29athrowaway wrote:
| What are the chances that your entire engineering team is
| entirely composed of low performers or people with bad attitude
| or whatever you designate as "fucking morons"?
|
| It's more likely that you are bad at managing, growing and
| motivating your team.
|
| Even if it was true, to refer to your team in this way makes
| you look like you are not ready for management.
|
| Your duty is to get the most out of the team, and that mindset
| won't help you.
| notadoomer236 wrote:
| Don't agree. Sometimes you can observe the world around you,
| and it's not pretty. Are they not allowed to observe the
| truth as they see it? What if they are right?
| ohyes wrote:
| Sometimes if everyone else is the problem you are the problem.
| gamblor956 wrote:
| _Our engineers are fucking morons_
|
| If your "dumbest engineer" got a job and a hefty raise going to
| Netflix, it means he was very capable engineer who was playing
| the part of moron at this Fortune 500 company because he was
| reporting to a manager who was calling him and the entire team
| morons and he didn't feel the need to go above and beyond for
| that manager.
|
| Also, highly likely that it was the manager that was the moron
| and not everyone around him.
| throwawaythekey wrote:
| > If your "dumbest engineer" got a job and a hefty raise
| going to Netflix, it means he was very capable engineer who
| was playing the part of moron
|
| It's also possible that there's very little correlation
| between capability, reputation and salary.
|
| Don't we all know someone who is overpaid? There are more
| than a few well known cases of particular employers who
| select for overpaid employees...
| krzyk wrote:
| > Don't we all know someone who is overpaid?
|
| Yes, usually managers.
| normie3000 wrote:
| > well known cases of particular employers who select for
| overpaid employees
|
| Not well-known enough, apparently. Where should I be
| applying?
| throwawaythekey wrote:
| There are different forms of overpayment but to give some
| examples:
|
| - The recent story of AWS using serverless for video
| processing comes to mind [1].
|
| - Google is renowned for rest and vest.
|
| - Many government jobs pay more than their private
| counterparts.
|
| - Military contractors
|
| - Most of the healthcare industry
|
| - Lobbyists
|
| [1] https://news.ycombinator.com/item?id=35811741
| johnisgood wrote:
| In terms of healthcare industry in Hungary: one worker
| does the same job for 700 USD a month and another for
| 1100 USD, the only difference is formal education and
| years worked in the industry. You can perform much better
| (by actually caring about the patients in those 12 hours
| you work) but you will get paid the same amount
| regardless. Of course if you have 3 kids (whether they
| are adults or not) then you do not pay taxes (or much
| less than someone who does not have kids or only has 2).
| reactordev wrote:
| Sounds like he got a better deal. If this is how you describe
| your team, I suspect they are all submitting their resumes
| hoping to get away from you.
| cultofmetatron wrote:
| > Our engineers are fucking morons.
|
| I interviewed at Netflix a few years ago; with several of their
| engineers. One thing I cannot say is that they are morons.
|
| their interview process is top notch too and while I was
| ultimately rejected, I used their format as the base template
| for how I started hiring at my company.
| silisili wrote:
| I don't have a dog in this fight, but you typically use your
| A players for hiring/interviews.
|
| It can be both true that Netflix has God tier talent and a
| bunch of idiots. In fact, that's probably true of most
| places. I guess the ratio matters more.
| parpfish wrote:
| oR god tier talent and a bunch of other god tier talent
| that decided to coast and cash their fat checks
| tayo42 wrote:
| what seemed good about it that makes different then any other
| hiring process that seems detached from the job?
| blinded wrote:
| You ever thought they were doing the bare minimum and studying
| at night to leave?
| justmarc wrote:
| Your job must be truly awful.
| throw10920 wrote:
| I'm going to avoid leaving a zero-effort response like
| "actually you're the problem" like half of the replies and
| contribute:
|
| Why do you call your engineers morons? Is it a lack of
| intelligence, a lack of _wisdom_ , a lack of experience,
| inability to meet deadlines, reading comprehension, or
| something else?
|
| I wonder if Netflix is just hiring for different criteria (e.g.
| you want people who will make thoughtful decisions while they
| want people who have memorized all the leetcode problems).
| _giorgio_ wrote:
| I don't understand why nobody here believes you.
|
| There's no reason to doubt what you say, probably people
| identify with the mistreated one. Why?
| horns4lyfe wrote:
| Because the idea that all the engineers that work at his
| large company are morons is absurd. Anyone in that situation
| that believes that and even more, states it, is just making
| their own character flaws apparent.
| sgarland wrote:
| It's hyperbole, like a teacher complaining to others, "my
| kids were all crazed animals today."
|
| I've worked with engineers where I had to wonder how they
| found their computer every morning. I can easily see how a
| few of those would make you bitter and angry.
| _giorgio_ wrote:
| Let me think about it...
|
| All the engineers in MY company are morons.
|
| They're just bureucrats.
| horns4lyfe wrote:
| Sounds like you're a good match for their team then
| dang wrote:
| (We detached this subthread from
| https://news.ycombinator.com/item?id=42154036.)
| harimau777 wrote:
| Currently trying to watch it and it's not loading at all for me.
| Re-subscribed specifically for the fight.
| Jabbs wrote:
| Looks like I'm playing Tysons Punchout right now
| selimthegrim wrote:
| Glass Jake?
| ralph84 wrote:
| Main event hasn't even started yet. Traffic will probably 10x for
| that. They're screwed. Should have picked something lower profile
| to get started with live streaming.
| pythonguython wrote:
| I don't work in tech. Is this something that engineers could
| respond to and reallocate resources to fix mid stream?
| GauntletWizard wrote:
| Not a chance. This level of infrastructure was set up days in
| advance - I would be unsurprised if they'd had a code freeze
| all week for this fight.
| tonygiorgio wrote:
| Had issues all stream but was perfect during the final
| fight.
| jdub wrote:
| They've done quite a bit of lower profile live streams...
| various events, and the Everybody's in LA chat show series.
| dools wrote:
| Can Mike Judge please stop predicting everything?
| RaftPeople wrote:
| I've been re-watching Silicon Valley the last few weeks and
| just watched the Nucleus live stream episode 2 days ago, pretty
| funny seeing it in real life.
| dools wrote:
| "Puts data compression in a rear naked chokehold"
| codethief wrote:
| Reference for everyone else:
| https://m.youtube.com/watch?v=9IGvzb-KCpY
| Jamie9912 wrote:
| Works in Australia. Maybe their CDN is under a lot of stress?
| gonzo41 wrote:
| Don't jinx it.
| geoffmunn wrote:
| In NZ, it was had maybe 2 low-quality moments, but never froze
| and was in high-definition for the rest of the time.
| jameson wrote:
| Dumb question
|
| Isn't live streaming at scale already solved problem by cable
| companies? I never seen ESPN going down during a critical event
| ilrwbwrkhv wrote:
| Yes, as I have said again and again on hacker news in different
| comments Netflix went overboard with their microservices and
| tried to position itself as a technological company when it's
| not. It has made everything more complex and that's why any
| Netflix tech blog is useless because it is not the way to build
| things correctly.
|
| To understand how to do things correctly look at something like
| pornhub who handle more scale than Netflix without crying about
| it.
|
| The other day I was having this discussion with somebody who
| was saying distributed counter logic is hard and I was telling
| them that you don't even need it if Netflix didn't go
| completely mental on the microservices and complexity.
| dilyevsky wrote:
| This is not the same streaming - netflix is doing that over
| HTTP. Totally different tech and scaling issues
| hmcq6 wrote:
| Yes and no. It's not the "same" but this _is_ a solved
| problem. Fastly regularly delivers the Super Bowl to 10x as
| many viewers.
|
| Netflix dropped the ball hard
| dilyevsky wrote:
| Fastly says they do 6M ccv for superbowl (i'm actually
| surprised they let them do the entire thing and don't mix
| different CDNs) and I'm not sure they do encoding and
| manifest serving - they might just cache/deliver chunks. Do
| you really think tyson vs other guy was only 600k ccv? I'd
| be shocked if Netflix can't handle this.
| bastard_op wrote:
| You would think, but technology always finds a way to screw
| things up. Cox Communications has had ongoing issues with their
| video for weeks because of Juniper router upgrades and even the
| vendor can't fix it. They found this out AFTER they put it in
| production. Shit happens.
| kalesh wrote:
| It's down permanently for me in India. We have Hotstar, which has
| a record of 58 million viewers during the cricket World Cup
| final. Way ahead.
| anakaine wrote:
| Probably less about the level of advancement and more about
| their ability to stream vs play VOD. Two different kinds of
| infrastructure optimisation.
| margaretdouglas wrote:
| Wasn't that the biggest concurrent stream ever?
| scrapcode wrote:
| They have absolutely shit the bed here, and of course their
| socials are completely ignoring it.
| quickslowdown wrote:
| Bet they wish they'd gone with middle out compression
| robertlagrant wrote:
| When they come up with that idea it's the most 18-rated and
| accurate way an engineer would think about it.
| ksec wrote:
| Netflix is good only on streaming ready made content, not _live_
| streaming, but;
|
| 1. Netflix is a _300B_ company, this isn 't a resources issue.
|
| 2. This isn't the first time they have done live streaming at
| this scale either. They already have prior failure experience,
| you expect the 2nd time to be better, if not perfect.
|
| 3. There were plenty of time between first massive live streaming
| to second. Meaning plenty of time to learn and iterate.
| crowcroft wrote:
| Yea, the issue here isn't just that they're having issues, it's
| that they're having the same issues they've had before.
| ilrwbwrkhv wrote:
| You can't solve your way out of a complex problem that you have
| created and which wasn't needed in the first place. The entire
| microservices thing was overly complex with zero benefits
|
| I spoke to multiple Netflix senior technicians about this.
|
| They said that's the whole shtick.
| iLoveOncall wrote:
| That's a ridiculous statement. PrimeVideo is the leader in
| terms of sports events streaming over internet and it is
| composed of hundreds of microservices.
|
| Live streaming is just much harder than streaming, and it
| takes a years of work and a huge headcount to get something
| good.
| sgarland wrote:
| Prime famously undid some amount of their microservices
| recently because it couldn't keep up, and was hideously
| expensive.
| iLoveOncall wrote:
| It was a single team for a very specific use case.
|
| To be clear when I said that PrimeVideo is composed of
| hundreds of microservices, I actually meant that it's
| composed of hundreds of services, themselves composed,
| more often than not, of multiple microservices.
|
| Depending on your definition of a microservice, my team
| alone owns dozens.
| crop_rotation wrote:
| This comment shows how a very random blog about a very
| small part of a product can dominate all conversation
| about it. Prime video famously did not undo anything. Out
| of 100+ teams one team undid one service. But somehow
| similar comments are common on HN. I am making no
| judgement or microservice or not just on this particular
| comment.
| that_guy_iain wrote:
| They have the NFL next month on Christmas day. So that'll be a
| big streaming session but I think it'll be nothing compared to
| this. Even Twitter was having problems handling the live pirate
| streams there.
| jonlucc wrote:
| > Even Twitter was having problems
|
| Is that a surprise? They're not who I would think of first as
| a gold standard for high viewership live streams.
| that_guy_iain wrote:
| Well, considering it was multiple small streams I would
| expect them to keep them up. No have their entire streaming
| service have issues.
| hughesjj wrote:
| Yeah didn't they crash on love is blind or one of their reality
| shows recently-ish?
| freefaler wrote:
| The problem is that provisioning vast capacity for peak
| viewership is expensive and requires long-term commitment. Some
| providers won't give you more connectivity to their network
| unless you sign a 12 month deal where you prepay that.
|
| Peak traffic is very expensive to run, because you're building
| capacity that will be empty/unsused when the event ends. Who'd
| pay for that? That's why it's tricky and that's why Akamai
| charges these insane prices for live streaming.
|
| A "public" secret in that network layer is usually not
| redundant in your datacenter even if it's promised. To have
| redundant network you'd need to double your investment and
| it'll seat idle of at 50% max capacity. For 2hr downtime per
| year when you restart the high-capacity routers it's not cost
| efficient for most clients.
| treflop wrote:
| Then sign a contract with Akamai, who has been in business
| for 25 years? You outsource if you aren't planning to do
| something very often.
|
| There is no middle ground where you commit a mediocre amount
| of resources, end up with downtime and a mediocre experience,
| and then go "but we saved money."
| freefaler wrote:
| Well, they didn't want to spend the money or more likely
| their own technical team/boss promised that they can do it
| themselves.
|
| They indeed have a great CDN network, but it's not very
| good for this particular type of traffic. May be they will
| know/fix/buy next time...
| curiousDog wrote:
| Utter incompetence from senior leadership at Netflix. They had so
| much time to prepare for this.
| PittleyDunkin wrote:
| I want to index everyone sneering at this situation and never
| work with any of them.
| seizethecheese wrote:
| Eh, punching up, while still punching, doesn't seem that
| distasteful to me.
| robertlagrant wrote:
| There's no up. There's just punching, and making excuses
| for punching.
| kpw94 wrote:
| yep, especially knowing this isn't their first rodeo... 18
| months since https://time.com/6272470/love-is-blind-live-
| reunion-netflix/
|
| > But the real indicator of how much Sunday's screw-up ends up
| hurting Netflix will be the success or failure of its next live
| program--and the next one, and the one after that, and so on.
| There's no longer any room for error. Because, like the newly
| minted spouses of Love Is Blind, a streaming service can never
| stop working to justify its subscribers' love. Now, Netflix has
| a lot of broken trust to rebuild.
| purpleidea wrote:
| I think they must be noticing the issues, because I've noticed
| they've been dropping the stream quality quite substantially...
| It's a clever trick, but kind of cheap to do so, because who
| wants to watch pixelated things?
| nomilk wrote:
| To be brutally honest if it's a choice between pixelated and
| constantly buffering, pixelated is way less bad. Constantly
| buffering is incredibly annoying during live sports. (but this
| doesn't negate your main point which is that if people paid to
| watch they expect decent resolution)
| nomilk wrote:
| It will never not annoy and amuse me that illegal options
| (presumably run by randoms in their spare time) are so much
| better than the offerings of big companies and their tech
| 'talent'.
| Bilal_io wrote:
| I have Netflix purchased legally with hard earned money. But
| because I had issues I looked for illegal streams, and they
| were bad, crashes, buffering.. you name it. So I went back to
| Netflix and watched it at 140p quality.
| samschooler wrote:
| This twitter stream was the most reliable for me. Completely
| took Netflix out of the equation; just some dude at the event
| with his phone: https://x.com/i/broadcasts/1mrxmMRmXpQxy
| koolba wrote:
| > just some dude at the event with his phone
|
| Antonio Brown is not "just some dude". He's a national
| treasure.
| bayarearefugee wrote:
| Its a good thing he's rich and famous otherwise there
| might actually be consequences for him illegally
| broadcasting this.
| rk06 wrote:
| Illegal options would have lot less active users. So it is not
| a fair comparison
| hughesjj wrote:
| Yup. Also a bit more latency since it's effectively
| restreaming unless it's someone at the actual event.
| boramalper wrote:
| Illegal options also have lot less resources (revenue,
| service providers who are willing host/facilitate illegal
| activities, and so on) so it's a fair comparison in my
| opinion.
| 0x0000000 wrote:
| > service providers who are willing host/facilitate illegal
| activities
|
| At least for NFL pirate streams, it seems they tend to use
| "burner" tenants from Azure and AWS. Of course they get
| shut down, but how hard is it to spin up another one?
| boramalper wrote:
| They still have to put it behind a privacy-friendly proxy
| to hide their IP address from litigators right?
| johnny_canuck wrote:
| It's far from perfect here in Canada, I keep having to pause it
| or go back and then load it again.
|
| Oddly having watched PPV events via the high seas for years, it
| feels normal...
| owenpalmer wrote:
| I'm watching the event as I'm writing this. I've been needing to
| exit the player and resume it constantly. Pretty surprising that
| Netflix hasn't weeded out these bugs.
| miek wrote:
| I switched to watching on the android app and it's been
| flawless. Sad, but workable
| callc wrote:
| I couldn't watch a show a couple days ago. Long time customer,
| and first time I've considered cancelling. Broke the basic
| contract of I give $ and Netflix give show.
| bluedino wrote:
| It's been fine since 11:00 EST, I wonder if they started using
| the CDN more effectively and pushed everyone back a few minutes?
| unsnap_biceps wrote:
| Mine just crashed and reloaded to "Envoy Overloaded"
| JoyfulTurkey wrote:
| Netflix has some NFL games on Christmas Day. Wonder how those
| will go for them.
|
| I remember when ESPN started streaming years back, it was awful.
| Now I almost never have problems with their live events,
| primarily their NHL streams.
| zoklet-enjoyer wrote:
| Wow I feel scammed. I paid for a Netflix subscription
| specifically for this but it's not loading so I'm watching on an
| illegal streaming website
| djbusby wrote:
| Mine is glitchy, but if I refresh i get a good steam for a bit,
| then it gets low res, then freeze. If I wait for auto-reconnect
| it takes forever. Hard refresh and I'm good. Like, new streams to
| new server, then overloaded, then does as if their cluster is
| crashing and healing is rapid cycles. Sawtooth patterns on their
| charts.
|
| And then all these sessions lag, or orphan taking up space, so
| many reconnections at various points in the stream.
|
| System getting hammered. Can't wait for this writeup.
| magic_man wrote:
| Illegal streams are working but netflix is not. That is crazy.
| gurjeet wrote:
| > envoy overloaded
|
| That's the plain-text message I see when I tried to refresh the
| stream.
|
| Follow-up:
|
| My location: East SF Bay.
|
| Now even the Netflix frontpage (post login,
| https://www.netflix.com/browse ) shows the same message.
|
| The same message even in a private window when trying to visit
| https://www.netflix.com/browse
|
| The first round of the fight just finished, and the issues seem
| to be resolved, hopefully for good. All this to say what others
| have noted already, this experience does not evoke a lot of
| confidence in Netflix's live-streaming infrastructure.
| smitelli wrote:
| Ah, envoy. Now that is a name I have not missed.
| tinyhouse wrote:
| It's not lagging for me. It crashed and not coming back.
|
| Update: Switched to the app on my phone and so far so good.
| KingOfCoders wrote:
| Not enough chaos monkey engineering.
| margaretdouglas wrote:
| There was some blog post on HN the other day where someone said
| they don't do chaos monkey anymore... Even then, how do you
| chaos test a novel event ahead of time?
| betaby wrote:
| Sounds like a scene from: Silicon Valley - Nucleus fails
|
| https://www.youtube.com/watch?v=9IGvzb-KCpY
| Sateeshm wrote:
| My kid woke me up complaining internet is not working. Turns out
| he is trying to watch the fight and it's not working at all here
| in India.
| blinded wrote:
| It's a learning experience! I remember Conor and Floyd broke hbo
| and the ufc. It's a hard problem for sure!
|
| Some buffering issues for us, but I bet views are off the charts.
| Huge for Netflix, bad for espn, paramount, etc etc
| Bilal_io wrote:
| You're lucky you only had some buffering issues. This is got
| the case for many people, I don't know the percentage, but many
| people on reddit were complaining.
|
| This is bad for Netflix imo.
| blinded wrote:
| I guess it kinda depends on viewer counts?
| everly wrote:
| Cable TV (or even OTA antenna in the right service area) is
| simply a superior live product compared to anything streaming.
|
| The Masters app is the only thing that comes close imo.
|
| Cable TV + DVR + high speed internet for torrenting is still an
| unmatched entertainment setup. Streaming landscape is a mess.
|
| It's too bad the cable companies abused their position and lost
| any market goodwill. Copper connection direct to every home in
| America is a huge advantage to have fumbled.
| ericcholis wrote:
| The Masters app is truly incredible, I don't know if it gets
| enough praise.
| mmooss wrote:
| What's so great about it?
| crote wrote:
| The interesting thing is that a lot of TV infrastructure is now
| running over IP networks. If I were to order a TV connection
| for my home I'd get an IPTV box to connect to my broadband
| router via Ethernet, and it'd simply tell the upstream router
| to send a copy of a multicast stream my way.
|
| Reliable and redundant multicast streaming is pretty much a
| solved problem, but it does require everyone along the way to
| participate. Not a problem if you're an ISP offering TV,
| definitely a problem if you're Netflix trying to convince every
| single provider to set it up for some one-off boxing match.
| muddi900 wrote:
| I can't see the fight right now.
| gamblor956 wrote:
| After a few buffering timeouts during the first match, the rest
| of the event had no technical difficulties (in SoCal, so close to
| one of Netflix's HQs).
|
| Unfortunately, except for the women's match, the fights were
| pretty lame...4 of the 6 male boxers were out of shape. Paul and
| Tyson were struggling to stay awake and if you were to tell me
| that Paul was just as old as Tyson I would have believed it.
| shrubble wrote:
| On X.com someone had a stream that was stable to at least 5
| million simultaneous viewers, but then (as I expected) someone at
| Netflix got them to pull the plug on it. So I would expect this
| fight to have say, 50 million + watching? Maybe as many as
| 150-250 million worldwide, given this is Tyson's last fight.
| wayoverthecloud wrote:
| I watched the whole fight with a 2 minute delay. That was
| frustrating and it didn't help that Tyson lost.
| smegger001 wrote:
| Well he is almost 60 and the average retirement age for pro
| boxers is in the mid 30s. He is well past his prime and in
| physically demanding sport that is very hard on participants.
| Dem_Boys wrote:
| What do you think were the dynamics of the engineering team
| working on this?
|
| I'd think this isn't too crazy to stress test. If you have 300
| million users signed up then you're stress test should be 300
| million simultaneous streams in HD for 4 hours. I just don't see
| how Netflix screws this up.
|
| Maybe it was a management incompetence thing? Manager says
| something like "We only need to support 20 million simultaneous
| streams" and engineers implement to that spec even if the 20
| million number is wildly incorrect.
| margaretdouglas wrote:
| Has there ever been a 300m concurrent live stream? I thought
| Disney+ had the record at something like 60m.
| JCharante wrote:
| probably an esports match hosted on bilibili
| markus92 wrote:
| World Cup final, if you add up all streams worldwide?
| nonameiguess wrote:
| There's no way 300 million people watched this, especially if
| that number is representing every Netflix subscriber. The
| largest claimed live broadcast across all platforms is last
| year's Super Bowl with 202 million unique viewers for at
| least part of it, but that includes CBS, Nickelodeon, and
| Univision, not just streaming audiences. Its average viewers
| for the whole game was 123 million, which is second all-time
| to the Apollo 11 moon landing.
| vitus wrote:
| FIFA claimed the 2022 World Cup final reached 1.5 billion
| people worldwide, but again that seems like it was mostly
| via broadcast television and cable.
|
| As far as single stream, Disney's Hotstar claimed 59
| million for last year's Cricket World Cup, and as far as
| the YT platform, the Chandrayaan-3 lunar landing hit 8
| million.
|
| 100 million is a _lot_ of streams, let alone 300. But also
| note that not every stream reaches a single individual.
|
| And, as far as the 59 million concurrent streams in India,
| the bitrate was probably very low (I'd wager no more than
| 720p on average, possibly even 480p in many cases). It's
| again a very different problem across the board due to
| regional differences (such as spread of devices, quality of
| network, even behavioral differences).
| iamjackg wrote:
| Just adding a data point, here in Canada on my nVidia Shield it
| went down to 360p a dozen times or so, but never paused at all. I
| guess I got lucky.
| jaarse wrote:
| Everyone pointing out that their illegal streams, X streams, etc.
| work fine are kind of missing the point.
|
| These secondary streams might be serving a couple thousand users
| at best.
|
| Initial estimates are in the hundreds of millions for Netflix.
| Kind of a couple of orders of magnitude difference there.
| samatman wrote:
| Piracy is distributed, yes.
|
| I think that is the point, in fact.
| robertlagrant wrote:
| Everything's easy when it's someone else's content.
| samatman wrote:
| I have to assume this is some snarky way of saying
| "violating copyright is Bad, m'kay".
|
| Because taken at face value it's false. Any technical
| challenges involved in distributing a stream cannot
| possibly be affected by the legal status of the bits being
| pushed across the network.
| robertlagrant wrote:
| They're not used to live. I imagine that's it. All their caching
| infrastructure is there assuming the content isn't currently
| being generated.
| impulser_ wrote:
| Hopefully they fix it because they are hosting two Christmas NFL
| games this year and if you want to really piss people off you
| have buffering issues during NFL games lol.
| 0xpgm wrote:
| Maybe this was a stress test for the NFL games?
|
| I'd expect the NFL games to have a largely American audience,
| but today's boxing event attracted a global audience.
| uptownfunk wrote:
| It was so bad. So so bad. Like don't use your customers as guinea
| pigs for live streaming. So lame. They need a new head of content
| delivery. You can't charge customers like that and market a
| massive event and your tech is worse than what we had from live
| broadcast tv.
| subless wrote:
| I ended up turning my TV off and watching from my phone because
| of the buffering/freezing. The audio would continue to play and
| the screen would be frozen with a loading percentage that never
| changed.
|
| I have Spectrum (600 Mbps) for ISP and Verizon for mobile.
| iamzycon wrote:
| From my limited understanding, the NFL heavily depends on the
| Netflix Open Connect platform to stream media to edge locations,
| which is different from live streaming. Probably, they over-
| pushed the HD contents.
| odinthedog wrote:
| Does anyone have any thoughts besides "bad engineering" on what
| could've gone wrong? It seems like taking on a new endeavor like
| streaming an event that would possibly draw many hundreds of
| millions of viewers doesn't make sense. Is there any obvious way
| that this would just work, or is there obviously a huge mistake
| deeply rooted in the whole thing. Also, are there any educated
| guesses on some fine details in the codebase and patterns that
| could result in this?
| normie3000 wrote:
| Did anyone else see different behaviour with different clients?
| My TV failed on 25% loaded, my laptop loaded but played for a
| minute or two before getting stuck buffering, and my iphone
| played the whole fight fine. All on the same wifi network.
| rdtsc wrote:
| Live streaming and streaming prerecorded movies is a whole
| different ballgame.
|
| In fact, optimizing for later can hurt the former.
|
| Would be interesting to read any postmortems on this failure.
| Maybe someone will be kind enough to share the technical details
| for the curious crowd.
| pests wrote:
| Amazon had issues last year too when they started broadcasting
| TNF but its fine these days.
|
| I'm sure they will get it figured out.
| thr0waway001 wrote:
| Shoulda used middle out compression.
| new_user_final wrote:
| I am curious about their live streaming infrastructure.
|
| I have done live streaming for around 100k concurrent users. I
| didn't setup infrastructure because it was CloudFront CDN.
|
| Why it is hard for Netflix. They have already figured out CDN
| part. So it should not be a problem even if it is 1M or 100M.
| because their CDN infrastructure is already handling the load.
|
| I have only work with HLS live streaming where playlist is
| constantly changing compared to VOD. Live video chunks work same
| as VOD. CloudFront also has a feature request collapsing that
| greatly help live streaming.
|
| So, my question is if Netflix has already figured out CDN, why
| their live infrastructure failing?
|
| Note: I am not saying my 100k is same scaling as their 100M. I am
| curious about which part is the bottleneck.
| vitus wrote:
| > Why it is hard for Netflix. They have already figured out CDN
| part. So it should not be a problem even if it is 1M or 100M.
| because their CDN infrastructure is already handling the load
| ... Note: I am not saying my 100k is same scaling as their
| 100M. I am curious about which part is the bottleneck.
|
| 100k concurrents is a completely different game compared to 10
| million or 100 million. 100k concurrents might translate to
| 200Gbps globally for 1080p, whereas for that same quality, you
| might be talking 20T for 10 million streams. 100k concurrents
| is also a size such that you could theoretically handle it on a
| small single-digit number of servers, if not for latency.
|
| > CloudFront also has a feature request collapsing that greatly
| help live streaming.
|
| I don't know how much request coalescing Netflix does in
| practice (or how good their implementation is). They haven't
| needed it historically, since for SVOD, they could rely on
| cache preplacement off-peak. But for live, you essentially need
| a pull-through cache for the sake of origin offload. If you're
| not careful, your origin can be quickly overwhelmed. Or your
| backbone if you've historically relied too heavily on your
| caches' effectiveness, or likewise your peering for that same
| reason.
|
| 200Gbps is a small enough volume that you don't really need to
| provision for that explicitly; 20Tbps or 200Tbps may need
| months if not years of lead time to land the physical hardware
| augments, sign additional contracts for space and power, work
| with partners, etc.
| junior44660 wrote:
| This is probably a naive question but very relevant to what we
| have here.
|
| In a protocol where a oft-repeated request goes through multiple
| intermediaries, usually every intermediate will be able to cache
| the response for common queries (Eg: DNS).
|
| In theory, ISPs would be able to do the same with the HTTP.
| Although I am not aware of anyone doing such (since it will
| rightfully raise concerns of privacy and tampering).
|
| Now TLS (or other encryption) will break this abstraction. Every
| user, even if they request a live stream, receives a differently
| encrypted response.
|
| But live stream of a popular boxing match has nothing to do with
| the "confidentiality" of encryption protocol, only integrity.
|
| Do we have a protocol which allows downstream intermediates eg
| ISPs to cache content of the stream based on demand, while a
| digital signature / other attestation being still
| cryptographically verified by the client?
| dboreham wrote:
| What you describe is called a CDN and has been widely used for
| 20 years.
| ilaksh wrote:
| I guarantee this is a management issue. Somebody needed to bear
| down at some point and put the resources into load testing. The
| engineers told them it probably won't be sufficient.
|
| I assume this came down to some technical manager saying they
| didn't have the human and server resources for the project to
| work smoothly and a VP or something saying "well, just do the
| best you can.. surely it will be at least a little better than
| last time we tried something live, right?"
|
| I think there should be a $20 million class action lawsuit, which
| should be settled as automatic refunds for everyone who streamed
| the fight. And two executives should get fired.
|
| At least.. that's how it would be if there was any justice in the
| world. But we now know there isn't -- as evidenced by the fact
| that Jake Paul's head is still firmly attached to his body.
| 2-3-7-43-1807 wrote:
| I thought it's only the best of the best of the best working at
| Netflix ... or maybe we can just put this myth to sleep that
| Netflix even knows what it's doing. The suggestions are shit, the
| UX is shit, apparently even the back end sucks.
| skwee357 wrote:
| I'm very disappointed.
|
| Woke up at 4am (EU here), to tune for the main event. Bought
| Netflix just for this. The women fight went good, no buffering,
| 4K.
|
| As it approached the time for Paul vs Tyson, it started to first
| drop to 140p, and then constantly buffer. Restarted my chromecast
| a few times, tried from laptop, and finally caught a stream on my
| mobile phone via mobile network rather than my wifi.
|
| The TV Netflix kept blaming my internet which kept coming back as
| "fast".
|
| Ended up watching the utterly disappointing, senior abuse, live
| stream on my mobile phone with 360p quality.
|
| Gonna cancel Netflix and never pay for it it again, nor watch
| hyped up boxing matches.
| hiyer wrote:
| On a tangential note, the match totally looked fixed to me -
| Tyson was barely throwing any punches. I understand age is not on
| his side, but he looked plenty spry when he was ducking, weaving
| and dodging. It seemed to me he could have done better in terms
| of attacking as well.
| adamtaylor_13 wrote:
| Yeah the biggest thing to me, and the commentators mentioned
| this as well, his legs looked REALLY wobbly.
|
| All your attacking power comes from your legs and hips, so if
| his legs weren't stable he didn't have much attacking power.
|
| I think he gave it everything he had in rounds 1 and 2.
| Unfortunately, I just don't think it was ever going to be
| enough against a moderately trained 27 year old.
| nubinetwork wrote:
| I would argue Tyson has a shorter reach, Jake was whiffing a
| lot of superman punches, and all that does is waste energy.
| Jake might be able to throw punches, but he clearly wasn't
| interested in taking them. If they stood closer and slugged it
| out, the fight could have gone either way.
| nightowl_games wrote:
| How is this story not on the front page anymore? 375 comments.
| Seems like a big story to me.
| npilk wrote:
| I believe HN's algorithm tends to relatively downrank stories
| with a high comment-to-upvote ratio, because they are more
| often flamewars on divisive topics.
| yodon wrote:
| Another major algorithmic down-ranker is vote wars on
| comments.
|
| If lots of people are upvoting and downvoting the same
| comments, that's treated as a signal the topic is contentious
| and people are likely to start behaving badly.
|
| HN is very clear they prioritize good behavior as the long
| term goal, and they are as a result comfortable having
| contentious topics fall in the ranking even if everyone
| involved in the discussion feels the topic is important.
| ajdude wrote:
| It wasn't even just buffering issues, the feed would just stop
| and never start again until I paused it and then clicked "watch
| live" with the remote.
|
| It was really bad. My Dad has always been a fan of boxing so I
| came over to watch the whole thing with him.
|
| He has his giant inflatable screen and a projector that we hooked
| up in the front lawn to watch it, But everything kept buffering.
| We figured it was the Wi-Fi so he packed everything up and went
| inside only to find the same thing happening on ethernet.
|
| He was really looking forward to watching it on the projector and
| Netflix disappointed him.
| mmooss wrote:
| > My Dad has always been a fan of boxing
|
| What did your Dad think about the 'boxing'?
| manquer wrote:
| [delayed]
| beanjuiceII wrote:
| i thought they did DSA interviews at netflix what happened? I had
| to watch the fight on someone streaming to X from their phone at
| the event and it was better than watching on netflix..if you
| could watch at all. extremely embarrassing!
| bobobob420 wrote:
| Its because they use medium-hard leetcode instead of hard. I
| suggest 8 rounds instead of 4
| taneq wrote:
| Surely more whiteboard questions are the key? Reversing a
| binary tree is so last year, they should make candidates
| reverse an octree.
| meesles wrote:
| My theory is they've so heavily optimized for static content
| and distributing content on edge nodes that they were probably
| poorly setup for live-streaming.
| moritonal wrote:
| This, I feel bad for their engineers who were told every
| bonus would be a matter of how low can they get the cost-per-
| GB of transferred data, leading to the glorious Netflix-in-a-
| box (https://openconnect.netflix.com/deploymentguide.pdf) and
| then management casually asks for a live show with a wildly
| different set of guarantee requirements.
| gosub100 wrote:
| someone's going to have to reverse a linked list of
| subscriptions.
| sylens wrote:
| Reminds me of Nucleus stuttering during UFC
| xerox13ster wrote:
| I could hear Gavin Belson screaming during the broadcast when
| my stream was freezing as they were each making their entrance.
| Mike Judge is a prophet.
| criddell wrote:
| I watched on an AppleTV and the stream was rock solid.
|
| I don't know if it's still the case, but in the past some devices
| worked better than others during peak times because they used
| different bandwidth providers. This was the battle between
| Comcast and Cogent and Netflix.
| eqvinox wrote:
| Your device type has no influence on your provider and its
| bandwidth characteristics. If you're on Comcast, Apple can't
| magically make it not suck.
| fidotron wrote:
| Native apps have a lot more scope for client side load
| balancing due to having a different security model than
| browsers.
| immibis wrote:
| If you're on Apple, Comcast can make it magically not suck.
| Not sure if it's relevant though.
|
| Cogent just seems to love picking fights with everyone (see
| Hurricane Electric). Why are they still in business?
| criddell wrote:
| It isn't Apple, it's Netflix.
|
| Remember back in 2014 or so when Netflix users on Comcast
| were getting slow connections and buffering pauses? It didn't
| affect people who watched Netflix via Apple TV because
| Netflix served Apple TV users with a different network.
|
| > In a little known, but public fact, anyone who is on
| Comcast and using Apple TV to stream Netflix wasn't having
| quality problems. The reason for this is that Netflix is
| using Level 3 and Limelight to stream their content
| specifically to the Apple TV device. What this shows is that
| Netflix is the one that decides and controls how they get
| their content to each device and whether they do it via their
| own servers or a third party. Netflix decides which third
| party CDNs to use and when Netflix uses their own CDN, they
| decide whom to buy transit from, with what capacity, in what
| locations and how many connections they buy, from the transit
| provider. Netflix is the one in control of this, not Comcast
| or any ISP.
|
| https://www.streamingmediablog.com/2014/02/heres-comcast-
| net...
| eqvinox wrote:
| Ah, Thanks, I see what you're saying now and it makes a lot
| of sense. Just didn't grok it from your previous post,
| Sorry!
| js2 wrote:
| > I watched on an AppleTV and the stream was rock solid.
|
| For me it was buffering and low resolution, on the current
| AppleTV model, hardwired, with a 1Gbps connection from AT&T.
| Some streaming devices may have handled whatever issues Netflix
| was having better than others, but this was clearly a bigger
| problem than just the streaming device.
| booleandilemma wrote:
| Was this their first time doing live content? I figured something
| would go wrong. I'm sure lots of people were watching.
| mmiyer wrote:
| Not their first time, but the first time at this scale.
| sgarland wrote:
| OK, but the last time they tried a livestream (a reality show
| reunion), it also fell over. I suppose to their credit, my
| stream never outright died yesterday, it just went to potato
| quality.
| l33t7332273 wrote:
| I thought Netflix's biggest advantage was the quality/salary of
| its engineers.
|
| I think that every time I wait for Paramount+ to restart after
| its gone black in picture on picture, and yet, I'n still on
| Paramount+ and not Netflix, so maybe that advantage isn't real.
| A4ET8a8uTh0 wrote:
| Sigh, none of the competitors are much better. Disney, who has
| more than enough cash to throw at streaming, is a near constant
| hassle for us ( after 3 or more episodes it throws an
| inscrutable error on Playstation ). I would drop it, but this
| is the only remaining streaming service and wife is not willing
| to drop it ( I guess until 1 it is one error per one episode ).
| immibis wrote:
| I keep hearing that a lot of people are switching to the
| illegal competitor for better service.
| A4ET8a8uTh0 wrote:
| Eh, the beef that I have is that I am already a paying
| customer. Why does it seem like I am getting subpar service
| in terms of delivery? I know it is a tired conversation on
| this forum, but corporations big and small do what they can
| do mess with experience to eke out few more cents from
| customers. It almost does not matter which industry one
| looks at, it is the same story; the only difference is
| typically how egregious things get[1].
|
| [1]https://www.marketwatch.com/story/i-opt-to-fly-private-
| no-ma...
| IncreasePosts wrote:
| Do you think YouTube couldn't handle it?
| grapesodaaaaa wrote:
| I think this was true at some point, but I've been disappointed
| in the quality of the OSS Netflix tools recently. I think
| before k8s and a plethora of other tools matured, they were way
| ahead of the curve.
|
| I specifically found the Netflix suite for Spring very lacking,
| and found message oriented architectures on something like NATS
| a lot easier to work with.
| dylan604 wrote:
| People just do not appreciate how many gotchas can pop up doing
| anything live. Sure, Netflix might have a great CDN that works
| great for their canned content and I could see how they might
| have assumed that's the hardest part.
|
| Live has changed over the years from large satellite dishes
| beaming to a geosat and back down to the broadcast center($$$$$),
| to microwave to a more local broadcast center($$$$), to running
| dedicated fiber long haul back to a broadcast center($$$), to
| having a kit with multiple cell providers pushing a signal back
| to a broadcast center($$), to having a direct internet connection
| to a server accepting a live http stream($).
|
| I'd be curious to know what their live plan was and what their
| redundant plan was.
| diggan wrote:
| > People just do not appreciate how many gotchas can pop up
| doing anything live.
|
| Sure thing, but also, how much resources do you think Netflix
| threw on this event? If organizations like FOSSDEM and CCC can
| do live events (although with way smaller viewership) across
| the globe without major hiccups on (relatively) tiny budgets
| and smaller infrastructure overall, how could Netflix not?
| phyrex wrote:
| Scale changes everything, I don't think it's fair to shrug
| this off
| tiluha wrote:
| This is true, but scale comes after production. Once you
| have the video encoded on a server with a stable connection
| the hard part is over. What netflix failed to do is spread
| the files to enough servers around the globe to handle the
| load. I'm surprised they were unable(?) to use their
| network of edge servers to handle the live stream. Just run
| the stream with a 10 second delay and in that time push the
| stream segments to the edge server
| dylan604 wrote:
| This right here is where I'd expect the failure to occur.
| This isn't Joey Beercan running OBS using their home
| internet connectivity.
|
| This is a major broadcast. I'd expect a full on broadcast
| truck/trailer. If they were attempting to broadcast this
| with the ($) option directly to a server from onsite,
| then I would demand my money back. Broadcasting a live IP
| signal just falls on its face so many times it's only the
| cheap bastard option. Get the video signal as a video
| signal away from the live location to a facility with
| stable redundant networking.
|
| This is the kind of thinking someone only familiar with
| computers/software/networking would think of rather than
| someone in broadcasting. It's nice to think about
| disrupting, but this is the kind of failure that
| disruptors never think about. Broadcasters have been
| there done that with ensuring live broadcasts don't go
| down because an internet connection wasn't able to keep
| up.
| shrubble wrote:
| Lumen has their Vyvvyx product/service which uses fiber
| for broadcast television.
| chgs wrote:
| I've been using vyvx since it was called global
| crossing/genesis, it was fairly unique when it started,
| but point to point ip distributon of programs has been
| the norm for at least 15 years. Still have backup paths
| on major events on a different technology, you'd be
| surprised how common a dual failure on two paths can be.
| For example output from the euro football this summer my
| mai paths were on a couple of leased lines with -7, but
| still had a backup on some local internet into a
| different city just incase there was a meltdown of the
| main providers network (it's happened before with ipath,
| automation is great until it isn't)
| mmooss wrote:
| > Once you have the video encoded on a server with a
| stable connection the hard part is over.
|
| The hard part is over, and people new to the problem
| think they are almost done, but then the next part turns
| out to be 100x harder.
|
| Lots of people can encode a video.
| diggan wrote:
| Yeah, I agree with this, especially the _everything_ part.
| Netflix isn 't exactly a scrappy FOSS/hackers organization
| or similar.
| BlueTemplar wrote:
| Last I checked, p2p solves a lot of the scaling issues.
|
| Haven't Asian live sports been using p2p already two
| decades ago ?
|
| (What is the biggest Peertube livestream so far ?)
| dylan604 wrote:
| > how much resources do you think Netflix threw on this
| event?
|
| Based on the results, I _hope_ it was a small team working
| 20% time on the idea. If you tell me they threw everything
| they had at it to this result, then that 's even more
| embarrassing for them.
| throw0101b wrote:
| > _If organizations like FOSSDEM and CCC can do live events
| (although with way smaller viewership)_ [...]
|
| Or, for that matter, Youtube (Live) and Twitch.
| bostik wrote:
| The CCC video crew has its fair share of geeks from
| broadcasting corporations and studio houses. Their combined
| institutional knowledge about live events and streaming
| distribution is probably in the same ballpark as that of
| giant global TV networks.
|
| They also have the benefit of having practiced their craft at
| the CCC events for more than a decade. Twice a year. (Their
| summer event is smaller but still fairly well known. Links to
| talks show up on HN every now and then.)
|
| Funky anecdote: the video crew at Assembly have more
| broadcasting and live AV gear for their annual event than
| most medium-sized studios.
| akira2501 wrote:
| > Their combined institutional knowledge about live events
| and streaming distribution
|
| Now if they could just get audio levels and compression
| figured out.
| bena wrote:
| It is weird because this _was_ a solved problem.
|
| Every major network can broadcast the Super Bowl without issue.
|
| And while Netflix claims it streamed to 280 million, that's if
| every single subscriber viewed it.
|
| Actual numbers put it in the 120 million range. Which is in
| line with the Super Bowl.
|
| Maybe Netflix needs to ask CBS or ABC how to broadcast
| ironhaven wrote:
| Do you live stream the superbowl? Me and everyone I know
| watch it over antenna broadcast tv. I think it is easier to
| have millions of tvs catch airwaves vs millions of point to
| point https video streams.
| dylan604 wrote:
| If you watch it over cable, you're live streaming it. Let's
| face it, that's where the vast majority of viewers see it.
| Few people view OTA even if the quality is better.
|
| Live sports do not broadcast the event directly to a
| streamer. They push it to their broadcast centers. It then
| gets distributed from there to whatever avenues it needs to
| go. Trying to push a live IP stream directly from the
| remote live venue rarely works as expected. That's
| precisely why the broadcasters/networks do not do it that
| way
| throw0101b wrote:
| > _If you watch it over cable, you 're live streaming
| it._
|
| Which is probably done over the cableco's private network
| (not the public Internet) with a special VLAN used for
| television (as opposed to general web access). They're
| probably using multicast.
| toast0 wrote:
| Is cable video over IP now? Last time I looked (which
| _was_ forever ago), even switched video was atsc with a
| bit of messaging for the cable box to ask what channel to
| tune to, and to keep the stream alive. TV over teleco
| systems seems to be highly multicast, so kind of similar,
| headend only has to send the content once, in a single
| bitrate.
|
| Not really the same as an IP service live stream where
| the distribution point is sending out one copy per viewer
| and participating in bitrate adaptation.
|
| AFAIK, Netflix hasn't publicly described how they do live
| events, but I think it's safe to assume they have some
| amount of onsite production that outputs the master feed
| for archiving and live transcoding for the different
| bitrate targets (that part may be onsite, or at a
| broadcast center or something cloudy), and then goes to a
| distribution network. I'd imagine their broadcast
| center/or onsite processing feeds to a limited number of
| highly connected nodes that feed to most of their CDN
| nodes; maybe more layers. And then clients stream from
| the CDN nodes. Nobody would stream an event like this
| direct from the event; you've got to have something to
| increase capacity.
| tredre3 wrote:
| > Is cable video over IP now?
|
| Over the US and Canada it mostly is, though how advanced
| the transition is is very regional.
|
| The plan is to drop both analog signal and digital (QAM)
| to reclaim the frequencies and use them for DOCSIS
| internet.
|
| Newer set top boxes from Comcast (xfinity) runs over the
| internet connection (in a tagged VLAN on a private
| network, and they communicate over a hidden wifi).
| akira2501 wrote:
| > If you watch it over cable, you're live streaming it.
|
| Those are multicast feeds.
|
| > Trying to push a live IP stream directly from the
| remote live venue rarely works as expected.
|
| In my experience it almost always works as expected. We
| have highly specialized codecs and equipment for this.
| The stream is actively managed with feedback from the
| receiver so parameters can be adjusted for best
| performance on the fly. Redundant connections and
| multiple backhauls are all handled automatically.
|
| > That's precisely why the broadcasters/networks do not
| do it that way
|
| We use fixed point links and satellite where possible
| because we own the whole pipe. It's less coordination and
| effort to setup and you can hit venues and remotes where
| fixed infrastructure is difficult or impossible to
| install.
| tempest_ wrote:
| When Netflix started it was the first in the space and
| breaking ground which is how they became a "tech" company
| that happens to stream media however it has been 15 years and
| since than the cloud providers have basically build "netflix
| as a service". I suspect most of the big streamers are using
| that instead of building their own in house thing and going
| through all the growing pains netflix is.
| ukuina wrote:
| You know they were commoditized when "Build Netflix" became
| a system-design interview question.
| Taylor_OD wrote:
| Solves differently though, right? Cable broadcasts are not
| the same as a streaming video over the internet, right?
| colesantiago wrote:
| This is the whole point of chaos engineering that was invented
| at Netflix, which tests the resiliency of these systems.
|
| I guess we now know the limits of what "at scale" is for
| Netflix's live-streaming solution. They shouldn't be failing at
| scale on a huge stage like this.
|
| I look forward to reading the post mortem about this.
| dylan604 wrote:
| Everyone keeps mentioning at scale. I seriously doubt this
| was an "at scale" problem. I have strong suspicion this was a
| failure at the origination point being able to push a stable
| signal. That is not an "at scale" issue, but a hubris of we
| can do better/cheaper than broadcasting standard practices
| zinodaur wrote:
| If it was a problem at origin, why did it get better/worse
| as viewership fell/rose?
| kristjansson wrote:
| As counterpoint, I observed 2-3 drops in bitrate, but an
| otherwise fine experience. So the problem seems to have
| been in dissemination, not at the origin.
| kortilla wrote:
| I highly doubt this. Netflix has a system of OCAs that are
| loaded with hard disks, are installed in ISP's networks,
| and serve the majority of those ISP's customers.
|
| Given than many people had no problems with the stream, it
| is unlikely to have been an origin problem but more likely
| the mechanism to fanout quickly to OCAs. Normally latency
| to an OCA doesn't matter when you're replicating new
| catalogs in advance, but live streaming makes a bunch of
| code that previously "didn't need to be fast" get promoted
| to the hot path.
| woobar wrote:
| I've tried to watch an old Seinfeld episode during this
| event. It was freezing every few minutes even at downgraded
| bitrate. A video that should be on my local CDN node.
| mmcgaha wrote:
| I am not sure that it is an issue with the origination
| point. In fact I just thought it was my ISP because my
| daughter's boyfriend was watching and doing facetime with
| her and my video was dropping but his was not. I have 2gb
| fiber and we regularly stream five TVs without any issue,
| so it should not have been a bandwidth issue.
| xyst wrote:
| You are making excuses for a multibillion dollar company that
| has been in this game for many years. Maybe the first to market
| in streaming.
|
| This isn't NFLX's first rodeo in live streaming. Have seen a
| handful of events pop up in their apps.
|
| There is no excuse. All of the resources and talent at their
| disposal, and they looked absolutely amateurish. Poor optics.
|
| I would be amazed if they are able to secure another exclusive
| contract like this in the future.
| DrillShopper wrote:
| A company that readily admits it burns out SWRs and SREs in
| exchange for the big bucks.
|
| Just what the fuck are these people doing?
|
| If I were a major investor in them I'd be pissed.
| Xenoamorphous wrote:
| Sorry for the off topic but what's this thing that I only
| come across in Hacker News about referring to a company by
| their stock exchange name (APPL, MSFT, etc) outside of a
| stock context? It seems really weird to me.
| rashabd wrote:
| merriam-webster.com/dictionary/brevity
| Xenoamorphous wrote:
| Writing APPL instead of Apple doesn't get you any fewer
| keystrokes.
| justinsaccount wrote:
| also, the symbol for apple is AAPL.
| SilasX wrote:
| Ugh. Similar (huge) pet peeve about people who say "n.B."
| instead of "note".
| egypturnash wrote:
| I just assume it's people who spend too much time thinking
| about the stock context.
| abduhl wrote:
| I have always assumed that a focus on stock tickers is the
| natural result when your primary user base is a group of
| people hyper focused on "total compensation" and stock
| grants. The name hackernews is merely a playful reference
| to the history of the site. Like the name "Patriot Act."
| umanwizard wrote:
| In-group signaling for people who like playing or thinking
| about the stock market. Similar to how people who make
| travel a big part of their identity refer to cities by
| their airport code.
| chgs wrote:
| You're talking about the contribution from the venue to the
| boardcast centre, increasingly not a full program but being
| mixed remotely.
|
| That's a very different area to transmission of live to end
| users.
| selimnairb wrote:
| Is multicast a thing on the commercial internet? Seems like
| that could help.
| _fat_santa wrote:
| When you step back and look at the situation, it's not hard to
| see why Netflix dropped the ball here. Here's now I see it (not
| affiliated with Netflix, pure speculation):
|
| - Months ago, the "higher ups" at Netflix struck a deal to stream
| the fight on Netflix. The exec that signed the deal was probably
| over the moon because it would get Netflix into a brand new space
| and bring in large audience numbers. Along the way the
| individuals were probably told that Netflix doesn't do
| livestreaming but they ignored it and assumed their talented
| Engineers could pull it off.
|
| - Once the deal was signed then it became the Engineer's problem.
| They now had to figure out how to shift their infrastructure to a
| whole new set of assumptions around live events that you don't
| really have to think about when streaming static content.
|
| - Engineering probably did their absolute best to pull this off
| but they had two main disadvantages, first off they don't have
| any of the institutional knowledge about live streaming and they
| don't really know how to predict demand for something like this.
| In the end they probably beefed up livestreaming as much as they
| could but still didn't go far enough because again, no one there
| really knows how something like this will pan out.
|
| - Evening started off fine but crap hit the fan later in the show
| as more people tuned in for the main card. Engineering probably
| did their best to mitigate this but again, since they don't have
| the institutional knowledge of live events, they were shooting in
| the dark hoping their fixes would stick.
|
| Yes Netflix as a whole screwed this one up but I'm tempted to
| give them more grace than usual here. First off the deal that
| they struck was probably one they couldn't ignore and as for
| Engineering, I think those guys did the freaking best they could
| given their situation and lack of institutional knowledge. This
| is just a classic case of biting off more than one can chew, even
| if you're an SV heavyweight.
| Loughla wrote:
| >First off the deal that they struck was probably one they
| couldn't ignore
|
| If you can't provide the service you shouldn't sell it?
| immibis wrote:
| You've never worked in a startup have you? Or any business
| for that matter. You have to promise something first, then
| build it.
| colesantiago wrote:
| No joke, is this actually true?
|
| Do startups really do this? I thought the capability is
| built or nearly built or at least in testing already with
| reasonable or amazing results, THEN they go to market?
|
| Do startups go to other startups, fortune 500 companies and
| public companies to make false promises with or without due
| diligence and sign deals with the knowledge that the team
| and engineers know the product doesn't have the feature in
| place at all?
|
| In other words:
|
| Company A: "We can provide web scale live streaming service
| around the world to 10 billion humans across the planet,
| even the bots will be watching."
|
| Company B: "OK, sounds good, Yes, here is a $2B contract."
|
| Company A: "Now team I know we don't have the capability,
| but how do we build, test and ship this in under 6
| months???"
| ImPostingOnHN wrote:
| Many do, as far as initial investment goes. It makes
| sense when you think about the capital intensive nature
| of most startups (including more than web startups here,
| e.g. lab tech commercialization). It also accurately
| describes a research grant.
|
| That's for startups that can't bootstrap (most of them).
| For ones which can, they may still choose to do this with
| customers, as you describe, because it means letting
| their work follow the money.
| tky wrote:
| "Aspirational sugar" is as common in startup culture as
| in Fortune 500 sales contracts, they're just messaged and
| "de-risked" differently.
| 999900000999 wrote:
| Startups absolutely sell things they haven't made yet and
| might not even be capable of doing.
|
| Next thing you know it's 9pm on a Sunday night and your
| desperately trying to ship a build for a client.
|
| Netflix isn't some scrappy company though. If I had to
| guess they threw money at the problem.
|
| A much better approach would of been to slowly scale over
| the course of a year. Maybe stream some college
| basketball games first, slowly picking more popular
| events to get some real prod experience.
|
| Instead this is like their 3rd or 4th live stream ever.
| Even a pre show a week before would of allowed for
| greater testing.
|
| I'm not a CTO of a billion dollar company though. I'm
| just an IC who's seen a few sites go down underload.
|
| To be fair no one knows how it's going to go before it
| happens. It would of been more surprising for them to
| pull this off without issues... It's a matter of managing
| those issues. I know if I had paid 30$ for a Netflix
| subscription to watch this specific event I'd assume I
| got ripped off.
| whstl wrote:
| If anything, startups are more transparent about it.
|
| In the enterprise sector this is rampant. Companies sell
| "platforms" and those missing features are supposed to be
| implemented by consultants after the sale. This means the
| buyer is the one footing the bill for the time spent, and
| suffering with the delays.
| yorwba wrote:
| You don't necessarily have to make _false_ promises.
|
| You can be totally honest and upfront that the
| functionality doesn't exist yet and needs to be built
| first, but that you think you understand the problem
| space and can handle the engineering, provided you can
| secure the necessary funding, where, by the way, getting
| a contract and some nominal revenue _now_ could greatly
| help make this a reality...
|
| And if the upside sounds convincing enough, a potential
| customer might happily sign up to cover part of your
| costs so they can be beta testers and observe and
| influence ongoing development.
|
| Of course it happens all the time that the problem space
| turns out to be more difficult than expected, in which
| case they might terminate the partnership early and then
| the whole thing collapses from lack of funding.
| _proofs wrote:
| i imagine this is why a lot of products, and startups,
| fail.
| _fat_santa wrote:
| My speculation here is this was just classic SV cockiness.
| The team that closed this deal probably knew that they didn't
| have the capability but I'm sure the arguments for doing it
| anyways was something along the lines of: "we have the best
| engineers in the bay area, we can probably figure this out"
| mbesto wrote:
| There are endless amounts of stories and situations in which
| selling something before it really exists has helped
| businesses. It's totally plausible that a team working on
| video streaming at the scale of Netflix could figure out live
| streaming.
|
| Pre-optimization is definitely a thing and it can massively
| hurt (i.e. startups go under) businesses. Let's stop
| pretending any businesses would say 'no' to extra revenue
| even before the engineering team had full assurance there was
| no latency drop.
| RicoElectrico wrote:
| Execs _never_ listen or even ask engineers about feasibility
| of projects they sign up to. Hope the exec in question will
| be let go.
| pilotneko wrote:
| I mean, the ones that do ask don't proceed to signing up. I
| think we are seeing a form of survival bias.
| throw0101b wrote:
| >> _First off the deal that they struck was probably one they
| couldn 't ignore_
|
| > _If you can 't provide the service you shouldn't sell it?_
|
| Then how will the folks in Sales get their commission?
|
| Besides, not providing the service hasn't stopped Tesla from
| selling FSD, and their stock has been going gangbusters.
|
| /s
| js2 wrote:
| > They now had to figure out how to shift their infrastructure
| to a whole new set of assumptions around live events
|
| It wasn't their first live event. A previous live event had
| similar issues.
| TheAceOfHearts wrote:
| This isn't Netflix's first foray into livestreaming. They tried
| a livestream last year for a reunion episode of one of their
| reality TV shows which encountered similar issues [0]. Netflix
| already has a contract to livestream a football event on
| Christmas, so it'll be interesting to see if their engineers
| are able to get anything done in a little over a month.
|
| These failures reflect very poorly on Netflix leadership. But
| we all know that leadership is never held accountable for their
| failures. Whoever is responsible for this should at least come
| forward and put out an apology while owning up to their
| mistakes.
|
| [0] https://time.com/6272470/love-is-blind-live-reunion-
| netflix/
| crazygringo wrote:
| > _But we all know that leadership is never held accountable
| for their failures._
|
| You've never heard of a CEO or other C-suite or VP getting
| fired?
|
| It most _definitely_ happens. On the other hand, people at
| every level make mistakes, and it 's preferable that they
| learn from them rather than be fired, if at all possible.
| TheAceOfHearts wrote:
| Accountability can take many forms. I don't think they
| should be fired for making a mistake, I think they should
| release a statement recognizing their failure along with a
| post-mortem. Not a particularly high bar, but most
| leadership failures are often swept under the rug without
| any public accountability or evidence that they've learned
| anything.
|
| We have evidence of prior failures with livestreaming from
| Netflix. Were the same people responsible for that failure
| or do we have evidence of them having learned anything
| between events? If anything, I'd expect the best leaders
| would have a track record that includes failures while
| showcasing their ability to overcome and learn from those
| mistakes. But based on what information is publicly
| available, this doesn't seem to be the case in this
| situation.
| JumpCrisscross wrote:
| Livestreaming is a solved problem. This sounds like NIH [1].
| (At the very least, hire them as a back-up.)
|
| [1] https://en.wikipedia.org/wiki/Not_invented_here
| gregorygoc wrote:
| Saying live-streaming is a solved problem is like saying
| search is a solved problem.
| JumpCrisscross wrote:
| > _Saying live-streaming is a solved problem is like saying
| search is a solved problem_
|
| It is. You can hire the people who have solved it to do it
| for you.
| talldayo wrote:
| > It is. You can hire the people who have solved it to do
| it for you.
|
| "GPGPU compute is a solved problem if you buy Nvidia
| hardware" type comment
| JumpCrisscross wrote:
| > _" GPGPU compute is a solved problem if you buy Nvidia
| hardware" type comment_
|
| You're replacing the word hire with buy. That
| misconstrues the comment. If you need to do GPGPU compute
| and have never done it, you work with a team that has.
| (And if you want to build it in house, you scale to it.)
| moralestapia wrote:
| >"GPGPU compute is a solved problem if you buy Nvidia
| hardware"
|
| Which is valid? If your problem can be solved by writing
| a check, then it's the easiest problem to have on the
| planet.
|
| Netflix didn't have to put out 3 PhD dissertations on how
| to improve the SOTA of live streaming, they only needed
| to reliably broadcast a fight for a couple hours.
|
| _That_ is a solved problem.
|
| Amazon and Cloudflare do that for you _as a service_ (!).
| Twitch and YouTube do it literally every day. Even X
| started doing it recently so.
|
| No excuses for Netflix, tbh.
| namaria wrote:
| Landing on Mars is a solved problem. Nuclear bombs are a
| solved problem. Doesn't mean anyone can just write a
| check and get it done and definitely doesn't mean any
| business model can bear that cost.
| moralestapia wrote:
| Of course it means that!
|
| You only need a big enough check.
| oehpr wrote:
| Look. I'm a small startup employee. I have a teeny tiny
| perspective here. But frankly speaking the idea that Netflix
| could just take some off the shelf widget and stuff it in
| their network to solve a problem... It's an absurd statement
| for even me. And if there's anyone it should apply to it
| would be a little startup company that needs to focus on
| their core area.
|
| Every off the shelf component on the market needs
| institutional knowledge to implement, operate, and maintain
| it. Even Apple's "it just works" mantra is pretty laughable
| in the cold light of day. Very rarely in my experience do you
| ever get to just benefit from someone else's hard work in
| production without having an idea how properly implement,
| operate, and maintain it.
|
| And that's at my little tiny ant scale. To call the problem
| of streaming "solved" for Netflix... Given the guess of the
| context from the GP post?
|
| I just don't think this perspective is realistic at all.
| JumpCrisscross wrote:
| > _the idea that Netflix could just take some off the shelf
| widget and stuff it in their network to solve a problem_
|
| Right. They have to hire one of the companies that does
| this. Each of YouTube, Twitch (Amazon), Facebook and TikTok
| have, I believe, handled 10+ million streams. The last two
| don't compete with Netflix.
| hunter2_ wrote:
| I believe this is the spirit of the "solved problem"
| comment: not that the solution is an off-the-shelf
| widget, but that if it has ever been solved, then that
| solution could technically be used again, even if
| organizing the right people is exorbitantly expensive.
|
| Offering it for sale != having solved it.
| ikiris wrote:
| There are multiple companies that offer this capability
| today that would take a few weeks to hide behind company
| branding. This was a problem of netflix just not being set
| up for live stream but thinking they could handle it.
| namaria wrote:
| "Solved" merely means you don't need to invent something new
| to solve it. It doesn't mean trivial nor easy. And it
| definitely doesn't mean the problem is above trade-offs.
| intelVISA wrote:
| Not sure why Netflix is held in high regard - this proves
| they're just as much clowns as the other 'big players' in the
| circus.
| thinkingkong wrote:
| They arent clowns at all. Ita a totally different engineering
| problem and you cant just spin up live streaming capacity on
| demand. The entire system end to end isnt optimized for live
| streams yet.
| draw_down wrote:
| I mean, maybe? You just made all this up.
| grapesodaaaaa wrote:
| I hope they do a postmortem
| yarrowy wrote:
| they should also do a business postmortem, how did the exec
| greenlight this without livestreaming infrastructure in place?
| ldjkfkdsjnv wrote:
| We all know netflix was built for static content, but its still
| hilarious that they have thousands of engineers making 500-1M in
| total comp and they couldnt live stream a basic broadcast. You
| probably could have just run this on AWS with a CDK configuration
| and quota increase from amazon
| hawk_ wrote:
| I thought Hooli was Google, but may be it was Netflix after all.
| J05ephu5M13r wrote:
| https://www.livemint.com/sports/news/mike-tyson-v-jake-paul-...
|
| "Netflix streamed the fight to its 280 million subscribers"
|
| Perhaps the technology was over-sold.
| Trasmatta wrote:
| Silicon Valley predicted this: https://youtu.be/ddTbNKWw7Zs
| cryptozeus wrote:
| Everyone here talking like this something unique netflix had to
| deal with. Hotstar live streamed india va Pakistan cricket match
| with zero issues with all time high live viewership ever in the
| history of live telecast. Why would viewers paying $20 month want
| to think about their technical issues, they dropped the ball pure
| and simple. Tech already exists for this, it's been done before
| even by espn, nothing new here.
| al_borland wrote:
| The Independent reports 35m viewers of that cricket match [0].
|
| Rolling Stone reported 120m for Tyson and Paul on Netflix [1].
|
| These are very different numbers. 120m is Super Bowl territory.
| Could Hotstar handle 3-4 of those cricket matches at the same
| time without issue?
|
| [0] https://www.the-independent.com/sport/cricket/india-
| pakistan...
|
| [1] https://www.rollingstone.com/culture/culture-news/jake-
| paul-...
| dilyevsky wrote:
| Majority of superbowl viewers watch it on cable. Streaming
| gets fewer than 10M concurrents
| achow wrote:
| India - Australia is the one of interest, scored cricket's
| highest concurrent audience ever, 59 Million.
|
| https://www.icc-cricket.com/news/biggest-cricket-world-
| cup-e...
| lxgr wrote:
| But that's exactly the point: Netflix didn't do this in a
| vacuum, they did it within Netflix.
|
| It might just have been easier to start from scratch, maybe
| using an external partner experienced in live streaming, but
| the chances of that decision happening in a tech-heavy company
| such as Netflix that seems to pride itself on being an industry
| leader are close to zero.
| colesantiago wrote:
| If you're going to be having intense algorithm interviews, paying
| top dollar for only hiring senior engineers, building high
| intensity and extreme distributed systems and having SRE
| engineers, we best see insanely good results and a high ROI out
| of it.
|
| All of the conditions was perfect for Netflix, and it seems that
| the platform entirely flopped.
|
| Is this what chaos engineering is all about that Netflix was
| marketing heavily to engineers? Was the livestream _supposed_ to
| go down as Netflix removed servers randomly?
| nova22033 wrote:
| Amazon prime streams the Thursday night NFL game and they seem to
| have no problem.
| margaretdouglas wrote:
| Isn't the scale a bit different though? Surely this event was
| an order of magnitude more concurrent viewers than some NFL
| game.
| jarsin wrote:
| I blame RTO and AI
| ironhaven wrote:
| This is why we need ipv6. If ipv6 was fully rolled out this
| livestream could have been an efficient multicast stream like
| what happens with ipTV.
| BrandoElFollito wrote:
| One similar crash I remember very well was CNN on 9/11 - I tired
| to connect from France but is down the whole day.
|
| Since then I am very used to it because our institutional web
| sites traditionally crash when there is a deadline (typically the
| taxes or school inscriptions).
|
| As for that one, my son is studying in Europe (I am also in
| Europe), he called me desperate at 5 am or so to check if he is
| the only one with the problem (I am the 24/7 family support for
| anything plugged in). After having liberally insulted Netflix he
| realized he confirmed with his grandparents that he will be
| helping them at 10 :)
| softwaredoug wrote:
| The way to deal with this is to constantly do live events, and
| actually build organizational muscle. Not these massive one off
| events in an area the tech team has no experience in.
| _dark_matter_ wrote:
| Agreed. This is a management failure, full stop. Unbelievable
| that they'd expect engineering to handle a single Livestream
| event of this magnitude.
| mbrumlow wrote:
| I have this argument a lot in tech.
|
| We should always be doing (the thing we want to do)
|
| Somme examples that always get me in trouble (or at least big
| heated conversations)
|
| 1. Always be building: It does not matter if code was not
| changed, or there has been no PRs or whatever, build it.
| Something in your org or infra has likely changed. My argument
| is "I would rather have a build failure on software that is
| already released, than software I need to release".
|
| 2. Always be releasing: As before it does not matter if nothing
| changed, push out a release. Stress the system and make it go
| through the motions. I can't tell you how many times I have
| seen things fail to deploy simply because they have not
| attempted to do so in some long period of time.
|
| There are more just don't have time to go into them. The point
| is if "you did it, and need to do it again ever in the future,
| then you need to continuously do it"
| parasti wrote:
| This is golden advice, honestly. "If you don't use it, you
| lose it" applied to software development.
| andai wrote:
| This is great, but what possible counterargument is there? We
| should prolong indefinitely a spooky ambiguity about whether
| the system works or not?
| mplewis wrote:
| The common and flawed counterargument is "when we deploy,
| outages happen." You'll hear this constantly at companies
| with bad habits.
| ukuina wrote:
| Finite compute, people, and opportunity cost.
|
| It is just a reframing of build vs maintain.
| macintux wrote:
| In some environments, deploying to production has a massive
| bureaucracy tax. Paperwork, approvals, limited windows in
| time, can't do them during normal business hours, etc.
| josho wrote:
| Those taxes were often imposed because of past
| engineering errors. For example, Don't deploy during
| business hours because a past deployment took down
| production for a day.
|
| A great engineering team will identify a tax they dislike
| and work to remove it. Using the same example, that means
| improving the success rate of deployments so you have the
| data (the success record) to take to leadership to change
| the policy and remove the tax.
| kortilla wrote:
| Deploying is expensive for some models. That could involve
| customer facing written release notes, etc. Sometimes the
| software has to be certified by a govt authority.
|
| Additionally, refactor circle jerks are terrible for back-
| porting subsequent bug fixes that need to be cherry picked
| to stable branches.
|
| A lot of of the world isn't CD and constant releases are
| super expensive.
| jerf wrote:
| Easy: Short term risk versus long term risk. If I deploy
| with minimal changes today, I'm taking a non-zero short-
| term risk for zero short-term gain.
|
| While I too am generally a long-term sort of engineer, it's
| important to understand that this is a valid argument on
| its own terms, so you don't try to counter it with just
| "piffle, that's stupid". It's not stupid. It can be
| shortsighted, it leads to a slippery slope where every day
| you make that decision it is harder to release next time,
| and there's a lot of corpses at the bottom of that slope,
| but it isn't _stupid_. Sometimes it is even correct, for
| instance, if the system 's getting deprecated away anyhow
| why take any risk?
|
| And there is some opportunity cost, too. No matter how
| slick the release, it isn't ever _free_. Even if it 's all
| 100% automated it's still going to barf sometimes and
| require attention that not making a new release would not
| have. You _could_ be doing something else with that time.
| rconti wrote:
| The counterargument is obvious for anyone who has been on
| call or otherwise responsible for system stability. It's
| very easy to become risk-averse in any realm.
| skybrian wrote:
| Doing dry runs regularly makes sense, but whether actually
| shipping it makes sense seems context-dependent. It depends
| on how much you can minimize the side effects of shipping a
| release.
|
| Consider publishing a new version of a library: you'd be
| bumping the version number all the time and invalidating
| caches, causing downstream rebuilds, for little reason. Or if
| clients are lazy about updating, any two clients would be
| unlikely to have the same version.
|
| Or consider the case when shipping results in a software
| update: millions of customer client boxes wasting bandwidth
| downloading new releases and restarting for no reason.
|
| Even for a web app, you are probably invalidating caches,
| resulting in slow page loads.
|
| With enough work, you could probably minimize these side
| effects, so that releasing a new version that doesn't
| actually change anything is a non-event. But if you don't
| invalidate the caches, you're not really doing a full
| rebuild.
|
| So it seems like there's a tension between doing more end-to-
| end testing and performance? Implementing a bunch of cache
| levels and then not using it seems counterproductive.
| lxgr wrote:
| It's very hard to do a representative dry run when the most
| likely potential points of failure are highly load-
| dependent.
|
| You can try and predict everything that'll happen in
| production, but if you have nothing to extrapolate from,
| e.g. because this is your very first large live event, the
| chances of getting that right are almost zero.
|
| And you can't easily import that knowledge either, because
| your system might have very different points of failure
| than the ones external experts might be used to.
| SketchySeaBeast wrote:
| True. And if you only fix the first element that failed
| at 10% capacity it won't help you catch the element that
| fails at 50%, which in turn hides the element that fails
| at 90%.
| unoti wrote:
| > 1. Always be building: It does not matter if code was not
| changed...
|
| > 2. Always be releasing...
|
| A good argument for this is security. Whatever
| libraries/dependencies you have, unpin the versions, and have
| good unit tests. Security vulnerabilities that are getting
| fixed upstream must be released. You cannot fix and remove
| those vulnerabilities unless you are doing regular releases.
| This in turn also implies having good unit tests, so you can
| do these builds and releases with a lower probability of
| releasing something broken. It also implies strong monitoring
| and metrics, so you can be the first to know when something
| breaks.
| MisterBastahrd wrote:
| The WWE is moving their programming to Netflix next year. If I
| were them, I'd be horrified at what I saw.
| geor9e wrote:
| They've been doing live events since 2023. But it's hard to be
| prepared for something that's never been done by anyone before
| -- a superbowl scale event, entirely viewed over the internet.
| The superbowl gets to offload to cable and over the air.
| Interestingly, I didn't have any problems with my stream. So it
| sounds like the bandwidth problems might be localized, perhaps
| by data center or ISP.
| burntalmonds wrote:
| Yeah, I think people are incorrectly assuming that everyone
| had the same experience with the stream. I watched the whole
| thing and only had a few instances of buffering and quality
| degradation. Not more than 30 seconds total during the
| stream.
| DharmaPolice wrote:
| Even if it was only 30% of people had a problem that's
| still millions of unhappy users. Not great for a time
| sensitive event.
|
| Also, from lurking in various threads on the topic
| Netflix's in app messages added to people's irritation by
| suggesting that they check their WiFi/internet was working.
| Presumably that's the default error message but perhaps
| that could have been adjusted in advance somehow.
| mastazi wrote:
| Maybe they considered this event as a rehearsal for the
| upcoming NFL streams which I am guessing might have a wider
| audience
| ignoramous wrote:
| > _...the tech team has no experience in_
|
| Unless Netflix eng decides to release a public postmorterm, we
| can only speculate. In my time organizing small-time live
| streams, we always had up to 3 parallel "backup" streams
| (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt
| they could simply summon any of these providers in, but I guess
| Akamai / Cloudflare would have been up for it.
| giantg2 wrote:
| Wow, building talent from within? I thought that went out of
| fashion. I think companies are too impatient to develop their
| employees.
| simple10 wrote:
| I watched the event last night and didn't get any buffering
| issues, but I did notice frequent drop in video quality when
| watching the live feed. If I backed the video up a bit, the video
| quality suddenly went back up to 4k.
|
| I had some technical experience with live video streaming over 15
| years ago. It was a nightmare back then. I guess live video is
| still difficult in 2024. But congrats to Jake Paul and boxing
| fans. It was a great event. And breaking the internet just adds
| more hype for the next one.
| lunatuna wrote:
| I wonder how localized the issues were. I watched the
| Taylor/Seranno fight and the Paul/Tyson without issue and the
| picture quality was the best in every seen for live sports. Was
| blown away by how good it was. No where near what I'm getting
| with steaming NFL. This is what I want the future of live
| sports to look like. Though the commentary was so so.
|
| I'm in the Pacific Northwest. I wonder if we got lucky on this
| or just some areas got unlucky.
| ctvo wrote:
| It's insane the excuses being made here for Netflix's apparently
| unique circumstances.
|
| They failed. Full stop. There is no valid technical reason they
| couldn't have had a smooth experience. There are numerous people
| with experience building these systems they could have hired and
| listened to. It isn't a novel problem.
|
| Here are the other companies that are peers that livestream just
| fine, ignoring traditional broadcasters:
|
| - Google (YouTube live), millions of concurrent viewers
|
| - Amazon (Thursday Night Football, Twitch), millions of
| concurrent viewers
|
| - Apple (MLS)
|
| NBC live streamed the Olympics in the US for tens of millions.
| csallen wrote:
| _> They failed. Full stop._
|
| It's not full stop. There are reasons _why_ they failed, and
| for many it 's useful and entertaining to dissect them. This is
| not "making excuses" and does not get in the way of you,
| apparently, prioritizing making a moral judgment.
| y-c-o-m-b wrote:
| Amazon had their fair share of livestream failures and for
| notably less viewers. I don't think they deserve a spot on that
| list. I briefly worked in streaming media for sports and while
| it's not a novel problem, there are so many moving parts and
| points of failure that it can easily all go badly.
| deanCommie wrote:
| There is no one "Amazon" here, there are at least 3:
|
| * Twitch: Essentially invented live streaming. Fantastic.
|
| * Amazon Interactive Video Service [0]: Essentially "Twitch
| As A Service", built by Twitch engineers. Fantastic.
|
| * Prime Video. Same exact situation as Netflix: original
| expertise is all in static content. Lots of growing pains
| with live video and poor reports. But they've figured it out:
| now there are regular live streams (NHL and NFL), and other
| channel providers do live streaming on Prime Video as a
| distribution platform.
|
| [0] https://aws.amazon.com/ivs/
| margaretdouglas wrote:
| Doesn't twitch almost fall over (other non-massive streams
| impacted) when anyone gets close to 4-5m concurrent
| viewers? I remember last time it happened everything
| started falling over, even for smaller streams. Even if
| Netflix struggled with the event, streaming other content
| worked just fine for me.
| ukuina wrote:
| IVS does not scale past 1080p: https://ivs.rocks/
| fredgrott wrote:
| it could be that they made use of the same advice X followed :)
| mikeryan wrote:
| The big difference of all the examples you've mentioned is
| dedicated full-time crews on the ground where the events are
| produced.
|
| I'm pretty confident that when the post mortem is done the
| issues are going to be way closer to the broadcast truck than
| the user.
| mfiguiere wrote:
| The examples given here are not on the same scale. The numbers
| known so far:
|
| - 120m viewers [1]
|
| - Entire Netflix CDN Traffic grew 4x when the live stream
| started [2]
|
| [1] https://www.rollingstone.com/culture/culture-news/jake-
| paul-...
|
| [2] https://x.com/DougMadory/status/1857634875257294866
| prasadjoglekar wrote:
| Disney HotStar managed to stream ~60M livestreams for the
| Cricket world cup a year ago. The problem has been solved.
| Livestreaming sports just have a different QoS expectations
| than on demand.
|
| https://www.geeksforgeeks.org/how-did-hotstar-
| managed-5-9-cr...
| achow wrote:
| Size of Hotstar team = ~2000. Enggrs will be less than that
| number.
|
| https://www.linkedin.com/company/disney-hotstar/people/
| margaretdouglas wrote:
| I wouldn't say it's a solved problem, how many other
| companies are pulling off those numbers? Isn't that the
| current record for concurrent streams? And wasn't it mostly
| to mobile devices?
| silisili wrote:
| I guess one question I have is did Netflix partner with other
| CDNs?
|
| Despite their already huge presence, Amazon for example has
| multiple CDNs involved for capacity for live events. Same for
| Peacock.
| Thaxll wrote:
| Netflix has 280m subscriber highly doubt half of them tuned
| in to watch the match, is that 130m figure official?
| toast0 wrote:
| I don't disagree that Netflix could have / should have done
| better. But everybody screws these things up. Even broadcast TV
| screws these things up.
|
| Live events are difficult.
|
| I'll also add on, that the other things you've listed are
| generally multiple simultaneous events; when 100M people are
| watching the same thing at the same time, they all need a lot
| more bitrate at the same time when there's a smoke effect as
| Tyson is walking into the ring; so it gets mushy for everyone.
| IMHO, someone on the event production staff should have an eye
| for what effects won't compress well and try to steer away from
| those, but that might not be realistic.
|
| I did get an audio dropout at that point that didn't self
| correct, which is definitely a should have done better.
|
| I also had a couple of frames of block color content here and
| there in the penultimate bout. I've seen this kind of stuff on
| lots of hockey broadcasts (streams or ota), and I wish it
| wouldn't happen... I didn't notice anything like that in the
| main event though.
|
| Experience would likely be worse if there were significant
| bandwidth constraints between Netflix and your player, of
| course. I'd love to see a report from Netflix about what they
| noticed / what they did to try to avoid those, but there's a
| lot outside Netflix's control there.
| freefaler wrote:
| As a cofounder of a CDN company that pushed a lot of traffic,
| the problem with live streaming is that you need to propagate
| peak viewership trough a loooot of different providers. The
| peering/connectivity deals are usually not structured for peak
| capacity that is many times over the normal 95th percentile.
| You can provision more connectivity, but you don't know how
| many will want to see the event. Also, live events can be
| trickier than stored files, because you can't offload to the
| edges beforehand to warm up the caches.
|
| So Netflix had 2 factors outside of their control
|
| - unknown viewership
|
| - unknown peak capacities outside their own networks
|
| Both are solvable, but if you serve "saved" content you
| optimize for different use case than live streaming.
| JSDevOps wrote:
| Why didn't they use Netflix AI to solve the problems?
| frankharv wrote:
| How dare you insult the AI Gods
| aucisson_masque wrote:
| Honestly you didn't miss much, every (real) boxing fan thought of
| this as a disgrace and a shame when announced. putting a 58 year
| old Tyson against a crackhead filled with steroids (Jake Paul) ?
| Either case it would have been a shame on Jake Paul for even
| getting in the ring with such an old boxer.
|
| In boxing you are old by 32 or maybe 35 year old for heavy
| weight, and everything goes down very very fast.
|
| End of rant.
| chx wrote:
| People still pay real world money to Netflix after they cancelled
| and how and why Warrior Nun just to see grandpa being beaten up.
|
| I guess in the year when Trump is being reelected this is hardly
| a surprise.
| bloomingeek wrote:
| Serves Netflix right for killing my beloved DVD rentals.
| duxup wrote:
| I'm a little amused at folks tuning in for meme / low quality
| personalities doing things ... and getting the equivalent
| production values.
| eigenvalue wrote:
| They should have partnered with every major CDN and load balanced
| across all of them. It's ironic how we used to be better at
| broadcasting live events way back in the day versus today.
| badgersnake wrote:
| I'm not sure buffering was the biggest issue with this event. How
| was as 58 year old Tyson fighting a man in his 20s?
| xyst wrote:
| This wasn't a "real" fight in the ring. It was clearly a
| hype/money fight only. The late 20 year old boxer has a massive
| following (or hate following) with younger age groups; and Mike
| Tyson brings the older age groups out. Mike has earned somewhat
| of a legendary status.
|
| Leading up to the fight, there were many staged interactions
| meant to rile up the audience and generate hype and drive
| subscription revenue, and subsequently make ad spots a premium
| ($$$).
|
| Unfortunately, American television/entertainment is so fake. I
| can't get even be bothered to engage or invest time into it
| anymore.
| phatfish wrote:
| It may as well have been a WWE special. As good as scripted. It
| was a business venture not a competitive fight.
| xyst wrote:
| What a massive blow to NFLX. They have been in the streaming game
| for years (survived COVID-19) and this silly exhibition match is
| what does them in?
|
| I didn't watch it live (boxing has lot its allure for me) but
| vicariously lived through it via social feed on Bluesky/Mastadon.
|
| Billions of dollars at their disposal and they just can't get it
| right. Probably laid off the highly paid engineers and teams that
| made their shit work.
| wannacboatmovie wrote:
| > Probably laid off the highly paid engineers and teams that
| made their shit work.
|
| More likely overpaying a bunch of posers without the chops, a
| victim of their own arrogance.
| thesausageking wrote:
| It's not a "massive blow" at all. Consumers will only vaguely
| remember this in a month. Netflix got a lot of new signups and
| got to test out their streaming infrastructure to figure out
| what needs work.
|
| The fight itself was lame which worked in their favor. No one
| really cared about not being able to see every second of the
| "action". It's not like it was an NBA game that came down to
| the last second.
| Thaxll wrote:
| In 2012 Youtube did the Red Bull stratos live stream with 8m
| concurrent users. We're 12 years later, Netflix fucked up.
| MichaelZuo wrote:
| The average quality of talent has gone way down compared to
| 2012 though.
|
| E.g. the median engineer, excluding entry level/interns, at
| YouTube in 2012 was a literal genius at their niche or quite
| close to it.
|
| Netflix simply can't hire literal geniuses with mid six figure
| compensation packages in 2024 dollars anymore... though that
| may change with a more severe contraction.
| spaceywilly wrote:
| To me the difference is that in 2012, you had companies
| focusing on delivering a quality product, whether it made money
| or not. Today, the economic environment has shifted a lot and
| companies are trying to increase profits while cutting costs.
| The result is inevitably a decline in quality. I'm sure that
| Netflix could deliver a flawless live stream to millions of
| viewers, but the question is can they do it while making a
| profit that Wall Street is happy with. Apparently not.
| Mistletoe wrote:
| The funny thing is I was just reading something on HN like
| three days ago about how light years ahead Netflix tech was
| compared to other streaming providers. This is the first thing
| I thought of when I saw the reports that the fight was messing
| up.
| lxgr wrote:
| But is there a way that Netflix might have learned from all of
| Youtube's past mistakes?
|
| The only reasonable way to scale something like this up is
| probably to... scale it up.
|
| Sure, there are probably some generic lessons, but I bet that
| the pain points in Netflix's architecture (historically grown
| over more than a decade and optimized towards highly cacheable
| content) are very different from Youtube, which has ramped up
| live content gradually over as many years.
| throwawayUS9 wrote:
| Why no one mentioned the term "vaporware"? Isn't this a classic
| example of one?
| sourcecodeplz wrote:
| This livestream broke the internet, no joke. youtube was barely
| loading and a bunch of other sites too. 130M is a conservative
| number given all the pirate streams.
| SG- wrote:
| don't confuse your ISP breaking with every other provider or
| the rest of the Internet. It was more than fine here.
| m3kw9 wrote:
| I would have just made it simple, delay the live stream a few
| seconds and encode it into the same bucket where users already is
| playing static movies. Just have the player only allow start at
| the time everyone is at.
| vouwfietsman wrote:
| Reading the comments here, I think one thing that's overlooked is
| that Netflix, which has been on the vanguard of web-tech and has
| solved many complicated problems in-house, may not have had the
| culture to internally admit that they needed outside help to
| tackle this problem.
|
| A combination of hubris and groupthink.
| kaptainscarlet wrote:
| Not invented here syndrome works at first but as time
| progresses the internally built tools become a liability
| nikolay wrote:
| The arrogant Netflix! They always brag about how technologically
| superior they are, and they can't handle a simple technological
| challenge! I didn't have a buffering issue, I had an error page -
| for hours! Yet, they kept advertising the boxing match to me!
| What a joke! If you can't stream it, don't advertise it to save
| face with people like me who don't care about boxing!
| notimetorelax wrote:
| I think you're oversimplifying it. Live event streaming is very
| different from movie streaming. All those edge cache servers
| become kinda useless and you start hitting peering bottlenecks.
| YZF wrote:
| Edge caches are not useless for live streaming. They're
| critical. The upstream from those caches has no way of
| handling each individual users. The stream needs to hit the
| edge cache and end users should be served from there.
|
| A typical streaming architecture is multi-tiered caches,
| source->midtier->edge.
|
| We don't know what happened but it's possible they ran out of
| capacity on their edge (or anywhere else).
| StressedDev wrote:
| Every organization makes mistakes and every organization has
| outages. Netflix is not different. Instead, of bashing them
| because they are imperfect, you might want to ask what you can
| learn from this incident. What would you do if your service
| received more traffic than expected? How would you test your
| service so you can be confident it will stay up?
|
| Also, I have never seen any Netflix employees who are arrogant
| or who think they are superior to other people. What I have
| seen is Netflix's engineering organization frequently describes
| the technical challenges they face and discusses how they solve
| them.
| gunapologist99 wrote:
| Nucleix needs to focus on fixing middle-out compression instead
| of kicking cameras.
| RyeCombinator wrote:
| Chaos testing, nothing to see here.
| pchwalek wrote:
| Sounds like a job for Pied Piper
| manav wrote:
| It seemed to be some capacity issue with the CDNs. When I stopped
| and restarted the stream it worked again. Perhaps they do not use
| real time multi-cdn switching.
| gardenhedge wrote:
| So much for Netflix engineering talent aura
| sporkland wrote:
| Was this the plot of a silicon valley episode?
| abc-1 wrote:
| I thought Netflix engineers were the best and could even do
| mythical leetcode hards. What happened? Why are they paid half a
| million dollars a year?
___________________________________________________________________
(page generated 2024-11-16 23:00 UTC)