[HN Gopher] Netflix never used its $1M algorithm (2012)
___________________________________________________________________
Netflix never used its $1M algorithm (2012)
Author : reqo
Score : 144 points
Date : 2024-01-10 20:48 UTC (2 hours ago)
(HTM) web link (thenextweb.com)
(TXT) w3m dump (thenextweb.com)
| teruakohatu wrote:
| They got far more than $1M in marketing from it. It would have
| been worth it for twice as much.
| taeric wrote:
| I'm curious how you quantify that?
| kamikaz1k wrote:
| at least for the internal goal of hiring top ML talent, you
| can probably just calculate how long it took to hiring the
| good ones, and then how much money that team made.
| taeric wrote:
| Are they objectively better at ML for movie recommendations
| than they were beforehand? I confess I've never been too
| impacted by their recommendations. Outside of new releases,
| but that doesn't really use any ML.
| kamkazemoose wrote:
| You can think, how much would their marketing team have to
| spend to get the same results that the algorithm contract
| gave. I'm not a marketing expert, but I'm sure they have
| metrics like consumer sentiment, name recognition, number of
| users visiting the site, google search trends, etc. There
| could also be benefits in recruitment, and that can be
| estimated based on how much you'd have to pay an external
| recruiter to bring in candidates the applied, or other things
| like that.
|
| It was in the news a lot, and was discussed on a lot of tech
| sites. Plus it gets people talking about their recomendation
| algorithm, and makes people thing Netflix subscription is
| more valuable becasue it recommends good shows. It wouldn't
| be cheap to get the amount of media that they got through
| more traditional marketing.
| taeric wrote:
| Right, but that is a lot of what I'm asking. What makes you
| think those are all better due to this contest? If I
| recall, a lot of why it was making the news was because
| Netflix was already popular in the industry. They certainly
| weren't that new of a name.
|
| The main one I don't think I would have doubts about is the
| recruitment. But, I don't recall them being a place that
| needed recruitment help, even at that time.
|
| To be clear, I found the thing fun to consider. I certainly
| am not upset that they did it. I do harbor a gut feeling
| that its ROI is greatly overstated.
| gen220 wrote:
| I'm not sure you need to.
|
| Not picking on you at all, just going on a tangent because
| I've been thinking recently about the good ideas that end up
| on the cutting floor because their impact, while positive, is
| intractable or completely impractical to quantify.
|
| Some ideas (like this Netflix one!) are "obvious" winners,
| because they have a diffused, positive impact in many
| important dimensions - each of which is almost impossible to
| measure, but the integral of which is almost _certainly_
| greater than the idea 's cost, probably by one or two OOM. If
| there's high conviction in an idea being a 10x idea, and
| attempting it costs quite little, it's better to do it now
| and maybe consider measuring it later. Who cares if it's 9x
| or 12x ROI, the point is it has a big margin of safety and
| large expected returns on capital.
|
| But in a "we can't greenlight this project unless we can
| directly attribute it to positive motion in our KPI" org,
| these flavors of idea are dead on arrival. The double-kicker
| is that the cost of trying to measure these ideas is often
| greater than the cost of the idea.
|
| For some ideas, especially when it's a rounding error on the
| company's annual budget, it should be OK if we don't invest
| much effort into quantifying the results. It's one of those
| rare set of ideas where it's anathema to the
| professional/investor culture of SV of the last decade, while
| simultaneously being how plucky Seed/Series A companies can
| punch above their weight.
| taeric wrote:
| I would push back on this specific case, though. What makes
| this an obvious winner? Netflix was already clearly on the
| upward swing. At the time of this contest, I'm not even
| sure I remember who their competition was.
|
| Now, I think it is fully fair to say that this was very
| cheap for them to do. In which case, why not do it? But I
| think you would be hard pressed to give me a counter
| factual world that is believable where not doing this had a
| meaningful impact on Netflix's future.
| travisjungroth wrote:
| Hiring. They got a lot of attention for this.
| taeric wrote:
| As I said in another reply, I don't remember them having
| trouble hiring in the first place?
|
| I can certainly mostly agree with this. I just also think
| them producing their own shows was likely more impactful,
| and had basically nothing to do with this.
|
| Don't get me wrong, plenty of moves in life are "win
| more." Such that I could see this one fitting that
| description. But a lot of "win more" choices are
| surprisingly low on the ROI side, when you control for
| everything else in that situation.
| travisjungroth wrote:
| Coca-Cola doesn't have trouble selling soda, but they
| still have commercials.
|
| Of course making their own shows was more impactful in a
| bunch of ways. But, the investment is also so much more.
| A season of a show is in the tens to hundreds of
| millions.
| sosborn wrote:
| Their competition was the old outdated business model. As
| easy as it was to say that streaming was the future, they
| still had to convince millions of people to learn about
| and sign up for it.
| chmod775 wrote:
| Yup. $1m is nothing. We live in a world where movies have
| $100m marketing budgets, game companies invest hundreds of
| millions into esports events and other material, and tiny
| companies are bought out "just in case" for similar
| amounts.
|
| $1m to generate some publicity and attract talent? Sure.
| They're probably paying headhunters similar amounts in
| comissions each year.
| gathersnow wrote:
| Netflix became so user-hostile and it is truly baffling to me.
| They act as if they are paid by amount of time you spend watching
| movies and have all of these dark patterns shoving the user all
| over the place and disorienting them. I remember when they had
| tools to see what your friends were watching and to discover
| hidden gems which was always such a fun experience.
|
| Does Netflix have such content nowadays? I really wouldn't know.
| They optimize the experience for binge-watching and nothing else.
| I wish so badly that they'd do a letterboxd thing and allow for
| curation of their catalog but it all goes back to the ephemeral
| nature of their content and how they have to hide that fact.
|
| They care so little about that approach now it's no wonder they
| never used this algorithm.
| rocky1138 wrote:
| I agree. Netflix died the day they switched to thumbs-up and
| thumbs-down versus 5-star rating content. My recommended list
| was so solid for so long and all of that went down the drain.
| gathersnow wrote:
| Can you even vote now? I think they do a lot of it by
| inferring from what you watch.
| iwontberude wrote:
| Their death was presaged when their executives decided to
| "become HBO before HBO can become us" and when they spun out
| Roku for lack of appetite to be a proper tech company. Reed
| Hastings wasn't capable of running that business and so their
| vision shrank to match.
|
| Netflix sold a lot of people a lie, their customers, their
| employees and their investors. We jumped on board thinking
| Netflix was serious about the living room experience in a
| wholistic way.
| anthonypasq wrote:
| how was netflix ever going to continue being a "tech
| company." Video streaming is a commodity at this point,
| they are not longer doing anything technologically
| revolutionary.
|
| Once every other media company could do what they were
| doing they realized they have to become an entertainment
| company. It was shrewd foresight on their end, otherwise
| they wouldnt exist anymore. Unless they tried to license
| out their infrastructure or something
| iwontberude wrote:
| You can say streaming is commodity and uninteresting now
| but back in 2012 when these decisions were being made,
| they had the ability to pivot into making proprietary,
| higher-end TVs and set top boxes that improved streaming
| significantly to where it is today or further. I
| understand they didn't want to become TiVO, but this is
| where Netflix had the ability to change things by
| publishing exclusive content directly to their customers
| without relying on cable/satellite providers. I can't say
| it would be easy -- and that is what I mean about Reed
| Hastings not being capable.
|
| Strategically, Netflix is beholden to its manufacturing
| partners and never had a wholistic story for the living
| room the way they claimed to want because they never had
| control. They never had control because they didn't come
| up with a compelling prototype. This would necessarily
| include gaming and other content.
| seabass-labrax wrote:
| Netflix's technology is incredibly advanced. The term
| 'revolutionary' is somewhat subjective, but what isn't
| subjective are achievements like streaming video at
| 400GB/s with a single server[1], no, make that
| 800GB/s[2], and writing a FUSE-based filesystem that runs
| at gigabit speeds despite storing data on AWS[3].
|
| [1]: https://papers.freebsd.org/2021/eurobsdcon/gallatin-
| netflix-...
|
| [2]: https://papers.freebsd.org/2022/eurobsdcon/gallatin-
| the_othe...
|
| [3]: https://netflixtechblog.com/mezzfs-mounting-object-
| storage-i...
| CoastalCoder wrote:
| IIRC, they pushed pretty hard on the idea that the thumbs
| up/down system was objectively better.
|
| I was never convinced. Partly because I wanted to see the
| actual number of stars when choosing a video, and partly
| because their motives at the time were highly suspect (one of
| their own produced videos was getting bombed with 1-star
| reviews, IIRC).
| resters wrote:
| The proof was(n't) in the pudding. Movielens has given me
| superb recommendations and led me to watch content that
| became some of my all-time favorite content. Netflix has
| never offered a useful recommendation at all.
| rabuse wrote:
| It would actually be slightly useful, if you could actually
| see the thumbs up/down ratio, but they pulled a Youtube.
| ldjkfkdsjnv wrote:
| "Its too crowded, no one goes there anymore"
| pi-e-sigma wrote:
| Could still be true, though if nobody _you care about_ goes
| there
| wharvle wrote:
| What's wild is that it's not like they're matching industry
| norms. Most of the other big streaming platforms are far less
| chaotic and confusing. Even the shitty new "Max" interface is
| better than Netflix. It's the only one I _will not_ use to just
| browse, I only open it with a purpose because the UI is so
| unpleasant.
| heleninboodler wrote:
| > Even the shitty new "Max" interface is better than Netflix
|
| I have to disagree. I don't love the Netflix interface, but
| Max is truly terrible. It is egregiously slow and it's
| difficult to locate the things that I want. And when the
| thing I want happens to be highlighted at the top of the
| homepage, it's always a dumb "just play more of this" instead
| of a link to the show info page, which almost always plays
| the end credits of the last episode I watched, and it takes
| 60 seconds to load that, then another 60 seconds to start the
| correct episode when I find it. If I get a take-out burrito
| and sit down to watch TV and eat it, I always think to myself
| "I wonder if I can actually find and start a show before I
| finish my entire burrito," which is an exaggeration, but not
| by a lot. All the accumulated UI "wait for something to load"
| moments add up to minutes. Netflix, on the other hand, is
| almost always more or less instantaneous (2-3 seconds maximum
| per UI navigation, 5ish seconds to start streaming).
| ryankrage77 wrote:
| One thing I dislike about Netflix is that there's a category
| for 'top ten in [your region]'. This doesn't really show the
| aggregate top ten pieces of media for a region, it is actually
| personalised to you. If I compare across accounts, the top ten
| are are different, sometimes overlapping but usually in a
| different order. This dark pattern exploits the 'join the
| crowd' bias
| (https://en.wikipedia.org/wiki/Argumentum_ad_populum).
|
| Also, constant price increases and laying off large numbers of
| employees...
| resters wrote:
| True, that is anti-personalization
| mvdtnz wrote:
| A simple "don't show me this thing ever again" button would
| make Netflix 100x more usable, but it would quickly demonstrate
| just how small their catalog is.
| bilsbie wrote:
| I wonder if it paid off for recruiting though?
| avelis wrote:
| I remember seeing the leaderboard rankings before the prize was
| awarded. If I recall correctly the top two teams were very close
| to the coveted goal. They ended merging to become one team and
| spit the prize once they achieved the goal.
| n2d4 wrote:
| Those contests are never about the actual code/algorithms they
| produce, it's always about recruiting the people who write them.
|
| For the price of $1mil (plus a bit of engineering time) they got
| a list of talented engineers who have too much time, and also
| everyone now knows that they are the company with the really
| challenging problems. Forward the list to recruiting, and you got
| a bunch of new hires, much more effective than paying for job
| ads.
| ffhhj wrote:
| > It's now extended beyond the US and in to Canada, 43 Latin-
| American countries
|
| There are about 21 LatAm countries.
| bryanrasmussen wrote:
| probably there's an advanced algorithm at work here that says
| otherwise.
| mushufasa wrote:
| in Reed's "No Rules Rules" book, they discuss how this contest
| was really a way to recruit top-quality engineering talent, which
| is a key assumption in how they ran their culture (highly paid
| small teams with lots of freedom, trusted to know what's best
| from the ground-up rather than top-down).
|
| They didn't really need an algorithm in the first place, they
| just brainstormed what would sound the coolest to developers.
| shermantanktop wrote:
| Everything that Netflix says or does externally wrt to tech
| appears to be for this purpose, including open-sourcing some
| things. It is all catnip for the HN crowd to come work for
| them.
| mrguyorama wrote:
| Netflix pays in the same ballpark as other FAANG companies
| but the product is so much clearer (stream people movies and
| shows, maybe recommend them something to watch) and the
| complete lack of grey ethics like in AdTech or a Monopoly
| like Microsoft is the real catnip.
| cybrox wrote:
| They use a lot of the same concepts as AdTech but only for
| their own content without shilling for everyone who is
| willing to pay.
| lulznews wrote:
| Which is odd given their app is highly trivial.
| rabuse wrote:
| All the top talent they've acquired and money spent still can't
| save them from their complete garbage lineup of content they
| push out, all while limiting account access and raising prices.
| xeromal wrote:
| People can call it garbage all they want. Their raised prices
| resulting in more revenue for them
| Uptrenda wrote:
| I kind of agree with you. I feel like there's a narrow set of
| content there worth watching and when you get through it all
| it's just like: okay, now what? There's nothing else worth
| watching. I haven't been able to use my netflix account for
| months now. I just go to youtube now and subscribe to
| channels I like. Seems much more interesting than netflix.
|
| Currently following some interesting permaculture projects. I
| find the type of work these people do so useful and clever
| really.
| cqqxo4zV46cp wrote:
| Netflix obviously since enjoyed a period of success. You may
| need to reconcile that with your overly negative portrayal of
| the situation which honestly seems more about your personal
| feelings about Netflix rather than more objective measures of
| success.
| kredd wrote:
| You call it garbage content, but every other person is
| somehow watching them. I'm probably not their target
| audience, but looking at my friend circles, those shows are
| definitely being consumed.
| caballeto wrote:
| Neflix revenue went from 4.3b in 2013 to 31b in 2022.
| Astounding success by any measure. That means they are doing
| something right, including their talent strategy. "garbage"
| is subjective. Neflix employees are likely paid bonuses on
| views / subscriptions generated, which from their revenues
| looks like they know how to hit.
| ben_jones wrote:
| At the same time it became a running joke about Netflix
| greenlighting everything as captioned in pop culture like a
| Rick and Morty episode. The scandal of them throwing money
| away to a novice producer didn't help either.
|
| Large revenues that are a function of macroeconomics does
| not indicate merit - a crushing statement to tech execs the
| world over.
| conradev wrote:
| Top engineering talent likely doesn't make content decisions
| is_true wrote:
| At least in my country the have the best catalog for my
| family.
| lr1970 wrote:
| A little secret is that the content rights holders negotiated
| draconian streaming contracts that are outright prohibitive to
| Netflix. Netflix is trying to do everything possible to hide
| the fact that their streaming library is small and shrinking.
| In the old hey-day of Netflix mail-in red envelopes with DVDs
| you were hard pressed to find a movie they did not have. These
| days your search returns mostly crap.
| Wowfunhappy wrote:
| This is why I'm still kind of surprised they separated out
| (and eventually killed) their disc-by-mail service. I
| understand the visceral feeling that discs are an antiquated
| relic--but you just can't beat that catalog.
|
| Combining discs and streaming in one subscription provides
| customers with the best of both worlds. Need something to
| watch _right now?_ No problem, we have an excellent streaming
| catalog and you 're going to _love_ Stranger Things! Want a
| _specific_ movie or show, maybe Friends or The Lion King? We
| 've got that too, it'll be in your mailbox in a couple days.
|
| And if some customers choose to forgo either the digital or
| physical side of the package, it doesn't cost Netflix
| anything.
| munificent wrote:
| I'm sure they just ran the numbers and realized that being
| both a data streaming software company _and_ a physical
| package logistics company just didn 't add up. Every dollar
| they spent on the latter would have net them more profit if
| they instead invested it in the former.
| Wowfunhappy wrote:
| Maybe (I of course don't have their numbers), but
| remember that Netflix kept offerring discs up until last
| year, just under a separate subscription. So they were
| still investing some money.
|
| I think Netflix got cocky. When Netflix separated the
| plans, streaming rights were cheap and Netflix's
| streaming catalog was growing quickly. They didn't
| foresee the market becoming so competitive.
|
| Continuing to offer discs would have allowed Netflix to
| hedge their bets and avoid reliance on rights-holders.
| The original content push did that too, but at a much
| higher cost. If keeping DVDs meant Netflix could produce
| a few less original shows per-year, that seems worth the
| trade.
| closewith wrote:
| I'd say the majority of their customers wouldn't have a way
| to play the discs if sent.
| Pxtl wrote:
| Right? I have a half-dozen old devices around my house
| that could play a DVD, but I've never owned anything that
| can play bluRay, much less 4K disks (I've never even
| _seen_ one of those IRL). And DVDs are no longer worth
| bothering with mostly because of the aspect ratio hassle
| if nothing else.
| piperswe wrote:
| Many people own an Xbox One S/X, Xbox Series X, or PS5,
| which can all play 4K Blu-ray. Many more people own a
| PS3, PS4, or regular Xbox One, which can all play regular
| Blu-ray.
| Wowfunhappy wrote:
| I agree, but largely because Netflix stopped renting
| them.
| pixl97 wrote:
| I disagree, because I stopped watching disks long before
| I lacked devices to play them.
| munificent wrote:
| _> A little secret is that the content rights holders
| negotiated draconian streaming contracts that are outright
| prohibitive to Netflix. Netflix is trying to do everything
| possible to hide the fact that their streaming library is
| small and shrinking._
|
| I think that's fairly well-known at this point, though I
| agree it should be more widely known.
|
| I think the real secret is that Netflix discovered that their
| users aren't trying to watch the best movies, they're just
| trying to kill time. Apparently you can substitute good films
| for generic junk and people will keep their subscriptions.
|
| Really, the business model for Netflix has some weird
| perverse incentives. They make money from monthly
| subscriptions. They lose money when actually streaming
| content to users. So the ideal user from their standpoint is
| one who keeps paying but never watches anything.
|
| Thus, the company is incentivized to put out _just_ enough
| good stuff that you can 't miss that you don't cancel, but
| otherwise produce garbage you don't want.
| LegitShady wrote:
| My favorite part about this is that a lot of the rights
| holders are losing their shirts on their alternative
| streaming services, because it turns out making content is
| different than running a profitable streaming service.
| Peacock and Disney+ are losing huge money, Amazon is
| introducing commercials or an upcharge to their service
| because they've been spending on their own content with
| abandon (I can't imagine why WoT costs per episode what I
| read it costs), and I don't know how appletv+ is doing in
| 2023 but in 2022 and 2021 they were also losing huge sums of
| money. Is netflix the only major streaming service that makes
| money at this point?
|
| Everyone decided "I can cut out the middleman and keep the
| money in my pocket" and they all lost hugely on that
| strategy.
| slt2021 wrote:
| This prize/competition has become a meme in itself and inspired
| hordes and hordes of people to dabble with recommendation
| systems.
|
| The advance of recsys is in big part thanks for Netflix because
| it attracted a lot of people to the field, all companies started
| developing inhouse RecSys just like today you hear about inhouse
| LLM/chatgpt wrappers etc
| minimaxir wrote:
| The Netflix Prize competition (2006, completed in 2009) was a
| Kaggle competition before Kaggle competitions (2010), with the
| same business-side incentives.
|
| Given the meteoric rise of ML/AI in the past few years, I'm
| surprised that Kaggle doesn't come up more often. It was all the
| rage 2013-2018...then most I heard about it was that it allowed
| free access to TPUs.
| htrp wrote:
| I feel like kaggle is a copycat wasteland for most projects
| now. You don't quite get top talent, rather you get a bunch of
| people xgboosting their way up your leaderboard.
| erhserhdfd wrote:
| "...rather you get a bunch of people xgboosting their way up
| your leaderboard" -->
|
| I don't see that as a universally bad thing. I feel this is
| actually very representative of the majority of ML projects
| that most organizations encounter. Most organizations don't
| have petabytes of data and a huge compute budget to train a
| DNN. They typically have megabytes to gigabytes of somewhat
| crappy data and need something that can be developed and
| deployed relatively quickly for low cost.
| AeroNotix wrote:
| Most companies spend a ridiculous amount of money for not much at
| all. The crazy amount of waste in boom times is, frankly,
| disgusting.
|
| You all have witnessed it, some of you may have even tried to
| stem the tide of horrific spending, some of you are to blame.
|
| Go to the billing page of whatever third-party thing you use and
| just recoil in horror how much you spend.
| resters wrote:
| Not surprising. I'd been using movielens for recommendations
| prior to the prize, and even today Netflix still does not provide
| high quality content recommendations.
|
| Worse yet it seems to think I want to watch the three things that
| are most popular on the platform.
|
| This indicates that the Netflix UX business is being
| significantly mismanaged.
| mvdtnz wrote:
| I suspect the Netflix recommendations system would be excellent
| if they had a catalog to back it. But the Netflix catalog is of
| such poor quality that even a perfect recommendation system
| ends up recommending garbage because it's all they have.
| zoover2020 wrote:
| Isn't their algorithm also very intendedly only catering to
| Netflix originals by default?
|
| I find its algo absolutely rubbish and most of my initial
| excitement to go watch something with my wife on the couch
| instantly fades as soon as we face the doomscrolling of
| finding content to watch.
|
| It's like zapping but worse
| onli wrote:
| For me it's the missing quality markers. Netflix provides
| no way at all to see whether a show or movie is good. So
| when one starts scrolling and does not find immediately
| something one likes, there is no reason to ever find
| anything good - because nothing seems good, and all alike.
|
| Since I added a userscript to see IMDB ratings directly
| next to the netflix shows it's way easier to find something
| promising.
| cortesoft wrote:
| I always hear people say this about the Netflix catalog, but
| I never understand what metric or criteria people are using.
| I find there is a ton of great content on Netflix, and I have
| way more things I want to watch on there than time to watch
| it.
|
| I always wonder if people who say this either have very
| particular interests, or are trying to find specific tv
| shows/movies and not finding them on Netflix. If you are not
| set on a particular thing, it seems to me that there is a ton
| of great content.
| londons_explore wrote:
| I think many people want to watch what their friends are
| watching (so they can talk about it at work/school
| tomorrow).
|
| That means if their friends are watching stuff on Amazon
| Prime, then netflix can never meet their needs.
| wharvle wrote:
| I have really broad interests in film--it blows my mind
| when people complain that not enough good or original films
| come out in a given year, I can't even make it through half
| the probably-good ones each year, I don't know what they're
| looking at or what their tastes must be like to come to
| that conclusion--but definitely find Netflix's catalog to
| be notably weak compared to Hulu or (HBO)Max, and it's been
| that way for years. I'd class it with second-tier services
| like Paramount, but from what I've seen on there lately,
| even that may be better.
|
| It's got Apple TV beat, I guess, but I wouldn't call that a
| major streaming service.
|
| [EDIT] Oh and I'd say it's overall worse than D+, for that
| matter, even though that's basically an umbrella for a few
| niche services. Like if I could only have one of those two,
| I'd take D+. And D+ kinda sucks.
| chuckadams wrote:
| The way Max is going, it's going to make Netflix look
| like the Criterion Collection in a few years.
| wharvle wrote:
| Oh, I know. Matter of time before I drop that one.
| They've inexplicably decided to destroy their own
| streaming service, on purpose.
| hedora wrote:
| That's especially true since I've noticed stuff max
| yanked has popped up on netflix.
|
| If netflix wants to lean in even further, they should
| pick up rights to produce additional seasons of most of
| the stuff Warner Brothers Discovery cancelled last year
| (they're the parent company of HBO and have subpar
| management, even by Hollywood exec standards).
| echelon wrote:
| > it blows my mind when people complain that not enough
| good or original films come out in a given year
|
| I have a weird take on this.
|
| Before I started working on my startup, I had a lot of
| time to watch and even make movies. I typically would
| watch about ~50-100 films a year. When I was younger, it
| was even more. I was voracious.
|
| I tend to like indie and art house films, but I'll also
| enjoy popular fare. I'm not a snob. As long as the
| choices of the writers, directors, actors, and editors
| make sense, I'm typically able to assign a rating and
| slot the film into some kind of tier list.
|
| _But I 'm always left unsated by the decision envelope._
|
| The film will make its choices, but I always find myself
| wanting to go in other directions. Sometimes wildly
| different ones. I feel more like an active participant,
| critic, or perhaps even script doctor or producer.
|
| When the story beats of a film change, my imagination
| keeps predicting outcomes along various trajectories.
| Sometimes I'm a perfect predictor. In these instances,
| films tend to be full of lazy tropes and are completely
| unsatisfying. Other times, my predictor is way off. If
| the dissonance is huge, sometimes I'm also left upset by
| the departure from expectations. It depends on whether
| the rest of the work makes sense as a whole.
|
| I can't turn this off. It's always running.
|
| I only see a handful of films a decade that will sit with
| me for days or weeks. The themes rhyme and have an
| ethereal logic that transcends me. These are the films
| that play my predictor like an instrument. Their choices
| are _better_ than mine. Surprisingly and delightfully so.
| They reach my soul and change who I am as a person. These
| are films that I wouldn 't change in the slightest. That
| I wish I could forget and watch again, and again, and
| again with fresh eyes.
|
| I enjoy film, but I can't stop my overactive analysis. As
| I've gotten older, this attitude has only deepened. It
| makes it hard to call watching film entirely passive or
| enjoyable.
|
| I enjoy the activity, but not always the film. The
| feelings are fleeting, and I'm left wanting for more.
|
| Except occasionally. And I cherish those.
| chucksmash wrote:
| Different strokes for different folks.
|
| I don't stream a lot. Every few months I'll just want to
| zonk out for a while though. The past few times I've wanted
| to watch "nothing in particular" it's gone something like:
|
| 1. Sit down, open Netflix.
|
| 2. Scroll through all the categories and suggestions on the
| home screen, finding nothing interesting.
|
| 3. Search for a few things I thought of while scrolling.
| Search returns things I can watch instead since what I
| searched for isn't on Netflix.
|
| 3. Open Prime Video, see something I'm interested in
| suggested pretty quickly, pay $ to rent or buy.
| WhyOhWhyQ wrote:
| I second this. I also experience this with Youtube. If
| Youtube monetized the part where you're scrolling looking
| for a video they would 10x their ad revenue.
| NavinF wrote:
| That used to my experience many years ago before I
| started using the "not interested" button aggressively.
| Now every video on my YouTube front page is 10/10
| gryn wrote:
| depends on which country you live in, and what you've
| already watched so far.
|
| I'm not one of the people complaining, but I've definitely
| seen my self scrolling a lot with Netflix recommending I
| re-watch a lot of the stuff I liked before only to end up
| giving up and switching to something else.
| consumer451 wrote:
| This raises an interesting point. Streaming has become so
| fragmented that I wonder if there enough space for a "$1M
| algorithm" which spans across all of the streaming platforms.
|
| I am not sure how that data would be gathered, maybe via an
| intermediary player like Plex, Roku, Android TV, etc..?
| cqqxo4zV46cp wrote:
| Not remotely viable.
| consumer451 wrote:
| For my own education, may I ask why?
| eddd-ddde wrote:
| The best recommendation algorithm they could provide is the
| one that recommends cancelling your subscription.
| itsoktocry wrote:
| > _Worse yet it seems to think I want to watch the three things
| that are most popular on the platform._
|
| Once they went heavy into content creation, this was expected
| and optimal for them, unfortunately.
| vkou wrote:
| Unfortunately, if they didn't go heavily into content
| creation, they'd be dead as a business, as all the rights
| holders are no longer selling them streaming rights for a
| song.
| huytersd wrote:
| That's by design. They used to have a perfectly reasonable
| system where you could search by category and sort by
| popularity and rating. They killed that specifically because
| otherwise no one would watch the buckets of random trash they
| have on there.
| dfxm12 wrote:
| They want to present to you what they want to serve you. To
| save money, they want to serve everyone as much of the same
| content as possible. Their algorithm isn't optimizing for you,
| it's optimizing for them.
| gosub100 wrote:
| I was going to make a similar comment to this:
|
| How does their licensing structure work? Do they pay a flat
| fee for unlimited license to some/all shows? Or do they pay a
| base + a micropayment for each view? If the latter, then of
| course they dont want to optimize for you, they want to find
| the optimization point between maximizing subscriptions vs
| cheapest content. Similar to the recurring conversations
| about the economics of buffets: you want a lot of people to
| come in, but not the ones who just eat all the expensive
| things.
| jvanderbot wrote:
| How have I been a movie buf and computer science pro this long
| and never found movielens. _thank you_!
| bazil376 wrote:
| Isn't $1m just like 2 eng salaries for Netflix?
| dev-tacular wrote:
| If levels.fyi is to be trusted, it like one L7's total
| compensation: https://www.levels.fyi/companies/netflix/salaries
| rappatic wrote:
| Ironically, Netflix's recommendation algorithm is now notoriously
| bad. In my experience, it seems to heavily push whatever
| "original" they just dumped $100 million into as well as what's
| most popular on the platform at the time. But it makes sense,
| since people increasingly rely on social media for deciding what
| to watch. A dollar Netflix spends improving their recommendation
| algorithm simply won't stack up to the dollar
| TikTok/Instagram/etc. spends improving theirs, because social
| media apps have so much more data to work with. It's probably
| more economical from their POV to let broad social media trends
| dictate what people watch.
| kevincox wrote:
| I suspect their algorithm has other success metrics than "did
| user enjoy movie". For example if the movie is a Netfix
| Original or exclusive movie then it is more valuable to Netflix
| because they may discuss or recommend it, which leads to more
| signups. Similarly they may prefer newer shows that are more
| likely to be discussed and raise hype than older movies that
| even if the user loves them may be less likely to advertise to
| others.
| 972811 wrote:
| in my experience from the outside it's difficult to separate
| their rec algorithm quality vs. the quality of their inventory.
| you're making the assumption they have a ton of great stuff
| that they're just not showing you, but they may not.
| VikingCoder wrote:
| If any streaming service wanted to give my family good
| recommendations, they would let me select ALL of the people who
| are watching right now.
|
| Husband.
|
| Wife.
|
| Husband and Wife.
|
| Husband and 4th grader.
|
| Wife and Pre-K.
|
| Husband, Wife, 4th grader, and Pre-K.
|
| 4th grader.
|
| Pre-K.
|
| 4th grader and Pre-K.
|
| Each of those has a _totally_ different viewing pattern and
| preferences.
| zzbzq wrote:
| Using a recommendation algorithm doesn't make any sense for
| Netflix' new business model where they produce their own content.
| It should be easy. They greenlight or buy the rights certain
| content for different demographics to maximize the coverage of
| their user base that has their needs satisfied enough to stay
| subscribed. If you're a 18-35 male, they've surely got people
| there working daily to make sure there's a content pipeline
| coming just for you. They shouldn't need AI to tell me that, they
| just need to identify when I'm in the demographic of one of their
| shows coming, and tell me about it. Maybe like a list. Still,
| they seem like they're doing a bad job at it, as I never really
| know what new shows Netflix is making for me. Sometimes I find
| out about them years later.
|
| For example, compare this to Disney+. I vaguely know about every
| Marvel and Star Wars thing 2 years in advance, and usually
| vaguely know what order they're coming out, and I don't even know
| how I know this, somehow I just know. Okay, that's easy because
| it's basically 2 IPs. But ultimately Netflix is doing something
| similar behind the scenes, why haven't they succeeded at making
| me aware of what content I'm supposed to be hyped for?
| cqqxo4zV46cp wrote:
| This feels like a disingenuous argument. "Young to middle-aged
| man likes Marvel and Star Wars" is such an obvious pick that it
| just about invites parody. As someone with at this point zero
| interest in Star Wars / Marvel content I a) wouldn't retain the
| information that you did and b) wouldn't find it to be that
| impressive that D+ was shoving it into my face.
|
| To not understand the usefulness of recommendation algorithms
| almost feels intentionally contrarian.
| cybrox wrote:
| To be honest, I would be happy if their 20+ people UX team
| would create an experience where I can easily find what I was
| watching instead of shoving stuff down my throat that I have no
| interest in.
|
| Then again, as long as people pay for that experience, it will
| continue to be as unbearable as it is.
| lubesGordi wrote:
| This is the problem across the board, regardless of service
| almost. Everything is some crappy recommendation system,
| instead of enabling me to find what I want.
|
| In addition to sucking in general, recommendation systems
| have this great property of radicalizing people politically.
| 0xfae wrote:
| Right? Like can we please get a user story on some agile
| board somewhere that says: "As a user, if the last 10 times I
| opened netflix I watched the _same show_ , I should be shown
| the option to resume that show very prominently"
|
| The number of times I've had to scroll or even search to find
| the thing that I very obviously want seems intentionally
| maddening.
| dangerboysteve wrote:
| Personally, I never thought they would have in the first place. I
| was thinking this was more for recruiting.
| eachro wrote:
| incentive problem
| chris-orgmenta wrote:
| > Well, $1m may sound like a lot of money for Netflix to pay for
| something that wasn't really used
|
| The title to me sounds analogous to 'I didn't use that piece of
| code I wrote on the weekend (but I probably learned something
| from it)'.
|
| If it were 10m, maybe relevant? But hardly to their bottom line.
|
| https://www.macrotrends.net/stocks/charts/NFLX/netflix/reven...
| ...
|
| Most companies that size (in evolving industries at least) should
| have 10-100 of those moonshot/R&D/marketing projects on the go.
| And I'm speaking to the choir here - exactly: surely most readers
| of thenextweb would be thinking the same here.
| liendolucas wrote:
| I don't get the obsession of some companies to recommend things.
| I'm happy not only for not paying Netflix but also for not
| wasting time watching crap. I have my own manual recommender:
| browse IMDB or Rotten Tomatoes of past directors, cast, etc that
| I've enjoyed. Also from a good old friend that from time to time
| recommends me films as we have very similar preferences. Randomly
| browsing genres worked very well too. That has become a near
| perfect movie recommender over many years.
| slig wrote:
| They push very aggressively for whatever new TV series / movie
| they just released that is "top 1 in your country".
|
| Spotify is the same with whatever crappy playlists and podcasts
| they want you to listen.
|
| YouTube doesn't even bother showing results that you searched
| past the 10 or so results, it's "content you might like" which
| is crap and unrelated to anything I've watched, 100% of the
| time.
| liendolucas wrote:
| We live in a crap-stream world LOL.
| cqqxo4zV46cp wrote:
| Is this a serious comment? Netflix does not and should not base
| its strategy on how you specifically choose to consume media.
| Surely you understand that most Netflix users don't use IMDb/RT
| _at all_ , let alone as a recommendation 'engine'.
|
| Netflix, especially Netflix back then, is highly reliant upon
| in-product recommendations because if people sought out
| specific pieces of media they'd often find that Netflix didn't
| have it.
| jakearmitage wrote:
| Exactly. Give me proper filters in the search tool, I'll find
| my own movies.
| IshKebab wrote:
| I dunno I think _when they work_ recommendations are actually
| quite good. YouTube 's recommendation system is very dumb but
| even so it's quite good (sorry Googler's; I'm sure you have
| optimised the wrong metric extremely well). Amazon's "people
| also looked at" is very useful - more useful than the actual
| search in my experience.
|
| I think it really falls down on Netflix because it doesn't have
| anything good to recommend to you 90% of the time. Doesn't
| matter how good your recommendation algorithm is if your
| catalogue is 90% dross and the user has already watched the
| remaining 10%.
| codelikeawolf wrote:
| Hang on a sec, it looks like BelKor's Pragmatic Chaos and The
| Ensemble had the exact same Best Test Score and % Improvement.
| Did BelKor win because they submitted their solution _20 minutes_
| earlier than The Ensemble? Or is that Best Submit Time something
| else?
| Irishsteve wrote:
| Yes - from what I remember the team that came second lost out
| due to something like 6 decimal places and the time diff in
| submission. It was pretty close!
| codelikeawolf wrote:
| Wow! That's bananas! Imagine being 20 minutes and 6 decimal
| places away from a million bucks.
| thaumasiotes wrote:
| Being six decimal places away from winning means the winner
| was one million times better than you.
| gyudin wrote:
| Netflix just needs decent movies in it's library :)
| tunesmith wrote:
| Interesting that the answer is basically "because of akrasia". I
| wish more of these recommendation algorithms were based more off
| of who we'd like to be rather than how we behave.
| softwaredoug wrote:
| As someone who works in search relevance, having just a great
| algorithm isn't worth much.
|
| You need all the team and know-how that has the maturity to
| maintain such an algorithm. Not just the ML skills. But all the
| bazillion ops, data quality, and many other things that go around
| it.
|
| I've worked with a lot of teams that have one smart person
| building stuff off to the side, in R or a Notebook, and then
| nobody knows how to productionize it. They try to throw the
| algorithm over the fence. Even if the team somehow succeeds in
| getting it into an A/B test, it eventually falls by the wayside,
| unless they can build a team and workflow around that person /
| algorithm / methodology.
| whimsicalism wrote:
| > I've worked with a lot of teams that have one smart person
| building stuff off to the side, in R or a Notebook, and then
| nobody knows how to productionize it. They try to throw the
| algorithm over the fence. Even if the team somehow succeeds in
| getting it into an A/B test, it eventually falls by the
| wayside, unless they can build a team and workflow around that
| person / algorithm / methodology.
|
| In my view, this marks a cultural failure - and certainly not
| on the part of the 'one smart person.'
| LudwigNagasena wrote:
| Anyone who has experience with ML wouldn't be surprised by that.
| Oftentimes ML competitions are about combining dozens of models
| together to juice the extra 0.01%--something that isn't viable in
| a production environment as the quote in the article confirms.
|
| AFAIK, modern ML challenges try to combat that by moving from
| answer-only submissions to code submissions and putting
| constraints on compute.
| bob_theslob646 wrote:
| Dr. David Belanger (RIP) was a part of advising this team
| (BelKor)and knew all about this, but never ever bragged about
| until one day I stumbled upon it and asked him about his
| involvement.
|
| https://www.thrillist.com/entertainment/nation/the-netflix-p...
|
| Also, they definitely used code from the first two competitions
| in the company.
|
| "The first year of the competition, in 2006 and 2007, the
| technical advancements that were made by us and the other big
| teams I think were really significant in the field of recommender
| systems," says Volinsky. He thinks the idea that Netflix didn't
| use the results is a misconception. "We gave them our code. They
| definitely did implement and use those breakthroughs that we made
| in the first year."
|
| For more that are interested , I can speak about it to the best
| of my ability. https://ieeexplore.ieee.org/author/38180399800 (
| that is some things about him) Essentially he would say that they
| initially shelved it over time as it was not needed, but they
| definitely used. Belanger tragically died November 18, 2022.
| gwern wrote:
| Their 2015 paper on how they were doing recommenders also
| mentions that they used bits and pieces of the prize work:
| https://gwern.net/doc/reinforcement-learning/exploration/201...
| nashashmi wrote:
| Engineers called it "Not worth my valuable time" kind of
| engineering effort. An early version of the 10x engineer (who
| refuses to do lots of meaningless work).
| gwbas1c wrote:
| I remember meeting someone at a networking event who believed
| they could win the competition and get > 90% accuracy.
|
| I tried to explain that you can't read minds; you can't account
| for the fact that I'm going to watch a movie that I saw in a
| random internet post that I didn't know that I liked...
|
| Which was my real beef with the contest: Computers can't read
| minds. They can't analyze the content of the movie and have the
| emotional experience that a person can.
| jfengel wrote:
| Streaming gives them much better information than a user's
| voluntary stars. Did you actually watch it? All of it? How soon?
| All at once?
|
| User ratings are fraught. The things a user actually does are
| more likely to be sincere. And it doesn't put any burden on the
| user. I have a feeling that even the simple thumb up/down means
| less than whether you actually finished watching the movie.
___________________________________________________________________
(page generated 2024-01-10 23:00 UTC)