[HN Gopher] How Netflix Accurately Attributes eBPF Flow Logs
___________________________________________________________________
How Netflix Accurately Attributes eBPF Flow Logs
Author : simplesort
Score : 70 points
Date : 2025-04-08 18:21 UTC (4 hours ago)
(HTM) web link (netflixtechblog.com)
(TXT) w3m dump (netflixtechblog.com)
| butlike wrote:
| All that logging and they cant figure out why people are going to
| other streaming services
| neogodless wrote:
| I mean, I haven't been a subscriber for a couple years but are
| they losing a lot of subscribers?
|
| https://backlinko.com/netflix-users
|
| I see some slowing growth particularly across 2021/2022 but as
| of this report (April 2024) they were still growing through
| 2023.
|
| 260M subscribers. They aren't exactly hurting.
| EwanToo wrote:
| It's now over 300 million subscribers, as you say they're
| doing OK.
|
| https://ir.netflix.net/investor-news-and-events/financial-
| re...
| ASalazarMX wrote:
| Despite their awful UX, I'm always impressed with how reliable
| their service is, technically speaking. Video is always good
| and responsive even on less-than-stellar connections, you can
| leave a show paused for hours, and resume it almost instantly.
| Their fast.com speed test is always much faster than your
| regular internet access, I guess thanks to their Open Connect
| Appliances.
|
| It must be great to work for them in infrastructure and
| backends.
| yuters wrote:
| I have an old fire tv and never tried to stop automatic
| updates on it, it has become so slow and unresponsive that
| I'm barely able to switch inputs to use something else.
| Netflix is the only app that still works on that tv.
| faitswulff wrote:
| I can only remember one major outage from them in the past
| ~10 years (in the 2020s, not the 2012 outage), and if I
| recall correctly, it was fixed in short order...and they
| never released a postmortem
| silisili wrote:
| Generally I'd agree, but you must not have seen their
| attempts at live events :(.
| seneca wrote:
| No one I know uses Netflix anymore, and I haven't for a while,
| but from what I've seen their subscriber numbers are actually
| doing quite well.
| temp0826 wrote:
| I'm not sure I know anyone that _doesn't_ have it
| steve_adams_86 wrote:
| Like seneca, I also don't know anyone who uses it. That's
| interesting. I haven't used it for around 2 years.
|
| I wonder who uses it when I see their reports, because I
| don't know them. I probably have a weird group of friends.
| I'm sure if I asked, some of my coworkers use it.
|
| My kids would love if we used it. Perhaps it's big among
| younger people.
| temp0826 wrote:
| Maybe that's part of it?...my sibling has a toddler and
| they use it for a lot of children's shows. Conversely, my
| mother watches it often too (not for kids shows!)
| itishappy wrote:
| Do you use something different? Anecdotally, I find
| Disney+ to be a major divider. Friends of mine have kids
| and Disney+ and/or they're still using a decade-old
| Netflix subscription.
| blinded wrote:
| lol wat?
|
| They make 10 billion+ a <b>quarter</b>.
| thewisenerd wrote:
| so they didn't want to pay for AWS CloudWatch [1]; decided to
| roll their in-house network flow log collection; and had to re-
| implement attribution?
|
| i wonder how many hundreds of thousands of dollars network flow
| logs cost them; obviously at some point it is going to be cheaper
| to re-implement monitoring in-house.
|
| [1]: https://youtu.be/8C9xNVYbCVk?feature=shared&t=1685
| Hikikomori wrote:
| Because vanilla flowlogs that you get from VPC/TGW are nearly
| useless outside the most basic use cases. All you get is how
| many bytes and which tcp flags were seen per connection per 10
| minutes. Then you need to attribute ip addresses to actual
| resources yourself separately, which isn't simple when you have
| containers or k8s service networking.
|
| Doing it with eBPF on end hosts you can get the same data, but
| you can attribute it directly as you know which container it
| originates from, snoop dns, then you can get extremely useful
| metrics like per tcp connection ack delay and retransmissions,
| etc.
|
| AWS recently released Cloudwatch Network Monitoring that also
| uses an agent with eBPF, but its almost like a children's toy
| compared to something like Datadog NPM. I was working on a
| solution similar to Netflix's when NPM was released, was no
| point after that.
| DadBase wrote:
| I recall a time when we managed network flows by manually
| parsing /proc/net/tcp and correlating PIDs with netstat
| outputs. eBPF? Sounds like a fancy way to avoid good old-
| fashioned elbow grease.
| nikolay_sivko wrote:
| At Coroot, we solve the same problem, but in a slightly different
| way. The traffic source is always a container (Kubernetes pod,
| systemd slice, etc.). The destination is initially identified as
| an IP:PORT pair, which, in the case of Kubernetes services, is
| often not the final destination. To address this, our agent also
| determines the actual destination by accessing the conntrack
| table at the eBPF level. Then, at the UI level, we match the
| actual destination with metadata about TCP listening sockets,
| effectively converting raw connections into container-to-
| container communications.
|
| The agent repo: https://github.com/coroot/coroot-node-agent
| zX41ZdbW wrote:
| If you are interested in network monitoring in Kubernetes, it's
| worth looking at Kubenetmon:
| https://github.com/ClickHouse/kubenetmon - an open-source eBPF-
| based implementation from ClickHouse.
| nimbius wrote:
| i refuse to believe a company that wasted $320 million dollars on
| "the electric state" could ever manage to do anything correctly.
| stripe the parking lot? stock the breakroom? clean the toilets?
| simply not possible.
| ZeWaka wrote:
| Beautiful art and book, but what a unfaithful travesty of a
| production that absolutely trodded on the original work.
| slt2021 wrote:
| Question to the Netflix folks: I saw a lot of in-house developed
| tools being quoted, do you guys have service mesh like linkerd ?
|
| Have you guys evaluated vendors like Kentik?
|
| I would love to get more insight into what do you guys actually
| do with flow logs? for example if I store 1 TB of flow logs, what
| value can I actually derive from them that justify the cost of
| collection, processing, and storage.
| r3tr0 wrote:
| we are working on a similar product that is eBPF powered and can
| extract flow logs:
|
| https://yeet.cx
___________________________________________________________________
(page generated 2025-04-08 23:00 UTC)