[HN Gopher] Noisy neighbor detection with eBPF
___________________________________________________________________
Noisy neighbor detection with eBPF
Author : el_duderino
Score : 104 points
Date : 2024-09-11 18:11 UTC (4 hours ago)
(HTM) web link (netflixtechblog.com)
(TXT) w3m dump (netflixtechblog.com)
| Triphibian wrote:
| I had an obnoxious neighbor with a tv on his back porch who would
| watch Love is Blind so loud I could hear it throughout my house.
| Was kind of hoping this would be about that.
| ibash wrote:
| Simple solution: figure out which episode they were on and play
| the next episode even louder.
|
| Also it's a fantastic show.
| xyst wrote:
| > fantastic show
|
| More like a fantastic show for rotting your brain. Just like
| any other "reality" show out there.
| ulnarkressty wrote:
| Back in the day when I used to live in a multi-apartment
| building someone was being so loud as to wake me up in the
| middle of the night and I could never figure out who it was. It
| was an old building so the noise would transfer across many
| floors/walls. I was trying to come up with an engineering
| solution to this but in the mean time I got a new job and had
| to move anyway. Probably a microphone array would work to
| triangulate the source, but it would also be hard to explain to
| the police.
| xyst wrote:
| > Probably a microphone array would work to triangulate the
| source
|
| https://www.youtube.com/watch?v=IRELLH86Edo
| efitz wrote:
| Go low-tech: call 911, report a noise complaint, and then the
| police locate the source :-D
| password4321 wrote:
| Then they come straight to your door for your statement.
| akira2501 wrote:
| Learning to sleep with ear plugs in was the best investment
| of time I've ever made. Of course they don't totally kill out
| the sound but they lower it down by enough dB that you can
| comfortably ignore it, it takes piercing annoying noises and
| drowns them down to an ignorable level.
| xyst wrote:
| > could hear it throughout my house
|
| Got to love American built homes that have poor or non-existent
| insulation.
| BenFranklin100 wrote:
| I blame NIMBYism, at least for older multi-family. (Shoddy
| SFH with poor noise insulation is a different story.) If
| zoning were more relaxed, we would be tearing down 100+ year
| old multifamily in places like Massachusetts and putting in
| new homes with modern sound proofing between tenants.
| wiseowise wrote:
| Try Amsterdam.
| UniverseHacker wrote:
| I was hoping it was about some tech to triangulate coordinates
| of illegal fireworks or gunfire. My wife has PTSD and our
| neighborhood has become almost unlivable because of illegal
| commercial grade fireworks year round.
| Terr_ wrote:
| Is there any tech that reliably tell the _difference_ between
| fireworks and gunfire?
|
| I live in a neighborhood where the question has come up a few
| times.
| gruez wrote:
| If shotspotter[1] is anything to go by, doing this sort of
| stuff accurately is non trivial.
|
| [1] https://en.wikipedia.org/wiki/ShotSpotter#Accuracy
| magundu wrote:
| Is it open source?
| efitz wrote:
| Yes, the link is in the article:
| https://netflixtechblog.com/announcing-bpftop-streamlining-e...
| packetslave wrote:
| that's a related tool for monitoring the utilization of BPF
| programs within the kernel, but NOT the focus of the article
| -- detecting noisy neighbors in _non_ -EBPF workloads.
| striking wrote:
| I think the code as given in the article is all you get.
| bieganski wrote:
| can someone explain to me, isn't it the kernel's responsibility
| to preempt the CPU-heavy userspace thread (noisy neighbour in our
| case) after fixed slice of time anyway?
| jeffbee wrote:
| If a sleeping low-latency task becomes runnable due to events
| (network, storage, timers) then ideally it would start running
| straight away, preempting a throughput-oriented task.
| mrguyorama wrote:
| Is this not the case in Linux land? On Windows 7, I could run
| a CPU based bitcoin miner that pegged every core (including
| hyperthreads) to 100% and still browse the internet with zero
| stutter, latency, or slowdown, because the Windows scheduler
| had no issues giving whatever time was needed by the UI or
| Window Event processing loop and just feed whatever was
| leftover to the number crunching app.
| gruez wrote:
| I think they have priority boosting for apps that are in
| focus. It doesn't work for tasks in general though.
| Network/disk throughout takes a hit when your cpu is
| pegged, for instance.
| ruthmarx wrote:
| Part of that was likely due to Windows doing its graphics
| at kernel level instead of in userspace.
| jeffbee wrote:
| This is nifty, but not really congruent with my understanding of
| the "noisy neighbor" phenomenon. This work seems to reveal when
| there are more runnable threads than CPUs, leading to tasks
| waiting to run. The way I use "noisy neighbor" is that it is a
| concurrent task that trashes some microarchitectural resource,
| forcing the victim process to use more CPU time. For example, a
| process on another CPU core in the same cache domain that trashes
| all of the shared cache lines, or fills up all the load/store
| slots, or that uses more thermal power causing a global clock
| speed slowdown.
| eigenform wrote:
| I thought so too - it seems like this is more about "who is
| being preempted by who" (although maybe noisy neighbor in the
| sense of "hogging up CPU time" does often imply "polluting
| hardware resources" to some degree, especially considering
| these machines probably have SMT)
| Thaxll wrote:
| Isn't noisy neighbor less of a problem on AWS since nitro? Where
| I work we monitor that with some CPU steal metrics and it's very
| rare to see it nowdays.
| wmf wrote:
| In this case the noise is coming from inside the house, er, the
| VM so it's not Nitro's problem.
| paxys wrote:
| Why on earth is the Netflix tech blog still on Medium? When I
| open the link half my screen is blocked by a popup asking me to
| sign up and upselling me on a membership. Why?
| tptacek wrote:
| _Please don 't complain about tangential annoyances--e.g.
| article or website formats, name collisions, or back-button
| breakage. They're too common to be interesting._
|
| https://news.ycombinator.com/newsguidelines.html
| MichaelZuo wrote:
| Is it 'too common'?
|
| I've never seen another corporate public facing blog asking
| readers to sign up with some third party service and so on,
| not even as a banner ad.
| ruthmarx wrote:
| I agree it is worth commenting on and not common. It makes
| no sense at all for a company like Netflix to use any
| external service to blog.
| __turbobrew__ wrote:
| I wonder if they A/B tested the EEVDF scheduler? Allegedly it is
| supposed to reduce noisy neighbour effects in multi tenanted
| environments.
|
| It would be interesting if kubernetes had stronger CPU isolation
| techniques. There is the static cpu manager which will dedicate
| cores for a specific container but I believe it isn't smart
| enough to know about hyperthreads and so you can still get a
| noisy neighbour running on the other hyperthread of the core.
|
| Ideally I would like to see kubernetes slice out a certain number
| of cores for the system daemons/kernel, IRQs, and NIC buffers.
| Some very low latency systems running on top of kubernetes
| recommend using isolcpus to do this but it would be nice to have
| something built into the kubernetes CPU manager to do this. A
| container running inside the guaranteed QoS class should have
| exclusive access to the cores and never be pre-empted.
___________________________________________________________________
(page generated 2024-09-11 23:00 UTC)