[HN Gopher] How We Found 7 TiB of Memory Just Sitting Around
___________________________________________________________________
How We Found 7 TiB of Memory Just Sitting Around
Author : anurag
Score : 62 points
Date : 2025-10-30 18:25 UTC (1 days ago)
(HTM) web link (render.com)
(TXT) w3m dump (render.com)
| shanemhansen wrote:
| The unreasonable effectiveness of profiling and digging deep
| strikes again.
| hinkley wrote:
| The biggest tool in the performance toolbox is stubbornness.
| Without it all the mechanical sympathy in the world will go
| unexploited.
|
| There's about a factor of 3 improvement that can be made to
| most code after the profiler has given up. That probably means
| there are better profilers than could be written, but in 20
| years of having them I've only seen 2 that tried. Sadly I think
| flame graphs made profiling more accessible to the unmotivated
| but didn't actually improve overall results.
| zahlman wrote:
| > The biggest tool in the performance toolbox is
| stubbornness. Without it all the mechanical sympathy in the
| world will go unexploited.
|
| The sympathy is also needed. Problems aren't found when
| people don't care, or consider the current performance
| acceptable.
|
| > There's about a factor of 3 improvement that can be made to
| most code after the profiler has given up. That probably
| means there are better profilers than could be written, but
| in 20 years of having them I've only seen 2 that tried.
|
| It's hard for profilers to identify slowdowns that are due to
| the architecture. Making the function do less work to get its
| result feels different from determining that the function's
| result is unnecessary.
| Negitivefrags wrote:
| I think the biggest tool is higher expectations. Most
| programmers really haven't come to grips with the idea that
| computers are fast.
|
| If you see a database query that takes 1 hour to run, and
| only touches a few gb of data, you should be thinking "Well
| nvme bandwidth is multiple gigabytes per second, why can't it
| run in 1 second or less?"
|
| The idea that anyone would accept a request to a website
| taking longer than 30ms, (the time it takes for a game to
| render it's entire world including both the CPU and GPU parts
| at 60fps) is insane, and nobody should really accept it, but
| we commonly do.
| javier2 wrote:
| its also about cost. My game computer has 8 cores + 1
| expensive gpu + 32GB ram for me alone. We dont have that
| per customer.
| nitinreddy88 wrote:
| The other way to look is why adding NS label is causing so much
| memory footprint in Kubernetes. Shouldn't be fixing that (could
| be much bigger design change), will benefit whole Kube community?
| bstack wrote:
| Author here: yeah that's a good point. tbh I was mostly
| unfamiliar with Vector so I took the shortest path to the goal
| but that could be interesting followup. It does seem like
| there's a lot of bytes per namespace!
| hinkley wrote:
| Keys require O(logn) space per key or nlogn for the entire data
| set, simply to avoid key collisions. But human friendly key
| spaces grow much, much faster and I don't think many people have
| looked too hard at that.
|
| There were recent changes to the NodeJS Prometheus client that
| eliminates tag names from the keys used for storing the tag
| cardinality for metrics. The memory savings wasn't reported but
| the cpu savings for recording data points was over 1/3. And about
| twice that when applied to the aggregation logic.
|
| Lookups are rarely O(1), even in hash tables.
|
| I wonder if there's a general solution for keeping names concise
| without triggering transposition or reading comprehension errors.
| And what the space complexity is of such an algorithm.
___________________________________________________________________
(page generated 2025-10-31 23:00 UTC)