[HN Gopher] Some Reasons to Measure
___________________________________________________________________
Some Reasons to Measure
Author : cdwhite
Score : 182 points
Date : 2021-08-27 11:00 UTC (1 days ago)
(HTM) web link (danluu.com)
(TXT) w3m dump (danluu.com)
| _tom_ wrote:
| damn, that's a lot of interesting content. I meant to go to sleep
| an hour ago, and I'm still reading.
| k__ wrote:
| _" When the reporter doesn't have a repro for the bug, which is
| quite common when it comes to distributed systems, the bug will
| be written off as non-existent"_
|
| This.
|
| If this is your modus operandi, then please think twice.
|
| For me, as the person with the problem, this just sounds like "I
| don't want to be bothered with this right now".
|
| Most maintainers don't see this as a possibility to improve their
| project, but as trouble for their brilliant idea. Grow up!
| vajrabum wrote:
| Looks like he wasn't kidding, at least for Java shops with
| untuned JVMs. This was linked from the article and from Dan Luu's
| work at Twitter. "We spent one day1 building a system that
| immediately found a mid 7 figure optimization (which ended up
| shipping). In the first year, we shipped _mid 8 figures per year
| worth of cost savings as a result_. The key feature this system
| introduces is the ability to query metrics data across all hosts
| and all services and over any period of time (since inception),
| so we 've called it LongTermMetrics (LTM) internally since I like
| boring, descriptive, names.
|
| https://danluu.com/metrics-analytics/
|
| I'll add that I worked in systems at a large php shop which
| didn't have good metrics for utilization. After the adoption of
| HHVM their utilization dropped enough that they ended up buying
| extra hardware in the 6 figures. They knew that performance was a
| lot better but they didn't quantify it. So surprise! All it took
| to figure that out and start making better decisions was to
| collect SAR data and dump it to a central log. So, yeah. Boring
| is good or at least a good place to start.
| shele wrote:
| Talking about measuring time I would put a date on the blog post
| vincnetas wrote:
| If you go to homepage, all entries are with date, its just that
| date is not displayed in article itself.
|
| https://danluu.com/
| dang wrote:
| Dan doesn't do that, but from
| https://web.archive.org/web/*/https://danluu.com/why-benchma...
| I gather that it's recent.
| zinekeller wrote:
| Dan does put dates... with month-level granularity and only
| in the home page: https://danluu.com/
| dang wrote:
| I had no idea! Thanks!
| amadeuspagel wrote:
| Measuring is undervalued, particularly vs writing, because it
| doesn't show off creativity.
| danielmarkbruce wrote:
| Additionally it's hard work. Creating the right tests, getting
| the data into the format you need to do analysis, collating it
| and finding the important things, explaining it.
| mcrad wrote:
| Very hard work and often takes a lot of creativity. I always
| thought measurement was the essence of engineering but these
| days find it to be a lost art. Too often the market for the
| measure takes priority over the scientific piece.
| Karrot_Kream wrote:
| Most folks don't naturally like to measure things. Most people
| view the world through their own personal lens and they
| categorize phenomena through their experiences. It makes it
| hard to "sell" the idea of measurement, since most people only
| see the things they're predisposed to see.
| IAmEveryone wrote:
| You should really measure that.
|
| Because, as it is, it just sounds like "advertisement doesn't
| work on me, because I'm better than normal people".
| scns wrote:
| If i read parent charitably i find things i can agree too,
| would phrase them differently thought.
|
| > Most folks don't naturally like to measure things.
|
| Heuristics save a lot of energy in contrast to critical
| thought.
|
| > Most people view the world through their own personal
| lens and they categorize phenomena through their
| experiences.
|
| Our perception is often coloured/distorted by our beliefs,
| which then lets information through that agrees with our
| presuppositions.
|
| > It makes it hard to "sell" the idea of measurement, since
| most people only see the things they're predisposed to see.
|
| I often experienced people reacting annoyed when i cited
| studies, could have been my tone, could have been me/them
| wanting to win/be right.
| Karrot_Kream wrote:
| I think my tone came across as more elitist than I meant. I
| certainly suffer from all of these issues! I enjoy working
| with data and ML so I'm aware of and try to actively combat
| them in myself but I'd be lying if I were to say I haven't
| succumbed to these problems myself in the past and don't
| continue to do so.
| gopalv wrote:
| > Measuring is undervalued
|
| Goodhart's law has been a problem with measures.
|
| "When a measure becomes a target, it ceases to be a good
| measure."
|
| The problem is that a competitive measure tends to overshadow
| the utility of the measure (look at anything that starts with a
| TPC- or the first rotation of a Tesla roadster tire).
|
| So the biggest impact of a measure is immediately after it is
| introduced, because the market hasn't engineering itself to
| overfit to that measure.
|
| And that is what Dan's post clearly says, that immediately
| after he brings up something, there is an actual change in the
| industry practice that happens.
|
| > it doesn't show off creativity.
|
| But something like Jepsen is amazingly useful because it is a
| creative nothing-up-my-sleeves end-run around first-party
| testing - actual third party testing for the product is the
| red-team to a QE team doing the blue-team work. The problem is
| that a lot of people leave that to the same vendor who is
| selling the product with its promises from their sales team -
| it is hardly an open book into the bugs open list (I work in
| open-source, so I have to wash my dirty laundry in public).
|
| As a side not to this, I had a meeting in the last 2 days where
| someone told me that the "QE team is doing a good job, because
| the test have been consistently green as the release
| approaches".
|
| And I didn't know exactly how to respond to that.
___________________________________________________________________
(page generated 2021-08-28 23:02 UTC)