[HN Gopher] Campbell's Law: The dark side of metric fixation
___________________________________________________________________
Campbell's Law: The dark side of metric fixation
Author : open-source-ux
Score : 138 points
Date : 2021-11-21 07:51 UTC (15 hours ago)
(HTM) web link (www.nngroup.com)
(TXT) w3m dump (www.nngroup.com)
| majkinetor wrote:
| Covid metrics are glaring case to the point.
| KarlKemp wrote:
| Of all the metrics we have mortality is the least likely to
| suffer from these problems: death is eminently measurable, and
| won't be fudged by wishful thinking, expectations, or placebo
| effects. Outright manipulation is hard, because you can just
| count the fresh graves in the cemetery to get a good estimate,
| because it's is such an important event, and also because there
| are laws and a bureaucracy for measuring it. And it gets
| awfully close to what you really want, living long & happy.
| While it doesn't measure happiness, every study on happiness
| shows that it is relatively resilient to changes in
| circumstances. Therefore, avoiding death is less of a _proxy_
| and more of a real thing than anything Google Anaytics will
| ever tell you.
| majkinetor wrote:
| Its IMO the most problematic one - you die for totality of
| stressors you carry, not just because of the Covid.
| Jiro wrote:
| It's still fudgeable, because of things like whether everyone
| who dies while having Covid should count as a death from
| Covid.
| KarlKemp wrote:
| You should have been here last year, when it was still
| believable you had not heard of the excess death
| statistics: https://www.cdc.gov/nchs/nvss/vsrr/covid19/exce
| ss_deaths.htm...
| dehrmann wrote:
| Those still capture things like untreated cardiac
| episodes because someone was afraid to seek treatment.
| Accounting for these things is hard.
| dehrmann wrote:
| At this point, my gripe with covid metrics is that we're past
| the point where they control long-term outcomes. Covid will be
| endemic, you will either get vaccinated, covid, or both, and
| masks and lockdowns are only tools for buying time to get more
| people vaccinated because once restrictions are lifted, it will
| spread again. The only metrics that matter are vaccination rate
| and hospitalization rate, and for vaccine-rich countries, the
| latter is just a question of now or later.
| d0mine wrote:
| It reminds me about Soviet central planning: <<Economist Paul
| Craig Roberts recalled, "a famous Soviet cartoon depicted the
| manager of a nail factory being given the Order of Lenin for
| exceeding his tonnage. Two giant cranes were pictured holding up
| one giant nail."
|
| ... The singular focus on a metric imposed from the top-down
| obliterated any incentive to innovate or differentiate production
| according to what customers might actually want.>>
| https://www.forbes.com/sites/emilychamleewright/2020/01/15/w...
| ogogmad wrote:
| The grand "eBay strategy" for Google Ads was to advertise on
| "long tail" keywords. This in practice meant that eBay would
| place its adverts on Google searches for extremely specific
| keywords like "Charizard pokemon card". This ended up
| maximising metrics like Click-Through Rate and Conversion Rate,
| and eBay's advertising division was famous in the advertising
| industry for its success in doing this. It was literally known
| as the "eBay strategy" and was treated with some respect. It
| took one insistent individual to convince eBay to turn all
| their advertising off and see if it decreased profits; it
| increased profits*. The problem is that right below the advert
| (which eBay was paying a lot of money for) was a link to the
| same page which eBay didn't need to pay for. The same page was
| a legitimate Google search result, and so the Google Search
| advertising was losing eBay a bunch of money.
|
| [edit]
|
| * - Thanks to the commenter above for the correction.
|
| Note: I've tried to be more careful in not implicating eBay's
| entire advertising division.
| Spooky23 wrote:
| It must have some value, as I searched on duck duck go for a
| Soviet fighter plane as the top result was an ad: "we have
| SU-27 in stock, seriously"
| djbusby wrote:
| Sounds likes it reduced expenses rather than increase
| revenue. Maybe I'm pedantic but those are important
| difference
| Jensson wrote:
| That works if you are the top, a competitor would probably
| earn a lot from that strategy until they become the top.
| compiler-guy wrote:
| This is exactly what happens. On the very long tail there
| may be no one willing to bid against you so there is no
| reason to buy the top slot, but if someone is, then
| suddenly it makes sense.
| DoneWithAllThat wrote:
| I've said on here before, an additional extremely negative
| incentive that comes from an overemphasis on measurable metrics
| is that a lot of (important, subtle, meaningful) metrics are very
| difficult to dumb down for non-technical audiences, or even
| technical audiences lacking domain knowledge.
|
| As a result the metrics that tend to be tracked are not just the
| measurable ones, but the ones for which the importance of them
| and the meaning of their values are comprehended by sometimes a
| very low common denominator audience. Usually needing to be tied
| directly in some way to money. (INdirectly doesn't cut it, that
| ends up being too confusing a lot of the time).
| burnafter187 wrote:
| Thinking about it, in the instance of important but esoteric
| metrics, do you think that the most frequent case has to do
| with the [expected] ability to impact or exploit it
| meaningfully? I don't do a lot of work with numbers, but it
| seems to me that in terms of affectation, those dumb statistics
| are the low hanging fruit with low investment cost in terms of
| manipulation, thus yielding "better" results and isolating them
| to the use case while alienating the complex metrics. It's a
| lot easier to take a metric that isn't contextually affected
| than the obverse, and up/downregulate it.
|
| I think my example would be the quasi-metric of the single-
| sample workup you'd get at a hospital. You get a single
| datapoint. Compare that to a longer term study where samples
| are taken at 4 hour intervals. The latter brings up a lot of
| subsequent questions and has a lot of interconnections with the
| world at large and the individual's biological characteristics.
| E.g. getting caught by a train 2/5 days, boosting cortisol and
| norepinephrine, or low receptor affinity for X, thus producing
| the illusion of high X levels. As you modify you also
| complicate with an increasingly large list of rules. Or you
| just write a script for statins and pat Tom on the ass, noting
| next visit that serum cholesterol is down.
| anotherevan wrote:
| I recently listened to an interview with David Manheim on
| "Goodhart's Law and why metrics fail"[1] and the key takeaways
| for me were:
|
| - You need to be using several metrics, not just one or a few.
|
| - You need a certain amount of noise in the system.
|
| [1] http://rationallyspeakingpodcast.org/240-goodharts-law-
| and-w...
| tomlue wrote:
| The whole world suffers from researchers optimizing on H-index.
| sillysaurusx wrote:
| A bit sad today, so I wanted to open a random thread and wish you
| well. Remember to take a moment to appreciate the people around
| you.
|
| One surprising thing is that a lot of people go through life with
| serious existential dread, and that you can't really empathize
| with what it's like until you experience it yourself. Sometimes I
| wonder what the point is -- we evolved that behavior for a
| reason, and I can't think of a single way it improves one's
| ability to produce grandchildren. (Apparently the best
| evolutionary traits are those that let your children have
| children, rather than you having children directly, which was
| sort of counterintuitive.) In my experience, the existential
| dread seems to do nothing but chip away at people and their
| relationships.
|
| Life gets strange at 30+. If you're around 20, my advice would be
| to relax and enjoy yourself, and truly realize that almost
| nothing you do now will matter in the long run. The reason that
| pushing yourself seems like the path to success is because you
| don't hear about the 9 out of 10 cases of ruined lives as a
| result of unrealistic expectations. But it's very easy to get
| caught up in being driven, as if it's religion.
|
| Just find something small that makes you happy. Even if it's a
| pointless number puzzle, it's still yours. And that's good
| enough.
| olau wrote:
| I don't think it's on purpose. I think the problem is that in a
| modern society, you are so far removed from the habitats that
| our ancestors evolved in that a lot of people simply can't
| cope. We're designed to spend time grabbing food, building
| shelters, making and taking care of babies, those sorts of
| things, all in small social groups. We're designed for
| immediate needs and feedback.
|
| You could try this series: http://rickroderick.org/200-guide-
| nietzsche-and-the-postmode...
|
| Seems like Nietzsche already saw this coming with science
| dealing a death blow to the old myths that kept people going,
| and with the advent of mobile terminals and apps optimized to
| keep our attention, I think we've just gotten closer to that
| vision.
| bobnamob wrote:
| Gpt-3?
| AussieWog93 wrote:
| I think they've just lost somebody that they care about. :(
| sillysaurusx wrote:
| Thankfully not. Just in a pensive mood. But my programmer
| friend has been unable to use the computer for two solid
| months due to some kind of sickness -- possibly pneumonia.
| I'm worried sick about him, because he lives in some kind
| of war-torn country with no functional medical system. I
| didn't realize how much I'd miss him till he was gone for a
| couple months on the verge of dying.
|
| But that kind of thing can happen at any time. You just
| don't really notice it till you get older, so you never
| slow down to appreciate the people around you while they're
| there. Or at least, I didn't, for a long time. Been trying
| to change that in recent years.
|
| Anyway, I really don't care about seeming weird. The
| weirdness is life itself, being a descendent of a monkey on
| a spinning speck of dust in the cosmic soup, worried about
| an abstraction we call money just to climb a societal
| ladder that doesn't really matter much in the end.
|
| Everyone has to walk their own path, but my goal was to get
| one of you to stop and think about how you want to spend
| the next few years of your life. I've seen so many people
| allow themselves to think that work = life, and most of
| them seem to regret it. So I try to consciously avoid that
| in my own. Maybe you will too.
| treyfitty wrote:
| I am in the process of preparing for Facebook PM interviews. All
| their product sense and product execution "tips" hyperfixate on
| goals, metrics, and tying to the North Star.
|
| In one sense, it seems as if the interview process is to
| indoctrinate a cult from the onset. I'm willing to bet there are
| people who go through this process, get rejected, and think the
| Facebook framework is THE right framework.
|
| As a non Facebook user, I'm conflicted between hating their
| products to the core... on the other, their TC is salivating.
| padolsey wrote:
| Yeh honestly being an SWE at FB just killed my soul (in part)
| because the job became so fixated on quantitative measures and
| experimentation...what we cannot measure cannot be important,
| surely....! :/ I think the only PMs/ICs who survive at FB are
| those who are either numbed or not aware of data fallacies like
| Campbell's Law.
| toss1 wrote:
| >>what we cannot measure cannot be important, surely
|
| Which concisely explains why the algos have zero correlation
| with moral compass, societal value, common decency, or caring
| for the commons upon which FB earns its wealth.
|
| None of that can be measured, so it is all cut down and falls
| at the feet of the engagement algorithms. The result is an
| org whose business model is poisoning the common well for
| profit.
| maneesh wrote:
| What does TC mean in this context?
| tejohnso wrote:
| Total Compensation
| orange_joe wrote:
| Total compensation
| mellavora wrote:
| > the interview process is to indoctrinate a cult from the
| onset
|
| Yes, I would say this is true in general at all companies,
| though I prefer the phrase "company/team culture" to "cult"
| (the phrases aren't that different in what they describe, just
| in emotional tone).
|
| Because it is not in itself a negative. _every_ team has a
| culture, by definition of "culture".
|
| If they can make this explicit in the interview, then you as
| the candidate get a good sense of what this culture is and if
| you want to be a part of the team.
|
| If they can not make this explicit in the interview, then you
| as the candidate know that they don't have control over this
| variable, again giving you valuable information on if you want
| to be part of the team.
| trendy-modern wrote:
| Reading the headline, never heard of champbell (sure 'soup' ^^) i
| thought this article may be about calculating (there was
| something about harmonization of markets) between us-mile, feet,
| inches, trillions and billions... and the metric fixation (in
| Europe for an example).
|
| But BTT, won't sound offending - 'Reflexion is not everywhere a
| fountain for respect", not ?
|
| Maybe i was wrong on the toppic... P-:
|
| > Comic (in German Language)://ibb.co/wJKGmNQ (-;
| bitcharmer wrote:
| I love how you referred to metric system as European
| "fixation". You're American, aren't you?
| grzm wrote:
| The phrase is taken from the title of the submission.
| liftm wrote:
| > > _posts German comic_
|
| > You must be American
|
| Maybe it's just easy to read the title that way?
| bluenose69 wrote:
| As someone once said, "you cannot fatten a pig by weighing it".
| lotsofpulp wrote:
| It should be "you cannot fatten a pig solely by weighing it".
|
| The purpose of weighing the pig is to see if what you are doing
| to the pig is working to fatten it.
| 5tefan wrote:
| I'll update my collection and add that one.
|
| Some metrics are useful. For the most pert I hate metrics. But
| you must deliver to managers if said managers lost touch with
| operations. They think they know what's going on by checking
| their metrics.
| bonniemuffin wrote:
| The only thing worse than metrics is not having metrics.
| 5tefan wrote:
| Le roi est mort, vive le roi!
| cryptica wrote:
| > Campbell's law states that the more important a metric is in
| social decision making, the more likely it is to be manipulated.
|
| This also holds for the fiat monetary system; a metric by which
| all economic value is measured. It has been manipulated in
| completely unjust ways to serve the interests of capital holders
| (and corporations) and that corruption is hidden behind mind-
| numbing complexity.
|
| Having worked both inside and outside of the finance sector. I
| sensed that as I got closer to the money printers in my career,
| people's logic seemed to break down and they seemed to understand
| less and less what's going on. You start to meet people with
| unbelievably huge blind-spots. It's weird. It's as if people were
| selected for these roles based on their tunnel vision and
| unwavering faith in authority.
| darawk wrote:
| Isn't this much better known as Goodhart's Law?
|
| https://en.wikipedia.org/wiki/Goodhart%27s_law
| codeflo wrote:
| I agree, and the article doesn't really explain a difference,
| either. Two quotes:
|
| > Campbell's law states that the more important a metric is in
| social decision making, the more likely it is to be
| manipulated.
|
| > Goodhart's Law, states that "When a measure becomes a target,
| it ceases to be a good measure".
|
| These are basically paraphrases if you accept that "target" =
| "important in decision making", and "likely to be manipulated"
| = "not a good measure".
|
| That's fine, important discoveries are often made multiple
| times independently, but let's not pretend there's some
| important difference where there isn't any.
| kodah wrote:
| Goodhart was an economist and Campbell was a social
| psychologist. The fact that they independently made similar
| observations in different fields is substantial. Though,
| maybe it's more efficient to remember the over-arching idea,
| it also makes sense to identify them independently.
| dpierce9 wrote:
| I agree they are basically saying the same thing but there is
| a subtle difference. Goodhart is observing a fact about
| measures, Campbell is offering an explanation of Goodhart's
| observation, that is, why the target ceases to be a good
| measure. Goodhart is making a very general observation which
| might be true for a number of reasons. One alternative to the
| Campbell version is that value of a measure ages so, by the
| time the measure has become a target, the world has moved on
| and the target isn't any good.
| itsdrewmiller wrote:
| This is discussed in the article, but my summary would be that
| Goodhart says "if it (a measure becoming a target) happens it's
| bad", and Campbell is saying "it will happen if it's
| important".
| [deleted]
| bregma wrote:
| They express the same abstract intent but differ in context for
| the literalists in the crowd.
| asplake wrote:
| Similar but different: one emphasises social decision making,
| the other, economics. The domain of Goodhart's law is economic
| policy.
| thewakalix wrote:
| There's a difference?
| dsr_ wrote:
| Sure: how a person frames the argument limits what kinds of
| evidence they are willing to consider.
| ojilles wrote:
| Also differentiated in the article itself, FYI.
| willvarfar wrote:
| I infer an undercurrent of "politics is about manipulating
| people not representing them" from Campbell and "Wally is going
| to code himself a minivan" (Dilbert ref) from Goodhart. So
| Campbell's might be Goodhart with a politics context but it has
| a slimier feel to me.
| SideburnsOfDoom wrote:
| > I infer an undercurrent of "politics is about manipulating
| people not representing them"
|
| There's a scene or two in The Wire about "Juking the stats"
| (1) - its about fooling the people in order to look good and
| meet campaign promises. Manipulating indeed. More like "Mayor
| Carcetti's going to juke himself a second term"
|
| > "Wally is going to code himself a minivan" (Dilbert ref)
| from Goodhart
|
| Not really, Goodhart was talking about UK monetary policy in
| 1975. It's political from the start. (2)
|
| These aren't different.
|
| 1) https://medium.com/@roshanrevankar/juking-the-
| stats-5926eaf5...
|
| 2) "Goodhart's law is an adage named after British economist
| Charles Goodhart, who advanced the idea in a 1975 article on
| monetary policy in the United Kingdom, Problems of Monetary
| Management: the U.K. Experience:
|
| Any observed statistical regularity will tend to collapse
| once pressure is placed upon it for control purposes."
|
| (Wikipedia)
| pydry wrote:
| Code coverage. Subject to campbell's law and goodhart's law.
| ksec wrote:
| There are so many of these advice "if you can't measure it, you
| can't manage it" that are wrong and yet every one keep using it.
|
| Another one is "Execution eats strategy for breakfast".
|
| The past 10 - 20 years there are so many of these being fed to
| people and it often took a long time before the sentiment shift.
___________________________________________________________________
(page generated 2021-11-21 23:02 UTC)