[HN Gopher] Metrics-driven product development is hard
___________________________________________________________________
Metrics-driven product development is hard
Author : kiyanwang
Score : 60 points
Date : 2021-12-27 09:53 UTC (13 hours ago)
(HTM) web link (blog.doubleloop.app)
(TXT) w3m dump (blog.doubleloop.app)
| ergest wrote:
| That's because metrics are great for optimization and fine tuning
| features but terrible for innovation
| rch wrote:
| One reason might be that great innovators don't naturally
| gravitate towards middle management (or management consulting),
| so the metrics are flawed out of the gate.
| dwaltrip wrote:
| We don't have the tools to measure the complex dynamics and
| factors that result in innovation.
| kqr wrote:
| Do you have evidence for this claim? I think metrics are
| absolutely invaluable for innovation.
|
| What is innovation but throwing 100 things at the wall and
| seeing which two of them stick? And to quickly get a sense of
| which things have stuck to the wall, metrics are the only
| reliable way.
| smugglerFlynn wrote:
| It probably depends on the type of innovation. If you are
| trying to _do things differently_ , metrics are invaluable --
| most modern internal innovation teams use metrics to build
| and manage an idea funnel.
|
| However, if your innovation strategy is to _do different
| things_ , you will likely be focused on market
| differentiation, customer development etc. Common practice is
| to refer to AARRR metrics for this, but it's not like metrics
| themselves are the main focus.
| dwb wrote:
| Metrics are great for testing a quantifiable hypothesis, but not
| all worthwhile hypotheses are quantifiable, and in ignoring the
| latter we discount an essentially human side to creativity and
| that's a big turn-off for me.
| black_13 wrote:
| karaterobot wrote:
| The last company I worked at was very metrics-driven, and used a
| really methodical, numbers-driven approach to make big,
| unpopular, counter-intuitive bets that failed catastrophically,
| while ruining morale, and trust in leadership, and gutting the
| company.
|
| After that experience, my view of metrics-driven product
| development is that it's a way to offload the burden of thinking
| too hard or understanding where you're going. When people talk
| about metrics-driven product development, I think of drivers who
| blindly follow their GPS' directions into a lake. Obviously,
| metrics can be a useful _supplement_ to a deep understanding of
| the product, the domain, the users, and the company, but keep it
| on a leash.
| hinkley wrote:
| Accidental or otherwise, we lost something. We had a situation
| where we had a three legged stool of product/marketing, QA, and
| Development. If any of the three were too far out of line, the
| other two could drag them back.
|
| But we killed QA, and Ken Schwaber gave us a treadmill to run
| on forever where structurally, nobody has time to think past
| the end of next month. So Product has been running
| progressively more roughshod over everything for the last 10
| years. And since 50% of devs have 5 years or less of
| experience, that means most people don't know that things used
| to be better.
| thewarrior wrote:
| I recently had a revelation that merely using a product can
| often reveal the organizational structure and the cultural
| incentives that went into making it.
|
| A company that is famous for being metrics driven and having
| teams that optimise for their own metrics is Facebook. You can
| tell that the product has "seams". Each page and widget feels
| like it was individually optimized and then stitched together.
| Overall it somehow feels off.
|
| Apple puts design on top and has a kind of dictatorial veto
| over all aspects of the product. Hence products made by Apple
| feel seamless and cohesive. They are also more polished because
| when you solely rely on metrics you make it as good as it needs
| to be and then stop.
| karaterobot wrote:
| Sounds like you've independently discovered Conway's Law, one
| of the most terrible, true things you can say about software
| organizations:
|
| https://en.wikipedia.org/wiki/Conway%27s_law
| akiselev wrote:
| _> After that experience, my view of metrics-driven product
| development is that it 's a way to offload the burden of
| thinking too hard or understanding where you're going. When
| people talk about metrics-driven product development, I think
| of drivers who blindly follow their GPS' directions into a
| lake._
|
| There's also the flip side - metrics-driven development used to
| keep other people from thinking too hard and scrutinizing work.
| At my last company, many product decisions were justified using
| a litany of metrics including AB tests, which people just
| swallowed because "the data speaks for itself!"
|
| I once took a look at the app we were using to run the AB tests
| and not a single one had reached statistical significance. The
| company was basing its decisions on data considered garbage
| even by the AB app's naive p-value algorithm. I thought that
| after dozens of A/B tests at least _one_ would be statistically
| significant just by dumb luck but no, not a single test had
| even the veneer of credibility.
| ford wrote:
| Metrics are great as an objective measure of feature
| releases/goals/success, but they don't provide the whole picture
| in terms of product success or direction.
|
| Years ago I interned on a messaging team at a FAANG company where
| _everything_ was driven by ~3 metrics. We had a screen with the
| trailing 30d average of our metrics, future work was prioritized
| based on the estimated impact to the metric, and the success of
| our team was based on these 3 metrics.
|
| My intern project involved some machine learning modeling using
| the (anonymized/scrubbed) content of the messages sent with our
| product, and despite the team having existed for >12 months, this
| was the first time anyone on the team had read a message sent
| with the product.
|
| In this case, the sharp focus on metrics was super useful for
| prioritizing features and evaluating our success - but was done
| so to a fault when the team didn't have a concrete idea of what
| people were trying to do with our product.
|
| The usual solution to this is to spend more time talking to users
| - it'd be interesting if there was a process that can help teams
| know how much time to spend on metrics vs qualitative feedback
| tlarkworthy wrote:
| metrics driven development has killed facebook.com. On its own,
| it is too short sighted.
|
| How do you measure long term success? hint: u can't.
|
| Blind application of ML is therefore, also dangerous for similar
| reasons.
|
| metrics are useful for tightening the tactical decisions like,
| fuck, our landing page conversion is low so let's invest there,
| but strategically metrics are unable to be creative and
| innovative and all the other things you need for a great product.
| coldcode wrote:
| At my last (very large but not FAANG) employer we collected huge
| amounts of metrics in our apps, ignored them entirely, and based
| everything on executive whims. Is this Whim-Driven Development?
| foobiekr wrote:
| This is sometimes called "reality."
| kqr wrote:
| Known as HiPPO -- highest-paid person's opinion.
| [deleted]
| indymike wrote:
| Metrics are really good, and really can help you reason about
| what will happen to usage when you add, remove or change a
| feature. There are two better / essential tools: First, dog food
| / use your product if you can. I build recruiting software. When
| I hire developers, I am a user, just like my customers. You'll
| really feel the impact of defects and bad assumptions have, and
| it will really help you understand how to make users happy. The
| second is to work closely with a few customers. This way you have
| someone who can heat check the metrics and confirm they are
| having the same experience you do when you dog food, to make sure
| you are not being led to an alternate reality.
| debarshri wrote:
| We have been using metrics driven product development lately.
| What we have been doing is hypothesize what will grow our product
| feature based on org identity, talking to customers, looking at
| the market. We define some metrics which we believe represents
| the success metrics in the best way possible. Some are based on
| domain expertise, some are just best guess. We use these metrics
| as a way to ascertain the hypothesis. We are not using it as
| indicator to build the next thing. The thing is that the success
| hypothesis or strategy might be wrong so we are willing to throw
| that away and start all over again. Not sure if it is the best
| way to do it.
| baybal2 wrote:
| Metrics driven software development gets more meaningless, the
| bigger is the customer base, and frequent are releases.
|
| A good way to make a big organisation chase meaningless things
| without your bosses realising, and even get a promotion for it.
| nowherebeen wrote:
| Can you give an example of this? I am having a difficult time
| understanding your meaning.
| baybal2 wrote:
| Google Chrome is a great example "developers in the hamster
| wheel" of metrics.
|
| Keeps getting constantly redesigned for seemingly no reason,
| and certainly not all for the better.
| Nextgrid wrote:
| Google Chrome's (or any other established software's)
| useless changes are absolutely metric-driven. The metric is
| some employees' salaries.
| beebmam wrote:
| Metrics-driven product development sounds like a good idea but in
| practice ends up being Cargo Culting.
|
| That's usually because people rarely reflect on the metrics
| they're deriving meaning from, rarely abandoning ambiguous
| metrics as misleading, and rarely imagining new metrics to find
| utility. Those are very hard tasks that take time and deep
| thought.
___________________________________________________________________
(page generated 2021-12-27 23:01 UTC)