[HN Gopher] The 'flawed five' engineering productivity metrics
___________________________________________________________________
The 'flawed five' engineering productivity metrics
Author : aard
Score : 66 points
Date : 2022-05-16 15:56 UTC (7 hours ago)
(HTM) web link (leaddev.com)
(TXT) w3m dump (leaddev.com)
| imwillofficial wrote:
| I noticed no solution was offered. Some metric ton s usually
| better than no metric
| btrettel wrote:
| In the past, I was a patent examiner at the USPTO. Patent
| examiners have their own problematic performance metrics. In Oct.
| 2020, the metrics went through a big change that made them a lot
| more complex. I suspect that part of the motivation was to make
| the system more opaque so that it'd be harder to game. But in
| practice I think it added just as many if not more ways to game
| the system.
|
| I quickly figured out that under the new system, you could
| increase the amount of time you get for a particular patent
| application through a particular reclassification procedure
| called a C* (pronounced C-star) challenge. I'm surely not the
| only one who figured that out. The reason the C* challenge exists
| is to reclassify a patent so that it can be transferred to a more
| qualified examiner. But if it's not transferred then the amount
| of time you get can be changed. That's not necessarily nefarious
| as many applications have the wrong classification and would give
| you a lot less time than if they had the right classification.
| But examiners aren't incentivized to switch an application to the
| _right_ classification. They 're incentivized to change the
| classification so that the application gets transferred or change
| the classification so that they get more time. In the latter case
| I'd intentionally avoid adding (or even delete) any
| classifications that would reduce the amount of time I got. I
| don't suspect the long-term dynamics of this system are what
| USPTO management intends.
| xyzzy4747 wrote:
| The best metric is your own intuition about how productive people
| are being and their output, subjectively thinking about the
| quantity, quality, and impact. It becomes pretty obvious who is
| contributing a lot and who isn't. You don't need to track
| metrics.
| Traubenfuchs wrote:
| This also doesn't work and just gives social, extroverted and
| eloquent people a huge advantage.
| xyzzy4747 wrote:
| Not really. If social and extroverted people provide more
| value to the company, they should get better performance
| reviews and paid more.
| teeray wrote:
| I'll add another one: code coverage. Coverage is, at best, a
| proxy metric for how easy it is to test your code. If you make it
| easy to write tests, coverage generally takes care of itself.
| gary_0 wrote:
| "When a measure becomes a target, it ceases to be a good
| measure." - Goodhart's Law. See also, the Cobra Effect:
| https://en.wikipedia.org/wiki/Perverse_incentive.
|
| If you set up an incentive system, people will (perhaps not even
| consciously) start trying to game that system. Maybe there's some
| way to use machine learning to turn performance metrics into a
| black box that considers every possible data point? (Now there's
| a terrifying idea.)
| foolfoolz wrote:
| velocity points is not useless. it can't be used in isolation but
| points delivered by individuals is a great starting place to
| identify outliers in your org. generally if someone is delivering
| far higher or far fewer points they are making an outsized impact
| to the team (either positive or negative). it's not perfect, you
| must take context with it, but with averages and on long time
| scales it's quite reliable
| prepend wrote:
| I think velocity points are useful within a team over time.
| They are locally useful.
|
| But they are stupid to measure across teams or to compare teams
| or productivity. Velocity points are just an estimating tool,
| not a measure of value. It's useful to know that a team usually
| produces 10 points per sprint but this sprint is 5 or 20. It
| just lets you know if your team is producing "normal" or not.
|
| It's useless to try to calculate that out of 20 teams the
| average velocity points are 10 per sprint.
| Traubenfuchs wrote:
| They are COMPLETELY useless and are actively gamed by clever
| devs to reduce workload and reduce output expectations. I
| always nudge fellow devs to overestimate the tickets I will be
| working on by overstating the complexity and risks, sometimes I
| prime them with higher numbers, etc.
| hinkley wrote:
| > So not only is this metric inaccurate, but it incentivizes
| programming practices that are a counter to building good
| software.
|
| There is a critical error in this statement that some would label
| as 'subtle' but it's about as subtle as a rusty axe to the
| forehead. Judging people by lines of code doesn't 'incentivize []
| practices that are counter to building good software'. It
| _punishes you for writing good software_.
|
| While we do need to be aware of the negative consequences of
| inaction, that's a problem to be solved once you have stopped
| actively digging a hole. That's a problem to be solved once you
| have stopped pushing people into that hole. That's a problem to
| be solved once you've stopped publicly congratulating people for
| not being pushed into the hole. By you. With an audience. There
| is nothing remotely subtle or nuanced about that distinction.
|
| There's an old adage that if you make laws people can't respect,
| then they will stop respecting the law. This is but one route to
| that problem.
| madcaptenor wrote:
| My kid is figuring out pooping in the potty. We have a chart
| where we make a check mark when she does it. She likes check
| marks, especially when she can make them herself. Over the past
| couple weeks her poops have gotten smaller and more frequent.
| rightbyte wrote:
| You should make bigger marks for bigger deliveries to not give
| the wrong incentives.
| cwilkes wrote:
| Sounds like she's going up for promo!
| [deleted]
| kodah wrote:
| The problem with _all_ of these metrics is that they assume that
| there 's an engineering team, or someone on an engineering team,
| somewhere that's _not doing shit_. Whether the business believes
| they 're _not doing shit_ because they refactor more than they
| write features, because they release twice a year instead of
| every week, or because they have less frequent merges. The entire
| emphasis of measurement is squarely a technical one which comes
| from the (management) belief that _some engineer somewhere_ isn
| 't contributing and that's bringing the team down. I've rarely
| encountered these kinds of engineers, much less teams, and
| designing an incredibly painful system that senselessly costs
| people their jobs and livelihood all in the hapless pursuit of
| identifying them seems fraught.
|
| Where these metrics would be useful is if the business actually
| looked at itself first when teams or team members underperform.
| In all honesty, I've never met a manager with this kind of
| mindset; they're usually captured by the belief of the above in
| some way.
|
| Businesses do have an old way of determining whether a team is
| meeting its goals: KPIs. If the team is responsible for a
| succinct domain, problem, or stack then these KPIs are easy to
| draw and measure against because they reflect business outcomes
| rather than trying to normalize for how everyone on a team
| contributes.
| prepend wrote:
| I've worked with teams and individuals that do nothing. Like
| literally nothing, they do shit. I once had to wrap up a
| product release for contract close out or something and it
| involved getting all the code and commits from developers who
| were rolling off. It amazed me how many had zero code they had
| written in months. I had one developer that had never committed
| anything in the three months he was there.
|
| Obviously this is a management problem and it's not a single
| individuals fault. But the manager had like 90 contractors
| reporting to them and didn't care that people had zero lines of
| code written.
|
| The developer's job was to code. I wanted to mention that as
| there are developers and designers who don't code but are
| productive in other ways.
| hinkley wrote:
| I have had devs who did next to nothing, I've also had devs
| who had their repository settings wrong such that all their
| commits were either ascribed to no-one, the dev who helped
| onboard them (or wrote the docs) or a team lead.
|
| Having some metrics here can help sort this out before you
| call some senior dev at 12:30 in the morning for code they
| didn't even write.
| kodah wrote:
| I believe you, I'm sure they exist, my argument is that
| they're not common enough for this level of pain. Personally,
| the way that org is sounds like it was by design.
| dijit wrote:
| There's some for SRE's and Sysadmins too:
|
| * Cost reduction; usually by a very arbitrary amount, despite you
| having no control over what's needed
|
| * Uptimes; last job told me that I had to get 99.998% uptime,
| Googles global load balancer is only 99.9%
| MichaelBurge wrote:
| You might as well just promise 100% uptime. If you don't meet
| it, most SLAs you're only liable for a couple bucks in service
| credits anyways.
| dijit wrote:
| I'm mostly talking about a metric by which my teams
| performance would be judged.
|
| Externally to customers we had no promises of availability.
| anotherhue wrote:
| It's probably easier to measure anti productivity than software
| productivity.
|
| Excessive or poorly timed meetings, scope change, poor WFH
| distraction management, excessive support load, etc.
|
| I'd like to think that we can assume people will be productive if
| we set them up for it.
| jameshart wrote:
| All of these metrics are like trying to measure progress on a
| building project based on the volume of noise produced.
|
| 'I don't hear hammering! There should be more hammering!'
|
| 'Lines of code' is a useful metric for 'likely ongoing
| maintenance cost'. 'Impact' is a good proxy measurement for
| 'likelihood the change introduced a bug'.
|
| If you encourage teams to increase those numbers you will get
| what you deserve.
| bawolff wrote:
| Did anyone i the last 2 decades ever think otherwise? These
| aren't just flawed but some of the most infamously flawed
| metrics.
| jrockway wrote:
| > Then there's the naming of it. Calling a metric 'Impact' sends
| a strong signal about how it should be used, particularly by
| managers. And this makes it very easy to misuse.
|
| Impact is pretty clear to me. You can find the site of an impact
| by looking for the smoke coming out of the crater.
| vannevar wrote:
| I think the real message here is not that metrics are bad, but
| that they are misused. Imagine if every time you went to the
| doctor with a fever and they took your temperature, the doctor
| prescribed an ice bath to bring your temperature down. You
| wouldn't conclude that thermometers are evil, you'd switch
| doctors. Same goes for most of the metrics here.
|
| Metrics are useful to navigate BY, not to navigate TO. If you
| have skilled and experienced managers, you can get a lot of value
| out of all of the metrics listed in the article.
| owlbite wrote:
| Same old story - as soon as you start using a metric to
| incentivize people, they optimize to the metric. If the metric is
| not well aligned with what you actually wanted (and I mean
| optimizing it is what you want, not currently correlated before
| you incentivized it), you are not going to be happy.
|
| It always amazes me how otherwise very smart people don't think
| through the consequences of "paid by the X".
| dtagames wrote:
| The most useful metric in my view, and one I learned at IBM, is
| fixes applied over shipped lines of code. Multiple fixes over the
| same lines of code is exponentially bad.
|
| IBM started measuring code defects vs working code in this way
| because productivity studies they did showed that fixes took much
| more time per LOC than new code and had other costs (customer
| sat, doc changes, reputation) besides.
| prepend wrote:
| This seems easily gamed by just producing lots of lines of
| code.
|
| Adding inline documentation would improve this metric.
|
| This does explain some absolute dogshit products IBM made as
| maybe they were optimizing for this metric by having 1000 lines
| when one would do. I'm bitter from having to decompile and
| debug websphere in the 90s and 00s.
|
| I think this runs into a problem is that programmers are good
| at minmaxing. So any rote metrics will end up being gamed
| pretty quickly.
| dtagames wrote:
| Everything was peer reviewed, so no extra code. Also, every
| single line had a comment, so no padding with comments.
|
| On the "why they failed" aspect, 100 years is a pretty good
| run. There are lots of reasons IBM is less relevant today,
| but buggy software isn't one of them.
| renewiltord wrote:
| But ultimately, the outcome is that IBM is a dinosaur corp and
| no one looks to them for technical leadership of any sort. They
| don't deliver and they're not well known for writing
| particularly good code. So that calls the value of the metric
| into question.
| wahnfrieden wrote:
| This is a variant of DORA's change failure rate
| prepend wrote:
| Does anyone actually use these as continuous variables and
| evaluate them. I've worked for 10 orgs for almost 30 years and
| while these existed, I've never even heard someone propose to use
| them to measure productivity.
|
| #commits are useful as a binary metric that a developer is alive,
| but trying to say one is more productive than another because
| they had more commits is pure madness that permeates an org so
| that I would detect it during an interview and avoid.
| skeeter2020 wrote:
| We track all of these and publish them on a continual basis
| (dashboards), with the exception that we capture deployment
| frequency not commit frequency. They're directly if weakly
| correlated. but deployments is closer to what you care about.
|
| I don't think anyone is saying you should only look at the
| metrics and not the qualitative factors (what's in all those
| frequent commits?) but they definitely help drive conversations
| and decisions. The alternative (pure qualitative & gut feeling)
| is much harder to get consistent across an entire engineering
| department.
| ftio wrote:
| I worked as a PM on internal developer productivity at Google for
| a few years. As I've said in previous comments, compared to my
| former colleagues, I'm an infant in this area, so take this with
| a heaping of salt. (Opinions my own.)
|
| I do not believe in the possibility of a "General Theory of
| Productivity," and management-by-numbers-alone is actively
| harmful, but I do believe in the possibility of measuring
| productivity in a useful way. Even "bad" metrics like _commits
| per engineer per week_ can be useful at the right granularity,
| e.g., to do high-level velocity forecasting over a large,
| representative group of engineers during different times of year.
|
| If you're wondering: different metrics are suited for different
| use cases, but as a baseline, I think the DORA metrics[1] are a
| reasonable starting point.
|
| 1. https://cloud.google.com/blog/products/devops-sre/using-
| the-...
| jqcoffey wrote:
| I'm surprised that folks are still considering metrics like LoC
| and commit frequency to measure developer productivity, even more
| so due to the (anecdotal, from my XP of 25 years in industry)
| fact that as developers gain in seniority they are typically
| spending more time with people than with code.
|
| IMHO, developer productivity is best judged by the humans they
| work with.
| skeeter2020 wrote:
| as long as you're looking at the content of frequent commits I
| think this is valuable as it encourages smaller task sizes. If
| you use a PR/MR approach it also corelates with other important
| metrics like WIP and how long the coordination work takes. LoC
| is not something I was aware people are still tracking.
| yodon wrote:
| My favorite quantitative metrics for engineering teams:
| - Avg time from code review requested to code review picked up
| - Avg time to complete code review - Avg time from eng done
| to first customer using it - Avg time from eng done to full
| production release - Fraction of tasks started that never
| reach a customer
|
| These are loosely based on the Japanese concept of Muda (waste),
| as personified in the physical logistics world via the acronym
| TIM WOOD (or TIM WOODS)[0] and are similar but not identical to
| the DORA metrics.
|
| The time to complete a code review is there (for example) not to
| focus on the amount of time actually spent performing the code
| review but to focus on all the waiting around the actual code
| review, which is typically much much longer than the time spent
| doing the review itself. It's not uncommon to see organizations
| where an engineer will submit code for review and then have to
| wait a day or more for someone to pick up their request and then
| another day or more for that other person to get around to
| reviewing it. If there are comments on the commit that need to be
| responded to, you can see additional delays. These "minor
| inefficiencies" can have huge impacts on the poor dev who is
| trying to get their code merged, and cumulatively they result in
| significant increases in feature latency, the total calendar time
| required to ship a feature.
|
| [0] https://www.shmula.com/28695-2/28695/
| kqr wrote:
| I use a similar set:
|
| - Time between idea first recorded and idea in production.
|
| - Time between idea merged to main branch and idea in
| production (in some places this is negligible, in others where
| ideas spend most of their time hanging out -- done, but
| collecting dust).
|
| - Time actively spent on task over the time between work
| started on idea and idea merged to main branch. (This is "flow
| efficiency" in lean terminology.)
|
| Other than that I use the DORA metrics too. I measure defect
| rate by looking at proportion of merge requests that fix
| defects, because mostly that is a stable distribution.
| puchatek wrote:
| Just the fact that these are team-oriented and not individual-
| oriented makes them above average valuable already
| dilyevsky wrote:
| So just rubberstamp your pr with a nit to score awesome on the
| first two?
| hatware wrote:
| You can game any system. I think the point is to give
| yourself guardrails that help you make good decisions for the
| present and future.
| dilyevsky wrote:
| Wouldn't even require bad actors - people will naturally er
| on the side of stamp and forget if you announce this metric
| as goal
| yodon wrote:
| > So just rubberstamp your pr with a nit to score awesome on
| the first two?
|
| If you're saying you can CR your own commits, that's a
| different problem to discuss unrelated to the numbers.
|
| If you're saying you're coordinating so closely with the
| other devs on your team that they're picking up your CR's
| immediately and they're able to legitimately approve them
| with just a trivial nit, then congrats - you're "gaming" the
| system in exactly the way it's designed to be "gamed." In
| general, if devs are talking to each other and aware of what
| each other is doing, efficiencies are high and latencies are
| short. If no one is talking to each other, efficiencies are
| low and latencies are high. Talking to each other and
| coordinating with each other is a good thing.
| dilyevsky wrote:
| So you're becoming "aware of what each other is doing" by
| half reading the pr title and going straight to approve
| button? I think "legitimately " is doing a lot of lifting
| here too
| yodon wrote:
| You seem to presuppose that the people involved in the
| team conversations about these metrics are stupid or
| clueless to a point of willfulness or intentionally bad
| actors. If that's the case, as with the case where you're
| somehow able to approve your own CR's, you face problems
| unrelated to the numbers. If those are the kind of
| problems you face, you need to fix them well before you
| try to bring in any processes designed to help a well
| functioning organization function better, including the
| case you called out where devs are approving code reviews
| without understanding them.
| dilyevsky wrote:
| Never said anything about approving your own prs nor bad
| actors. It's just basic human psychology especially when
| money is involved and if you're going to pretend it does
| not exist you got another thing coming
| marcosdumay wrote:
| > Fraction of tasks started that never reach a customer
|
| If you optimize this down, you punish any kind of innovative or
| ambitious task.
| lukeramsden wrote:
| Depends how you measure "reaching", i.e. all refactorings and
| optimisations all reach the customer technically "reach" the
| customer as soon as they're deployed (essentially, anything
| that meets its definition-of-done is "delivered). I think the
| wider idea is to measure work being interrupted and
| abandoned, or being abandoned due to not being necessary,
| both of which could be dubbed signs of inefficiency.
| marcosdumay wrote:
| Hum, no. I'm talking exactly about abandoned work.
|
| If you don't abandon work, you either don't take any risky
| task, or you create a really shitty environment and will
| lose effectiveness soon because of it anyway.
|
| And yes, for some software you don't actually need to take
| any risky task. At least for a while. But if you are in a
| position to set that policy, you are very likely not in a
| position to have a clear enough view to know if your
| software is such one.
| jrockway wrote:
| This is worth a read: https://apenwarr.ca/log/20171213
|
| Specifically, look at the "PM can't make up their mind"
| graphs (https://apenwarr.ca/log/20171213#slide13). That's
| the state that you're trying to avoid.
|
| I think it's fine to have tasks like "investigate the
| feasibility of X" and finding it infeasible. To me, that
| outcome is similar to delivering a feature (your
| customers will pay less for that than an actual feature,
| though, probably). What you want to avoid is "do X at any
| cost", and then the first steps are "integrate with third
| party service Y" "get a support contract for service Y"
| "do a giant refactoring to better use service Y", and
| then finding that X is actually infeasible. Now you have
| complexity, cost, and wasted months without having
| anything to give to the user.
|
| (Conflict of interest note: go ahead and do that. More
| work that never ships = more developers needed = higher
| pay for developers!)
| yodon wrote:
| In practice I generally see the abandoned work fraction
| dominated by much less interesting and much less valuable
| poor planning decisions, false starts, and misalignment
| between teams. Genuine high-value risk taking is
| typically already so rare in most organizations that it
| doesn't even show up significantly in a count like this.
| Equally importantly, when those genuine risk taking
| activities are terminated they typically happen in the
| form of an experiment that reached a customer at some
| level as part of the decision to terminate the effort. If
| it reached a customer in some manner like that it's not
| in the bucket of tasks started that never reached a
| customer.
| kqr wrote:
| These are great!
|
| How are you using the last one? I feel like fraction of tasks
| started that reach customer could easily become misleading: you
| _want_ to quickly abandon tasks once you realise they 're no
| longer viable.
|
| In fact, the development effort should be partially about
| finding reasons to stop working on the thing, so you can toss
| it out as soon as possible, instead of waiting for the customer
| to turn out not to use it.
|
| I would even say that canceling many started tasks is directly
| correlated with a quick cycle time, by Little's law.
| district487 wrote:
| not the OP, but I would use it as a leading indicator that
| there is something wrong with the way we're scoping and
| prioritizing tasks. If 100% of our tasks are no longer
| viable, it's a signal that we're doing something
| fundamentally wrong earlier in the chain, e.g. are we
| misunderstanding the user problems? the market? the wrong
| prioritization? not enough scoping done?
| yodon wrote:
| > Why are you using the last one?
|
| The last one is definitely in a different bucket for me than
| the first four. For starters, all of these are there
| primarily to encourage conversation. That said, the first
| four can make a lot more sense to try to graph and track and
| optimize. The last one tends to be more purely about driving
| a conversation.
|
| At the top level, if you're genuinely doing a ton of learning
| along the way to shipping the right feature to the customer,
| then arguably that value is ultimately reaching the customer.
| If on the other hand you can't decide what the goal is and
| you keep changing your mind (as is often the case), then you
| tend to end up with a lot of dev investment made in things
| that simply never ship.
|
| It's also worth differentiating technical "spikes" from
| feature "experiments." In my vocabulary, spikes are things
| where you're internally assessing a question like "could we
| do this" and experiments are things where you're externally
| assessing "do customers want this/does this have the impact
| we want." If you have a lot of experiments that don't reach
| customers, you're burning a lot of dev time on things that
| aren't actually experiments (because by this definition
| experiments need to reach the customer surface to deliver
| data). That's a signal you should probably be looking at.
| Spikes generally only reach the customer indirectly (through
| an eventual shipping feature), but if you have a lot of
| spikes that don't ever reach the customer in any way that's
| also a signal you should probably be looking at.
| feoren wrote:
| > - Avg time to complete code review
|
| Doesn't this depend heavily on the size of each PR? Sometimes
| it makes sense to have small changes, sometimes to have
| sweeping refactors. These would have vastly different times to
| complete a code review, for reasons unrelated to team
| productivity.
| kqr wrote:
| Have you measured this in your organisation? In my
| experience, every team has a fairly distinct cut-off point
| where if you make PRs larger than that, they start to hang
| around in someone's review queue forever.
|
| Every time I've made a larger PR I've regretted it and
| subsequently split it up into smaller ones. I get higher
| quality feedback and faster feedback to boot that way.
| dkersten wrote:
| > if you make PRs larger than that, they start to hang
| around in someone's review queue forever.
|
| Or you just get a "ship it" without more than a cursory
| glance.
| yodon wrote:
| > Doesn't [Code Review time] depend heavily on the size of
| each PR?
|
| The amount of active "eyeballs on code" time to complete a
| code review can of course vary tremendously. That said, when
| I see code reviews taking a week to complete, only very very
| very rarely is that a sign that developers are routinely
| spending a solid week of eyeballs-on-code time completing
| those code reviews. Most of the time it involves a relatively
| small amount of focused review time and a huge amount of not
| actually getting around to starting doing the code review
| time.
|
| If everything in your shop is so well handled that you're not
| wasting any time waiting for code to get reviewed, in
| practice your shop is probably also already making good
| choices about how to structure code for efficient reviews as
| well (but that's more an anecdotal observation about
| correlations, not directly an observation about causality).
| mdoms wrote:
| Large PRs are one of the most serious anti patterns in day to
| day work.
| jrockway wrote:
| Personally, if I get a small PR, I review it in about 30
| seconds. If I get a giant PR, I review it in a matter of
| hours. You are usually aware that such things are coming;
| that's why people provide daily-ish status updates and that
| sort of thing.
|
| I have also found that high review latency begets large code
| reviews. It's kind of a vicious feedback loop. If you always
| review PRs immediately, people are less afraid of splitting
| up their work, and PRs will become smaller. (It's hard to
| "stack" reviews, so people avoid it. That means that 3 PRs
| become 1 PR, so they can be mentally done with the task and
| start something brand new while waiting a week for your code
| review. The alternative is hoping that the bottom of your
| stack is approved without any requests to change anything,
| but what happens is that large changes are requested and the
| two floating PRs are now impossible to merge back in, or are
| irrelevant. If the bottom of the stack could be reviewed in
| an hour, then all that work wouldn't have been wasted, and it
| reinforces the good habit of doing 1 small thing at a time.)
|
| I look at all of this as a systemic issue, and not a personal
| issue. The team needs to set an aggressive SLA, guided by
| what the engineer is expected to do while waiting for a
| review. Then engineers need to treat the objective seriously;
| code review is as important as any code you're writing.
| That's why you're part of a team, and aren't a 1 person army
| (which is a totally viable approach; I write a ton of code in
| my free time by myself, but I don't necessarily think that
| that's how I should behave at work.)
|
| Personally, I have Github's Slack notifications on, and
| almost always start the review as soon as I get the
| notification. I know that the person waiting for the review
| has no work to do other than to build on top of this
| foundation; the less they do without feedback, the better.
| (Plus, they want to get their code Out There, that's why they
| wrote it.) I don't think your organization has to be that
| aggressive, but I'd aim closer to "hard interrupt" than
| "within 1-2 business days". (No, I don't review PRs that come
| in at 7PM on Friday night. Those wait until Monday morning.)
|
| The other metrics listed a few comments above are really a
| function of organizational policies. Make your releases easy,
| so you do them often. Make sure that your PMs know what users
| want, so you don't implement code and throw it away. Make
| sure that CI is fast, so there isn't a delay in pushing to
| production. And make sure your process is realistic. My team
| used to spend a lot of time manually testing stuff in staging
| before releasing to production. It never found any issues
| that the test suite didn't find, and we prefer the automated
| tests anyway, so eventually we just made every commit to
| master a deployment to production. Click the "merge" button
| and it's out to the users. No unusual number of surprise
| production outages, just a vastly reduced time from idea to
| being in the hands of the customer. (Code reviews ended up
| being the limiting factor. People really don't like doing
| them!)
| imoverclocked wrote:
| Just starting the review can take a while. However, engineers
| who might be reviewing code instead of other activities may
| be providing more value/higher velocity/better quality with
| those other tasks than breaking to just review a PR.
|
| Someone once argued to me that each feature PR should have
| very few surprises if the feature has been planned and
| communicated well with affected parties before implementation
| even begins. The opposite is often true in some
| organizations.
| Brian_K_White wrote:
| What do I care about start to finish time?
|
| The only way to shorten that latency would be to have so many
| employees that they can all spend all day just waiting to
| pounce immediately on their inbox. It's just the same or worse
| waste distributed differently.
|
| A developer with a full pipeline seems the most efficient to
| me. Let there be 15 projects all in different stages of
| progress.
|
| It's fundamentally not a synchronous process and I say a
| mistake to try to make it one.
|
| Edit to address a sense I'm getting from several comments at
| once, that I may be a manager who doesn't understand or care
| about developers:
|
| I AM the developer (& sysadmin) with a bunch of fresh and stale
| projects and it doesn't bother me at all. I always have work to
| do but it's not stressfull because that is about your boss(es)
| not about the number of open items. If anyone tried to say all
| that stuff needs to be done yesterday, sure that would be a
| problem. But all they really are is a spectrum of priorities.
| Some are just ideas little more than stubs for possible future
| interest, some would be nice but may never be justifiable in a
| strict easily quanifiable sense, you can't bill any customer
| for the hours, and yet would still be nice, would be one of the
| things that sets your product apart and attracts the customer
| in the first place, and some have normal priority, and when
| something high priority comes along, it simply displaces the
| rest. It just requires a boss who doesn't say that every single
| new item is always the most important.
|
| That stack of available projects, and the age of half of them,
| doesn't bother me at all. Half the time, a different project is
| as good as a walk in the park for the reset / fresh look factor
| when stuck on something.
| Brian_K_White wrote:
| I take my idle time whenever I want, either by simply not
| working right then, or working on something low priority that
| I just want to work on. The size of my pool of available
| projects has no bearing on that positive or negative.
| kqr wrote:
| This is a common conflict!
|
| One perspective says that idle engineers cost money, so you
| should load up your engineers to 99 % utilisation with a huge
| backlog of tasks. The consequence of this is that tasks take,
| on average, months to complete because they sit idle most of
| the time, not being worked on.
|
| The other perspective is that idle tasks is what's expensive.
| Thus you must keep your engineers lightly loaded (say 70 %)
| so they can, in your words, pounce on new tasks. With this
| setup, most tasks spend a negligible time waiting and most
| time actively worked on.
|
| Now, how can idle tasks be more expensive than idle
| engineers? There are whole books dedicated to explaining
| this. I'd recommend starting with Reinertsen's _Principles of
| Product Development Flow_.
| dcow wrote:
| Some developers are very much not efficient in a "fully
| pipelined" stage. Lots of context switching can slow down
| everything in the pipeline. Even if you're waiting on a code
| review and burn a few hours, it can still be more efficient
| to stay focused on driving the task at hand to completion
| than trying to spin up a small task during that time and
| switching back to respond to review comments every so often.
| lordnacho wrote:
| You need idle time as an engineer. Otherwise if a higher
| value item appears in the pipeline after a lower value item
| you'll be working on the lower value thing first.
|
| Idle time is also where you do useful things like checking
| for updated dependencies, running a linter, generally reading
| up on the tech you're using, and making the code ready for
| future edits. Not so idle at all but not really the same as
| working on a ticket either.
| the_jeremy wrote:
| Do you work somewhere that gives you this idle time? We
| explicitly have tickets for any tech debt or dependency
| updates we need to do, because otherwise we don't get time
| to do that.
| gerberthomas wrote:
| Reading the comments here, I see 2 things most of us seem to
| agree on: 1. team metrics are more useful than individual
| metrics; that makes sense because a team is an expression of a
| shared context while people come and go, so team metrics are
| inherently more valuable to the company in the long term 2. PR
| cycle time and some version of lead time (time to reach
| production) are often cited as 2 important metrics; also makes
| sense because those are the main critical, serial steps in
| software delivery
|
| Now, I would contend that the rest of the productivity metrics
| depend on what the team and its parent structures are trying to
| achieve, and should be DIFFERENT over teams and over time. Maybe
| a team with a spaghetti-like legacy project will want to track
| LOC or cyclomatic complexity for a while. Maybe some other team
| will want to track the amount of transitive dependencies. Maybe
| some will want to optimize for onboarding (time-to-10th-PR or
| something like this). 1 size will not fit all, and the team and
| its management must work to figure out what success looks like
| w.r.t productivity based on the context at hand.
| ptudan wrote:
| Amazon promos and firings are based a lot around the amount of
| lines of code you write, the number of code reviews you do (and
| the percentage of the time you review when asked), the number of
| merge requests you have, and the number of iterations per review.
| If you average more than 2 iterations per MR, you're on the
| chopping block as it means you're "sloppy".
|
| Its ridiculously dumb. I've heard those numbers matter less as
| you gain tenure and seniority. But for the new grads and lower
| level engineers, all the savvy ones were gaming these metrics. It
| sucked to work in that environment and was a big reason why I
| left.
| hammock wrote:
| My high school summer jobs was working the phone at an inbound
| call center, where there is a constant queue of customers
| calling and you answer each call in turn.
|
| The software used would spit out individual metrics like
| "number of calls taken," and our supervisor used to look at
| that metric to make sure we were working and not slacking off
| at our desks, not on the phone.
|
| People figured it out and would just pick up and hang up the
| phone ten times in a row to game the metric. In response, the
| supervisor started looking at "average call time" in addition
| to the number of calls, to make sure this wasn't happening.
|
| So what people did instead would be pick up ONE call, and leave
| the line open for an hour, long after the customer had already
| hung up, in order to game that metric.
|
| Seems like something similar could be done with these Amazon
| metrics.
| ptudan wrote:
| Oh for sure. There were tacit agreements between engineers.
| Splitting MRs into 3 for "readability", adding nits that
| explicitly "could be fixed without another iteration", having
| your personal testing repo be public so any random commits
| you dump in there count.
|
| And then on the even worse side, you have Amazon's stack
| ranking coming into play. Engineers can screw each other by
| blocking merges with needless comments, slow dripping reviews
| (eg problems that were in iteration 1 aren't brought up until
| iteration 3).
|
| I won't deny that maybe I'm a paranoid person and that I read
| into patterns that weren't there sometimes. But the fact that
| those patterns _could_ exist was enough to hurt my mental
| health to the point that I wanted to leave.
| wnolens wrote:
| Are you a manager at Amazon employing these or seeing your
| peers employ these metrics?
|
| I've never heard of such a thing. It's leadership principles
| only AFAIK.
| ptudan wrote:
| I worked there for a bit over 2 years, pretty recently. I
| heard this from other engineers and from managers not
| directly in my chain.
|
| The iterations per MR metric is one I know for a fact is used
| in PIP docs. Heard this directly from a friend that was
| PIPed.
| shawnb576 wrote:
| > I heard this from other engineers and from managers not
| directly in my chain.
|
| OK so this is hearsay, your original post did not frame it
| as such, and you walk it back further here.
|
| In addition other Amazon people seem to not agree, maybe
| don't post something like this next time.
| jerf wrote:
| I've never been seriously threatened with being measured by
| these metrics, but I have said that if I ever am, my next work
| task will be some code that takes a single commit and turns it
| into one commit per line of code changed.
|
| If anyone complains, I will take the extra time to turn it into
| one commit per character of code changed.
|
| You want commits? I can give you commits. It's not what I'd
| really like to do, but by golly, I can do it. Commits by the
| thousands, by the millions (whoops, did I just import a library
| and run my code over it? sorry), anything you like.
| buscoquadnary wrote:
| I've been thinking about doing something like that. You'd
| also want to space the commits over the day as well so you
| can complain about how you were up till 2 working on things
| and your commit history will bear it out.
| twblalock wrote:
| > Its ridiculously dumb. I've heard those numbers matter less
| as you gain tenure and seniority. But for the new grads and
| lower level engineers, all the savvy ones were gaming these
| metrics.
|
| The company care more about weeding out the really bad junior
| engineers than it does about rewarding the good ones.
|
| Honestly I don't have a big issue with that because I've seen
| what happens when bad people stick around. It's never fun for
| the good people, though.
| ptudan wrote:
| Weeding out bad engineers is necessary. But these metrics are
| usually used to justify firings on the face than actually
| using them as an evaluator. Then the directors can pat
| themselves on the back that they fired someone justly.
| koliber wrote:
| Decision making purely based on such metrics is wrong. It's
| management by numbers, and similarly like coloring by numbers,
| while relatively easy, will not produce great results.
|
| At the same time, metrics do have a place. Even flawed metrics,
| like the ones this article describes can provide value. When used
| together with qualitative evaluation and thoughtful analysis, it
| provides a more complete picture of what is going on in a team.
|
| Metrics such as these provide an addition perspective on a team.
| A manager knows what their team should be doing. A good manager
| should have an intuitive feel of what is going on. A manager
| should have a good qualitative idea of how their team is doing.
| If the metrics do not align with the other perspectives,
| something may be off.
|
| If a manager believes a person should be coding, the person is
| not bringing up any challenges, is reporting progress, and they
| produced 3 small commits over the past month, it is time for a
| conversation to find out more about what is going on.
| hinkley wrote:
| Metrics are for asking questions, not answering them.
| kqr wrote:
| Yup. Performance measurement is like planning: the outcome
| (metrics, a plan) is useless. The process you take to getting
| there (discovering, questioning, measuring, imagining,
| simulating) is everything.
| jph wrote:
| IMHO teams works best when they choose their own key performance
| indicators, and match these up with the real-world success of the
| team's users, customers, and stakeholders.
|
| These kinds of metrics can involve people (e.g. add a feature to
| increase customer satisfaction by X points), performance (e.g.
| optimize a path to increases throughput by Y%), processes (e.g.
| fix a bug so security continues to match commitment Z), etc.
| Normal_gaussian wrote:
| When leading these are the metrics you should care about:
|
| Oldest MR - this should _always_ be less than 2 weeks. This
| should normally be less than 1 week, but its not worth caring
| about at less than 2.
|
| Unfinished sprints - sprints should finish with enough time left
| over to cope for an incident in the week. The extra time should
| be used for planning and continuous professional development
| (CPD). When someone is trapped in overflowing sprints it means
| they are deprioritising and undercompleting work that will come
| back to bite you.
|
| Track other metrics for at most 3 months each, ideally only a
| month or a single week spot check. This prevents gaming and
| obsession whilst letting you reason about more nuanced
| behaviours.
| shakezula wrote:
| For fuck's sake, can we just stop trying to measure developer
| productivity like we're an assembly line?
| ok123456 wrote:
| But then how will non-technical managers justify their salary?
| ethanwillis wrote:
| I know the question is probably sarcastic. But being serious:
| Simply by being good at their jobs of helping the people they
| manage and having those people want them to be part of the
| team. And then on the flipside as well where higher ups trust
| them to help be a good translation layer that enables teams
| to meet organizational objectives.
| hinkley wrote:
| If you have to chose between two managers based on metrics,
| then you're already fucked one way or another. Either
| because you can't actually afford to lose either of them,
| or their both so awful that just asking people doesn't get
| you a good answer.
|
| I've said it before and I'll say it again: finding ways to
| characterize people as bad at their jobs is about keeping
| salaries down, whether by accident or on purpose. Because
| those are metrics your boss's skip level manager looks at.
| And they 'work' until they don't, by which point the
| manager can move up or out and get a reset on the numbers
| being used against them.
| brazzy wrote:
| So how _do_ we measure it?
| xyzzy4747 wrote:
| Imagine you own a small company (I actually do) and the
| company's money is basically your money.
|
| Now imagine who you want to pay to do stuff and how to
| measure them. The best way is to see their actual performance
| unfold before your eyes and use your own intuition. See who
| gets things done and who doesn't. Who needs more handholding
| and who can work more independently. Some workers are much
| better "deals" than others from a company perspective.
| cbozeman wrote:
| The same way you accurately measure anything... with a lot of
| hard work and attention to detail.
|
| The best physics professor I ever had assigned a fuckload of
| homework every single day of class and graded it all himself.
| He probably worked 65+ hours a week, minimum. But you got
| feedback on all homework and that gave you an understanding
| of how well you were grasping concepts and on what you needed
| to work.
| closedloop129 wrote:
| Your professor grades the homework against the solution
| that he knows.
|
| How can a manager do code review when he doesn't even have
| the capacity to know the details of the tasks?
| shakezula wrote:
| Why do we need to? No, seriously, why?
|
| I have yet to see any meaningful increase in a team's
| productivity after they start tracking "developer
| productivity".
|
| Each time it results in a blow to developer morale and a
| pretty dashboard that management uses to retroactively
| justify their decisions.
| closedloop129 wrote:
| Have they just started tracking "developer productivity" or
| do they also spend time on identifying good developers and
| enabling them to teach their skills?
|
| If you want some form of rational decision when you develop
| a team, how else but with metrics do you make decisions?
| hinkley wrote:
| Illusion of Control.
|
| It's a terrible answer, in no small part because it's true.
|
| How do you manage people when you're bad at managing people
| and can't/won't look at it? You manage safe, crisp, numbers
| instead.
| mdoms wrote:
| Because as a business we seek efficiency. And how can we
| know if we're efficient if we can't measure output? It's
| only natural to want to get the most out of your dollar
| from expensive resources like developers.
| dandare wrote:
| One thing that baffles me in corporate IT is not just the snake
| pace of development but rather the fact that nobody seems to be
| bothered by the snake pace. There is zero effort to measure or
| speed things up. The only important thing is to be nice to
| everyone, any mention of productivity is considered hostile
| behaviour.
|
| (For reference, I am talking about cases where a team of 5 devs
| takes 2-3 months to deliver a feature that would take a single
| independent developer maybe 2-3 days.)
| rightbyte wrote:
| > snake pace
|
| You mean snail or like moving in an "S"?
| Traubenfuchs wrote:
| A slow pace makes it possible to slack off more in peace. If
| you plan to take a week to do feature A and you finish it in a
| day, you have 4 free days. If you plan for 1 day and it takes 2
| because it was harder than expected, plans get messed up, you
| need to work faster on the next feature and look bad.
|
| I always encourage fellow engineers to vastly overestimate
| tickets. That's also important to set a comfortable pace with
| the business people who have zero clue how hard our work really
| is and prevent them from making us work hard.
| Copenjin wrote:
| I can't say that I've never used the first three to spot people
| with absolutely no useful output, sadly they are pretty good
| metrics for that.
___________________________________________________________________
(page generated 2022-05-16 23:02 UTC)