https://leaddev.com/reporting-metrics/flawed-five-engineering-productivity-metrics Skip to main content Homepage link Homepage link * Team Back + Culture, engagement, motivation + Diversity & inclusion + Hiring, onboarding, retention + Managing distributed teams + Mentoring, coaching & feedback * Tech Back + Architecture & microservices + Building better software + Legacy, technical debt, migrations + Scaling software & systems + Technical direction & strategy * Process Back + Agile & other ways of working + Code reviews & documentation + Monitoring & observability + Productivity & Eng velocity + Technical decision making * Your Career Back + Communication & relationships + Continuous learning + Professional development + Self-care & burnout + Skills for new managers * Events Back + Conferences + LeadDev London 2022 + LeadDev London Meetup 2022 + LeadingEng London 2022 + LeadDev San Francisco 2022 + LeadingEng San Francisco 2022 + LeadDev Berlin 2022 + Group Leadership Courses + LeadDev Together + Virtual Events + Workshop: Be a brilliant people developer - September + Live broadcasts + Investing in your personal development as an engineering leader Menu Log in Sign up * Log in * Sign up Open search dialog box Search LeadDev.com [ ] Search The 'flawed five' engineering productivity metrics Reporting & metrics Scroll down [arrows-dow] [arrows-rig] Posted: 06 April 2021 * Posted in: Reporting & metrics * Written by: Abi Noda Abi Noda Abi Noda @abinoda * Share: twitter linkedin Abi Noda Abi Noda @abinoda 5 mins You have 0 further articles remaining this month. Join LeadDev.com for free to read unlimited articles. Which commonly-used metrics aren't working and why? 2022 Conferences & Events Events graphic element 2022 Calendar - Join us in London, San Francisco and Berlin this year Get unparalleled access to industry leaders. Learn from diverse voices in tech. Develop your skills as an engineering leader. When I began researching engineering metrics several years ago, one of the first metrics I looked into were commits. While working in GitHub, they provided graphs showing commit counts so it was easy to track. I knew it wasn't going to be the perfect metric, but it felt useful. For starters, if an engineer had no commits it would definitely be a red flag. If there are no commits, no code is getting shipped; on a software team, that's a problem. graphs showing commit counts See how useful this metric is already? It gives you an awesome way to see how much work is getting done. And it gets even better. You see, developers that I spoke with agree that making small, frequent commits leads to better-designed code. And so if you increase your number of commits, that will, in effect, result in smaller, more frequent commits - which is great! I thought I was onto something, so I showed it to some other CTOs and they thought it sounded kind of interesting. But then I brought it up with my dad over a family dinner - he thought it was garbage and said that no developer would ever want to be measured like this. For context, my dad had been a developer for 30 years. And throughout his career, he'd seen many situations where managers would roll out terrible metrics and anger everyone on the team. He told me that tracking commits was a horrible idea because it said nothing about the actual value of the work delivered. And if someone wanted to, they could easily game the metric by creating extra commits. Number of commits don't tell anything about the value and quality of those commits. Please don't measure yourself with this poor metric. -- Jaana Dogan yanadogan (@rakyll) February 26, 2019 I've highlighted how commits are problematic, but it's not the only metric that spells trouble. There are four others that I see a lot of companies using as a way to measure productivity. But these metrics are flawed. I call them, 'the flawed five'. They are: 1. Commits 2. Lines of code 3. Pull request count 4. Velocity points 5. 'Impact' Lines of code The number of lines of code is a metric that has been around for decades - but it's a really bad measure of productivity. For starters, there are different languages and formatting conventions that greatly vary in the number of lines of code they generate. So, three lines of code in one programming language might be exactly the same thing as nine lines in another. Additionally, any good developer knows that they can code the same stuff with huge variations in lines of code, and that refactoring code (which is good) results in less code. So not only is this metric inaccurate, but it incentivizes programming practices that are a counter to building good software. Unfortunately, lines of code is still a really common metric used in our industry. I come across companies that use it as a way of evaluating developers' contributions to their team, even determining stack ranking and terminations based on it. I think we'd all agree that this isn't good practice, but it's surprisingly common. We need to move away from this. Number of lines of code written is not a measure of your value to a project. -- Scott Hanselman (@shanselman) April 13, 2019 Pull request count Another metric that I see being used to measure productivity is pull request count. Counting pull requests seems to be a more recent trend - I was at a meetup last year and a manager said to me, 'Pull request count is the new vanity metric'. And I completely agree with him. It's not a good way of measuring how much work is getting done. Tracking the number of pull requests created or merged doesn't factor in the size, effort, or impact of that work; it tells you almost nothing other than the number of pull requests created. Like lines of code, this metric can encourage counterproductive behaviors. For example, this metric could encourage developers to unnecessarily split up their work into smaller pull requests, creating more work and noise for the team. I've seen this metric spreading like wildfire across our industry. A recent example I came across is GitLab's engineering OKRs. These are published on their website. Engineering OKRs In this OKR, their objective is to improve productivity by 60%. They intend to achieve and measure this by increasing the number of merge requests created per engineer by 20%. I don't think this is a good practice. Counting pull requests might seem less offensive than counting lines of code, but both metrics suffer from similar flaws. Velocity points Velocity points can be an unpleasant subject. I think a lot of developers see them as a necessary evil. Personally, I'm a big fan of velocity points and think that they can be an outstanding way of sizing and estimating work. However, when you try to turn velocity into a measurement of productivity, you will run into problems. Jira tracks individual "productivity" using points per Sprint, thereby destroying the careers of highly productive people who happen to be working on very hard problems. -- Allen Holub (@allenholub) August 5, 2019 When you reward people or teams based on the number of points they complete, they are incentivized to inflate their estimates in order to increase their number. When this happens, it makes the estimates and the number of points you are completing, meaningless. In essence, as you start using points to measure productivity, points become useless for their designed purpose. Impact 'Impact' is a new, proprietary metric offered by several prominent vendors in the engineering analytics space. 'Impact' is an evolved version of lines of code. It factors in things like how many different files were changes, and what number of changes were new code vs. changing existing code. All these factors are combined to calculate what is called an 'Impact' score for each developer or team. I've observed many companies that have tried this metric, and developers almost always hate it. Not only does this metric suffer from the same flaws as lines of code, but it's really difficult to understand because it's calculated using a number of factors. Then there's the naming of it. Calling a metric 'Impact' sends a strong signal about how it should be used, particularly by managers. And this makes it very easy to misuse. Do you remember the story about my dad? This is exactly the kind of stuff he was terrified about. Conclusion Identifying metrics for developer productivity is difficult. There are many things we can measure in software, but few that we should. But why is it that these five metrics are still so prevalent? Why is it that we keep using them despite their flaws? What is the definition of developer productivity? And what metrics can we use? It's time to work together to answer these questions. Go to the next featured series slide Go to the previous featured series slide Related content Four best practices for leveraging data responsibly 11 May 2022 Article Four best practices for leveraging data responsibly Hillary Nussbaum Hillary Nussbaum 6 mins Why engineering teams should stop counting Agile story points 19 Apr 2022 Article Why engineering teams should stop counting Agile story points Kevin Dallaire Kevin Dallaire 6 mins Four ways to empower your team through data 04 Apr 2022 Article Four ways to empower your team through data Hillary Nussbaum Hillary Nussbaum 5 mins Using SLOs to align business and engineering goals 30 Mar 2022 Article Using SLOs to align business and engineering goals Irving Popovetsky Irving Popovetsky 8 mins Five mistakes to avoid when setting OKRs for your team 02 Mar 2022 Article Five mistakes to avoid when setting OKRs for your team Dan Persa Dan Persa 11 mins Balancing delivery speed with engineering health 10 Feb 2022 Article Balancing delivery speed with engineering health Claire Tran Claire Tran 13 mins Getting your engineers engaged with OKRs 03 Feb 2022 Article Getting your engineers engaged with OKRs Rafael Esquicato Rafael Esquicato 9 mins Using metrics to communicate the state of your engineering org 14 Oct 2021 Article Using metrics to communicate the state of your engineering org Bryan Helmkamp Bryan Helmkamp Pat Kua Pat Kua +1 more 6 mins Engineering metrics at every level: An engineering director's guide 30 Sep 2021 Article Engineering metrics at every level: An engineering director's guide Pat Kua Pat Kua James McGill James McGill +1 more 7 mins How managers can use metrics to improve planning and run better standups 22 Sep 2021 Article How managers can use metrics to improve planning and run better standups Khan Smith Khan Smith Pat Kua Pat Kua +1 more 8 mins How to foster data-driven tech leadership 19 Aug 2021 Article How to foster data-driven tech leadership Carlos Coehlo Carlos Coelho 7 mins How Netlify, Splice, Code Climate, and more use engineering metrics 19 Aug 2021 Article How Netlify, Splice, Code Climate, and more use engineering metrics Cody Lee Cody Lee Leslie Cohn-Wein Leslie Cohn-Wein +2 more 13 mins What engineering leaders should be measuring 16 Jun 2021 Video What engineering leaders should be measuring Dalia Havens Dalia Havens Cat Swetel Cat Swetel +2 more 46mins The importance of effective status reports 01 Apr 2021 Article The importance of effective status reports Jason Wong Jason Wong 7 mins How to run a great software incident post-mortem 29 Mar 2021 Article How to run a great software incident post-mortem Katie Wilde Katie Wilde 12 mins A primer on the OpenTelemetry collector 10 Mar 2021 Article A primer on the OpenTelemetry collector Ted Young Ted Young 3 mins The health of your business depends on the health of your engineering team 09 Nov 2020 Article The health of your business depends on the health of your engineering team Hillary Nussbaum Hillary Nussbaum 3 mins Using metrics to remove bottlenecks and support your team 22 Oct 2020 Article Using metrics to remove bottlenecks and support your team Hillary Nussbaum Hillary Nussbaum 4 mins How to develop engineering metrics with people, process, and tools in mind 12 Oct 2020 Article How to develop engineering metrics with people, process, and tools in mind Dalia Havens Dalia Havens 8 mins Creating efficient, accurate, software estimations 12 Aug 2020 Video Creating efficient, accurate, software estimations Daiany Palacios Daiany Palacios Dominika Rogala Dominika Rogala +2 more 46 mins How synergy between product and engineering can solve the problem of software estimation 12 Jun 2019 Video How synergy between product and engineering can solve the problem of software estimation Jonathan Rigby Jonathan Rigby 11 mins Taking a fresh look at setting objectives and key results for your engineering team 11 Jun 2019 Video Taking a fresh look at setting objectives and key results for your engineering team Whitney O'Banner Whitney O'Banner 12 mins Plug in to LeadDev Want to get regular updates on all things LeadDev? Sign up to our mailing list [ ] Submit To find out more about how your data is handled check out our Data Promise [diagonal-purple] Pink chequered Home * About LeadDev * Community * Events * Advertise * Event partnerships * Call for Proposals * Contact us * Scholarships * Newsletter * Media Kit * (c)2022LeadDev * Data Promise * Terms * Twitter * Youtube * Linkedin * Slack scroll top