back
32 comments
> We need to think about measuring productivity in a holistic and multifaceted way, not in a reductionist, unidimensional way. Accordingly, we need to measure productivity using more than one metric, and we need frameworks for selecting metrics (for example, SPACE from Forsgren et al.) that enable us to understand tradeoffs.

> We must think about productivity both in the short term and the long term; for example, we need to understand the effects that biases against underrepresented groups have on developer productivity in terms of getting code submitted and also on retaining skilled employees by treating them fairly and affording them the same opportunities that others enjoy.

There's many issues with "human-centered" anything. Mostly on the idea that we "start with people", but we don't realize our own flaws as humans. Humans have finite willpower and creativity. They acknowledge that but still continue on persisting that this problem can be solved. It can't.

So we're trying to measure how much we produce when we should be measuring how efficient or effective we are with that finiteness. Again, quite a relative and impossible task.

The object isn't to make something (i.e. be productive). It is to be in that wonderful state in which making something is inevitable.

No metric or metric framework is going to help you here. There is no single proven "artist way" to do this either. Everyone has different routines and approaches to their creative finiteness.

Attrition is especially harmful at any company because you lose a flavor of creativity giving other creators perspective on how to create as budding artists. You can't really replace it easily and it takes time to mature in your talent.

Personally I think chasing an answer to these problems is a fool's errand. Many before have tried and failed. Not everything in life needs to be measured or improved. Especially something like creativity where it already is pretty much magic.

> The object isn't to make something (i.e. be productive). It is to be in that wonderful state in which making something is inevitable.

In the absence of at least some reasonable system (despite its flaws) to recognize a software engineer’s contributions, you’re likely to have a situation where the team as a whole is falling short. Managers will be under pressure to turn up the temperature on the whole team, and the environment will risk becoming more akin to a sweatshop. I’ve seen it firsthand more than once.

So I’d wager the contrary to what you suggest here. The measurements are what enable management to keep some greater distance without being a distraction. Creativity might even become inevitable if the right system is in place, but such a thing is not easily accomplished no matter the approach you try. I feel that data trumps the lack of such in this case though.

Though this is a nice topic for debate and I’d love to see some more literature or studies on it.

I've been interested in this topic lately, given the economic outlook, which in turn is pressing more and more companies to look at impact, productivity, "doing more with less"

Here's some other related links i've looked at before, but i'd be interested in gathering further research on this as well.

- SPACE framework, https://queue.acm.org/detail.cfm?id=3454124

- DORA metrics, https://cloud.google.com/blog/products/devops-sre/using-the-...

From a programmer's perspective, I'd say "if you want to go faster, release the brakes". So I think the first job of a manager is to "remove the brakes", which implies that they have to find where they are first. A measuring network could help here - and would less likely fall under Goodhart's law.
> The object isn't to make something (i.e. be productive). It is to be in that wonderful state in which making something is inevitable.

Wow, that's so aptly put. I love seeing these pithy quotable lines on HN from time to time. Thank you.

I think a good way to explain to non-engineers is look at writing software like writing a novel. How easy is it to measure the amount of good work produced by a writer? How reasonable is it to expect that any two average writers will be able to cooperate and produce a good work?

Then you can ask yourself, how would you help make an organization full of collaborative writers more productive? Would you measure their lines per day? Or would you spend your effort trying to make sure they like and understand each other?

This reminds me of the quote: "If I had more time I would have written you a shorter letter" (attributed to various people, e.g. Twain and Churchill).

In my domain (developing research software for small-scale scientific projects) more lines of code rarely correlates with more functional/readable/trustworthy software. Instead I frequently find myself thinking: If I had more time I would have written less code!

We measure books based on number of copies sold. The more books sold roughly means you're more successful than a comparative author.

If I was managing an organization full of book writers then I very well might measure number of lines per day. I've heard people discuss (on podcasts) their process to writing their book and it distills down to the commitment to write N number of lines per day until the book is complete.

> We measure books based on number of copies sold.

But is this a good metric or just Goodhart's? Let's say you're writing a technical book. How do you measure the "best" book? What is best? Number of sales just correlates and can happen just because a big school picks up a book and other schools copy. But that doesn't mean it is good.

> If I was managing an organization full of book writers then I very well might measure number of lines per day.

Please don't manage anyone. We don't need any more Dickens style books, and especially not code.

You realize books also have lines of text? Why do you think # of lines = # books sold?

Not # books sold = # of copies of software sold?

>We measure books based on number of copies sold.

Who is 'we'? That is certainly not how I measure a book.

I would be cautious in using this parallel. Many writers get paid per published word (but also get contracted to write x amount of words)
One can't measure something, much less discuss and downselect ways to measure that something, without first defining that something. This article doesn't define productivity and only implicitly addresses what productivity might loosely mean.

Productivity is often difficult to measure because no one ever defines it. It's no surprise that the ways to measure it as such are so diverse and often conflicting, because most are measuring something different.

Productivity is a highly dynamic measurable that will change depending upon the tasks. Are the tasks adding new features, performing maintenance, fixing bugs, writing documentation, or what? A naive approach to productivity is to simply ask, are the things that people agreed upon needing to be done getting done?

It seems a lot of these articles focus on software engineering alone, but I rarely, if ever, see reference to other industries. How do architects, designers, artists, professional engineers, hardware engineers, accountants, lawyers, etc. measure productivity? What makes software engineering so special?

In many years I haven't seen a better study of the issues in measuring anyone's performance than 'Measuring and Managing Performance in Organizations', Robert Austin.

One key difference is between informational metrics (e.g. using a scale to guide your own planning about diet and exercise) and motivational metrics (e.g. reporting your weight and being subject to weight loss targets associated with carrots and sticks.) Very different dynamics, most of which are harmful.

> Software Developers Are Humans

perhaps not so much, these days?

I'd still say very much yes, "car makers" are all humans even if the physical construction of the car is almost entirely done by machines - you still have people making and maintaining the machines so that becomes the action of making a car
I'm a software engineer turned software engineering manager. What I'm about to say is not gospel but a way I've found/used to measure productivity within my current and previous organization.

The number of pull requests and/or work items completed.

It's not perfect, but when the numbers are crunched over the course of 6+ months or longer then it approximately aligns with our anecdotal observations about "our best developers".

Metrics that (I believe) are important to track.

1. number of days (for a new employee) to complete first pull request.

2. number of pull requests complete within first thirty days.

3. number of pull requests complete within first six months.

4. number of kickbacks (a work item that goes back to development because it failed testing).

#4 is kind of difficult to measure. The work item systems that I have experience with do not have an easy way to track this metric so it takes effort to produce/track.

Finally, I'm transparent with my direct reports about this information and that I believe it's important. I ask them if they feel it's fair, or if there is another metric/measurement I should use or look at to assess productivity/performance. Thus far, no one has yet to bring anything serious to me so either people don't really care or they feel this is fair/sufficient (or maybe I'm a giant smug jerk who is hostile to feedback from subordinates, I don't believe this to be the case but this is Hacker News after all so very prepared for this response).

This is an interesting approach. I wonder how many EMs are currently following this approach. DO you use any tool to track these data?
Number of pull requests can be gamed and hard to measure - easy vs complex tasks. I'm surprised you did not mention anything about Velocity points.
I agree with hnrodey, and here’s the thing: trying to game this metric is a pretty good way to turn into an actually productive programmer! People who crank out lots of small, easy tasks build up a lot of the very skills they need to take on large, complex projects.

Another way to put this is, I’ve never encountered a highly productive programmer whose approach was to occasionally land a big important change. It just doesn’t work out that way.

I'm not surprised to see your criticism that is missing an alternative way to measure. The answer is not "measuring developer productivity is impossible".

Otherwise, I'd challenge you to provide a metric that cannot be gamed. Humans respond to incentives and thus there will always be people that could/will intentionally try to rise above the rest based purely on the measurement.

Measuring individual developer productivity is impossible without also creating perverse incentives that destroy business value. The smallest unit you can usefully measure is a single Scrum team (and even at that level most metrics will have wide error bars and a lack of statistical significance).
> Measuring individual developer productivity is impossible without also creating perverse incentives that destroy business value.

Probably should be printed at the top and bottom of every page of every performance review.

My alternative suggestion would be to worry less about individuals and more worry about if the organization is producing something of value. After that, you can start to pick and prod at the individual contributions if you want, but expect that any team will likely have a step function that is their effectiveness. As such, you may remove the least functioning member of a team, and then find that you dropped the entire team to nothing.

It is almost certainly true that there can be drags on the morale of a team. People that are causing things to go slower in detrimental ways. It is also almost certainly true that sometimes having someone around that is just fun to talk to for the rest of the team is enough to keep the entire team functioning.

> The number of pull requests and/or work items completed

Good idea in an environment such as an assembly line where the same tasks are repeated over and over.

Probably useless in a software development project where the tasks are different every time.

You must be soo easily fooled by low-hanging-fruit hogs
Likely weighted by (at least) story points (I hope).
This is a great discovery! Have you tried tying developer pay directly to these metrics?
Many companies have a "Human-Cent" approach to developer productivity. It is "Human-Centipede" instead of "Human-Centered"
This is cool! Measuring anything within the social sciences is hard but the act of trying to do so is, in my view, a worthy endeavor.