Heisenberg Principle of Developer Productivity
I fell in love with physics watching The Black Hole. It was not the greatest movie, but I was just a young kid and was thoroughly hooked. I became obsessed with how such a thing as a black hole could exist in the universe and started reading every book on the subject.
My dive into physics took me on a wild ride to the quantum side that still mystifies me today. There was the spooky science that Einstein scoffed at, thought experiments like Schrodinger’s Cat, and my favorite mind twister, the Uncertainty Principle.
While I am oversimplifying, German physicist Werner Heisenberg stated that you cannot know both the position and velocity of an object at the same time. This is because of the duality of matter which exhibits properties of a wave and particle. From our perspective, we never notice this duality. At the atomic level however, it’s a very different story where we only operate in the realm of probabilities.
The more well-known consequence of the Uncertainty Principle is that of measurement. Even the effect of measuring objects at the atomic scale causes enough disturbance that capturing one measurement makes the other impossible to pin down.
In a similar way, what we measure in the everyday world can have the effect of making what we measure meaningless. One of my call center customers shared a story that highlighted the perverse nature of incentives and metrics. In the drive to boost call center agent productivity, management changed the way targets were set to reward based on number of calls. A month into the change, they reverted back to the original system.
What happened to force management to make such a dramatic turnaround? Churn rates spiked and customer satisfaction scores tanked. Agents rushed calls or even hung up on customers to drive their call numbers up. They gamed the system because the metrics allowed them to.
Goodhart’s law is a concept that economist Charles Goodhart coined to describe the perverse nature of measurement. It is better known by this quote:
“When a measure becomes a target, it ceases to be a good measure.”
We see this idea play out in the real world. There was the NYC COMPSTAT system which helped reduced crime, but also increased the number of frivolous arrests because that is how precincts and police officers were measured. When executive pay was shifted to equity over cash in recent decades, executives focused more on short-term stock gains over longer-term goals. As I often say, the incentives drive the behavior.
Measuring developer productivity faces many of the same challenges. Lines of code or number of code commits were historical ways to measure productivity but have flaws. Are more lines of code or commits necessarily better than less? Is a 2000 line piece of code better if the same function could be performed in 200 lines? What we end up with is a metrics that rewards the production without factoring in quality.
This leads to another well-known developer metric, measuring bugs and defect rates. I will leave this cartoon strip to explain why incentives based on bugs is dangerous. Defect rates seem to be an improvement, but this fosters a culture of code protectionism and shifts the focus away from new ideas and pushing the envelope to smaller and safer ideas.
With the adoption of Agile and scrum teams, story points became a popular metric for productivity. However, this begins to look more like measuring hours worked. The focus is on completing the story points, no more and no less. Developers will work to the limit of the measure, not to the grandeur of what is possible. As a result, innovation is stifled.
The error of these metrics is that they value production over outcomes. It would be like a car company measuring number of cars produced but neglecting quality. The end result is angry customers taking their vehicles to the repair shop, which was exactly the situation that was happening in the US during the 70’s and 80’s.
How does an organization measure productivity then without unintentionally creating incentives that negatively change behavior? This is where physics provides a useful insight into a framework that gives us a better view into productivity.
If you want to measure the position and velocity of an object such that it captures the dual nature of its wave and particle features, you create a wave packet. By layering waves together, the scientist creates a wave packet with a clear wavelength in a defined area.
In our analogy, the waves in a wave packet are simply different software delivery metrics. By combining these metrics, we create an overall view that gives us a clearer view into the outcomes of developing software. The question then becomes, what are the metrics that we would use to build this view?
Two years ago, Nicole Forsgren, Jez Humble, and Gene Kim published Accelerate. You may be familiar with their work on the State of DevOps Report. Their research led them to a set of metrics that connected the work of software delivery and operations to business value:
- Lead time — time from code committed to production release
- Deployment frequency — number of deployments to production
- Mean time to restore (MTTR) — the time to restore a production service
- Change fail percentage — the rate of deployment failures
You may object that the metrics go beyond just the work of developers. That is point. The only way to get a true sense of productivity is through outcomes, and the only outcomes that should matter in driving the right behaviors are outcomes that deliver value to the business. That means starting with production code and working back to activity.
The data from the State of DevOps Reports shows a clear difference between elite performers versus medium and low performers in terms of software delivery. The best performing teams consistently hit high marks across all four metrics by keeping the focus on outcomes, not by narrow focusing on one particular activity.
It is important to keep in mind however that much like our analogy of the wave packet, there is built-in uncertainty. This is a feature, not a bug! You can get a more accurate measure of position, but that increases the uncertainty of velocity, and vice versa. In the same way, what the business prioritizes will have an impact on other metrics. The benefit though is that if we are tracking the four key metrics, we can see what that variability is.
How do you go from these key metrics however to things that directly impact productivity specifically for developers? There are a whole set of other levers that I will discuss in the next post to dive deeper into approaches into development tying activity at the developer level to the four key metrics mentioned here.
How are you measuring developer and team productivity today? If you were to use the four key metrics, where do you feel you would be strongest/weakest in?
Episode #8 — Building a high-performing engineering culture with Jason Milkins
Episode #8 - Building a high-performing culture of excellence in engineering Co-hosts Justin Arbuckle and Mark Birch…
Revisiting past episodes before our next season, this one is with Jason Milkins to share his experience and thoughts about building a high-performing culture of excellence in engineering.
Wondering what is up with the HERETECHS podcast? Season Two is coming up next week and we are very excited to feature an awesome tech startup founder!!! Also if you want to be a guest on the podcast and have unique thoughts on the state of IT and Engineering, we would love to have you on the show!
We help IT leaders in enterprises solve the cultural challenges involved in digital transformation and move towards a community based culture that delivers innovation and customer value faster. Learn more about our work here.