But only if you're measuring the right things
What Do You Measure?
‘Measure what matters’ is advice that you hear again and again in businesses of any size. The truth is, however, when it comes to software development that often transmutes into “Measure what’s easy.”
We spend a lot of time measuring the wrong things. Or, to put it another way, we spend time measuring things that really don’t matter.
Outcomes matter. Customer experience matters. Outputs are easier to measure. Guess which one organisations usually focus on?
There are two key questions we should always be asking ourselves from the outset of any project:
- What are we trying to achieve?
- How will we know if we’re successful?
If we can’t frame the answers to those questions in terms of actionable results or tangible customer benefits, then we are thinking in outputs, not outcomes. This is one of the reasons it is important to frame hypotheses and expected results when considering potential areas for investment.
Making Sense
In their book Sense & Respond, Jeff Gothelf and Josh Seiden tell the story of how they worked with Taproot to deliver a new system for matching volunteers to organisations. Taproot wanted a list of features, but they eventually agreed to a contract where delivery was framed around outcomes. As a result, the development team were able to build a prototype quickly and learn what features were actually required. Some of the assumptions they had proved to be incorrect when tested, and they saved significant time and effort not building unnecessary features. This enabled them to deliver to a scalable solution that “far exceeded the performance goals written into the contract.”
The authors note that it’s unusual for a company to reach this sort of arrangement with an external service provider. I believe it’s almost as unusual within organisations that build their own software. Stakeholders often want tangible results in the form of features delivered so that they can demonstrate progress to the business. Unfortunately, this doesn’t always translate into successfully solving customer problems.
OKRs to the Rescue?
Objectives and Key Results (OKRs) enable organisations to frame their desired outcomes (objectives) and to state the measures or metrics they will use to assess the success of any initiative. Used well, OKRs can enhance the work of software development teams, enabling them to take a data-informed approach. A key step in being successful in using OKRs or any other way of assessing organisational performance, is understanding what makes a good metric.
What Makes A Good Metric?
In their book Lean Analytics, Alistair Croll and Benjamin Yoskovitz describe the key features of a good metric.
- Comparability: it must demonstrate progress, e.g. a 5% increase in conversions since last month is more useful than 2% conversion
- Understandability: a metric must be easily understandable by everyone in the organisation
- Ratios: the most useful metrics are ratios rather than numerical.
- Behavioural: a good ratio tracks customer behaviour but also will challenge the organisation’s behaviour
For me, organisational metrics must be based in customer outcomes and should be stated as ratios. It’s important to watch out for vanity metrics (e.g. number of clicks) which may make you feel good about yourself, but don’t offer any insight into customer behaviour or action you can take which will impact it. A related symptom of vanity metrics is overhyping internal measures of success (e.g. revenue) at the expense of customer outcomes.
Vanity metrics being emphasised as the highest-order problems detract from the mission of the company. A loss of perspective on who your customer is and what they need can lead to an organisation losing its way and ultimately failing, as happened to crowdsourcing innovation startup Quirky in 2015.
In the same book, Croll and Yoskovitz also introduce the concept of the One Metric That Matters (OMTM). What’s the one thing that you can measure that will tell you that you are on track to achieve your desired outcome? The discipline of being guided by what’s essential is important here. There are many things you can track. Recording and reviewing different measures is easier than ever with developer tooling and analytics software. But what really moves the needle?
There Can’t Be Only One?
I’m not convinced there is only one metric that matters in all situations. For example in a two-sided marketplace, you’re likely to need a metric for each side of the market. Equally, there are other businesses that may need to consider more than one outcome per product area. Don’t get caught up in there only being one, but you should value an extremely low number of metrics above all others. I would say three is the absolute limit. This isn’t to say that you should only measure three things, but you need to make a decision in advance about what few things really matter to you. What change in customer behaviour do you need to drive, and how do you know it’s been accomplished?
For example, I once worked on a product which had consistent traffic, but only a 9% conversion rate. In this case, conversion was considered to be a customer creating an account AND taking further action. By focusing on removing frictions in the account creation flow, the team increased the conversion rate to over 25% and it stayed there, even as the number of unique visitors didn’t change a great deal over time. If we had focused on increasing site visits, we would never have solved for adoption.
What About Internal Measures?
Nobody would argue that you shouldn’t also measure what happens in product development. There are key metrics you can use as an organisation that will give you insights into the health of your product development process. You absolutely should care about developer productivity, but prizing predictability, features and outputs over delivery, solutions and impacts is misguided. The metrics that matter are about what drives customer experience and actions we can take to improve it.