Why can’t Metrics just be Indicators?

Humans like to measure things. It gives people a sense of understanding, and a related feeling of control. On the topic of measuring things, I’ve been through a range of opinions: I started believing that we just needed to improve performance metrics, so they better fit with their aims; moved to a nihilistic view that the focus on targets was negative, so removing measurements would improve performance; but I now believe that what’s most important is a level beyond that – understanding what the measurement is really measuring, what any results mean and why we should care.

Firstly, though, why the spread of the culture of measurement? Organisations do this for a number of reasons: it allows them to use targets to motivate staff; as entities get bigger, it’s more difficult to see what’s going on in your business; automation allows us to measure more than ever; they are easy to show to shareholders and other stakeholders; and, if I dare to be so bold, the more you measure, the more chance of finding a metric showing a positive trend.

So what’s the problem? Primarily that these measurements can lead to more and more negative behaviours – they can drive different parts of organisations to act against each other, actions that don’t fit with overall aims, time consuming analysis and unnecessary stress.

These issues arise for reasons such as: wanting to measure the performance of the smallest possible organisational unit clashing with the need to maximise the organisation’s overall delivery (this can lead to a lack of co-ordination between areas of the business, creating messy hand-offs and reducing overall efficiency); metrics either not measuring what they’re meant to (or thinking they measure something different); the clouding of a metric’s meaning due to uncontrollable or unforeseen variables; a misdirection of incentives and/or focus; unnecessary proliferation of data; and the creation of too many or inappropriate targets.

Those kinds of considerations led me to my first standpoint – we just need to invest more time and money in measuring the right things and that’ll lead to better results. It’s true that most organisations massively under-invest in their approach to performance metrics. It’s simple to keep using the same ones, it allows an easy comparison to other years and, potentially, across organisations and it’s easy for everyone to understand. And often the longer a metric has been in existence the less we remember about the subtleties of what’s really being measures, and the more we try to flex how we use it. Ultimately, however, I could see no way to reconcile all the different motives for metrics into one cohesive model.

Bill-Walsh-300x208
Bill Walsh – the 3 time Super Bowl Champion

This led me to move towards the Bill Walsh and John Wooden philosophy that the “score will take care of itself”. This supposes that focusing on any figure will lead to negative results, even if that figure is the bottom line (in their case the final score). They suggest that the focus should be on processes rather than outputs. If all the processes are performed as well as possible, then the final result will take care of itself. It’s a powerful and empowering approach. It brings activity right down to its basic components and puts emphasis on individuals, rather than numbers. It defines success as each individual achieving as near to their potential as possible, removing ability biases that can demoralise those who are less talented (because they can never “win”) and make the more talented become lazy (because they can “win” without delivering to their potential) It’s also the approach that I would take (and do take) when coaching sport. It makes it easy to be fair, helps push people and removes pressure.

However, it can’t work everywhere – in a large organisation you can’t see every employee, so you have to rely on other people’s judgements. But that then leaves you unable to compare performance across areas because people’s standards are all different. It also works in sport because, however much you try to ignore it, there is ultimately a way to quantify how you’re doing – just look at the score. There is a clear motivation and there is quick feedback on how the team is performing. This isn’t true of a team within a larger business. Finally, it leaves you with only the most basic outputs to communicate to others – if you only have your balance sheet then it’s very difficult to explain what’s really happening.

Thinking about metrics didn’t get me too far, so I started thinking about thinking about metrics. Our workplace culture has led us to view every metric as a key performance indicator and every key performance indicator (KPI) as an output in its own right. The easiest way to justify success is to show a number that’s getting better and the clearest way to be seen as failing is to oversee a range of figures all getting worse. The more we measure the more figures that become targets and the more time we have to spend focusing on how to make them better, rather than trying to achieve real organisational goals.

We’ve forgotten that KPIs are only indicators – they show us what is happening within an organisation and help us to identify areas where we’re performing well or poorly. A change in any metric is an insight into what’s happening, but only as much as any other source of information, such as the organisation’s overall delivery or the reports that you get from your staff (let’s not pretend that numbers can be just as biased as any other sort of information). And, even worse, we increasingly believe that anything we can measure is important.

This is what ultimately leads to the negatives that I noted earlier – an over-reliance on numbers, a misunderstanding of what they reflect and a failure to explore what the numbers are showing us. To shift back to American sports, an NBA team might have their interest piqued by someone who can jump 45 inches high and run the three quarter sprint in under 3 seconds. But they’d then go and watch the player actually play.

This limits what people can achieve – they are confined to act in such a way that increases “their figures”. Not only does this mean we stop people doing what’s right, but we disengage them as well – they begin to feel a lack of trust and an inability to do what they think is best. It can also lead to us rewarding people who aren’t the most deserving – if two salespeople both pitch perfectly, but one pulls in a million pound deal, while the other can’t make a sale then we’d see the first rewarded, when really they’ve both performed identically – leading to further disengagement and encouraging inappropriate behaviours (like trying to close other people’s sales).

Some organisations are better at this than others, but I believe the vast, vast majority can improve. This can only happen by seriously looking at the figures being generated and reviewing how they’re used. This takes time and is an investment, but it enables a workforce to deliver more and leaders to make better, more informed decisions. This change has to be top-down – when you stop being assessed on non-essential metrics, then you are able to free others.

Do I believe this is an easy thing to do? No – it’s a big cultural shift and removes the illusion of having sight of all parts of a business, but I do believe it’s possible. And we still need to look for better metrics, which I’ll talk about in later posts, and maintain a focus on the process. Overall it reflects where a lot of organisational issues come from; in line with Nietzsche’s quote above, everything else starts to become more important that what we actually set out to achieve.

Advertisements

5 thoughts on “Why can’t Metrics just be Indicators?

  1. George Margesson February 12, 2015 / 3:31 pm

    Hi James,
    Interesting stuff. This is topical for me as I’m working on a KPIs project at the moment. I agree that the distinction between metrics and KPIs is significant: it seems to me that measuring too many things is only a problem if the numbers become targets, not just explanatory context (apart from the costs of collection and analyis of course). At that stage they start motivating behaviour, in both positive and negative ways.

    You’ve mainly focused here on the negatives of misuse of metrics here, but what’s your perception of the scale of the negatives relative to the positives? As you say, complex organisations need something to emulate the sporting scoreline. On one level, I feel that improving the quality of KPIs is in the realm of marginal improvements to something that is already (very) useful. But the ‘nihilistic’ position suggests there are no benefits (or at least countervailing disbenefits) to measuring things. It would be interesting to consider what the value is of having even substandard metrics, or how bad things have to be for the costs you outline to outweigh the benefits. My feeling would be that even very bad measurements would be (slightly) better than nothing, but perhaps you disagree?
    Cheers!
    George

    Like

    • James Armitage February 13, 2015 / 9:39 am

      Hi George – at the risk of sounding a bit woolly, I think it depends on the situation. Overall, I’d agree that it’s important to measure things, particularly in an organisation that’s either starting up, going through transformation or hasn’t been monitored before – in those situations it’s important to be able to act quickly to exploit or mitigate any unexpected results and having lots of information gives you more chance of spotting something. So when there is a “paradigm shift” I think it’s very important to have metrics and KPIs. Even in this situation, I still think Goodhart’s law (http://en.wikipedia.org/wiki/Goodhart%27s_law) applies – with credit due to Ian for telling me about the work on this. In summary, KPIs stop being useful when they become targets. Knowing your current environment, I think you may have an advantage in stopping this contaminating operational work – where there is lots of live data and everything is highly digitalised, it becomes difficult for this not to flow down the chain of command and become a “target” (even when those at the top don’t want it to – as middle management starts to feel judged on it).

      Where organisations are in a more marginal gains type situation, then I think metrics become less important than honing approaches and processes – if you already know the basics of what’s going on and you have an approach in place, then focusing on metrics often drives negative behaviours to improve the results because there isn’t any transformational change to justify big changes, yet people still feel the need to improve. This is the phase that sport is often in – because the rules are established and basic principles in place – which is another reason very successful teams have tried to avoid being held to account for specific numbers (including number of wins!). In that situation taking away the focus on metrics can enable the freedom to concentrate on how best to do the work. I still think that you need KPIs and further metrics at strategic levels, so you can identify change (for example, in the external environment that needs your response to change) and make informed decisions about future transformational change.

      I don’t believe that we should get rid of metrics – I think they can really help if used properly – and will write later about how we can create better metrics (including for targets, where use of variance-removing statistics, like regression analysis, makes it easier to hold people to account). More importantly, I don’t know the answer to the best way to use metrics. But, I think there is a better and more flexible way than we, as a broad generalisation, use them now and the only way to get there is for people to think about it rather than stick with the status quo. What do you think?

      Like

      • George February 27, 2015 / 12:51 pm

        Hmm. I suppose I think that even when measures are targets, they are more often than not broadly beneficial. For example, sales targets do motivate people working on commission, and only sometimes lead the the PPI situation. (Or, perhaps they act as a signal? Ie potential sales recruits are out off if they couldn’t hack the commission environment, so you end up with employees who are inherently suited to chasing that target). Admittedly that is at quite an extreme end of the scale.

        Another thought – I wonder if designing a system where only managers knew the figures would be helpful? It’s not exactly empowering the workforce but on the other hand it would limit the gaming effects you discuss.

        Like

      • James Armitage February 27, 2015 / 8:54 pm

        For some reason I can’t reply to the comment, so putting my thoughts here.

        I think firstly there is a difference between metrics that are targets and metrics which are “if…then…” targets which result in financial incentives. The second are designed primarily for motivation rather than quality of response to the metric – they are used where people aren’t intrinsically motivated by the work and you need to use money to make people do something (see my latest post for analysis on financial and intrinsic motivation – there was a bit of discussion here https://www.linkedin.com/groups/look-motivation-how-getting-paid-1992832.S.5975728650060861441?trk=groups_items_see_more-0-b-ttl).

        There I start to look at where money can drive useful things and how it can hinder useful behaviours – financial rewards make you more focused on your specific task and make you work quicker (if your initial impulse about how to do something is right), but it limits your ability to think laterally, solve problems and be creative (see here for research on the link between pay and performance – http://www.bostonfed.org/economic/wp/wp2005/wp0511.pdf).

        Anyway my concern was about Key Performance Indicators also being targets – in that situation I think they stop indicating “true” performance (as in they stop being as useful for making strategic decisions about the organisation) and start indicating the best people are able to make the stats look. It’s also then difficult to know whether improvement is due to people finding a way to manipulate stats or desirable improvement. So I think there are situations where both KPIs and targets can be useful, but I think it’s dangerous if they’re the same thing.

        That leads to what you suggest – a system where some metrics are used centrally for making strategic decisions or used locally for training, process management etc decisions (i.e. can be kept away from the front line operational staff). If it’s the right situation then targets that drive pay (and thus motivation) are useful too, but it’s worth having a look at when’s that right (again see the “bostonfed” paper above)

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s