Do your IT KPIs Reflect Your Business Needs?
One of the questions we always try to answer early is: “What does success look like?” But we don’t usually stop there. We also try to understand why that metric defines success.
Thinking carefully about what you’re measuring is vital to achieving what you’re intending to achieve.
The Freakonomics podcast touched on this recently. Looking at medical treatments, they found many of the approved medications were very effective at treating one of the core metrics of diabetes: Blood sugar levels. But a deeper dive into the studies showed that the studies really stopped there.
“All of those drugs have been approved onto the market with only evidence showing that they improve your blood sugar. But none of them have got evidence showing that they reduce your risk of heart attack or renal failure or eye problems or any of the actual, real stuff that patients with diabetes care about,” said Dr. Ben Goldacre.
Freakanomics Podcast
And Goldacre found that the correlation between blood sugar levels and the issues weren’t perfect. So you can get a false positive of effectiveness by choosing this KPI.
Now let’s unpack this a little bit from an IT setting.
You roll out a secure file sharing system to solve the issue of compliance violations of sharing sensitive data in a way that violates specified rules and regulations. The goal you decide to measure is adoption rate: The number of people who are using the new file system.
And you’ve succeeded: 95% adoption! You feel good about that, right?
But a deeper dive reveals everyone uses it when they have a large file that won’t fit in corporate e-mail. Sensitive data, however, is still regularly shared over e-mail.
You had the wrong KPI.
So play devil’s advocate when setting up KPIs. Don’t simply ask if that KPI will capture the behavior you’re trying to measure. Ask how that measure could fall short and give a false positive.