Cassandra.org
"Not everything that can be counted counts, and not everything that counts can be counted." -- Bruce Cameron
People like to put on an air of superiority when they say: "We are a data-driven company." Now, this is not at all a bad thing, and it is something you should always aspire to, but you have to be driven by the right data, not just any data. I have seen, too often, the incompetent attempt to adopt this philosophy and apply it blindly.
I have been pressed for metrics in contexts where it is hard or near impossible to measure the impact directly. These data-driven neophytes develop a desperate need for something to measure and will settle for anything, regardless of whether it is meaningful. Delivering "a" metric is their only goal. They just want a number and usually wind up with something that is tangential, indirect or just plain wrong. The applicability of the metric is not a primary concern for them.
Even if you believe that there is always something meaningful you can measure, more often than not, the effort and cost involved with getting that meaningful metric is prohibited. I have often explained how to get meaningful measurements for a project, how many people have to be involved and how long it is going to take. This is often more time that the work under consideration and almost always viewed as too complicated. Thus, for expediency, only the simpler, flawed metrics will be considered.
Trying to point out that some ill-conceived (but easy to measure) metric is not that meaningful meets with all sorts of lame rationalizations. Here's a few of them:
- "It is better than nothing." - No, that is not true. Measuring the wrong thing is harmful. If a metric is not measuring exactly the behavior you want, but a side-effect of a behavior, then you will alter the behavior to something you do not want. Try measuring a software developer by number of lines of code they create or number of bugs they fix and you will quickly have the developers doing all sorts of interesting things.
- "Do you have something better?" - Oh, it's all my problem now? And if I cannot come up with something, that gives you the right to use a bad metric? Some things cannot be measured and you sometimes have to trust people. Unfortunately, most company organizations are systems rooted in distrust.
- "This will not be the only thing we use to measure success." - Yes it will be. It always happens that these things take on a life of their own. Everyone forgets, or was not privy to all the caveats and discussions about the flaws of this metric. They see the number, they focus on the number, they forget the details.
More often than not, there is pressure from above to show progress with a number. Somehow this number is more trusted than if the person themselves just said "It was a success" or "It was worth the effort". This over-importance of a number happens despite the fact that the same, presumably untrustworthy, person has many way in which they can manipulate this number and likely no way for the higher-ups to even validate this number. There are plenty of mathematical and semantic tricks that can be used to get the metric to be the number you want to show.
"We tend to over-value the things we can measure and under-value the things we cannot." -- John Hayes