In a previous post, I discussed the importance of measurement in growing software companies. The core argument: while technical debt slows you down, measurement debt may send you in the wrong direction entirely.
Okay, so just measure everything, and you're good to go, right?
Not quite. Measurement has pitfalls, and there are legitimate reasons why a company may not measure. One of the most important: measurement often captures what's easy to measure, not what's important.
This is the drunkard's search principle:
A drunk man is searching for his keys under a streetlight when a police officer walks up and starts helping him look. After searching for a while, the officer asks, "Are you sure you lost them here?""No," says the drunk man, "I lost them in the park.""Then why are we looking here?" asks the officer."Because this is where the light is!"
Organizations can fall into a drunkard's search in several ways:
Optimizing page load time while users struggle with confusing workflows and poor information architecture
Celebrating record daily active user numbers while customers can't achieve their core goals in the product
Achieving perfect security compliance scores a month before a major data breach
Driving app store ratings to 5 stars through aggressive prompts while their core user experience deteriorates
Notice the common thread: getting caught up in easy-to-track metrics instead of focusing on the actual value users get from your products. Quality, maintainability, and user experience often suffer in the rush to optimize measurable numbers.
Does this mean measurement isn't worth it? Far from it. These pitfalls exist, and are often worse, without measurement. Without data, teams can easily convince themselves they're succeeding while missing major problems. At least with metrics, you may notice when improvements in one area correlate with degradation in others.
Sometimes simple metrics are exactly what you need. If your site takes 30 seconds to load, that page load time metric is probably the right thing to focus on. But it shouldn't be the only thing you measure, and it definitely shouldn't be the only thing you care about.
The key is keeping sight of what actually matters to your users and your business. Ask yourself:
Does this metric reflect real user value, or just what's convenient to measure?
What important aspects of user experience might we be missing?
Are we measuring leading indicators of problems, or just trailing indicators of failure?
Even when we can't measure the most important things directly, we can often piece together understanding through multiple indirect measurements. I'll talk about strategies for doing this effectively in my next post.
Have your metrics ever led you astray? What were you measuring, and how did it diverge from the value created? Hit reply and let me know - I read and respond to every email personally.