Meaningful Metrics

Recently, a few people have pointed to me as completely opposed to metrics in software development or testing.

I wouldn't say completely opposed - for example, when the technical people gather their own metrics in order to understand what is going on and improve, really good things can happen.

No, I would say concerned about the use of simplistic measures that fail to measure the entire scope of work, or or act in the place of some harder to measure thing(*), or lack construct validity(**).

Most of the ardent fans of software metrics like to quote Tom DeMacro, and his book Controlling Software Projects. For example "You can't control what you can't measure." Now, for years, I've been confused by these quotes - as DeMarco spent the second half of his career writing books that refute the premise behind that quote, such as Peopleware, The Deadline, Waltzing With Bears, Adrenaline Junkies. He even titled one of them Slack. No, I'm serious.

So I always found the copious use of that one quote a little unsettling.

Well, folks, there's good news. Twenty years after he wrote "You can't control what you can't measure", Tom DeMarco just wrote a column for IEEE software explaining his current thoughts. Here's a quote:

"My early metrics book, Controlling Software Projects played a role in the way many budding software engineers quantified work and planned their projects. In my reflective mood, I'm wondering, was its advice correct at the time, is it still relevant, and do I still believe that metrics are a must for any successful development effort? My answers are no, no, and no."

Now, DeMarco isn't saying that Metrics and Control are /bad/, as much as they may not be the brass ring we should be striving for in software work.

So what should we be striving for? DeMarco appeals to us to shoot for the big idea - the multi-billion dollar concept, where if you blow your budget by 200% it still changes the world and enables you to retire.

Ok, fair enough; that's a decent chunk of the focus of this blog. But let's go back for a moment. What metrics do I like?

Well, for a software company, I do like revenue and expenses as metrics. They are real, hard things that have construct validity, they are not in the place of something else, and without them you go out of business. But that's not the only thing I like. What if we measured the number of heart attacks and divorces our teams experienced, and expected them to go down?

Now, you might have concerns about that for legal or PC reasons (discriminating against divorced people) - but I think it's really interesting as a thought experiment - as the idea that the whole person matters. And at least one company has done it.

Bravo, Obtiva. Bravo.


--heusser
(*) - Classic example as lines of code used to approximate productivity
(**) - Not all "test cases" are created equal

UPDATE: I just have a conversation with Markus Gaertner, a german collegue I work with. He had never experienced this desire for metrics to evaluate and control that I discussed above. We talked for some time about dysfunction and he did a blog post on it. My post certainly assumed a few bits of shared understanding about metrics - and I could be wrong about my assumptions. If you don't "get it" either, let me know in comments, and I can expand.