Goal Question Metric - The Yellow Brick Road?

I posted this on the software-testing email group yesterday.

The replies have been fascinating; but I'm curious what you think:

Hello Folks.

Many people here have heard my own life stories about programming and testing; how, essentially, I kept getting "patted on the head' and told that I "Didn't Get It" because I opposed big extensible designs, rituals, signoffs and handoffs in the development process, and expensive, heavyweight test case programs.

I stopped worrying about it when I realized that my projects were far more successful than my peers. Eventually, I started talking about it openly.

Metrics are currently on that list. I have in my study the handbook of software quality assurance, 3rd edition, that contains a list of about 150 qualities (like scalability, security, etc) that can be measured. Then it tells you that one or two metrics will cause dysfunction, you need a balanced scorcard. And that the easy-to-gather metrics are also easy to game and bad, but that the good metrics are expensive to measure. Oh, and be careful, because your engineering staff will rebel if they have to spend too much time gathering metrics instead of doing work.

To summarize, this is what the book has to say about metrics:

"Good Luck."


Which brings me to my next sacred cow: Goal Question Metric.

GQM is a framework written by Victor Basili; you can google it. The basic idea is that instead of gathering a bunch of metrics, you actually figure out your goal (like "faster production"), ask a question that will help measure that goal, and turn that into a metric.

I have to grant that this is an intellectually valid framework, and it beats the pants off of mindless gathering of numbers. For software testing, the idea has been endorsed by people I respect like Cem Kaner and Lee Copleand.

Here's my problem: This idea has been around for a long time. When It comes to software testing, I've read a great deal of the literature, been to the conferences, read a lot of blogs.

Except for a few examples from people like Lee Copeland and James Bach here's what I always see: "If you want metrics, use GQM. Since all contexts are different, I can't give you an example."

pschaw. Is it too much for me to ask for a case study before I invest time, energy, and effort into a metrics program? One with positive ROI? Enough positive ROI that I wouldn't be better off working on other projects, or sticking the money I would have spent in a CD?


It's been 13 years since the first GQM paper was published. I haven't seen GQM provide it's value in a software testing context.(*)

Have you? I would be really interested in success stories, please.

Regards,


--heusser
(*) - Please don't say NASA. They work under an entirely different set of constraints than commercial software development. And even then, the business case is shaky.


UPDATE: Dr. Kaner replied that he doesn't really 'endorse' GQM as much as he simply mentions it during talks. His overall comments are along the lines of "GQM looks interesting, it's more grounded than nothing - if it works for you, good for you."