Agile Metrics

Someone on the Agile-Testing list asked about test metrics for his SEI/CMM compliance effort. Here's my reply:

I have one graph, which is a stacked-line graph. On the X axis I have time. On the Y axis I have deliverables.

Each deliverable has phases - need requirements, in dev, in software engineering test, in customer acceptance test, waiting for prod, and in production.

I update the chart every monday. Of course, I am an agile guy, so I dev as I test, so spending a lot of time in SE test tells me something. (If I move to SE test on tuesday and promote to CA test the next day, it never shows up on the spreadsheet. That's good.)

Looking at this sheet, I should see the size of the delivered features go up regularly. Now that is what I care about. It also shows the relative size of the work-in-progress inventory.

It helps the devs. The testers. The requirements people ... and it's holistic.

Now, if I see things start to stack up at a specific point, (especially testing), I know something is going on, and I put more effort in eliminating the bottleneck; that's basic constraint theory.

Of course, it's a first-order approximation. Some deliverables are done in a few days, others are a few weeks. I could weight them, but that seems like 'good enough' for now. Of course, the graph tells a story, so I show it to people in context only.

That has nothing to do with the CMM(I) - it's what I actually do to make my life easier. It has pros and cons, but it gives me and my management visibility into what I am producing, and opportunities for feedback.

As for the CMM(I):

I just spent a considerable amount of time reviewing the CMM(I) Integrated Version 1.1 for Systems Engineering and Software Engineering, looking for a tie between metrics and testing.

I couldn't find it. Of course, the thing is so poorly written that it's probably in there.

The one thing I did find was that for level 4, it could be argued that you need to measure your adherance to the defined process. ("Quantitative Project Management"). Now that is relatively easy, and counting test cases doesn't help you get there, unless you have a standard policy of X test cases per 100 lines of code, or something like that.

I don't know your environment, but in mine, I would want a CMM(I) assessor who believed that our environment changes so rapidly that common approaches to test metrics would be nieve and premature, and that we could get all of the level 4 goals accomplished without them. (Ref: Handbook of SQA, 3rd ed, Schulmeyer/McManus)

But, to be honest, I have fundamental problems with the CMMI. I suspect that you might be better off reading "Quality is Free" for yourself.

So please take this with a grain of salt. I did a best-effort attempt at a answering your question, but my head hurts now.

Good Luck.