Schedule and Events

March 26-29, 2012, Software Test Professionals Conference, New Orleans
July, 14-15, 2012 - Test Coach Camp, San Jose, California
July, 16-18, 2012 - Conference for the Association for Software Testing (CAST 2012), San Jose, California
August 2012+ - At Liberty; available. Contact me by email:

Wednesday, September 09, 2009

Test Management Tools

Allright folks, I'll admit it.

I'm not excited about test management tools.

Oh, you could argue that I should be. After all, Test Management tools are purchased by test managers and executives. Test managers and executives have money; they control the budget and decide who goes to what training when. Finding someone's pain point - and taking the pain away - is a perfectly legitimate business strategy. (If they have money to spend, why, that's even better, right?)

Yet I'm still not excited. Why?

Well, let's take a frank look at the thinking behind a test management tool, by which I mean something specific: A keeper of 'test cases', and a tracker of which test cases have been run against which codebase.

It starts with this thinking:

(A) We can define all our 'test cases' up front,
(B) When those test cases pass, our codebase is 'good' (Or, alternatively, when some fail but some decision maker desices to ship anyway),
(C) /Recording/ which test cases have run, and which are yet to run, in precise detail, has some value in and of itself

I reject the premise behind all of these arguments.

Here's an alternative, that we use at Socialtext:

1) Create a single wiki (version, editable web page) page for a release
2) Mark down each type of testing you want to do in every significant combination
3) For example, break the app by major piece of functionality, then further by browser type
4) Add all the automated suites or unit-test results if those matter
5) Have the technical staff 'sign up' for which pieces they will test
6) When testing on a component is completed, the tester writes 'ok' and the bug numbers he found, or, perhaps 'skipped' and the reason why.

For what it's worth, we've been doing this at Socialtext for nearly two years, since before I was hired. We are constantly tweaking the process.

This one-page overview is a higher-level view than a test management tool might provide. It shows you what matters - the failures - not 5,000 "ok" results. It assumes that the test ideas are located somewhere else that the test can find if needed. It assumes the tester actually did the testing and leaves open the possibility that the tester can explore the functionality. It leaves the tester responsible for what 'ok' means, instead of a spreadsheet or document.

This isn't a brand new idea; James Bach recommended something similar in 1999, called a "low tech testing dashboard", only he suggested it be done on a whiteboard. Other people have suggested using a spreadsheet, but that has version and read/write problems.

A wiki is just one more step forward; it provides version control, transparency, and creates a permanent artifact that could be audited. In my mind, this provides some of the benefits of test management tools with much less time investment.

So no, I'm not excited about most test management tools on the market. In many cases, I am suggesting they swat a fly with a sledgehammer. Yet I recognize that test managers and executives have legitimate problems. So let's not rush off to build something to get money; let's come up with real solutions and see if the money flows from there.

Who's with me?


Joe Harter said...

I'm with you! We use Quality Center at my company and I think that the tool has contributed to us doing worse testing, not better because we adjust our processes to make use of the tool instead of using the "best" process.

I call this "Death by Quality Center", and I'm working on a post about it. I'll be sure to link back to this post.

Rick said...

When the test team at my current place went looking for a test management tool, I was kind of perplexed, because I didn't see the need for it. It's ultimately up to them to pick what they prefer (after all, they'll be the ones using it) but as for myself, I'd rather just keep test cases in a wiki or under a version control system, as appropriate. When a release comes along, it's easy to scrape up a quick report, and that avenue leaves a lot more room for results like "This succeeded, but..." or "There's a (trivial|major|massive) problem here" instead of just a binary PASS/FAIL.

Gil Bloom said...

You are right in the sense that many test management tools are an 'overkill'. So what if you find a tool that isn't, is very easy to use (intuitive, no-training required, 5 minutes from download to start), does only what it should as a test management tool and its costs are (very) small? wouldn't you re-consider?
Of course I'm not objective :-)

Justin Rohrman said...

I think that this is the right path to facilitating thoughtful manual testing and reporting on the testing, but it also implies / requires a large amount of trust in the expertise of the testers by the people seeking information on the current state of the product as well as a good understanding of metrics and how they are used.

The metrics derived from a test management tool are often used into deluding people into a false sense of comfort that stifles further inspection of what is really going on. The trust aspect will be required in the sense that the testers will now be reporting on the state of the product to management where as before the numbers could be derived 'objectively' from the tool.

Parimala Hariprasad said...

One of the reasons why Test managers and executives stress on Test Management Tools is that it facilitates easy reporting of metrics to them. In other words, it makes their job easy to share these meaningless metrics with the higher management. Instead of worrying about what the critical issues are, should we ship the product with these critical defects(which are of course 'FREE') etc, they are more interested in the numbers. How many test cases executed, how many passed, how many failed, how many blocked, how many bugs filed and so on. These Test managers and executives fail to ask 'How good is the quality of the product to ship?'

Parimala Shankaraiah

Joel Montvelisky said...

A tool is only a tool, and a fool with a tool is still a fool... But in the same way that not all men are fools or are created equal so not all Organizations, Processes or Tools are the same.

Many teams work great with a simple Excel Spreadsheet and they should continue doing so for as long as they can. But many others, as they grow and evolve, start running into order, communication and coordination issues that challenge them to look for a more structured approach and a process-oriented tool.

I love the phrase "swatting a fly with a sledgehammer" (I personally prefer "using a Bazooka to kill a fly") since it perfectly describes when someone uses an overkill process, tool or approach to solve a simple problem. We should strive for the simpler solution to solve our current problems.

There are many tools out there that learned from the mistakes of over-killing processes with QualityCenter and the like. We believe we have such a solution if someone is interested in checking it out -

Rikard Edgren said...

I think your "starts with this thinking" for test management tools isn't too benevolent.
A) you can enter test cases or test ideas after the test has been executed in a test management tool
B) Pass/Fail results can be a start of an analysis that cares about details
C) Knowing which test have been run might not have great value in itself, but it surely can help you decide which tests to run the next time. A good test management tool makes it easy to re-run tests that failed (or has a Re-Run flag), and helps the next executor by showing the Notes from the last executors.

Excel is a good tool, but when you have released many versions, and want to re-use test ideas fast, it is cumbersome.

On the other hand, I haven't yet seen a really good test management system.
Maybe it is because "Test Management tools are purchased by test managers and executives", so the main focus isn't on helping the testers.
Or maybe the usage between different comanies is so different that it is too difficult to create a generic, good system.

Anonymous said...

I recently came accross your blog and have been reading along. I thought I would leave my first comment. I dont know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.


Craig Brown said...

Here here.

We've just been 'mandated' to use a bunch of Microfocus (formerly Compuware) tools, which are full of features, but have had the effect of optimising test management practices at the cost of imparing the whole project system. Sheesh!

Dave said...

In my opinion, a test management tool should never drive the testing process. Tools are there to help and aid, thus as IT guru's, we have to figure out how to best incorporate them. A good test tool replaces many small tools, thus helping to ensure testing task do not fall through the cracks. Quality Center, for instance allows "one stop" testing. You have a built in Requirements Tracebility Matrix, test case designer, execution platform and defect tracking. At my current company, we also utilized a WIKI page for reporting testing status. But guess what, someone had to update it, after testing was complete in the testing tool. Now I know you are thinking, well, that may take 20 minutes to update daily, but that translates into over 1 1/2 hour weekly, or roughly 6 hours monthly, which approaches a full work day. Also, when followign a strict test process or CMMI process, you need concrete results reporting. Test tools can also control the integrity of the test bed, denying users the ability to make modifications. Personally, I love test tools, I JUST NOT SOLD ON EVERYONE HAVING THE CAPABILITY TO IMPLEMENT THEM CORRECTLY.

Project Management Team said...

Really like the useful information about software testing. It is very popular now a days. Test Management tools control the budget and accomplish the goal of the organization.

Nathaniel @ project management test said...

Great post! You have given a great alternative!

And I am sure I'm with you! Besides, the quality of the output of our projects is way much better than gaining money. Because by the time your work is great, money will just flow endlessly, right?

Thanks for the post!

gal said...

I am 100% agree with you!
Most of the test management tools are an 'overkill'.
We are focusing on startups and
Our vision is to provide a simple, intuitive and very low cost test management and ALM solutions

Feel free to try it for free and in case you are a startup or open source projects feel free to use it for free

seva said...


I read this post two times.

I like it so much, please try to keep posting.

Let me introduce other material that may be good for our community.

Source: Time management tools

Best regards