(Taken From a recent post fo the software-testing email list)
It always amazes me when strong people come out and say publicly something that I have been mulling on for a few months.
James Bach's comments on sapient processes do that for me. Let me give you the back story ...
I do a lot of developer-facing test automation. That's things like asserting the return values of a F-to-C temperture conversion function.
This has a lot of value to developers. It provides a functional spec to the maintenance programmer. It allows us (I am a tester/dev) to change the internals of the code and have some confidence that we didn't break anything. It allows the developer to experience the API and change it to be easier to use.
Just about all of those are benefits to the developer - not the end customer. For the most part, they are design, maintenance, and software engineering benefits. Increasingly, I am wary of test automation that promises to "prove that the software works." I find that on every release, the things that I need check are different - they are the things that have changed. Thus, the old test suite has less and less value as we add new functionality - and providing "hooks" and automation can be very expensive.
I think that Brian Marick's Recent Comments echo this thread. Customer-Facing test automation isn't all that great at, well, finding bugs, or proving that the software works.
Next month, I'll be out in New York, trying to make this (and similar) points at the Google Test Automation Conference. Wish me luck ...
Tuesday, July 10, 2007
Subscribe to:
Post Comments (Atom)
2 comments:
"Customer-Facing test automation isn't all that great at, well, finding bugs, or proving that the software works."
I take you to mean what I'd refer to as a regression test. A test of mostly the old functionality that maybe did not change much.
So yeah. I have to admit there's a low likelihood of finding bugs with this. What nags at me though about this idea is that I still feel like regression tests are pretty important. We want them *not* to find bugs after all. We want to know we did not break anything inadvertently, indirectly, by what we did change.
This is a point I've been mulling on myself for a long time, years maybe. So now I have to ask myself "why?" Why would it be so bad if we broke something inadvertently, indirectly, that we did not change?
Focused on customer usage as I am, I answer myself that if it's something people don't really use much, maybe it's not so bad. If it is something used often, other tests will find it right? They better, or we have a lot of crappy, useless tests.
So maybe it's just the fear of an unknown bug lurking out there that we could have easily found with a regression test, or so we believe.
Fear countered with belief. Like that's going to help.
Weighing all this against the great effort of maintaining *automated* regression tests... Yeah, maybe it's not really worth it...
Anyway, I'm not sure I'm done mulling this one yet.
-Charlie Audritsh
I hope you dont mind me adding to this blog but i think you have identified one of the major industry challenges. I often describe it as the change/risk triangle – a nice stable base, automation possible down here. Higher up, less volume, more changes to the applciation, and thus more risk, and the testing is difficult to automate. So automation is usually only successfully directed at the base of the triangle - the area of lowest risk.
However, I suppose that just because it is low risk, does not mean it is unimportant and hence is still worth testing to ensure nothing unsuspected has occurred. The consequences of an error anywhere on a customer facing application or website can be massive, so as testers we need to maximize coverage and minmize risk – If automation can do it for you, then why not?
That leads on to the second point – automation is great if it works, or works at a reasonable cost (for cost read effort). Too often the cost is high in people skills, in creating tests and in maintaining them. Re-doing scripts everytime there is a change to the app can be a real pain.
The recent new trend for automation solutions with more built-in intelligence (avoiding scripting languages and employing scripts that can update themselves after application change) are promising and have enabled the automation base of the change/risk triangle to be made deeper by making automation more cost effective more more parts of the application....
You only have to have been the “buck stops here” project manager on the sleepless go-live eve of a business critical change to know that fingers-crossed feeling of “I wish we had had more time for testing”.
Post a Comment