Schedule and Events

March 26-29, 2012, Software Test Professionals Conference, New Orleans
July, 14-15, 2012 - Test Coach Camp, San Jose, California
July, 16-18, 2012 - Conference for the Association for Software Testing (CAST 2012), San Jose, California
August 2012+ - At Liberty; available. Contact me by email:

Wednesday, October 08, 2008

Complete test automation is the Holy Grail?

Recently, Phil Kirkham mentioned a comment he'd heard that he was puzzling over:

"Of course, complete test automation is the Holy Grail of software development."

The speaker was talking about interactive (user-driven) systems, and probably meant that automated test execution (do-this-compare-to-that), giving complete confidence in the push of a button, was this thing people searched for that had the magical ability to solve all your problems.

As I am a bit of an Authurian Legend buff, this intrigued me.

But what is the Holy Grail, really? WikiPedia says it is the cup (or maybe plate) used at the last supper - the one dipped in by both Christ and Judas. Or maybe it was a cup that held Jesus's blood. In whatever case, it's magical, and can heal people. Maybe. We think. Sorta.

As a Catholic, I do believe in the possibility of relics and sacred tradition, so I looked it up on Catholic Encyclopedia. You can read the entire article here, but just check out the summary:

A word as to the attitude of the Church towards the legend. It would seem that a legend so distinctively Christian would find favour with the Church. Yet this was not the case. Excepting Helinandus, clerical writers do not mention the Grail, and the Church ignored the legend completely. After all, the legend contained the elements of which the Church could not approve. Its sources are in apocryphal, not in canonical, scripture, and the claims of sanctity made for the Grail were refuted by their very extravagance. Moreover, the legend claimed for the Church in Britain an origin well nigh as illustrious as that of the Church of Rome, and independent of Rome. It was thus calculated to encourage and to foster any separatist tendencies that might exist in Britain. As we have seen, the whole tradition concerning the Grail is of late origin and on many points at variance with historical truth.

Let me put that in testing terms:

"As we have seen, the whole tradition concerning the test automation legend is of late origin and on many points at variance with historical truth."

In other words, the Holy Grail is is the stuff of fairy tales, said to have mystical powers but never actually seen by anyone. King Arthur's legend was an interesting story from my youth, a fun little adventure to pretend as a child - but when I became a man, I put away childish things.

Perhaps you could say that test automation is the Holy Grail of software development, after all.


AgileTester said...

Clearly software test automation has benefits, but (as you hinted) by itself it is not the answer. I like that it enabled quicker feedback regarding regressions and side-effects from one newly implemented function on another, but I assert that any planned test will only catch the bugs you plan to find. I'm a firm believer in using exploratory testing as a method for supplementing any planned tests. This allows the tester to use his/her tests as feedback into what next areas to explore, and in my practice, has found more bugs and found them faster than any other method. By testing new features this way, I learn more about the software domain and can provide more valuable feedback to developers and product owners in a timely manner.

I really like unit test automation in that it provides a significantly better product out of the gate into any testing. Functional test automation is effective as long as it is designed properly and does not become a maintenance burden, and as long as the tests aren't so complex as to cause problems themselves. Automation that extends the reach of the tester clearly results in a better product.

Holy Grail, not by a long shot.

Bob Clancy

AbbotOfUnreason said...

Ha! Ha! He said "assert"

Unknown said...

If it is the Holy Grail, mine is especially holy. It keeps leaking faster than I can patch it up.