Schedule and Events

March 26-29, 2012, Software Test Professionals Conference, New Orleans
July, 14-15, 2012 - Test Coach Camp, San Jose, California
July, 16-18, 2012 - Conference for the Association for Software Testing (CAST 2012), San Jose, California
August 2012+ - At Liberty; available. Contact me by email:

Tuesday, December 04, 2007

You keep using that word ...

Last week I wrote that:

There's something going on here with the way we use terms like "Test" and "Requirement" that causes confusion and misunderstanding. Fundamentally, various groups, like the traditional test community, the "Agile" Developer Community, the Scrum People, and so on, are looking for different benefits from testing. Thus, in conversations, we "miss" each other or leave unsatisfied ... Perhaps that's a post for another day.

Then I spent the next week in a private email discussion over this subject, started by Ben Simo. During that discussion, we identified two major different kinds of tests:

A) Examples to help with the construction process. For example, in a simple Yards-To-Meters conversion program, examples can show the bounds for input, can demonstrate how many decimal places to take the rounding, what to do with garbage input, and provide samples to check the forumula. You could argue that tests as examples can act as requirements. (You Would Be Wrong, but you could argue it). Personally, I hold that these kinds of "construction/example" tests can augment requirements, and that's a real good thing.

B) Tests to learn things about the software. These tests are investigative in nature, and make sure we are not "fooled" into thinking things like "The Software Works" without reason. These investigative tests generally require much more aggressive and advanced critical thinking skills, often in real time.

---> One problem that I've seen is that we talk about these things, "tests", but don't expressly talk about which of the two classes we mean - and, of course, there are more than two classes. On the discussion list, I think Ben summed it up best:

I think we have people thinking that automated "acceptance tests" can replace traditional critical testing. We now have some testers thinking that developers are going to be more involved in the critical testing. People coming from the two viewpoints don't seem to realize that they aren't all talking about the same thing. ... Although there are some testing ideas that are not compatible, I do believe that things like TDD and automated acceptance tests are good as long as people don't think that automated execution replaces restless thinkers. If I had to have one or the other, I want the restless thinkers. However, I hope we can have both.
- Simo

---> Now, for my thoughts. I've spent most of my career with a business card that said "Developer." When I started doing programmer-testing, I did actual critical thinking and "does it work" testing.

Then I ran into various Agile dev/testers that, well, weren't. They were using tests as examples and not finding bugs - or writing blogs, articles, or books about "testing" that didn't really talk about critical thinking.

My initial response to this was:

"What the hell is wrong with you people?"

After some time, I realized that a different philosophy about software testing leads to different results.

If those are the results you actually want, well ... I guess that's ok.

In the free market of ideas, these ideas will compete - and sometimes, complement each other.

May the system that delivers that best result win.

1 comment:

Unknown said...

In any area where people are to some extent figuring out what they need to do as they go along (like software dev), there aren't going to be clear-cut, ready-made, and widely agreed upon terms for everything. Words are used without scrutinizing their applicability too closely, and this creates the confusion you describe, imho.

I might argue that "tests" to help with the construction process aren't *really* tests. 'Experiments' might be a better word. But they seem enough like tests that people are going to call them that.

I said I might argue that because it's hair-splitting over confusion and not realy here nor there... But I definitely appreciate the confusion you point out. That said, I think words of clarification, like your post, are the only way to clear this confusion up. There is a tendency to want to define and enforce specific meanings of terms, but that really doesn't work any better, it's just more heavy handed...