Does anybody else remember the Middle School Science Fair? The idea is to apply the scientific method to something practical, then show off all the budding scientists.
If I remember the scientific methods as I was taught, it was essentially this:
- Collect some documented observations of your subject
- Make a hypothesis – a theory that should predict behavior (Make sure you pick the right one!)
- Conduct an experiment – Conduct it properly, so to easily repeatable.
- Evaluate the results of the experiment – do they support your hypothesis?
- Draw Conclusions
When done, you should have some artifacts that document the experiment. This is usually a big poster-board with a page for each step in the method, plus a cutesey visual display. We were then evaluated on our displays, documentation, and how valid our experiments were.
... and I suspect that the entire approach is wrong. Wrong. Wrong.
What if we were not taught to do one experiment meticulously and correctly, but instead taught how to do lots of experiments – dozens, hundreds – very quickly and very sloppily? Then, if we find something interesting, we can come back and do the experiment again. If the results are interesting, then play the documentation game.
Done this way, our students actually have a shot at real innovation, instead of pseudo-science. Thus we can judge competitors not by the quality of the documentation, but by weather or not they actually found anything interesting.
This test early, test often, test always idea of science is not new; actually, it’s the basic idea that Thomas Edison used to bring the light bulb to market. In fact, when I looked up the Scientifc Method in Wikipedia this morning, it had a large section on "Evaluation and Iteration" and "Testing and Improvement." It turns out, that’s the real scientific method after all.
The problem is that we are so seeped in this culture that we confuse the accidental things like documentation with the essence – discovery and experiment.
We have this problem in software testing.
We call it the test case.
2 comments:
I keep hearing that documentation and repeatability are keys to good testing. I also hear that we should not waste any time running unnecessary tests. Instead we should over discuss (time is better spent in meetings than testing), over analyze, and over document everything before we waste time executing tests. Right? ;)
I think there is a worthwhile counterargument hiding in Nikola Tesla's bitter commentary on Edison's life, as quoted on Wikipedia.
"His method was inefficient in the extreme, for an immense ground had to be covered to get anything at all unless blind chance intervened and, at first, I was almost a sorry witness of his doings, knowing that just a little theory and calculation would have saved him 90 per cent of the labor."
Tesla should not be considered an objective commentator on Edison (he hated Edison with a passion), but his phrase "inefficient in the extreme" was the key point for me.
I think that if a test case can be described simply enough to allow experimentation, with enough room for development and capture of knowledge, then a test case serves its purpose well and is not as likely to be "inefficient in the extreme".
I realize the argument is weak, based on a poor analogy and an opponent's embittered comments about a brilliant inventor, but I don't want to surrender the scientific method too easily, even in my exploratory testing.
When I am exploring an area, one outcome will probably be bug reports. Repeatable bug reports have generated better results for me in the past than unrepeatable bug reports. Getting to repeatable bug reports seems to trigger at least some portion of the scientific method. I don't want to abandon that portion of the scientific method that helps me remember to remember things, and remember to observe things, and remember to reason about things...
Post a Comment