Testing Encyclopedia - I

Meta: I am currently working with Chris McMahon on a series of short articles on testing terminology. Of course, we covet your feedback, and would like to open them up to the world for review. Your comments are welcome ...

Each month, we takes testing jargon and puts it in plain English, allowing you to respond to questions like "Did you exhaustively test this?", "Are we doing an SVT after our BVT?", or Does the performance testing pass?" Our first installment is on fundamental concepts in software testing.


The Impossibility of Complete Testing
Imagine the simplest of computer programs: One that takes a given date and generates the next calendar day; we call this GetNextDate. Assuming the software works for year 4000BC to 4000AD, there are 365*(4000+4000), or 2.9 Million possibilities – and that is assuming the dates don’t include timestamp, and that does not include any error cases.
The typical computer program is much more complex than GetNextDate(), and quickly spirals to billions and trillions of possible test scenarios. One of the central challenges of software testing is that we have a finite amount of time to run an infinite number of possible tests. As Bill Hetzel once put so well "The only exhaustive testing there is is so much testing that the tester is exhausted! "
Many approaches and techniques are designed to overcome this impossibility – to determine the cheapest and most valuable tests, and how to use the information those tests provide to assess the software.

The Mission Problem
Once your team determines the impossibility of complete testing, the next logical step is to identify the goal, or mission the team can strive toward. Two common missions are the bug hunt – to find the most valuable bugs quickly – or to assess the readiness of the software to ship.
While they may sound similar, those two missions can lead your team in different directions. For example, in a bug hunt, the team may go after the most broken and defect-laden modules, while an assessing team might state that the module is known defective and move on to assess other modules.
Other missions include complying with documented process or documenting work-arounds prior to shipping the software. Over time, a team may have many different missions. An implicit or vague mission could mean that your team is not doing what management expects, which can lead to conflict, friction, and strife.

The Oracle Problem
In order for a test to pass, the tester has to know what the "right" answer is. For a complex financial, graphical, or database system, the right answer might not be obvious. The classic example of this is trying to test a word-processor – how do you know that the 12-point font on your screen is actually 12-point?
If you have documented requirements, these can serve as an Oracle. Other popular oracles include "the product owner said so", "The developer told me that is correct", "That is the way our competing product that we are supposed to be compatible with does it.", and the dictionary.

Equivalence Class Partitioning (ECP)
ECP is an extremely popular way of taking the problem of complete testing and limiting it to a reasonable set of test cases. The idea behind ECP is to break all inputs down into classes, then test at least one of each class. In our GetNextDate example, we might break problems into:

The First day of a New Month
The Middle of a Month
The Last Day of a Month
The Last Day of a Year
The 28th of February, Non-Leap-Year
The 28th of February, Leap-Year
The 29th of February, Leap-Year

We could also add test cases for BC, AD, and the boundaries between the two. This technique allows us to go from millions of test cases to a few dozen, but it is not perfect. Equivalence Class Partitioning does not address malformed dates such as the 32nd of January or the –1st of April – nor will it catch bugs created by local optimization. (More about that next month)


For an in-depth introduction to testing concepts, we recommends the Fundamentals of Testing course offered by the Association for Software Testing: http://www.associationforsoftwaretesting.org/drupal/courses/foundations
Future Articles will cover testing process, test automation, exploratory testing, and coverage in more depth.