What's a "Test Framework"?

Shrini Kulkarni has been after me to define my terms; after all, I keep writing about "Test Frameworks" but I've never defined the term.

Wikipedia defines a framework as "A basic conceptual structure used to solve a complex issue. It also warns that "This very broad definition has allowed the term to be used as a buzzword."

When I use the term, I mean any support, infrastructure, tool or "scaffolding" designed to make testing easier, and (often) automated.

For example: Let's say you have a simple program that converts distance from Miles to Kilometers. The application is a windows application. Every time we make a change, we have a bunch of tests we want to run, yet we can only enter one value at a time, manually. Bummer.

Yet we could think of the software as two systems - the GUI, which "just" passes the data from the keyboard into a low-level function, and "just" prints the answer, and the conversion formula function.

If we could somehow separate these two, and get at the formula function programmatically. Imagine that the formula function is struck in a code library, which can be shared with many different programs.

Then we could write a "tester" program, which takes an input file full of input values and expected results. The "tester" program simply calls the library, compares the actual result to the expected, and prints out a success or failure message.

This is basically how I test a lot of my code, using a program called Test::More. You could call Test::More and it's friends (Test::Harness, and so on) a "framework."

We can go dig into the details and test at the developer understanding level, or bubble up and only test things that have meaning to the customer. One popular framework for these higher level, business logic tests is FIT/Fitnesse

Of course, there is more to the application and just the business logic. The GUI could accept the wrong characters (like letters), format the decimals incorrectly, fail to report errors, handle resizing badly, or a half dozen other things. Even with one "framework", we still have the problem of testing the GUI (not to be forgotten) and testing the two pieces put back together again - "Acceptance" testing, or, perhaps, "System" testing.

This "outer shell" testing can also be slow, painful, and expensive, so there are dozens of free/open or commercial testing frameworks that allow you to 'drive' the user interface of windows or a web browser. With the big commercial tools, people often find that they are writing the same repetitive code, over and over again, so they write libraries on top of the tool like Sifl.

The big dog web browser automation systems are Selenium and Watir.

Years ago (back when he was at Microsoft), Harry Robinson once told me that MS typically had two types of testers: Manual Testers, and Developers who like to write frameworks. The problem was that the Developers would write frameworks that no one was interested in using. His assertion (and mine as well) is that people who straddle the middle - who like to test, and like to write software to help them test faster, can be much for effective than people entrenched on either side.

Thus, you don't set out to write a framework - instead, you write a little code to help make testing easier, then you write it again, then you generalize. Over time, slowly, the framework emerges, like Gold refined through fire.

But that's just me talking. What do you think? (And, Shrini - did I answer your questions?)