Interesting People at GTAC - I

Or: Test Eye for the Framework Guy

Douglas Sellers, The Inkster Guy gave a talk on writing a domain-specific language for testing. His idea is to write the test cases first in a language that looks like a English, then figure out how to write an interpreter for those test cases.

(Stereotype warning / warning / you’ve been warned ...)

In my mind, this is a huge part of the problem with test frameworks. Developers write test frameworks so that someone else ("those QA people") can write test cases.

The problem is, writing the test case in XML or java is just no fun. Really, it is no fun. It's so not fun that the devs don't want to do it. In my mind, this is a problem with process design – when you design a process for somebody else to do, the resulting process isn't very good, because you trade all the good and fun things away to get management-y things.

Here's my conclusion: If you want to write a test framework, first write some test cases, and keep tweaking them until they are fun to write – or at least easy. Then write the framework.

Then implement a real, non-trival test project in your framework. Yourself. Yes, you. The developer. If you can build a test suite and actually use it, then open-source and talk about your tool.

Another neat person I met at GTAC was Will Roden, author of the Software Inquisition blog. At lunch on the second day of the conference, will demoed SiFL for me. SiFL is a "foundation library" that will wrote on top of Quick Test Pro, in order to make writing test cases in QTP quick, easy, and, well, fun.

Basically, it's just a series of reusable functions that make driving a browser really really easy. The thing is, they are designed to be easy for the tester, and optimized for the tester. This is a real library that, when combined with QTP, looks just as useful to me (more useful) than some of the frameworks that we see projected on the big screen.

Why? Because it was developed of the tester, by the tester, for the tester. No XML test cases for you ...

UPDATE:I found a great little quote that explains the whole framework-authors-vs-testers issue:

Strathman's Law of Program Management: Nothing is so easy as the job you imagine someone else doing.

GTAC - Bonus Section #3:

Another speaker claimed that "The only Exhaustive Testing is when the tester is exhausted" - we agree. By Exhausive Frameworks, I mean the ability to do everything. The problem is that by being _able_ to do everything, we use abstractions that make it hard to do any specific thing.

Sometimes, a quick-and-dirty, only-works-for-this-app, only-works-for-this-screen automation provides the highest ROI.

My approach is rapid test automation - but do what works for you.

(The concept is similar to James Bach's Rapid Software Testing - Used with permission)

GTAC - Bonus Section #2:

We have seen a wonderfully isolated, encapsulated, poly-morphed, design-patterned, auto-tested, mocked app ...

- That could have been written procedurally in 500 Source Lines of Code
- But now consists of 10 classes and 20 files spread over 4000 SLOC

Using mock tools results in software with more code (Pettichord, "Homebrew Test Automation", 2003)

If you can keep everything in your head, you don't need radical separation. Radical separation is tool to use when your components get big and unmanageable, and results in a *lot* of components, with each individual component being smaller than you had to start.

GTAC - Bonus Section #1

Our GTAC Talk evolved over an extended period, and had a lot more material than the time allowed. So, just for you Creative Chaos readers, I'm going to blog our bonus section.

Comments on the Slide Above:
If you want to mock out ('trust') every domain - including database on db intensive apps, the filesystem on file intensive apps, and the internet on web-service apps - tread lightly.

Steve Freeman, co-author of Mock Roles not Objects, points out that his main use of mocks is to help him discover roles and relationships between objects in his domain code, rather than speeding up external dependencies. He tends to write little integration tests to check things like that the database is hooked up correctly. (Editors note: Steve actually wrote that paragraph.)

So he's saying use mocks for design - de-emphasizing the use for testing. Mocking out the boundaries doesn't help you with that design decision, because you are not designing the hard boundary.

So don't do this unless you're testing some otherwise untestable requirement - like handling exceptions when the filesystem is full up, when it is unrealistic to fill up the real file system. (We have a slide on this on the YouTube Talk called "Mocks As Simulators")

If you must mock out the core system, have a balanced breakfast, cover your tests somewhere else, or run the "real" tests overnight.

Why GTAC is different

As I write this, it’s 3:36PM on August 23rd, and I am sitting at the New York Google Office, just after co-presenting a talk on interaction-based testing.

I am sick. Exhausted. Drained. Barely able to give the follow-up speakers the attention they deserve – but I’m trying, they did it for me.

And, to borrow a phrase, this is also "The Awesome."

GTAC is fundamentally different . There is one track (no "concurrent sessions"), so everyone has an identical frame of reference. Attendance is capped at 150, so you have a real chance to meet everyone. The conference is run at a financial loss by Google as a gift to the community – so instead of attracting paying customers, Google can be selective about attendees. Because it is capped at 150, they can be very selective.

Moreover, this is no junket in Nantucket. From the moment we arrived until it was time to sleep, Google has had events. The "business day" for the conference days run 7:30AM to 7:30PM – following by mingling, followed by a reception, which doesn’t end until 10:00PM. If you came to New York to see the Statue of Liberty (or to appear on the Today Show) you’re probably out of luck – but if you want to talk about software testing, c’mon, sit down, here's a salt shaker, let's test.

Finally, the conference ends on a Friday, which means if people want to fly home, they have to do it on Saturday. Again, people who don't really care but want a junket are not very likely give up "their" Saturday for travel.

Bottom line: GTAC is the most impressive test automation conference I know of, period. It's been an honor to speak, it was fun, and I'm glad that my speaking portion is done and I can enjoy the rest of the conference.

By the way, if you are into speaking, GTAC is also one of the best audiences that I have ever presented for before. Forgiving, interested, actively listening, thinking critically, and consisting of true peers. I have to say, this is just about everything I could ask for in an audience – and that makes a huge difference. (Come to think of it, the only conference I've had a better audience with was the Indianapolis QA Association, who have all that and mid-west sense of humor ...)

UPDATE: If you couldn't be in New York city RIGHT NOW, you can watch the talk on Interaction Based Testing by Heusser and McMillan.