Interesting People at GTAC - I

Or: Test Eye for the Framework Guy

Douglas Sellers, The Inkster Guy gave a talk on writing a domain-specific language for testing. His idea is to write the test cases first in a language that looks like a English, then figure out how to write an interpreter for those test cases.

(Stereotype warning / warning / you’ve been warned ...)

In my mind, this is a huge part of the problem with test frameworks. Developers write test frameworks so that someone else ("those QA people") can write test cases.

The problem is, writing the test case in XML or java is just no fun. Really, it is no fun. It's so not fun that the devs don't want to do it. In my mind, this is a problem with process design – when you design a process for somebody else to do, the resulting process isn't very good, because you trade all the good and fun things away to get management-y things.

Here's my conclusion: If you want to write a test framework, first write some test cases, and keep tweaking them until they are fun to write – or at least easy. Then write the framework.

Then implement a real, non-trival test project in your framework. Yourself. Yes, you. The developer. If you can build a test suite and actually use it, then open-source and talk about your tool.

Another neat person I met at GTAC was Will Roden, author of the Software Inquisition blog. At lunch on the second day of the conference, will demoed SiFL for me. SiFL is a "foundation library" that will wrote on top of Quick Test Pro, in order to make writing test cases in QTP quick, easy, and, well, fun.

Basically, it's just a series of reusable functions that make driving a browser really really easy. The thing is, they are designed to be easy for the tester, and optimized for the tester. This is a real library that, when combined with QTP, looks just as useful to me (more useful) than some of the frameworks that we see projected on the big screen.

Why? Because it was developed of the tester, by the tester, for the tester. No XML test cases for you ...

UPDATE:I found a great little quote that explains the whole framework-authors-vs-testers issue:

Strathman's Law of Program Management: Nothing is so easy as the job you imagine someone else doing.

GTAC - Bonus Section #3:

Another speaker claimed that "The only Exhaustive Testing is when the tester is exhausted" - we agree. By Exhausive Frameworks, I mean the ability to do everything. The problem is that by being _able_ to do everything, we use abstractions that make it hard to do any specific thing.

Sometimes, a quick-and-dirty, only-works-for-this-app, only-works-for-this-screen automation provides the highest ROI.

My approach is rapid test automation - but do what works for you.

(The concept is similar to James Bach's Rapid Software Testing - Used with permission)

GTAC - Bonus Section #2:

We have seen a wonderfully isolated, encapsulated, poly-morphed, design-patterned, auto-tested, mocked app ...

- That could have been written procedurally in 500 Source Lines of Code
- But now consists of 10 classes and 20 files spread over 4000 SLOC

Using mock tools results in software with more code (Pettichord, "Homebrew Test Automation", 2003)

If you can keep everything in your head, you don't need radical separation. Radical separation is tool to use when your components get big and unmanageable, and results in a *lot* of components, with each individual component being smaller than you had to start.

GTAC - Bonus Section #1

Our GTAC Talk evolved over an extended period, and had a lot more material than the time allowed. So, just for you Creative Chaos readers, I'm going to blog our bonus section.

Comments on the Slide Above:
If you want to mock out ('trust') every domain - including database on db intensive apps, the filesystem on file intensive apps, and the internet on web-service apps - tread lightly.

Steve Freeman, co-author of Mock Roles not Objects, points out that his main use of mocks is to help him discover roles and relationships between objects in his domain code, rather than speeding up external dependencies. He tends to write little integration tests to check things like that the database is hooked up correctly. (Editors note: Steve actually wrote that paragraph.)

So he's saying use mocks for design - de-emphasizing the use for testing. Mocking out the boundaries doesn't help you with that design decision, because you are not designing the hard boundary.

So don't do this unless you're testing some otherwise untestable requirement - like handling exceptions when the filesystem is full up, when it is unrealistic to fill up the real file system. (We have a slide on this on the YouTube Talk called "Mocks As Simulators")

If you must mock out the core system, have a balanced breakfast, cover your tests somewhere else, or run the "real" tests overnight.

Why GTAC is different

As I write this, it’s 3:36PM on August 23rd, and I am sitting at the New York Google Office, just after co-presenting a talk on interaction-based testing.

I am sick. Exhausted. Drained. Barely able to give the follow-up speakers the attention they deserve – but I’m trying, they did it for me.

And, to borrow a phrase, this is also "The Awesome."

GTAC is fundamentally different . There is one track (no "concurrent sessions"), so everyone has an identical frame of reference. Attendance is capped at 150, so you have a real chance to meet everyone. The conference is run at a financial loss by Google as a gift to the community – so instead of attracting paying customers, Google can be selective about attendees. Because it is capped at 150, they can be very selective.

Moreover, this is no junket in Nantucket. From the moment we arrived until it was time to sleep, Google has had events. The "business day" for the conference days run 7:30AM to 7:30PM – following by mingling, followed by a reception, which doesn’t end until 10:00PM. If you came to New York to see the Statue of Liberty (or to appear on the Today Show) you’re probably out of luck – but if you want to talk about software testing, c’mon, sit down, here's a salt shaker, let's test.

Finally, the conference ends on a Friday, which means if people want to fly home, they have to do it on Saturday. Again, people who don't really care but want a junket are not very likely give up "their" Saturday for travel.

Bottom line: GTAC is the most impressive test automation conference I know of, period. It's been an honor to speak, it was fun, and I'm glad that my speaking portion is done and I can enjoy the rest of the conference.

By the way, if you are into speaking, GTAC is also one of the best audiences that I have ever presented for before. Forgiving, interested, actively listening, thinking critically, and consisting of true peers. I have to say, this is just about everything I could ask for in an audience – and that makes a huge difference. (Come to think of it, the only conference I've had a better audience with was the Indianapolis QA Association, who have all that and mid-west sense of humor ...)

UPDATE: If you couldn't be in New York city RIGHT NOW, you can watch the talk on Interaction Based Testing by Heusser and McMillan.

... and Steve Poling Replies

This went out over SW-IMPROVE, my software discussion list. I will post my reply tomorrow ...

I heartily agree that the first two items are anathema to effective software development.

However, I see (with significant disclaimers) some benefit in the last two. When you say "hand off to the next person in the chain" that premises a serial relationship between workers who, assembly-line fashion, perform tasks that incrementally transform raw materials into some manufactured good. This might work if software were manufactured goods. It's a remarkably stupid way to approach software development. Whereas rivets and holes and the like in a manufactured good is standardized, and subject to no uncertainty or renegotiation at the point of production. However, if you and I work together between two parts of a software system, we'll may start with a rough interface between our stuff, but we'll be working out details between us.

I assert the value of the "artifacts that are intermediate work products" to record the AS-BUILT connections and theory of operation of each piece. A few reasons come to mind: some poor slob is going to have to maintain this mess. It's nice to have a written starting point to help that guy get into the "Zen" of the code. Especially, when that poor slob is either of us six months after our initial development push.

Second, suppose one of us gets hit by a truck or leaves the company. The only true reality of a software solution is the object code. But when you look at the source code, you easily lose track of the forest for the trees.

After a software system exists for a few years someone gets it in his head to "rewrite this mess, and get it right this time." I have fingerprints on three generations of software that solves similar/overlapping problems. Would you believe I've seen similar bug reports issued on each? Each time we ran into the same issues that are inherent to the problem being solved. What often looks like kruft in the code are bug-fixes. Those krufty looking bug-fixes reflect difficult parts of the problem being solved, or limitations of the initial design. The fact that they're bug-fixes indicates that the requirements and design were inadequate at these points. I have often heard "build the new one just like the old one, but..." If I can meaningfully study the old one's requirements and design while aware of the bugs of that implementation, I can write more complete requirements and design.

It's my opinion that the most enduring aspect of any software development system is the requirements discovery. I have picked up the requirements for a DOS system written in the 1980s and used them to directly identify test cases for three-generations-later rewrite. Thus when we discovered that we had no decent requirements statement for a project (written in the early 1990s.), I told the woman tasked with this, "Managers come and go. (I looked at my boss.) Programmers come and go. (I pointed to myself.) But the single most important thing that you will do in your entire tenure with this company will be this requirements statement."

This happened in the context of our having an existing code base that has been making the company money for over 15 years. Making the new one exactly like the old one would not work, because in so doing, we found differences that reflected undeniable errors I made in the early '90s. Back then, me and a fellow who left the company sweat bullets on the requirements coming up with "a nod and a handshake" agreements at each point. We didn't write it down; I just coded it that way. Thus, I had nothing to go by, except code known to have bugs and fading memories, when I did the rewrite this spring. My fading memories are a sucky basis for establishing requirements. Even less would be available to the company had I died of cancer five years ago. Conversely, if we had documented what we agreed to back then, we'd have saved the company a LOT of rework when the new implementation introduced variations where unstated business rules were ignored.

With this in mind, I happily assert, "if you didn't */record/* it, it didn't happen." I know you like to satisfy documentation requirements "creatively" so I'll gladly accept VHS (or Betamax) tapes of conversations between principals where requirements are discovered and concomitant decisions are made. Similarly, I'm cool with the meeting minutes of design meetings and reviews.

Interlude

This looked so interesting I had to post it:

"Myths in Software Engineering" -

A few of my favorite myths:

- Software development consists of discrete, separate activities that can be organized into phases

- The best way to make the *overall* process effective is to have efficient specialists for each phase

- These specialists should produce artifacts that are "intermediate work products", to be handed off to the next person in the chain

... and my personal favorite:

- If you didn't write it down, it didn't happen.

If you enjoy this kind of thing, there is a great little book called "Facts and Fallacies of Software Engineering" that goes into much more detail. If I recall correctly, fallacy #1 is "Without metrics, you can't manage."

What are some of your favorite myths in software engineering?

Testing the Stapler

To answer the Stapler question, Mike Kelly referred me to this post where he had 144 tests. Yes, 144 tests, and, offhand, they look to be real, valid tests. To get there, Mike starts by looking up the manufacturer of the stapler, getting the specification, and seeing if the physical object conforms to the spec.

David Drake took a different tactic: He started with the Egg, and asked what purpose I wanted to use the Egg for.

A different way to say this would be "What are the requirements?" which is a solid way to start the discussion. For example, Mike could have tested the stapler to the Stanley Bostich Specification, but our requirements could be for a industrial-strength heavy duty stapler. In other words, we bought something Commercial, Off the Shelf that wasn't fit for purpose.

That might not be quite fair to Mike, as I am sure he asked about fitness for purpose before starting his list - but I would be remiss if I did not mention it.

To do this at a conference, I would probably have a thin, pointed knife sitting next to a stack of envelopes, and say something like "Take your pick" between the egg, stapler, shalt shaker, and knife. When picking up the knife, by assumption is that more people would lick the envelopes and try to open them, and few would ask me what are the requirements.

At which point, I reveal that I'm looking for a butter knife.

You may say "No fair! That's misleading!!"

To which I say, hey, it's a simulation.

If some says that it's an unrealistic simulation, I would ask:

'Have you ever actually tried to build software in corporate America?'

:-)

More importantly, I think the response to the challenge is important. Both Mike and David Drake impressed me. I've seen a spectrum of answers and put them in general categories. More about that tomorrow.

PS: If you've got a conference and would like to see more of this thing, let me know. Otherwise, I'm thinking of setting up shop in the hallway at a few places next year ...

... And Carry a Big Stick

Yesterday I copied over my iTunes Library as well as some key conference CD notes. Big Mistake; I'm up to 2GB and haven't started yet. :-)

Both James Bach and Jon Kohl Recommended Portable Apps, a complete, 'lite' software suite designed for a USB key. The programs are both small and can run by double-clicking - without installing, without needing write access to C:\, without having to be an administrator.

Portable Apps includes Mozilla FireFox, Putty (a telnet and SSH client), WinSCP3 (A Secure file copy utility), and Open Office. I'm not 100% happy with these options - just about every system allready has a web browser (even if it's a bad one), and Microsoft Office is just about ubiquitous. So saying "Look, I bought a USB key so I can use (1) May Favorite Apps in case (2) Equivalent Apps aren't available" doesn't sound so great when (1) The Differences Ain't Much and (2) Equivalenta Apps are pretty much everywhere.

I'll keep looking. For the time being I am considering taking those conference materials off the keyfob. In the mean time, it's nice to have a home for my passwords, writings/presentations, and a backup of my website. My next move is probably to move to an encrypted password manager, and surprise, there is one available for portable apps. (Can anyone recommend a good, free, windows automation tool besides Tasker?) I may also download the encryption/backup software that comes with portable apps.

As for the stapler, egg, and salt shaker - more tomorrow ...

UPDATE: The 'standard install' of portable apps seems to have a lot of applications I am not really interested in, like games and open office. Also, I'm not quite sure how to un-install the things that are installed. I'll keep playing around; more reviews to come.

Look Ma, the works of Shakespeare on a stick!

At STAREast this year, James Bach had a "portable testing training kit." Roughly the size of a small purse (or, er ... a "man's bag"), I'm sure it had various test challenges in it.

I didn't get the chance to see it, but if pressed to make my own, it would be things like a stapler, a salt shaker, and a hard boiled egg. I would hand these to my student and say something like:

"Test This"

(This also, by the way, is my general litmus test for a tester. When it comes to actually talking about the real work of testing a thing - If the person is utterly confused, or smiles genially and changes the subject, or asks me what _I_ think - well, that tells me something. If they roll up their sleeves and dive in, that tells me something ...)

But I've been thinking about a different testing toolkit lately.

I keep my source materials, some tools, and a copy of my website on a USB Drive. The drive has recently been pushing its limits, so I just bought a replacement - a 4 GB drive, for thirty bucks from BestBuy.

4 GB is a lot of space. A lot.

This got me thinking about what I do when I come into a new company. The first thing I do is download a bunch of free tools - Putty, WinSCP3, Tasker, SnagIt, TextPad, ActivePerl, Dia, GVim, Audacity. I also have a bunch of PDF and Word documents that I read and re-read every year. With 4GB of space, I could put all of those on a memory stick, and more.

These are testing tools, but there are also security testing tools - and tutorials - that would fit easily on the stick. Snort, Crack, intrusion detection, SQL injection, and other tools come to mind.

With 4GB of space, I could put all these tools and more on a stick. If they were good enough, I could sell the stick as a value-added tool or, more likely, just have an interesting howto list on a website. Yes, getting a booth at DefCon and selling security tools has occurred to me, but for the time being, I'll keep my hat white, thank you very much.

So here's my two questions, take your pick:

1) If you developed a testing tool on a stick, what free (or cheap-ware) tools would you include? What is missing from my list? What entire categories are missing? If there a different kind of stick to develop? Yes, I could do a developer stick with apache, php, mysql, but most of those come with Linux Anyway.

2) If you don't like that, here's another one: The stapler, the hard-boiled egg, or the salt shaker. I've asked you to test it. What do you do?

Models - II

(There aren't the Requirements Models you are looking for ...)

Sometimes, the models we use are predictive (like yesterday) or are analogies. For example, e-mail is a pretty straightforward analogy to snail mail; you have a send, receiver, addresses, transfer mechanism, routers, and so on. In many cases, the analogy (or prediction) is helpful; you what to expect and how to behave.

Then again, like the famous outside-the-box problem, analogies can limit our thinking. Consider, for example, the term 'requirements gathering', which for me always brought an image of a child on an easter-egg hunt. The requirements are right there; just go collect them. In that model, requirements are a contract – customers tell the technicians what they want, then we haggle over the price (schedule), sign the contract, and develop.

Except in software, the contract is not static – it's fluid. Oh, and English is a vague and ambiguous language, so when we communicate solely in written words (like a typical contract), we are losing information. Plus, with this model, if the developers have a better idea "gosh, if we knew zip code, we could call out to the mapquest web service and ..." they do not have a mechanism to contribute the idea to the software. (And don’t even think about testers contributing ideas!)

The contract model can work for certain kinds of software – for example, avionics software that has to go on a new helicopter for the department of defense. Given an incremental improvement to a well-defined, state-based system, you can both state requirements precisely and limit change. That is not software development; it is software engineering.

... And most of my project just aren’t like that. On most of my projects, the contract model is a joke. Instead, I prefer the defense-lawyer model, which works like this:

The customer has a problem. He does not understand the technical jargon and doesn't have the education or tools to solve the problem. So he hires a lawyer. The lawyer makes the plan, and the customer can over-rule the lawyer on key decisions (how to plead), or, if he is not happy, fire the lawyer. Ideally, the two work together in collaboration, admitting that if they do not, the project will have much less chance of success.

For commercial software with at least hundreds, if not tens of thousands of customers, you don't even have a customer to write the "contract." Instead, you have to invent roles like business analyst or product owner. That's fine, but sometimes the limits imposed by the model aren't real – they are simply part of the model.

Now, on a real contract, all elements of the contract can be negotiated – not just price and schedule. By not mentioning that earlier, I’ve limited the model by omission, making it worse.

We humans do this all the time. As much as this is a problem to avoid, it’s also a huge opportunity. When your competitors get trapped in an analogy, *you* can break out of the analogy.

This doesn't sound like much, think of what it could mean for version control software, email, file systems, or any other piece of software trapped in a metaphor. Look at what google did with search, using back-links instead of a dewey-decimal system metaphor. Think of what they did with email with "search, don't sort". Think of what developers are doing with AJAX and mashups – taking two different web services and combining them in interesting ways – or what testers are doing with random, automated testing instead of a traditional list-checklist-verify model.

Sometimes, people get it wrong. Sun spent years trying to change the model from PC-centric back to the mainframe using the term "thin client." (Eventually, that happened anyway, due to the web browser, not Sun.)

As I mentioned above, models can have a lot of value. But, often models can be like the stock market: By the time everybody is buying, it's probably time to sell and find something else ...

Real Science

Does anybody else remember the Middle School Science Fair? The idea is to apply the scientific method to something practical, then show off all the budding scientists.

If I remember the scientific methods as I was taught, it was essentially this:

- Collect some documented observations of your subject
- Make a hypothesis – a theory that should predict behavior (Make sure you pick the right one!)
- Conduct an experiment – Conduct it properly, so to easily repeatable.
- Evaluate the results of the experiment – do they support your hypothesis?
- Draw Conclusions

When done, you should have some artifacts that document the experiment. This is usually a big poster-board with a page for each step in the method, plus a cutesey visual display. We were then evaluated on our displays, documentation, and how valid our experiments were.

... and I suspect that the entire approach is wrong. Wrong. Wrong.

What if we were not taught to do one experiment meticulously and correctly, but instead taught how to do lots of experiments – dozens, hundreds – very quickly and very sloppily? Then, if we find something interesting, we can come back and do the experiment again. If the results are interesting, then play the documentation game.

Done this way, our students actually have a shot at real innovation, instead of pseudo-science. Thus we can judge competitors not by the quality of the documentation, but by weather or not they actually found anything interesting.

This test early, test often, test always idea of science is not new; actually, it’s the basic idea that Thomas Edison used to bring the light bulb to market. In fact, when I looked up the Scientifc Method in Wikipedia this morning, it had a large section on "Evaluation and Iteration" and "Testing and Improvement." It turns out, that’s the real scientific method after all.

The problem is that we are so seeped in this culture that we confuse the accidental things like documentation with the essence – discovery and experiment.

We have this problem in software testing.

We call it the test case.

Test Automation - IV

Right now one of the "louder voices in the room" for test automation is the "Agile" test automation voice. And by that I mean something very specific - that you create a large series of clerical tests that go from acceptance level all the way to unit; that you can run those tests at the click of a button, and get a green or redbar (pass/fail) very, very quickly. If any component in the system is slow, you can mock or stub it out to get that fast feedback. A few people who talk about this idea admit that you may want to actually run the system end-to-end, for example by running the full, slow test suite overnight.

This kind of test automation can be very valuable, especially for software developers, but it isn't the only kind.

Earlier this week, Ben Simo and I were kicking ideas around and we came up with:

1 Any tool that eliminates manual, clerical work is test automation (diff, loadrunner, tasker)

2 Finite-State-Machine, Random-Number Generating Tests are test automation (In the Harry Robinson sense, or, for that matter, the Ben Simo sense)

3 "Big FOR loop with an external, trusted Oracle" test automation

4 Develop it twice, run it on both systems, and compare the results test automation (Create two, not-very-trustworthy oracles)

5 Monkey Test Automation. Press Random buttons, record the order, wait for a crash, and then dump the log.

6 Pairwise Test Generation tools are test automation.

7 Any test case generator (perlclip) is test automation

This is a quick, sloppy list. What are some other forms of test automation? What am I missing?

It would be awful nice to come up with a consistent model for different types of testing tools. Failing that, I'd settle for a vision, or, to be honest, a "good start."

Elisabeth Hendrickson just announced the Agile Alliance Functional Testing Tools Visioning Workshop, in Portland, in October. I think all of these could fall into that vision. If I could only cover the travel expense and lost income, i'd be there. As it is, I look forward to see what falls out.

If you can make it, I'd encourage any regular Creative Chaos readers to attend. In other words, if you like this blog, I think you'd like the workshop, and, I would hope, vice versa ...

Where have all the sapient processes gone?

Most agile test automation is, well, clerical. To borrow an analogy from James Bach, it views testing as something like an inventory clerk at a Grocery Store. "It says here we should have thirty cans of peas. How many cans of peas do we have? Thirty? Good! Check. Greenbar."

In addition to counting them, a real tester would be doing a lot of different things - looking at how the cans are stacked, checking the labels to make sure none have peeled off - "noticing" things that are just not quite right. It's relatively easy to automate the clerical part of testing, but very hard to automate the sapient part.

I find that when I talk about Sapient testing, some people get really uncomfortable. I've spent some time analyzing this, and I think it goes back to Frederick W. Taylor. In the early 19th century, Taylor proposed scientific management, suggesting that we separate the worker from the work. Henry Ford credited Taylor's work as leading to the assembly line, and making Ford a billionaire.

So, hard-coded in the business DNA of North America we have this idea that Management works ON the system, and contributors work IN the system. In that world-view, management is responsible for both eliminating general-cause variation (machines break down periodically, so invent a maintenance schedule) and handling special cause variation like customer complaints.

If management is working on every single exception to the process, then workers should be able to work by rote; contributors should be able to simply follow prescribed procedures. (This is a big part of what Aldous Huxley was talking about when we wrote Brave New World)

The problem is, what worked great for Taylor in 1907 isn't working so well a hundred years later. Today's high-tech technical contributor has more education, depth of decision making, and specialization than yesterday's foreman or line supervisor.

Instead of trying to predict the process, we need to recognize that the white collar worker *is* a manager of sorts. And that's not me talking; I'm paraphrasing the words of Peter Drucker.

What does that have to do with software testing? The best minds in the business are saying that dumbing-down software testing to prescriptive, proceduralized process is foolish. The the same time, the same discussions are happening in the development world, the requirements world, and the project management world.

Still, when I talk about sapient processes, people get edgy. Getting Frederick W. Taylor out of the IS Shop is going to be an formidable challenge.

So let's buckle up. I can't promise that it will be an easy ride, but it certainly will never be boring ...