New Testing Challenge - II

Have you ever played pac-man? In pac-man, there is a simple, ostensible game: Eat the dots before any of the ghosts get you. However, there is a hidden game - a game-within-the-game - that is - figure out what makes the ghosts change direction. If you can do that, you can 'trick' the ghosts into going the wrong way, and, suddenly, the regular game gets much easier.

Software Development has it's own meta-games: Good project and product management can help us define success in more concrete terms, which makes it a lot easier to hit between the goal posts.

Does testing have a meta-game? I think it does, and I think it can be explained, described, and trained on.

Last week I posted a New Testing Challenge. On first blush, it's a simple problem: Make sure that the following list of business rules are enforced at a local grocery store.

I to be super-tempting for the reader to dive into problem-solving the surface problem. Why, first off, you test each of the business rules individually. Then you look for interesting bounds conditions, then you look for combinations of rules ... and so on.

And there's nothing really wrong with that - it's the eat-the-dots problem. Sometimes, you'll be presented with a "just test it!" sort of statement in the field, and knowing how to generalize to explicit rules and test those rules (and combinations) is a genuine skill. I would argue that when pushed to "just test it!", a good exploratory tester can usually come up with a dozen quick, easy-to-run test cases ideas, to make it look like they are testing, to buy time to figure out the meta-game.

Yes, the Meta-Game. The Meta-Game is figuring out who the customer is, what matters to him, and how much time, effort, and energy he is willing to invest in testing.

Knowing the customer helps you figure out what to test.
It helps you triage test ideas.
It helps you come up with test ideas.
It helps you know when to stop.

So, when presented with something like my testing challenge, a meta-game player will trying to engage someone - preferably the customer or product owner - in a conversation, along these lines:

"Who is the customer?"
"What problems are they worried about?"
"What is your expectation of 'good enough'?"
"How much time/money do you want me to invest?"
"Tell me about the project?"

Notice I say a conversation - because the customer may have uninformed expectations about estimates and quality expectations. It'll be a good bit of give and take - probably more than one round of it. I was very pleased to see Michael Bolton as the first responder, but even more pleased that he tried to actively engage me in the meta-game. I'll try to generalize the answers for everyone's benefit below.

The Answers
In the story below, you've been brought in as a consultant by Weiss's Management. They have recently had a number of complaints about just one store - the one in Frederick, and gave management a month to clean up the ship before calling you. The complaints are around the laws. One person noticed the store selling beer on a Sunday and cigarettes to minors, another complained because could not buy beer at 9:30 (something about daylight savings time and a clock), a third complained because he was charged tax on bread. A fourth complained that the store would not honor a sale price listed in the weekly circular.

Management wants to get these complaints resolved before they escalate to the newspaper or worse, the state attorney general's office. You live about six miles from the store, and have about a month to provide a written summary of known, systematic training issues along with recommendations for corrective action.

They expect you to work on this part-time, about 10 hours a week; your total budget is $1,000 US Dollars. (This is about $25/hour, which, in 1982 dollars, is reasonable consulting rates for a grocery store.) You live about 2 miles up the road from Weiss's and can also expense mileage.

What to do tomorrow
Ok, I've given what I believe is a good description of the problem. We could have some more give-and-take conversation, but I haven't intentionally left out any spoilers. You've let the Avante-Garde Michael Bolton types take a swing at this. Go back to Maryland in 1982. You just got out of your car and are headed into Weiss's Market. (Or, you can be at home planning, it's up to you.) What would you do?

Finally
If you look at the initial blog post as "requirements" to be "tested", I would say my initial description of the problem was probably at or above industry average. But it did not tell you any of the why that would guide your testing. Many such documents lack this context - and notice that you can not have a conversation with a document.

The way any organizations do development, it's hard to impossible to play the testing meta-game I've described above. That is just sad. The ghosts'll getcha every time. Or, to put it more plainly: without the meta-game, we are much more likely to find a bunch of bugs that don't matter and get called into the big meeting where the boss says "Why didn't QA find (the one that did matter)?"