So let's see here. We have a simple test project on it's face. Management hired us to figure out if the store if following a set of rules.
I hope you agree this is a real, critical thinking test problem. If we substitute "rules" for "requirements", we find this is very similar to a real test problem.
First of all, just because one cashier follows the rule is no /proof/ that the cashier will do so on the next transaction. Nor that the cashier next to her will do so.
For that matter, a typical grocery store has thousands of items. So finding out that ten or fifteen ring up at the correct price doesn't give us proof that the rest will be correct.
The best we can do is to say that the the process is capable of success - a term I borrowed for auditors a few years ago.
So what would I do? First of all, I'd walk into the store and look around. Yes, look around. Are the floors clean? Are the items stocked correctly? Are the lowest-paid workers, the baggers, wearing clean clothes with shirts that are tucked in? When the lines are down, do they go bring in carts from the parking lot, or do they sit around?
Then I'd go get my three cousins, Joe, who's 20 and looks 23, Billy, who's 21 and looks 25, and Sally, who's 22 and looks 18. Together, we would:
(A) Have Joe wring up a modest set of snack food and "forget" his id; she if he can buy beer. Have him use a different cashier, give his ID, and she if the cashier notice he just turned 20 last week and is 20, not 21. Try to argue with her about the math is she says 'no.' (Also, check the snack food for pricing)
(B) Have sally try the same trick with cigarettes.
(C) But a set of regular groceries, including items on sale, one from each major department. Carry a calculator with me, write down what I bought and the correct amount. Also buy two each of the items that should not be taxed (staples). Go to the checkout counter, and see if the correct amount is subtotaled, correct*0.05 is charged for tax, and sub+tax = total.
(D) Have the cousins try (C) on different shifts with different cashiers.
(E) Each of my and the cousins would try to buy beer at 6:10 AM, noon, 9:50PM, 10:10PM, and on several times on Sunday. We'd all try to buy non-food items on Sunday
(F) Clarify what tax should be applied on things that cost $0.19 (I think nothing) and $0.89 (I think four cents). Buy 100 items at $0.19 each, expect to pay $0.85.
(G) I'd try to buy cigarettes and beer; back then I appeared to be 45 years old. (Yes, I got younger. Hey, it's my story.)
I'd also try to get an appointment with the manager and see if we could have a friendly chat about the inspection, instead of doing it in a clandestine, us-vs-them way. I'd ask what systems he's put in place to educate and train the workers.
At this point, I'd have three types of feedback:
#1: I'd have looked and paid attention to the store (touring). There may be problems that senior management needs to be aware of outside the initial test. Also, a sloppy environment would tell me to spend more time looking carefully at the business rules themselves, because /some/ people at the store aren't doing their jobs.
#2: I'd have performed a first, sloppy pass at the initial requirements handed to me. If the store passed all those, I'd breathe a sigh of relief and start to wonder about the cashiers we didn't test. If some failed, I'd start looking and what failed to consider what to test on my next run.
#3: My walkthough or inspection with the manager would give me some idea of what the risks and weaknesses of the store might be - and it's strengths.
I would take the results of my initial round of testing and plan the next round. We can talk about that next time.
In the mean time
That was my sloppy first pass, and it's an honest first pass. Oh, perhaps I've given it more thought because I had to in order to /write/ the test, but it's not any kind of perfect, down-from-the-heavens answer.
Now's the part where you come in: You get to tell me what a terrible test it is, and what I've missed. Comments, anyone?
More to come.
Tuesday, December 09, 2008
Subscribe to:
Post Comments (Atom)
4 comments:
I guess my beef (not really a beef, actually--an observation) is that when there is variability on the part of the staff (software) there is no way that you can test to give assurance that the overall operation of the staff (software) is according to the rules (requirements).
Imagine a software product whose source code changes every few minutes. Every use of the software becomes a test.
So, you are correct when you talk about whether the store has the capability of following the rules. However, that doesn't say a lot about whether the store is actually following the rules.
Consider the governor of a state whose junior senator has just been elected President but has not yet taken office. The state itself may look, on the surface, to be well-run, productive, a great place to bring up a family, relatively crime-free, and all the rest. However, beneath the surface the governor seems to be trying to sell the Senate seat about to be vacated to the highest bidder, benefitting him personally in the process. You couldn't say anything about the underlying problem with that state when you look at the externals. (Not that this example could ever happen in real life...)
Conversely, while an untidy shop may look like it couldn't possibly follow the rules (requirements) and continue in business, many antique bookstores (for example) are run in that ideosyncratic way. Books piled up in corners, books filed every-which-way in the shelves, proprietor looks like an academic down on his luck, but he stays in business year after year because of the opportunity the shop presents to customers of finding a gem in the middle of the pile of dross the owner calls his "bookstore".
If the shop is untidy but the manager has had a friend or relative who was injured by a drunk driver, then perhaps the rules are more likely to be followed there than in a shop where the manager doesn't care about the rules on selling liquor but is a neat freak.
Tests of the nature that you describe are, I believe, carried out on behalf of large store chains by companies that specialise in such testing. Part of this is to ensure that the shop follows the chain's rules for presentation, pricing, and the like. Another part is to try to establish whether the average shopper will enjoy his/her experience there.
So what will your next pass be like?
First, management must have some suspicion that the rules aren't being followed, as they've hired you to verify the rules' application in this store. It would be a good idea to get them to tell you what evidence they have (financial irregularities showing up, stock being lost, the police have come to them saying that they suspect the store is selling beer to teenagers) before going in again.
Second, unless the staff are automatons you will be able only to demonstrate that on the particular occasions you chose to test the shop's application of the rules the shop either applied them consistently and correctly or did not do so. No general conclusions can be drawn from your tests, as the personnel can change (new shop assistant is young and lets his/her young friends buy alcohol or cigarettes) or the same personnel can be in different modes of awareness (experienced shop assistant tied one on last night, comes in with a massive hangover, forgets some of the rules or just can't be bothered to apply them correction; experienced shop assistant is short of cash and takes backhanders to allow teenagers to buy beer; experienced shop assistant discovers that his/her colleague is going to have him/her fired and decides to start selling liquor to minors to possibly have the shop's liquor license removed by the police).
Requiring the management that hired you to articulate what their real motives are for testing the shop would help in designing tests that would carry out their desires in the most efficient way.
I don't believe your tests were terrible, by the way. Just that the first pass would have been much more profitable if the motives of the management for hiring you were clear.
To bring this into the realm of actual software testing, a consultancy that I worked for in a galaxy far, far away was hired by a company that had contracted out the construction of a new software product to run physicians' offices to a firm that had once been part of their company but was now part independent. The contract firm was told that we were there to ensure that the software they produced was fit for purpose. The real motive, articulated to me when I was assigned there as test lead, was to demonstrate that the software was so badly constructed that it could not be fixed in a reasonable amount of time with a reasonable amount of money. The software was difficult if not impossible to install correctly, the receptionist couldn't make appointments, the doctors couldn't write prescriptions (but the system administrator could), and if a patient had two appointments in a day, the second one wiped out all record of the first. My opinion to the management was: "If this software is released, it will kill people." Management was pleased and terminated the contract. I would not have been able to produce the information that management needed to make its decision had they not told me what their real motive in engaging us was. (Had I said that the software was repairable, they would have accepted that verdict, by the way.)
This has gone on too long--sorry about that. Hope it's helped.
Any true testing problem will be embedded in the murk of a project problem. What is the project problem, here?
Examples:
- Maybe I have 100 stores to check in one week. If so, I might not even stop at that particular store. If I stop, I might do one simple test and leave.
- Maybe the store belongs to me, and everyone there knows me, and I am checking their awareness of the rules.
- Maybe this store is the subject of a pending lawsuit (perhaps a divorce action, and I work for the wife of the owner trying to get dirt on the husband... or perhaps I work for a competing store and want to put this one out of business), and I must carefully test to find any possible violation.
Project considerations come prior to settling on a specific strategy. That said, I think you have some interesting ideas.
One thing you didn't mention was testing under stress conditions. I suspect that many stores that would card a minor during slow times may not card when there's a long line of customers clamoring to check out. Similarly, taxes that are properly calculated by the point-of-sale equipment may be incorrectly calculated if that equipment breaks down and the clerks have to process things by hand.
-- James
Similarly, taxes that are properly calculated by the point-of-sale equipment may be incorrectly calculated if that equipment breaks down and the clerks have to process things by hand.
This has cropped up here in the UK, where the government has reduced the rate of VAT (value-added tax, a kind of sales tax that is added to the price of goods and services) from 17.5% to 15% for a little more than a year.
All the tills have been primed to calculate VAT at 17.5%, and there was little or no notice that the rate would go down. So stores and online shops have been scrambling over each other to lower their VAT but have found that the tills are "hard wired" to calculate VAT at 17.5%. They've mostly tinkered with the final prices so that the VAT is calculated correctly for the original price. This has meant that some stores have had incorrect prices, so have just left it up to the cashier to do the calculation and hand back the correct amount of money, and some have not done much at all and are still charging 17.5% but are only responsible for paying 15% of that to Her Majesty.
Sorry Matt, I missed the post where you defined the context. I actually saw it, but unfortunately, I read it using the preview feature of Google Reader, which did not show the whole post. So I missed your specific context clarification.
I guess that's part of the project context, too-- people who miss meetings and don't read the whole spec!
Hey, in case it's not obvious-- I love your blog and your ideas. You are shaping up as a true leader in the field.
Post a Comment