I've been spending a good deal of time lately thinking about the cognitive process of testing: What makes a good test case? How do we "know" that the software is fit for use? And why is that an area of such little interest in many test communities?
One thing that can help are examples that we can draw rules of thumb from. The typical example is the triangle problem, where we have certain rules for types of triangles, an input for sides, and try to figure out if the software is calculating the correct type of triangle.
The triangle problem is a great example -- as long as you did well in Geometry in high school. I've been in search of other examples that don't require telling a scalene from a isosceles.
That said, you can test anything that consists of rules and examples. So here goes ...
Picture It: Maryland, 1982. You've been hired as an auditor to check to see if Weiss's Grocery Store in Frederick is enforcing the sale laws correctly. You turn on your Plymouth Horizon and leave Middletown, taking Route 40 east, over Braddock Heights - in the distance you can just barely make out the famous clustered spires of Frederick.
... and then you're turning into the Weiss's parking lot. Before entering the store, you grab your clip-board and review the rules:
Weiss's five basic types of products:
A) Staples - Break, Milk, Cheese, Water, Flour, Sugar, Juice)
B) Non-Staple Food such as Soda Pop, Ice Cream, Frozen Food, Candy Bars, Everything else
C) Alcoholic Beverages like Wine and Beer
D) Cigarettes
E) Other items like toys, magazines, pencils, paper cooking supplies, and so on
The following laws apply:
1) Maryland State Sales Tax of 5% applies to all items except staples
2) Only food items (staples and non-staples) can be sold on Sundays
3) All times must have a white tag that lists the price of the item; that price must be correctly charged by the cahsier
4) Listed prices in the weekly 'sale' circular must be displayed with a yellow that; that price must be correctly charged by the cashier
5) You must be twenty-one years of age to purchase alcoholic beverages; the cashier should card you if you appear to be under thirty-five
6) Alcoholic Beverages cannot be sold between 10PM and 6AM
7) You must be eighteen years of age to purchase cigarettes; the cashier should request identification if you appear to be under thirty
Your job is to ensure that the correct laws and taxes are applied, and items are sold for the correct amount. (If you really want to be testing software, imagine that the rules must be applied through self-service checkout stations.)
Challenge: How would you test this? Defend your answer.
UPDATE: Yes, I welcome clarifying questions, and NO, you don't need to test the website. I've linked to that solely as background information. The test is theoretical and set in 1982.
Friday, November 28, 2008
Subscribe to:
Post Comments (Atom)
10 comments:
There are many conditions to check here, but in order to be helpful, I need to know some things first. I'm going to assume that you're my client.
0) Is it okay if I ask you some questions? Assuming Yes,
1) Do you want a quick, deep, or a practical answer to the question, "How would you test this?"
2) Has anyone else tested this?
3) What's my timeframe?
4) Is it Sunday? When will it next be Sunday?
5) What are, in your estimation, the most serious risks?
6) What resources are available to me?
7) Who else can I talk to about this? Clerks? Customers?
8) If I see violations of laws other than the ones you've set out, are you interested?
9) What are my references for correct prices, categories, sale items, and so on?
10) What do you want my report to look like?
---Michael B.
This is a compliance question. In a store that becomes a task for managemnt enforcement. In software it becomes a case of defining rules, generating outcomes, and detecting violations. Since I can only generate the outcomes by having people shop, it is ipso facto a store, and hence a management enforcement problem, not a software task.
If you want a testing question that will stick in the software realm try recasting the compliance requirements as something to enforce in the software, not in a store. E.g., accessibility standards, or some other compliance requirement.
So how would I test this? I would hire a time and motion team with retail expertise, a management accountant, and an inventory contractor.
Here are my initial tests:
1) Maryland State Sales Tax of 5% applies to all items except staples
A. A variety of non-staples items - does the total tax end up 5%?
B. A mix of non-staple and staple items - is the total tax 5% of the non-staple items only?
C. Variety of items with regular/white/yellow tags - appropriately taxed?
D. Ring up various items that are taxable - remove items - does tax total reflect the changes?
E. Total up the sale, add a few more items that are taxable - does tax total reflect the changes?
2) Only food items (staples and non-staples) can be sold on Sundays
A. Attempt to buy alcoholic beverages on Sunday - is this item rejected?
B. A variety of "other" items along with staples/nonstaples - is any item rejected?
C. Attempt to purchase non-Sunday allowed items with yellow/white tags - purchase allowed?
D. Self service - if an override is needed from management before a customer continues to cash
out, is an alcoholic beverage then allowed to be scanned and purchased?
3) All times must have a white tag that lists the price of the item; that price must be correctly charged by the cashier
A. Items with only white tags - correctly charged/taxed?
B. Variety of tags - correctly charged/taxed?
C. Variety of tags and non-staple/staple items - correctly charged? appropriately taxed?
D. Alcoholic beverage with white tag on Sunday - purchase available?
4) Listed prices in the weekly 'sale' circular must be displayed with a yellow that; that price must be correctly charged by the cashier
A. Items with white tags, yellow tags, taxable/non taxable - correctly charged/taxed?
B. Only yellow tagged items both taxable/non taxable - correctly charged/taxed?
C. Yellow tagged alcoholic beverages on Sunday - purchase available?
D. Does removal of a yellow-tagged item remove the weekly "sale" price or the original price of
item?
5) You must be twenty-one years of age to purchase alcoholic beverages; the cashier should card you if you appear to be under thirty-five
A. Attempt to purchase by 20 year-old who will be 21 within short time (days/weeks/months) of the sale - purchase available?
B. Attempt to purchase by 21 year old who has no ID - purchase allowed?
C. Attempt to purchase by 35 year old with ID - purchase allowed?
D. Attempt to purchase by 35 year old with no ID - purchase allowed?
E. Attempt to purchase by individual with false ID - purchase allowed?
6) Alcoholic Beverages cannot be sold between 10PM and 6AM
A. Attempt to purchase alcoholic beverage when in line at 9:58 and cashing out at 10:01 pm -
purchase allowed?
B. Attempt to purchase alcoholic beverage at 10:01AM - purchase allowed?
C. Attempt to purchase alcoholic beverage at self-service cash register by underage person
with no ID - purhcase allowed?
D. Attempt to purchase alcoholic beverage at self-service cash register at 10:05PM when
marked with white or yellow tag - recognized as alcohol? purchase allowed?
E. Attempt to purchase alcoholic beverage at 5:59AM - purchase allowed?
F. Attempt to purchase alcoholic beverage at 5:59PM - purchase allowed?
G. When purchase of alcoholic beverage is disallowed - do the rest of the items cash up
correctly?
7) You must be eighteen years of age to purchase cigarettes; the cashier should request identification if you appear to be under thirty
A. Same types of tests as for alcohol with age modified to 18.
B. Self- service: ID checked for cigarettes, then alcoholic beverage rings up - recognize need for ID check/age requirement?
C. Self-service: Purchase of alcoholic beverage with appropriate ID check, then purchase of cigarettes - recognize no need for another ID check?
Hello Matthew,
You convinced me pick up your challenge. So I started with the meaning the be short in text. Sorry I failed on this one.
Looking to the triangle paper one statement triggered me to come to an approach: “There is no one correct theoretical answer to any testing problem. The answer depends on the
assumptions and judgement calls which the test planner makes.”
This gives me some freedom to define my approach and perhaps preventing me to fail on your challenge.
Another statement you made: “…and then you're turning into the Weiss's parking lot. Before entering the store, you grab your clip-board and review the rules”
This made me assume that I didn’t spent time already to think about a testing approach or test cases. As the action reviewing rules indicates to me that the customer has already some expectations. This expectation in my opinion can be starting with testing as a proof of certain quality should be given as soon as possible.
As all rules and types are available I can imagine that the customer believes I have all the information needed, otherwise I would have ask for more information before I started the journey towards Weiss’s Grocery Store.
When looking at the rules I noticed that some rules might have impact as I have to assume that for instance the system should be able to validate the age. This means that on a card some kind of technology must be available.
Based on these assumptions and statements I want to claim that the customer is asking me the following: “Please help me, now!”
Although this is a short statement, it is strong: as it tells me that the customer needs my help and expertise. And he want is now and also the exclamation mark tells me that it is urgent and no time to lose.
This statement tells me also that I don’t have time to waste by investigating the project, system and organization thoroughly to define a solid TMap minded approach.
In this case I’m suggesting to tell the customer what I’m going to do and how I’m going to do it.
In short this approach can be defined by the following steps:
1. Check which laws are applying to the system under test
2. Tell in simple phrases what you are going to test
3. Explain that after every test I’m explaining him if I think more tests are needed in that area.
4. Check with the customer if he thinks I explained well enough if I tested enough.
5. Check if he sees more opportunities to check
6. If he sees more opportunities in a certain area, explain him that executing those extra tests other areas might be skipped.
7. If he decided to continue, I continue
8. After the race is over I explain him that I did everything I think was necessary and possible within given timeframe
Statements to start with:
1. There are types of products available
2. Those products are classified under the correct type
3. There is a tax mechanism in the system
4. All items can be sold
5. Products have a price listed
6. A product can be sold for a correct price
7. There is an age validation mechanism available in the system
8. There is a time check mechanism available in the system
9. There is a pricing mechanism available in the system
10. The price mechanism is able to count and calculate total price including taxes.
11. There is an option to print the price-coupon
12. There is a barcode reader to scan products which are bought
Proposing these statements I would tell the customer how much time I think I need. If he thinks it is too much I will adapt my strategy.
During verifying these statements I might become more specific as after trying some types of products in different order. Perhaps adding and removing items from the list
After verifying these single statements I would try to challenge the system. What about:
1. It is possible to buy cigarettes and buying alcohol is denied
2. It is possible after a denial of an item due to age to remove this item from the list and continue
3. It is not possible to buy items when the credit card has no money on it
4. If one error is shown, there must be others there in the same area or same logic.
5. It is possible to buy items which are covering almost all rules and leaving the store without chased by the police.
I’m sure that this is not the whole list. As there are always questions to ask, situations to define, and risks to cover.
Mainly the point of this approach is that the customer needs your help. You can spend a lot of time on investigation how to start. You can come up with lots of reasons why not to start. And there might be even more reasons why you will not be successful. At least you should be able to convince that you are the person who can help the customer. Also you should be able that you are able to tell the customer why you are doing for what reason.
I think it is better to give the customer clarity in what you are tending to do as soon as possible then letting him pay for the time you needed and afterwards you cannot be successful.
I have to admit that reading this challenge first I had the feeling that it is hard to explain in detail and completely how to approach this challenge. I didn’t expect that it was this hard. And this bothers me a bit as this is one signal to me that I might have made it too complex and losing some important points.
With regards,
Jeroen
I was quick draw McGraw with whipping out test cases that I was running in my mind, but there has since been a few revelations to me...
1. Jeroen's comments about the customer made me have to scold myself for forgetting "the people that matter" in the challenge.
2. Matt provided a link to Weis Markets, here there is information that provides a valuable insight into the what and why of the situation. (Go on, click on the link...)
3. Based on what I have seen on the Web Site, the customer will need to have an understanding that although testing can cover a lot of ground, it will be incomplete. There is no way that every situation and every combination of items, taxes, etc. can be tested.
4. I am still creating test cases in my head :)
***The people that matter is a phrase I borrowed from the Rapid Software Testing course by James Bach and Michael Bolton.
I have been sleep deprived for close to 24 hours. Here's are some questions that made it through the serotonin:
1) Are pharmaceutical items sold? If so, would there be a problem selling copious amounts of cough syrup
to minors?
1.1) If the above is permitted, would they allowed to be sold on Sunday?
2) What if I ordered items from another location like Washington D.C., or NoVa. How is the 5% sales tax
handled? If it is supposed to be left off, is it _really_?
3) Can items C, D, and E be sold during Religious Holidays? This is assuming that the restricted items on
Sunday is due to a moral observance of some kind (e.g. church, cultural day of rest, etc).
4) What is the Maryland Sales Tax on staples? Is it non-existent? Higher? Lower?
5) How would the cashier go about "correctly charging" for the price listed on the item's white tag? What
are the consequences of failing?
6) When are the weekly sale prices in effect? When do they revert?
7) Would it be a problem if the cashier were color-blind - unable to discern White from Yellow?
8) Are White and Yellow distinct enough so that they are not mistaken for each other?
9) Will the yellow tags be stored away from harsh light sources that would eventuallymake it fade?
10) Will the white tags to stored in suc a way that they do not yellow with age?
11) Who is responsible for determining the price and tagging the appropriate items?
12) If a customer buying alcohol appears to be under 3 feet tall, will should they be carded by they cashier?
13) Alcohol cannot be sold between 10PM and 6AM. Is this an 8 hour time frame? A 32 houur time frame?
14) When does the store open? When does it close? What Holidays does it observe, if any, that would cause it to be closed?
15) If a customer buying cigarettes appears to be under 2 1/2 feet tall, will should they be carded by they cashier?
16) What visible indicators would consistently identify under-age people.
Hi,
I am always up for a challenge. I have created a blog that has my questions/tests to the challenge so others can see it too.
Check it out at:
http://www.allrealms.com/profiles/blogs/testing-challenge-are-you-up
Thanks,
Jay
Not enough time to play the full game, but wanted to dump a few thoughts.
1982 is a bit of a challenge since it means that the store is using a regular register rather than some sort of point of sale device (i.e. something programmable).
The reported errors are are human errors - ideally, we'd want to find a way to prevent them...
but I just remembered that I'm playing the role of the auditor and not the "fixer"
In that case, I'd build a moodel (fancy speak for drawing a flow chart with constraints), then I'd check to make sure that the rules were being followed. Of course, I'd discover that errors were still being made, so I'd call my cousin Guido and tell them there's a store in Frederick that's just about to hire a consultant to help them stop making so many mistakes.
So Guido would come into the picture and figure out if he could jury rig the register to add tax to only the appropriate items, ask for ID on all purchases that contain alchohol and refuse to ring up alchohol between 10 & 6 (there's a boundary issue there that would need some additional discussion).
If you can't rig the cash reegister, you could work out a motivational system where bonuses could be based on workers making fewer mistakes, and perhaps even fun a party for all employees when going a certain amount of time without making a mistake. Feedback is important here, so you could even have a sign that says "n days without a customer complaint" visible to all employees. The feedback system could encourage them to be a bit more careful. This isn't a trainining issue - it's a human performance issue (again, assuming that I can't create software for the cash register), so I would approach it as an organizational change model (or at least that's what Guido would do).
Interesting example of transposition between the original Creative Chaos requirements list and the one on Jay's site.
My favourite: "items" become "times".
Lesson here for those who would have distinct requirement and test documents.
Posted without reading other comments, so apoligies for any duplication, also sort of stream of consciousness:
Really I'd need a LOT more info than this to come up with a good test plan.. So the first step is asking questions.
How much resources do we have for this testing? what timeframe to complete it?
Which areas is the store most concerned about, where do they have the most concern that they are not in complience
Which areas pose the greatest risk? e.g. if they mischarge sales tax on a few items and have to make it up, that might be small potatoes, whereas if they face a substantial fine for selling alcohol or tobacco to a minor, it could post a substantial risk to the store (perhaps even having their right to sell such things revoked)
Item 4 a yellow what? tag on the item, tag on the shelf.
Do we have access to definitive standards for the classification of things (e.g. boundaries between staples vs other food ideams)
For transactions that are time gated, what is the authoritative standard for the 'time' (does register have a clock, is there clock on wall visible from checkstand)..
Also for time gated transactions, the transaction takes time (to ring up whole order etc) does the process need to start before the allowed time occurs, or must it complete before that time occurs.
Is there any grace period on the time gated items..
What kind of register system are we talking about, (some by NCR IBM etc were fairly sophisitacted at that time) and is the register itself expected to act as gatekeeper on any of this or is the checker expected to know what is taxable, staple, etc.
(this is important because if it's the checker, then we have to test these things against each checker)
Is checker or some other system expected to be able to identify fake id's or is merely asking for the ID and checking age according to it enough.
Does the register support UPC codes? (UPC has been around for about 8 years in 1982) if so how are non UPC items entered?
Do we need to test the shelf labeling etc.
are product price labels expected to indicate the product's category?
How extensive does the testing need to be? e.g. do we need to verfiy that every items is wrung correctly, or just a sampling from each category,
Do we need to validate that the white tags with the price are in fact correct, or only that the price input to the register matches the tag (even if it's pretty obviously mispriced such as 1cent or 100$ for a bag of flour)
I could go on, but you get the idea..
(hmmm interesting, your first commentor Michael had many similar questions)
Post a Comment