Do you document your test cases?

We are debating the value of software testing standards right now on the context-driven testing list.


Here's my latest post ...

>and by the same token writing test cases doesn't make
>your testing worth any more than if you wrote
>NOTHING AT ALL.

For the broad, general case, James, I agree with you.

However, (to borrow a phrase) can you imagine a situation where this is not the case?

For example - instead of two pages of MS word documents per page, image one row in a spreadsheet, with five columns -

Who you should do
What we expect to happen
What actually happened
Pass/Fail
Notes

Your program is a Fahrenheit to Cecilius conversion. The requirements talk about the formula and give cases to 0, 32, 212 and 100 - but don't cover bounds or rounding.

The test cases cover bounds and rounding, and the customer views them and agrees.

In this case, test cases are a form of documentation. Heck, I wrote an article on it!


http://www.ddj.com/architect/199600262

My main problem with this is that using this side-effect logic, you aren't adding value to forensic and investigative process of figuring out if the software works.

In other words, your "test documentation" may help with something, but, at this point, it is not helping you test. So why call it test documentation?

Luckily, I can think of other examples. Say you have an API that does the conversion, and a test suite that looks like this:

my ($blnOk, $msg, $convert) = FahrToCel(5001);
ok(!$blnOk,'Limit of function is 5000');
ok($msg eq 'FahrToCel Limit Exceeded','And error message makes sense');
($blnOk, $msg, $convert) = FahrToCel(5000);
ok($blnOk,'5000 and under work fine');
ok($convert,2760);

----> These examples not only provide basic regression, they provide examples of the basic API for the maintenance programmer, and they get the easily, simple bugs out of the way so that we can focus on finding the real hard ones.

Sadly, I have to agree with James and Cem's comments. In the years that I have heard the mantra of "You must document your test cases", the few examples I saw had much more complexity and needless detail than the examples above, and we never automated at the API level with simple, straightforward code - meaning that the Return On Investment for the practice went way down.

Again, sadly, I suspect that's because the gurus had never actually, well ... done much of the stuff in the field.

And that, in a nutshell, is why I am involved in the context-driven community. :-)

Microsoft Tester Center

Microsoft announced it's new 'community' for software testers last week at STARWest.

The 'community' (which is a fancy name for a combination webzine, blog, and forums) is located here.

Cynical people will point out that this is Microsoft's attempt to win the hearts and minds of software testers, thus, in five years, increasing the number of QA Managers who purchase Microsoft test products and hire and train people who use those products. This creates a 'gravity well' for Microsoft products.

To which I say: Bah.

Testers are pretty independent thinkers by nature. At least Microsoft is trying. It could be worse: They could be ignoring us.

While I may be skeptical, it can't hurt to check out the content on the site; I notice that they have Scott Barber listed on the syndicated blogger page.

And for those of you who are really cynical: No, nobody from Microsoft paid me any money, gave any favoritism, or any special favors for this link. In fact, Alan Page asked me to take a look, and, if it was valuable, pass it on.

But enough about what I think - check it out, and let me know in the comments if this link is valuable to you.

Still to come: More Technical Debt ...

Technical Debt - III

I promised to offer three ways to limit technical debt – Personally, how to impact members on the team you contribute on, and how to impact members on the team you manage. I would like to start with the first. Please be aware: This is not a guide to managing your manager, but how to avoid sticking yourself in the victim role. This is about how to do good work that you are proud of, so you can take responsibility for it.

As Steve Poling pointed out in the previous comments - sometimes you have to take on debt to stay afloat. You need to ship the product today to bring in the revenue; some of that revenue can be used to pay off the technical debt. I don't mean to stick you in a binary thought pattern - the question of technical debt is not yes or no but "how much."

These posts are just about techniques to use to keep you in control of your process, instead of becoming it's victim.

Let's start by analyzing a conversation that is happening, right now, all over the world:

Manager: "We need the foo feature by Friday."
Do-er: "I need two weeks."
Manager: "We need it Friday."
Do-er: "To do it right would take two weeks."
Manager: "Look, it doesn't have to be great. Just Done."
Do-er: "Well, I guess I could just hack in a conditional ..."
Manager: "Can you have that done by Friday?"
Do-er: "I guess. If I work in the evenings a little."
Manager: "GREAT! Do It."

Before we re-write this conversation, a little applied psychology here: By backing down and "finding a way", the contributor is training his manager to ask for unreasonable things. Think about it. The manager asks for more than what is possible, the contributor puts up a faux fight ... then yields. (Worse, the contributor is saying that his first estimate should not be trusted.)

Next time the manager needs something done "real quick now", who on the team is he going to go to? And what is he going to expect?

Also, notice that the quality shortcut is implicit. The contributor never says "In order to hit the date, I will short quality. I will add in a hard-to-understand, hard-to-test feature that will be undocumented and hard to maintain. I will increase the cost of all future development on this module by about two percent."

Instead, he said "Ok, boss", then maybe went home and kicked his dog. We don't know, but the cycle of miscommunication that starts with the boss (what does "gotta have it" mean, anyway?) has been continued.

As an industry, we have to stop doing this. We have to grow up.

Let's try this again:

Manager: "We need the foo feature by Friday."
Do-er: "I can have it done the following Friday – Oct 14"
Manager: "We need it Friday."
Do-er: "As it stands, given our departmental quality standards, I don’t know how to do what you are asking. I can, however, present a few options."
Manager: "Go on."
Do-er: "We could assign Sarah to write the frobble sub-feature. Joe could do the system and integration tests, so I could write the code. We could skip our quality standards, or we could ship on October 14"
Manager: "Sarah is busy doing XXY. Joe is busy doing ABC. We need that code on Friday!"
Do-er: "Ok. Send me an email approving the quality slip, and I'll do it."
Manager: "um ... what?"
Do-er: "I can hit the date if I take certain risks. If you approve and take responsibility for the risk, I can do it; I just need an email."
Manager: "What happens if it doesn't work?"
Do-er: "Well, you took responsibility for it and approved the risk. I'd say it's about 10% that we have a minor bug, 20% that it’s a show-stopper."
Manager: "There can be no show-stoppers!"
Do-er: "Ok. We could assign Joe to do the system and integration tests ..."

You see where this is going. The contributor is offering real options to management to make explicit tradeoffs. The manager, of course, doesn’t WANT to make those tradeoffs, and doesn't want to take responsibility for those tradeoffs – but we're not letting him off the hook. (Yes, I'm tough on managers here. Don't worry, I will be tough on contributors later.)

Is this confrontational? Yes ... a little. The key is to present three to five very valid options for manager. That's not a confrontation; it's asking the boss to be boss, to make the big decision.

Now, consider: If anything was less important, you could find someone else to help out. Thus, if everyone else in the department is working on something else and can't be spared, this functionality needed on Friday is actually the least important thing the team is working on! Also Consider: If we back down at this point, we sacrifice our life, make a compromise that is a violation of our professional standards, and incent management to continue the practice.

Finally, consider: Assuming you are competent and this is work on a legacy system, there really is no one else who can learn this. If the boss fires you tomorrow (he won't) and assigns someone else, it will take them two weeks to even figure out what the software is doing. He would be a fool to remove us. Unless annual review season is this very week, we have nothing to fear, and even then, we're talking about losing a percent or two off our raise; three at most.

In order to vastly decrease technical debt at the personal level, all we have to do is grow a spine. Of course, that is simple, not easy. And when it is a hack that will keep the cash flowing and prevent layoffs, it may just need to happen. (Hint: Those don't really happen that often.)

The example above got into some pretty ridiculous circles at the end. Sometimes, a manager or executive will not 'play fair', and will try to get the circles to continue until you give up, frustrated and tired. They may use coercion, power, or the threat of power.

By framing our shortcuts as choices with explicit compromises, we can show the true cost of cutting quality, and, six times out of ten, the result will be a choice with less technical debt. Maybe not none, but less.

The example above was contrived, so I will end with a real conversation from last week:

Project Manager: "Will we have the bar function today?"
Me: "No."
PM: "But ... we need it today!"
Me: "It doesn’t even have customer signoff. I couldn’t release it if I wanted to."
PM: "We promised bar TODAY!"
Me: "Was Johanna, our customer acceptance tester, in the room for that commitment?"
PM: "No."
Me: "Was I?" (I honestly did not remember; you could tell by my tone of voice)
PM: "No."
Me: "Was the engineering team lead?"
PM: "No."
Me: "Well, then, think of it this way. When the full-time analyst and the full-time developer for the bar function left, Johanna and I took over, in addition to our other full-time responsibilities, and we are going to deliver bar two or three days late. I don’t think that is a problem. I think it is something to celebrate!"

And I walked away quickly.

Sometimes, the way to be most effective in your job is to act as if you don't care about it.

Technical Debt II.5 -

I've been struggling for the past two weeks to put up a post on technical debt that doesn't sound like passive-aggressive "how to manage your boss" or "how to trick your boss."

That is not my intent.

So, while I may post one of my working copies later, I'd like to make this clear now.

Imagine a carpenter who is a professional craftsman. The carpenter is brought into a job and told to "hurry it up - we have to have the framing complete by November 1st."

Now, carpentry is more fungible than technical work; if you have a standard pattern, it is much easier to throw a half-dozen extra bodies on it to hit the date. The carpenter brings this up to the general contractor, who says "No. Work unpaid overtime if you have to, but HIT THE DATE!!"

What is the carpenter going to do?

Well, he could slack off a _bit_. He might save five or ten percent of his time by doing a poor job, that would decrease the lifespan of the house substantially.

But as a homeowner, you don't want him to do that, do you?

Moreover, the carpenter probably went though an apprentice program for three or more years, where he saw reasonable standards, and, possibly, heard a journeyman say "no" a couple of times.

If he loses the job, so be it. In most cases, carpentry is contract work, and there are plenty of people looking to build houses. The point is, he is responsible for doing high quality work to an ethical standard. To paraphrase Richard Bach, you can either own your process ... or be it's victim.

We lack and understand of craft in Technical work. The job training programs we have are largely academic, not On The Job Apprenticeships - we don't know how to respond.

So we blame management, take shortcuts we should not, and act like victims.

Shame on us.

Shame on us!

Please keep that spirit in mind when I post the next article in the series. I am still not 100% confident in it. If you know me personally, please feel free to email me and ask for a preview before I post it; I am interested in your opinions.

--heusser

Technical Debt - Interlude

Something to read while I am on vacation ...
Tom DeMarco's Clasic, Excellent, Famous Essay "Why Does Software Cost So Much?"

Update: I am creating a new label (or "tag") for this post, titled "Software Engineering Classics." As I add links to these classic articles and books, you will be able to search for "Software Engineering Classics" on the blog and get the whole list.

My goal is to provide a quick link of everything you should read in a good master's program in Commercial (not-government) Software Engineering.

--heusser

Technical Debt - II

Before I jump to solutions, I’d like to take a moment and talk about entropy.

(From the site "If you assert that nature tends to take things from order to disorder and give an example or two, then you will get almost universal recognition and assent. It is a part of our common experience. Spend hours cleaning your desk, your basement, your attic, and it seems to spontaneously revert back to disorder and chaos before your eyes. So if you say that entropy is a measure of disorder, and that nature tends toward maximum entropy for any isolated system, then you do have some insight into the ideas of the second law of thermodynamics." -> It's a good read.)

If you think about it, every human accomplishment is a temporary battle against entropy.

Great Buildings are built, last hundreds of years ... and fall.

Cities become kingdoms that become empires, reign thousands of years ... and fall.

Even the great sphinx of Egypt, built in a warm, dry climate with little weather, is feeling the effects of age.

So it is with software. The typical software system might last ten years, fifteen if it is wonderful, but, eventually, the center will not hold. The code will not work on the new operating system, the company will fold, or the company will merge and turn off the legacy system in favor of the new.

In some ways, developing software is a battle against entropy. In some cases, just getting the code into production is a losing battle.

Once the code is in production, the battle starts all over again. Each maintenance change is the opportunity to write a clever hack or make an improvement. In each case, we can trade a little time now for a system that is slightly worse. With each change, the overall impact is un-noticable – but add them up, and we’ve got a mess.

At the same time, no one wants to develop the perfect, new new thing, with beautiful code and perfect documentation – only to be six months late, miss a market window, and have the company go out of business.

The question isn't "to accumulate technical debt, yes or no?", but instead "Just how great can we make this system given our constraints?" (Little things like time and money), "How can we trade off or ignore those constraints in order to have less debt?"

In other words "How much can we invest in this system to save time and money later?"

That is a hard question. I assert that is must be asked and decided consciously. Socrates said the unexamined life is not worth living, and I would add that the unexamined process results in ... crappy products.

We all know the "right thing to do." It's just hard to do it.

So I would like to propose three ways to lead conversations towards less technical debt, focused on –

A) What *you* can do as a technical contributor
B) What you can do as a technical contributor to influence other members of the team
C) Options for management

See you next time ...