Budgets, Badges, and Badgers - III

Earlier in the week I introduced the Malcolm Baldridge Quality Award, and my doubts about it -- yesterday I posted a quick summary of my opinion.

It is a serious subject and deserves a serious answer -- I do believe it is time to get specific.

What is the Baldridge award?

According to it's homepage, the mission of the Baldridge program is to "improve the competitiveness and performance of U.S. organizations." Mr. Borawski defined it by saying that:

The Baldrige Program serves to:

1) Identify and recognize role model organizations
2) Establish criteria for evaluating improvement efforts
3) Disseminate and share best practices

To put this into my own words, I suppose the best, most competitive companies have ideas that can be used by other companies. So we should find those organizations, hold them up, and share their ideas. If everyone were to share these ideas, why, we would see increased productivity, which means more goods and services created, which means an improvement in quality of life in our communities and increased competitiveness abroad -- everybody wins. Sounds good to me, eh?

Except ... wait.

Exactly who is deciding what 'best' means?

In Wall Street, we have one way of deciding who's best: The wisdom of the crowd. People buy shares in companies they like, and sell stock if they do not like them. This creates 'winners' and 'losers.'

Likewise, on main street we have another kind of voting: The pocketbook. People purchase services from companies they like, and can complain about or boycott companies they don't like. If enough people stop buying your stuff, you go out of business; if lots of people buy your stuff (and you don't mess up along the way), you can become Wal*Mart. (Or at least Target, maybe?)

I call these sorts of systems "market based" because they allow people to vote with their wallet. This means a company needs to sell goods or services people want at a price they can afford -- or the company goes out of business.

The Baldridge Award replaces this measure with it's own wisdom. Now, for public service organizations (a police station) and non-profit organizations, you might have to do something like that.

But that's not the point Matt, sharing of best practices is!

The ASQ (and most other Baldridge defenders) are quick to point out that the program is not about the award, it is about sharing of best practice.

ahh, there is that word. "Best Practices." I have to tell you honestly, that term creeps me out.

Best practices is not an engineering term.

Engineers of any stripe, software or mechanical, do not talk about best practices. They talk about tradeoffs, about losing something less important to our group at this point in time in order to get something more important to our group at this point in time.

The groups and the points in time might change, so no practice is ever "best."

In fact, I belong to a group called the context-driven school of software testing that censures the term best practices.

By censure, I mean outlaw. If you use the term at one of our conferences, you'll likely be told to use another term. (More likely, your submission won't be accepted, and you'll get an email explaining why.)

When I hear the term "best practices", what comes to mind is marketing, sales, hype, and sloppy thinking.

Sure, the baldridge program might share practices - but are the cash-handling practices for a bank going to apply for a gas station? How about for a one-person small engine repair shop?

It's likely that they will not - that implementing the practices can cause more harm than good.

What is the criteria for the program?

The criteria for the business side of the Baldridge award ("performance excellence") is a seventy-seven page PDF that provides a framework for evaluating a business. The evaluation terms include things like "Measurement, Analysis, and Knowledge Management", "Workforce Focus", "Customer Focus", and "Results."

Now this is where I have to get a little personal and base my opinions on my experiences, just a little bit. You see the Baldridge program does have a relatively small budget - so to evaluate companies it relies on volunteer "examiners." Of all the aspects of the award, I probably like the volunteer/examiner program the best.

Ideally, I should become an examiner and take the training myself - but I've met enough examiners and talked to them to have some idea of what the program entails. Suffice to say that the program evaluates companies according to a value system - to see if the companies work is stable, predictable, and measured enough that it can experiment with a change and known numerically if the results are sufficient.

The examples I have seen were around hospital wait time, and time-to-execute on certain recurring operations, like perhaps a blood draw. Yet there are significant challenges with measuring knowledge work, which is an increasingly large part of the American economy. Even if the work is repeatable, I might question if it is valuable -- for example, Barnes and Nobles had a great, stable, repeatable system to produce books in 1995 ... right up until Amazon.com came along with a disruptive innovation and took away their business model.

That idea of disruptive innovation being more and more dangerous to a business that becomes more and more highly specialized -- isn't mine alone -- it is a hallmark of modern risk management.

So sure, running your company "by the numbers" is one ideal of business management, but it's not the only one, and it bothers me that our government would institutionalize it. (This is probably the one area I know least about the program, but I am open to learning more, and I calls 'em like I see 'em.)

Should our government be doing this?

I may not be a constitutional scholar, but I've read the thing and I see that the government has certain powers elaborated in the constitution and those not elaborated are delegated to the states. I do realize that recently, as a nation, we have not paid a whole lot of attention to the document, especially it's intent. Further, I realize that the Federal Government is granted the right to regulate interstate commerce, and that power is used to justify several large government organizations like OSHA, and NIST, compared to which Baldridge is a drop in the bucket.

Further, our Federal government is one of the world's largest employers; I think we employ something like two million people, and that's before adding government contractors like LockHeed-Martin.

On the business side, I can't see a reason the Federal government would see spreading performance excellence "best practice" as within it's role. The appeal to 'national interest' seems to go against the historical reason we exist as a country -- we exist because we wanted the government out of our business. In addition, it seems to smack of centralized planning to me -- something the russians tried after the second world war -- to "decide" the "right way" to do things and to spread that out to every company, instead of letting the free market decide.

It didn't work out that great for the Russians.

We need less of this, not more. According to my value system, we need lless scripted behavior and more thinking -- the group we need to hold up as exemplars is most likely the liberal arts tradition.

But that's my opinion. I don't need government money to compete; give me a microphone called the internet and let people respond if they want to.

Conclusions

By now you realize that I'm not keen on the Baldridge award. Of course we should de-fund it. More than that, we should ask how a program like that ever got to be funded in the first place!

But that brings an interesting question. If Quality is what Paul Borawski calls "the set the concepts, techniques, and tools that connect good intention with realized and sustainable outcomes", how is it possible that we have such a different understanding the role and benefits of the Baldridge award? Wouldn't you hope we came to the same conclusion, not conclusions that were wildly different?

This is a contradiction and it's probably wise to check our assumptions.

The simplest explanation I can think of is that Paul and I have different values; that he believes that more centralized planning (or "sharing" if you have a lighter touch) combined with management by the numbers will lead to better outcomes for the United States, or even the world.

I've made my case against this worldview.

I would be pleased to see a strong reply; I'm interested in the discussion, or, possibly, an explanation of what I am missing - the benefits that the Baldrige programs adds that I am failing to take into account.

Either way, as a tiny little niche industry, I suspect we 'quality' people have a fair bit of work cut out for ourselves.

It's an exciting time to be a tester.

Budgets, Badges, and Badgers - II

Last time I introduced the Malcolm Baldridge Quality Award, and my doubts about it.

I wrote a serious, detailed response that I will post tomorrow. In the mean time, though, I would like to give the five-minute version. It goes something like this:

I have a number of concerns about the Baldridge award, but chief among them is the worldview it seems to be advancing: One in which the ideal business has a defined process and can be managed 'by the numbers.'

In my experience, this kind of business is *both* especially vulnerable to black swan problems, but also vulnerable to disruptive innovations. It pursues a form of maturity that I do not agree with.

I'll debate the details tomorrow - for now, Barry Schwartz's presentation "Practical Wisdom" says it all for me:



For the record: I'm with Barry.

Budgets, Badges, and Badgers - I

I don't talk about politics much on this blog, but there are some interesting things going on right now with the USA Federal budget that seem relevant.

Consider, for example, our massive annual debt. Every politician seems to agree this is a problem -- but have you noticed that few of them have any detailed ideas on how to cut it? If you push hard they'll come up with a statement such as "going over the budget "line by line", but nobody wants to get specific.

Here's why: Every line item on the federal budget has a special interest group supporting it; that is why the line item exists. If you threaten to cut that item, you've just made an enemy of that special interest group.

Threaten to cut medicare or social security, and the baby boomers and senior citizens won't vote for you. Cut medicaid and you'll lose the disabled and lower-incomes -- same with Head Start or Welfare. U.S. unemployment is hovering around ten percent; add family members supported by unemployment, and you've just ticked off a large group of people.

In other words, if you want to cut anything, you can't get elected. So we come up with silly ideas like a federal pay freeze that will save a billion or two, but combine in with stimulus spending that adds up to hundreds of billions a year. (To help visualization, here's short video explainng the last spending "cut".)

Philosopher's call this "the tragedy of the commons." By each group lobbying for it's own individual best interest, we slowly destroy the system as a whole.

And by every special interest, I mean it. Did you know the 'quality' special interest has our own line-item?

It's called the Malcolm Baldridge Award, a federally-chartered award to recognize performance excellence in the categories of public, private, and non-profit organizations.

A little googling shows me that the Baldridge award program costs out Federal Government about twelve million dollars annually. As a taxpayer aware of the tragedy of the commons, I'd be inclined to sacrifice the award off the bat.

Then I read this blog post by the executive directory of the American Society for Quality, taking the opposite position. It made me pause and reflect.

What criteria should we use to judge the Baldridge award?

A few things occur to me. First of all, we know the cost, but what is the value? In order to make an informed decision, we would want to subtract the cost from the value -- to find out of the award is a good investment for the American people. We would want to find out if the award is good for society. If that comes out positive, I'd want to ask if the award is within the role of government -- is it the kind of thing the government should do, and, if yes, if it is the kind of thing allowed by the Federal Republic defined in our constitution.

All that said: Let's take a look.

More to come.

Test Management Certification

So I'm trying to figure out my 2011 (and beyond!) professional development plan.

I've got a lot of ideas -- I like to try a lot of things at the same times and see what sticks.

In 2009, I started a formal, zero-profit, non-commercial school for testing known as Miagi-Do, and that has gone well. So well, in fact, that in a recent email thread on test certifications, someone wrote:

I confess that all I know about Miagi-Do is that all the people who have mentioned they are Miagi-Do rated in some way are people I respect highly. This leads me to believe it is a good program.

That was nice.

That got me to thinking about certifications, and risk.

Think about it the main arguments for test certification: The it reduces the risk to the company in the hiring decision, flattens expectations, maybe reduces some of the communications friction because people use the same words and know what those words mean. Mostly, though, I think it is about risk.

Test certifications create /some form/ of differentiation, allowing an HR department without discernment to winnow two-hundred resumes down to twenty with relative ease.

But that's the problem: The department lacks discernment.

So what if, instead of a test-er certificate, we came up with a test management certificate?

It wouldn't have to be limited to test managers, of course. A development manager could earn it to demonstrate his expertise in the discipline.

So I took the idea to the Rebel Alliance List (an informal group of software testers) and we kicked the idea around a bit.

Certifications have problems.

What does a certificate mean?

The words "certified", imply to me that some authority has decided something about you. So a certified test manager would mean that this authority (whoever it is) claims the person has the skills, tools and abilities to be successful in a certain role.

For some very specific jobs that are well historically defined, like plumbing, bricklaying, or for an electrician, it seems reasonable to me to have a certification.

But in testing, I've seen far too many people be successful in one environment, jump ship, and the very things that made them successful in one environment made them fail in the next.

So the best think we could do is say something like "If you company has this sort of values, and if they are doing this sort of testing, we think this person has the abilities to have success."

Nothing is ever guaranteed, of course, but I do think that within our community we could find the skills to do an evaluation to make such a statement that stands up to scrutiny.

But that kind of statement is too complex for the lazy HR person who wants to check a box.

Which, as my friend Joe Harter pointed out, is a problem - a test management certification might enable someone to be lazy, but at best that is treating a symptom, not a root cause. (I said "at best"; Joe was ... more choice in his wording.)

Conclusions

After looking into the issue seriously, I don't think a test management certification is something I can reasonably pursue in 2011. It is appealing, and I won't rule it out for the future, but it's not the top of my stack for next year, and I don't think it should be. Moreover, if you do pursue a certification, you might want to ask the people offering the cert if they have wrestled with the questions above -- and what answers they came up with.

If you get a reply that is a sort of sheepish grin, handwaving, or "mature organizations don't have those sorts of problems", well, you can probably figure out what I think of the cert.

Yet there is another, less often discussed, benefit of certification: It offers a concrete development plan, combined with some sort of sense of accomplishment. Those are good things, and important things, and I don't want to downplay them.

What I recommend instead, though, is that you write your own plan. One place to start is with reading, and piles of it. If you're here, you're in the right place, and I could drop a suggested reading list as a blog post anytime.

Why I'm thinking of that, though, is mostly from an interview I did recently with Jurgen Appello on Software Management, in preparation for his upcoming book on Management 3.0.

Or to paraphrase James Bach -- "If you can't find a certification with integrity, go certify yourself."

More to come.

One way to transition to agile

There's been a little bit of discussion in the blog-o-sphere lately about how to "sell" agile, or how to convince senior managers to "adopt" agile, how to get buy-in and so on.

I pushed back against this; couldn't you just do it? Does Senior Management even know what the software team is doing, anyway?

The answer to that was that Agile is an investment; a typical team might take six to eight months to build infrastructure (CI, Test Driven Development, deployment tools, etc) and culture -- eight months before the team is again productive. (Reference)

Now there are a couple of different ways to transition to Agile; a team might, for example, make a series of small, incremental changes, each of which pays for itself quickly. But I thought it might be nice to share my favorite "sell the CFO" transition to Agile Story:

A long time ago ...

Senior Manager: "We need you to be the technical project manager on the Super-wiz ERP upgrade, Dave, so we can sell (new product) by (date). If we aren't int he market by (date) (competitor) will eat our lunch. Due to government regulation, we need to file a plan by (date1) and start selling on (date2), or we miss a ONE YEAR market window, by which competitor will have sewn up the market. Dave, you are the man. Only you can do it."

Dave: "I need a war room and for the entire team to be physically co-located, 100% of the time."

Senior Manager: "Well, I don't know about that."

Dave: "If you want me to have a chance to hit your date, I need a war room."

Senior Manager: "I'm sure you can do it. We have confidence in you."

Dave: "If you want me to have a chance to hit your date, I need a war room."

Senior Manager: "Dave, politically speaking, it's impossible. I could probably get you all the technical folks in one room, but then we'd have to find the room. No, you'll have to make it work.

Dave: "If you can't find a war room, then I'm not the person to manage this project. Perhaps you can find someone else willing to take that problem on. I am not."

Senior Manager: "But you're the best! There is no one else who can do this."

Dave: "I need a war room."

Dave got his war room.

The Moral of the Story

One time that organizations are willing to drop "the way we always do it" and try something new is immediately before an oncoming crisis. If you step into the void and offer to take responsibility given certain reasonable changes, you've got a real shot at impacting long-term change.

Another way to get chance is to deal with an organization that is profitable enough that they can experiment.

The classic example of the intersection of those two problems is the creation of the IBM Personal Computer.

The Scrum literature is full of examples of this sort of game-changing project; the classic example is probably in "Wicked Programs, Righteous Solutions" by DeGrace and Stahl.

The Drake Equation of Software Testing

In the 1960's a scientist named Frank Drake came up with a formula to predict the probability of life on other planets -- specifically the chance they would evolve to the point that we could contact them. That equation came to be known as the Drake Equation.

The drake equation is roughly this:

N = R * f(p) * n(e) * f(e) * f(l) * f(i) * f(c) * L

Where:

N = the number of civilizations in our galaxy with which communication might be possible;
R = the average rate of star formation per year in our galaxy
f(p) = the fraction of those stars that have planets
n(e) = the average number of planets that can potentially support life per star that has planets
f(l) = the fraction of the above that actually go on to develop life at some point
f(i) = the fraction of the above that actually go on to develop intelligent life
f(c) = the fraction of civilizations that develop a technology that releases detectable signs of their existence into space
L = the length of time such civilizations release detectable signals into space.

This sounds impressive. I mean, if we could just determine those other variables, we can determine the chance of life on other planets, right?

But ... Wait

It turns out that Drake's equation really says that one unknown number that can only be guess can be calculated as a function of seven numbers ... that we don't really know either and can only be guessed. (And if you try, you run into Kwilinski's law: "Numbers that are 'proven' by multiplying and dividing a bunch of guesses together are worthless.")

Now think about this: As an actual menaingful number, drake's formula is pretty useless. If any of the guesstimates are off by a wide margin (or you can't even really predict them at all), then your answer is a non-answer. Worse than no answer, a wrong answer wastes your time and pushes you toward bad decisions.

Yet what if we didn't try to come up with a 'solid' number, but instead use Drake's equation as a modeling tool -- to help us better understand the problem? To help us figure out what questions to ask? To guide our research?

Suddenly, Drake's equation has some merit.

Back to software testing

Over the next few months I plan on doing some work in the area of the economics of software development -- testing specifically, but also other aspects. To do that work, I intend to throw out some illustrative numbers to help model the problem.

I don't claim that those numbers are "right", nor that any numbers are "right"; the economic value of the software project will develop on what the project is, who the customer is, what the technology stack is, the value of the staff, the time in history ... illustrative numbers are overly simplistic.

So I'm going to abstain from "proving" final answers using those numbers, instead using them as illustrative numbers, to tell as story - that it could be conceptually possible for a certain technique to work if things turned out like the example.

With that foundation model in place, we can make different decisions about how we do our work, and see how it impacts the model.

I want to be very clear here: I'm going to throw up ideas early in order to get feedback early. The ideas I throw out may be cutting edge -- they will certainly be wrong, because all models are wrong. But they might just have a chance to positively impact our field.

I figured it's worth a try.

Plenty more to come, both here and on the STP Test Community Blog.

Types of testers and the future - I

Ok, ok, it's not literally called boutique tester. But listen to this:

You: someone who can make guesses about how a website or application is going to fail, prove that you're right, and then communicate it clearly and effectively to the folks who need to fix it. Ideally, you also have the ability to predict the things about our products that will confuse or dismay a new customer.

As an example of what we do, imagine we have four months to write, test, and ship two applications on a brand-new hardware platform (with no specimens of said hardware in the building) while still updating and maintaining our released products on both platforms. Could you keep up without going totally crazy in the process?

If so, we'd like to hear from you — we're looking to expand our QA department by adding another Software Test Pilot.

Here's the ad, along with the companies "Jobs" site. The position is in Seattle, Washington, and I suspect the position is everything it claims to be.

About the future of software testing

For the past couple of year I have been engaged in a sort of shadow-boxing match of ideologies with some folks. Predominantly in the United States it has been with the idea of the tester/developer - that "the tester of the future" /will/ be writing test automation code, not doing actually testing, and the customer-facing tester will "go away."

Now certainly, I grant that testers will become 'more' technical over the next decade or so, but then again, our entire society is becoming more technical.

I still think it will take a variety of skill sets to find defects on different types of projects. We might have more tester/dev-ers in certain places, but I think "going away" is a bit strong.

In fact, I believe there are some simple economic conditions that make a boutique tester a valid choice for a company like "the omni group."

More to come. Or maybe more on test estimation to come. Either way, I'm writin' again. :-)

CAST 2011 - August 8, 2011, Seattle Washington

If you've been waiting with breathless anticipation for announcements on CAST 2011, wait no more: It's August 8-10 at the Lynwood Convention Center in Seattle, Washington.

Note: I am not running the conference, and this is not some sort of official announcement. I have been, however, monitoring the intarwebs closely, waiting for such an announcement, and wanted to be the first to link to it once it came out. :-)

You heard it here first, folks.

Alternatively, maybe you didn't, in which case, I admire your testing-web-search-foo.

So perhaps I should say "you probably heard it here first?"

I am unaware of a Call for Proposals. As soon as I know of a public one, I'll link to it.

See you in Seattle in about nine months?

Quality is a slogan

Some long-time readers may recall that I am a member of the American Society for Quality, one of the larger, older and more influential non-profits in the Quality Space. ASQ has historically focused on manufacturing quality, but ASQ does have a journal and certification program on Software Quality.

This year ASQ is starting a new initiative on "raising the voice of quality", including blog posts from it's executive director, Paul Borawski.

As someone active in the quality community on the intarwebs, the ASQ leadership asked me to participate in a program called "Influential Voices." As much as I am enjoying the test estimations series (with more to come), I hope you can agree this opportunity was too good to pass by.

The first assignment was to write a response to Mr. Borawski's first blog post. Here goes ...

---

On the ASQ Blog, Paul Borawski wrote:
"Here’s my conundrum. Over the past 25 years I’ve heard some of the world’s best leaders extol the virtues of quality to improve bottom lines, top lines, and whole organizations. I’ve been in rooms of thousands when hospital executives, school superintendents, and city mangers all tell their stories of discovery, learning, trial and success. They tell their stories of transformation from the edge of failure to the summit of success. In the ‘80s I listened to Roger Milliken say, if quality is right, everything else follows. In 2010 I heard Alan Mulally (CEO, Ford Motor Company) in essence say the same thing. I get inspired and excited. Quality really does offer answers to what organizations most need.

So, what would it take to get the world’s attention to focus on that truth? What would it take to have the world realize the full potential of quality?"

---

There's certainly a lot in there, but let's start with this idea that many big, important people have championed the value of quality.

Well, um, of course they have, right?

I mean, can you imagine the CEO of Ford stepping up and saying "Quality? Nah, not really. I mean, that's not /really/ important."

I have a hard time imagining any corporate executive saying that. After all, Quality is sort of like Mom, Apple Pie, or "The Children Are our Future", the kind of thing you can appeal to and get easy, universal agreement.

Why, right now, I'm looking at the side of a five-dollar pizza box, from a company known to use big vats of pre-processed cheese, pre-processed tomato sauce, and pre-cooked, frozen pizza crusts. (In other words, the cheapest pizza with the simplest assembly-line process.)

What does the side of the box say?

---

Our QUALITY! QUALITY! Pledge
(company name) uses only the finest ingredients to provide top-quality products

---

No, seriously, that's what it says.

Clearly, something is going on here.

I'm all for quality ... as long as it doesn't cost me anything

That is to say, Americans want quality, but they don't want to pay for it.

Maybe it started with Phil Crosby's little book, Quality Is Free, in 1979. The basic theme of Crosby's work was sound: If you do a better job, you'll have less defects to fix and correct later, and defects slow you down. So by investing in quality, we actually get payback, whereas trying to go fast right now actually slows things down in the long term.

The problem is, Crosby's book wasn't called "Quality is an investment", it was called "Quality is free."

I doubt I am the first person to observe that our culture wants quality -- as long as they don't have to pay for it.

Quality is ... wait, what is quality, again?

Let's be honest. We don't agree on what quality is. Joseph Juran told us quality is is fitness for purpose, Crosby says it is conforming to requirements, while Jerry Weinberg suggest that quality is value to some person. (I'm with Jerry, for the record.)

That lack of definition introduces some problems. For one thing, it brings us people counting defects, arguing a lower number of defects is a more "quality" piece of software.

But there is another, more sinister problem: If you've never defined what quality is, yet agree it is good, then we've got a situation. This "desire" for "that thing, you know, that quality stuff", creates a market opportunity for consultants, gurus, authors, and all kinds of leaders.

That's not necessarily bad. I'm a guru of sorts, and, on a good day, I have had a few people tell me they have succeeded with my ideas, which is very gratifying. But a market without discernment ... that troubles me.

And it gets worse

To most folks in quality, the "Japanese revolution" is old news, so I will be brief. Over a period of a few years, the Japanese changed their reputation from making cheap plastic junk to making high-efficient, low-defect, last-a-long-time electronics and automobiles.

They went from followers to leaders.

About that time (surprise surprise), back in the United States, we developed a market for "that quality stuffs", but, as the secret reply also went "...keep in mind, we're busy with the business of production. So let's hire some bright young engineer and give him the role of director of QA."

Another common secret statement was "But don't change the way we do business, or interrupt production!"

Can you imagine why that didn't work?

Shocking.

Another monkey-wrench in the works

About a hundred years ago, a man named Frederick W. Taylor wrote a book entitled "The Principles of Scientific Management"

I would be reluctant to call the book science, by any stretch of the term, but it is worth reading, if only to know your enemy.

Taylor was kind of jerk.

His basic idea was to "separate the worker from the work" by having an elite group, the scientists, design the work to be done a specific, exacting way.

Another term for this is "defined process."

And, while Taylor's "Scientific Management" might have worked in 1910 to move iron from one place to another (though modern historians debate that), it certainly doesn't work well to create intellectual property, like books or software.

Imagine a publisher telling an author or artist when to sit down, when to write, and when to take breaks. It just doesn't work that way.

My research into the Japenese Revolution, specifically Toyota, shows a style that is roughly the opposite of Taylor. The Toyota way involves engaging the workers in the work by giving the team interesting assembly problems and asking them to find the best way to build the product.

For that matter, forget "best"; the Toyota way asks the team to continually improve the process of building the product.

So we Americans sent people over to Japan to figure out what they were doing and systematize it, just as Ray Croc had done with McDonalds.

So we came back with Lean and Six Sigma and standardization.

But something happened along the way. By translating these ideas from the far-east, we injected some of that old "Tayloristic" ideology. America, fresh from the great success of McDonalds, was in love with systems and process and standardization. We were also in love with command and control methods of management.

Nobody seemed to recall that McDonalds had to experiment first, to find a good burger that could be made quickly; we just thought standardization was good.

Now on Toyota's teams, it is the team that decides what things are worth writing up standards for, to help the new guys. On a Toyota team, making a standard pre-maturely is a kind of waste; it causes you to make rules today that will be out of date tomorrow. Or worse, causes you to build things in an old, out-of-date, slow, error-prone way because "that is what the process book says."

Yet I've read several books by Americans on "Lean Manufacturing", and can not recall ever reading anything about just-in-time standards. Instead I read "standardize, standardize, standardize"

Why, we even have a "quality system", ISO-9001, that is all about documenting processes.

About ISO-9001

The basic belief system of ISO-9001 seems to be: If you do something, you should do it the same way, every time. You should write down how you do things. Then you can hire people to come into your shop and audit, to see if you are in fact doing things the way the book says.

All of this pre-supposes that it is possible to write down a process that will lead to zero defects. With hardware, where you can "insert tab A into slot A" in a cookie-cutter process, that might be possible. In software, where the process has to say, at some point "... and then the developer writes the code", or "... and then the tester does some testing", it'll be impossible to have that kind of certainty with defined process.

Also, this pre-supposes that having new ideas, and changing our process, in the moment, is less valuable than having a stable, predictable, repeatable process.

And that, since the process doesn't change too much, the cost of documenting and maintaining change control over the process will not be overly onerous.

System Effects

So we've got a system where we define "quality" as good and no further, which creates a market demand for a product that is hard to evaluate. We have these leftover ideas of class warfare from the 19th century that are clouding our vision. We have traditional command-and-control managers that want "that quality stuffs" as long as that doesn't mean any change to them or their power base.

So people who come in selling "quality" as something that doesn't require change, that reinforces command and control, will likely have more success than those trying to reform things.

To go back to Mr. Borawski's charge, I have to say, the question should not be "Why hasn't quality had more impact?" but instead, we should celebrate the little impact we have had.

My fix? Well, I actually have some good news

The Good News

Outside of the government and some elements of the health care sector, most of North America is living in a free market economy. That means we get to compete against other companies for products and services.

If Quality is value to some person, as Weinberg tells us, and the customer is that person, then those of us who build better products will sell more of them, make more money, and have successful businesses.

Of course there are exceptions; big brands getting by on reputation, companies creating monopolies or manipulating the market. But, by and large, those focused on the "accidental" side of quality - meaningless metrics or silly process - will not do as well as those focused on excellence in product development, operations, or sales.

In other words, in the long term, the good guys win.

Now as for the other concern -- that this idea of doing quality work can lead to better products, more products for less cost, and lower defects -- well, I agree. If you think about it, "if everybody did it", we would create an abundance of resources (by which I mean, actual stuff) and make the world a better place.

Where can we start? Well, once place to start, I think, would be by giving examples and case studies of successful organizations, along with exploring ideas behind systems thinking.

Sure, Six Sigma and ISO-9001 might be tools, and they might have a place, but they are relatively shallow tools with a niche place. Let's radically de-emphasize the role of those tools, choosing instead to focus on systems thinking.

To do that, we would first model our workplace as a system, then see what the right tool is for the job, focusing all the while on the customer and customer needs.

Of course that's only a start. We'll have a whole year to debate these ideas back and forth, and I suspect we'll end with something strong than that.

But it is a start.

Now get out there and change the world!

On Skill

I just got back from the software test and performance conference in beautiful Las Vegas, Nevada - conference write-up here.

Just after I got back from the conference, we started talking about skill on the software-testing discussion list, and I posted this:

---

--- In software-testing@yahoogroups.com, "Elena" wrote:
>
> The rest I cannot quite explain to myself -- is it intuition,
>having a bigger picture/snapshots of what it's supposed to be
>like in my head, etc?
>

There's an old saw that an apprentice butcher is sitting with an old butcher.
You know, they grey-hair type who's working in the shop hit entire life.

Customer comes in "I would like two pounds of pork."

Chop Chop Chop. "Here you go."

Suddenly the kid pipes up "No way that is /exactly/ two pounds! You didn't even
weigh it!!"

"Ok, kid, you weigh it then."

Sure enough, 2.01 pounds on the first slice.

"How did you /*do*/ that?"

"I'm not exactly sure, kid, but, now that you mention it ... I suspect cutting
meat five days a week for thirty years had something to do with it."

Moral: Don't let other people label your hard-won experience as "intuition" or
"talent."

No, I suspect you /earned/ and /developed/ skill over time.

Skill takes work.

To be precise, the human brain has a lot of functions, one of which is a really
big neural network. So we recognize patterns.

You recognized a pattern that no one else could. That's part of what good
testers do.

I'm happy for you, man.

Don't let the anti-skill ... goofballs get you down.

all my best,


--heusser

---

Then this afternoon Justin Dessonville posted this video to his twitter account.

It is a five-minute video of people doing the seemingly impossible: Multiple far-court free-throws in a row in basketball, a guy who throws a playing card and uses it to blow out a candle - multiple story jumps, tony hawk's impressive skateboarding feats.

I'm not sure how some of those were done; the guy might have spent six hours in front of the video camera to record one take. But some of them, like the basketball player and Tony Hawk, were clearly the result of a lifetime of practice.

Now compare that to your typical software ideology of having a defined, standard, predictable, repeatable process.

How do you write down a process called "be an awesome skate-boarder?"

You don't.

You can, however, defined moves and create a place to practice consciously. You can memorize, and repeat, and build from low-skill to high skill exercises.

I'm pleased to say that, at this point, I see a substantive part of the test-o-sphere moving from these shallow notions of repeatability into something more meaningful. If it's testing dojos or weekend testers or something else, we're finally getting there.

I'm pleased.

Now, if you'll excuse me, I gotta go practice my javascript blackflips on IE6 ...

Software Test Estimation - VII

So we've had a bit of an adventure; I've covered three different estimation models, each with a basis that isactually sound.

And there's a problem.

The problem is simple: We are presupposing that "tested" and "not tested" are binary statements. That there is some amount of testing that is "right" and it's universal. Doing less is irresponsible; doing more is waste.

I suspect there is more to the story.

---

First, consider this situation. A leader, executive, or customer hires you as a boutique tester. He ells you that "I'd like you to evaluate this product for one hour and then share with me what you know. I'll pay you a big pile of money."

In that case, I might want to be very clear about what responsibility I am taking on. If I were testing missile-launch software, I might have quite a few concerns.

For the most part, though, I don't have a problem taking on a project like this, especially if he pile of money is large enough.

In that case, an estimate is pretty easy, right? An hour, maybe two if you want a debrief and you'd like me to write up some documentation.

Most software testing projects are somewhere between "Take an hour and tell me what you find" and "how much time do you need?"

In many cases, it might be wise for us to consider test negotiation, not test estimation.

---

Test Negotiation

Management knows how long they want testing to take - they want it to be free. Or at least ridiculously cheap; they'd certainly prefer the one hour, the one day, the one week test 'phase' to whatever reality offers.

And that's okay.

So one thing we can do when it comes to testing is offer the cafeteria.

Hopefully you're familiar with the idea of cafeteria-style dining - each piece is sold separately, and you only pay for as much as you want.

The way the buffet works is that we come up with a range of plans, at a range of price points, that address a range of risks. We use the tools in the previous six posts to estimate them. Then we ask management "Which (testing) services would you like to pay for?"

Based on that discussion, we draw up a test strategy. Along the way, we make recommendations to management, but let them sign off.

This does two things for us. First of all, it transforms our roles from roadblocks and nay-sayers to consultants and enablers.

Second off all, it moves risk management decisions to the senior management layer.

I have been in meetings where someone asks "Why didn't QA find that?" and the VP of Engineering replies "I signed off on QA skipping that test; we decided to take a calculated risk to hit the deadline."

Suddenly, once the senior executive takes credit for the defect, it's not such a terrible thing that "QA missed that bug." Instead, it was a business decision.

I could get sarcastic about that, but the thing is: It's actually true. It was a calculated, explicit business decision.

Bringing our customers and stakeholders into the process will help them understand what we do, and understand the great task of "complete testing." (Whatever that means.)

It's a better place to be.

As to how to get there, and how to test under iterative and incremental models, well ... more next time.

Interlude - About Software Engineering

(More Test Estimation to come, but first: An Interlude)

By now many of us know the standard 'barbs' - for example, that when Winston Royce designed the waterfall model, he said it was "risky" and "invites failure".

Or that Tom DeMarco, author of the oft-quoted "Controlling Software Projects: Management, Measurement, and Estimation" has essentially recanted his claim that "You can't control what you can't measure."

(To be more accurate, DeMarco actually argues that the quote may hold. It just turns out that control isn't all that important on software projects these days.)

Then I saw this video by Glenn Vanderburg at the Lone Star Ruby Conference:



Once again someone has taken some of those ideas, deconstructed them, and re-packaged them in a way greater than the sum of their parts. In the video, Greg goes way back, explaining not only what Winston Royce wrote, but the how and why it could have been perverted in the "waterfall" "standards based" and "software engineering" approaches of the 1980's and 1990's -- and what we should do about it.

I think you'll enjoy it.

Next time: Still more test estimation to come.

We're getting there. Really.

Test Estimation - VI

So far, we have two ways to predict project outcome:

First by comparing the test effort to other projects, and suggesting it is "between the six of project X and projec Y", this giving us a range.

Second by calculating out test costs as a percentage of the development (or total project) effort, looking at the official schedule and projecting out our expected project length. If we're smart, we also take into account slips to get that percentage.

A third approach I can suggest is to predict the cycle time - time to run through all the testing once. I find that teams are often good at predicting cycle time. The problem is they predict that everything will go right.

It turns out that things don't go right.

Team members find defects. That means they have to stop, reproduce the issue, document the issue, and start over -- that takes time. More than that, it take mental brain energy; the tester has to "switch gears." Plus, each defect found means a defect that needs to be verified at some later point. Some large percentage of defects require conversation, triage, and additional mental effort.

Then there is the inevitable waiting for the build, waiting for the environment, the "one more thing I forgot."

So each cycle time should be larger than ideal - perhaps by 30 to 40%.

Then we need to predict the number of cycles based on previous projects. Four is usually a reasonable number to start with -- of course, it depends if "code complete" means the code is actually complete or not. If "code complete" means "the first chunks of code big enough to hand to test are done", you'll need more cycles.

If you start to hear rhetoric about "making it up later" or "the specs tooks longer than we expected, but now that they are solid development should go faster", you'll need more cycles.

(Hint: When folks plan to make it up later, that means the software is more complex, probably buggier, than the team expected. That means it'll take more time to test than you'd hoped, not less.)

---

So now we have three different methods to come up with estimates. With these three measures we can do something called triangulation - where we average the three. (Or average the ranges, if you came up with ranges.)

When that happens, it's human nature to tend to throw out the outliers - the weird numbers that are too big or too small.

I don't recommend that. Instead, ask why the outliers are big or small. "What's up with that?"

Only throw out the outlier if you can easily figure out why it is conceptually invalid. Otherwise, listen to the outlier.

Which brings up a problem -- all the estimating techniques I've listed so far have a couple of major conceptual flaws. And I haven't talked about iterative or incremental models yet.

They are just a start.

Still more to come.

Test Estimation - V

So one way to estimate the testing phase (if you have such a thing), or at least testing activities, is to compare the test effort to the development effort or overall effort on other projects.

Examples:

"We spent about ten solid months on MaxStudio, and only spent two months testing. So I think testing should be about 20% of the overall dev budget."

"We spent a year on StudioViz from kick-off meeting to release, and about a month testing. So I think testing should be less than 10% of overall budget."

---

Both of these examples are real.

The thing is, after release, we spent the next six months fixing MaxStudio, and took a serious hit in the marketplace or reputation.

Likewise, we developed StudioViz incrementally, with many stops along the way to bring the work-in-progress up to production quality. StudioViz was also a browser-based application - well - sort of. It ran in browser control inside a windows application. So we were able to 'lock down' the browser to at least modern versions of Internet Explorer.

What all this means is that if you pick history to do a percentage of effort measurement, make sure the development model - the "way you are working" is relatively similar. Big changes in teams, in technology, technique, or method can render these sort of projections obsolete pretty easily.

But now we have two methods: Comparing test effort to test effort on similar sized projects, and using test effort as a percentage of dev effort. (That is, what percentage was it of dev effort for previous projects, look at dev effort for this project, multiply by percentage, get test effort.)

Of course, both of those measurements assume that you have roughly the same portion of developers to testers - but like I said, changing things makes projections based on past experience less and less accurate.

Of, and successful organizations tend to change. A lot.

Another method, that I've hinted at, is percentage of the overall project. Now for this to work you have to be careful, because it's very hard to measure the effort if you go back to when the idea was a gleam in someone's eye. When I've done this, I've tried to go back to when the initial kick-off happened - at which point the project probably had a full-time project manager, 'assigned' technical staff, maybe a business analyst.

---

Here's another quick exercise for the folks that complain "that sounds great, but we don't have the data":

Start tracking it.

Seriously. Get a pen, pencil, maybe your email box out, and start tracking just the projects you are on or the 'huge ones' swirling around you. This is easy enough to do in excel. If you want to get really fancy, start recording when the project was created and predicted due date, along with when the due-date slips occur, and how much much they slip by.

It turns out that this trivial-to-gather data can be extremely valuable when used to predict the performance of future projects.

More on that next time.

Test Estimation - IV

So far, I've pointed out various issues in test estimation, pointing out the fallacies involved in simple 'naive' test estimation. I also pointed out that is possible to do some kind of estimate, even if all you say is "um, yeah boss -- two weeks."

A couple of people expressed surprise at this. Laurent Bossavit, a board member of the Agile Alliance, was concerend that people wouldn't take me seriously.

But if you've been around the block a few times, you might just remember a test project or two where a wild guess made without context was the test strategy. Someone with no context walked in, said "testing -- hmm. Four weeks." and walked away. They were the boss, and it is what it is, and the team made the best of it.

Hopefully that isn't what actually happened.

It might have looked like that to an outsider, but there was likely something more going on. Perhaps the manager needed to ship the product in four weeks in order to hit some deadline. Perhaps he needed the staff to stop billing on project X because in four weeks, you'd run out of budget.

Or, perhaps, it's possible that some sort of rational process was going on in his head that we could not see.

I'm going to start with the rational approaches. Don't worry, we'll cover the social approaches, and even agile and iterative models -- but I think the next step is to have some logical reason to believe it will take X, instead of wishful thinking.

---

It might have looked like "Four weeks" was made up, but it's very possible that some complex process was going on in that manager's head. For example, he might have thought:

Hey, this project is about as big as project A. I've got the exact same technical teach as project A. We haven't had much turnover in the dev ranks either. We've actually learning something since project A, and I believe the team will make less mistakes. How long did project A take to test? Oh yeah, four weeks. Okay. I'll use that as estimate.

Check it out -- our theoretical manager might actually had a reason for believing in the four week number.

It's also possible that the manager was considering a number of projects and averaging the lengths, to come up with four weeks. It's unlikely the data was available, but the human mind is surprisingly capable of doing those averages, even subconsciously. I prefer to do it with a pen and paper and a little math, to have something to explain to someone who asks "where did the four week number came from?", but many a tester, lead, or manager can do this sort of comparison in their head without even realizing it.

This happens in other disciplines too. Consider the expert Butch who has spent his entire adult life in the field. You ask him for two pounds of ham and he goes slice slice slice - weight it and it's exactly 2.002 pounds. Ask him how he did it, and he'll likely say "I don't know. I suspect cutting meat five days a week for thirty years had something to do with it.

---

But we can do one better than that. Write down a half-dozen projects. It's unlikely that any of them are specifically like project X. Project A had a different dev team, project B had more testers, project C was riskier with a more complex infrastructure, project D was the first time we had ever used the new webserver, and so on.

So you end up with a range. "Given our previous projects, we expect this test project to take three to five weeks."

Of course, there's all kinds of reasons that's a bad estimate. But it's better than the first strategy - to make something up - right? And it's likely better than functional decomposition, for reasons we already discussed.

This idea - past history - is a lever, a way to come up with an estimate. In other words, it's a model from which an estimate will "pop out".

When this series is done, you should have several levers. You'll be able to apply several models to your project, get several estimates, and consider which one is correct.

More to come.

Test Estimation - III

So previously I posted that factors outside our own control make accurately estimating the total test schedule impossible. In addition, I posted that even estimatingf simple, known, static things by breaking them into tasks, then adding up the tasks is much less accurate than you might think.

Example Three:

Imagine your typical workday that ends at 5:00 think about it. Sure, I'll likely be home from my drive at 5:30, but I might hit traffic, my boss might ask me a question at 4:55PM, someone in the parking lot might need a jump start. So if you want a commitment from me to predict when to invite the neighbors of when to pull the turkey out of th oven, I should likely say 6:00PM.

That's a 100% timing padding.

One hundred per cent.

On a task I've done hundreds, if not thousands of times with known physical 'resources' and very few dependencies.

Which reinforces the conclusion that test accurate estimation is impossible. At least in the sense of estimating a single number for total time without context.

Yet we are tasked with doing it anyway -- better yet, creating that context. So where do we start?

---

About a year ago my good friend Ben Simo pointed out to me that even if you know nothing at all about a project, you can always estimate it. Here's an example:

Q: "How long will testing take?"
A: "Two Weeks."

See that? I know absolutely nothing at all about the software. Nothing at all. But I made up a number.

For any given project, we likely know more than nothing. So we can certainly do a better estimation than "two weeks."

Next: I'll start talking about how I do that.

On Test Estimation - II

Another post I made to the Agile-Testing Group recently:

---

Here's a simple estimation excercise. My honest advice is don't just read it; actually try it. It takes about two minutes.

To start, think about the space between the bottom of your feet and your knees. Write it down. Then think about the space between your knees and your middle. Write that down.

Then estimate the and write down the size of your torso, then your neck, then your head.

Next, add up all five numbers.

Now compare that to how tall you /actually/ are.

That difference - between how you imagine things and how they actually are - is a picutre difference between task estimates and how long things will actually take.

Except of course, you can see and touch your body and it's been about the same height for decades, whereas code is new and fresh and symbolic and 'tests' are an even-more abstract concept.

When you think about it, tests are a first-order derivative of the code itself. Also, most testing is exploratory in nature, EG early predictions are not the best predictions.

So would I be reluctant to make task estimates on a testing task, given the typical American shorthand that estimate==commitment? Certainly.

---

I like to think of this as the test estimation rabbit hole. First, we have to have the bad news that test estimation is conceptually impossible.

Then we figure out how to do it anyway.

More to come.

On Test Estimation - I

I posted this yesterday to the Agile-Testing List, thought I would share it here as well:

--- In agile-testing@yahoogroups.com, "daswartz@..." wrote:
>
>
> Can you help us understand why the QA people care whether
> you estimate in hours or points? I'm sure they have a reason, which
> should help us better answer the context for your question.
>

I'm not the Original Poster, but consider you are testing feature X. You break it down into tasks and say it will take 5 hours to "test."

The first build is total garbage. You can't even click the submit button.

The next day, you get a new build. You find five bugs. You get a new build
late in the day - four of the five bugs are fixed, and you find three new ones.

You get the fixes in the morning on day three. You find another bug. At noon,
your boss comes up: "You said this would take five hours to test and you are on
DAY THREE of testing? Wassup with that?"

---> Bottom line, there are elements in how long it takes to do testing beyond
the testers control. It's generally possible to estimate a test /cycle/ with
some accuracy, but estimating the entire test /process/(*) is rarely possible
unless you know the devs very well and have had some stability in delivered
software quality for some time.

Estimating in terms of points 'smooths' those gaps.

That's one possible explanation, anyway ...

--heusser
(*) - Yes, this pre-supposes that testing is a separate and distinct activity, I
know, we should be involved up front, whole team, etc. I'm with you. But
you gotta walk before you can run. Let's have that discussion on a separate
thread, ok?

Sheep of a different fold

For a few years now I've been listening and reading to the work Mary Poppendeick has produced with increasing appreciation. Then last year I had the opportunity to interview Mary and Tom for InformIT, as part of InformIT's coverage of the Agile 2009, where Mary was giving a keynote speech.

(Mary and Tom) and I have very different backgrounds. We worked in different kinds of organizations and our careers and interests took us in very different directions.

Yet here was this other person that had both studied the history of the philosophy of management -- and studied the actual effects of those ideas in practice -- and come to the same conclusions as I had.

For that matter, the way that the Poppendeick's approach the subject is different than my stuff, and I think worth studying. So when I found out that they had a google tech talk, I just had to link to it here:



I've spent tens, if not hundreds of thousands of words trying to explain the ideology of "process improvement" and some of my concerns about it. For a quick summary introduction, I gotta say, this video by Mary is shockingly good. Throw in a copy of The Management Myth by Matthew Stewart and you end up with a very good survey of the literature in two source documents.

sigh.

If you need me, I'll be in the corner, licking my wounded pride, trying hard not to cry.

:-)

Seriously, this is a good stuff, and I am pleased to recommend it.

UPDATE: A cursory glance at Poppendeick LLC website finds several 'sound byte' level things that you might take issue with in regards to /testing/. My advice: Ignore the sound bytes that are so easy to misconstrue; watch the video instead. Check out what she actually /says/ about software development, management, and leadership. I expect it will resonate with you. It did with me.

What Matt has been up to

Woa. Some cobwebs on Creative Chaos, eh? Seems like Matt Hasn't been very busy, doesn't it?

Gosh, I sure hope you don't think that. Let me bring you up to speed:

a) I've split up my blogging into two places - here (always free, no-registration required) and for the Software Test Performance Collaborative - still free, registration required.

b) That blogging now includes recruiting, running and managing a weekly podcast "This Week in Software Testing" - also free for the new shows. To get the old content, you'll need to be a paid member of SoftwareTestProfessionals.Com. Don't want to pay? You can download every episode as they come out, or you can participate in various contests on the blogs. Or write an article for the magazine.

c) I'm still micro-blogging on twitter under the username mheusser.

d) I'm still producing a column for the magazine, now known as "Software Test & Quality Assurance" magazine, or STQA. After two years of writing a encyclopedia-style column were we defined key terms used in the "theme" of the issue, we decided to shake things up a bit and write an interview column. Each issue we'll gather questions from the community for a 'test expert' and have them answer. For the August issue we interviewed Michael Bolton; the extended interview is now available on-line. I'm at the point now where I need recruits to interview, and shortly after that, I'll need questions ...

e) Beyond working with the fine folks at STP, occasionally I get a bit of time to work with other publishers, including the folks at SearchSoftwareQuality. That includes a few podcast-style interviews, a tutorial on the Selenium IDE, a good place for classic testers to learn about Selenium, as well as a two-part tutorial on Selenium RC, which is a start for programmer-types. (Link to Part I and Part II here). I also just wrote a small piece on Effective Bug Reporting Techniques for SSQ.

f) We finally put the Conference for the Association for Software Testing (CAST 2010) to bed last night with a conference retrospective. We held it in Grand Rapids, Michigan in early August. Instead of presenting, I helped out with the local logistics, recruiting some sponsors, and organizing and funding the evening receptions. CAST 2011 will be in Seattle, Washington. With Jon Bach as the conference chair and James as the program chair, I suspect it will be amazing.

g) With one conference to bed, it's time for me to worry about the next one! STPCon is going to be October 19-21 in Last Vegas, Nevada. It starts the 17th if you grab a two-day pre-conference tutorial. I'm a "track chair" for the hands-on track sessions, I'm running one of the track sessions, running a panel on how to decrease costs in testing, and organizing a lightning-talk like session. Oh, and there will likely be a Monday night reception.

h) Day job! Full time as a member of the technical staff specializing in test for Socialtext. At least I managed to skip the commute, otherwise this stuff would be impossible.

i) It's about time for me to re-start teaching religious education for fourth and fifth grade at my Church during the school year, plus coaching soccer for this fall. (See, I have a life outside of work. Really. Occasionally. Sorta.)

j) I just wrapped up a two-year night teaching position at Calvin College. It was really great, but due to a-i, plus not commuting into Grand Rapids anymore, something had to give. I have small children at home; it would be nice to occasionally see them.

... and then I went crazy.

No, at least semi-seriousy. Based on a discussion on the LinkedIn Discussion list, I just signed up to be the lead editor on a collection of essays on how to reduce he cost of software testing to be published by CRC Press in early 2011.

We've got a good team. We had some solid progress before we signed the contract.

But I did just sign and email the contract last week, and our completed, publisher-ready draft is due Nov 1st.

More to come; at the very least, I'll try to blog pointers to interesting work elsewhere.

But forgive me if I haven't been blogging here much. As I hope you can see, I've been ... kinda busy.

And that was before I went crazy. :-)

Kaner On Testing

Those of you who read this blog know I've probably spent tens, if not hundreds of thousands of words discussing the applicability of hard metrics to the management of software development.

You likely know that I'm not keen on it.

Yet I struggle to make the point sharply and quickly. Cem Kaner wrote something on metrics today that summed it all up in a hundred words or so:

Capers Jones sometimes talks disparagingly about the (claimed) fact that 95% of American software companies have no metrics program. On the surface, this sounds terrible. But what I saw as a consultant was that most software companies have tried a measurement program, or have executives with lots of experience with metrics programs in other companies. The problem is that their experiences were bad. The measurement programs failed. Robert Austin wrote a terrific book, Measuring & Managing Performance in Organizations. When you start measuring something that people do, people will change their behavior to make their scores better. People will change what they do to get better scores on the measurements—that’s what they’re supposed to do. But they don’t necessarily change in ways that improve what you want to improve. Often, the changes make things worse instead of better (a problem commonly called “measurement dysfunction.”) This problem happens more often, and worse, if you use weak, unvalidated metrics. I keep meeting software consultants, especially software process consultants, who say that it’s better to use bad measurements than no measurement at all. I think that’s’ a prescription for disaster, and that it’s no wonder that so many software executives refuse to harm their businesses in this way.

I thought it was brilliant.

If you want more, you can read the source that quote comes from - part I of a series of interviews with Cem on uTest.com.

Or come to the Conference for the Association for Software Testing - CAST 2010 - next week in Grand Rapids, Michigan, where Cem is giving a keynote-level speech.

UPDATE: I've been thinking about it, and it certainly depends how you define your metrics. For example, one kind of measurement that I am in favor of is the "slip chart." In other words, you look at every deadline the team has committed to and how late they are, and you figure the rough percentage of how late they /always/ are. With that, you can predict when you will really be done. The folks in the extreme programming community codify this into story points and burnup charts, and that's fine. I don't have a problem with these used as approximations; as first-order measurements. The problem comes when they are reduced to 100-word silver-bullets without the context of how they can be used well.

So I wouldn't say I'm totally opposed to metrics are part of a balanced breakfast. I'm just leery of the common ideology of measurement by numbers alone, without context.

Is Your Software Development Organization Agile?

Elisabeth Hendrickson gave a wonderful keynote on Agile-Testing at STAREast this year, and my Friend Dan Mondello took her definition of Agile and codified it with an article. It's a nice read.

We were sitting together for the keynote, and Dan threw in a picture of us at the bottom of the article. You can see Selena Delesie and Lanette Creamer at the left of the photo, Dan at right, and me in the middle.

If you're not interested in the article, check out the photo. That someone as dopey as me has managed to have some modest amount of success in this field should be encouragement to dopey people everywhere! :-)

Selenium IDE

The folks at TechTarget just asked me to write an article on Selenium IDE, the integrated, simple, easy-to-use, free browser-driving automation tool for FireFox.

And they just published it!

You can get the article right now at http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1516589,00.html. (Free Registration Required)

Matt's Big CAST Announcement - Part II

Did you know that CAST will be serving breakfast, lunch, and snacks? They are come free with your conference registration.

More than that, we are currently working on sponsors to provide pop and snacks at dinner every night -- Monday, Tuesday, and Wednesday. And it's possible those turn into dinners.

If you are attending CAST on your own dime, you'll be able to load up on free food and only have to pay for a minimal number of dinners. Likewise, if the company is sending you, between hotel discounts and all the food, the cost will be significantly less than your typical "Big Box" conference with the sixty dollar dinner buffet. You can practically afford to send two for the price of one. I will be personally sponsoring snacks in the Rebel Alliance hospitality suite on Tuesday night. Don't say I never gave ya nothin'. :-)

Of course, since CAST doesn't work with rebels the way, say, a 'STAR' conference might, we may go for a different theme. "The CAST Aways" probably works. Just don't call me "little buddy" and hit me with your hat!

Other stuff - there's a tool called "Is the website down or is it just me?" that seems handy-dandy, and I've got a lot more blog material up at the Software Test Professionals site. If you're hungry for posts from me, check out that site. I'll have a blog post every week (or more), plus, new, a "This Week in Software Testing" podcast up once a week or more.

Matt's Big CAST Announcement - Part I

So CAST - the Conference for the Association for Software Testing - 2010 is coming up, and it's going to be about 40 miles from my place, at the Prince Conference Center at Calvin College.

Last time I checked, the Prince Center was very close to booked solid; people keep asking me where to stay.

I've got two concrete suggestions for you:

#1 Country Inn and Suites - Here's the details from an email forward for 35% off!

Book your stay by July 13, 2010 and enjoy a 35% discount at participating Country Inns & Suites By CarlsonSM hotels when you stay at least two consecutive nights during select dates between July 14 and August 31, 2010. See all participating Country Inns & Suites By Carlson hotels or check out participating hotels in your favorite regions: U.S. - Midwest U.S. - Northeast U.S. - Southeast U.S. - Central Midwest U.S. - Southwest U.S. - West U.S. - Mountain Costa Rica Hurry, these great rates are only available for a limited time! Book your stay at countryinns.com today and earn bonus Gold Points® for every online booking.

You'll want the Country Inn and Suites in Grand Rapids, Michigan, on East Beltline Road. It's about 3 miles north of the Prince Conference Center.

#2 Any Marriott You'd like

Specifically, the FairField Inn Grand Rapids, which is maybe 1.5 miles from Prince, probably more like 1. If you really feel like pushing it, you could sign up for a Marriott Rewards credit card, get the points and certificate, and use those. It should be about enough combined for two free nights stay -- but read the fine print. The points often aren't credited to you until the first statement is cut. So you might want to stay at Mariott next time, and use the 35% off the Country Inn and Suites this time.

I hope that helps.

Technical Debt: Refired

Phil Kirkham has the cover story of T.E.S.T. magazine this month, talking about technical debt. It's a good article and I recommend it. One thing I like about it is that Phil tries hard to provide a framework for thinking about tech debt, that he borrows from Martin Fowler.

All of it reminds me of the technical debt workshop we ran at Calvin College in 2008. Not a whole lot poured out of that workshop -- it was only two days long, and we spent most of the time trying to come to get to the 'gelled' state so we could make progress. Ron and Chet weren't keen on the term, Chris McMahon came out against analogy shortly thereafter. There wasn't a lot published afterward; I saw a few proposals go to magazine editors but they weren't wild about the formats we proposed. There were a few good presentations we recorded, but I'm afraid there's lots of editing required, and the videos from the event remain locked on a hard drive on my desk.

It wasn't until a week after the conference, on the e-mail discussion list, that we started to see some of the collaborative, building comments I had hope to have during the conference.

Perhaps, if it had been three days, we'd have gotten there. Perhaps, if it had been three days, I'd be saying "if it had only been four days." I don't know.

Using metaphors to describe our work does have certain risks, but I still think that in many cases the tech debt metaphor can have more value than the risk it creates.

It may be time for me to start writing about this again -- or considering a 2011 or 2012 workshop. I'm not sure.

In the mean time, I've got three other new projects. First, a series of interviews with testers called "Testers at work", currently on the back-burner. Second, a book project on changing the cost/value ration of software testing, that i've announced here, and third, a (near) weekly series of podcast interviews with testers I'm nick-naming "This Week in Software Testing", or TWiST, the first of which is up here.

In other news, you can now catch my blogs in two places -- both here and at www.softwaretestpro.com/blog. The majority of my blogging will likely be at STP, but if the topic is the kind of thing that needs a disclaimer, you'll likely find it here.

More to come.

"Welcome all my friends to the show that never ends; I'm so glad you can attend. Come inside, come inside ..."

A couple of articles, coming at you

I've always felt that the test automation literature is a little ... odd.

On the one hand, you have the floofly high-level tutorial, written so that it applies to every test tool. By trying to apply to every tool, the tutorial isn't specific enough to provide step-by-step instructions.

Then you buy something and try to use it. UGH. That's no fun. So maybe you hire some consultants to build the framework and basic test suite. That's great, but it costs you $8,000 a week - and - $320,000 and six months later, and the consultants leave, do you have the expertise to maintain the suite?

All too often, two years later, you've got some really expensive shelfware.

So I'm interested in tutorials about test tools. The thing is, once you get specific, the market is much smaller -- so it's rare to find a mass-market book about testing with a specific test tool.

But that's the other thing. With some test tools today, you don't need one. All the readers need is an article bit enough to get them started -- or a short series to keep them learning.

That said, the folks at SearchSoftwareQuality invited me to do a small series on Selenium, and have published the first two parts - a two-part mini-series on Selenium RC. Read all about it - part one and part two, online, for free.

The next article in the series will address Selenium IDE, the development environment for Selenium. I hope you find them enjoyable and helpful.

Awards and Decorations

A little over ten years ago I interviewed for a developer job with Steve Hoek, at a tiny division of McGraw-Hill. Steve asked me if I saw myself collecting an award in five years - what would it be?

I replied that I'd like it to be some sort of industry award, but those sort of things don't really exist for do-ers. I hope I never forget how Steve replied -- he agreed with me. Those kind of things do not exist for do-ers.

Maybe they should.

Oh, yes, there are lots of problems. There are conflicts of interest. The inner ring-ers tend to get them disproportionately, without making significant or meaningful contributions to testing. Plenty of people do good work, dedicated work, for decades without doing any PR, and are virtually unknown. But I do think that a good faith effort, on balance, would be better than none at all. The Agile movement has the Gordon Pask Award; can Software Testing do something similar?

I think we can.

So I was very please earlier in the month when the folks at Software Test & Quality Assurance Magazine announced the Luminary Award. Please allow me to read the qualifications from the nomination page:

The software test and quality community will decide who receives the award, and the expectation is the community will choose a nominee with some or all of these qualities:

•A person that has inspired individuals to be better at their profession
•A person that has dedicated their career for the betterment of software testing and quality
•A person that has shown exceptional leadership in promoting the advancement of our industry
•A person that has shown exceptional ability in promoting and educating concepts for the advancement of the industry
•A person that has published and promoted the industry that has resulted in greater recognition and respect for the industry
•A person deserving of an award based on merit not particular to personality (although nominees may be very popular we want this to be about career achievements)

When I read those requirements, a few words jump out at me. Some appear on the page, others are just how I summarize the tone:.

* Testing AND Quality
* Published
* Community
* Lifetime

When I think of those words, one name comes to mind: Cem Kaner.

Why Cem Kaner? Well, he's been around a long time. In the early 1980's he got a PhD in Cognitive Psychology, and he has been advancing the cause of software as a human-driven activity ever since. After doing some programming and testing in that PhD Program. Dr. Kaner moved to Silicon Valley where he worked as a tester, programmer, test manager, documentation manager -- and get this -- he got a job at night in a retail software store while he held those jobs so he could get closer to the customer.

In 1988 Dr. Kaner wrote the first edition of "Testing Computer Software", a landmark book in the field, which he brought in two co-authors to help revise into the 2nd edition in 1993. After Silly Valley, Dr. Kaner got a law degree, practiced law for a short time, and went on to write Bad Software, one of very few books about the legal implications of defective software. In 2000 he moved to Florida where he started teaching software engineering at Florida's institute of technology, the only school in the nation to my knowledge to offer an earned minor in software testing. Some of Dr. Kaner's former students are heavily involved in the test community today.

Then there is testingeducation.org, the source for free and open testing training materials. There is the Association For Software Testing (AST) that Dr. Kaner helped champion and organize. There is the Black-Box Software Testing (BBST) course that the AST runs for free to it's members, that Dr. Kaner helped create. There's Lessons Learned in Software Testing, the most recent book Dr. Kaner has served as lead author on. Not to mention his publication list, which would likely use up more than all of the ink currently in my printer. And I haven't even mentioned the LAWST-Style peer conferences that Dr. Kaner lead, inspired, and served as an initial host for.

That's the short list; I've tried to hit the highlights. But notice something about those publications: He was almost always the lead author. They weren't done alone; they were done as a part of a community. And he doesn't do the classic professor trick of sticking his name on top and letting the "little people" do the work: Dr. Kaner is a do-er through and through.

Now look at the dates - this is a gentleman who views career as a marathon. He started his first testing job on his resume in June of 1983, and hasn't stopped since. I already mentioned his publication list.

I don't think it takes a genius to connect the dots back those initial requirements, nor to see how Dr. Kaner is uniquely qualified for the luminary award.

Yes, I said uniquely.

When I think of a 'luminary award', sure, there are other names that come to mind, but when I think of those four bullet points, only one name passes the test. But, just as an exercise, let's talk about some of the folks who might deserve an award, just not this one:

Dr. Jerry Weinberg: Author of over forty books on computing, consulting and quality, Jerry has also ran an enviable marathon career, going back to being the test lead on the Apollo spacecraft in the 1960's! Jerry built his own special community and inspired a generation of consultants. James Bach went so far as to refer to him once as the "Prince of Testers." But that other requirement is to focus your career on quality and testing, and I'm afraid I can't say that was Jerry's focus. Consulting, certainly, and Delivery, yes, and maybe Quality, but testing is not the main thing of the main thing of the main thing for Jerry the way it is for Cem.

Dr. Paul Jorgensen: My old professor at Grand Valley, Paul Jorgensen spent twenty years in industry testing telephone switches before getting his PhD and teaching software engineering for twenty more. Paul is also the author of Software Testing: A Craftsman's Approach, now in it's third edition. Massively published, long career, testing and quality, yes, and I have great admiration for Paul. What I can't say is that he's built a community around him the way Cem has. Every one of Cem's project is collaborative, every one seems to get new people involved. Paul deserves and award for individual effort, certainly, and has done a great deal to foster community in his decades of service. I just don't think he's a perfect fit for this particular community award.

Buccaneer Scholar James Bach: Is probably the closest. James also contributed to create AST, he's keynoted at every single international test conference, he also co-authored LLST, and he's invested a fair amount of time in helping new testers get into the craft. Like Cem, his publication list is a mile long, and he's done a significant amount of training of testers, as well as doing. Yet he started later, and he was a contributor on one book, not a lead author of three. I've no doubt James is worthy of a luminary award, I'm just not sure it should be the very first one. Also, I expect in the years to come, he will become more and more worthy.

There are a few other names you could mention. Michael Bolton, for example, probably falls into the same category of James, and so do my friends Mike Kelly, Ben Simo, and a handful of others. There are also several people like Glenford Meyers, Richard Bender, or Boris Beizer who did very promising work but didn't stick around to wrestle with how their ideas were implemented in practice, to see if they worked. One thing Cem and James have done is come back to revise those ideas in light of feedback from the field - both in terms of what works, and how to explain the concept so there is a high transference rate.

Likewise, there are some agile-testing names that may be eligible for the award in a few years, but right now, I'm looking at the intersection of lifetime, published, community, testing and quality. And I'm looking at Cem Kaner.

Now I am open to other people interpreting those requirements differently, and I am open to debate. You can suggest anyone you please, and nominate and later vote for anyone you please. But I hope that, after some reflection, you will join me in nominating Dr. Cem Kaner for the STP Luminary Award within the next week. After nominations, STP will open the community to vote on the top three or four selections.

The nomination form is up on the web right now; please join me in nominating someone you respect, value, and appreciate.

Thank you!