Quality Metrics (...and other Fine Myths)

When I look at our North American (or at least developed nation) public systems, I am, well ... sad.

Education, Health Care, Government, pick any area of public life and you are likely to see waste, declining performance indicators and increasing cost.

It seems strange. For twenty-five years we have had this resounding gong of public education reform, but despite our best effort, things keep getting worse. It's all most as if we are applying Weinberg's 1st Law Of Management: If something isn't working, do more of it.

Yet I have hope.

You see, while I believe things are getting work in many public sectors, I believe that is accident, not essence. I believe it is possible that those things can be managed well. To borrow a line from Dr. Deming, the systems we have set of guarantee a certain outcome based on human nature, but it may be possible to change the system.

If I had to give a label to this idea of improvement, one label that might fit is quality. I am a strong believer in the idea of quality, in everything from pride in work to effective work to continuous improvement; I am even a card-carrying member of the American Society for Quality.

Now over years there have been a number of ways to look at quality. You can look at it as a toolbox of techniques, or a philosophy, or, as I prefer, a sort of systems thinking approach to problem solving. In any event, the term can mean different things to different people and I am okay with this; the root of qual is sort of an intangible that defies measurement. (If you could objectively measure it, it's not qual anymore, it is quant.)

Enter Quality

Now about this idea of quality improvement as systems thinking. That suggests that instead of numerical targets, we seek to understand what is actually going on in the process, optimizing for the things we value the most, using numbers as indicators but not for control.

I may have worded that poorly, but something like that is what a great number of systems thinkers advocate, from Deming to Weinberg, and, most recently, the British Consultant Mr. John Seddon.

Notice the inherent conflict between systems thinking and "management (control) by the numbers." With control by the numbers, you can look at numerical targets and, if the process is within specific control limits, well, you don't have to worry about it. The numbers are a sort of report card which allows you to have an understanding of the system without having to be directly involved in the work. Many control-oriented management thinkers often believe that separating management from the work is essential for success.

The problem comes when the report card itself is causing the dysfunction. John Seddon put it this way in a recent lecture:

Twenty-five years ago or more I was studying peoples behavior in organizations. You know, it's kind of odd that you take honest, God-fearing people off the street, put them in a building, call it an organization and they behave in very strange ways. It wasn't until I got to the work of Deming that I saw that it is actually the system that governs their behavior ... do you remember in the 1970's people went to Japan to find out what they did? They woke up to the Japanese Miracle so they go over and 'let's have a look', and they came up with the idea that this was quality circles and suggestion schemes and then it became kind of TQM and it failed, didn't it? Because they couldn't see the thing you need to see, which is a different way of thinking about the design and management of work.

Now back to my words

If we take Mr. Seddon seriously (and I do), then we will want to scrutinize applications of 'quality tools' to make sure the measurements make sense, and, even if they do make sense, will they introduce dysfunction into the organization. (The most common form of dysfunction I have seen is exploiting the difference between what is measured and the actual desired outcome, but another common form is when optimizing individual parts of a process hurts the whole.)

All this brings me to Mr. Paul Borawski's recent post on the ASQ national quality blog about the Pewaukee School District. In Paul's words:

Dr. Sternke is a passionate and committed leader. She has personally invested in understanding her school district as a system and applied the tools of improvement in the conduct of the district’s mission. It’s all there: A mission, objectives, strategy, and metrics. And over time, by concentrating on the vital few, Dr. Sternke, and the devoted staff of the school district have driven performance to ever higher levels. Success is measured in the classroom, but the business of the school system to support educational excellence is managed, too.

All this sounds pretty good, right? Here you have a professional looking at education as a system and overhauling it; that's good. The plan is public; that's good too. They have a defined strategy they can track against, which is good. So let's look at those goals.

What's your goal?

I originally had several thousand words here, listing some of the goals of both the local district and the school. The district requirements are exhaustive, and include everything from revising 100% of the curriculum to comply with "power standards" to getting a certain number of twitter followers to training a percentage of the teachers to use computers at a certain level of competence.

Instead of going through all of those, though, I decided to save your eyes and review just one document; the high level review of the high school performance in 2008-2009:

Academic Goal #1: Identify essential concepts and skills in all subject areas by June, 2009.

Results: Collaborative Department Teams developed Power Standards in each subject area in preparation for the 2008-09 school year. Power Standards were articulated to students via course syllabi and many staff members posted their standards on classroom walls for display and review. Essential concepts and skills were continuously reviewed and updated in all subject areas. Essential concepts/skills are clearly articulated in unit Curriculum Maps and they are the basis for the development of daily Learning Objectives.

Academic Goal #2: Evaluating the Pewaukee School District’s non-fiction writing program.

Results: Data collected during the 2008-09 school year focused on the frequency of non-fiction writing assigned. Individual teachers collected student samples and anchor papers for each formal writing exercise. Curriculum Maps and end-of-unit assessments were revised to reflect our focus on non-fiction writing and data clearly indicates that students wrote consistently and continuously throughout the year. PHS staff members played an integral role on the District Literacy Committee where the overall writing program was evaluated.

Academic Goal #3: 50% of all Common Unit Assessments will be comprised of mid to upper level critical thinking questions/activities as measured by Bloom’s Taxonomy.

Results: Data collected at the end of the 2008-09 school year indicates that nearly two thirds of all questions/activities found on end of unit assessments require students to think critically.

Is this kind of improvement really going to change a school system?

Forming a strategy

Notice that all of these goals are process goals -- none of them are actually tied to outcome. To get on my soapbox just once, this is a classic middle-management mistake. Measuring the process gives management a sense of control, but it does very little (or less!) to indicate that the outcome will be good.

When you think about it, measuring the process will tend to put our focus on the means (the education process) not the ends. (An educated citizenry? A productive citizenry?)

What is the real goal of the school system? With a strategy written in this way we don't know.

Now If I were looking at forming a higher-ed strategy, I'd start with asking the teachers what they need. My guess was it might involved new textbooks, guest speakers, time to develop materials. Then I would ask the teachers, parents and students what success means to them.

Somehow I doubt it would mean a percentage of questions have a higher level in Bloom's Taxonomy, that the school had created across-the-board standards, or that a certain percentage of classes had complied with a new special way to teach.

Indeed, instead of "standards", I would likely let the teachers experiment with whatever they would like, the use whatever external tools I have (state standard tests, pass/fail rates, AP test scores, student and parent qualitative evaluation, peer review) to measure outcome.

Then I would ask the high-scoring teachers what was working for them.

You see, first I would study the system to see what was working, then ask how we could improve improve.

Along the way, I might identify some obstacles in the nature of the work; for example, many schools are set up like a factory, with a "batch" size of thirty students or so, instead of using lean concepts like one-piece flow. Ideally, it would be nice to customize education to meet the child, so I would be talking about balancing that one-piece flow against other economic realities. I'd be talking about tough choices, about why we had to done thing and not another. (I can't help but notice the term budget does not appear in the goals, nor does opportunities or obstacles.)

I wanted a quality plan, but, I am afraid to say, we got a

C'mon, Matt, do you really believe this batch size vs. one-piece flow stuff?

Well, let me tell you this: I did not realize it at the time, but when my parents moved me from a school with a 30-to-1 student/teacher ratio to a private school at ten-to-one, it was in order to avoid the stigma of my failing out of 2nd grade. Within a few years of private school I was dramatically outscoring my peers.

Being a believer in continuous improvement, we decided to homeschool our children, driving the student/teacher ratio even lower. So yes, it does matter.

Except, of course, you will see no mention of that in the strategy document for the district.

I can't really blame the administrators or leadership of the Pewaukee School district. Given what they were given, I don't even think they did a terrible job.

You see, the school district lives within a greater system, the American Public Education system. That education system has it's regulations, mandatory tests, departments, laws and state offices, that frames the problems of education, and creates rewards and incentives, that make this kind of strategy very sad ... but all-too predictable.

Over the part decade we have seen a huge increase in education delivery by internet, in charter schools, homeschools, and in competition. It is competition in which I have hope. Competition is the opposite of standardization; it allows a thousand flowers to bloom, and then us to pick the best flowers by outcome, not process.

Perhaps, over time, we can pick the methods that seem produce good outcome. But to do that we need to experiment.

When you think about it, without the Pewaukee School district trying something, and doing it in public, we never would have had this essay. They are experimenting. In many ways, they should be applauded.

But make no mistake; you can't just make up a bunch of new standards and give them numerical targets and call it "quality."

It's got to be, like, good and stuff, ya know?

UPDATE: I shared the first draft of this interview with a few of my fellow "Influential Voices Program" Bloggers. One of them, Aimee Siegler, pointed out to me that her children in Wisconsin, in a district near to Pewaukee, are involved in programs that try to provide individualized attention, adjusting the program based on the child's ability level and less on age ... decreasing the batch size.

So it is happening. For some reason, though, that isn't on the radar for "continuous improvement" in the school system.

I think it should be.