Long-time readers of Creative Chaos will know that I'm not a big fan of X-drive-Y processes. Sure, Test-Driven Development was great, but now we've got so many abbreviations poured on top that things are getting a bit silly. (For the record, I think X-driven-Y jumped the shark at Bacon Driven Coding, but that's just me talkin')
But, after some time of opposing Behavior Driven Development as a "bunch of nothing'", I have to admit, I do believe I was wrong. Let me explain.
Several years ago, around 2005-2006, I noticed a disturbing trend of super-isolated unit tests that weren't actually testing the return results of functions - they were instead testing what the function was doing. So your function would call int() once and printf() three times - that would be the test. According to the unit testing zealots, actually having real objects, connecting to a real database, etc - well, that was "not a unit test."
I found this resulted in real gains in /design/ of the software, but the regression test suite it produced - not so much. The suite was brittle and not good at catching bugs. I found ways to inject bugs a suite would not catch, or refactor the code so that it worked but the suite would trip an error. (For example, change from three print()s to instead build up a combined string, then just print() once. Code still works, regression test suite registers a 'failure'.)
My colleague, Sean McMillan, and I developed a few cautionary tales on this form of ultra-low-level "testing and presented it at the Google Test Automation Conference in 2007:
Our conclusion? This is interesting stuff, but we wouldn't call it "testing" - perhaps "isolation-based design" would be more accurate.
So this intersects nicely with Behavior Driven Development, specifically, the falvor of BDD that gave rise to a /behavior/ framework called RSPEC.
You see, under RSpec, you don't call it a test. You talk about the /behavior/ of the software at a low level, replacing words like "test" and "assert" with "should" and "ensure." BDD is about design and doesn't claim to be about testing.
Sound familiar?
So, I think this is huge. Some of the the BDD people took the same observations I did about low-level testing and design and did something positive about it. David Astels, I salute you.
Now, there is a different flavor of BDD, one that is higher-level, where the requirements are expressed as a specification of the form "Given ... When ... Then." If the team has objects with concrete nouns ("The customer", "A membership packet") and verbs ("Requests"), it's relatively easy to automate those tests expressed in near-English.
From what I can tell, the jury is still out on BDD at the customer level. A few people I respect (Including Chris McMahon) are doing it. I'm cautiously optimistic, in that I suspect the process will have some value, but I'm not exactly sure what.
But, hey, as you can tell from above, I've been wrong before.
So - is anyone here using BDD at the higher levels? What do you think of it? And how long have you been doing it? I'm curious who has used it for more that a couple of years in a row, and what tweaks are required in the process as it gets older.
Monday, August 24, 2009
Subscribe to:
Post Comments (Atom)
6 comments:
For one of our simpler components I started with some BDD style Slim tests lately. The resulting test suite looks fine for me, though I'm not overly too ambitious for dealing with a more complex component/application in that mood.
For my Rails projects, I got infected with BDD and RSpec because it allowed (forced?) me to separate so many concerns, and I could focus much better on the design of the code. When I finally got around to doing RSpec seriously, Cucumber was mature enough to work at the higher level, to make sure that things were plugged together properly. I know of several Rails shops that use the RSpec-Cucumber combo.
I've also used higher-level testing on Java code. For some reason, I can't give up JUnit (so I remain TDD, not BDD, in Java), but I've taken up mock objects, so higher-level tests are necessary. I've used FitNesse for these in the past, but it doesn't work well for my domain (interpreters and compilers), so I wrote my own tool CIAT.
Supposing we testers worked from the premise that how programmers do their work was none of our business? That knowing how they do their work might be interesting and informative as to how we test, but that we testers might choose to observe the outcomes rather than having overly strong opinions about the methods that produce those outcomes?
What often gets left out of these discussions, for me, is this: I've seen programmers do great work test-first, and without test-first; using TDD or using unit tests; using BDD or writing the code from scratch. Unless I'm managing a programmer, I try (Lord, I try, though it's hard sometimes) not to be presumptuous about How Programmers Should Do Their Work. Apart from the principled stances of asserting No Best Practices and allowing other people to drive by their own headlights, there's another reason: I'm tired of hearing programmers telling testers how to do their work.
---Michael B.
Michael -
To speak to Michael's point, Please remember, in the 2005/06 timeframe, I /was/ a programmer who loved to test. Or, at least, my business card said "developer" on it.
So I went to GTAC to talk about developer-facing tests that might be of some interest to some testers, something I was skilled and qualified to do.
Some who test, also program. Not all, and I don't think it's required. I hope my post came off in the right spirit ...
Alistair Cockburn talked in his keynote at Agile 2009 about testers making decisions around system correctness. Interestingly, this happens _after_ the program has already been written, meaning that the only way in which those decisions can play into the code is through rework (we call them bugs).
Chris Matts also addressed this in the Real Options comic he handed out. He moves the testers' input to sit before the developers', so that the maximum amount of information can be found before commitment. The question I asked Alistair was really Chris's: is there any way to get those decisions about system correctness into the system before implementation?
Alistair reckons there are only 10 companies he's heard of who have managed to put the acceptance tests first as "fine-grained requirements". I suspect he doesn't know about several companies out there using BDD with Fitnesse, JBehave, Cucumber, Concordion, or just plain old XUnit. It's getting better.
To Michael: I'm happy for you to do your work in any way that suits you. I'd like to redefine what your work is, rather than how you do it. Instead of giving feedback on the developer's understanding of the system _after_ they start coding, would you be able to do it _before_? The developers, after all, are the most immediate consumers and customers of your work.
BDD is one way to shift the focus of these decisions. Good old fashioned Acceptance Test Driven Development is another; the only real difference is the language used (Given / When / Then over Setup / Exercise / Verify, etc) which I've found makes conversation with the stakeholders of the system much easier.
To Matthew: If you want to know how it works at high levels, please give it a try and share your problems on the newbie-friendly BDD dev list:
http://groups.google.com/group/behaviordrivendevelopment
I discovered RSpec as part of a Selenium-Ruby framework I got to work on and from there started to learn more about BDD.
The BDD perspective blew me away and I'm convinced it's a real step forward in joined up development thinking.
As I belong to a test consultancy that does just testing it's confusing for us to talk about BDD ("...but we mean testing as we don't do dev")to clients. However, we adopted the BDD perspective over the last year to our UI centric / Web based testing.
Howver, when we talk about what BDD looks like as we engage with it and mirror the thinking and practice for testing we call it BDT, Behaviour Driven Testing.
My colleagues and I have used it as the main approach on four engagements now and what a difference.
There's a presentation on my site and I've blogged a number of times about it.
The approach uses Cucumber, RSpec, Selenium, Ruby and the companies test management tool.
Michael - "I'm tired of hearing programmers telling testers how to do their work." It was actually that thought and the way BDD literature was developer centric in it's presentation that prompted the writing of the BDT material.
Post a Comment