OK, let’s talk, for the ten thousandth time, about when story (acceptance) tests should be automated, and when they should not. I’m patient, in a manner of speaking. I can keep doing this until you get it.
When I say “story” test, I mean a test which has been provided to help the customer see that she has gotten what she asked for. Here’s an example, in a classic story format:
- As a purchaser,
- I want to see an itemized bill for my order,
- So that I’ll know what I’ve ordered and what it will cost,
- For example:
- CD: “Beatles White Album” $12.00
- Book: “XP Installed” $14.00
- Shipping: $4.00
- Total: $30.00
We use examples like this to ensure that we understand what our customer wants, and we show the results to our customer so that they’ll see what we’ve done. Ideally, they’ll see that we’ve done what they want; sometimes they’ll see that we’ve done something that they don’t want.
When should such a test be automated?
Frankly, the answer is simple: any test which is going to be done more than once should be automated.
There are a few kinds of objections to this fact (;->) that I’d like to talk about.
We shouldn’t automate the test because we might see something different the next time we run it.
Yes, if we’re going to look for something different, we quite likely shouldn’t automate it. Therefore we are not planning to do the same test more than once. The rule does not apply.
We shouldn’t automate the test if it is only going to be run once.
We cannot ever know that we will only need to do the test once, even if we plan to throw the code away after running it once. Why? Because we don’t know it will pass the test the first time. If it doesn’t pass … we’ll need to run the test again.
The more common case, of course, is that the feature is being written to be left in the system. Therefore there is a chance we’ll break it. Therefore we need to run the test more than once. Oh sure, we can guess, estimate, assume, possibly even sometimes calculate that we probably haven’t broken this feature. Would you stake your life on it, for the rest of the duration of your project? I wouldn’t.
There is no test that is guaranteed to be needed only once.
We shouldn’t automate the test if it would cost more to automate than to perform it manually as many times as we will perform it.
The whole point of automation is to make things faster and more repeatable than human manual processing. Will it take longer to automate this test than to do it repeatedly by hand? Well, we’re not very bloody good at automation, then, are we?”
It comes down to this:
Every story test, by definition, needs to get the same answers every time. We can never be sure that a story test will not be able to be run more than once, and most of them clearly do need to be run more than once.
Therefore the only excuse for not automating these repeated tests is that we aren’t very good at automation.
I’m not very comfortable saying that the reason I’m not going to automate my next story test is that I’m not very good. Are you?
