Testing is not part of the product

I run into the idea that test code (could be infrastructure, a tools, functional built into the product and others) is optional. Because it doesn’t ship with the product, it is a second class citizen.

On one hand, I totally agree. No customer will ever see or even care about how the product was tested. They just want it to work and expect that it has been tested carefully.

On the other hand…this doesn’t seem to work. Code that is testable seems to be better. It takes a long time to figure this out, at least with the products that I have worked with. The product has to ship and customers have to be unhappy. It doesn’t usually come down to a single big (or well publicized) bug. It’s the odd behavior that wasn’t thought through well enough or the odd an intermittent crash/hang/hiccup.

The big, awful bug is actually pretty easy to deal with from a development standpoint. Everyone sees it, you fix it, you’re done. Well, the lawyers may not be.

Anyway, the problem bugs are the ones that are subtle, not easily reproducible and cause a general sense of low quality or instability. In the end, they are all just bugs, with steps to reproduce. But, with systems as complex as they are today, this is not easy.

In fact, it requires a carefully built test infrastructure. Good product design, good development practices, an eye for testability, hooks, logs and functions built into the code to make it more transparent, a layered approach where you test each piece, test pieces together, do integration testing, stress testing and finally stability testing.

Skipping some of these steps can give you quite a shock. Instead of a quick fix, tiger team or some other tactical approach, what you may find is that you simply have no idea where the problems are or how to fix them. You find that your test tools are buggy and give bogus results. You find areas that have been completely skipped.

This isn’t necessarily a business disaster. There are plenty of examples of very successful, buggy and unstable products. But, is that the kind of product you want to work on?

My suggestion, if it isn’t obvious already, is that the core test tools and a usable test environment are part of the product. They should be developed, tested and maintained at the same level as the product itself. And, the expectation should be that these tools are of the same quality level, in terms of design an implementation, as the rest of the product.

It’s hard to do. And, it’s easy to waste large amount of money if you try, but don’t do it correctly (i.e. spend the money, but have a bad design or implementation). But, once your product ships, it will be far more expensive (in terms of time and money) to go back and put quality in.


No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: