Archive for March, 2008|Monthly archive page

Why Test?


Why test?

It seems like an easy question.

Maybe quality? Desire to make the best product possible? Pride of workmanship? Zero-defect policies?

I don’t think so.

Looking through the noise, I think the reasons are more practical.

Here is the hierarchy of testing (think Maslow’s Hierarchy of Needs) http://en.wikipedia.org/wiki/Maslow’s_hierarchy_of_needs), in order from most to least important.

1. Compliance
2. Certification
3. Liability
4. Viability
5. Meets the Specification
6. Pride

Compliance
Tests that must be executed and passed in order for the product to be sold legally. I question whether this is testing at all, in fact. It looks like testing. But, if you were able to pay 1/2 the cost of testing to be automatically certified, would that happen on your product? Probably. In which case, Compliance is an *expense* and expenses are managed down to the bare minimum. You’re not looking for bugs. You are trying to pass the Compliance criteria. If you *happen* to find bugs along the way that help your product, great. But, it isn’t the actual goal or focus.

Certification
Tests that must be executed and passed in order to sell the product on a private network, with a special logo or to be sold to the government. Some products are not viable without certification (like cellular phones on a specific carriers network – you can’t sell it unless it passes). Others, like Windows Logo, expand the number of customers available. For enterprise products, it’s almost a requirements. For games, it’s a nice-to-have. Certification and Compliance are very similar and have the same focus.
Liability
Does it work? Does it do what you say it does? Does it cause damage (delete random files)? Does it expose the user to liability (bad security model that exposes personal information)? Whereas Compliance and Certification are black and white: you either pass or not, liability is a judgement call.  Don’t be confused by “no exceptions policies” or “mandatory improvement plans” or whatever corporate-speak is popular today–if the executives in your company think you can sell the product, you pass), 

Viability
Does it actually work? Is it likely to sell? How does it do against the competition? Is it as least as “pretty” as the competitors? A marketing call, although test data is often required to guide the decision. Testing doesn’t usually add to the viability of the product, unless one of the main differentiators is that this product is less buggy than your competition.

Does it meet the specification?
We’re quite a ways down for a reason most people think of first. Think about it though: if you have a product that fails one of the above, but meets the spec, will it ship? Usually not. But, if it meets all of the above and does meet the spec, will it ship? Usually, in my experience. So, meeting the spec is a nice-to-have. It’s also tremendously difficult, because you have to have a complete and up-to-date specification and have had enough time to consider how you would test it in detail. There are probably industries or areas that move slowly enough for this to be possible, but I haven’t ever actually seen one.

Pride
This is not usually an actual requirement, no matter how much lip-service it receives (at companies). Individual developers, however, especially those writing open-source software, may strive for perfection and test until they are satisfied. If you see it at companies (and, the first four have to be complete in order for this to be a real test), it will probably be a smaller company with a strong founder/technical lead.
I consider this the only “pure” area of testing. The *only* goal here is to find bugs in the product, with the goal of making the product better.

What about software that is sold for $0? (open-source, free software, etc.)
The same rules apply, for the most part. Even though the developers might not take responsibility for having their software brought into compliance or certification, the person or organization that uses it still must. In other words, you can’t skip the hierarchy without incurring some kind of penalty.

What are the implications of the hierarchy of testing?

It provides a clear framework for understanding and communicating objectives. When asking a test team to perform a specific task, it is important for those doing the asking (executives or development managers, usually) to know exactly what they are asking. For the test team, it is important to understand the objectives so that the appropriate trade-offs can be made.

To often, I find expectations for what testing can achieve to be far greater than reality suggests. The fault is often shared, because the test team is motivated by noble and abstract goals like “delivering a high-quality product” and not simply “get this thing certified.”  The executives/managers also seem reluctant to give clear direction. To them, “deliver a high-quality product” is crystal clear: ship a product that will make us tons of money as soon as possible!

They also have some specific ideas about what contributions the test team should make, like “manage the certification process and get through any roadblocks” and “make sure there aren’t any bugs that we care about” (like ones that will stop us from selling the product or getting sued). But, it’s not what they ask for, which leaves the prioritization of testing tasks, ranking of bugs, etc. to be pushed further down in the organization.
So, why test? You tell me.

 

Advertisements