Complete Naming Model for Test Automation

Language is tough. Paul Graham wrote an entire essay with the premise that Philosophy is impossible because you get bogged down with language, as opposed to math, which essentially is a super-precise language.

As someone working in Test Automation, I find that discussing the way tests are constructed and grouped can stand in the way of progress. After all, if you can’t agree on what Test Case is, how can you do any serious work? (just in case I’m not being clear: you can’t!)

So, I believe establishing a clear and precise vocabulary is essential to having an intelligent conversation about Test Automation.

This is my proposal for the language to use when talking about Test Automation. I have a page dedicated to an Test Automation Glossary, but don’t think it offers enough context. Also, I’ve been inspired by Paul’s essay to try and make it airtight; like math. (If I was a math geek. Which I’m not)

I would like it to be broad enough to cover all types of testing (manual or automated), encompass the testing that is done at most companies and is still precise enough that you could build an actual Test Automation System around.

Let’s start small and build up.

Test Case
The basic building block of testing is the Test Case. It can be explicit (i.e. the steps are written down) or implicit (as in exploratory testing). In Automation, there must be code somewhere. It’s usually a script that tests a single thing. It can also be a script that tests many things based on data inputs.

The configuration includes the build of the software under test, the hardware, the network, the network settings, the operating system, the version of your test, etc. It is the least well developed of my concepts so far, because it is so open-ended.

Test Case Data
These are the values or data you pass to your test case, if any. For example, if you have a browser test, there will usually be a set of URLs to test. These are not tests themselves, but rather data to be passed into the “open URL” test case.

Test Instance
This is a combination of three things:

  1. A single Test Case
  2. A single piece of Test Case Data
  3. A single Configuration

Simply, when you go to run a test case, it is an “instance” of that test case; it’s what you’re actually going to do, as opposed to the test case itself, which is something you “could” do. Anyone with any object-oriented programming will understand this easily, but it may not be clear to others.

Test Result
This is what you get from a Test Instance. You perform the Test Instance and you get a Test Result. Simple! See my recommendations on what I think these should look like.

This covers a lot of ground for a basic case, but I believe that it is universally true and that any of the 100’s of SQA systems, languages and techniques can be mapped into this language.

Test Suites, Scenarios, Groups, Catagories, Tags and so on

Test Cases can be arranged in many ways. I’ve seen flat lists, categories with test cases inside, groups, multiple level hierarchies and tagged by type, then combined into groups. I’m sure this only scratches the surface of the possibilities. The point for this model is that it just doesn’t matter. This is all Test Case meta data. Organize them how you like. Think of iTunes and songs. You can have the same song in multiple albums, in multiple encoding formats and with a picture or not. In the end, the song, like the Test Case, is the same.

Test Run
A Test Run can span the range from very simple to very complex. In our example above, if you were to run the Test Case with its Test Case Data on a particular Configuration, this would be a Test Run. Usually, though, you have a list of tests in a Test Suite and run them. This is also a Test Run.

A more complex variation, for example, is a matrix for browser compatibility testing. Say a test suite with 20 Test Cases that you plan to run on three different browsers and three different platforms. Let’s say that one of the tests is the openURL test we mentioned earlier and that it has an attached spreadsheet with 50 URLs to visit. This is getting to be quite complex (and very realistic!).

But, it can still go much, much further. Think of a Test Run in which you have a group of three test suites (groups of Test Cases) that you would like to run multiple times, in random order, with a random selection of Test Case data appropriate for that Test Case and repeat until failure (or forever). On one hand, this would be really tough to define.

At the lowest level, though, it’s just a list. In fact, I would define a Test Run as a flat list, possibly infinitely long, that contain lines like so:

Test Case + Test Case Data + Configuration

Because of the complexity, a Test Run needs to be a program (or at least have the possibility to be a program), to provide the necessary flexibility.

So, that’s it!

I would like to expand with examples of how some test harness fit into this model. But, enough for now.


No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: