Brian Estes

I signed up for one of the Comcast low-cost-for-six-month deals, giving me a significantly faster Internet connection. I haven’t actually *switched* yet, as Comcast offers only about 1/10 the value as my current ISP, meer.net. But…the speed is nice.

So, I thought I was see what it was like to use the Apple TV in a normal situation (for my family): we’re interested in watching a movie and don’t have one everyone can agree on in-hand. Since I was excited to try this out, I strongly suggested with give the Apple TV a try. My kids scrolled through the selection for a while and finally settled on Jimmy Neutron. We asked the kids to put on their PJ’s while daddy tried to figure out the Apple TV (to stall for time).

I went ahead and bought the movie (I could have rented, but it looked like it was the kind of movie we would watch more than once) and it said I could start watching it about 5 seconds later. Wow.

I didn’t actually start it for another 10 minutes or so and we were able to watch it without any problems. Since this was the first time I’ve purchased (or rented) anything non-HD, it transferred back to my Mac as well. Everything went as smoothly as possible.

Next experiment: HD!

I’ve had both the AppleTV and iPhone for a while now and thought I would share a couple of my impressions.

AppleTV

I’ve had mine since 1.0 and have used it on a regular basis ever since. My kids like to be able to watch cartoons and some TV shows I downloaded for them. I find it much better than TV itself because it starts and stops (and pauses) exactly when we want. It’s also commercial-free, which is a big deal to me.

High Definition movies is HUGE for me. I have one of the early Mitusubishi 65″ widescreen TVs. It is fantastic and I’ve enjoyed it for years. But…and this is a biggie…it’s not hobbled with the HDCP copy-protection scheme. For all of you out there who are not earlier adopters, this means I can’t watch any high definition content, like BluRay.

The AppleTV, though, has component video outputs. I was really anxious to find out if they would pass high definition content and am thrilled to say Yes! they do. So, this is my one and only source for better-than-DVD content.

On the downside, the policies for this content are really harsh. When I first read about it, it didn’t seem too bad. But, in practice, the rental service doesn’t work for me at all. Let me explain.

When you download a movie, you get 30 days to watch it and once you start, you have 24 hours to finish. High Definition content can ONLY be downloaded from the Apple TV and cannot be transferred back to your Mac or to your iPhone. You also can’t buy it, only rent.

I have a pretty slow Internet connection, so to watch, I need to download the movie a full day ahead of time. No spur-of-the-moment rentals for me (Standard def? Sorry, not interested. I’ll drive to the video store to get the better picture from DVD). Then, I need to make sure I have time for the whole movie once I touch play.

So, I did this as soon as the service came out. I rented No Reservations and sat down with my wife and a friend to watch it right away. I was impressed. The quality was definialty better than DVD, the sound was Dolby Digital and sounded great. I later saw  BluRay and would classify this picture as about half-way between DVD and BluRay. In other words, really nice!

So, I rented a second movie. I can’t remember the title. 30 days later, it was gone. In 30 days, I never found a night where I was sure I could finish it. This sucks. I feel ripped off. For whatever reason, this $5 is very annoying, in the same class as ATM fees. I’ll do (almost) anything to avoid them.

So, until these policies get updated (and it doesn’t have to be a big change) or I get an Internet connection that is fast enough that I can rent and watch a high definiton movie in the same night, I’m afraid I really can’t use it.

The other feature of 2.0 that is really cool is the way it will show you your whole iTunes library and simply use the AppleTV as a display device. My home network is fast enough that this is seamless. Unfortunately, I can’t use this feature either, because the biggest user of my Apple TV are my kids, and the only way I can limit content is to make a special playlist and only sync that. 

iPhone

Some surprising findings:

I watch a lot of video on my iPhone. I’ve never done this with a mobile device before.

WiFi isn’t very useful. I *thought* that because I have WiFi at home and at work and am usually at Starbucks when not in either of those places, that the slow speed of EDGE wouldn’t really be an issue for me. Well, it turns out that WiFi doesn’t work very well at all. Technically, it’s great. The iPhone remembers your favorite network, switches automatically and seamlessly and is much faster. But…most places are covered with WiFi now and they all require “click here to accept” or “it’s free, just watch this dumb ad” buttons. If I *happen* to be using my browser when this happens, I can deal with it. But, if I’m getting mail, looking at the weather, etc. it just doesn’t work. By the time I figure out what’s going on, get in there and turn off WiFi or whatever is appropriate, I’ve moved on. Oh well. EDGE isn’t really that bad.

I’m hoping with the SDK, somebody will write a little program to “auto-accept” all of these (un)welcome screens for me.

I have many more things to share, but this is long enough already!

Language is tough. Paul Graham wrote an entire essay with the premise that Philosophy is impossible because you get bogged down with language, as opposed to math, which essentially is a super-precise language.

As someone working in Test Automation, I find that discussing the way tests are constructed and grouped can stand in the way of progress. After all, if you can’t agree on what Test Case is, how can you do any serious work? (just in case I’m not being clear: you can’t!)

So, I believe establishing a clear and precise vocabulary is essential to having an intelligent conversation about Test Automation.

This is my proposal for the language to use when talking about Test Automation. I have a page dedicated to an Test Automation Glossary, but don’t think it offers enough context. Also, I’ve been inspired by Paul’s essay to try and make it airtight; like math. (If I was a math geek. Which I’m not)

I would like it to be broad enough to cover all types of testing (manual or automated), encompass the testing that is done at most companies and is still precise enough that you could build an actual Test Automation System around.

Let’s start small and build up.

Test Case
The basic building block of testing is the Test Case. It can be explicit (i.e. the steps are written down) or implicit (as in exploratory testing). In Automation, there must be code somewhere. It’s usually a script that tests a single thing. It can also be a script that tests many things based on data inputs.

Configuration
The configuration includes the build of the software under test, the hardware, the network, the network settings, the operating system, the version of your test, etc. It is the least well developed of my concepts so far, because it is so open-ended.

Test Case Data
These are the values or data you pass to your test case, if any. For example, if you have a browser test, there will usually be a set of URLs to test. These are not tests themselves, but rather data to be passed into the “open URL” test case.

Test Instance
This is a combination of three things:

1. A single Test Case
2. A single piece of Test Case Data
3. A single Configuration

Simply, when you go to run a test case, it is an “instance” of that test case; it’s what you’re actually going to do, as opposed to the test case itself, which is something you “could” do. Anyone with any object-oriented programming will understand this easily, but it may not be clear to others.

Test Result
This is what you get from a Test Instance. You perform the Test Instance and you get a Test Result. Simple! See my recommendations on what I think these should look like.

This covers a lot of ground for a basic case, but I believe that it is universally true and that any of the 100’s of SQA systems, languages and techniques can be mapped into this language.

Test Suites, Scenarios, Groups, Catagories, Tags and so on
Test Cases can be arranged in many ways. I’ve seen flat lists, categories with test cases inside, groups, multiple level hierarchies and tagged by type, then combined into groups. I’m sure this only scratches the surface of the possibilities. The point for this model is that it just doesn’t matter. This is all Test Case meta data. Organize them how you like. Think of iTunes and songs. You can have the same song in multiple albums, in multiple encoding formats and with a picture or not. In the end, the song, like the Test Case, is the same.

Test Run
A Test Run can span the range from very simple to very complex. In our example above, if you were to run the Test Case with its Test Case Data on a particular Configuration, this would be a Test Run. Usually, though, you have a list of tests in a Test Suite and run them. This is also a Test Run.

A more complex variation, for example, is a matrix for browser compatibility testing. Say a test suite with 20 Test Cases that you plan to run on three different browsers and three different platforms. Let’s say that one of the tests is the openURL test we mentioned earlier and that it has an attached spreadsheet with 50 URLs to visit. This is getting to be quite complex (and very realistic!).

But, it can still go much, much further. Think of a Test Run in which you have a group of three test suites (groups of Test Cases) that you would like to run multiple times, in random order, with a random selection of Test Case data appropriate for that Test Case and repeat until failure (or forever). On one hand, this would be really tough to define.

At the lowest level, though, it’s just a list. In fact, I would define a Test Run as a flat list, possibly infinitely long, that contain lines like so:

Test Case + Test Case Data + Configuration

Because of the complexity, a Test Run needs to be a program (or at least have the possibility to be a program), to provide the necessary flexibility.

So, that’s it!

I would like to expand with examples of how some test harness fit into this model. But, enough for now.

(I’m using Firefox this time so that the WordPress editor has a chance of working.)

It seems like every test harness has a different set of possible results. I’ve seen Passed, Failed, Skipped, Abort, Error, Expected Failure, Unexpected Pass and many others. Seems exciting and technical at first. But, what do you *do* with this data? I’ve found that there are really only two possible actions for each test result:

1. Ignore the result

2. Do something about the result

There are many other associated actions, like looking for trends, measuring progress, getting warm fuzzy feelings, etc. I usually find that these are management functions. To actually make the product under test better, you have to follow up the issues and fix them.

So, I favor a very simple set of possible results:

Passed: You can ignore this test. It worked. Nothing to see here. You run the risk of getting false positives and this is important. But, I’ve yet to work on a project where we can actually keep up with all the failures and so have never done any significant research into method for capturing these more subtle problems.

Failed: This is a BUG. Getting to root cause may be difficult, but there *is* a problem. It might be a problem in the script, a bug in the product, a conscious design change or any of a thousand other reasons. In the end, though, it needs to be fixed, and therefore is a bug.

Skipped: The test isn’t going to run for a good reason. The test case itself knows this good reason. To make up an example, you might have a test suite for browser compatiblity. There may be some tests specificialy for Internet Explorere and some other tests specifically for Firefox. If you run the full suite on both, the tests designed for the other browser should skip, not fail.

One other result type I have had difficulty avoiding is the “performance” test. Usually it gathers data, which can then be analyzed later. I don’t find that this is the best use of resources, though. Whatever the performance test is doing, it is a *test*. So, it should also pass, fail or skip. It needs to *also* keep the metrics it was designed for, but why waste a testing opportunity?

My cell phone service was cut off Friday afternoon and so I needed to get new service ASAP. Having kids in school and day-care means having a reliable cell-phone mandatory these days. Plus, I need to surf the web while standing in line at Starbucks! I started doing research on the plans, coverage and other options, but didn’t get very far. It turns out my wife was getting me an iPhone for Christmas and so I got to open my present early this year!Opening an Apple product is always quite an experience for me. I like to sit down in the middle of the room, maybe have a glass of wine, relax and then carefully open the layers. The iPhone did not disappoint. First, the box is very small. I guess most Apple products come in small boxes these days, being environmentally sound and all. Although it was small, it was quite heavy. It felt nice. I would guess it felt to me like a box of jewelry from Tiffany would feel to a woman–perfect, luxurious, substantial and valuable.I can’t help but smile just thinking about it. The iPhone itself was nicely displayed and everything else was tucked under a black box under the iPhone itself. Very classy. When I turned it on, it said I needed to connect with iTunes. I knew this from reading about it and so was not surprised. I would have liked to surf the web a bit over my WiFi connection or maybe just look at the menus, but no big deal. Once I connected,  the activation part started. At this point, the entire experience turned sour. I knew it was coming. I’ve done it before. But I haven’t actually gone through a cell phone activation process for many years.So, what is it I object to so much? First, the phone number. I have to have a phone number to get a cell phone. My whole being screams out in frustration and anger at this. Why in the world do I need a phone to get a phone!? I know the answer, of course. It’s just so they can sell it to people to make sales calls or tie it together with other information in order to sell me stuff. Oh, and so the NSA has an easier time tracking me, although I’m sure this is just a nice side benefit for them and not actually part of the business model. Do I have a phone? Yes. Although I didn’t once before when I first experienced this (maybe 10-12 years ago). I had just moved to the bay area, but didn’t have a place yet, therefore no phone. More on that later. So, I gave it and provided my number. It looks like they actually take the time and effort to verify it as well, so no 555-1212’s (I didn’t try this, just guessing). Next, they needed my billing and home address. OK, so they have to send me a bill, I guess. Most other services I deal with these days don’t, but this is a pretty old and crusty company and I’m sure it never crossed their mind to just let me punch in my Visa or Paypal or whatever. So, I give the billing address. But, that’s not good enough. They want my home address to. Why? Are they going to come to my house?This really makes me mad. In order for them to “loan” me the $100 or whatever they are subsidizing the iPhone with, I have to give them my life. Why can’t I just pay them the $100? Or, put it on a Visa? Or give it to them and get it back later if I manage to pay my bill for 24 months? Obvious, again. They want this information to sell. And, they do some serious checking on it too. They have some kind of database that has addresses and you have to be in it in order to activate (or call; catch-22 again).Finally, and so unbelievable that it doesn’t even upset me that much, is the requirement of a social security number. I’m not sure I’ve put my social security number on anything except my tax returns for years. I had forgotten what it was like to be poor and at the mercy of these monopolistic corporations (not that I wouldn’t mind if *I* was the monopolistic corporation). But, still. This sucks. Again, I’m sure this is just for money. The personal profile they build up is very complete and of high quality, so I’m sure it is worth quite a bit. I’ve looked at my credit reports in the past and much of the information is either totally wrong or out of data. Fortunately, it hasn’t been wrong in a bad way, but I know it’s just luck.  With this personal information strip search, I guess it will be more up-to-date now. I managed to get through this process (my wife was not happy that I was so upset and threatened to take the iPhone back a couple of times if I didn’t get over it). So, I did. And, it is wonderful, just like I expected. It is beautify to look at. The finger controls are really fun to play with. The set of built in applications is compelling (which is good, since I can’t add any others yet!). My phone was activated before I finished reading the page on the screen that was telling me about activation. I was very impressed. It found my WiFi and just started using it.I made my first phone call tonight and it was very clear. My music and videos transferred over without any issues, along with my bookmarks, calendar and contacts. I’m thrilled–even though I had to sell a bit of my soul.

Wow.Joyent went from this to this.As someone who has dealt with both Sun and Dell, my experience was similar in some ways, opposite in others. Dell was very excited to talk, responsive, has a nice web site and low prices. On the downside, they have the whole “medium and large business” scam going on (same stuff; higher prices). Their idea of same-day on-site service: have *me* solve the problem, then ship out a part that comes days later. And, it took weeks to get the 16 servers I ordered to actually arrive. If you count down to the last cable required to make it work, it was more like two months.When I called Sun and said I had $100k to spend (seemed like a lot to me!), they immediately passed me off to a VAR. For my needs, this actually worked fine. We worked with Jeremy at Groupware Technologies and they were fantastic. The machines came, worked fine, no issues. None have broken, so I don’t know about their service.I hope the Dell Joyent deal provides the wake-up call that Sun needs to actually take some action on this problem!

I’ve been writing quite a bit on my blog, but don’t seem to be able to actually finish any of my posts. They’re all sitting waiting for more editing. I’ve become very impressed with Joel and other who blog on a regular basis about current topics.

Anyway, I’ve always been interested in servers. Both the software and the hardware. The software I prefer is almost completely open-source now. The hardware has had a few changes.

In the past, I liked name-brand, powerful (expensive!) servers. Sun, Compaq ProLiant, etc. Then, I found that white-box systems were unbeatable in terms of price/performance. And, by having more boxes, you could get as much or more reliability.

But, things have shifted again (for me, at least). I’ve grown tired of so much integration work. This processor *should* work with this motherboard. This network adapter *should* work with this switch. And so on.

My last big hardware purchase was for Sun x4100’s and I’ve been thrilled. They are powerful and have had exactly zero maintenance so far (it’s been over a year).

I’ve also been curious about how the Google Apps work. I haven’t had much of a need for them, but when I was dreaming about new hardware, I realized that I could give them a try.

I created a “Fantasy Server” in Google’s Spreadsheet and have published it. It’s a work in progress. I don’t have an actual application (breaking rule #1 of building a server!). But, it’s still fun!

Fantasy Server

The latest Sun box, the 4450, seems really powerful. The biggest thing I’ve noticed with the x4100’s is that having four CPU’s (well, two CPUs with two cores) is already outstripping the disk I/O by a wide margin. Even with 16GB of RAM.

I have an Apple MacPro for my workstation, which has two quad-core CPUs at 3Ghz each. Even though I have a couple of Raptors and the Apple default drive, the processors just don’t have much to do. It’s all about the hard drive.

So, in my latest server design, I included a LOT of spindles. I also went with the 2.5″ SAS drives. Many people seem uncomfortable with these and cite capacity, cost, etc. To me, the last thing I want is a drive that is so big that it takes a long time to get all the data from it. I want lots of little drives, so I can break up database file, log files, system files, etc. in appropriate ways.

Anyway, take a look and give me any feedback if you have it.

I run into the idea that test code (could be infrastructure, a tools, functional built into the product and others) is optional. Because it doesn’t ship with the product, it is a second class citizen.

On one hand, I totally agree. No customer will ever see or even care about how the product was tested. They just want it to work and expect that it has been tested carefully.

On the other hand…this doesn’t seem to work. Code that is testable seems to be better. It takes a long time to figure this out, at least with the products that I have worked with. The product has to ship and customers have to be unhappy. It doesn’t usually come down to a single big (or well publicized) bug. It’s the odd behavior that wasn’t thought through well enough or the odd an intermittent crash/hang/hiccup.

The big, awful bug is actually pretty easy to deal with from a development standpoint. Everyone sees it, you fix it, you’re done. Well, the lawyers may not be.

Anyway, the problem bugs are the ones that are subtle, not easily reproducible and cause a general sense of low quality or instability. In the end, they are all just bugs, with steps to reproduce. But, with systems as complex as they are today, this is not easy.

In fact, it requires a carefully built test infrastructure. Good product design, good development practices, an eye for testability, hooks, logs and functions built into the code to make it more transparent, a layered approach where you test each piece, test pieces together, do integration testing, stress testing and finally stability testing.

Skipping some of these steps can give you quite a shock. Instead of a quick fix, tiger team or some other tactical approach, what you may find is that you simply have no idea where the problems are or how to fix them. You find that your test tools are buggy and give bogus results. You find areas that have been completely skipped.

This isn’t necessarily a business disaster. There are plenty of examples of very successful, buggy and unstable products. But, is that the kind of product you want to work on?

My suggestion, if it isn’t obvious already, is that the core test tools and a usable test environment are part of the product. They should be developed, tested and maintained at the same level as the product itself. And, the expectation should be that these tools are of the same quality level, in terms of design an implementation, as the rest of the product.

It’s hard to do. And, it’s easy to waste large amount of money if you try, but don’t do it correctly (i.e. spend the money, but have a bad design or implementation). But, once your product ships, it will be far more expensive (in terms of time and money) to go back and put quality in.

Elements of a Test Automation Tool

I have someone show me or tell me about a new automation tool a couple of times a year. Sometimes it is something the person made themselves or a product they read about somewhere. Other times, it is a partner that is showing off their automation and tries to say that it covers everything.

The tool usually comes down to a single part of what I’m about to describe. The ability to write a script, capture a screenshot, push buttons or for the code-centric, a library, harness or development environment addition is usually a long ways away from having all the pieces required to do automation.

I’m sure I’ve forgotten some things. I’m also sure that some are unnecessary in certain environments. But, most of these are generic and are required in some form.

So, here is my list:

Provide a way to program (from least to most sophisticated)
–API’s for use with C
–vendor script (Like 4Test that is used with SilkTest)
–open source language with interface (Perl, Python, Ruby or PHP)
–action word (or even multi-level action word) system

UI Elements
–identify all UI elements native to the OS
–provide a way to identify custom elements, gadgets and other non-native elements
–associate each instance with a name (or multiple names, for robustness)
–give them names (provide a naming structure). Sometimes call declarations.
–provide a bitmap and coordinate system as a fallback

Framework hooks
–start a script
–start a testcase
–end a testcase
–end a script

Test case management system
–way to identify a test case
–way to identify a test suite
–way to feed data into a test case (making it a “data driven” test case)

Test Execution
–Ability to execute a single test case (from the command line)
–Ability to execute a test suite
–Ability to execute a group of test suites

Logging system
–Pass, Fail, Skip
–Capture error messages
–Ability to log warnings and other debugging text
–Ability to save log to a file or to a remote system/database

Validation function (did the test pass?)
–Bitmap compare
–String compare
–Window exists

Agent (or other method to accomplish the same thing)
–Run the least possible code on the system under test
–Have a way to communicate with the system under test (via TCP/IP, serial, USB, etc.)
–Have another way to communicate with the system under test (you may want to test the communication channel you usually use for control, so there needs to be a way to switch to a secondary channel).
–Have a way to automatically start and stop the agent code (at boot or at application launch)
–Have a way to install and uninstall the agent without user interaction (including it in the build is not good enough, unless you are willing and able to ship it as part of the product).
–Have source code to the agent. This is essential if you are working on a non-product OS (i.e. you are developing an OS or parts of an OS and not just an application).