Jared's Weblog

Main Page - Books - Links - Resources - About - Contact Me - RSS

Jared Richardson Most Popular
Help! I've Inherited Legacy Code
Testing Untestable Code
Continuous Integration... Why Bother?
Mock Client Testing
The Art of Work
Technical Idiot Savants
Targetted Skills Aquition vs Free Range Chickens
The Habits of Highly Effective Developers

Blog Archive

Sun, 12 Feb 2006

Problems with Test Automation

When I spoke at Adobe last week the title of my talk was It's All About the Platform. The presentation was about how to effectively leverage test automation and continuous integration to validate your platform products. After the talk a former Microsoft employee asked me why I didn't talk more about the problems with test automation.

I think I addressed the concerns at the time, but it's a good question that bears repeating.

First, you can only cover so much in a sixty to ninety minute talk. I already had more material than I could do justice. Trying to present the case for test automation, then continuous integration, then presenting some counter-points... it was beyond the scope of the talk.

Secondly, I've never encountered a shop that wasn't top-heavy on manual testing. In my experience, there aren't many shops abusing automated testing. So I present on topics like this I try to push hard towards automation, not because I think it's all you need, but because I'm trying to restore balance.

If your shop has only manual or only automated tests, you lack balance. You need a good mix of both. In pushing people towards test automation, I'm trying to establish that balance, not wipe out manual testing.

That leads into another good point. Test automation can't replace a skilled interactive tester. It can eliminate a lot of boring, repetitive work. It can keep the basic product rock solid, freeing up the (presumably) smart, creative human to look for another class of problem. You need testers and test automation. Neither stand-alone.

It takes a different sort of mind to do interactive, manual, black box testing than it does to do programming or test automation. Some people take a shot at programming and they just Get It. Other people are the same way when it comes to intelligently probing a program's limits. When your company decides to initiate an automation effort, please don't blindly convert your interactive testers to test automaters. Just because both titles contain the word test doesn't mean they require the same skill set. Some people may overlap the two jobs, but many others will not. If you do a blank convert of an entire department, you'll likely condemn your effort to fail and frustrate some good people along the way.

Remember to provide some structure when you start creating automated tests. A little design and architecture has been known to greatly increase the value of software. ;)

On the other hand, turning a dozen people loose to write automated tests (for the first time in their life) might create some really bad tests. Be sure to adopt a strategy that can guide the team and make sure they are doing the best job they can do. Blitzkrieg Testing is a great starting point for providing that structure.

Test automation is an extremely powerful tool that I rarely see properly utilized. It's not a magic wand or silver bullet, but it's a tool that you can use in your shop everyday.

So, when's the last time you wrote an automated test? When's the last time you wrote one?


posted at: 23:06 | path: | permanent link to this entry

Thank You Adobe for the Warm Hospitality

I spent some time last week on our west coast. San Jose to be exact. The good folks at Adobe invited me to speak at their QA Summit at the San Jose Convention Center (thanks to Jordan Dea-Mattson and Serena Zhou). It was an eye-opening experience for me. Adobe devotes an entire week to a series of technical conferences each year, and brings together their employees from all over the world, as well as a few external speakers like myself. I'm told Tim O'Reilly spoke earlier in the week... I really hate that I missed that!

Some managers fail to understand the value of getting their people together to talk, share, and learn, but Adobe "gets it." When I gave my keynote on platform validation, the room was packed... there were five to six hundred QA professionals in the room. Then we moved to an adjacent room for a series of round table discussions. Even though I led one of the tables, it was fascinating to watch Adobe employees from Ottawa sharing solutions and ideas with employees from India and Seattle. I answered some questions, but I also listened to a lot of good information being shared.

You have to admire a company that makes such a strategic investment into their employees. Ultimately this investment pays off company-wide. Way to go Adobe!

Does your company do the same for you? Are they as successful as Adobe? Don't you think there might be a correlation?


posted at: 23:01 | path: | permanent link to this entry