Which tests should be automated?

I don't know how many times at work I've had this topic come up. Typically a program manager (the one controlling the staffing) muses about how if we could just automate 100% of our testing then wouldn't everything be grand!  Think of the savings!

Of course this is an unreachable dream. Why can't 100% of tests be automated? Here are a few reasons that come to mind:

Ten Reasons Why 100% Test Automation is only a Dream

1. There are an infinite number of possible test cases.
2. We don't have enough trained staff to create the automated tests.
3. There isn't enough time in the schedule to create the automation.
4. The more tests created, the more time is needed for test updating and maintenance.
5. The greater the number of tests in our test run, the more time is needed to analyze the test run results.
6. Some test cases are too hard to automate, such as image or layout validation.
7. The test automation team doesn't have comprehensive product detail knowledge.
8. We don't have a comprehensive test plan.
9. There is no detailed functional specification.
10. Our test framework doesn't support manipulation of custom controls.

Well that is a bit of a discouraging list.  So at this point I need to decide how to prioritize our test automation effort. Here's how I usually approach the question of which tests to automate first.

Automate pri-1 "happy path" test cases first

We have a manual test team that knows the product better than the test automation engineers. If they have spent time preparing a test plan for manual test execution, great!  We steal some of that work and go through the documentation making a list of Pri-1 test cases. These are the most important "happy path" functions that the software or website supports, so we plan to automate these test cases first.

I talk to the manual testers and try to understand which test cases are the best candidates for automation. I key in on those cases that are simple to execute but involve a lot of repetitive effort, large forms to fill out, or generally mind numbing to repeatedly execute.  I have to use my experience to guage how much effort would be required to automate the test scenario. The best automation candidates are stable features that aren't frequently changed -- which would require updating the automation.

These pri-1 automated test cases also become excellent candidates for a regression test suite. I've been using Visual Studio recently and like to use both test lists and test categories for organizing our tests into sets that can be used for different purposes. I create a test list for "smoke tests" that includes only reliable deterministic tests that run fast and verify core functionality.

I create other test lists that include every test in our arsenal.  These we script to run during off-hours using Windows Task Scheduler because they will run for several hours.

 

Next automate pri-2 tests including negative test cases

If we have automated all the pri-1 cases we scan for secondary scenarios or important negative test cases. Another good source of automation cases is found in the bug reports. For example, we found that the developers had inserted a debug code into our website when a translated string was not found and the test team was finding these codes popping up all over the website. We automated a test that crawled the entire website looking for this code and saved our test team a ton of effort.

Our team rarely gets past automating a few pri-2 cases before the schedule moves us on to the next project.  What's your experience?