Then again, a test for it would be easy to include. And programmer tricks such as copy/paste errors happen all the time. If you feel comfortable that one-time manual testing does the job and that the risk of future failures doesn’t justify automating regression tests, don’t automate them. If your decision turns out to be wrong, you’ll get another chance to automate them later. If you aren’t sure, and it’s not terribly difficult to automate, go for it.
If you’re testing a life-critical system, even a very small risk of a regression failure is too much. Use risk analysis to help decide what tests should be automated.
See Chapter 18, “Coding and Testing,” for more about risk analysis and how it relates to testing.
One-Off Tests
Most times, manually executing a one-off test is sufficient. If automating a test doesn’t have payoff, why do it? Sometimes automation is worth doing for a one-off test.
Lisa’s Story
We recently did a story to pop up a warning message dialog when posting a payroll, but the message should only come up during the first two weeks of January. Automating a test for this functionality would require some way to simulate that the current date was between January 1 and January 15. That’s not terribly hard to do, but the consequences of a failure were fairly trivial, and we had more critical stories to deliver that iteration. Automating that test at that time just didn’t have enough value to justify the cost, and the risk factor was low. We decided to test it manually.
There are other cases where doing a one-off test seems the most intuitive but automation is a better choice. We host sites for different business partners, and each one has unique content, look, and feel. Values in the database drive the correct behavior and content for each brand. Some of the data, such as fee schedules based on asset values and numbers of participants, are highly complex. It’s much easier and much more accurate to verify this data using FitNesse tests. We have a set of fixtures that let us specify keys for the partner “brand” that we want to test. We can easily plug in the appropriate expected results from the spreadsheets that the business development staff creates for each new partner. These tests aren’t part of our regression suite. They’re used one time only to validate the new brand.
—Lisa
Tedious tasks may be worth automating, even if you don’t do them often. Weigh the automation cost against the amount of valuable time eaten up by manually doing the test. If it’s easy to do manually, and automating wouldn’t be quick, just keep it manual.
What Might Be Hard to Automate?
When code isn’t written test-first, or at least with test automation in mind, it’s much harder to automate. Older systems tend to fall into this category, but no doubt plenty of new code with the same untestable characteristics is still being produced.
If you’re faced with working on existing code that doesn’t already have automated tests, you’re in for an uphill battle, but a winnable one. Legacy code may have I/O, database access, as well as business logic and presentation code intertwined. It may not be clear where to hook into the code to automate a test. How do you get started automating tests on such a system? You certainly can’t plan on automating everything below the GUI, because much of the logic is in the presentation layer.
There are at least a couple of different approaches that work well. The “hump of pain” we talked about in Chapter 13, “Why We Want to Automate Tests and What Holds Us Back,” is intimidating, but it can be overcome, and then test automation will become much easier. Michael Feathers’
Lisa’s team decided on a different but equally effective approach. The team members started “strangling” the legacy code by writing all new features in a new test-friendly architecture. They’re gradually replacing all of the old code with code written test-first. When they do work on old code to fix bugs, or in the cases where the old code needs updating, they simply add unit tests for all of the code they change. A GUI smoke test suite covers the critical functions of the rest of the legacy system that has no unit tests.
Chapter 7, “Technology-Facing Tests that Support the Team,” goes into more detail about different agile approaches to legacy code.
As with any automation project, approach the hard-to-automate code one piece at a time, and address the highest risk areas first. Solve the testability problem and find a way to write unit-level tests. The effort will pay off.
