One reason it takes so much longer to perform a test than to use the system in real life is that the tester must carefully observe the results and record them as the test is performed. Both of these tasks, observing and recording, can be difficult, depending on the application and environment. Difficulties arise when the results of a test are not easily observable, such as in the case of an internal calculation that is never seen by the end-user, but drives the functions that are subsequently performed by the user.

There is a basic rule in testing that says if you can’t observe something, you can’t test it. This means you need a tool or another way to see the outcome of a test to know whether it worked. Dirty systems tend to hide a lot of functionality from the user, which means that a lot goes on behind the scenes that the user is not aware of and may not be able to access. To observe results in these cases, a tool such as an interactive test and debug tool is needed. Alternatively, the tester may place special code into the application being tested. This code will display the values of data, which the tester needs to see to verify a test result. There are problems with this method.

One problem with added code is that it is intrusive to the application, and its presence can cause unintended negative effects. Another problem with added code is that removing it can inject defects into the application.

Then, there is the task of reporting what is observed during the test. If the test is a simple and observable cause/effect test, then printing a couple of screen images could handle the matter. However, if the test involves a series of steps or a large volume of data, the job becomes very difficult. There is also the consideration of transmitting and storing large volumes of test documentation, either physically or electronically.

One solution would be to minimize the amount of information reported for each test. However, the tester never knows when a defect will manifest, and it may be impossible to recreate the circumstances exactly so the developer can see them. The problem is twofold. Recreations can be faulty due to human error on the part of the tester. Additionally, test recreations can be faulty due to the high complexity of dirty systems. This high level of complexity almost ensures that the tester will not get precisely the same results twice in a row unless the environment is very tightly controlled. Tight control is not characteristic of dirty systems.

A solution that we have used in the test lab is to use an image capture tool, such as Snagit or Camtasia, to capture a series of images. If the test passes, the image file can be overwritten with the next test. However, if a defect is observed, you have it documented. 

Capture/playback tools are a possibility to document the test, but most of these tools are recording keystrokes and user actions to be replayed against software. This means that you will need to recreate the test conditions exactly to replay the captured script to see the same results as before. The expense of capture/playback tools is another issue if all you are using them for is to document the test results.

With any tool for documenting test results, you need to consider the ability to be able to view the electronic version of the results at a later date. Consider how quickly operating system and test tool versions have changed over the past 5 years, and it’s easy to see how test documentation could be inaccessible simply due to technical obsolescence. For some people, this may not be an issue. However, if you are in a regulated industry, you may be required to show the results of tests in years past.

Solution Strategies

  • Use an interactive test/debug tool for observing hidden results in interactive applications. While this gets a tester down to the coding level, this may be the only way to see some results.
  • Use screen capture tools for capturing a series of screen images. I prefer to use a tool like Camtasia for the purposes of test documentation because a) it is cheaper than a capture/playback tool and b) it doesn’t depend on the current state of the software for playback; therefore, it is an accurate record of what happened during the test.

A personal guideline and practice for me is to try to replicate a failure before I report it. If I can replicate the failure, then I can report that the failure is reproducible.

Conclusion

Be sure to allow extra time in the testing process for evaluating and reporting results. Also, make sure you have adequate physical and electronic storage capacities for the reporting files. It’s impossible to know in advance where and when a defect will manifest, so it pays to record the test each time rather than spending an hour or more trying to recreate the defect.

The better your test reporting, the better the developers will be able to isolate and, hopefully, fix the problem!