24 Jan No Need To Test©

Introduction

For approximately twenty years I have been a tester, a quality assurance consultant, and a verification and validation manager. In more recent years I have facilitated workshops and taught at a large Washington DC university. I have always advocated for software testing. Not just testing but formalized and structured testing. For all that time I had never met a software project that I didn’t want to test. Recently, I have begun to rethink this mantra and now I understand that there are software projects for which testing may be, at best, a waste of time and resources.

During these years in the software quality assurance arena I encountered numerous projects where software testing did not meet its objectives and only a few projects

where testing did meet its objectives. Of these, I have worked on only one project where testing achieved its objectives and where the project delivered a stable and functioning application within the budget and schedule constraints. Each time a project did not meet its testing objectives I evaluated what had been done in order to mitigate continuing or future risks. As might be expected, patterns began to emerge.

Patterns

Whenever the manager of a rapid development effort did not value engineering documentation the test team found itself testing what had been built rather than what was supposed to be build or what was needed by the client. Code Word: no engineering documentation.

Whenever management allocated four to six months for development but only one or two weeks for testing the test team was unable to perform comprehensive testing and was never able to perform regression testing. Code Word: insufficient schedule.

Whenever development continued to add new functionality without allocating resources to fix known defects the test team was often faced with an unstable application and cannot perform any testing. Code Word: no ongoing bug fixes.

Whenever development occurs without having documented requirements the test team found itself testing what had been built rather than what should have been built or what the client expected to receive. Code Word: no requirements documentation.

Whenever development took place without making the marketing claims part of the requirements the test team was faced with testing for functionality that had not been included in the application. Code Word: insufficient requirements

Whenever development took place without creating user scenarios the test team was faced with testing for functionality that had not been included in the application. Code Word: insufficient requirements.

Whenever developers were given a Visual Basic form and told to code it without also being told what result (from completing the form) was expected by the user the test team were correct in expecting the application to be unstable and non-functional. Code Word: no engineering documents

This list is based on few of my most memorable projects. It certainly is not meant to represent the universe of problems that may negatively impact on testing. My first inclination was to take steps or cause others to take steps that would mitigate the impact on testing. For example:

Where the schedule was not ample enough to allow sufficient testing I attempted to negotiate for more time. When I could not negotiate for the needed time to do regression testing, new feature testing, and several iterations of bug fixes then I gained management agreement to redefine the test effort. This might mean testing only the critical requirements or features rather than all of them. Or it might mean testing only those components that contained a majority of the custom code.

Or if the problem was that of missing or insufficient requirements I tasked the test team, including myself, to derive requirements from any existing documentation. Sometimes requirements would be derived based on testers’ experience with similar applications. For example, one application that was Internet accessible yet only supposed to be used by a selected group of employees had no documented security (logid/password) requirements. In fact, the first release of that application did contain a logon with password dialogue box. However, it could be by-passed by selecting the key. If there was absolutely no existing documentation and no functional knowledge of the application then I explained to management that testing would not sign off on the approval document with a notation that it was not tested.

Actually, I expended a great deal of energy and time attempting to create change in areas I had no authority to create change. I spent energy and time attempting to change the thinking of managers (mid- and senior- level) who had no vested interest in changing their thoughts. Finally, I understood that some things were not going to and some people were not going to change – at least not in the foreseeable future.

Conclusion

From these experiences I have learned that there are some software applications that need not be tested. Applications that serve no business need, have no documented requirements, have not been designed, or have been built with significant known defects because there was no time to fix the defects should not be tested. Applications that that are pushed forward so

each participant in the process can check it off of their list need not be tested. And the resources that would have been allocated to testing then can be allocated toward customer support.