24 Jan Small Company IV&V

Introduction

For approximately twenty years I have been involved in Information Technology (IT). I have been a software tester, a quality assurance consultant, and a verification and validation manager. In more recent years I have facilitated workshops and taught at a large Washington DC university. During this time I have worked for small organizations, both public and private. All these organizations identified similar concerns. They wanted to know how they could institute all the things that would allow them to become a quality organization. Some of them wanted to be ISO certified and others wanted to become compliant with the SEI-CMM model. They wanted to do the right thing but they anticipated that their smallness would be a deterrent

These concerns were voiced by organizations that provided services and by those that developed products. My job was to help them institute a quality program that, at the beginning, encompassed a minimum number of verification and validation techniques. Their program could then be expanded as the organization grew.

The Beginning

One organization was a small start-up with only twenty employees. They were in the business of developing a software product that would allow customers to store fax, voicemail , and email messages in the same “box.” The user could then pick up their messages (in whatever form) from any telephone or from a computer and have the fax sent to any number they wanted. For the first year or so they were satisfied with letting the developers test their own code. But then the unthinkable happened. An evaluation copy of the software product sent to a large company failed to perform correctly. The start-up company readily agreed to fix the problem(s) and re-send a working evaluation copy. Because they were not able to fix the problem in a timely manner they lost a potential customer. Their potential customer was one of the Baby Bells.

This loss created an awareness that the old way was no longer viable. A Quality Manager was hired to create a process to ensure that a non-functioning product would not be sent out. The Quality Manager attempted, without success, to install a large organization process. The organization rejected these attempts. The Quality Manager contacted me to establish a formal testing group.

Later On

Working with the Quality Manager, I determined that a formal testing group would need a basic infrastructure before the results of testing would be trusted by senior management. This infrastructure could not require the participation of any of the hardware or software engineers because they worked for the Engineering Manager. And the proposed infrastructure had to be implemented by the existing staff while they continued to do basic functional testing. The proposed structure contained:

• Maintaining control of the test platform
• Documentation describing the testing
• Change control for the software to be tested
• Reporting and tracking of defects

Maintaining Control

The Quality Manager requested and received a server that was set up by engineering. A representative of the test

group recorded the model and version numbers of all of the cards and other pieces of the hardware. Also recorded were the version and build numbers of each software application, including the operating system. From that day forward, the test group installed each version and build of the product under test.

On occasion, the senior engineer requested time on the test server. This time would be granted if it did not conflict with testing. In addition the senior engineer had to first make a copy of the files to be changed, so that the test system could be easily restore to its original configuration. The senior engineer was told that if the system was not returned to its original configuration, engineering would no longer have access to the test server.

Documentation

By requesting input from all members ( one full-time and two part-timers) of the testing group, I secured agreement from everyone for a test case template based on IEEE Std 829. Each test case contained the

same elements (even if the element was not applicable for a specific test case) and included the step-by-step for executing that test case. The junior tester had begun to document all of his tests and volunteered to put all of his previous work into the new format. I discovered that his writing skills were very good and asked him to take on the responsibility of editing and clarifying the test cases

written by more senior test staff. It took a few months until all of the functional tests for the current level of functionality were documented. We then printed them out and placed them in a loose-leaf binder. Working together, we identified those test cases that formed the core for future regression testing.

While talking with the test group members and the Quality Manager I discovered that each version of the product was treated as a brand new product. After meeting with the engineering group to learn how this “product” worked, a team meeting was held to discuss how to approach testing. Since each version merely added additional functionality, this process was not really efficient. The suggestion that each version was the same product was well received by the Quality Manager and the test group. I took it upon myself to write a Test Plan based on IEEE Std 829. The Test Plan had provide the information necessary to manage the test effort and yet be a living document because new versions were released every 12-15 weeks. This was done by describing the general purpose of the product including the current functionality in the appropriate section of the test plan document. This document specified that all new functionality and appropriate test strategy would be contained in subsequent appendices.

Both the test cases and the test plan documents were completed before the next release. Based on information provided by the Engineering Manager, I generated the test plan appendix for the coming release. When the next release came out, we were able to use the test case documentation for structured regression testing.

Change Control

Prior to this time the Engineering Manager would compile the code using one of the well known change control tools. Because the developers code was not maintained under change control the code was compiled with missing or out-of-date units of code. This resulted in an unstable product being given to the testing group. It often meant days of delay until the test group would receive stable code. The engineers would then install the product in the test environment, when they had time.

The solution to the problem was twofold. First, the test group took over the responsibility of installing each new build. This required learning the complexity of the installation. This task was assigned to a senior level tester who was also tasked with creating a step by step procedure document. Second, the Engineering Manager began to enforce the requirement that developers fill in the field in the change control tool specifying the units of code that they had changed. If the field was not completed the modified code was not accepted by the tool. The next problem faced by the Engineering Manager was code that had been taken for modification but not returned.

These small improvements decreased from five working days to one or two working days the time it took from compilation to installation to testing in the test environment.

Defect Management

Spending money for a tool was not a high priority. We found a freeware tool on the Internet. It took a few hours of programming effort to customize the data entry screen to look the way we wanted it to look. It took another few iterations to have the query and reporting capacity we needed.

With our new tool, testers and engineers could both open a new defect report and add information to an existing defect report. The ability to close a defect report was restricted to the test group. We could search for a report by number, by originator name, by version/build number, level of severity, or by status. We could limit the query by searching for

a combination of these criteria.

With our new ability to manage the defects and to report on the status, the test group could better track the stability of each version. With this ability we could recommend that the product under test be or not be released into production; and document why we made that recommendation.

The Next Stage

The generation of test documentation, improved change control and defect management were accomplished within six months. The next set of improvements included gaining the cooperation of Engineering to place the development environment under change control. This would preclude engineers from turning in new or modified code and then “improving” it the night before the compile and build is to take place. Concurrently a formal Configuration Management (CM) process will be established. The primary purpose of the CM process is to establish a Board comprised of the senior engineers and the senior tests. This Board will be responsible for deciding when a product is ready for shipping to the customer. The Board will also be responsible for deciding what defects will get fixed immediately and which will be delayed. The Board may also identify which new functionality will be implemented in the next Version

and which will be implemented in subsequent versions.

Summary

This small company, with limited resources, was able to create a structured test environment. Although these steps were not enough to brings this small company into conformance with SEI-CMM Level 3, they did fulfill some

of the requirements of level 2. Perhaps more important for them is that they now can deliver stable and functional software to their customers. They also see that they have the potential for

achieving first Level 2 and then Level 3.