05 Dec The Elevator Speech about IV&V

For about twenty years, I have been involved in Information Technology (IT) in both the public and private sectors. I have been a business analyst, a software tester, a test manager, and an independent verification and validation (IV&V) manager. I am now a principal for a consulting company providing independent verification and validation services. I have taught software

quality assurance at a Washington DC university. I am involved in establishing the standards for both testing and IV&V having co-authored the IEEE Std 829-2008, System and Software Test Documentation and I was involved in the writing of IEEE Std 1012-2004, Standard for Software Verification and Validation . I am currently part of the working group that is developing IEEE Std. 1012-2010 (estimated date) to enhance best practices regarding system and hardware verification and validation.

Introduction:

I was attending a networking meeting not too long ago.As is the custom I circulated through the crowd, introduced myself, and attempted to hand my business card to everyone I met and to get their card in return.At some point, someone announced that we were all to get into a circle and introduce ourselves and our companies to everyone in the group.This was the time for the three-minute elevator speech with which I present my hopes and aspirations in a short period of time.I took a deep breath.When my time came I presented it just as I had rehearsed it. I was well spoken and succinct.However I was not brilliant because I had neglected to take the audience into account.I neglected to frame my information for a non technical audience.As a result no one understood what I did or what kind of contacts I needed.

Afterward, I had the opportunity to speak with a number of people and to explain exactly what kind of contacts I needed and what I did.Since that day I have been striving to put the words of that one-on-one into another elevator speech.The speech should summarize the nature of the services offered, the benefits to the user, and the type of organization that could benefit from these services.

Services:

We provide a variety of IV&V services.The important thing is that we are independent.That means that we have a budget that is independent from the development budget, a management that does not report to the manager of development, and that the staff performing IV&V tasks is not the same staff as those performing development tasks.As a result we are able to provide non-biased feedback to the client organization.We base our assessments on consensus standards and so the feedback is both substantive and informed.We provide feedback on both the quality of the product and the likelihood that the system being produced will meet the business needs of the organization.

Our IV&V services include:Management Support; Engineering Support; Test Support; Review and Audits Support; and Process/Procedures Support.Thus the client organization can acquire all or some of these services.The details of the specific tasks included in each of these services can be found in another “musings on the deck” — IV&V Services, published on July 4, 2008.

This means that in each of these categories of IV&V services we evaluate documents and other products that provide a foundation for the system under development.

Management Support Services

For example, the management category generally involves program/project plans, schedules, milestones, and proposals. It also may include various management plans such as quality assurance and configuration management.By evaluating these, IV&V identifies inconsistencies, errors, or other issues that may present a risk to the program/project, thus providing management with an early identification of potential problems.

Engineering Support Services

In the engineering category, IV&V evaluates individual technical/development products to insure that the

content, at a minimum, conforms to the applicable IEEE or other consensus standard.IV&V also evaluates the adequacy and completeness of the products based on good engineering practices.One of the earliest assessments performed by IV&V and

the most important is the assessment of the system requirements because these are incorporated in the subsequent design documents and ultimately into the software code or the computer hardware.These evaluations identify the adequacy, completeness, consistency and correctness of the content so that technical risks are identified prior to the creation of software code or manufacture/assembly of computer hardware.

Test Support Services

In the test category, IV&V evaluates developer and/or integrator test documentation including plans, test cases and test procedures.These are assessed against the applicable IEEE or other consensus standards for consistency, completeness, and correctness.IV&V determines that the test documents correctly link to each other and correctly reflect the business needs, security, performance and other requirements based on the level of testing.Additionally, IV&V often witnesses the actual testing performed by the development and/or integration contractor.After testing is completed, IV&V may assess the test report prepared by the development and/or integration contractor.As a result of the test activity, IV&V identifies possible risks, if any, for taking the system under test, into production.

There may be occasions when IV&V is tasked to perform its own testing.Whenever this occurs, IV&V must generate its own test plans, test cases, and test procedures.Once test execution is completed IV&V will write its own test report.

Review and Audit Support Services

Although projects may have many reviews and audits these are typically either management reviews or technical reviews.Management reviews may be performed at different intervals by various level of management and are most often concerned with schedule and budget issues.IV&V often provides input concerning the adequacy of project artifacts or lifecycle processes, in the form of reports or presentations, in support of these reviews.IV&V may provide a special report for that specific review or an IV&V End of Activity Report may be provided in support of the management review.Management has the responsibility for deciding if the project should go forward and if so what is the risk or the cost and if the project should not go forward what is the risk or the cost.

Technical reviews are sometimes called milestone reviews.Technical reviews are held based on the project schedule and/or at the completion of one or more specific products.IV&V provides input, in the form of task reports, in support of technical reviews.Technical reviews are held to determine if a specific product meets its requirements and to determine if the project can move to the next activity based on completion of

approved criteria.One technical review is often called the critical design review.This event determines if the requirements and the design documents are complete and that the project can move to the implementation (code generation) process.The technical review provides project manage to assess the technical risks involved with going forward.

One type of audit may examine a process and the results of that process such as a Configuration Audit.Another type of audit may compare the system as-built to the system as-tested such as a Functional Configuration Audit.IV&V may participate in these audits.

Benefits:

An immediate benefit of IV&V is that upper management is made aware, on an on-going basis, of potential risk to the budget or schedule while project management is appraised on a continuing basis on the quality of the deliverables.Some deliverables, if defective, will have substantial negative

impact on the ability to build the right

system and to do so in a correct manner.Other deliverables, if defective, will have a substantial negative impact on the ability to monitor and control the project.All defective deliverables should be corrected as early in the life cycle as possible in order to mitigate the risk to the project and/or system.

A more long-term benefit of IV&V is the avoidance of costs associated with defective systems or software.These costs might include excessive maintenance and support costs due to an increase in HELP requests and/or the unplanned software releases.Additional costs may include those associated with the creation of a negative corporate image among potential customers based on negative reviews by professional review organizations or bloggers.There may be future costs associated with current customers leaving and going to competitors. Finally, defective systems or software may result in lawsuits for breach of contract or even damage to organization assets.

The cost of IV&V over the development life cycle is said to average 15%; however, once the cost of IV&V is spread over the life of the system or software then the actual cost is significantly lower.A study at the NASA IV&V facility compared the Return on Investment (ROI) of IV&V on several different projects to be 1.9 on the low side and 4.9 on the high side.

Benefitting Organizations:

If you own or represent an organization that produces software for its own use or for others your organization will benefit from allocating funds for IV&V because testing the product to assure its functionality only assures

the functionality of the system or software as-built.It does not assure the functionality that should-be.IV&V will assure the functionality that you need is present.

If you own or represent an organization that acquires systems and or software produced by others you will benefit from allocating funds for IV&V because you are not in the business of developing systems or software and likely have neither the skills nor the time to oversee the development. At the same time you probably have a low level of trust that the development contractor will deliver what you need and what you

paying for without stringent oversight of the development products.IV&V will provide the oversight so that your get the system or software that you need and that you are paying for.

You can acquire as many or as few IV&V services as you need or your budget permits.No matter how much or how little IV&V you acquire you will enhance the quality of your product.