This manual defines the standards for software development testing and the entry criteria for systems testing.

This manual defines and documents the testing procedures for developers who must develop or maintain software.

These testing procedures provide a standard framework for developers (including contractors working for Applications Development organizations) to follow and complete before releasing software to Production or Test, Assurance & Documentation (TAD) personnel for further testing.

The terminology used in this manual is defined in Exhibit 2.5.2-13

The controls established in this Internal Revenue Manual (IRM) apply to Agency personnel and contractors responsible for developing or maintaining IRS Software Systems. Agency personnel who contract for development or maintenance of these software systems shall ensure contracts comply with these controls.

The following documents have been referenced in this manual:

Proper preparation of the Software Product improves the quality of testing to be performed.

Testing preparation should include but is not limited to the following activities:

Test results should be included in the project folder with the exception of taxpayer data. Refer to IRM 10.8.8, Information Technology, Live Data (LD) Protection for guidance.

Complete lines 1 a - f of the Testing Checklist, Exhibit 2.5.2-3.

The project folder(s) is a requirement for all projects. The project folder(s) contains all project and program documentation and correspondence and provides pointers to the location of all current project related information. Information that cannot be incorporated in the project folder, such as boxes of printed output, computer files, interface tape(s) and/or file backups must be cross-referenced in the project folder checklist to expedite location and access.

For the purpose of this IRM, the project folder term is used interchangeably to reference both project and program needs.

The project folder(s) is a useful resource document for auditing purposes, project planning, allocation of resources, and process improvement.

The project folder(s) must be maintained throughout the project lifecycle by the development team and at a minimum updated at each milestone. Subsequent reviews may be conducted as needed for test monitoring and oversight, and to verify the folder is updated on a regular basis to reflect tasks accomplished.

The project folder(s) must be stored in Document Management for Information Technology Projects (DocIT) or an approved repository or secure environment as designated by the project manager, Contract Officer Technical Representative (COTR) or supervisor. The project folder must be retained for two years beyond the scheduled program implementation date.

Content of the project folder should be verified by the developer prior to Unit Testing. During Unit Testing, the project folder checklist should be completed to reflect artifacts of Unit Testing.

The project folder(s) will be reviewed by the designated management representative at established intervals. At a minimum, the review must occur prior to production implementation.

For an example of a Project Folder Checklist, See Exhibit 2.5.2-1.

Depending on the project type (major, non-major or small/other), all artifacts listed on the project folder checklist may or may not be applicable. The project folder checklist may be tailored for project needs while ensuring the integrity of the checklist is preserved.

.

Complete line 2 of the Testing Checklist, Exhibit 2.5.2-3.

Create/update the Requirements Traceability Verification Matrix (RTVM) to ensure that all requirements are met and enable the development of test plans. The RTVM is developed in phases over the course of the life cycle.

An RTVM is a mechanism to identify requirements and track satisfaction of the requirements through development and test phases.

Requirements Traceability maps requirements to business processes, system development document(s) and test cases. The RTVM correlates system requirements with test cases to ensure appropriate test coverage.

The RTVM is a required deliverable as part of the Test Plan

A new RTVM should be created for each release/production implementation to document a new baseline. For legacy projects, documenting the new requirements would be the first step in creating a new baseline of the existing (as-is) project.

AD projects that are currently using the Requirements and Demand Management (RADM) template (Excel Requirement Repository Spreadsheet) or RequisitePro should continue to maintain these repositories and are not required to complete the RTVM workbook template.

The RTVM, defined in the RTVM workbook template, is composed of the following worksheets: Requirements, RTVM (Unit Testing), RTVM (ISAT) and RTVM (SAT).

A clear audit trail must be documented in the RTVM when a requirement moves from one test status to another.

An RTVM workbook will provide, at a minimum, the following information:

Create/Update RTVM steps:

An excel spreadsheet containing detailed instructions for completing the RTVM and the four worksheet templates (Requirements, Unit Testing, ISAT, and SAT) is stored in DocIT. See IRM 2.5.2.3(3) for the DocIT number. The RTVM (Unit Testing) Worksheet and instructions can be found in Exhibit 2.5.2-11 and Exhibit 2.5.2-12, respectively.

An RTVM must contain a Configuration Identification Index (CII). The CII is a unique identifier used to correlate documentation to the appropriate CI and will be placed in the footer of the RTVM. For example, CII identifier OS:CIO:AD:PMO:BPI:TMP-RTVM-V1.4-01262010.

The RTVM must be stored in DocIT or an approved repository designated by the project manager.

Complete line 7 on the Testing Checklist, Exhibit 2.5.2-3.

Perform a Desk Check of the source code to eliminate logic and syntax errors.

A Desk Check of the software must be performed after developing the source code and completing preliminary tests to achieve a clean compile.

Desk Check Software steps:

Complete line 3 on the Testing Checklist Exhibit 2.5.2-3.

Conduct a Peer Review on software and documentation to identify any defects and improvements.

Peer Reviews should be conducted if the complexity of change warrants it or when an inspection of test results indicate an unusual problem.

Four major types of Peer Reviews are:

Peer Review process consists of the following six stages:

The project manager will assign a peer review coordinator to perform the following steps:

Complete line 4 on the Testing Checklist, Exhibit 2.5.2-3.

The Test Plan describes the overall approach, procedures, and defines the conditions to be tested based on functional requirements including conditions that exercise every path of new or changed logic.

After the requirements have been approved through the WR process, develop a Test Plan. See Exhibit 2.5.2-5 for an example of a Test Plan.

The following structure is required for a Test Plan:

Creating and approving Test Plan steps:

Execute Test Plan steps:

Complete line 5 on the Testing Checklist, Exhibit 2.5.2-3.

To specify and document the conditions to be tested, create Test Case Specifications.

Test Case Specifications should be developed from the Test Plan and designed to "try to break the code" (test outside the normal or expected function) to find defects.

Each Test Case Specification must:

Test cases must meet the following attributes in order to be reusable in retesting, to provide an adequate test audit trail, and to perform problem resolution:

Create Test Case Specifications for:

Test Case Specifications must include all applicable components:

Verify Test Case Specifications are complete and consistent with software requirements.

See Exhibit 2.5.2-7 for the template of the Test Case Specifications.

Place a copy of the Test Case Specifications in the project folder.

Complete line 6 on the Testing Checklist, Exhibit 2.5.2-3.

Confirm that all systems documentation has been received and that requirements have been reviewed and accepted prior to test data creation.

Develop or acquire sufficient data to test that all requirements have been satisfied.

Test Data should be consistent with the software requirements to be tested.

Test Data should provide the input as required by modules, programs, applications, or systems.

Test Data must be protected as "Official Use Only" information and is intended solely for testing purposes.

Determine which of the following test data types are appropriate for the test:

Develop/Acquire Test Data steps:

Complete line 8 on the Testing Checklist, Exhibit 2.5.2-3.

Coordination of the testing environment will vary depending on the degree of complexity of the software changes, type(s) of testing required, practices of the customer, methodologies used to develop the software, and impact of the software being tested.

Identify the type of equipment, operating system(s) and all communications media needed.

Coordinating test environment steps:

Complete line 9 on the Testing Checklist, Exhibit 2.5.2-3.

Unit Testing is a procedure used to validate that individual units of source code are working properly.

Unit Testing will test the smallest unit of the software and verify that the unit of code meets the detailed design specifications and is working properly.

The objective of Unit Testing is to test not only the functionality of the code, but also to determine whether the code is structurally sound and robust, and able to respond appropriately in all conditions.

Unit Testing should be conducted on individual modules and programs by the developer prior to integration. Unit tests are usually conducted in the development environment prior to system integration.

Unit tests should satisfy the design of the application and be traceable to the design documents and the RTVM.

The process of conducting the Unit test consists of the following steps:

When all tests have been executed and errors corrected, re-execute the full unit test to confirm that no new errors were created during the correction.

The unit test is completed when:

Complete line 10 on the Testing Checklist, Exhibit 2.5.2-3.

Compatibility testing is conducted to determine whether the program correctly exchanges files with other software.

Parties responsible for the programs impacted by the change must participate in Compatibility Testing.

The Section Chief or designee of the Originating Section and any Impacted Section must sign off on the Files Communication Status Report (FCSR).

See Exhibit 2.5.2-9 for the template of the FCSR.

Compatibility Testing steps:

Complete line 11 on the Testing Checklist, Exhibit 2.5.2-3.

Integration Testing verifies that the system integrates properly and functions as required.

The purpose of Integration Testing is to accept, integrate and test system and software components until the entire system is operational and all agreed upon customer requirements have been validated.

Integration Testing occurs after unit testing has been completed for all units contained in the subsystems being tested.

Integration Testing takes modules that have been unit tested, groups them into larger aggregates, applies tests defined in an Integration Test Plan to those aggregates, and delivers as the integrated system ready for system testing.

The procedure for Integration Testing is as follows:

Test cases for Integration Testing are constructed to test that all components within assemblages interact correctly.

When creating the Integration Test Plan review the detailed design to determine which units (and therefore, which subsystems) are changing.

Complete line 12 on the Testing Checklist, Exhibit 2.5.2-3.

Determination on whether to conduct a Developer Integration Test (DIT) will be made by management.

After Unit Testing, a DIT combines the parts of an application to determine if they function together correctly. In its simplest form, two units that have already been tested are combined into a component and the interface between them is tested.

A component is an integrated aggregate of units or components. Multiple units are combined into components, which are in turn aggregated into larger components and then into the application. The idea is to test combinations of components to find interface defects and eventually expand the process to test all the integrated parts of the application.

The purpose of DIT is to accept, integrate and test application components until the entire application is operational and all agreed upon customer requirements have been validated. DIT verifies that the application system integrates properly and functions as required.

DIT can start when units appropriate for integration have been tested. DIT can be done in parallel with some unit testing.

The procedure for DIT is as follows:

A Test Readiness Review (TRR) will be conducted after the DIT.

Complete line 13 on the Testing Checklist, Exhibit 2.5.2-3.

Determination on whether to conduct a DSIT will be made by management.

Development System Integration Testing (DSIT) is a logical extension of DIT testing. After the individual projects have been independently integrated into applications, tested in a DIT, and meet their respective requirements, they will be integrated into a complete system and tested in DSIT.

DSIT occurs after both unit testing and DIT have been completed

DSIT verifies that all component application systems integrate properly and function together as required.

The procedure for DSIT is as follows:

A Test Readiness Review (TRR) will be conducted after the DSIT has been performed.

Complete line 14 on the Testing Checklist, Exhibit 2.5.2-3.

Final Integration Test (FIT) is the integrated test of multiple systems that support the high level business requirements of the IRS.

A determination to include a particular system in FIT is negotiated between TAD and the customers after evaluating interfaces, dependencies and the business role of the System.

No one outside of TAD can perform FIT Testing. If FIT is an appropriate test for your system, complete and submit a Final Integration Test Request Form for approval by using the following URL: http://tad.web.irs.gov/fit/AB_About/FIT_Form-New.asp

FIT is conducted:

Complete line 15 on the Testing Checklist, Exhibit 2.5.2-3.

External Trading Partner Testing is special testing that is conducted on programs that exchange files with an entity outside the IRS. All security guidelines must be adhered to. See IRM 2.5.2.2.25 Coordinate Security Testing for additional information and references.

External Trading Partner testing is required if the program exchanges files outside the IRS.

The External Trading Partner must create and make available external trading partner's data.

The development organization will coordinate with all External Trading Partners to ensure that a thorough test is conducted between IRS and external processes.

The Section Chief or designee of the Originating Section and any Impacted Section must sign off on the Files Communication Status Report (FCSR).

See Exhibit 2.5.2-9 for a template of the FCSR.

External Trading Partners Testing steps:

Complete line 16 on the Testing Checklist, Exhibit 2.5.2-3.

Review test results to verify accuracy.

The complexity of the requirements will determine the level of testing.

Developers should encourage customers to assist in reviewing the test results.

Review Test Results steps:

Complete line 17 on the Testing Checklist, Exhibit 2.5.2-3.

Coordinate Systems Acceptability Testing (SAT) with TAD.

TAD will independently assess the quality of the application software by testing with controlled data to determine conformance to customer requirements and to aid the customer and developer in determining the system's production readiness.

SAT is usually performed after unit and (initial) Compatibility/Integration Testing has been completed by the development organization.

Complete line 18 on the Testing Checklist, Exhibit 2.5.2-3.

Independent Systems Acceptability Test (ISAT) is required if SAT is not performed by the TAD organization.

ISAT will assess the quality of the application software by testing with controlled data to determine conformance of the system to customer requirements and to aid the customer and developer in determining the systems' production readiness.

ISAT is used primarily by Current Processing Environment (CPE) projects.

ISAT activities may include but are not limited to the following:

Complete line 19 on the Testing Checklist, Exhibit 2.5.2-3.

Regression testing is performed to determine whether changes to the application have adversely affected previously tested functionality. Regression Testing demonstrates system integrity after changes are made to the configuration or environment of a system.

Regression Testing repeats key tests performed in the past and compares results of the new tests to the recorded actual results of the previously executed tests.

Perform regression testing when there are changes to:

Regression testing can be performed for the complete product or it can be selective.

Data must be consistent from test to test to make the results comparable.

Test cases for regression testing should be selected based on:

Only portions of the systems affected by modifications/enhancements and/or problem corrections are tested. However, in many cases, changes to one portion of the system may necessitate retesting, verifying, and updating other applicable documentation.

Regression Test steps:

Complete line 20 on the Testing Checklist, Exhibit 2.5.2-3.

The goal of Usability Testing is to analyze the user experience through direct observation and evaluation.

At its simplest level, usability testing will determine whether a system works from the users perspective or whether it requires rework.

Usability Testing usually requires specific lab configurations and testing equipment.

Usability Testing should ideally involve direct user feedback and indirect feedback (observed behavior).

The activities listed below are designed to obtain user feedback through usability testing, thereby producing elements for the Usability Review, which summarizes the results of testing:

It is the development project's responsibility to ensure that all usability activities are completed.

Complete line 21 on the Testing Checklist, Exhibit 2.5.2-3.

The Rehabilitation Act of 1973, as amended, requires Electronic and Information Technology (EIT) developed, maintained, used, or procured by the Federal government be accessible to persons with disabilities.

In 1998, Congress amended the Rehabilitation Act to add Section 508, which eliminates barriers in information technology. This made available new opportunities for people with disabilities, and encouraged the development of technologies to achieve these goals. The enforceable provisions of Section 508 apply to a broad range of EIT, from computer hardware and software to office equipment, such as fax and copy machines. Under Section 508 (29 U.S.C. 794d), agencies must give disabled employees and members of the public access to information that is comparable to the access available to others.

Accessibility testing determines whether EIT satisfies federal law for accessibility to people with disabilities. Typical disabilities can be classified into the following four groups, each of them with different access difficulties and issues:

Complete line 22 on the Testing Checklist, Exhibit 2.5.2-3.

Performance testing determines whether the system undergoing testing can effectively process transactions under expected normal and peak workload conditions, within acceptable response time thresholds.

During systems evaluation planning or systems test planning activities, the Project/Program manager, TAD, system owner and other stakeholders make the decision whether to execute a performance test.

Performance testing is designed to uncover any bottlenecks and capacity constraints that may not have occurred during normal functional testing.

Effective performance testing can identify the nature or location of a software-related performance problem.

Performance tests can be broadly classified into:

Each type of performance test plays a critical role in fine-tuning the system, discovering degradation points, and eliminating system bottlenecks.

A Load test identifies peak load conditions at which the program will fail to handle required processing loads within the required time. The number of users is gradually increased until the application fails.

A Stress test is conducted to evaluate a system or component at or beyond the limits of its specified requirements to determine when it will fail and how it behaves at failure. Often Stress testing is performed using the same process as Performance testing but employing a very high level of simulated load. Stress testing is load testing over an extended period of time to validate an application's stability and reliability.

Volume tests will seek to verify the physical and logical limits to a system's capacity and ascertain whether such limits are acceptable to meet the projected capacity of the organization’s business processing. Volume tests are also executed to ensure that the anticipated numbers of transactions are able to be processed and that they satisfy the stated performance requirements.

Capacity tests are performed against different aspects of a system. It is best to focus testing effort on one aspect of the system at a time. Capacity testing is related to Stress testing. It determines the server’s ultimate failure point.

Complete line 23 on the Testing Checklist, Exhibit 2.5.2-3.

The Application Network Review (ANR) process is performed to:

The ANR process requires the development organization and the Enterprise Networks organization to define the application operational performance requirements and to ensure the application and network infrastructure are able to achieve the performance objectives in the production environment.

Application performance issues used to be considered a matter of hardware resources only. Performance problems are now more likely to originate with system configurations, network and server architecture, application code, application messages between servers and the user, and external systems.

Adding network bandwidth or server hardware to resolve performance issues in production is costly and often may not correct the root cause of the poor performance.

Application performance testing will be required for all mission critical systems. The following systems will also be required to undergo ANR testing:

The above criteria for ANR testing are taken from Appendix A, ANR Screening Guidelines from the ANR Process Description.

The requirements and guidance for performing ANR may be found in the ANR Process Description. The document can be found on the Process Asset Library (PAL) by using the following URL: http://paig.ds.irsnet.gov/pal/assets.htm

Complete line 24 on the Testing Checklist, Exhibit 2.5.2-3.

Security Test and Evaluation (ST&E) is conducted in the IRS production environment and consists of activities designed to ensure that the system’s security safeguards are in place and functioning as intended.

ST&E is conducted by MITS Cybersecurity. Cybersecurity usually assigns security experts to support the ST&E and they maintain the standards for conducting the test.

Program/Project Manager must coordinate with MITS Cybersecurity and Enterprise Services early in a project’s life cycle to ensure security requirements are addressed.

For additional information regarding Cybersecurity, visit the MITS Cybersecurity web site by using the following URL: http://mits.web.irs.gov/Cybersecurity

The Privacy Impact Assessment (PIA) is a process the system owner and information developer use to address privacy issues in a program or Internet website under development.

The PIA process provides a means to assure compliance with applicable laws and regulations governing taxpayer and employee privacy.

The approved PIA is required as part of the overall Certification and Accreditation process necessary for all IRS programs collecting personally identifiable information.

Complete line 25 on the Testing Checklist, Exhibit 2.5.2-3.

Web testing is a computer-based test delivered via the internet and written in the "language" of the internet, HTML and possibly enhanced by scripts .

The Communications and Liaison Department (C&L) is responsible for managing the IRS intranet. The content and usability standards are implemented on all IRS intranet sites and updated periodically. These standards are intended to help ensure that intranet sites present content that is properly managed and easy to use and understand. Refer to the C&L Division web site, http://cl.no.irs.gov/intranet

Testing considerations for web applications may include, but are not limited to the following:

Testing tools can improve testing reusability, predictability and testing time. The IRS organization will evaluate tools, gain an understanding of the expected benefits offered by different categories of tools, ensure that selected tools are appropriate for their environment and applications, and train testers in the use of the tools.

Web sites must be tested after development and periodically while in maintenance to ensure compliance to accessibility standards. Accessibility standards are established to ensure that potential users with disabilities have the comparable access to web sites as users without disabilities.

IRS approved testing tools must be used to verify accessibility on MITS/AD Web sites. The following are information accessibility standards that MITS/AD enforces:

JAWS (Job Access with Speech) is an assistive technology product for people with visual impairments. It provides users with information about software and websites via speech. JAWS relies on software and web developers to make sure their products conform to accessibility standards. JAWS is used for testing the functional performance of a product and identification of accessibility issues that may not be easily caught via manual (keyboard only) or software testing (AccVerify).

Web Services standards for the development of IRS websites include content authoring languages, programming/scripting languages, and back-end applications, along with applications that provide developing interfaces.

Development Methodology standards refer to the tools, applications, and languages acceptable for use in the production and maintenance of web sites in the IRS Intranet. These standards are primarily determined by the Portal Program Management Office. The Portal Program Management Office has the responsibility for providing detailed guidance on uniform database naming conventions, script naming, and software components within the IRS Intranet environment.

Complete line 26 on the Testing Checklist, Exhibit 2.5.2-3.

Review the test package to determine completeness and adherence to systems development standards.

The Test Package must be reviewed to determine whether the overall test strategy and test package elements validate the software requirements, and whether the test case specifications provides sufficient coverage for any new software functionality.

Prepare and Review Test Package steps:

Complete line 27 on the Testing Checklist, Exhibit 2.5.2-3.

While most projects transmit software into production under configuration management control of the Source Code and Documentation Control (SCDC) Branch within TAD, there are some that do not. Historically, each project not under SCDC control has used its own method for transmittal. This section describes the procedure that development projects using the services of SCDC must follow.

Once testing is completed and the test results are satisfactory, complete project folder and transmit software to SCDC in accordance with IRM 2.6.1, Test, Assurance & Documentation (TAD) Standards and Procedures.

SCDC Branch serves as an independent version control entity for all of the source code running in SQuA, ENDEVOR and ClearCase. SCDC serves as a control center for the movement of transmittals from the development environment to the SAT, FIT, Martinsburg Application Development Systems (MADS) production files, and the Submission Processing Center (SPC) for production processing.

Identify and document the software version control procedures used to transmit the software.

ECC-Martinsburg (ECC-MTB) or ECC-Memphis (ECC-MEM).

Transmittal systems are set up with their own checklists to be included with the transmittal.

The ENDEVOR ACM tool provides a robust and controlled environment for test, development, and storage of software elements and documentation. It provides version control for the IBM Program Source Code, JCL, memo, and etc. This tool allows developers to simultaneously work on multiple versions of the same source code element. Control is exercised by designating code in production libraries as the official version and level.

ENDEVOR Package Processing:

ClearCase is a configuration management tool designed to help software development teams track the files and directories used to create software. ClearCase enables you to manage the development and build process and to enforce your site-specific development policies.

SCDC Tiers 2/3 Version Control Process using ClearCase describes how SCDC performs version control and promotes software changes through the Version Description Document (VDD) build process. It focuses on the primary activities performed to process VDDs submitted by development projects.

Software Quality Assurance (SQuA) Automated Configuration Management (ACM) tool ensures that controlled Production libraries contain the official version, or baseline, of all existing IRS UNISYS application source code. SQuA facilitates easy access to the baseline version of the source code as a starting point for the development activity. It also significantly reduces the effort required to transmit software to the production sites after development and testing and ensures that modified source elements do not move into production libraries until the appropriate implementation date.

Transmit Software steps:

Complete line 28 a - b on the Testing Checklist, Exhibit 2.5.2-3.