System testing is conducted to ensure system components are free of logic and design errors while satisfying the customer requirements. It is conducted for new code and existing applicable code.

The objective of system testing is to validate not only functions of the system, but also to ensure that the system is structurally sound and robust, and capable of responding appropriately in all conditions.

System testing is conducted to validate the core and specific functions within a system to satisfy requirements and confirm that changes work correctly when receiving data from and sending data to external systems.

System Testing results are traceable to customer requirements and functional design documents.

Conduct a system test for new requirements and existing requirements. Effective system testing should be repeatable.

The process of conducting the system test involves the following steps:

Re-execute test cases in the regression test suite to ensure no new errors are created by corrections. (For more information on regression testing See IRM 2.6.1.3.1.4.).

The system test is completed when:

Compatibility testing is conducted to ensure that unit and string tested programs accurately process data received from external programs/systems and accurately pass this data forward to subsequent programs/systems. The emphasis of this testing should be on data format, physical media, blocking factors (block/record length), padding and controls.

Compatibility testing is conducted if the system exchanges files with other Software systems.

Parties responsible for the systems impacted by a requirement participate in Compatibility Testing.

Upon a coordinated team review of requirements and design, a designee for the system sending changed data enters an agreement for data exchange with the receiving system using a SAT Interface Database ( See IRM 2.6.1.7.).

The process of conducting the Compatibility test involves the following steps:

External trading partners testing is special testing conducted on programs that exchange files with an entity outside the IRS.

The External Trading Partner must create and make available external trading partner's data.

The Team Lead or designee will coordinate with all External Trading Partners to ensure a thorough test is conducted between IRS and external processes.

The Section Chief or designee of the Originating Section and any Impacted Section must sign off on the Files Communication Status Report (FCSR). (For an example of the FCSR, See Exhibit 2.6.1-18. ).

Regression testing is conducted to ensure changes to the system applications have not adversely affected previously tested functionality.

Regression testing repeats key tests performed in the past and compares results of the new tests to the recorded actual results of the previously executed tests.

Regression testing demonstrates software application integrity after changes are made to the configuration or environment of a system.

Perform regression testing when there are changes to:

Regression testing can be performed for the complete product or it can be selective.

Data must be consistent from test to test for comparable results.

Test cases for regression testing are selected based on:

Only portions of the software applications affected by modifications/enhancements and/or problem corrections are tested.

The process of conducting the regression test includes the following steps;

The Rehabilitation Act of 1973, as amended, requires Electronic and Information Technology (EIT) developed, maintained, used, or procured by the Federal government be accessible to persons with disabilities.

In 1998, Congress amended the Rehabilitation Act to add Section 508, which eliminates barriers in information technology. This made available new opportunities for people with disabilities, and encouraged the development of technologies to achieve these goals. The enforceable provisions of Section 508 apply to a broad range of EIT, from computer hardware and software to office equipment, such as fax and copy machines. Under Section 508 (29 U.S.C. 794d), agencies must give disabled employees and members of the public access to information that is comparable to the access available to others.

Accessibility testing is the mechanism for ensuring EIT is accessible to people with disabilities. Typical disabilities can be classified into the following four groups, each of them with different access difficulties and issues:

The goal of User Acceptance Testing is to analyze the user experience through direct observation and evaluation. A usability test ensures the collection of systematic, recorded, quantifiable data and observation of user behaviors.

At its simplest level, user acceptance or usability testing will determine whether a system works from the user’s perspective or whether it requires rework.

User Acceptance Testing requires specific lab configurations and testing equipment.

User Acceptance Testing should ideally involve direct user feedback and indirect feedback (observed behavior).

The user acceptance testing methodology is based on conducting usability activity that coincides with the enterprise lifecycle. User Acceptance Testing must occur early in the life cycle to provide adequate time to resolve identified issues to ensure the cost of changes can be minimized.

The activities listed below are designed to obtain user feedback through usability testing, thereby producing elements for the Usability Review, which summarizes the results of testing:

It is the team lead’s responsibility to ensure that all usability activities are completed, as necessary.

Web testing is a computer-based test delivered via the internet and written in the "language" of the internet, HTML and possibly enhanced by scripts .

The Communications and Liaison Department (C&L) is responsible for managing the IRS intranet. The content and usability standards are implemented on all IRS intranet sites and updated periodically. These standards are intended to help ensure that intranet sites present content that is properly managed and easy to use and understand. Refer to the C&L Division web site, http://cl.no.irs.gov/intranet

Testing considerations for web applications may include, but are not limited to the following:

Testing tools can improve testing reusability, predictability and testing time. The IRS organization will evaluate tools, gain an understanding of the expected benefits offered by different categories of tools, ensure that selected tools are appropriate for their environment and applications, and train testers in the use of the tools.

Web sites must be tested after development and periodically while in maintenance to ensure compliance to accessibility standards. Accessibility standards are established to ensure that potential users with disabilities have the comparable access to web sites as users without disabilities.

IRS approved testing tools must be used to verify accessibility on MITS/AD Web sites. The following are information accessibility standards that MITS/AD enforces:

JAWS (Job Access with Speech) is an assistive technology product for people with visual impairments. It provides users with information about software and websites via speech. JAWS relies on software and web developers to make sure their products conform to accessibility standards. JAWS is used for testing the functional performance of a product and identification of accessibility issues that may not be easily caught via manual (keyboard only) or software testing (AccVerify).

Web Services standards for the development of IRS websites include content authoring languages, programming/scripting languages, and back-end applications, along with applications that provide developing interfaces.

Development Methodology standards refer to the tools, applications, and languages acceptable for use in the production and maintenance of web sites in the IRS Intranet. These standards are primarily determined by the Portal Program Management Office. The Portal Program Management Office has the responsibility for providing detailed guidance on uniform database naming conventions, script naming, and software components within the IRS Intranet environment.

Requirements analysis is the cornerstone activity of a SAT. A requirement at its lowest decomposed level, provides specific information for a single function that is to be performed by the software application. A finite process exists by which a person or machine can verify that the system built meets the requirement (that is, the requirement is testable). Taken collectively, requirements completely define all system capabilities.

Functional requirements describe capabilities that meet user needs; non-functional requirements address other capabilities, such as performance, security, and availability capabilities that must be included in the new system. A RTVM is developed to ensure that all requirements can be traced to specific capabilities identified in the applicable test cases for completeness and accuracy.

High-level requirements are usually provided via a UWR using the following procedure:

.

The RTVM ( See IRM 2.6.1.4.2.5.) is created to track requirements and test conditions that must be validated by the end of the test.

For location of the PIA, WRAF and RTVM templates See IRM 2.6.1.5., Testing Templates.

Documentation Review is the process of confirming the translation of requirements into documentation associated with the system development effort. The review serves to ensure test case requirements are mapped to functional documentation processes. The review also confirms operations for the system prior to the test execution phase.

The following describes the documentation review activities to be performed by the test analyst:

Work product reviews are performed to ensure accuracy, completeness, and consistency in the test life cycle work products such as the WRAF, RTVM, SAT Plan, WBS, and EOTSR.

Work product reviews are to be conducted as needed throughout the testing life cycle.

Types of Work Product Reviews:

The test project manager will assign an analyst to coordinate the following activities:

To document comments and/or questions generated from a work product review, ( See Exhibit 2.6.1-9., Work Product Review Form)

For location of the Work Product Review Form template, See IRM 2.6.1.5., Testing Templates.

To ensure a testing environment is available and ready, coordination of the testing environment is made with the following organizations POCs:

Support is also provided by project POCs and test participants.

Determine support resources assigned roles to ensure environment is fully established and viable for testing. All participants are invited to any coordination meetings to ensure accuracy of environment information and environment build out scheduling.

The test environment needs to emulate the production environment to the greatest degree possible in which the software is implemented. If the required test environment is in existence, verify that the environment is correct. Coordination of the testing environment will vary depending on the degree of complexity of the software changes, type(s) of testing required, practices of the customer, methodologies used to develop the software, and impact of the software being tested.

Identify the type of equipment, operating system (OS), software (SW), configuration tools, portal and all communications media used in the test environment.

The Test Environment Checklist will help to determine if the test environment is already established or needs to be established. ( See Exhibit 2.6.1-10., Test Environment Checklist.)

For location of the Test Environment Checklist template See IRM 2.6.1.5., Testing Templates.

Test environment coordination steps:

Environment Descriptions:

The test analyst should coordinate interface database agreements with other test teams and projects. ( See IRM 2.6.1.7. TAD Interface Database for more information to initiate an Interface Database Agreement).

A Test Readiness Review (TRR) is a milestone signifying the transition from the programmer's test to the SAT and ensures that all stakeholders agree that requirements, design, software, environment, and test products are at a level of readiness to commence SAT.

Two weeks prior to the scheduled TRR, the initiator sends a TRR Agenda and Checklist to all participants.

A TRR is conducted five workdays before scheduled program delivery and serves to validate entries in or to complete the TRR Checklist. ( See Exhibit 2.6.1-11.)

A TRR is a formal review that is conducted in partnership with the Applications Development, Business Customer, TAD and other stakeholders key to the test effort, which may include operations, networks and infrastructure. Collectively the group determines the readiness of the system to proceed with testing. The decision to proceed with testing is made based on a successful TRR outcome. If the TRR is not successful, follow-up meetings are conducted to address all outstanding items and new revised dates must be updated in the SAT Plan. ( See IRM 2.6.1.4.2.6. Systems Acceptability Test (SAT) Plan for more information)

Participants in the TRR includes:

The following documentation is delivered by the participants at least five workdays prior to the TRR, which includes but is not limited to:

TRR participants are to review all documentation prior to the TRR meeting.

For location of the TRR template See IRM 2.6.1.5., Testing Templates.

A Problem Ticket describes a problem or discrepancy that is detected during documentation review, test execution, or output review. These problem tickets will be reported using the Information Technology Asset Management System (ITAMS). The problem may be one of the following:

When a problem is detected, the analyst will:

Post-Implementation Review (PIR) is a formal end-of-test meeting held with developers, customers, and TAD representatives to review the test and identify process improvement areas. A PIR meeting will be held within 60 days after the completion of each test to allow sufficient time to assess the status of Production start-up.

Items to be discussed with customers, developers and TAD Management include, but are not limited to:

An open forum will be provided to discuss any other concerns or issues.

Two weeks prior to the scheduled PIR meeting, the initiator will send an agenda and discussion sheet to all participants. ( See Exhibit 2.6.1-17.)

The PIR Discussion Sheet will be used to document any findings and will be placed in the project folder.

For location of the PIR Agenda and Discussion Sheet template See IRM 2.6.1.5., Testing Templates.

The Work Breakdown Structure is a tool used to aid in planning and monitoring each TAD test through its life cycle. Use the current standard software application for project planning to prepare the Work Breakdown Structure. The use of the WBS is dependent upon the guidelines of each TAD Branch.

Each SAT project has phases and tasks that occur during the test. The Work Breakdown Structure is used to identify, schedule and track activities performed during each TAD test. ( See Exhibit 2.6.1-2.), which provides the phases and significant tasks and contains the dates noted in (c) below.

For location of the WBS template See IRM 2.6.1.5., Testing Templates.

The creation of the Work Request Analysis Form (WRAF) begins in the Initiation Phase, as applicable per the project, when any of the following requirements document(s) listed below, but not limited to, are received:

The WRAF is prepared by the SAT analyst assigned to a particular UWR.

The purpose of this form is to ensure the analyst is familiar with the requirements document(s), understands the requirements and what changes or additions/deletions of functionality are being requested.

The WRAF is used to clarify and articulate the analysts complete understanding of the requirements documented in the UWR/CR/CRL.

The analyst reviews and analyzes the assigned requirements document(s) and any attachments. Based on their analysis answers the 11 questions in the WRAF (See Exhibit 2.6.1-3.).

Completion of the WRAF occurs before testing begins. The analyst may need to consult with their developers and other test areas that may interface with the project and UWR to complete the form.

The completed WRAF is then reviewed by the Team Lead and technical or senior analyst, for accuracy and to verify the analyst is clear on what they are being asked to test. A Work Product Review of the WRAF is then conducted to address any outstanding questions or issues.

Once the analyst has competed the WRAF, then the analyst is ready to create the Requirements Traceability Verification Matrix (RTVM).

For location of the WRAF and RTVM templates See IRM 2.6.1.5., Testing Templates.

The Project Impact Assessment (PIA) Form is used to document the impact of a UWR on each project, documented by staff day and internal order number or program code.

When a UWR is sent through the branches in the TAD domain, each project within the branch will input the estimated staff days needed to test the UWR if impacted, and enter them in the appropriate fiscal year and internal order number or program code. ( See Exhibit 2.6.1-4. )

The PIA's are then sent back to the primary supplier or input directly into the UWR management system.

The Project Impact Assessment (PIA) is primarily used to provide the estimated number of staff days that are required to effectively test the systems impacted by the UWR.

The PIA is required when final analysis of the UWR has taken place. Completion needs to occur upon final acceptance of the UWR.

For location of the PIA template See IRM 2.6.1.5., Testing Templates.

The project folder baseline is the completed folder, which has been reviewed and approved by the Program/Project Manager, Section Chief, or designee, and contains all the required items pertinent to the specific test. The project folder provides a history of the test. It is a useful source document for auditing purposes, project planning, and allocation of resources, and proves improvement. The project folder contains copies of all critical test documentation and products.

The project folder contains all project and program documentation, correspondence and provides pointers to the location of all current project related information.

The project folder may vary in volume depending on the size and/or complexity of the project. Information that cannot be incorporated into the project’s on-line DocIT project folder, such as boxes of printed output computer files, interface tape(s) and/or file backups must be cross-referenced in the project folder checklist to expedite location and access.

The project folder is a requirement for every test and must be stored in the DocIT. It must be retained for 240 days beyond the scheduled program implementation date.

The project folder is reviewed by the Program/Project Manager, Section Chief, or Team Leader prior to the receipt of programs for testing. Subsequent reviews are conducted as needed for test monitoring and oversight, and to verify the folder is updated on a regular basis to reflect tasks accomplished.

Requirements Traceability maps requirements to business processes, system development documents test cases and test results.

The first five columns of the RTVM (ISAT) worksheet or the RTVM (SAT) worksheet are populated by the applications development organizations. This information must be provided by the development organizations 30 days prior to the scheduled delivery of applications to the organization performing systems acceptability testing. Instructions for what is input to these columns are listed in the Instructions spreadsheet of the RTVM Workbook.

When systems acceptability testing is performed by the Test, Assurance and Documentation organization, the remaining columns of the RTVM (SAT) worksheet will be completed to document system acceptability testing results. Otherwise, the organization performing the independent systems acceptability test must complete the remaining columns of the RTVM (ISAT) worksheet to document system acceptability testing results.

For an example of the RTVM (SAT) worksheet, See Exhibit 2.6.1-6

For location of the RTVM Workbook template See IRM 2.6.1.5., Testing Templates.

Test cases are created to specify and document the conditions to be tested and to validate that system functions meet customer requirements as translated into documented functional design. Test cases also test outside the normal or expected functions in order to find defects.

Each test case must:

To provide an adequate test audit trail, and to perform problem resolution, test cases must meet the following attributes to be usable in retesting:

Create test cases for:

Verify test cases are complete and consistent with software requirements.

Disposition of all test cases is required by the end of the test. Disposition by any of the following:

Confirm all systems documentation has been received and requirements have been reviewed and accepted prior to test data creation.

Test data is developed or acquired to verify that all conditions are met.

Test data is consistent with the software requirements to be tested.

Test data emulates inputs as required by modules, programs, applications, or systems.

Test data must be protected as "Official Use Only" information and is intended solely for testing purposes.

Determine which of the following test data types are appropriate for the test:

Develop/Acquire Test Data steps:

To ensure test results are accurate, test results must be reviewed. These results must be consistently recorded, documented and verified during test execution. The complexity of the requirements will determine the level of testing. Testers should encourage customers to assist in reviewing the test results. Perform the following steps to review test results:

A SAT Plan describes the project/system to be tested, the scope of the test, and the requisite skills, roles and responsibilities required to successfully complete the test. Additionally a SAT plan defines the various types of tests that are performed during a SAT. See IRM 2.6.1.3.1. Test Types, defines the type of tests that are addressed in a SAT Plan.

A SAT Plan:

SAT Plan Revisions, if necessary, should only be made if the test execution phase has not started.

A SAT Plan must be revised and a new version distributed for one or more of the following reasons:

The baseline of a SAT Plan is the initial version, Version 1.0 each subsequent revision establishes a new baseline.

The Work Breakdown Structure is a critical component of a SAT Plan. The WBS defines the time-line that drives the completion of activities and tasks to performed throughout the test life cycle.

For an example of a SAT Plan and Cover Memorandum, ( See Exhibit 2.6.1-7.).

For location of the SAT Plan template See IRM 2.6.1.5., Testing Templates.

Electronic copies of SAT Plans are stored on the TAD web-site at http://tad.web.irs.gov/.

The End of Test Status Report (EOTSR) provides information regarding test results to allow customers and developers the ability to assess the quality of the product. The EOTSR is a TAD/ELC deliverable, that is part of the project folder. The EOTSR addresses:

The EOTSR is submitted to the TAD Program/Project Manager, Section Chief and Branch Chief for approval and concurrence. The EOTSR is submitted with sufficient time to allow distribution to all stakeholders within five workdays after the scheduled test completion date.

A revised EOTSR will be prepared if a project has reached its test completion date and the EOTSR has been written, but a significant number of open Problem Tickets remain to be retested. A revised report will be issued upon completion of the test.

The EOTSR should be revised and a new version distributed if significant information becomes available that was missing at the time the original EOTSR was written.

Revisions to the EOTSR should be submitted to the TAD Section Chief, Program/Project Manager and Branch Chief for review and approval as soon as completed.

The revised EOTSR Memorandum, Revision History Page, and EOTSR should be distributed to all stakeholders.

For an example of an EOTSR, ( See Exhibit 2.6.1-7.).

For location of the EOTSR template See IRM 2.6.1.5., Testing Templates.

Electronic copies of the EOTSR are stored on the TAD web-site at http://tad.web.irs.gov/.

The Weekly Test Status Report is driven by the RTVMs and used to reflect the status of test data preparation and test execution throughout the test. It includes the significant dates in the test life cycle, counts of problem reports and current status narrative. (For more information on the TAD Status Reporting System (TSRS), See IRM 2.6.1.8.3. )

The Release Readiness portfolio in the ProSight tool monitors projects throughout the development and testing life cycle. The Release Readiness portfolio identifies projects the TAD and Development Organizations have committed to test.

The TAD and Development Organizations negotiate critical milestone delivery dates, including the following:

The Development Organization is responsible for initiating a project on the Release Readiness Schedule and Status Report.

The Release Readiness Schedule and Status Report is updated as necessary in ProSight by TAD, Development, and the Program Management and Release Readiness Organization (PM & RRO).

The Release Readiness portfolio also produces Weekly Exception Reports which identifies projects which could be in jeopardy due to missing information, or milestone dates that have not been met. It is critical for TAD to promptly enter actual dates into ProSight for each milestone as it occurs.

The Release Readiness portfolio tracks automated (based on milestone) and manual status "color-coded" values.

Prior to manually adjusting the status of any project in ProSight, the test team must clear the manual status update through their management, and pre-coordinate all manual status over-rides with the appropriate project office, or programming team. When entering a manual status update a narrative explanation must also be entered into ProSight.

Major projects create a UWR to enlist TAD support in planning and scheduling a FIT. The Responses to the UWR need to be coordinated between all of TAD before it is sent out.

When a UWR is received in TAD for FIT support, it is assessed based on the following criteria:

Respond to each UWR indicating the status of the request as either "FIT" or " No FIT" .

Review and evaluate information contained in the FIT Requests submitted by the projects requesting their inclusion into FIT.

Ensure the following information is contained in the FIT Requests:

Respond to FIT Requests indicating the status of the request.

The FIT Request process is documented on the FIT web-site and can be accessed by using the following URL: http://tad.web.irs.gov/systbr/fit .

The FIT Project Manager decides when to establish a special FIT project and directs team members to begin the planning process.

All FIT personnel provides input to the scope of the Annual FIT, however, the FIT Project Manager makes the final determination when there is not a clear consensus.

The determination to support FIT for each request is documented in the corresponding response to the request. Additionally, for every FIT project there is a FIT Plan indicating the requests included in a FIT.