A TT&E program is an organized approach to coordinate and streamline activities that must occur when recovering and restoring applications, data, and/or systems during and after a major or minor disruption. The ISCP & DR Testing Checklist is an IRS internal document designed to assist BODs and support staffs in navigating through TT&E events. The Checklist shall be included in the training sessions scheduled prior to all testing and exercise events. See Exhibit 10.8.62-1 for the Checklist Template at the end of this document.

Exercises and tests offer different ways of ensuring that ISCPs provide viable and actionable procedures to recover or restore IRS systems and applications to their original state in the event of a disruption.

Refer to NIST SP 800-84, Test, Training, and Exercise (TT&E) Program for IT Plans and Capabilities, for guidance on establishing an effective ISCP TT&E program and the various methods and approaches for conducting TT&E activities.

All tests and exercises shall include some kind of determination of the effects on the organization’s operations and provide for a mechanism to update and improve the plan as a result.

The depth and rigor of ISCP TT&E activities increases with the FIPS 199 availability security objective. Refer to the ISCP templates (FIPS 199 low, moderate, high) in NIST SP 800-34 for details for conducting TT&E activities appropriate to their respective impact level.

The ISCP shall provide procedures and capabilities for recovering a system or application in the event of an information system disruption. The plan shall address the resources, roles, responsibilities, and procedures for restoration of information systems and recovery of business applications and processes after a disruption.

The disaster recovery keystroke procedures located within the ISCP are an information system-focused part of the plan that applies to major, usually catastrophic, events that deny access to the normal facility or information system for an extended period of time. The plan is designed to restore operability of the target system, application, or computer facility at an alternate site after an emergency.

The purpose of the disaster recovery keystroke procedures is to provide detailed step-by-step procedures to facilitate recovery of capabilities at an alternate site; the scope is information system-focused and limited to major disruptions with long-term effects.

As defined in NIST SP 800-84:

Tabletop exercises are discussion-based exercises only and do not involve deploying or recovering systems, equipment, or other resources. Personnel meet to discuss their roles during an emergency and their responses to a particular emergency situation. In exercising ISCP personnel also identify outdated information or procedures in the plan that need to be updated and corrected.

The objectives of any tabletop exercise are to validate the content of the ISCP and related policies and procedures, validate participants’ roles and responsibilities as documented in the plan, and validate the interdependencies documented in the plan.

Functional exercises allow personnel to validate their operational readiness for emergencies by performing their duties in a simulated operational environment. A functional exercise is designed to exercise the roles and responsibilities of specific team members, procedures, and assets involved in one or more functional aspects of a plan (e.g., backup procedures, communications, emergency notifications, information system equipment setup).

Functional exercises vary in complexity and scope, from validating specific aspects of a plan (e.g., backup retrieval, reading backup data, and validation of offsite storage) to exercising all plan elements in a simulation.

Functional exercises allow staff to execute their roles and responsibilities as they would in an actual emergency situation, but in a simulated manner.

In the context of DR, a test is the method used to evaluate the organization's readiness and ability to recover a system from varying degrees of non-functioning to its original functional state by following authorized ISCP/DR keystroke procedures. Components of tests are listed in these sections, such as using quantifiable metrics to validate the operability of an information system or system component in an operational environment specified in an ISCP.

Tests are used to measure the effectiveness and suitability of the processes and procedures contained in ISCPs related to the systems being tested and to evaluate compliance with an information system contingency. In the event of a disaster or disruption the goal is to be able to use tested ISCPs to ensure that following documented operational procedures and plans will result in successful recovery of business applications and systems.

The scope of tests can range from individual system components or systems to comprehensive tests of all systems and components that support an ISCP. Examples of tests are:

A test is conducted in as close to an operational environment as possible, testing components, or systems used to conduct daily operations.

If feasible, an actual test of the components or systems used to conduct daily operations for the organization can be used to comply with the TT&E program’s annual requirement to test.

Tests that result in components or systems malfunctioning or becoming inoperable could indicate problems in personnel training or in DR plans and procedures.

Each information system component shall be tested to confirm the accuracy of individual recovery procedures.

Each information system shall have a contingency plan that addresses the following areas, as applicable:

Additional test plan requirements:

Training refers to informing personnel of their roles and responsibilities within a particular information system plan and teaching them skills related to those roles and responsibilities, thereby preparing them for participation in exercises, tests, and actual emergency situations related to the information system plan.

The scheduling of training sessions will be coordinated closely with the schedules for ISCP tabletop exercises, functional exercises, and DR tests.

Training sessions will emphasize studying and understanding the following documents in preparation for participating in each test or exercise:

Recovery personnel shall be trained on the following plan elements:

The Checklist is a three part form that allows BODs and Support Organizations to document multiple exercise/test activities on one form to create one authoritative source to standardize and simplify the archival process.

Part A of the Checklist is the Tabletop Exercise, Part B is the Functional Exercise, and Part C is the DR Test or Production Operational Recovery which documents DR Testing activities. See Exhibit 10.8.62-1 for a copy of the Checklist.

The ISCP & DR Testing Checklist provides a step-by-step process to guide participants through the most pertinent sections of the ISCP. The Checklist provides an area to document changes for each section in the ISCP and changes to procedures that may be needed. The Checklist also provides areas to document the results from functional exercises and DR tests, if applicable.

The Checklist standardizes the walkthrough process for all applications and systems, and ensures that all testing activities and ISCP changes are noted and documented in the Checklist. The completed Checklist is then uploaded into TAF as the validated artifact with supporting documentation after the AO has been briefed and DRTEE receives the Checklist signed by the AO.

The Checklist is used to train personnel in their contingency roles and responsibilities with respect to their application or system.

Completion of the Checklist documenting performance of the required exercises and/or tests provides the artifact in TAF to validate that the following family of controls, if appropriate, are met (Reference NIST 800-53, Revision 3, Recommended Security Controls for Federal Information Systems):

Each BOD will be responsible for identifying a Data Collector who will be responsible for documenting the exercise and/or testing activities as they occur and populating the appropriate parts of the Checklist with the description of the activities.

The ISCP & DR Testing Checklist will be used as an artifact in TAF to document all tabletop exercises, functional exercises, and DR tests that are scheduled.

DRTEE will schedule and present training for all BOD and IRS IT participants to prepare them for the current FISMA Cycle ISCP exercises and tests. DRTEE will answer any questions the POCs may have about the test process or the Checklist.

Based on the approved testing schedule and IRS IT/BOD POC list, the assigned DRTEE Facilitator will send the standard conference call invitation to all participating POCs 10 days prior to the day of the exercise. The assigned DRTEE Facilitator shall attach the ISCP and DR Testing Checklist template to the invitation so that the Data Collector (who is assigned by the BOD Security PMO or AO POC as delegated by the AO) can populate the Checklist.

Using the most current version of the ISCP and POA&M stored in TAF the Data Collector populates items 1 through 7 and Part A on the Checklist, and if necessary meets with appropriate BOD or IRS IT personnel to complete this task. When exercising and discussing the ISCP, the Data Collector and BOD or IRS IT personnel should capture noteworthy changes prior to the tabletop. This promotes a more efficient exercise and discussion regarding how to recover the application/system.

After the Checklist is populated, and at least 5 work days prior to the tabletop, the Data Collector shall forward the checklist and the current ISCP to all recipients, including the DRTEE Facilitator. If assistance is needed, the Data Collector should notify the Facilitator or the designated DRTEE Contacts noted in the invitation.

During the tabletop exercise, the Data Collector is responsible for capturing on the Checklist all changes, observations, lessons learned, and summary findings that result from the tabletop discussions. The Date Exercise Completed block must be entered with the date the tabletop was performed.

The Data Collector then has 7 work days to update the Checklist with the result of the exercise. The Facilitator will coordinate with the Data Collector as needed to provide guidance and to compare notes taken during the exercise.

After the Checklist update is completed, the Data Collector shall send it to the *IT IT DR Mailbox with a copy to the BOD Security PMO. DRTEE shall enter it in the test tracking log and forward it to the assigned DRTEE Facilitator, who shall review the Checklist to ensure that all information has been recorded. If the Checklist is complete and no modifications are needed, the DRTEE Facilitator shall record the date and next steps in the test tracking log. If Checklist corrections are needed, the Facilitator will coordinate with the Data Collector to ensure that the modifications are made.

Depending on the required exercises or tests, the DRTEE Facilitator can hold the Checklist until all other testing has been completed and can be documented in Part B or Part C of the Checklist. If no other testing is required, the DRTEE Facilitator shall send the Checklist back to the Data Collector and the BOD Security PMO within 7 work days for final signature (digital signature is acceptable) by the AO or the AO Designee.

The AO or AO Designee has a 30 calendar day maximum timeframe for signing the Checklist unless the June 1 deadline is less than 30 days, then the checklist is due on June 1. The AO or Designee should return the signed Checklists to DRTEE as soon as possible to avoid delays in uploading the completed test packages into TAF prior to the end of the FISMA reporting cycle.

After the AO has signed the Checklist, DRTEE shall submit it to the *IT IT DR Mailbox, and the designated DRTEE Staff member shall load the Checklist and all supporting documentation into TAF.

BOD and IRS IT organizations have 90 calendar days from the signature date of the AO on the Checklist, or June 1, whichever comes first, to revise the ISCP with the changes identified in the tabletop exercise. After changes are made, the BOD/IRS IT designated POC shall send the revised ISCP to the *IT IT DR Mailbox for upload into TAF by the designated DRTEE Staff member. If no changes were noted, the existing version of the ISCP will remain in TAF unchanged.

Functional exercises are performed by IRS IT personnel or by the BOD’s information system personnel when the application is not supported by IRS IT. During the performance of the functional exercises, IRS IT personnel or BOD information system personnel will complete the ISCP & DR Testing Checklist Part B as they go through the exercise. (See Exhibit 10.8.62-1.)

See Exhibit 10.8.62-2, ISCP Functional Exercise Methodology and Procedures. This exhibit provides step-by-step procedures for a backup retrieval and sampling pull for functional exercise activities. All functional exercises will be conducted using the approved procedures in Exhibit 10.8.62-2.

As the production environment implements new technologies, strategies, and procedures, IRS IT and SRM shall assess when to modify Exhibit 10.8.62-2 procedures to ensure that functional exercises can be performed to accommodate the updated production environment.

During the functional exercise, the IRS IT or BOD information system personnel will take screen prints of the backup tool index header and tape or server listing to validate the method used to backup system files and/or application data files. Take additional screen prints to validate that the data on the backup media is readable. The IRS IT or BOD information system personnel will also provide evidence in the form of routing sheets, logs, or e-mail requests proving the length of time needed between the request for backup data from offsite storage and the receipt of that data at the test site.

IRS IT or BOD information system personnel shall also provide evidence to validate that documented backup procedures are in place including information about the backup frequency, encryption of backup media, offsite storage site, and timelines for receipt of backup media from offsite storage during normal working hours and after hours.

If no documented procedures describe the backup process, annotate the Summary Findings section in Part B of the Checklist to document this issue. Annotate the Summary Findings section if the backup tapes are corrupted or if evidence cannot captured for the exercise.

At the end of the functional exercise, the IRS IT or BOD information system personnel shall update the Checklist with results from the exercise. The IT personnel performing the exercise will submit the populated Checklist and supporting evidentiary documentation to DRTEE at *IT IT DR Mailbox within 10 work days from the completion of the exercise.

DRTEE will ensure that the populated Checklist received from the IT personnel who performed the functional exercise is consolidated with the Tabletop Exercise Checklist. DRTEE will meet with the BOD POCs and the AO POC to share with them the Checklist showing the results of the completed exercises in preparation for the AO signature.

The AO POC will present the completed test package to the AO for review of the exercise results, the summary findings, and final validation. The AO shall sign and date the Checklist and the AO POC shall return the signed Checklist to DRTEE at *IT IT DR Mailbox for final action.

Upon receipt of the signed Checklist and supporting documentation from the AO POC, DRTEE shall upload the Checklist into TAF as the validated artifact along with all supporting documentation.

IRS is required to perform DR tests on all applications with a High Availability Impact and for CIP assets. These tests are designed to evaluate IRS readiness to cutover, relocate, restore, or rebuild IRS systems/applications.

DR tests involve activities such as performing cutovers from one platform or system to another, relocation of systems/applications, or recovery of platforms and their hosted applications. As DR tests are performed on systems, sites, or platforms, hosted applications can benefit from these tests through coordination of the application ISCP review and the DR test activities.

IRS IT personnel perform DR tests unless IRS IT does not support the application. The BOD’s information system personnel perform DR tests when the application is not supported by IRS IT. During the performance of the DR Test IRS IT personnel or BOD information system personnel shall complete the ISCP & DR Testing Checklist Part C, and Test Case templates as they go through the test. (See Exhibit 10.8.62-1.)

The DRTEE Staff will coordinate with IRS IT organizations to identify components, systems, and/or comprehensive tests to be planned based on FISMA, Treasury, and NIST requirements, and IRS executive-level priorities.

Production operational recoveries can also be considered in meeting FISMA and DRTEE program requirements. The Service may also consider combining tests with planned operational activities, such as restoring a backup, moving a server from one room to another, upgrading or patching operating systems or applications, or changing hardware components (e.g., swapping hard drives, replacing a failed power supply). The results of this collaboration will define the scope and objectives for the tests.

The DRTEE Staff will collaborate with designated BOD POCs to determine if the tests identified in collaboration with IRS IT are compatible with the priorities and processing timeframes of the Business Unit. DRTEE will coordinate with BODs to determine the level of involvement required from the BOD POCs.

The DRTEE Staff shall create a test schedule based on IRS and FISMA requirements, FISMA timeframes, and business processing priorities.

The DRTEE Staff will coordinate the following activities with IRS IT and BOD POCs to ensure that the Test Case Template, Test Activities Worksheet, ISCP & DR Testing Checklist, Summary Report, and all testing documentation is completed before, during, and after testing. DRTEE will:

For the purposes of reporting on the progress of exercises and testing, DRTEE shall maintain at least two specific scorecards. One scorecard will document the progress of the ISCP tabletop and functional exercises, and a separate scorecard will provide the status of the DR tests.

DRTEE shall input all activities and documentation into TAF in a timely manner. All changes to the application or system must be recorded in TAF. DRTEE shall document changes identified during the testing process in the ISCP & DR Testing Checklist artifact or in the ISCP. As these artifacts are created and/or updated, update the Contingency Planning (CP) fields in TAF with completion dates.

Documentation for all activities and all actions performed must be completed in a timely manner. The results of each exercise/test must be fully documented using the ISCP & DR Testing Checklist and then uploaded into TAF.

The following TAF documentation and TAF CP fields are uploaded and updated after exercise/testing is completed: