2.14.2  Enterprise Incident Management Standards (Cont. 1)

2.14.2.7 
Incident Escalation/Notification

2.14.2.7.1 
Priority 1 and Priority 2 Incident Escalation

2.14.2.7.1.5  (03-17-2010)
Notification

  1. It is the responsibility of the ESD to forward information to their internal and external customers as directed by the Service Provider. The ESD will utilize a common e-mail distribution list to relay information in regards to Scheduled and Unscheduled Outages as well as the sharing of information. Each type of notification with a detailed description is listed below.

2.14.2.7.1.6  (03-17-2010)
Informational Alerts

  1. Informational Alerts (General Information) is the sharing of information prepared and submitted to the ESD by the Service Provider, that is not a Scheduled or Unscheduled Outage. The ESD will forward the Informational Alert to the alert distribution list. The Information Alert will be closed immediately following customer notification by the ESD (Not a Scheduled or Unscheduled Outage).
    The Standard Information Alert Template is located on the following URL: http://eues.web.irs.gov/ESD/ESD-MAIN.aspx

  2. The list below shows examples of Information Alerts:

    • Holiday Schedules

    • Holiday Mainframe Processing Schedules

    • Accelerated Cycle Instructions

    The Standard Information Alert E-mail Subject Line Format is:

    • SUBJECT LINE: SUBJECT @ LOCATION – Incident Alert #xxxxxxx

2.14.2.7.1.7  (03-17-2010)
Unscheduled Outage

  1. This is an outage to a system or service that does not have a designated date/time for being unavailable to customers. The Service Provider is responsible for providing to the ESD scope and impact via the Unscheduled Outage template that will be attached to the ITAMS P1 incident or accessible via the URL listed below. The Incident Alert will be issued by the ESD to the customers via the alert distribution list once the scope and impact has been defined. The ESD will issue closeout notification once the Service Provider has provided a valid resolution.

  2. Standard Incident Alert Template is located on the following URL: http://eues.web.irs.gov/esd/esd-main.htm .
    The list below shows examples of Incident Alerts:

    • Realtime unavailable (IDRS, CFOL, etc.)

    • Modernization Applications/Systems unavailable

    • Server unavailable impacting multiple customers

    • Command Code(s) unavailable

    • Generation of bad or incorrect data

    • Corruption of data base files

    • Any Unscheduled Downtime

    • Work Stoppages impacting MULTIPLE users


    The Standard Unscheduled Outage E-Mail Subject Line Format is:

    • SUBJECT LINE: SUBJECT @ LOCATION - Incident Alert #xxxxxxx - Incident #xxxxxxx

2.14.2.7.1.8  (03-17-2010)
Scheduled Outages

  1. A Scheduled Outage is an outage to a system or service that has a designated future date/time for being unavailable to customers. A Scheduled Outage is prepared and submitted to the ESD by the Service Provider before the scheduled outage takes place. The ESD will forward the Scheduled Outage Alert to the alert distribution list. The Scheduled Outage Alert will be closed once the scheduled end date/time has been met.

  2. Standard Information Alert Template is located on the following URL: http://eues.web.irs.gov/esd/esd-main.htm.

  3. The list below shows examples of Scheduled Outages:

    • Scheduled Maintenance - Preventative Maintenance

    • Scheduled Power Outages

      The Standard Scheduled Outage Alert E-mail Subject Line Format is:

    • SUBJECT LINE: SUBJECT @ LOCATION - Incident Alert #xxxxxxx

2.14.2.7.1.9  (03-17-2010)
Service Providers

  1. ESD Specialists can be notified via ITAMS SC mail (if the operator record has been flagged for notifications) when one of the following conditions for the incident activities listed below. All incidents assigned to the ESD are available in the Web queue or the EUES SD queue.

    Note:

    Due to volume most ESD Specialists do not have notifications flagged.

    • Assignment of incident

    • Shared Assignment of incident

  2. All other Service Provider groups will be notified via ITAMS SC mail, enterprise e-mail or designated network printer when one of the listed conditions applies:

    • Assignment of incident

    • Reassignment of incident

    • Shared Assignment of incident

  3. Notification settings are controlled in the Assignment Group, Operator list. To alter the setting an ITAMS incident should be opened to the local Problem Administration for action. The settings for Specialist should always be set for only internal notifications.

2.14.2.7.1.10  (03-17-2010)
Customer Advisory

  1. A Customer Advisory is issued enterprise wide by the Enterprise Advisory Team using the Standard procedures documented at http://mits.web.irs.gov/M_ProceduresGuidelinesStandards/ProcsGuidesStands_Advisory.htm .

  2. The Enterprise Service Desk will create an Info Alert incident in ITAMS for all Advisories. Since these advisories have already been distributed to the required recipients ESD will not forward the Enterprise Advisory to standard ITAMS Information Alert distribution list. At a minimum the following information will need to be included using the standard ITAMS Information Alert template:

    • What is planned?

    • Who will be affected?

    • How will you be affected? If you have incidents or need additional information, please visit the Enterprise Service Desk (ESD) at http://getservices.web.irs.gov or call ESD at 1-866-7HELP4U (866-743-5748) or TDD/TTY: 1-866-435-7486 (1-866-HELP4U6)

2.14.2.7.1.11  (03-17-2010)
Alert Notification/Escalation

  1. Alerts are built-in "clocks" that are turned on as soon as a incident is opened. The priority of the incident can determine the behavior of the alert process. The built-in alert processes are the escalation of the incident to a deadline alert.

  2. The Escalation of a incident refers to a preset timer that will systemically place the incident in an Alert Status and send e-mail indicating the change in status to a preset group of ITAMS users or an individual (depending on priority).

    Note:

    The preset /group individuals must ensure " Out-of-Office E-mail Rule" is activated in order for their designee to receive alerts.

  3. This clock begins when a incident is opened and the administrator, through the use of alert expressions, has set the time for the alert. Alerts will systemically reset every time the incident is updated except for Deadline "clock" times.

  4. Alert times are based on calendar time of the assignment groups calendar if there is a calendar set up for the assignment group. If no calendar is set up for the assignment group, it is assumed the assignment group works 24X7. If an assignment group works from 11:00 p.m. to 7:30 a.m., the alert could be set to change only during those hours. After alert stage one has passed, alert stages 2 and 3 will follow if the incident has not been updated within the predetermined time. This predetermined time is based on either the category or a combination of category and priority. The Alert messages are then sent to designated personnel as defined in the Alerts/Notification Section.

  5. The Deadline Alert process starts when the incident is opened and continues based on target resolution time and will trigger if target resolution time is exceeded regardless of timely updates. After a predetermined time the incident status will change to a Deadline Alert Stage and the designated personnel are notified through Enterprise e-mail of this status. The purpose of the Deadline Alert is to avoid keeping incidents opened for an extended length of time. It is also used to notify management of high priority incidents that are opened but not resolved. A Deadline Alert can never be altered and is calculated on the open time of the incident.

  6. Listed below are the Alert Basics:

    • Alerts are based on the business days of the primary assignment group calendar.

      Note:

      The calendar will only effect the times a notification e-mail will be sent. It will not alter the service level commitment for Priority 1 or Priority 2 and does not stop the manual escalation of the incident.

    • A incident can go into Deadline Alert even if it never goes into Stage 1 Alert. If a priority one incident has to be resolved in four hours, it will go into Deadline Alert regardless of how many times it was updated.

  7. Only Priority 1 and 2 incident ticket alerts are activated. Systemic notification will occur at the time intervals as defined in Exhibit 2.14.2-4. Priority 3 through 5 and R1 through R5 service requests alerts are not activated and are not scheduled to be activated.

2.14.2.7.2  (03-17-2010)
Problem Management: Root Cause Analysis

  1. Root Cause Analysis (RCA) facilitates identification, location, analysis, and documentation of repeated problems. The main goal of an RCA is to minimize the effects of system down time/lost productivity caused by errors in the IT infrastructure, and to prevent their recurrence.

  2. The Problem Management process is owned by Enterprise Operations (EOPS) and managed/facilitated by the EOPS Service Operations Specialist (SOS) Section.

  3. An RCA will be opened following the RCA Business Rules listed below:

    • A Service Restoration Team (SRT) activity was conducted for a particular incident and root cause was not determined;

    • An outage lasted for more than four hours;

    • A particular workaround was used for a recurring incident more than three times;

    • More than 100 users were impacted by an outage;

    • The incident was time sensitive in nature; e.g. it impacted the interest free cycle;

    • More than two applications were impacted by one incident; or

    • Request made by MITS management

  4. If the SOS PM determines the RCA request is a valid candidate, an e-mail will be sent to the requestor of the RCA asking them to identify the team members who will assist in identifying the Root Cause and permanent solution for the problem reported. Once the RCA Team Members have been identified, an RCA Kick-Off meeting will be scheduled.

  5. If the request is a valid RCA candidate and there is an existing RCA, the P1/P2 incident and Service Request will be associated to the existing RCA.

  6. If the request does not meet the criteria for a valid RCA, the requestor will be notified and an explanation provided. However, the request may be resubmitted to the next level of management for reconsideration.

  7. ROLES AND RESPONSIBILITIES

    Service Operations Specialist

    • Facilitate the problem management process through the identification of the root cause and workaround (if applicable) and the permanent solution

    • Schedule and facilitate conference calls

    • Record notes and action items

    • Update the RCA record on ITAMS

    • Post Action Plan and Notes on the EOPS RCA SharePoint

    • Identify next steps

    RCA Team Members

    • Identify additional team members and/or make corrections to the list of Team Members, if applicable

    • Attend scheduled calls or designate someone to attend in your absence or in your place

    • Review the notes and assigned action items

    • Follow-up on action items prior to the next scheduled call

  8. RCA UPDATES

    1. Status Updates

      The RCA record will be kept up to date through closure with actions being taken by the RCA team to identify the Root Cause and Workaround, if applicable, and Permanent Solution.

      The SDM Problem Management personnel will forward an e-mail to the RCA Team and MITS Executives after each RCA meeting to provide status of RCA and next steps. The RCA Team Member who is the liaison to the business customer will keep the business informed of status updates.

      Meeting Notes, Action Items and other RCA related artifacts are placed on the listed SharePoint site: http://ss.ds.irsnet.gov/sdccb/rca/2009%20RCA/Forms/AllItems.aspx

    2. Workaround

      If a workaround is available it will be entered into the RCA record changing the status of the RCA and systemically notifying the team members that a workaround has been identified. The ESD searches open RCA’s when opening P1/P2 tickets to determine if there is a workaround for the incident being reported.

      If a workaround is available, the ESD will provide assistance to the customer following the directions documented under the Workaround tab.

    3. Root Cause

      A Root Cause could be identified/entered at anytime during the lifecycle of the RCA.

    4. Resolution

      The resolution will be entered and the RCA closed once the RCA Team has identified and implemented a permanent solution.

  9. RCA CLOSE-OUT

    A close-out RCA e-mail will be issued to the team identifying the Root Cause (if known) and Resolution.

    The detailed Root Cause Analysis Business Process is located on the following link: http://ss.ds.irsnet.gov/sdccb/rca/default.aspx

2.14.2.8  (03-17-2010)
Customer Requests: Move, Add, Change (MAC) Category

  1. MACs are used to track customer request for products, services, as well as track MITS related projects. The category for MACs should be " CUST REQ" and the Sub-Category will vary depending on the product.

    • MAC ADD

    • MAC CHG

    • MAC PLAN

    • MAC ADD1 (SW requests through Get-IT tickets only)

  2. Please refer to Section 2.14.2.21 OS GetServices for complete ticket procedures to process customer requests and Section 2.14.2.3.2.2 for procedures to use MAC PLAN tickets for project planning.

2.14.2.9  (03-17-2010)
Service Requests: Priority R1-R5 Production Processes

  1. Service Request - This section is intended to establish uniform procedures for Service Requests related to Mainframe processing.

2.14.2.9.1  (03-17-2010)
Service Requests

  1. Service Requests:
    The vehicle for submitting Mainframe Service Requests is the Information Technology and Asset Management System (ITAMS). This process should be used to request changes to:

    • Real time schedules

    • Printed and electronic output schedules

    • Special utilities

    • Notice suppression and holds

    • Processing schedule changes

    • Reruns

    • Audit Trails

    • SACS TVT

    Note:

    This process does not override the Enterprise Incident Management Standards IRM process for incident resolution, file tracking or the System Change Request process for changes to system/program design.

2.14.2.9.1.1  (03-17-2010)
Service Request Responsibilities/Requirements

  1. All Internal Revenue Service (IRS) organizational components providing Information Technology (IT) Mainframe support to or receiving IT Mainframe support from the Enterprise Operations are responsible for adhering to these procedures.

  2. EUES, IDSE QSC, and Enterprise Operations will identify the personnel with the appropriate authority to request and negotiate changes that are not associated with the resolution of an incident.
    QSC - EOPS Incident resolution

    The EUES IDSE QSC Staff will:

    • Perform first-line diagnostics and recommend incident resolution;

    • Provide input to determine and track the best incident solution, monitor the impact of the corrective action and, if subsequent actions are necessary, coordinate with the Enterprise Operations;

    • Confirm the incident is not at the local level before the ITAMS tickets are shared outside the campus;

    • Work very closely with the staff at the Enterprise Operations and service campus to minimize the impact of the incidents;

    • Participate in conference calls and one-on-one technical discussions, if necessary;

    • Update the ITAMS ticket to indicate it has been reviewed;

    • Include, on the ITAMS ticket, any actions taken to identify the cause of the incident or to expedite the incident resolution;

    • Include, on the ITAMS ticket, the Liaison name and phone number as a point of contact;

    • Place the ticket in Pending status when the Enterprise Operations has performed their portion of the resolution until the customer is satisfied with the result;

    • Verify satisfaction of the incident resolution with the customer before closing the ticket;

    • Place the ticket back into Open status if the customer is not satisfied;

    • Ensure the tickets are closed within 24 hours of the final resolution.

    Customers requesting additions and changes to products approved by Division heads or their designee, will:

    • Request, through the MITS Enterprise Service Desk, additions or changes to products and service schedules defined in the Service Level Agreement;

    • Make requests for changes to the products and service schedules, e.g., real-time or additional prints, one week in advance of the requested change date, or as soon as possible for emergency changes, e.g., last minute processing changes;

    • Include in the requests, information such as: what is required, when it is required, and a name of a customer contact.

    • Indicate in the requests, the duration of the special request or if the change is permanent.

    The ESD will:

    • Document all request information on ITAMS;

    • Assign a priority to requests. (R1-R5);

    • Assign the ticket to the responsible service provider. The first assignment will be to the Integrated Document Management Solutions Enterprise Quality Support Center staff (IDSE QSC).

    • Respond to requests within the defined time frames and support special requirements within the target resolution times.

    The Enterprise Operations will:

    • Respond to the requests from the IDSE QSC Staff within the defined frames and support special requirements within available resources, both system and personnel;

    • Ensure all Service Request information is documented on ITAMS.

2.14.2.9.1.2  (03-17-2010)
Service Request: IRSC/Business Areas

  1. The following are procedures to be followed when Service Requests originate in the IRSC or business area:

    1. The authorized end user from a business area will contact the ESD organization with a MITS incident or request; define requirements; ensure that the authorized official has given approval; open an ITAMS Customer Request, subcategory Service Request ticket; ensure the ticket contains the specific requirements (including due dates and duration).

      Note:

      For routine releases only a contact name is necessary; ITAMS will generate the time.

    2. The ESD will ensure the incident ticket is assigned to the appropriate (Service Provider), in many cases this will be the QSC Staff servicing local Campus Document Management Center Staff (CDMC), and will assign a priority code.

    3. The QSC/CDMC Staff will perform an initial analysis with the customer to obtain detailed Information, to document all the necessary information on ITAMS ensuring a complete understanding of the incident, to determine any impact on others, to determine if Enterprise Operations (EOPS) intervention is necessary.

    4. The QSC/CDMC Staff will, if EOPS intervention is necessary, share the ticket with the EOPS.

    5. The EOPS will assign the incident ticket to the appropriate EOPS MITS support personnel and respond within the time frames defined in Exhibit 2.14.2-3 "Service Request Priority Definitions" .

    6. Both the QSC/CDMC Staff and the EOPS will work the issue and update the ITAMS incident ticket until a temporary change occurs and normal processing resumes, a permanent change is implemented, or the request is denied.

    7. When EOPS has completed their work, EOPS will place the resolved ticket into Pending status and notify the QSC/CDMC Staff.

    8. The QSC/CDMC staff will contact the customer with resolution information and ensure the customers satisfaction before closing the ticket.

    9. The QSC/CDMC staff will be responsible to inform EOPS that further intervention is necessary by placing the incident ticket back into Reassign status when the customer is not satisfied with the resolution or with the output received.

2.14.2.9.1.3  (03-17-2010)
Service Request: Enterprise Operations

  1. The following are procedures to be followed for Service Requests originating in the Enterprise Operations:

    1. The authorized EOPS personnel will open an ITAMS Service Request incident ticket by contacting the ESD for all processing changes that may have an impact on the customer. The ticket must contain the specific requirements including the ticket assignment to the EOPS Service Provider, due dates and duration.

    2. The ESD will share the incident ticket with the IDSE QSC Staff.

    3. Incident tickets should be shared if there is an impact to IRSC Business owners.

    4. The IDSE QSC staff will ensure the incident ticket is disseminated to impacted IRSC Business Customers.

    5. The IDSE QSC will perform an analysis with the customer to determine any impact on others and coordinate changes with the customer and with EOPS.

    6. Both the QSC Staff and the EOPS MITS will work the issue and update the ITAMS incident ticket until a temporary change occurs and normal processing resumes, a permanent change is implemented, or the request is denied.

    7. The EOPS Service Provider will contact the QSC Staff with resolution information and resolve the incident ticket. When EOPS completes their work, will place the resolved ticket into Pending status and notify the QSC Staff.

    8. The IDSE QSC Staff will verify the resolution with the customer and determine the customers satisfaction.

    9. The IDSE QSC Staff and the EOPS MITS support personnel will record all permanent changes as possible issues to be addressed at the next Service Level Agreement review and negotiation process.

    10. The EOPS will provide change status information on each daily status report until after the change occurs.

2.14.2.9.1.4  (03-17-2010)
Service Request: Root Cause Analysis

  1. Enterprise Operations customers will utilize the Service Request process when requesting a Root Cause Analysis.

    • R1 - if request is initiated from a P1 ticket

    • R2 - if request is initiated from a P2 ticket

  2. The SOS PM will review and evaluate the Service Request and will respond to the initiator via the Service Request following standard procedures for updating and closing Incidents.

2.14.2.10  (03-17-2010)
Enterprise Policy and Procedure: Tool Change Requests

  1. Tool change requests are handled by the UWR process. For guidance see the Enterprise Services Demand Management web site http://mits.web.irs.gov/ES/DM/ .

2.14.2.11  (03-17-2010)
Archive Process

  1. In an effort to manage the Service Center Incident Management (IM) database size and continue to have access to all data for analysis purposes, tickets will be archived as follows:

    1. Incident Management incidents will be available for 36 months - Open, Closed, Active. Incidents are archived after 12 months. There are two archives: incidents 13 -26 months old and incidents 26 - 36 months old.

    2. Call incidents (not Open) will be available to the Enterprise Service Desk (ESD) personnel following the same standards as Incident Management incidents listed above.

    3. All Change Management tickets are available.

2.14.2.12  (03-17-2010)
Downtime Recording

  1. The downtime time available on all hardware incidents will be prepared via ITAMS incidents for all IT equipment requiring remedial maintenance.

  2. The Downtime Tab (D/T) will be completed via ITAMS following the procedures for a hardware incident. The entry on ITAMS must include the time the Computer Engineer (CE)/vendor was contacted, the time the CE/vendor arrived, the time the repairs were completed, and an incident description identifying what was done to resolve the incident included in the description/resolution field.

2.14.2.13  (03-17-2010)
Inventory Discrepancy Incident

  1. When opening a hardware related incident and the inventory and/or vendor information is not accurate and/or does not populate, then an Inventory Discrepancy Category incident should be opened. The discrepancy incidents or events will be forwarded to the local inventory control group for research and correction in ITAMS Asset Center.

2.14.2.14  (03-17-2010)
File Tracking

  1. The Enterprise Service Desk (ESD) will open a File Tracking Category incident on each Problem File, as they occur. When the Problem file is identified as needing to be replaced, the incident will evolve into a File Replacement. The incident will stay opened until the File has been successfully replaced and the data is either input or printed. Only one incident per File will be tracked. The file tracking category is primarily used for mainframe file processing.

2.14.2.15  (03-17-2010)
Alternative Incident Processing

  1. During periods of ITAMS SC unavailability alternative incident processing must be used. An ITAMS Priority 1 incident must be recorded and relayed via phone to the appropriate Service Provider. There are two basic situations that will result in the use of this process.

2.14.2.15.1  (03-17-2010)
ITAMS Inaccessible

  1. Each Enterprise Service Desk (ESD) will immediately begin using alternative methods of incident reporting when ITAMS is unavailable. This can be manual incidents, the disaster recovery site and/or e-mail reporting and tracking. In using either medium the ESD Specialist must prepare the incident, sequentially record each incident and gather the following information:

    • Incident Start Time

    • Reported By Name

    • Contact Name

    • Telephone Number

    • Incident Site ID

    • Building Code

    • Priority Code

    • Project

    • Asset ID (Bar Code), if applicable

    • Incident Description

2.14.2.15.1.1  (03-17-2010)
Disaster Recovery Website for Reporting

  1. The ESD backup web site http://itssupport.web.irs.gov:8080/ITAMSBackup.asp is used as soon as it is determined that ITAMS SC is unavailable or is working in a degraded status. The site allows ESD Specialists the ability to create incidents for later import to the ITAMS SC system when available. This backup ticketing system is to be used to track issues and provide temporary incident numbers to customers during ITAMS outages or degraded status. Service Provider will be sent a incident via e-mail during an outage. The disaster recovery e-mail address list will be used to e-mail incidents to each territory and organization when ITAMS is down.

  2. Once ITAMS is available, the temporary incident information will automatically import into ITAMS SC and create an incident ticket with the temporary incident number in the key field. An ITAMS incident will be issued to replace the temporary incident. Customers will receive an e-mail with the new incident number. ESD Specialists and Service Providers are responsible for updating and closing incidents in ITAMS that were worked during the ITAMS outage. The Disaster Recovery site contains minimal table data to allow for accurate incident routing.

  3. At the first instance that ITAMS becomes unavailable an ESD Escalation Specialist will immediately send out an e-mail to the &MITS-EUES Enterprise Service Desk distribution list asking if other sites are experiencing the incident as well. Only ESD Specialists working Escalation for that day will respond. Once it is determined that there is an enterprise wide outage the Escalation Specialist whose group is assigned to e-mail processing for that week will immediately create a High Priority incident and issue the ITAMS Unscheduled Outage template. Since this is a Priority 1 for the ITAMS outage, notify the appropriate Service Provider via phone as documented. Use the escalation contact list for the appropriate manager contact.

  4. The recovery site accommodates the backup ticketing process for input of Fix-IT and Get-IT incidents. The site also contains the P&R Guide, MITS Reports, and Outage E-mail for usage during these outages by the ESD Specialists. Once the incident is submitted and a temporary incident number is obtained, the ESD Specialist will click on the "Email This Ticket" button and e-mail the incident to the appropriate territory e-mail group using the Territory E-mail Address spreadsheet posted on the disaster recovery site. Service Providers should start checking their territory e-mail inboxes for issues relating to their group as soon as ITAMS becomes unavailable.

  5. Any incidents initially resolved by ESD while working on the backup system will be assigned to the ESD assignment group and will be closed on ITAMS as soon as the system becomes available and temp incidents are imported.

  6. For Priority 1 and 2 incidents, the ESD Specialist will need to document the information in the temporary incident and obtain a temporary incident number; contact an Escalation Specialist with the issue and then e-mail the incident information to the responsible territory group. A copy of the e-mail should be shared with the Escalation Specialist for tracking purposes. The Service Provider group managers will receive phone calls from Escalation Specialists notifying them of high priority incidents. Managers should provide the name and number of a contact to the Specialist for further timely follow-ups. Once ITAMS is available, all work performed should be documented in the incidents and incidents closed accordingly. Please note, special consideration should be taken to accurately reflect the Incident Start Time and Stop Time if appropriate.

  7. The Service Desk Analyst Responsibilities for supporting Disaster Recover operations are listed below:

    • The Probe and Response (P&R) Guide should be extracted every 30 days from ITAMS and posted on the ESD desk guide site for disaster recovery.

    • The program table should be extracted monthly and posted to the ESD desk guide site.

    • The Service Provider directory table should be extracted monthly and posted to the ESD secure contacts web site.

  8. Any additions or deletions of assignment groups by ESD Problem Administrators will need to be coordinated to ensure the back up sites and reports are accurate. An e-mail can be sent to *MITS-EUES ESD Analysts Feedback for coordination.

2.14.2.15.2  (03-17-2010)
LAN/WAN Outage Affecting ITAMS Incident Management Systems

  1. The above procedures will be followed with the exception that manual incidents can be used and distributed during this time. These paper incidents will consist of a hard copy sheet or a stand alone excel sheet file which can be faxed/hand carried or e-mailed to Service Providers. The Service Providers with updates and resolutions will document it on the paper incidents until the ITAMS is available. The Service Provider will update all updates or resolutions from the Service Areas once ITAMS is available and the Specialists have entered the incident.

2.14.2.16  (03-17-2010)
ITAMS Outage Notification

  1. ITAMS Outage Notification

2.14.2.16.1  (03-17-2010)
Unscheduled Outages

  1. When an outage to the ITAMS system is encountered an e-mail (using the standard unscheduled outage e-mail template) will be issued to the following e-mail distribution lists: &MITS ACIOEUES ESD Unscheduled Info Alerts, &MITS ACIOEUES-ESD ITAMS and must contain the following information:

    • Employees will be unable to reset their LAN passwords using Identity Management Suite DIRECT! (gold key) or access their Password Management profiles (all PWM calls during this outage are considered EXCLUDED.) Employees will be unable to input help incidents via the web. Service Providers will be unable to update/resolve/close ITAMS incidents. Tivoli incidents will not be auto-generated for Modernization applications and infrastructure. Customers should continue to contact the Enterprise Service Desk (ESD) via the phone or e-mail to report incidents.

    • The ESD will revert to using the alternative ticketing process and will open an incident for each incident reported during the unscheduled outage. Individual incident information will be forwarded to Service Providers via e-mail (using the Disaster Recovery - DR Mailboxes for ITAMS Outages contact list located on the ESD backup web site) to be worked as they are opened. (There is no current capability to update the incidents in ITAMS or in the alternative system or to provide status on incidents while the alternative ticketing process is being used.) Once ITAMS is back up, the incident information collected in the alternative system will be imported into ITAMS. Service Providers are responsible for entering updates/resolutions and closing their incidents.

    • The ESD Escalations for high priority issues will utilize the Disaster Recovery - DR Mailboxes for ITAMS Outages contact list located on the ESD backup web site as well for any Priority 1 and/or Priority 2 incidents reported during this outage will have updates sent out by ESD Escalation (via e-mail) as they are obtained from Service Providers.

2.14.2.16.2  (03-17-2010)
Scheduled Outages

  1. The Scheduled Outage process in Section 2.14.2.7.1.8 should be followed with the following included in the summary:

    • Employees will be unable to reset their LAN passwords using Identity Management Suite DIRECT! (gold key) or access their Password Management profiles (all PWM calls during this outage are considered EXCLUDED.). Employees will be unable to input incidents via the web. Service Providers will be unable to update/resolve/close ITAMS incidents. Tivoli incidents will not be auto-generated for Modernization applications and infrastructure. Customers should continue to contact the Enterprise Service Desk (ESD) via the phone or e-mail to report incidents.
      The ESD will revert to using the alternative ticketing process and will open an incident for each incident reported during the scheduled outage. Individual incident information will be forwarded to Service Providers via e-mail (using the Disaster Recovery - DR Mailboxes for ITAMS Outages contact list located on the ESD backup web site) to be worked as they are opened. (There is no current capability to update the incidents in ITAMS or in the alternative system or to provide status on incidents while the alternative ticketing process is being used.) Once ITAMS is back up, the incident information collected in the alternative system will be imported into ITAMS. Service Providers are responsible for entering updates/resolutions and closing their incidents.
      The ESD Escalation for high priority issues will utilize the Disaster Recovery - DR Mailboxes for ITAMS Outages contact list located on the ESD backup web site. Any Priority 1 and/or Priority 2 incidents reported during this outage will have updates sent out by ESD Escalation (via e-mail) as they are obtained from Service Providers.

  2. The Scheduled and Unscheduled Outage standard templates are located at the following URL: http://eues.web.irs.gov/ESD/ESD-MAIN.aspx

2.14.2.17  (03-17-2010)
Personal Inbox

  1. Each ITAMS Service Center (SC) user has the capability to setup their own personal inbox to display incidents using their defined query. This functionality is provided to allow all users the ability to query incidents based on their own criteria. The inbox will only be viewable by the user who created the inbox.

2.14.2.17.1  (03-17-2010)
How to Create an Inbox in ITAMS SC

  1. Follow the steps below to create an inbox in ITAMS SC:

    1. 1. After entering your search criteria in the “Search Incidents Tickets” screen, click the search button to run the query. After the results are displayed go up to “Options” on the menu bar, from the drop down menu select “Save as Inbox.”

    2. 2. Answer "Yes" to message "“Save this record as an inbox?”"

    3. Enter an appropriate title in the field "What would you like to call this inbox?"

    4. Leave the Only Me entry.

    5. You can sort the data in your inbox if warranted.

    6. Click the Add button.

    7. Click OK.

      Note:

      You must log out of ITAMS SC and back in for your inbox to be available.

2.14.2.18  (03-17-2010)
Problem Administrator: Responsibilities/Requirements

  1. There are two types of Problem Administrators for the ITAMS SC module. The Enterprise Problem Administrators and the Field/Territory Problem Administrators:
    Enterprise Problem Administrators: have the responsibility to add/update/delete enterprise tables in conformance of Enterprise Problem Management Standards. A list of the majority of the tables:

    • Assignment Groups

    • Operator Profiles

    • Sub-Category

    • Project System Name

    • Service Provider

    • Holiday Schedule

    • Vendor Table

    • Desktop counts Table

    • Calendar Management

    • Software COTS Tables

    • Program Name Table

    • Application Program Registry

    • Probe and Response


    Field/Territory Administrators: are responsible for the following tables in relation to field office updates. These changes must be documented on incidents before system updates to the tables can be made.

    • Operator Profiles

    • Service Provider Tables

    • Operator adds and deletes in the assignment group

    • Operator Notification


    A quarterly review of the Service Provider Data will be required by each site.
    The ITAMS Problem Management Administrators Guide identifies the roles and responsibilities of the various types of ITAMS Problem Management Administrators as well as detailed instructions on the ITAMS Service Center Problem Administrator capabilities. The ITAMS Problem Management Administrators Guide can be located at the following URL: http://mitsreports.web.irs.gov/pr.asp . Select "ESD Problem Management Administrators Guide."

2.14.2.19  (03-17-2010)
ITS Customer Satisfaction Survey

  1. A customer satisfaction survey process is used for a MITS products and services tracked and delivered in the ITAMS SC incident management process. The Survey is maintained by the Strategic Plan & Perf Mgmt Div. Customer requests are sampled using a predetermined sampling strategy and maintained on an ITAMS Service Center table. Upon closure the incident it is analyzed to determine if it is a candidate for survey. If it is determined to be a candidate an e-mail is sent to the customer as the close out e-mail on the incident. The feedback link brings the customer to an external server to record the satisfaction results.
    Who/What gets surveyed?

    • Surveys are sent to the contact (reported for) when an ITAMS incident is closed.

    • CALL incidents and incident opened via OS GetServices or DR site.

    • Sampling rate (surveys to incidents) based on customer BOD effective FY05. Frequency is controlled by a SurveyCtrl file maintained by Strategic Plan & Perf. Management Division (formerly known as RAM).

    • Survey exclusions are managed by a Survey Exclusion Table which can be changed by select Service Center users. Incidents that are excluded from the survey:
      - Call incidents opened by GETITPWM (password resets).
      - Incidents not opened by GET-IT or WEBOPEN (web incidents).
      - Subcategory MAC-PLAN.
      - Resolution code NO CUST RESPNSE.

    • Additional Exclusions are performed after receipt:

      -"Late survey responses" - Survey responses received more than 21 days after survey was sent.

      - Duplicate survey responses.

    • Available data elements for slicing (ad-hoc only):

      Assignment Group, Building Code, Business Unit, Category, Subcategory, Priority, Resolution Code.

    • Satisfaction Rate:

      Number of survey responses equaling 4 or 5 divided by the total number of valid survey responses.

    • Survey Questions
      - Regarding your closed incident, how satisfied are you with...
      - The time it took to submit your request?
      - The professionalism of MITS staff?
      - The resolution of your issue?
      - The time it took to resolve your issue?
      - The overall experience?

    Note:

    Customers rate each question on a scale of 1-5

2.14.2.19.1  (03-17-2010)
ITS Enterprise Reports

  1. An Online report generation tool was developed to provide MITS employees and management a better understanding of their pending active ITAMS requests.

  2. The Reports Web Site has been enhanced to provide measures reporting for the three critical incident management measures: First Contact Resolved, Resolved On Time and Ticket Activity. These measures can be run on demand to show performance results based on a variety of filters and groupings.

  3. This application is directly based upon the Master Service Level Agreement (MSLA). All data is refreshed nightly. There are three major report types available via the MITS Web Reports Web Site - http://eues.web.irs.gov/ESD/Files/MITS%20Reports%20General%20Info.doc .

    The table below describes ITS Enterprise Reports in greater detail.

    ITS Enterprise Report Report Parameters
    Ticket Management Reports
    [Reports that can be run against open incidents to show the status of incidents based on numerous user selectable filters including, Service Area, Territory, Priority, Building, etc.]
    • HOT and Overage Tickets (open incidents that have exceeded will exceed the target resolution time within user-specified time in hours). Overage Tickets(open incidents that exceeded priority-based targeted resolution time) based upon the user specifying the number of day and hours that the incident has been in an overage status. Hot Tickets(open incidents that will exceed the priority-based target resolution time) based upon the user specifying the number of hours in increments of 4 from 4 hours to 168 hours.

    • Ticket Listing (all open incidents through the previous day).

    • Ticket Totals by Day - Summary of incidents opened by day by Service Provider (includes GET-IT total).

    Note:

    Since these reports are generated from the open incident inventory the results are potential overage and hot incidents. Additional analysis of the closed incidents are completed in the measures section of the reports sites against specific coding elements outlined in the MITS data dictionary located at http://mitsreports.web.irs.gov/userguide/datadictionary.doc

    Measure Reports
    [Reports are run against closed incidents to show performance results based on numerous user selectable filters and groupings (by priority, by month, by category, etc.)]
    • First Contact Resolved - Summary report showing incidents resolved at first contact, those that were not, and potential FCR. See data dictionary for complete definition.

    • Percent On Time - Summary report showing incident resolved on time, those that were not, and potential on time. See data dictionary for complete definition.

    • Ticket Activity - Summary report showing number of incident opened and number of incidents closed (see data dictionary for complete definition).

    MAC Plan Reports
    [Reports include Open Only and Closed Only. Reports are run against subcategory MAC-PLAN tickets and demonstrate level of effort and customers impacted as Service Providers work project tickets. These tickets are excluded from the measures reports.]
    • Mac-Plan - Open Only - Listing of Open MAC-PLAN master tickets with drill down to "child" tickets.

    • Mac-Plan - Closed Only - Listing of Closed MAC-PLAN master tickets with drill down to "child" tickets.

    • Mac-Plan All - Listing of Open and Closed MAC-PLAN master tickets with drill down to "child" tickets.

    Ad-Hoc Reports
    (http://mitsreports.web.irs.gov/adhoc.asp select Ad Hoc Reports from Links drop-down menu)
    Link to a variety of periodically updated reports, including percent on time charts.
    MITS Reporting Services
    (http://det0190cpitsrs1/MITSReports/Pages/Folder.aspx)
    Link to a variety of incident and asset management historical reports. Folders on the home page house special-purpose reports, most of which have been built to meet user defined requirements. Reports can be requested using the on-line request form at http://eues.web.irs.gov/tools/mitsrptreq.aspx
    Automated Call Distributor (ACD) Enterprise Reports
    [The ACD is used to manage calls to the 1-866-7help-4u toll free number for MITS ESD. This call routing system is used to classify calls and deliver the caller to the next available agent as prescribed by the call control tables maintained by the ESD QASC ACD staff. In the management of this system a number of reports are generated to evaluate the efficiency and effectiveness of the ESD call strategy. ]
    • Daily Application Summary - summarized call activity by application name and sorted by group.

    • Detailed Queued Calls - details number of calls offered by 1/2 hour and the % handled in X seconds (up to 120).

    • Detailed Abandoned Calls - details number of calls abandoned, average speed to answer by 1/2 hour.

    • Detail Call Profile - details number of calls offered and handled, average speed to answer, longest wait time, average wait time, and average handle time by 1/2 hour.

    Note:

    Reports are distributed by the Enterprise Service Desk, Quality Assurance and System Control.

    Automated Call Distribution Key Performance Indicator Reports
    [Automated Call Distribution reports summarizing performance measures]

    KPI Key Performance Indicators Reports
    • First contact resolution percentage (ESD and by Service Desk)

    • Web incident turn-around percentage (ESD)

    • Web incident turn-around average time (ESD)

    • Resolved on time percentage (ESD and by Service Desk)

    • Total web incidents processed (ESD)

    • Total calls handled (ESD)

    • Calls abandoned percentage (ESD)

    • Average speed of answer (ASA) for calls (ESD)

    • Total contacts handled

    Note:

    Reports are distributed by the Enterprise Service Desk, Quality Assurance and System Control.

    ITAMS REPORTS
    [In addition to the distributed ACD reports there are a number of reports cascaded regarded ITAMS incident management and status]
    Daily Priority 1 Incidents - Summarizes Priority 1 incident incidents opened within the last 24 hours, measured from midnight to midnight Eastern time. Information includes:
    • ITAMS incident number

    • Associated incidents/call incidents

    • ITAMS incident title

    • Incident site

    • Project and incident type

    • Open and close (if applicable) date and time

    • Current status

    • Assigned to assignment group

    • Resolution definition

    • Incident description and updates

    Daily Info Alerts - Summarizes information and incident alerts opened within the last 24 hours, measured from midnight to midnight Eastern time. Information includes:
    • ITAMS incident number

    • Associated incident/call incidents

    • ITAMS incident title

    • Open date and time

    • Current status

    • Assigned to assignment group

    • Incident description

    Note:

    Reports are distributed by the Enterprise Service Desk, Quality Assurance and System Control.

2.14.2.20  (03-17-2010)
Enterprise Self-Service Tools

  1. The MITS Enterprise Service Desk (ESD) support staff and its customers use a suite of self-service tools. These tools provide an alternative vehicle for requesting enterprise password resets, trouble ticketing and service request submissions via an intranet website portal. The tools also provide a vehicle to perform searches on a knowledge database to gain information on a certain topic.

  2. MITS has integrated a set of customer relationship management tools into the ITAMS Incident Management module. These tools consist of; OS GetServices, Password Management, and FindIT. They are available to the ESD Specialist as well as MITS end-users. The customer has the option of using the Incident Management Self Service tools or contacting the MITS Toll Free Number for incident reporting and/or service requests. The ESD Specialist will have access to the same suite of tools should the customer prefer assistance from the ESD. The end-user may choose to use the self-service tools as follows:

    • Report their own incident via the Web using OS GetServices

    • Submit a request for services using OS GetServices

    • Research and possibly resolve own incident by accessing FindIT via OS GetServices

    • Use self-service password management for resets and unlocks of LAN/Unix accounts.

    • Check status of incidents


    The tools are defined in the sections below.

2.14.2.20.1  (03-17-2010)
FindIT

  1. FindIT is an online application that contains short tutorials and information on information technology topics, from information about logins and passwords to how to communicate with the IRS network through remote communication software.

2.14.2.20.2  (03-17-2010)
Support Staff Tools

  1. Support Staff Tools consists of the following: Password Management: Support Staff Mode and Password Management Web Administration (Web AD).

2.14.2.20.2.1  (03-17-2010)
Password Management: Support Staff Mode

  1. Password Management is a web based tool that not only allows customers to self-service password but provides functionality for ESD personnel to reset/unlock passwords in the IRS network. This utility is called Support Staff Mode and allows an ESD Specialist to reset/unlock passwords. This tool also uses a product called Enable User Utility and is used to enable and disable an end users ability to self-reset/unlock a password. End-users have access to a self-service reset tool via an IRS networked PC using Identity Management Suite DIRECT!.

  2. ESD Specialists will use the support staff mode to support their customers with the tool. There are two utilities, Support Staff Mode that performs the reset/unlock and Enable User Utility which is used to control the ability of customers to use the self-service function to reset/unlock their own password.

  3. The OS GetServices Password Management, Support Staff Mode module will be used by the ESD Specialists to reset/unlock passwords for various platforms that are listed in the tool. The ESD Specialists will be required to validate and authenticate the user by asking a series of questions before proceeding with a password reset/unlock.

2.14.2.20.2.2  (03-17-2010)
Password Management Web Administration (Web AD)

  1. This tool is the primarily back-up tool in the event Password Management Support Staff Mode is unavailable. The only exception is servicing of student accounts, and the reset/unlock of Administrative Accounts using Web AD.

    Note:

    Web AD is the primary support tool used by the ESD Specialist to support enterprise account administration (i.e. establishing new passwords, modifying and reinstating accounts).

  2. Once the Specialist has reset the customer's password using PWM Support Mode or Web Administration (Web AD), the customer will be forced to reset their password. Unlocks will not require the resetting of a password. In addition the Specialist will not have to manually open and close a call incident for a reset/unlock as it is systemically generated.

2.14.2.20.3  (03-17-2010)
Desktop Support Tools

  1. Desktop Support Tools

2.14.2.20.3.1  (03-17-2010)
Tivoli Remote Control

  1. Tivoli Remote Control is the IRS standard software that enables MITS EUES support personnel to remotely service an end user without requiring them to physically go to the location of the computer. It allows support personnel to view and control the end users workstation once the end user grants permission.

    • ESD Specialist will utilize this function to perform initial triage and attempt First Call Resolution for desktop incidents.

    • Field IT Specialists will utilize this function for efficient troubleshooting and resolution of desktop incidents.

2.14.2.21  (03-17-2010)
OS GetServices

  1. The OS GetServices suite of tools is web based, and is accessed using an IRS Internet browser. The OS GetServices portal provides an entry point to request services from the Employee Resource Center (ERC) as well as MITS products and services. This IRM will outline the MITS standards only. The web incident tool will be launched from the portal at http://getservices.web.irs.gov.
    From the OS GetServices page, customers have the capability to:

    • Open a Fix-IT, Get-IT or Move-IT incident for himself or herself or a customer under My Technology selection.

    • Open, update, and close a Get-IT and Move-IT incident that has been reported by them.

    • Check the Status of any incident submitted to MITS or ERC.

    • Access the tutorials regarding the tools online.

    • Access frequently used IRS intranet sites.

    • Learn what to do and where to go to resolve password issues for IRS systems.

    • Provide feedback.

    Note:

    The Enterprise Service Desk (ESD) will close all Web incidents opened by Customers/Service Providers where they are requesting the incident to be upgraded to a Priority 1 or Priority 2.


    The Master Service Level Agreement and the OS GetServices Web Site indicates critical outages are to be reported via the MITS Toll Free Number. If a Priority 1 or Priority 2 is submitted the following actions should be taken:

    The Web incident will be closed and coded as follows:

    • Cause Code: CUST ERR

    • Resolution Code: Not Resolved

    • Incident Resolution: Your incident has been closed - The Master Service Level Agreement states all Priority 1 and Priority 2 incidents will be reported via the MITS Toll Free Number. Your new incident number is #########.

    • Closed web incident will be linked and Cross Referenced into New ITAMS incident.

    • The Incident Start Time will reflect the time of when the ESD retrieved the incident from the Web Queue and cross referenced.

    • The ESD will contact the customer and inform them of the new incident number.

    • The ESD will open a P1 or P2 incident in ITAMS and assign to the appropriate Service Provider.

    • If the request from the customer was to prioritize the incident as a P1 and the scope and impact was not defined, the ESD will request the Service Provider to complete the Priority 1 Confirmation Template (Unscheduled Outage).

    • If the incident is not name assigned or the Priority 1 Confirmation Template (Unscheduled Outage) is not completed within 30 minutes, the ESD will begin the Incident Management Escalation process.

2.14.2.21.1  (03-17-2010)
FIX-IT: My Technology

  1. The Fix-IT tool allows the customer to report incidents, via the intranet, directly to the ESD without dialing the toll free number. A Fix-IT selection should be used for incidents encountered with existing hardware and software. The customer can open an incident for themselves or a co-worker. Upon successful submission of the web incident, the customer will receive a incident number and recommended timeframe in which the incident should be resolved as well as the toll free number to call if they have additional questions. A confirmation e-mail is also sent to the customer.

    Note:

    The customer is instructed to call the toll free number if the incident being reported is a Severe Critical Work Stoppage, where immediate action is required, impacting multiple users, service to taxpayers, or issues impacting Safety or Health (i.e. shock from equipment).

  2. The incident is assigned based on the building code of the customer as well as the category / subcategory selections made by the customer during the creation of the OS GetServices incident. The incident will then be assigned to the associated Service Desk as an incident for resolution or assignment to a Service Provider:

    • ESD Specialists will use ITAMS Service Center (SC) to record all incidents and will not use the Fix-IT option in OS GetServices.

    • MITS Service Providers who receive opened Web Fix-IT incidents are required to validate coding and customer information during initial triage, including utilizing Probe and Response.

    • All MITS Service Providers are required to update and close incidents within ITAMS SC, not through OS GetServices.

    • All submitted Fix-IT incidents have default coding based on selection criteria. Coding can be viewed from the ITAMS resource page available under the User Utilities Tab in ITAMS SC.

2.14.2.21.2  (03-17-2010)
GET- IT: My Technology

  1. Get-IT is used to request NEW IT products and services. Get-IT MUST be used by the ESD Specialist when requesting IT products and services on behalf of a customer. This will ensure the generation of the request for managerial approval. (Specialists should NOT open tickets using ITAMS SC.)

    Note:

    GET-IT should not be used for emergency requests from IRS executives for standard IT equipment. These requests are discussed in section 2.14.2.22, Emergency Equipment Provisioning.

  2. The customer's manager MUST approve all requests. The employee making the request will designate a manager to approve. An e-mail will be generated to the users manager for approval. Upon successful submission of the request, an ITAMS SC ticket will be opened and the ticket number will be displayed on the web page for the caller/requester. A confirmation e-mail is also sent to the customer.

  3. The ticket will remain in the status code "Open-Approval Requested" until the customer's manager has approved or disapproved the request from the system generated e-mail. No MITS Service Provider should take any action on a ticket until APPROVED by the customer's manager which changes the status code to OPEN. The ticket status should only be changed as a result of the manager's response of approval or disapproval. Any requests not acted upon by the designated manager within sixteen (16) days will be systemically closed and the customer will be notified by e-mail. Tickets closed through this process will be coded with resolution code "No Approval" and cannot be re-opened. During the 16-day period an additional e-mail will be sent as a reminder to the manager if not approved / disapproved.

    Note:

    GET-IT tickets cannot be used to circumvent the UWR process requirements.

  4. Once the OS GetServices web page is launched the customer will select My Technology and follow the steps to submit the request based on their selection.
    Customer will select the Get-IT option for the product they are requesting:

    • Get New Hardware/Printers

    • Get New Upgrade Software

    • Get New Telephone/Faxes

      Note:

      Any request involving small equipment will be routed through the IT staff for a review of the EUES employee profile database. If the employee has not been designated by the BOD for the non-standard piece of equipment, additional approvals will be needed through the BSP and/or BOD designee before request can be fulfilled.

2.14.2.21.3  (03-17-2010)
MOVE-IT/Change Management

  1. Change Management is an ITAMS SC module that allows more complex work to be performed under one "umbrella" - the Change Request. The module allows the work to be analyzed, approved/disapproved, tasked out to multiple Service Providers, and so on.

  2. In order to move people and equipment from point A to point B, there are a number of MITS and AWSS support personnel involved. A move request starts with the customer opening a Move-IT ticket and ends after all of the MITS and AWSS work is completed.

  3. REFM (AWSS) uses the Incident Management module to record their activities in moving the customer. MITS will use the Change Management module to complete its portion of the customer's move request. These two modules are integrated along with OS GetServices to support the move business process.

  4. All move requests begin with the customer opening a Move-IT ticket. Through a"screening process" the customer defines the scope of the move request. Depending on the answers to the questions, a specific type of Change Request is opened from the ticket. The type of Change Request that is opened depends on the REFM subcategory.

    • PHYSICAL MOVE Change Request - consists of 5 phases: Analysis, Approval, Task Creation, Implementation, and Review.

    • MOVE CONSULTATION Change Request - consists of 2 phases: Estimate and Consult Review. A SPACE CONSULTATION (specific REFM subcategory used to handle large-scale move requests that often have funding and/or space implications) ticket will only generate a MOVE CONSULTATION Change Request.

  5. REFM (AWSS) is the move process owner and receives the customer’s move request ticket. REFM is responsible for evaluating the customer’s request, involving MITS by opening a Change Request, coordinating with the customer, and closing out with the customer.

    Note:

    Exceptions are Phone/VMS or Computer Equipment Only moves with no people moving and no space implications – these customer requests are routed directly to MITS.)

  6. MITS is responsible for completing its portion of the work required to complete a Physical Move request or to complete an estimate for a Move Consultation request.

  7. A Change Request is created by REFM from the original incident ticket generated by OS GetServices and contains the information needed for MITS to work its portion of the Move request. The Change Request will begin with a "C" followed by a number (i.e. C61). The EUES Change Coordinator will access the Change Request records and then create Tasks for the Service Providers.

  8. Tasks created by Change Coordinators to assign the work to Level 2 Service Providers. A Task will have a " T" in front of it, followed by a number (i.e. T70). There may be multiple tasks for one Change request. Once the Service Provider has completed their task, the Change Coordinator can close the Change Request, and the information will be posted on the original incident ticket. Note: Any ITAMS SC user can query for Change Requests or Tasks.

  9. Change Coordinators are generally Area and Territory EUES employees who have been involved in some capacity with completing move requests. The primary role of the Change Coordinator is to coordinate the MITS portion of a move request by working with the customer and REFM. Change Coordinators:

    • Analyze the work to be performed (Physical Move - Analysis Phase).

    • Involve MITS management as necessary.

    • Approve/disapprove the Change Request (Physical Move - Approval Phase).

    • Create tasks to be worked by service providers (Physical Move - Task Creation Phase).

    • Coordinate with the customer, MITS service providers, and non-MITS service providers (e.g. Real Estate/Facilities Management - REFM).

    • Follow up to ensure work is accomplished (Physical Move - Implementation Phase).

    • Review the completed work and close the Change Request (Physical Move - Review Phase).

    • Coordinate estimates for large-scale moves (Move Consultation - Estimate Phase).

    • Consolidate and provide estimates to REFM for proposed moves (Move Consultation - Estimate Review Phase).

  10. Service Providers are generally Desktop, Telecom, IEG, and other MITS employees who work ITAMS incident tickets. Service Providers will monitor the Task queue for their assignment group and complete the Tasks assigned by the move coordinator. This includes documenting inventory updates when equipment is moved.

  11. ESD is not involved in the Change Request process.

  12. For more information go to: http://eues.web.irs.gov/ESD/ChangeMgmt.aspx. Information on Change Management/Move Requests is also available in ITAMS SC under User Utilities → Guides/Documentation.

2.14.2.21.4  (03-17-2010)
Request Status / Approvals

  1. The "My Request Status" tab allows the customer to check the status and view any incident by number whether it is active or closed (less than 13 months old). It also allows the customer to close or update an incident with new information or attachments. When a customer closes or updates an incident via OS GetServices, an e-mail is sent to the assignment group members notifying them that an update/close has occurred.

  2. Once the OS GetServices web page is launched the customer will use the "My Request Status" tab located at the top of the page. Additional queries can be entered using the left hand navigation options.

  3. All update comments made by Service Providers in ITAMS SC that have been marked"Visible to Customer" will be appended to the customer's description and can be seen by the customer in OS GetServices.

  4. A list of incident numbers that are waiting approval can be opened by selecting the "My Approvals" tab from the left hand navigation options.

2.14.2.21.5  (03-17-2010)
Self Service Password Management

  1. OS GetServices Password Management provides the customer with the capability to securely reset and synchronize their password using Identity Management Suite Direct!. The customer will be required to build a user profile for themselves prior to using this application for the first time. This profile is created by using the OS GetServices portal and selecting the "Password Management" tab at the top of the screen. The customer should select "Profile Builder" and follow the prompts for building their profile. Additional information can be found by selecting the FAQ option.

  2. Once the customer is profiled, the reset/unlock program can be accessed via the Gold Key on the Task Bar (Identity Management Suite DIRECT!) or by selecting Cancel at the initial LAN login screen on any network PC. The customer will be prompted to answer questions that only they know for validation and authentication for access each time they initiate a password reset/unlock. If a customer has built their profile and calls the ESD for a reset/unlock, they will need to provide one or two questions to the ESD Specialist. No MITS personnel will have access to all of the profile data of the customers.

  3. This tool is integrated with ITAMS SC Incident Management and will generate an incident for each successful self-service password reset/unlock and automatically closes the incident upon a successful password reset. If a customer encounters a problem resetting a password after profile validation an ITAMS SC incident will be generated to the ESD.

    • Effective August 27, 2004, the Chief Information Officer mandated all employees use the Password Management self service tool for all resets and unlocks of LAN passwords. This policy was shared with all Business Organization Division DIO and BSP staffs before implementation. Currently a few technological barriers exist that requires some employees to be excluded. For a summary of the policy and exclusions please reference the CIO Policy memo and other relevant documents located at: http://eues.web.irs.gov/ESD/PWM.aspx .

    • Effective July 18, 2006, the CIO rescinded the requirements for managers to authorize ESD password reset/unlocks for employees who can use the PWM rest/unlock (not-excluded in the initial policy). This policy will be re-implemented if compliance rates fall under 85%. See the policy memo located at: http://eues.web.irs.gov/ESD/Files/PWM/password%20unlock%20reset%20memo.pdf .

2.14.2.22  (03-17-2010)
Emergency Equipment Provisioning

  1. (1) Requests for new IT products fall into two categories: Emergency Equipment Provisioning and Standard Provisioning requests. Section 2.14.2.21.2 discusses procedures for using GET-IT for making Standard Provisioning requests.

  2. Emergency Provisioning is an expedited service to accommodate emergency requests from IRS executives for standard IT equipment. Upon receipt of a request, MITS will provide standard IT equipment within one day when the receiving individual is co-located with a depot, or within three days when the equipment requires shipping.

    Standard IT Equipment:

    • Laptops

    • Desktops

    • Printers (desktop, portable, network, color network)

    • Flat Panel Monitors

    • Scanners

    • Fax Machines

    • Blackberry Devices

    • Cell Phones

    • Air Cards

    • Desktop Phone Sets

  3. Requests under the Emergency Equipment Provision must be submitted or approved by an Executive who declares an urgent need for the new equipment.

  4. Only one request is needed even if the individual requires several pieces of standard equipment.

    Note:

    This is not a substitute for expediting an existing repair or break/fix replacement incident.

  5. Executives follow these steps when requesting "Emergency Provisioning" service:

    1. Call the Enterprise Service Desk (ESD) at 1-866-743-5748.

    2. Identify themselves as IRS executives.

    3. State that they are requesting "Emergency Provisioning. "

  6. ESD will initiate a Priority 2 (P2) incident for all " Emergency Provisioning" requests. The Specialist will ask the executive the reason for the emergency so it can be documented within the incident for historical purposes.

  7. In addition to following normal P2 procedures and assignment of incidents, the Specialist will immediately notify their direct manager of the opened P2.

  8. Additional detailed information on Emergency Provisioning can be found at the following URL: http://eues.web.irs.gov/ESD/ITEquipmentProvisioning.aspx.

Exhibit 2.14.2-1 
Valid Sub-Categories by Category Type

A listing of current SubCategories by Category type is available through the URL: http://det0190cpitsrs1/MITSReports/Pages/Resource.aspx?ItemPath=%2fITAMS+Supporting+Info%2fCategory+SubCategory+Listing.xls .

Exhibit 2.14.2-2 
Priority Code Table

Note:

Priority Code table reflects response levels in the 2/26/09 draft MSLA http://mits.web.irs.gov/becs/BECS%20MSLA/index.htm .

Priority Definition Response Levels












1
Severe Mission Critical Work Stoppage or ANY issue relating to safety or health (fire, electrical shock etc). Impact on vital IRS Customer Commitments of National or area-wide scope, affecting MULTIPLE internal or external customers and impacting service to taxpayers. Immediate action required .
The following list is not all inclusive:
Real-time Unavailable:
CFOL
IDRS
ICS
ACS
Critical Tax Processing System Examples:
IDRS
CADE
CFOL
E-SERVICES
IRFOF
ISRP
MODERNIZED E FILE
SCRIPS
ACS
RICS
ICS
ELF
Critical Administrative Systems Communication
Internet Access to Modernization Applications
Voice/Data Outage
Health/Safety or Environmental Disasters
Unscheduled Power Outage
ITAMS Service Center Unavailable
ACD Outage
E-mail
VPN/ERAP - Central Site Connectivity Outage
VPN/ERAP - Central Site Tunnel Outage
Assignment of Incident:
No Later than 30 minutes
Updated through Incident Resolution:
Hourly
Target Resolution Time:
Within 4 hours
2 Potential work Stoppage. Could have a direct impact on the service to taxpayers or if its scope is multi-user and there is no work-around. Could lead to severe mission critical work stoppage if actions are not taken to resolve incident.
The following list is not all inclusive:
National Standard Applications down
Network device unavailable affecting multiple customers
Services/traffic on redundant equipment; hardware, communication equipment
VPN/ERAP - Fixed Site Connectivity Outage
VPN/ERAP - Fixed Site Tunnel Outage
Assignment of Incident:
No Later than 1 hour Updated through Incident Resolution:
No later than 2 hours
Target Resolution Time:
Within 8 hours
3 Work stoppage for ONE customer with NO work around.
The following list is not all inclusive:
Password resets/unlocks
Printer Incidents
Desktop/Laptop
VPN/ERAP - service disruption for single customer resulting in work stoppage
Assignment of Incident:
No Later than 1 hour
Updated through Incident Resolution:
No later than 4 hours
Target Resolution Time:
Within 2 working business days
4 A. Non-Critical, non-software incidents where it is not a work stoppage and there is a workaround.
Sample workaround scenarios:
Capability to route prints to an alternate printer
Capability to use an alternate Desktop/Laptop
Assignment of Incident:
No Later than 2 hours
Updated through Incident Resolution:
No later than 3 working days
Target Resolution Time:
Within 4 working days
 
B. Non-Critical software incidents where it is not a work stoppage and there is a workaround.
Assignment of Incident:
No Later than 2 hours
Updated through Incident Resolution:
No later than 3 working days
Target Resolution Time:
Upon agreement between customer and Service Provider
5 Requests for non-production related products and services
Example:
New software requests
Assignment of Incident:
No Later than 2 hours
Updated through Incident Resolution:
No later than 5 working days
Target Resolution Time:
Within 20 working days

Exhibit 2.14.2-3 
Service Request Priority Definitions

Priority Definition Response Levels
R1 Immediate response and action required. Assigned to all requests that severely impact Services provided (e.g., Notice hold Requests). Response Required:
Within 1-24 hours
R2 EOPS Support is required. Acknowledgement of Receipt:
Within 1-3 days
Status Updates:
Within 1-3 days
R3 Request can be resolved by local IT staff without EOPS support and may severely impact services. Acknowledgement of Receipt:
Within 1-24 hours
Status Updates:
Within 1-24 hours
R4 Request can be resolved by local IT staff without EOPS support. Acknowledgement of Receipt:
Within 1-3 days
Status Updates:
Within 1-3 days
R5 Request will be addressed as time and resources allow. There are no response levels.

Exhibit 2.14.2-4 
Alert Stages

ITAMS SYSTEMIC ALERT NOTIFICATION PROCESS

Incident Escalation/Alerts are configured using the Open time for aging the incident and deadline alert stage represents target resolution based on priority definitions. The calendar function for the assignment group is utilized when calculating alert timeframes.

In the event an incident is upgraded to a Priority 1 or a Priority 2, alerts will be calculated based on the open time of the incident.

Priority 1: Incident Assigned Timely
Example: Incident opened and assigned at 9:00 a.m.

ALERT STAGE 1 1 HOUR Issued at 10:00 a.m.
ALERT STAGE 2 2 HOUR Issued at 11:00 a.m.
ALERT STAGE 3 3 HOUR Issued at Noon
DEADLINE 4 HOUR Issued at 1:00 p.m. = Target Resolution

Priority 1: Incident Not Assigned to Assignee
Example: Incident opened at 9:00 a.m. and a Service Provider did not complete assignee field.

ALERT STAGE 1 30 MIN Issued at 9:30 a.m.
ALERT STAGE 2 1 HOUR Issued at 10:00 a.m.
ALERT STAGE 3 1.5 HOUR Issued at 10:30 a.m.
DEADLINE 4 HOUR Issued at 1:00 p.m. = Target Resolution

Priority 1: Incident Not Assigned to Assignee Timely
Example: Incident opened at 9:00 a.m., assigned to assignee at 9:45

ALERT STAGE 1 30 MIN Issued at 9:30 a.m.
ALERT STAGE 2 2 HOUR Issued at 11:00 a.m.
ALERT STAGE 3 3 HOUR Issued at Noon
DEADLINE 4 HOUR Issued at 1:00 p.m. = Target Resolution

Priority 2: Incident Assigned Timely
Example: Incident opened and assigned at 9:00 a.m.

ALERT STAGE 1 2 HOUR Issued at 11:00 a.m.
ALERT STAGE 2 4 HOUR Issued at 1:00 p.m.
ALERT STAGE 3 6 HOUR Issued at 3:00 p.m.
DEADLINE 8 HOUR Issued at 5:00 p.m. = Target Resolution

Priority 2: Incident Not Assigned to Assignee
Example: Incident opened at 9:00 a.m. and a Service Provider did not complete assignee field.

ALERT STAGE 1 1 HOUR Issued at 10:00 a.m.
ALERT STAGE 2 1.5 HOUR Issued at 10:30 a.m.
ALERT STAGE 3 2 HOUR Issued at 11:00 a.m.
DEADLINE 8 HOUR Issued at 5:00 p.m. = Target Resolution

Priority 2: Incident Not Assigned to Assignee Timely
Example: Incident opened 9:00 a.m., assigned to assignee at 10:15 a.m.

ALERT STAGE 1 1 HOUR Issued at 10:00 a.m.
ALERT STAGE 2 4 HOUR Issued at 1:00 p.m.
ALERT STAGE 3 6 HOUR Issued at 3:00 p.m.
DEADLINE 8 HOUR Issued at 5:00 p.m. = Target Resolution

Enterprise Operations

ALERT STAGE 1 Service Provider Section Manager
ALERT STAGE 2 Service Provider Section Manager/Branch Chief
ALERT STAGE 3 Service Provider Team Leader/Section Manager/Branch Chief/Division
DEADLINE Service Provider Team Leader/Section Manager/Branch Chief/Division/Director

EUES

ALERT STAGE 1 Service Provider Assignee/Section Manager
ALERT STAGE 2 Service Provider Assignee/Selection Manager/Ops manager/Territory Manager
ALERT STAGE 3 Service Provider Assignee/Selection Manager/Territory Manager/ Area Director
DEADLINE Area Director/Director - EUES

Exhibit 2.14.2-5 
Definitions and Descriptions

The table below displays key definitions and descriptions.

Term Definition
Assignee Service Provider Group or Individual assigned to the incident and responsible for incident resolution.
Assignment Service Provider group where the incident is assigned. The assignment groups names must be based on the organizational structure of the MITS Service Provider (ex No project name should be used for assignment group).
Call Incident Using the service management option, a call incident will be opened for every incident or request reported to the ESD. If the ESD Specialist resolves the incident while the customer is on the phone, the call incident will be closed. The call incident number will not be provided to the customer since first call resolution has occurred.

If it is required to perform a conference transfer to another Service Desk while in the process of opening a call incident, the call incident number will be provided to the Service Desk to which the call is being transferred.
Cause Code Code used to identify the cause of an incident.
Enterprise Service Desk (ESD) Responsible for receiving incident reports, defining the incident category, and determining the priority for all incident reports received, and overseeing the resolution process.
Customer An IRS employee/contractor who utilizes Information Systems services.
Incident If the ESD Specialist is unable to perform first call resolution an incident will be created using the incident management option. The incident will be associated to the call incident and assigned to a Service Provider for incident resolution. The customer will always be provided the incident number for future reference. Incidents are also created for all incidents created by the OS GetServices web site. These incidents have no call incident.
Resolution Code Code used to identify how an incident was resolved.
Service Provider
(a.k.a Assignment Group)
A Service Provider is defined as an MITS associate assigned to provide services, support, and incident resolution. ESD Specialists are also Service Providers for initial call resolutions. A Service Provider always has the minimum of level two privileges in ITAMS and may have level one based on management discretion.
Share Assignee Service Provider Group or individual assigned to the shared incident and who may be responsible for incident resolution and/or customer communication.
Share Assignment The Service Provider Group where the incident is share assigned and responsibilities for incident resolution and/or customer communication resides with more than one Service Provider group.

Exhibit 2.14.2-6 
ACD/Uniphi Connect Outage

Procedures identified below address actions to be taken in the event of identified failures. Four failure scenarios are addressed. Each scenario addresses one of the four possible points of failure (i.e., ACD failure, WAN/LAN failure, VPS failure, and telephony circuit failure). Also addressed below is call center/service desk evacuation. These procedures apply to users and business units utilizing the Aspect Call Center Automated Call Distributor (ACDs) currently administered by the Enterprise Service Desk (ESD) Quality Assurance and Systems Control (QASC) ACD Team and access through 1-866-7HELP4U (1-866-743-5748).

Failure Scenario: Aspect Call Center (ACD) Unavailable at Either Memphis or Fresno

Impact: A failure of the Aspect Call Center at either Memphis or Fresno will render that Aspect Call Center incapable of routing either inbound or outbound calls consistent with call control tables.

Contingency Plan: Memphis Aspect Call Center (ACD) not available for routing or handling calls

The Memphis Aspect Call Center is the primary Automated Call Distributor (ACD) for routing Employee Resource Center (ERC) calls to ERC Customer Service Representatives (CSRs) located at the Memphis ERC. The Memphis ACD also serves as the backup ACD for call routing to ESD services. The Memphis Aspect Call Center is a redundant ACD consisting of a primary and backup server.

The system is designed so that if either the primary or backup server fails, the Aspect Call Center will automatically switch to the operational server. Should power be lost to the ACD, the system will revert to backup power. Backup power should enable the system to operate for approximately 113 minutes on backup battery power. A total failure of the Memphis Aspect Call Center will necessitate ERC CSRs signing onto the Fresno Aspect Call Center via Uniphi Connect. ERC CSRs will take the following steps to sign on as remote Uniphi Connect users to the Fresno Aspect Call Center:

  1. Exit out of the Uniphi Connect client application (i.e., sign off and close the application).

  2. Re-open the Uniphi Connect client application.

  3. Select the Configure command button.

  4. The Configure Aspect Uniphi Connect dialog box will appear. Select the Network Settings tab and in the Callback Information window, enter the CSR's administrative telephone number (area code and telephone number).

  5. Within the Network Settings tab of the Configure Aspect Uniphi Connect dialog box and in the ACD Name window, if available, select the Fresno server number and click on the OK button. The Fresno server number is for the Fresno CallCenter ACD. If unavailable, select "Add" and a Configure Options dialog box will appear. In the Configure Options dialog box, make the following entries:

    1. Under ACD System Name enter the Fresno ACD number designation (supplied by a supervisor).

    2. Enter the Primary IP Address for the Fresno ACD system (supplied by supervisor).

    3. Check the box labeled "Redundant."

    4. Enter the Backup IP Address for the Fresno ACD system (supplied by supervisor).

    5. Leave both the Primary and Backup Port Number as the default and select the "OK " button.

  6. Back at the Network Settings tab ensure that the Fresno ACD number designation appears in the ACD Name window. Then enter the 10 digit call back number in the "Phone Number" field. In the "Instrument ID" field enter "0" and then enter the appropriate Virtual Instrument Group Number.

    Note:

    The virtual Instrument Group Number will be the same as the team number.

  7. After all above information has been entered, select the OK button to return to the Sign-on screen.

  8. At the Sign-on dialog box, log-on to the Fresno Aspect CallCenter ACD using the same extension number and password as for the Memphis Aspect CallCenter ACD, assuming that the CSR has not changed his/her password as assigned.

    Note:

    Changing a password on one Aspect Call Center does not change the password on the Aspect Call Center designed to provide disaster recovery call routing. The two systems are independent. It is recommend that upon changing a password that the CSR ensures that his/her password is changed on both Aspect Call Center ACD systems.

The CSR should connect to the Fresno Aspect CallCenter ACD. If incidents are experienced logging on, immediately notify the ESD QASC ACD team at 866-716-6702.

Since ERC disaster recovery sign-on capability is approximately fifty percent of normal business operations, it is anticipated that not all ERC CSRs will be able to sign onto the Fresno Aspect CallCenter ACD. ERC management, based upon available staffing at the time of the failure, will identify those ERC CSRs who should sign onto the Fresno Aspect CallCenter. The QASC ACD team will advise ERC management of unused and available ESD ACD/Uniphi Connect resources that may be used by ERC remote users on a temporary basis. ERC management personnel are responsible for ensuring that CSR accesses (additions, deletions and modifications) are communicated as they occur to the QASC ACD team; this ensure that CSR access to the Fresno Aspect CallCenter is available at the onset on the emergency.

Contingency Plan: Fresno Aspect CallCenter (ACD) not available for routing or handling calls

The Fresno Aspect Call Center is the primary Automated Call Distributor (ACD) for routing ESD calls to ESD Specialists. The Fresno ACD also serves as the backup ACD for call routing to ERC services. The Fresno Aspect CallCenter is a redundant ACD consisting of a primary and backup server.

The system is designed so that if either the primary or backup server fails, the Aspect CallCenter will automatically switch to the operational server. Should power be lost to the ACD, the system will go to backup power. Backup power should enable the system to operate for approximately 113 minutes on backup battery power. A total failure of the Fresno Aspect CallCenter will necessitate ESD Specialists signing on to the Memphis Aspect CallCenter via Uniphi Connect. ESD Specialists will take the following steps to sign on as remote Uniphi Connect users to the Memphis Aspect CallCenter:

  1. Exit out of the Uniphi Connect client application (i.e., sign off and close the application).

  2. Re-open the Uniphi Connect client application.

  3. Select the Configure command button.

  4. The Configure Aspect Uniphi Connect dialog box will appear. Select the Network Settings tab and in the Callback Information window, enter the Specialist's administrative telephone number (area code and telephone number).

  5. Within the Network Settings tab of the Configure Aspect Uniphi Connect dialog box and in ACD Name window, if available, select the Memphis server number and click on the OK button. The Memphis server number is for the Memphis CallCenter ACD. If unavailable, select "Add" and a Configure Server dialog box will appear. In the Configure Server dialog box, make the following entries:

    1. Under ACD System Name enter the Memphis ACD number designation (supplied by a supervisor).

    2. Enter the Primary IP Address for the Memphis ACD system (supplied by supervisor).

    3. Check the box labeled "Redundant" .

    4. Enter the Backup IP Address for the Memphis ACD server (supplied by supervisor).

    5. Leave both the Primary and Backup Port Number as the default and select the "OK" button.

  6. Back at the Network Settings tab ensure that the Memphis ACD number designation appears in the ACD Name window. Then enter the 10 digit call back number in the "Phone Number" field. In the " Instrument ID" field enter "0" and then enter the appropriate Virtual Instrument Group Number.

    Note:

    The virtual Instrument Group Number will be the same as the team number except for AG 346 (TTY) extensions in which case the Instrument number to be entered will be supplied by a supervisor.

  7. After all above information has been entered, select the OK button to return to the Sign-on screen.

  8. At the Sign-on dialog box, log-on to the Memphis Aspect CallCenter ACD using the same extension number and password as for the Fresno Aspect CallCenter ACD, assuming that the specialist has not changed his/her password as assigned by the ESD QASC ACD team.

    Note:

    Changing a password on one Aspect Call Center does not change the password on the Aspect CallCenter designed to provide disaster recovery call routing. The two systems are independent. It is recommend that upon changing a password that the Specialist ensures that his/her password is changed on both Aspect CallCenter ACD systems.

Specialist should connect to the Memphis Aspect CallCenter. If incidents are experienced logging on, immediately notify the QASC ACD team at 866-716-6702.

Since disaster recovery sign-on capability is approximately fifty percent that of normal business operations, it is anticipated that not all ESD Specialists will be able to sign onto the Memphis Aspect CallCenter. Based upon available staffing at the time of the failure, ESD managers will identify specialists to sign onto the Memphis Aspect CallCenter. The number of specialists designated to sign-on to the Memphis Aspect CallCenter will not initially exceed fifty percent of the ESD's authorized staffing. Priority should be given to those ESD Specialists that will be signing on and consistently handling inbound calls. The ESD QASC ACD team will allocate additional sign-ons based upon call volumes at the time of the failure and availability of remaining resources. ESD managers are responsible for ensuring that Specialist accesses (additions, deletions and modifications) are communicated as they occur to the QASC ACD team. The QASC ACD team will ensure that extensions (i.e., accounts) are created on both the Fresno ACD and the Memphis ACD.

Notification: A suspected failure of either the Memphis or Fresno Aspect CallCenter will be reported immediately, in as much detail as possible, to the QASC ACD team employing whatever means available (e.g., voice notification is preferred). Employees discovering the failure will also ensure that the incident is reported to the ESD. The ESD will open a Priority 1 ITAMS incident assigning it to the ESD QASC ACD team. Upon notification, the QASC ACD team will:

  1. Confirm the failure of the Aspect CallCenter.

  2. Upon confirmation, advise applicable management personnel to implement back-up sign-on procedures as identified above. Preferred means of notification is voice.

  3. Immediately notify the SPRINT System Management Center (SMC) at 1-877-387-3667 to implement the appropriate contingency route set. The pre-planned route set will redirect all inbound calls from the initial SPRINT VRU (voice response unit) to the operational Aspect CallCenter. The re-configuration to a pre-planned route set will take from five to fifteen minutes. The SMC maintains a list of personnel authorized to change command route sets; this list will be maintained by the ESD QASC ACD Team Lead.

  4. Immediately notify the Aspect Customer Support Center at 1-800-541-7799, Option 1, of the failure and ensure that an Aspect technician is dispatched to the downed Aspect CallCenter, and obtain the Aspect incident number. As time permits, advise the Joint Operational Center (JOC) at (678) 530-5014, Option 3, of the incident and that a incident has been opened with the Aspect Customer Support Center.

  5. Monitor back-up sign-on activities and advise applicable management personnel regarding availability of additional sign-ons.

  6. Update applicable management personnel regarding the status of recovery operations. Update the ITAMS incident in accordance with Enterprise Incident Management (CIM) Guidelines.

  7. Upon restoration of the Aspect CallCenter, advise applicable management personnel to return to normal business operations and close the ITAMS incident. Monitor return to normal operations.

  1. Failure Scenario: WAN/LAN Failure Affecting Memphis or Fresno Uniphi Connect Users

Impact: Uniphi Connect utilizes data and voice communications to deliver calls to available remote Specialists/CSRs. The failure of a WAN/LAN component (e.g., border router), depending upon the point of failure, may adversely affect remote Uniphi Connect connectivity at one or more call sites. Remote Specialists/CSRs, unable to connect to the supporting ACD, will not be able to register their call state on the appropriate Aspect CallCenter and hence, calls will not be delivered.

Contingency Plan: A failure of a WAN/LAN component affecting one or more call sites may necessitate programming the Aspect Call Center to function as a telephone exchange and dial a "virtual " telephone number (or telephone numbers if multiple call sites are experiencing the failure) according to a pre-programmed plan. For this reason, each call site should coordinate with local telecommunications personnel and set-up a "virtual" telephone for the call site. In most case, the "virtual" telephone number is referred to as a hunt group and is generally configured on the supporting PBX. The number should be so configured that when dialed, it rings on every extension identified in the hunt group. All call site Specialist/CSRs extensions should be associated with this single, direct, dialable hunt group or "virtual" telephone number. The "virtual" number should also be configured to go to voice-mail in the event that an Specialist/CSR is unavailable to take the call. ERC and ESD management personnel are responsible for communicating their locally assigned "virtual" telephone numbers (to include area code) to the ESD QASC ACD team. Emergency call control tables will be pre-programmed to route calls to identified "virtual " telephone numbers. Because the Aspect CallCenter is dialing out and then releasing the call, only selected call performance measures (e.g., numbers of calls, transfer time, etc.) will be available.

Notification: A suspected failure of a WAN/LAN component affecting one or more call sites will be reported immediately to the ESD QASC ACD team employing whatever means available (e.g., voice notification is preferred). Employees discovering the failure will also ensure that the incident is reported to the ESD. The ESD will open an ITAMS incident (priority dependent upon the effected user population) assigning it to the appropriate WAN/LAN assignment group. Upon notification, the ESD QASC ACD team will:

  1. Confirm the WAN/LAN component failure and determine the impact on the entire call routing architecture.

  2. Upon determining which call sites are affected, advise ESD or ERC senior management of alternatives. Because ESD a "virtual" CallCenter, the adverse impact of losing one call site can be reduced by reassigning Specialists from non-call inventory functions to call handling functions. Once a course of action is approved by senior management, the ESD QASC ACD team will implement that course of action and inform affected call sites.

  3. The ESD QASC ACD team will continue to monitor the situation and will advise call site and senior management upon restoration of WAN/LAN services.

    Failure Scenario: Failure of SPRINT Circuits

Impact: If a T1 trunk fails, it reduces the number of calls the system can handle simultaneously by 24. However, it does not impact calls coming in on the remaining T1s. The failure of one or two T1s would be transparent to callers, unless it happens to be the peak calling period. In that case, callers will hear a busy signal if all incoming circuits are in use. There is generally sufficient capacity built into the ERC-EUES call routing architecture to compensate for a failed circuit.

Contingency Plan: The failure of a group of circuits to a specific Aspect CallCenter may necessitate implementation of plans to redirect calls to the other Aspect CallCenter (see Section above for recovery plans for the Aspect CallCenter (ACD) Unavailable at either Fresno or Memphis). Consideration should be given to the number of failed circuits.

Notification: A suspected failure of a SPRINT circuit will be reported immediately to the ESD QASC ACD team employing whatever means available (e.g., voice notification is preferred). Upon notification, the Aspect System Administration team will:

  1. Investigate and attempt to confirm the outage. If applicable, implement procedures for Aspect CallCenter (ACD) Unavailable at either Fresno or Memphis (see Section above).

  2. Report suspect circuit outages to the SPRINT Management Center (SMC) at 1-877-387-3667, Option 1.

  3. Advise call site and senior ERC/ESD management of the outage and the impact of the outage on call routing.

  4. Update management on the status of restoration.

    Failure Scenario: Aspect Call Center Voice Processing System (VPS) Failure at Memphis or Fresno

Impact: The Aspect CallCenter voice processing system (VPS) is a voice storage and retrieval system used to record and save announcements or prompts and voice-mail messages. Announcements are used within the ACD to provide information to the caller (e.g., the availability of agents) and/or to solicit a response from the caller. While call routing is not directly affected by a VPS failure, recorded announcements or prompts cannot be played, hence the caller is queued in silence.

Contingency Plan: A VPS failure at either the Memphis or Fresno Aspect CallCenter will necessitate the implementation of a preprogrammed voice system down call control table. This call control table is designed to offer calls to the most available Specialist/CSR regardless of their agent group assignment. The voice system down call control table is designed to be an interim workaround to the VPS failure. Depending upon the nature of the incident causing the VPS failure, consideration will be given to procedures delineated in Aspect CallCenter (ACD) Unavailable at either Memphis or Fresno (see Section above).

Notification: A suspected failure of the voice processing system at either Memphis or Fresno will be reported immediately to the ESD QASC ACD team employing whatever means available (e.g., voice notification is preferred). Employees discovering the failure will also ensure that the incident is reported to the ESD. The ESD will open a Priority 1 ITAMS incident assigning it to the ESD QASC ACD team. Upon notification, the ESD QASC ACD team will:

  1. Confirm the failure of the Aspect Call Center voice processing system.

  2. Upon confirmation, advise applicable management personnel of the situation. Notify call site and senior ERC/ESD management of the outage and the impact of the outage on call routing. Based upon the circumstance, consider implementation of procedures as identified above (Aspect Call Center (ACD) Unavailable at either Memphis or Fresno). Preferred means of notification is voice.

  3. If the situation requires implementation of Aspect Call Center (ACD) Unavailable procedures, immediately notify the SPRINT System Management Center (SMC) at 1-877-387-3667 to implement the appropriate contingency route set. The preplanned route set will redirect all inbound calls from the initial SPRINT VRU (voice response unit) to the operational Aspect CallCenter. The re-configuration to a pre-planned route set will take from five to fifteen minutes. The SMC maintains a list of personnel authorized to change command route sets; this list will be maintained by the ESD QASC ACD Team Lead.

  4. Immediately notify the Aspect Customer Support Center at 1-800-541-7799, Option 1, of the failure and ensure that an Aspect technician is dispatched to investigate the VPS failure; obtain the Aspect incident number. As time permits, advise the Joint Operational Center (JOC) at (678) 530-5014, Option 3, of the incident and that an incident has been opened with the Aspect Customer Support Center.

  5. If required, monitor back-up sign-on activities and advise applicable management personnel regarding availability of additional sign-ons.

  6. Update applicable management personnel regarding the status of recovery operations. Update the ITAMS incident in accordance with Enterprise Incident Management (CIM) Guidelines.

  7. Upon restoration of the Aspect Call Center VPS, advise applicable management personnel to return to normal business operations and close the ITAMS incident. Monitor return to normal operations.

  8. Update management on the status of restoration.

    Failure Scenario: Emergency Evacuation/Closure of a Call Site

Impact: A variety of situations (fire, weather, etc.) may necessitate the short-term or long-term evacuation and/or closure of a call site. Call centers such as ERC and HR Connect that rely solely or predominantly on one call site to handle calls are operationally vulnerable to situations that may require the closure of their call site. On the other hand, "virtual" call centers such as the ESD can sustain business operations if one of their call sites must close. Calls are simply routed to available Specialists at those call sites that remain operational. The reduced staffing caused by the site closure, of course, adversely impacts service levels but the virtual desk can continue to provide services.

Contingency Plan: At those call centers that rely solely or predominantly on one call site to handle calls, calls will continue to be offered to the evacuated/closed site until the supporting Aspect CallCenter is directed otherwise. Emergency evacuation/closure call control tables have been pre-programmed for each of these call sites for short-term evacuations. Generally, these routing tables can be implemented simply by signing a pre-defined user onto the system and placing that user in an idle state or implementing a pre-programmed dynamic routing table; inbound callers are advised of an emergency at the call site and their calls are directed to voice mail. For long-term evacuations/closures, each call site manager should identify an alternate call site location with IRS LAN access and sufficient voice lines to support operations. If an alternate call site with LAN access is not available, sufficient voice lines should be set-up in a manner described for disaster recovery operations for WAN/LAN Failure Affecting Memphis or Fresno Uniphi Connect Remote Users. Call site managers are responsible for communicating short-term and long-term evacuation/closure plans to the ESD QASC ACD team. The ESD QASC ACD team will develop long-term evacuation call control tables consistent with each sites evacuation/closure plans.

Notification: The site experiencing the situation necessitating short-term or long-term evacuation/closure is responsible for notifying the ESD QASC ACD team as soon as possible, situation permitting. Employees reporting the evacuation/closure will also provide the name and contact telephone number of a contact person. The ESD QASC ACD team will provide as much assistance as possible to the site being evacuated/closed, to include ensuring that the ESD is advised of the situation. The ESD QASC ACD team will also remotely sign on the aforementioned pre-defined user and/or implementing pre-programmed dynamic routing table(s).


More Internal Revenue Manual