Accessibility Skip to Top Navigation Skip to Main Content Home  |  Change Text Size  |  Contact IRS  |  About IRS  |  Site Map  |  Español  |  Help  

Part 21. Customer Account Services

Chapter 10. Quality Assurance

Section 1. Embedded Quality (EQ) Program for Accounts Management, Compliance Services, Field Assistance and TE/GE

21.10.1  Embedded Quality (EQ) Program for Accounts Management, Compliance Services, Field Assistance and TE/GE

21.10.1.1  (10-01-2008)
Embedded Quality (EQ) Review Program Overview

  1. This section provides procedures for Campus Embedded Quality program level and site reviews, as well as front-line manager evaluative employee reviews of:

    • Telephone operations

    • Closed Paper case reviews

    • Responses to taxpayer correspondence

    • Outgoing correspondence and notices

    • Adjustment actions

    • Compliance Services Exam Operation determinations

    • E-Mail responses to IRS web site questions

    • Centralized Insolvency

    • Compliances Services Specialized Phones & Paper (Excise Tax and Estate and Gift)

    • Innocent Spouse

    • Centralized Case Processing

    • Taxpayer Assistance Center walk-up contacts

    Note:

    Campus Embedded Quality is defined as the system that is used by Accounts Management, Compliance Services, Electronic Products and Services Support, Field Assistance and Tax Exempt and Government Entities for their Embedded Quality Review Program.

  2. This section also provides procedures for:

    • Accessing, adding, editing, and correcting National Quality Review System (NQRS) and Embedded Quality Review System (EQRS) records,

    • Generating reports available through NQRS and EQRS,

    • Completing the IRS portion of the Customer Satisfaction Survey.

21.10.1.2  (10-01-2009)
What Is Quality Review?

  1. The Quality Review process provides a method to monitor, measure, and improve the quality of work throughout Accounts Management, Compliance Services, Electronic Products and Services Support, Field Assistance and Tax Exempt/Government Entities (TE/GE). Quality Review data is used to provide quality statistics for the Service's Business Results portion of the Balanced Measures, and to identify trends, problem areas, training needs, and opportunities for process improvement.

  2. The Embedded Quality (EQ) effort is a way of doing business that builds commitment and capability among all individuals to continually improve customer service, employee satisfaction, and business results. The EQ effort is based on three components:

    • Improving the way quality is measured, calculated, and then reported from the bottom to the top of the organization,

    • Creating accountability by connecting employee evaluations to the quality measurement and calculation process in a way that is fair and which enables managers and employees to act on the results in a meaningful way,

    • Improving the design and deployment of the quality resources dedicated to review, analysis, and improvement.

    In the Embedded Quality measurement and calculation process, the measures are calculated using the percent correct based on the number of opportunities for defect within each of five "buckets " . The buckets are defined as follows:

    • Customer Accuracy: giving the correct answer with the correct resolution. "Correct" is measured based on the taxpayer receiving a correct response or resolution to the case or issue, and if appropriate, taking the necessary case actions or case disposition to provide this response or resolution. For the purpose of coding, additional IRS issues or procedures that do not directly impact the taxpayer's issue or case are not considered.

    • Regulatory Accuracy: adhering to statutory/regulatory process requirements when making determinations on taxpayer accounts/cases.

    • Procedural Accuracy: adhering to non-statutory/non-regulatory internal process requirements when making determinations on taxpayer accounts/cases.

    • Professionalism: promoting a positive image of the Service by using effective communication techniques.

    • Timeliness: resolving an issue in the most efficient manner through the use of proper workload management and time utilization techniques.

  3. A "product line" is a major grouping of similar work that is reportable and is measured/captured in the Business Performance Review System (BPRS) or a snapshot report. All product lines listed below are further defined later in this IRM. The national and local quality reviews for these product lines will be entered on NQRS, and the managerial quality reviews for these product lines will be entered into EQRS. A "Specialized Product Review Group (SPRG)" is a subset of a product line that has a separate sample.

    Product Line Specialized Product Review Group (SPRG)
    Accounts Paper Adjustments
      IEAR
    Accounts Phones EIN
      General
      International
      NTA
      PPS
      Spanish
    ACS Phones ACS Phones
    ACS Written ACS Case Processing
      ACS Support
    AM Routing Default Screener
    AM Specialized Services CAF/POA
      EIN
      International Specialized
      Support Services
      Technical Services
    AUR Paper AUR Paper
      CAWR
      FUTA
      PMF
    AUR Phones AUR Phones
    ASFR Paper ASFR Paper
      ASFR Paper Reconsideration
      ASFR Paper Refund Hold
    ASFR Phones ASFR Phones (Including Refund Hold)
    Centralized Case Processing Collection Paper CCP Lien Paper
      CCP GCP
      CCP MMIA
    Centralized Case Processing Phones CCP Lien Phones
      CCP MMIA Phones
    Centralized Case Processing Exam Paper CCP Exam Paper
    CIO Paper CIO Paper
    CIO Phones CIO Phones
    COIC Paper COIC Paper
    COIC Phones Offer Exam
      Process Exam
    Collection Paper Collection Paper
    Collection Phones Withholding Compliance
    CS Specialized Paper Services Centralized Excise Tax Paper
      Centralized Estate and Gift Tax Paper
    CS Specialized Phone Services Centralized Excise Tax Phones
      Centralized Estate and Gift Tax Phones
    e-help Phones e-help Phones
    Exam Paper Area Office Support
      Classification
      Discretionary Programs
      EIC Programs
      Flow Through Entities
    Exam Phones Cold Calls
      Extension Routed
      Outgoing Calls
    Forms Order NDC
    Innocent Spouse Paper Innocent Spouse Paper
    Innocent Spouse Phones Cold Calls
      Outgoing
    Tax Law Phones General
      International
      Rmail Callback
      Spanish
    Tax Law Written ETLA
    TE/GE Phones TE/GE Telephone
    TE/GE Correspondence TE/GE Correspondence
    Field Assistance Tax Law
      Accounts
      Procedural
      Return Preparation

  4. New product lines must be approved by the EQ National Support Staff. In the request that an additional product line be reviewed, the function must be prepared to answer the following questions:

    • Is there an existing measure?

    • How many Full Time Equivalents (FTEs) will be needed, and is the function willing to provide all or part of them?

    • What is a detailed explanation of the function's need?

    • Who will perform the review (Centralized Quality Review System (CQRS), Program Analysis System (PAS), site quality analysts)?

    • What is the volume of work?

    • What is the sample size/frequency?

    • Will the review be by site, Business Operation Division (BOD), or enterprise?

    • Who is recommending this request?

    Note:

    For additional guidance and required next steps, refer to the EQ Website athttp://eq.web.irs.gov/ and click on "EQ Implementation Process" .

  5. Components of the Quality Review process include the Centralized Quality Review System (CQRS), Program Analysis System (PAS), site reviews for the national quality measure, and local reviews performed for quality improvement. The National Review is a review of an entire product, measuring the quality of service to all of the product's customers. Data from national reviews is entered into the National Quality Review System (NQRS), then rolls up, and provides the results for the Balanced Measures. These results translate into the monthly quality ratings for Customer Accuracy, Professionalism, and Timeliness. Data from local reviews performed for quality improvement is entered into NQRS as a Local Review and is not rolled up into the national quality and accuracy rates.

  6. The Managerial Review process creates accountability by connecting employee evaluations to the quality measurement and calculation process. Managers will use the system for planning purposes, and to track employee performance and training needs. Data from the managerial reviews is entered into the Embedded Quality Review System (EQRS), which maps to an employees critical job elements and aspects, then rolls up to identify overall employee, team, department and site scores in Accuracy, Professionalism, and Timeliness. It is not combined in any way with national or local quality and accuracy rates.

  7. Review data is compiled in NQRS and EQRS using a Data Collection Instrument (DCI).

  8. Contact Recording is a quality monitoring system that has proven effective in private industry and is now used in the IRS. The Contact Recording (CR) system records contacts between the Service and the customer. For telephones, the system records the complete conversation of every call, and randomly selects approximately ten percent of the calls per site, to simultaneously capture both the voice (conversation) and on-screen computer activity. For Field Assistance, the system records both the voice (conversation) and the on-screen computer activity for every contact. If a call that is selected for review is a call that is one of the random ten percent, managers and quality review staffs will be able to hear and see (through synchronized screen displays) the entire customer experience. For Field Assistance, all contacts will show the entire customer experience. The technology allows managers to provide substantive feedback and identify training needs for employees. Managers can access the recordings and allow employees to listen to their own interactions with customers, fostering personal improvements. Contact Recording is available for managers for most telephone and Field Assistance product lines. Contact Recording is available for some of the national quality review. Contact Recording records remain in the (CR) system, available for review, for 45 days.

    Note:

    Refer to the specific product line portion of this IRM for individual program guidance on the use of Contact Recording.

    • The Verint Ultra system of "Contact Recording" (CR) is a telephone tool used by Accounts Management, Electronic Products and Services Support and Compliance Services to record incoming "toll free" telephone contacts for the purpose of possible subsequent quality review monitoring. Incoming calls are answered with an additional announcement that states, "Your call may be monitored or recorded for quality purposes." The Verint Ultra system records the audio, and occasionally the video, of all telephone calls coming into the Service via the ASPECT Communication voice response unit and teleset.

    • Managers and Quality Analysts use CR to perform required random reviews (performance and product) of incoming telephone contacts. Contact Recording allows for a more cost effective review as there is no lag time between calls. Calls recorded in this system are available for National Review the next business day. The analysts are no longer required to wait on a trunk to access a call, but are now able to directly access the entire call for review at their discretion. Additionally, the system stores data by Standard Employee Identifier (SEID) for a maximum of 45 days.

      Note:

      At this time the COIC Phones SPRG does not use the Contact Recording system because they are not using the ASPECT Communication teleset.

  9. Sampling with the Contact Recording system is made up of the following.

    1. All calls received by the Operation are included in the review universe, ensuring a more statistically valid review.

    2. Statistic of Income Office (SOI) created a sampling algorithm, which was programmed into the Verint System. The algorithm provides each site with a statistically valid sample per SPRG.

    3. See IRM 21.1.1.10 , Contact Recording, for more information on Contact Recording.

21.10.1.2.1  (10-01-2008)
Centralized Quality Review System

  1. The Centralized Quality Review System (CQRS) is operated by Headquarters to provide independent quality review services for a number of product lines.

  2. Centralized Quality Review System (CQRS) measures the quality of:

    • Tax Law, Accounts, NDC, Default Screeners, e-help, TE/GE and ACS calls answered by assistors in all sites,

    • E-Mail (Electronic Tax Law Assistance – ETLA) responses to questions received through the IRS Website,

    • Field Assistance Tax Law, Accounts, Return Preparation and Procedural contacts.

21.10.1.2.2  (10-01-2009)
PAS and Site Reviews for the National Measure

  1. PAS and Site Quality Review measures the quality of:

    • Accounts Paper

    • ACS Support

    • ACS Case Processing

    • AM Specialized Services

    • ASFR Paper and Phones

    • AUR Paper and Phones

    • Centralized Case Processing Collection Paper and Phones

    • Centralized Case Processing Exam

    • CIO Paper and Phones

    • COIC Paper and Phones

    • Collection Paper

    • CS Specialized Paper and Phones Services

    • Exam Paper and Phones

    • Innocent Spouse Paper and Phones

    • Withholding Compliance Phones

21.10.1.2.3  (10-01-2006)
Local Reviews for Quality Improvement

  1. Local reviews for quality improvement may be performed to focus attention on areas that require improvement as identified by the measured reviews of CQRS, PAS, and site national reviews. The local quality reviews are performed by staffs reporting to the Quality Assurance Coordinator (QAC), PAS Manager, and/or other units that have quality assurance duties. Local quality reviews may also be used for employee development and on-the-job instruction. Accounts Management and Compliance Services functions may also request that local quality reviews be performed on collateral processes that are not subject to the national quality review.

21.10.1.2.4  (10-01-2007)
Managerial Reviews

  1. Managerial reviews on EQRS measure the quality of the work performed by their employees on all Specialized Product Review Groups (SPRGs).

21.10.1.2.5  (10-01-2009)
Objectives of Quality Review

  1. Quality Review data is used by management to provide a basis for measuring and improving program effectiveness by:

    1. Identifying defects resulting from site or systemic action(s) or inaction(s),

    2. Identifying drivers of Customer Accuracy,

    3. Analyzing the reason(s) for defect occurrence,

    4. Identifying defect trends,

    5. Recommending and initiating corrective action,

    6. Following up with process review and analysis to ensure the corrective action was effective,

    7. Providing input to Balanced Measures,

    8. Assisting management in efforts to improve quality of service provided by staffs.

  2. Managerial review data is used by management for some or all of the purposes listed above as well as:

    1. Tracking employee performance and providing input into employee appraisals,

    2. Identifying training needs for individual employees and for groups of employees,

    3. Planning workload distribution.

    Note:

    Managerial reviews evaluate individual employee performance and are performed independently from national and local quality reviews. National and local quality review results are never used for evaluation of individual employees. See IRM 1.4.16 (Accounts Management Managers Handbook), IRM 1.4.20 (Filing and Payment Compliance Managers Handbook) IRM 1.4.17 (Compliance Managers Handbook) and IRM 1.4.11 (Field Assistance Guide for Managers ) and IRM 1.4.18 (e-help Desk Managers Guide for more information on manager responsibilities in conducting managerial reviews.

21.10.1.2.6  (10-01-2008)
Quality Review Roles and Responsibilities

  1. The Quality Review process relies on the teamwork of all of the following:

    • Headquarters Product Line Analysts (PLAs) and Business Operating Divisions (BOD) Quality Analysts

    • CQRS

    • Accounts Management, Compliance Services, Electronic Products and Services Support, Field Assistance and TE/GE Operations

    • Quality Assurance Coordinators (QACs) and Quality Assurance Managers (QAMs) and Improvement Specialists or other quality staffs

    • Planning and Analysis (P&A) Staffs and PAS Managers

    • Quality Analysts and PAS Analysts

21.10.1.2.6.1  (10-01-2009)
Headquarters

  1. Headquarters will issue Strategy and Program Plans based on the Balanced Measures.

  2. Headquarters will review Quality Assurance programs as part of periodic reviews of Accounts Management, Compliance Services, Electronic Products and Services Support and Field Assistance programs.

  3. The Joint Operations Center (JOC) will operate the Centralized Quality Review System (CQRS). JOC will also house the National Support Staff and will maintain NQRS, EQRS, and the EQ Website.

21.10.1.2.6.2  (10-01-2009)
Accounts Management/Compliance Services Operations

  1. The P&A Chiefs for Accounts Management and Compliance Services are responsible for the Site Level Business Plans.

  2. The P&A Chiefs are responsible for site reviews of product lines that provide data for the Balanced Measures.

  3. If feasible, the QAC or QAM should report directly to the P&A Chief. This eliminates potential conflicts of interest which may occur when more than one department manager is accountable for a product or service.

21.10.1.2.6.3  (10-01-2009)
Quality Assurance Coordinator (QAC)/Quality Assurance Manager (QAM)

  1. This section only applies to remote and campus locations with functional areas where there is an existing QAC/QAM position. In offices/functional areas where no position exists, management will need to ensure these duties are appropriately addressed.

  2. The QAC/QAM is responsible for the overall planning, administration, and evaluation of the quality-related sections of the Site Level Business Plans. The QAC/QAM will identify problems and work with management to solve them.

  3. The Site Level Business Plans outline procedures for the review of all areas of responsibility. This review process, when combined with CQRS data and other functional data, will assist the QAC/QAM in evaluating the overall quality of operations and making recommendations for improvement.

  4. The QAC/QAM will serve as the Quality (QA) manager for the Operation, ensuring that designated quality resources are used to focus on quality improvement efforts.

    Note:

    Results of reviews performed by the Quality Review (QR) staff are not to be used in employee evaluations.

  5. The QAC/QAM will maintain the integrity and quality of the QR system by monitoring and reviewing the NQRS results. The PAS staff may assist in this process.

  6. By using trend analysis, the QAC/QAM will determine the causes that adversely affect quality. The QAC/QAM will assist the management team in initiating processes for employees to improve their quality of service. It is important that lines of communication remain open among the QAC/QAM, the QR team, and management in order to identify problem areas, take appropriate corrective actions, and re-evaluate quality to ensure corrective actions result in improved quality.

  7. The QAC/QAM or other designated person within the operation will log in and date completed review records that require rework, including reviews by CQRS or PAS where defects have been identified including when FLASH was annotated in the Feedback Summary Remarks field. Managers must return corrected work to the QAC/QAM within five working days of receipt. See IRM 21.10.1.7.7 (4) for additional instructions on FLASH remarks. The QAC/QAM will enter the completion date in the log. The QAC/QAM will monitor corrected work to ensure the timeliness and quality of responses to taxpayers.

  8. The QAC/QAM will act as the liaison between the QR/PAS staff and management and is responsible for communicating quality information to all managers in the operation.

  9. The QAC/QAM will identify training needs and recommend to the Training Coordinator and/or management, the type of training needed (e.g., on-the-job training or classroom instruction), and assist in the development of additional training exercises and workshops to meet those needs.

  10. The QAC/QAM and the QR/PAS staff will be responsible for the protection of NQRS DCIs and any supporting documentation from legacy systems. All documents and information (including taxpayer information) seen, heard, or handled must remain secure and confidential.

  11. The QAC/QAM will serve as the Embedded Quality liaison with managers.

  12. The QAC/QAM will work with managers to recognize and reward quality at all levels (e.g., employee-of-the-month, Quality Honor Roll, etc.), using the National Agreement for guidance.

  13. The QAC/QAM will be responsible for working with the Training Coordinator and/or management to ensure that EQ training is made available to all who need it. The QAC/QAM is responsible for training the quality staff.

21.10.1.2.6.4  (12-11-2008)
PAS Managers/Other Quality Review Managers

  1. Throughout this IRM section, the term "PAS Managers" will be used to include the duties of either PAS Managers or other functional Quality Review Managers who perform site reviews for the national quality measure and/or local reviews for quality improvement.

  2. PAS Managers are responsible for completing national and local reviews as described in this section.

  3. PAS Managers ensure that all applicable work is sampled and reviewed within prescribed guidelines.

    Note:

    Results of reviews performed by PAS are not to be used in functional employee evaluations.

  4. The PAS Manager maintains the integrity and quality of the review system by monitoring and reviewing the quality analysts and clerical support. The PAS Manager should routinely monitor the quality sampling selection process. Refer to descriptions of the product lines later in this section for more information on sample universes and sampling guidelines. The PAS Manager should also perform a periodic (at least quarterly) spot check of the work to confirm the case volume matches the count provided by the operations. For example: verify the review content of a single folder to ensure that the volume of the work matches the number of cases in the folder. If these figures do not match, contact the operation to address the reason(s) for the discrepancy and to discuss ways to improve the process accuracy.

  5. The PAS Manager and staff determine the causes that adversely affect quality by using trend analysis to identify the most frequently made defects and root causes. The PAS Manager recommends corrective action and/or improvements to functional areas. In sites or functional areas where a QAC, QAM or Improvement Specialist is present, the PAS Manager may not be required to perform this function.

  6. Depending on local procedures, the PAS Manager identifies sample cases that require rework (including cases where FLASH was entered in the Feedback Summary Remarks field) and forwards them to the appropriate functional area. This includes reviews by CQRS where defects have been identified.

  7. The PAS Manager is responsible for working with the Training Coordinator to ensure that the PAS staff has the appropriate training.

  8. If no QAC/QAM position exists in the functional area, the PAS Manager may be required to perform the duties described in the IRM.

21.10.1.2.6.5  (10-01-2008)
Quality Analyst/PAS Analyst

  1. For all product lines identified in this IRM, the quality analyst will thoroughly review and code sample cases using the guidelines in this IRM.

  2. The quality analyst performs an unbiased, consistent, and accurate review of all work. To ensure this effect, the Quality Analyst/PAS Analyst, or a clerk assigned to the Quality Review/PAS Unit, is responsible for the pulling of the daily sample.

  3. The quality analyst provides the QAC/QAM or PAS Manager with:

    • Any cases identified for rework

    • Weekly analysis of types of errors identified during review.

  4. The quality analyst will provide the QAC/QAM, Headquarters Product Line Quality Analyst, or National Support Staff, as applicable, with recommendations for corrections/improvements on:

    • IRM 21.10.1, EQ Program for Accounts Management, Compliance Services, Field Assistance and TE/GE.

    • Embedded Quality Review Training Material

    • The NQRS DCI.

  5. The quality analyst will:

    1. Identify the defect and code the appropriate attribute for that particular action, while considering the root cause of the defect; a single erroneous action on one DCI should usually not be coded with "N" in multiple attributes (avoid ripple/trickle down effect).

    2. Review work using valid sampling techniques as approved by SOI.

    3. Use consistent review techniques.

    4. Have knowledge of all equipment used in reviewing.

    5. Record complete review results using the NQRS DCI.

    6. Report all problems to the QAC/QAM or PAS Manager.

    7. Measure performance against established quality standards in the IRM, publications, and other approved reference sources.

21.10.1.2.7  (10-01-2009)
Quality Review/PAS Staffing

  1. Positions on the QR/PAS team will be filled by:

    • Formal detail,

    • Reassignment, or

    • Competitive announcement.

  2. All employees selected to sample and review work will need to complete training that is appropriate for the SPRG they will review. The following is a list of suggested training topics.

    • Embedded Quality Review Training,

    • Quality Review command codes,

    • Automatic Data Processing (ADP)/Integrated Data Retrieval System (IDRS) training modules, including on-line accounts and adjustments, and Accounts Management and Compliance Services,

    • Technical tax law (as appropriate),

    • Technical Probe and Response Guide,

    • Interactive Tax Law Assistant (ITLA),

    • Accounts Collection Service Guide,

    • Correspondence guidelines,

    • Taxpayer Advocate Service guidelines and criteria,

    • Oral statement guidelines,

    • Procedural guidelines,

    • Communication skills,

    • Probing skills,

    • Timeliness guidelines,

    • e-help and TE/GE Specialized Systems.

  3. The QR/PAS staff will not be limited to Accounts Management, Compliance Services, Field assistance and TE/GE personnel; however, all QR/PAS staff members must meet the requirements below as applicable to their assigned SPRGs to review:

    1. Technical competence in tax law and IRS procedures,

    2. Technical competence in all areas of accounts and/or functional work,

    3. Ability to maintain objectivity,

    4. Ability to make firm decisions and to use sound judgement,

    5. Ability to communicate tactfully.

  4. The number of analysts selected/assigned to the QR/PAS Staff will be determined by the volume of work to be reviewed. More analysts may be required in preparation for peak periods.

21.10.1.2.8  (10-01-2006)
Strategy and Program Plans

  1. The Strategy and Program Plans are vehicles used to monitor, measure, and evaluate activities consistent with the quality goals of the Balanced Measures.

  2. The Strategy and Program Plans should include:

    1. Action items that support Balanced Measures initiatives.

    2. Measurement data from CQRS, PAS, QR, local reviews, quality staffs, and managerial reviews.

      Note:

      Throughout each month, management must monitor/review the required minimum number of employee telephone calls/cases with taxpayers as set in the Site Quality Action Plan.

  3. The QAC/QAM or other local management will assist in the development of Site Level Business Plans which should include requirements to perform reviews to assess the quality of the work including:

    • Managerial reviews of employee work including telephone assistance and written work

    • Timeliness of completed work

    • Accounts Management Accounts and Tax Law work

    • Compliance Services Collection and Exam Operation work

    • Field Assistance contacts including Return Preparation

21.10.1.2.8.1  (10-01-2006)
Establishing the Strategy and Program Plans and Site Level Business Plans

  1. When creating the Site Level Business Plans, the QAC/QAM will:

    1. Review past Site Level Business Plans and the current year’s Balanced Measures recommendations to understand the program’s goals.

    2. Analyze statistical data, such as NQRS, and EQRS data to identify inconsistencies.

    3. Meet with management to discuss QA objectives.

    4. Establish QA objectives and set priorities using criteria provided by Headquarters and functional management. Local objectives can be added, if desired.

    5. Develop a schedule for implementing each objective of the plan and assign responsibilities.

    6. Work with management to prepare a draft of the plan for review.

    7. Consolidate suggested changes into a final draft for the Operations Chief’s concurrence, which certifies that the plan is operable.

    8. Communicate the applicable portions of the plan to all functional employees, ensuring they understand the plan’s objectives and their roles in the QA process.

  2. After implementation of the Site Level Business Plan, the QAC/QAM will monitor plan accomplishments and continue to analyze site data to determine if the operation is meeting objectives. The QAC/QAM is expected to recommend needed changes to the plan such as modification or discontinuance of certain objectives.

21.10.1.2.8.2  (10-01-2008)
Strategy and Program Plans and Site Level Business Plans Resources

  1. Following is a list of resources available to the QAC/QAM when creating the quality portion of the Site Level Business Plan. This list is in no particular order and is not meant to be all inclusive. It is designed to suggest the wide variety of data available for consideration.

    • Local work plans

    • NQRS data including local reports

    • EQRS data including local reports

    • Timeliness data

    • Prior years’ Operating Guidelines, including plans for QR and managerial involvement

    • Headquarters and functional business plans and reviews

    • Field Assistance data

    • Accounts Management/Compliance Campus reports on functional activities

    • Alert information previously provided to employees

    • Staff feedback

    • Statisticians feedback or reports

    • Taxpayer Advocate Service staff

    • Treasury Inspector General for Tax Administration reports

    • Results of Improvement Projects

    • Focus testing reports

    • General Accounting Office reports

    • ASPECT telephone reports/data

    • Customer satisfaction survey results.

21.10.1.3  (12-11-2008)
Quality Review Research Tools and Procedures

  1. The following paragraphs may not be all-inclusive, but they will provide a bibliography of the most frequently used research tools.

  2. A number of IRMs impact the work done by Accounts Management, Compliance Services, Field Assistance, Electronic Products and Services Support and TE/GE. The IRM 21 will often cross-reference these other manuals. Examples include:

    • IRM Parts 4 and 5 (Examining and Collecting Processes).

    • IRM 20.1, Penalty and Interest

    • IRM 3.42, Electronic Tax Administration

    • IRM Part 13, Taxpayer Advocate Service

    • IRM 11.3, Disclosure of Official Information

    • IRM 2.3, IDRS Terminal Response

    • IRM 2.4, IDRS Terminal Input

    • IRM 2.8, Audit Information Management Systems (AIMS)

    • IRM 20.2, Interest

    • IRM 25.6, Statute of Limitations

    • IRM 21.3.7, Processing Third Party Authorizations onto Centralized Authorization File (CAF)

    • IRM 1.4.11, Field Assistance Guide for Managers.

    • IRM 1.4.16, Accounts Management Guide for Managers

    • IRM 1.4.17, Compliance Managers Guide

    • IRM 1.4.18, e-help Desk Managers Guide

  3. A number of methods are used to communicate changes, clarifications or corrections to the IRMs and other published products. Among these are:

    • Servicewide Electronic Research Program (SERP)

    • IDRS Bulletins

    • IDRS Message File

    • IRM Update

    • Headquarters Alerts/Memoranda

    • Internal Revenue Bulletins (IRBs)

    • The EQ Website

    • The CQRS Website

    • Quick Alerts and e-help Desk Communications for e-help

  4. SERP allows for efficient research of many sources of information. The information available changes yearly. Following are examples of sources of information available:

    • IRM 21, Customer Accounts Services,

    • Other product line-specific IRMs (Parts 3, 4 and 5),

    • Probe and Response Guide (P&RG),

    • Interactive Tax Law Assistant (ITLA),

    • Publication Method Guide (PMG),

    • Electronic ACS Guide (EACSG),

    • Technical Communication Documents (TCD),

    • TE/GE Enhanced Job Aid,

    • Telephone Transfer Guide (TTG),

    • Taxpayer Contact Outline,

    • Taxpayer Information Publications,

    • Forms, Schedules, Instructions,

    • Correspondence Letters,

    • CP Notices,

    • Post-of-Duty Listing and Job Aid,

    • Lockbox Addresses,

    • Special Procedures Directory,

    • State Unemployment Tax Agencies,

    • VITAVolunteer Income Tax Assistance/Tax Counselors for the Elderly/American Association or Retired Person Sites,

    • On-Line Training Materials,

    • Miscellaneous documents of local interest,

    • Document 6209.

  5. Do not charge quality defects related to IRM or procedural changes until seven calendar days after the Servicewide Electronic Research Project (SERP) posting date of the IRM update/change.

    Note:

    For AUR, the seven calendar day period begins after the IRM update/change is issued on e-mail by the Headquarters Analyst, and for National Distribution Center (NDC), the seven calendar day period begins after the date on the Alert/Change.

    Note:

    For e-help, the seven calendar days begins on the date the e-help Communication or Quick Alert e-mail was first issued.

  6. During the seven day grace period, national analysts will code "Y" if the former or new procedure is followed. Informational feedback from local and national reviews will be shared with functional areas prior to the seventh day, but no defect will be charged unless no procedure or an incorrect procedure is followed.

  7. Because of the scope of paper reviews conducted in campuses, each QR/PAS function should establish or have access to a library of all necessary IRMs and Law Enforcement Manuals (LEMs). Area Offices (AOs) should establish a similar IRM library covering all the types of work performed in their site. In most cases, the IRMs can be found on-line through SERP.

  8. In many cases, the IRMs will contain the same information as LEMs. LEMs contain supplemental information of a more sensitive nature, such as tolerance amounts. Access to LEMs is restricted; therefore, not all caseworkers or employees may be in a position to readily research a LEM. However, where a relevant IRM references a LEM, the information contained in that LEM must be considered in evaluating quality.

  9. Any IRS publication can be cited as a reference source. IRS publications will often be the primary research tool for tax law issues. Procedural issues are often addressed in the publication which can be located electronically on SERP.

  10. Various websites can be accessed as additional reference sources.

  11. Responses to taxpayer questions may refer to specific forms and their instructions. They can be located electronically on SERP under Forms/Letters Pubs to verify the quality of the response. If erroneous information has been given to the taxpayer, cite the form or instruction to substantiate the defect.

  12. The IRS Electronic Publishing Catalog contains a number of documents that can be used for research purposes. One of the most frequently used is Document 6209, IRS Processing Codes and Information. Another is Document 9912, Post-of-Duty Listing and Job Aid. These documents can also be found on-line through SERP.

    Note:

    If there is a discrepancy between the Document 6209 and a specific IRM, the IRM takes precedence.

  13. Use the Probe and Response Guide or the Interactive Tax Law Assistant, or the Publication Method Guide in conjunction with Publication 17 and other reference materials to answer customers’ tax law questions. Its use is mandatory for review of the Tax Law product line.

  14. It is essential that all sites and functions follow the same guidelines for providing quality service. Local procedures are deviations from the IRM or other established practices that are implemented by a specific site or function. Only those local procedures that have received prior approval from Headquarters program management can be considered when evaluating quality. The procedure for requesting deviations is as follows:

    1. The Operations Manager in a site is alerted to a need for a deviation and forwards this need in memo format to the Site Director.

    2. The memo should state the reason for the deviation, what caused the situation to occur and what is being done to correct it, and how long the deviation will remain in effect (no longer than one year - see Note below).

    3. If the Site Director concurs with the deviation, it will be forwarded to the Business Unit Director for concurrence.

    4. Once the Business Unit Director concurs, the deviation memo is forwarded to the peer Business Unit Director for concurrence; the memo should be signed by both Business Unit Directors.

    5. The last step is to send the signed memo to the Headquarters (HQ) Policy and Program Analysts for distribution. The memo will be kept on file in HQ for a period not to exceed one year.

    Note:

    Deviations are only good for one year unless extended specifically by Headquarters. Deviations are from a starting date in the future and are not retroactive.

  15. Compliance Deviation Process from IRM procedures initiated by Headquarters must be approved by the impacted Business Operating Division (BOD) Director; the Director, Campus Reporting Compliance, or the Director, Filing and Payment Compliance. The complete procedures for requesting deviations are as follows:

    • The Headquarters Analyst, Headquarters Program Manager, or Campus Director requesting the deviation must forward the request in a memorandum to the appropriate Business Operating Division Director.

    • The memo must explain the reason(s) for the deviation, what caused the situation to occur, what is being done to correct it, and the starting and ending dates of the deviation. Deviations will not be retroactive and will remain in effect for the stated period but no longer than one (1) year;

    • Upon concurrence, the BOD Director will sign the deviation and forward it to his counterpart BOD Director for information. Should the receiving BOD Director elect to implement the deviation, an additional signature would be required. Deviations will only be considered in effect for the business unit upon signature by the BOD Director(s); and,

    • The deviation memorandum must be distributed to all Campus Directors affected by the change as well as to Quality Performance Measurement (QPM) to ensure no quality errors are charged during the period of the deviation

      Note:

      These procedures do not apply to regular manual updates that provide interim guidance and are processed through SERP Alerts, as these items officially replace the current manual. Also, manuals owned by Field Policy do not fall within this process, as deviations have a separate reporting structure. Finally, in order to remain valid, deviations currently in place must be brought into compliance with the aforementioned procedures by no later than January 31, 2008.

  16. Among the items that cannot be used to evaluate the quality of a contact are training materials, locally developed job aids, or commercial tax publications.

  17. Local reviews of telephone operations require access to the ASPECT system, Contact Recording and Account Management Services (AMS). Headquarters does not mandate how to use this equipment. Analysts should experiment with different techniques and use the techniques that are most comfortable for them and most effective for the desired review.

  18. Use various IDRS command codes to review sample cases. Refer to IRM 2.3, IDRS Terminal Responses, and IRM 2.4, IDRS Terminal Input, for complete details of all IDRS command codes.

  19. Users must have access to NQRS to input sample case review results. See IRM 21.10.1.7 for instructions.

  20. In conducting reviews, other automated systems may need to be accessed. Managers are responsible for contacting the appropriate system administrator to grant access rights to these systems. These items include, but are not limited to:

    • Automated Collection System (ACS)

    • Automated Underreporter (AUR)

    • Automated Lien System (ALS)

    • Integrated Collection System (ICS)

    • Automated Non-Master File (A-NMF)

    • On-Line Notice Review (OLNR)

    • Locator services, such as credit bureaus and state employment commissions

    • Automated Substitute for Return (ASFR)

    • Report Generation Software (RGS) used by Examination.

    • Automated Offer in Compromise

    • Correspondence Imaging System (CIS)

    • Withholding Compliance System

    • e-help Support System (EHSS)

    • Third Party Data Store (TPDS)

    • EP/EO Determination System (EDS)

    • Letter and Information Network User Fee System (LINUS)

    • TE/GE Correspondence Control System (TCC)

    • Automated Offer in Compromise (AOIC)

21.10.1.3.1  (01-29-2009)
Quality Review Sampling Guidelines

  1. While CQRS, PAS or sites perform reviews on product lines or SPRGs for the national measure, each site may also perform local reviews to help improve their quality. The Local button on the NQRS DCI is used to input local reviews.

    Note:

    Local reviews are not used in official measures of product lines and are not included in the calculations of Balanced Measures.

    • Resources for local review are allocated in a site's workplans. Some sites will have resources for both local and national reviews, while others will only have resources for local reviews. This will depend on where the site's national reviews are performed. Resources specifically allocated for local quality improvement reviews should be used to perform these reviews.

    • When performing local reviews for quality improvement, the overall local quality rates may be lower than the national quality rates since local analysts will focus on problem areas.

    • Using the local review option, other areas of work may be reviewed that are not covered under the Balanced Measures.

    • Local reviews for quality improvement are not necessarily expected to be statistically valid samples.

  2. See each specific SPRG for additional guidance on sample selection.

  3. A sample is a collection of cases selected from all cases in a given population. Sampling allows the IRS to determine the quality of service provided by an office without having to review every piece of work (i.e., a census).

  4. A simple random sample is a method of selecting 'n' cases ensuring that all "N" cases in the population have an equal chance of being selected in the sample.

  5. A systematic sample is one method of selecting a simple random sample. This method involves methodically selecting every ‘kth’ case in the population. The interval ‘k’, referred to as the ‘skip interval’, is calculated by dividing the population, ‘N’, by the sample size, ‘n’. The first case, referred to as the ‘random start’, is selected randomly and is between 1 and ‘k’. See IRM 21.10.1.3.1.1.(5) for different methods of determining a random start.

    Note:

    The exact case must be pulled based on the skip interval.

  6. Results from a sample can be used to estimate the quality of work across all ‘N’ cases in the population. In order to represent the actual quality, a sample estimate must be unbiased and consistent. An unbiased estimate is one where the average value of the estimate, taken over all possible samples of a given size, is equal to the actual quality in the population. A consistent estimate is one where the estimate approaches the actual quality in the population as the sample size increases.

  7. Because a sample does not include all cases in the population, any estimate resulting from a sample will not equal the actual quality in the population and will have some variability associated with it. For an estimate to be meaningful, a measure of variability should be included with results. A precision margin and level of confidence can be used to express the variability of an estimate. When added to and subtracted from the estimate, a precision margin identifies the range of values where the actual quality in the population most likely falls. The confidence level indicates how much confidence one can have that the actual population value is within this range. Many IRS quality measures are designed to achieve estimates of 5% precision with 90% confidence. That is, there can be 90% confidence that the actual quality in the population is within plus or minus 5% of the sample estimate.

    • While quality measures are designed to achieve a certain precision (e.g., 5%), the actual precision of an estimate must be calculated using the actual results from the sample.

    • Precision is directly related to sample size. As sample size increases, precision decreases.

    • Precision is also directly related to the estimate of quality. The worst case scenario for an estimate (precision-wise) is 50%. In other words, an estimate of 50% will have the highest precision of all possible samples of the same sample size. Estimates that are closer to either 0% or 100% will be more precise than estimates near 50%.

    • Precision margins should be taken into consideration when determining if a site met its goal.

    Example:

    Assume a site has a goal of 85% and that their sample estimate is 82% with a precision of 4%. Applying the precision margin to the estimate implies that the actual quality in the population is between 78% and 86%. Because 85% lies in this range, the site cannot conclude that they did not meet their goal nor can they conclude that they did meet their goal. However, not taking the precision margin into account would have led the site to conclude that they did not meet their goal.

    • Precision margins should also be taken into consideration when comparing quality estimates between offices or different time periods from the same office.

  8. Samples should be designed conservatively. One piece of information that is necessary when designing a sample is an estimate of the actual quality being measured. This estimate, often obtained from reviews from prior years, is used in the process for determining the sample size. An assumed quality of 50% will result in the largest sample size. Therefore, in order to get a more conservative sample size, either slightly increase or decrease the assumed quality rate so that it is closer to 50%. This will result in a sample size that should provide the desired precision.

    Example:

    If an office had an 88% quality rate for a certain SPRG during the prior year, then assuming an estimated quality rate of 80% will result in a sample size that is sufficiently large.

  9. When designing a sample, an office must decide how often quality estimates are necessary. The individual needs of the office, as well as the resources assigned to the quality review, will help determine whether estimates should be made on a weekly, biweekly, monthly, quarterly, or annual basis. Making estimates on a daily basis is usually not recommended. In addition, in many cases, a weekly estimate requires a sample size that is not practically possible. Often, monthly is the shortest time period for which IRS quality estimates are both practical and recommended.

  10. Some offices may consider merging several types of similar work. There may be several items at the site that need to be tested. However, none are large enough to justify an individual sample, and all are too large to take a census. A merged sample, producing a composite estimate, would be a possible solution in this situation. Information describing the different types of work would have to be included with all estimates from these merged samples.

21.10.1.3.1.1  (10-01-2008)
Selecting the Quality Sample

  1. Use the following steps to determine a sample size:

    1. Decide how often quality estimates are necessary (e.g., weekly, monthly).

    2. Decide the level of precision needed for estimates (e.g., 3%, 5%).

    3. Decide the level of confidence needed for estimates (e.g., 90%, 95%, 99%).

    4. Decide on a hypothetical "best guess" of the quality rate expected from the review. This guess should be conservative ( See IRM 21.10.1.3.1.) and can be obtained by reviewing historical quality rates for similar product lines or SPRGs.

    5. If 90% was selected as the confidence interval, then refer to the Table of Approximate Precision ( See Exhibit 21.10.1-2) to determine the sample size (n) required given the assumed quality rate (p) and desired precision. If a confidence level besides 90% was selected, please contact a Headquarters quality analyst for assistance in determining an appropriate sample size.

    6. If the resulting sample size is too large for the allocated resources, consider reducing how often quality estimates need to be made.

    Example:

    If the sample size for monthly estimates is too large, then switching to either a quarterly or annual estimate may require a more manageable sample size.

  2. Unless otherwise specified in IRM 21.10.1, all cases with a closing count must be made available for the National Quality Review System (NQRS) process. This includes cases reviewed by managers, On-the-Job Instructors (OJIs), and cases that are subject to 100% review.

  3. Determine whether the size of the population is known prior to implementing the sample. Whether or not the population is known will determine how the sample selection instructions should be presented to analysts. Unclear or inappropriate analyst instructions could lead to a sample that is not random, resulting in estimates that are neither unbiased, representative of the population, nor consistent. These issues could call into question the statistical validity of estimates.

  4. If the actual size of the population is known prior to selecting the sample, (or if a reliable estimate is available), then apply a skip interval to select the sample. See IRM 21.10.1.3.1.1(7) if the population size is not known.

    1. The skip interval 'k' is equal to the actual (or estimated) population, ‘N’, divided by the sample size, ‘n’.

    2. Calculate and use a random start number to select the first case ( See IRM 21.10.1.3.1.1(5). The random start is between 1 and the skip interval.

    3. Use the skip interval to select the rest of the sample See IRM 21.10.1.3.1.1(6).

    4. If all of the cases are not available at the same time, then sample cases as the work arrives.

    5. If all of the cases are available at the same time and are stored electronically, then software should be used to sort the list of cases either randomly or by a variable related to the quality measure (e.g., time, case ID) prior to applying the skip interval.

    6. If used correctly, a skip interval will ensure that a sample is spread appropriately across the population and provides estimates that are relatively unbiased.

    7. Skip intervals are particularly appropriate for written work.

  5. A random start for a skip interval can be determined using any of the following methods.

    1. Use a scientific calculator with a random number function. Use the random number function to get a random number between 0 and 1. Multiply this number by the skip interval, and round up to the nearest whole number.

    2. Use the RAND() function in Excel. Type "=RAND()" in any cell to get a random number between 0 and 1. Multiply this number by the skip interval, and round up to the nearest whole number.

    3. Use the RANDBETWEEN() function in Excel. Type "=RANDBETWEEN(1,k)" in any cell, where ‘k’ is the skip interval, to get a random number between 1 and the skip interval.

      Note:

      The RAND() and RANDBETWEEN() functions will only work in Excel after the Analysis Toolpak Add-In has been successfully installed. For help with installing Excel Add-Ins, contact your technical help desk.

    4. Use a standard table of 4-digit random numbers. ( See Exhibit 21.10.1-1.).

    5. The first time a table is used, select a starting point anywhere on the table, and decide on a method for working through the table (either down columns or across rows). Use this method to systematically work through the table each time a new random start number is needed, crossing off numbers as they are used. Do not skip around the table. To calculate a random start number, divide the number selected from the random table of numbers by 10,000 and then multiply the result by the skip interval, and round up to the nearest whole number. Then cross the four-digit number off on the table.

  6. Use the following procedures to select a sample using a skip interval.

    1. On the first day of the sampling period, use the random start number to identify and select the first sampled case.

    2. Use the skip interval to select the subsequent documents for review. In other words, select the ‘kth’ case after the random start case, and continue selecting every ‘kth’ case thereafter.

    3. If the population of cases spans more than one day, then the skip interval must continue between days. Begin each new day’s count with the number of cases remaining following the last document selected from the previous day.

      Example:

      Assume a skip interval of "8" and that there were 5 cases remaining after applying the skip interval over Monday’s entire population. Then, continuing the skip interval sequence of 8 into the next day, the case count would begin at 6 on Tuesday. Therefore, the first case selected on Tuesday would be case number 3, the second would be case number 11, the third would be case number 19, etc.

    4. A random start number is used only once per planning period, even if the skip interval changes during the period.

    5. If the skip interval is "1" , every document must be reviewed.

    6. If the required sample size has been met, continue applying the skip interval through the last case in the population. This will ensure that all work has an equal chance of being selected.

  7. If the actual size of the population is not known prior to selecting the sample (or if a reliable estimate is unavailable), then a skip interval cannot be calculated and another method must be used to select the sample. One possible alternative is to manually spread the sample among the population. This method can be implemented using the following steps.

    1. Establish the time period of the estimate and the sample size.

    2. Using the sample size and the number of business days in the sampling period, determine the average daily sample size

    3. Depending on the flow of the work for this particular review, manually spread the weekly sample appropriately among the days of the week. If you expect the work to be distributed evenly across the week and hours of the day, then divide the sample evenly among the days of the week. In addition, if you expect the work to be distributed evenly throughout individual days, then the sample can be split evenly between the morning and afternoon hours. If, on the other hand, it is known that certain types of work are more likely to occur on certain days (e.g., Mondays, Tuesdays) or during certain times of day (e.g., afternoons), then the sample can be shifted accordingly to follow the workload more accurately.

    4. Attempt to spread the sample appropriately among each member of the team or unit performing the type of work being reviewed.

    5. When selecting cases for review during designated dates and times, use one of the methods in See IRM 21.10.1.3.1.1. (5) to incorporate an aspect of randomness.

      Example:

      If a case must be selected from the 50 cases handled by a particular employee on Tuesday morning, then a random number table can be used to select a random number between 1 and 50.

    6. Document all decisions made and procedures used throughout the process of manually spreading the sample across the population.

      Note:

      Because sampled cases are selected without the use of a skip interval, it is not automatically ensured that the sample is spread appropriately across the time period being measured and among the assistors included in the review. It is also not ensured that all cases in the population will have an equal chance of being selected. For these reasons, samples selected using the above procedures will have some amount of bias. Selecting the "most random sample possible" given local resources will help minimize this bias.

21.10.1.3.1.2  (10-01-2007)
Revising Sampling Instructions

  1. If the actual size of the population is not known prior to selecting the sample, a site may opt to use a skip interval based on an estimate of the population size. However, if the site experiences much higher or lower volumes than predicted, the site will end up either exceeding or falling short of its goal sample size. This could lead to straining or under utilizing available resources. To alleviate these situations, a site may use the following guidelines to change its skip interval mid-quarter.

    1. Use the procedures outlined in See IRM 21.10.1.3.1.1.(4) to calculate a new skip interval.

    2. The new skip interval may be implemented ONLY at the beginning of a sampling month, NEVER in the middle of a sampling month. Because NQRS generates weighted reports monthly, skip intervals must remain constant within any given month.

    3. Never simply grab extra cases, drop selected cases, seek out cases of special interest, or use different methods to select cases in the same sample. Each of these situations could lead to a sample that provides biased results.

    4. Any hand-picked cases that were not selected by the skip interval should only be used for local review and should never be included in the data sent to Headquarters.

  2. If a site requires additional data for error analysis and feedback, then it can sample extra cases by changing the skip interval accordingly. By setting the sample size higher than the goal established by the National Review, the resulting skip interval will be smaller than the one provided by Statistics of Income (SOI). This smaller skip interval should be used during the entire month and cannot be changed again until the beginning of the next month. With this approach, the entire sample can be reviewed and used in NQRS.

21.10.1.3.1.3  (10-01-2007)
Weighted Sampling

  1. In sampling, every sampled case represents a certain number of cases in the population. The exact number of cases a sampled case represents will depend on both the sample size and the actual size of the population from which it was selected. When a quality estimate is a combination of two or more separate samples (e.g., a fiscal year report for a single SPRG for a single site), it is necessary to account for the fact that each sampled case included in the overall estimate may not represent the same number of cases in the overall population. Weighting is used to ensure that every sampled case has the appropriate amount of influence on the overall cumulative estimate.

    Example:

    A quality estimate for a single SPRG in a single site for a planning period may consist of three individual samples, one from each month. Therefore, the planning period quality estimate is weighted by the three individual monthly SPRG volumes. This will make certain that each month’s influence on the planning period estimate is directly related to the total number of cases handled during that month.

  2. NQRS provides both weighted and unweighted estimates of quality.

  3. Unweighted estimates that combine more than one site, time period, or SPRG are not considered statistically valid. Such estimates should only be used internally. Their statistical limitations should be taken into consideration when basing business decisions on them.

21.10.1.3.2  (12-01-2000)
Quality Review Time Reporting

  1. See IRM 25.8, OFP Codes Overview, for appropriate Work Planning & Control (WP&C) Organization, Function, and Program (OFP) time reporting codes.

21.10.1.3.3  (10-01-2008)
Quality Review Records Retention

  1. IRM 1.15.31, Records Control Schedule for Customer Service provides specific guidelines on the retention period for national quality review records. National quality review records may be destroyed one year after the current processing year, or when no longer needed in current operations, whichever is earlier.

21.10.1.4  (12-11-2008)
Quality Review of Phone Calls

  1. CQRS monitors recorded contacts through Ultra 10 and occasionally performs remote monitoring for all Accounts Management toll-free phone SPRGs (except for National Taxpayer Advocate), and ACS Phones for Compliance Services as well as e-help Phones for Electronic Products and Services. Correspondence Program Analysis System monitors the AUR Phones, Exam Phones and Innocent Spouse Phones SPRGs. PAS monitors the remaining Compliance Services Phone SPRGs. The data from these reviews may be used for the Balanced Measures report. Not all SPRGs have balanced measures.

  2. Managerial reviews of these phone calls are not included in the Balanced Measures calculations; they are used in employee evaluative documentation, to plan workload distribution, and to identify training issues.

  3. The basic roles and responsibilities for phone analysts are the same across all phone product lines. Calls are rated based on percentage correct in each of the five buckets.

  4. Analysts may monitor "live" calls from a trunk line or application line or use the Contact Recording software to review calls from the previous day. Managers may choose either method to review a specific employee.

  5. If the reviews are based on "live" monitoring, the calls should be scheduled equally throughout the week covering all hours of operation for the site. Leave some time each day to make up missed work. This allows for flexibility should any unforeseen problems occur (e.g., downtime). Every effort should be made to complete each daily sample (for national reviews).

  6. Contact Recording may be used to evaluate the contact for some SPRGs. The analyst will include the Contact Recording identification number or Router Call Key on the DCI.

    1. The Verint Ultra 10 system of "Contact Recording" (CR) is a telephone tool used by Accounts Management, Electronic Products and Services Support, and Compliance Services to record incoming "toll free" telephone contacts for the purpose of possible subsequent quality review monitoring. Incoming calls are answered with an additional announcement that states, "Your call may be monitored or recorded for quality purposes." The Verint Ultra system records the audio, and occasionally capture the screen, of all telephone calls coming into the Service via the ASPECT Communication voice response unit and teleset. See IRM 21.1.1.10 for more information on Contact Recording.

    2. Managers and Quality Analysts use CR to perform required random reviews (performance and product) of incoming telephone contacts. Contact Recording allows for a more cost effective review as there is no lag time between calls. Calls recorded in this system are available for National review the next business day.

      Note:

      In the event that the Ultra 10 system is unavailable or there is a call volume variance less that 90.1%, the Quality Analysts may be required to manually monitor calls until the system is available or the variance is above 90.1%.

    3. Additionally, the system stores data by Standard Employee Identifier (SEID) for 45 days on calls that are not reviewed, 60 days on calls that are reviewed and a maximum of 18 months can be requested.

      Note:

      At this time the COIC Phones SPRG will not use the Contact Recording system because they are not using the ASPECT Communication teleset.

  7. When sampling with the Contact Recording system, all calls received by the Operation are included in the review universe, ensuring a more statistically valid review.

    Note:

    Statistic of Income Office (SOI) created the sampling algorithms which is included in the Verint Contact Recording System. The algorithm provides each site with a statistically valid sample per Product Line/SPRG. The PAS/National analysts use a Shared In-Box to retrieve their daily sample.

  8. On a call that was Contact Recorded, if the taxpayer requests that the recording be stopped (aka: "Stop on Demand" ) PAS and CQRS will not review the call. If a "Stop on Demand" call is randomly selected for the national sample, it will be systemically rejected and replaced by Contact Recording. Although the call will not be reviewed, if the employee ignores the request or if the system malfunctions and the recording continues, PAS or CQRS will inform management via E-mail that the taxpayer requested the recording be stopped but was not accommodated.

  9. Good listening and note taking skills are critical when monitoring a live call.

  10. When performing a telephone review, the analyst will use the employee's identification number provided on the call:

    • If the analyst is unable to capture the employee's identification number on the call, the last name of the employee, as captured during the review, will be entered into the Employee’s Name field.

    • CQRS will capture the employee's SEID (as it appears on CR from the PBXID field on Ultra 10) as the identification number.

      Note:

      If an employee's workstation is not configured properly, their extension will show and will be captured instead of the SEID. When reviews are performed through remote monitoring, the employees' identification number will be captured.

    • PAS/National analysts will use the following identifiers in the Employee Name field if the situation warrants:
      1.U – If PAS/National analysts are unable to determine both the ID # and the last name. For example, if the employee speaks too fast and the analyst is unable to decipher the ID # or name,
      2.N – If the employee does not give either an ID # or a name at any time during the conversation,
      3.I – If the analyst could not capture the full ID # or last name, but was able to get a portion of the ID # (less than 7 Digits). An example for this situation would be I - 17-985XX (last 2 digits not captured). If the ID number is given and the analyst got the first two numbers and could not decipher the middle two numbers but got the last 3, the analyst should indicate I - 17XX521 in the employee name section.

      Note:

      PAS/National analysts will enter the applicable letter and a dash and then the ID number or XX information. The entry will not contain spaces between the letter and the numbers. For example: I-17-98521 or I-17-XX521.

      Note:

      A simple way to capture the name every time is to have the site ASPECT System Analyst (SA) add a column next to the Line information to allow ASPECT to automatically show the User ID and/or name of the employee connected to that telephone line. It should only take the SA a few minutes to update the Custom View canvass.

  11. For a sample call to be counted as a phone review, the taxpayer does not have to remain on the line until all adjustment actions are complete. Even if the employee does not complete all work until the next business day, the call is still counted as part of the sample.

  12. If an Accounts Phone call subject becomes a Tax Law or ACS issue, or vice versa, code the complete call for Professionalism and Timeliness, and code any issue(s) addressed for Accuracy. If the call is transferred and no action was taken to resolve the taxpayer's issue, code the case for all applicable buckets except Customer Accuracy, which will be "not applicable." When this happens, code attribute 004 Call Transfer.

  13. The Master Attribute Job Aid (MAJA) for the phone Product Lines and SPRGs are located on the EQ Website. The EQ Website address is http://eq.web.irs.gov/ .

21.10.1.4.1  (10-01-2003)
Accounts Phones Product Line

  1. The Accounts Phones Product line consists of six Specialized Product Review Groups, (SPRGs). The six SPRGs are Employer Identification Number (EIN), General Account Calls, International, National Taxpayer Advocate (NTA), Priority Practitioner Support (PPS), and Spanish calls.

  2. Reviewing Accounts Phones allows us to monitor and improve the quality of responses to a taxpayer's questions about his/her account.

21.10.1.4.1.1  (10-01-2006)
Accounts Phones Measure

  1. Accounts Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.7.3. for more information on the measures.

21.10.1.4.1.2  (10-01-2003)
Definition of EIN Calls SPRG

  1. EIN calls include any questions received on the Aspect EIN application(s) such as.

    1. Any call relating to a taxpayer’s request for an Employer Identification Number (EIN);

    2. Any call regarding procedural issues (how to complete Form SS-4, where to fax/mail Form SS-4 etc.);

21.10.1.4.1.3  (10-01-2006)
Sample Procedures for EIN Calls

  1. Statistics of Income (SOI) Staff develops sampling plans for EIN Phone calls monitored at CQRS. Samples from CQRS are valid at the site level on at least a quarterly basis and nationally on a monthly basis.

  2. Contact Recording and the associated screen shots may be used by managers or CQRS to evaluate the contact.

  3. A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included;. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1

  4. Managerial reviews are not subject to a sampling plan.

21.10.1.4.1.4  (10-01-2006)
Definition of General Account Calls SPRG

  1. General Account Calls include any questions received on the Aspect Accounts Phone Balance Due, Advanced Accounts or Procedural applications for General Account Calls. This does not include calls received on the designated Spanish or International Account applications. General Accounts calls include:

    1. Any call relating to a taxpayer's account (Individual Master File (IMF) or Business Master File (BMF),

    2. Any call regarding entity information, the processing of a tax return, corrections to errors found during processing, or corrections resulting from adjustments or audit assessments,

    3. Any call regarding procedural issues (where to file a return, when and where to make payments, etc.).

21.10.1.4.1.5  (10-01-2008)
Sample Procedures for General Accounts Calls SPRG

  1. Statistics of Income Staff develops sampling for Account Phone calls monitored at CQRS. Samples from CQRS are valid at the site level on a monthly basis.

  2. Contact Recording and the associated screen shots may be used by managers or CQRS to evaluate the contact.

  3. A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1

  4. Managerial reviews are not subject to a sampling plan.

21.10.1.4.1.6  (10-01-2003)
Definition of International Calls SPRG

  1. International Calls include any questions received on the designated ASEPCT applications for International such as:

    1. Any international (foreign, non-resident, etc.) call relating to a taxpayer's account (IMF or BMF)

    2. Any international call regarding entity information, the processing of a tax return, corrections to errors found during processing or corrections resulting from adjustments or audit assessments,

    3. Any international call regarding procedural issues (where to file a return, when and where to make payments, etc.).

21.10.1.4.1.7  (10-01-2007)
Sample Procedures for International Calls

  1. SOI Staff develops a combined sample plan for International Tax Law and Accounts Phone calls monitored at CQRS. Sample size from CQRS is valid at the site level on a quarterly basis.

  2. Contact Recording and the associated screen shots may be used by managers or CQRS to evaluate the contact.

  3. A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1.

  4. Managerial reviews are not subject to a sampling plan.

21.10.1.4.1.8  (10-01-2003)
Definition of NTA Calls SPRG

  1. National Taxpayer Advocate calls include any calls relating to a taxpayer's account (IMF or BMF) received on the designated NTA ASPECT applications.

21.10.1.4.1.9  (10-01-2008)
Sample Procedures for NTA Calls

  1. Contact Recording and the associated screen shots may be used by managers to evaluate the contact.

  2. A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1.

  3. Managerial reviews are not subject to a sampling plan.

21.10.1.4.1.10  (10-01-2003)
Definition of PPS SPRG

  1. PPS Calls include any call from a tax practitioner relating to his or her client's/taxpayer's account (IMF or BMF) or any other questions received on the Aspect applications for PPS.

21.10.1.4.1.11  (10-01-2006)
Sample Procedures for PPS

  1. Statistics of Income Staff develops sampling plans for PPS calls monitored at CQRS. Samples from CQRS are valid at the site level on at least a quarterly basis and nationally on a monthly basis.

  2. Contact Recording and the associated screen shots may be used by managers or CQRS to evaluate the contact.

  3. A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1

  4. Managerial reviews are not subject to a sampling plan.

21.10.1.4.1.12  (10-01-2003)
Definition of Spanish Tax Law and Account Calls SPRG

  1. Spanish Calls include any questions received on the ASPECT applications for Spanish Calls including:

    • Any call relating to a tax law question from the taxpayer

    • Any call relating to a taxpayer's account (IMF or BMF)

    • Any call regarding entity information, the processing of a tax return, corrections to errors found during processing, or corrections resulting from adjustments or audit assessments

    • Any call regarding procedural issues (where to file a return, when and where to make payments, etc.)

21.10.1.4.1.13  (10-01-2007)
Sample Procedures for Spanish Account Calls

  1. Statistics of Income Staff develops a combined sampling plan for Spanish Tax Law and Spanish Accounts calls monitored at CQRS. Samples from CQRS are valid (which includes Spanish Accounts and Spanish Tax Law combined) at the national level on a quarterly basis. Statistical validity at the site level varies for Puerto Rico based on time of year. For all other sites combined, the statistical validity is quarterly.

  2. Contact Recording and the associated screen shots may be used by managers or CQRS to evaluate the contact.

  3. A site may want to perform local reviews to aid in the quality improvement of the product line. Local review sampling guidelines have been included. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1.

  4. Managerial reviews are not subject to a sampling plan.

21.10.1.4.2  (10-01-2002)
ACS Phones Product Line

  1. Automated Collection System (ACS) Phone calls are reviewed to measure and improve the quality of our responses to taxpayer inquiries about balance due and return delinquency accounts.

21.10.1.4.2.1  (04-08-2008)
Definition of ACS Phones Product Line

  1. ACS is a computerized inventory system of balance due accounts and return delinquency accounts after normal notice routines occur.

  2. ACS Phones calls are defined as any call received on an IMF or BMF account in 22 or Taxpayer Delinquency Investigation (TDI) status assigned to ACS, and any other calls received on the ASPECT ACS application.

  3. Incorrect/incomplete action (per IRM guidelines) which results in incorrect calculations must exceed a $5.00 threshold before charging a defect for the national review

21.10.1.4.2.2  (10-01-2007)
Sample Procedures for ACS Phones

  1. SOI Staff develops sampling plans for ACS Phones calls monitored at CQRS. Samples from CQRS are valid at the site level on a quarterly basis and nationally on a monthly basis.

  2. Local reviews are not performed for the national measure of the ACS Phones product line and therefore, are not included in the sampling plan. However, if time and staffing permits, each site should also perform local reviews to aid in the quality improvement of the product line. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1 for more information on local reviews.

  3. Contact Recording and the associated screen shots may be used by managers or CQRS to evaluate the contact.

  4. Managerial reviews are not subject to a sampling plan.

21.10.1.4.2.3  (10-01-2003)
ACS Phones Measures

  1. ACS Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy, and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.7.3 for more information on the measures.

21.10.1.4.2.4  (10-01-2009)
Roles and Responsibilities of the ACS Phones Analyst

  1. The Centralized Quality Review System (CQRS) Analyst for ACS Phones will complete an unbiased, consistent, and accurate review of ACS Phones including follow-up actions taken once the taxpayer has hung up. Even if the call site employee does not complete all work until the next business day, the call is still counted as part of the sample.

  2. ACS Phones Quality analysts will complete an NQRS Data Collection Instrument (DCI) for each case reviewed. All appropriate fields will be completed to indicate quality standards having been met or not met. Analysts’ narratives will provide the basis for their findings and include applicable IRM references for coded defects. Whenever Attribute 715 is coded with a defect, the driver must be indicated in parenthesis immediately following "715" in the Feedback/Summary Remarks on the DCI. NQRS reviews will provide a basis for defect analysis and reporting error trends.

    Note:

    Before Attribute 508 Appropriate Procedural Action/Answer is coded, the analyst should research to see if another attribute describes the action/answer given by the employee. Only use this attribute as a last resort. When it is decided to code Attribute 508 either "Y" or "N" , clearly explain in the Feedback Summary section of the DCI why this attribute was selected. If a defect is charged, clearly describe the defect and provide an IRM reference to support the coding.

  3. Enter the word FLASH in the Feedback Summary Remarks section of the DCI to identify a defect that requires immediate (by the next business day) corrective action by the operation. For example, recalling a notice/letter before it is issued or correcting an adjustment to an account. See IRM 21.10.1.7.7, EQRS/NQRS Remarks section for additional information.

    Note:

    Defects requiring correction that are not annotated with FLASH are to be completed by the Operation within five working days. See IRM 21.10.1.2.6.3.

  4. Review data will be input within 24 hours of review to NQRS.

  5. Refer to the Embedded Quality Website http://eq.web.irs.gov weekly to glean updated information on the use of attributes in the ACS Phones review, obtain the latest Master Attribute Job Aid (MAJA) and Quality Grams, review the Issues Log, etc.

  6. Consult the ACS Phones Product Line Analyst for coding assistance or to interpret attribute usage, whenever necessary.

21.10.1.4.3  (10-01-2008)
ASFR Phones Product Line

  1. ASFR Phones are reviewed to measure and improve the quality of responses given to taxpayer inquiries received on the ASFR and ASFR Refund Hold toll-free lines.

21.10.1.4.3.1  (12-11-2008)
Roles and Responsibilities of the ASFR Phone Analyst

  1. The ASFR Phone PAS Analysts will complete an unbiased, consistent, and accurate review of ASFR and ASFR Refund Hold calls.

  2. ASFR Phone PAS Analysts will complete an NQRS Data Collection Instrument (DCI) for each case reviewed. All appropriate fields will be completed to indicate quality standards having been met or not met. Analysts’ narratives will provide the basis for their findings and include applicable IRM references for coded defects. NQRS reviews will provide a basis for defect analysis and reporting error trends.

  3. Enter the word FLASH in the Feedback Summary Remarks section of the DCI to identify a defect that requires immediate (by the next business day) corrective action by the operation. For example, recalling a notice/letter before it is issued or correcting an adjustment to an account. See IRM 21.10.1.7.7, EQRS/NQRS Remarks section for additional information.

    Note:

    Defects requiring correction that are not annotated with FLASH are to be completed by the Operation within five working days. See IRM 21.10.1.2.6.3.

  4. Review data will be input daily to NQRS.

  5. In a monthly report to be shared with the ASFR Operation, provide suggestions for improvement by:

    • Identifying most frequently occurring defects,

    • Analyzing root causes of defects,

    • Verifying sampling plans and guidelines,

    • Reviewing methods used to capture needed information.

  6. It is also recommended that NQRS analysts meet at least quarterly with the EQRS analysts to confer, compare and review attribute usage. These meetings should be used as a forum to discuss and agree on the use of each attribute in the ASFR Phones Smart DCI. The Product Line Analyst should be consulted for assistance in interpretation of attribute usage when necessary.

21.10.1.4.3.2  (04-08-2008)
Definition of ASFR Phone Product Line

  1. ASFR Phones are defined as any call received as a result of ASFR issued 30 day (2566) and 90 day (3219) letters or Refund Hold (CP63) letters generated from IDRS.

  2. Contact Recording and the associated screen shots (when available) may be used by managers or PAS to evaluate the contact.

  3. Incorrect/incomplete action (per IRM guidelines) which results in incorrect calculations must exceed a $5.00 threshold before charging a defect for the national review.

21.10.1.4.3.3  (10-01-2008)
Sample Procedures for ASFR Phones

  1. Statistics of Income Staff develops sampling plans for ASFR Phones monitored by the sites. Samples from the sites are valid at the site level and national level on a quarterly basis.

  2. The National Review will consist of a daily random sample of calls covering the entire telephone operational day. See IRM 21.10.1.3.1 and See IRM 21.10.1.3.1.1 for sampling guidelines.

  3. EQRS reviews are not subject to a sample plan.

21.10.1.4.3.4  (10-01-2007)
ASFR Phones Measures

  1. ASFR Phones will be measured for Timeliness, Professionalism, Customer Accuracy, Regulatory/Statutory Accuracy and Procedural Accuracy. These are the measures that are available and may be reported under the Balanced Measurement System. See IRM 21.10.1.7.3 for more information on the measures.

21.10.1.4.4  (10-01-2005)
AUR Phones Product Line

  1. AUR Phones is a separate Automated Underreporter product line review in NQRS and is reviewed within the Embedded Quality system.

  2. AUR Phones is reviewed to measure and improve the quality of responses given to taxpayer/caller inquiries received on the AUR toll-free telephone lines.

  3. Analysts and managers will primarily use IRM 4.19.3, IMF Automated Underreporter, IRM 21.10.1 and the Attribute Matrix Job Aids.


Next

More Internal Revenue Manual