7.11.3 Tax Exempt Quality Measurement System (TEQMS)

Manual Transmittal

July 15, 2019

Purpose

(1) This transmits revised IRM 7.11.3, Employee Plans Determinations Letter Program, Tax Exempt Quality Measurement System (TEQMS).

Material Changes

(1) Updated IRM 7.11.3.5, TEQMS Case Selection Process, to state that use of closing code 00 was discontinued and that Technically Screened Cases are also known as EP Merit Cases.

(2) Updated IRM 7.11.3.6, Statistical Validity Level/TEQMS Sample Selection, to replace reference to Rulings and Agreements Planning and Programs with reference to the Compliance Planning and Classification Group.

(3) Updated IRM 7.11.3.8, Consistency Reviews to Ensure Data Reliability, to eliminate reference to certain responsibilities of the primary reviewer on cases selected for review.

(4) Updated IRM 7.11.3.9.1, Case Return Criteria, to clarify that QA reviewers may not correct administrative-type errors on closing letters, and that letters superseding determination letters may not be issued on TEQMS cases.

(5) Updated IRM 7.11.3.9.2, Procedures for a Returned Case, to clarify that cases will only be reopened with managerial approval.

(6) Updated throughout for editorial and plain language changes.

Effect on Other Documents

This supersedes IRM 7.11.3 dated January 5, 2018

Audience

Tax Exempt and Government Entities
Employee Plans

Effective Date

(07-15-2019)

Robert S. Choi
Director, Employee Plans
Tax Exempt and Government Entities Division

Program Scope and Objectives

  1. Purpose: This IRM describes the Tax Exempt Quality Measurement System (TEQMS) and the TEQMS process for Tax Exempt and Government Entities (TE/GE) Employee Plans (EP) Determinations specialists and managers.

  2. Audience: EP Determinations and Quality Assurance (QA) staff

  3. Policy Owner: Director, EP

  4. Program Owner: EP

  5. Program Goals: The goal of EP Determinations is to ensure that plans comply with the tax laws by reviewing applications for determination letters, opinion letters, and advisory letters, and to protect the public interest by applying the tax law with integrity and fairness to all.

Background

  1. TEQMS is designed to measure the quality of the work product demonstrated by TE/GE employees using established quality criteria. It serves as the quality measurement in the business results part of the Balanced Performance Measurement System (BPMS).

  2. TEQMS results give interim quarterly and annual reports to management to identify improvement opportunities. Management is primarily responsible for analyzing TEQMS reports and taking appropriate action. TEQMS is designed to achieve statistical validity at the national level for EP Determinations.

  3. TEQMS for EP Examinations is in IRM 4.71.15, Employee Plans Examination of Returns, EP Special Review.

  4. This IRM includes:

    • Overview of TEQMS, including changes from the BPMS

    • Quality Assurance Staff/Role of TEQMS Analyst

    • Case selection process

    • Review and data submission process

    • Description of reports

    • Tools for assessing data reliability

    • Case return process

Acronyms

  1. These acronyms are used in this IRM:

    Acronym Term
    BPMS Balanced Performance Measurement System
    CJE Critical Job Elements
    EP Employee Plans
    EIN Employer Identification Number
    EDS EP/EO Determination System
    QA Quality Assurance
    RRA '98 Restructuring and Reform Act of 1998
    TE/GE Tax Exempt and Government Entities
    TEQMS Tax Exempt Quality Measurement System

History

  1. The enactment of the IRS Restructuring and Reform Act of 1998 (RRA '98) required the IRS to change its quality standards. The BPMS, which includes quality, is one of five key RRA' 98 modernization "levers of change." The levers are:

    1. Revamped business practices

    2. Four operating divisions

    3. Management roles with clear responsibility

    4. Balanced measures of performance

    5. New technology

  2. The balanced measures of performance lever is broken down into three-parts and used to assess organizational performance.

    • Business Results

    • Customer Satisfaction

    • Employee Satisfaction

  3. RRA '98 increased the IRS's awareness of the need to focus on customer service and customer satisfaction. This increased awareness has been incorporated into TEQMS.

TEQMS

  1. TEQMS is a/an:

    • Important component of the BPMS.

    • Organizational measurement.

    • Measure of quality as it relates to cases.

    • Indicator of improvement opportunities.

    • Tool for assessing components of merit and non-merit cases.

    • Calculator of quality scores for evaluation at the national level.

    • Method of providing interim quarterly and annual quality reports.

  2. TEQMS is not evaluative at the:

    • Area level.

    • Group level.

    • Employee level.

  3. In addition to serving as an organizational quality measure, TEQMS reviews result in the following benefits:

    • Improved customer service and customer satisfaction.

    • Reduced number of cases that Appeals returns for additional work.

    • Improved suspension rate in Appeals.

    • Increased consistency in determination reviews.

    • Created a way to increase communication.

  4. Quality results are reported in several high level documents, such as:

    • IRS Congressional Justification

    • TE/GE Strategy and Program Plan

    • TE/GE Business Performance Review Process

    • IRS Annual Performance Plan

    • Commissioner's Monthly Report

    • IRS Management Board Report

    • Executive Management Support System

  5. The success of TEQMS requires a joint effort between EP Determinations management and specialists.

EP Determinations Quality Assurance (QA)

  1. EP Determinations QA reviewers perform all TEQMS reviews and the program is administered by a TEQMS analyst.

  2. QA reviewers are senior determination specialists who have extensive knowledge of tax laws, guidance and policies for determination cases. QA reviewers must be objective and exercise independent judgment and individual initiative in performing their duties. They review EP Determinations cases and report the results in an electronic checksheet called a "survey." They are expected to communicate effectively with other specialists, managers, taxpayers and external customers to resolve issues in completing their assignments.

  3. The TEQMS analyst's role includes: overseeing the TEQMS sample selection process, monitoring case receipts for the sample selection, preparing interim quarterly and annual reports, and implementing consistency reviews and validity reviews, as appropriate.

TEQMS Case Selection Process

  1. The following cases are not included in the population for the TEQMS sample selection:

    • Volume submitter or master and prototype specimen plans.

    • Pooled/group trusts.

    • Non-lead multiple employer plans.

    • Cases that are not worked to completion (e.g., returned incomplete, withdrawn by taxpayer, substantially deficient, cases re-selected for TEQMS review, and correction disposal).

    • Auto-closure cases (closing code 13).

  2. EP Determination cases subject to TEQMS review are divided into two groups. EP Determinations management determines the type of cases included in each group.

    • EP Determination cases, also known as non-merit cases, closed with closing code 01. Use of closing code 00 was discontinued after September 28, 2015.

    • EP Merit cases, also known as Technically Screened cases, closed with closing codes 06 and 09.

      Note:

      The group's proposed closing code determines the type of review classification.

  3. Each Employer Identification Number (EIN) and plan number is treated as a separate unit in the population.

  4. Mandatory review cases (see IRM 7.11.9.2, Mandatory and TEQMS Case Reviews, paragraph 3) are included in the population of cases that are subject to the TEQMS selection process.

Statistical Validity Level/TEQMS Sample Selection

  1. At the beginning of each fiscal year, the TEQMS analyst determines the approximate sample size of each group described above based on the number of cases expected to be closed during the year based on information provided by the Compliance Planning and Classification group. The TEQMS analyst calculates an annual sample size for each group using statistical calculations with a 95 percent level of confidence and plus/minus 5 percent margin of error. Case review results are statistically valid at the national level; the TEQMS analyst computes the actual precision of the quality data at year-end.

  2. The EP/EO Determination System (EDS) randomly selects cases for the samples during EDS's closing process. The TEQMS analyst monitors the sample selection rate during the year and adjusts it as appropriate, typically once per quarter.

TEQMS Review Process/Quality Standards

  1. QA reviewers review cases in a closed status then complete the survey.

  2. The case may be returned to the specialist under IRM 7.11.3.9, Case Return Criteria Overview, after the QA reviewer completes the survey based on their initial review.

Data Capture Process

  1. QA reviewers evaluate cases and complete a survey (electronic checksheet), that contains the case review criteria. The survey has two sections:

    1. The administrative part which contains review information, case return criteria and case information.

    2. The quality standards/elements part which contains quality standards, elements, aspects, and reason codes.

      Note:

      The survey for EP Merit cases contains only quality elements and reason codes.

  2. Instructions to complete the surveys, and guidelines to apply the quality standards, elements, aspects, and reason codes, can be found in the TEQMS - Tax Exempt Quality Measurement System - EP Determinations Student Guide, catalog number 86711Q, at: http://core.publish.no.irs.gov/trngpubs/pdf/t4317-002--2005-10-00.pdf. Minor changes have been made to the surveys over time to either expand or supplement some aspects and reason codes for clarification purposes so the current surveys are not the same as in the guide.

Data Submission/TEQMS Database

  1. QA reviewers electronically submit the survey into the TEQMS database when they complete their case review.

  2. The database performs a series of validity checks before the system accepts the survey's data.

TEQMS Reports

  1. Each quarter, EP Determination and EP Merit surveys are summarized in various reports. The reports identify:

    • Measures of overall performance on quality standards.

    • Potential improvement areas.

    • Feedback on specific root causes.

    • Potential training/CPE needs.

  2. The various reports do not provide individual, group or area evaluative information.

  3. The reports include:

    • Percentage Yes (answers) by Standard/Average Total Score - this reflects the percentage of yes answers (standard was met) for each standard and the organizational score.

    • Percentage Yes (answers) By Standard/Average Total Score by Quarter - this reflects the percentage of yes answers for each standard and the organizational score for each quarter and cumulative fiscal year.

    • National Quality Standards - this reflects the percentage of yes answers for each standard/element/aspect.

    • National Quality Standards Detail - this reflects the number of yes, no, and n/a answers for each standard/element/aspect and reason codes/process measures.

    • Case Return Criteria - this reflects the number of cases returned under return criteria.

Report Analysis

  1. The TEQMS analyst evaluates the database's reports and prepares trend reports. The reports describe areas where EP Determinations has or has not done well, provides suggestions for improvement and analyzes trends based on the answers on the survey. A separate report is prepared for EP Determination cases and EP Merit cases, but a combined report is acceptable.

Quality Score

  1. The BPMS requires case quality to be scored numerically. This measure is an indicator of the IRS's progress in achieving high-level strategic goals. TEQMS offers a rating system that best measures the organization's case quality.

  2. TEQMS is distinguished from the Critical Job Elements (CJE) Performance Plan. CJE Performance Plans are documents that describe to employees the work performance expected of them and the performance standards applied to their work performance. Manager case reviews should assess an individual's strengths and weaknesses based on the CJE's. Although the TEQMS standards are closely associated with the CJE's, performance evaluation is not based on TEQMS criteria.

  3. The TEQMS database computes the organizational score based on the reviewer's answers as to whether the case met each standard (yes or no). EP Determination cases have six quality standards. Some standards contain elements and/or aspects. The maximum possible score is 100. EP Merit has five quality elements, which may contain reason codes. The maximum possible score is also 100. See the TEQMS - Tax Exempt Quality Measurement System - EP Determinations Student Guide, catalog number 86711Q, for more information about quality standards, elements and aspects at: http://core.publish.no.irs.gov/trngpubs/pdf/t4317-002--2005-10-00.pdf

    The organizational quality score for all EP Determination cases = Total Number of Standards Passed / Total Number of Standards Rated X 100

    The organizational quality score for all EP Merit cases = Total Number of Elements Passed / Total Number of Elements Rated X 100

    Example 1:
    During the fiscal year, TEQMS samples 400 EP Determination cases. The EP Determination survey has six quality standards. Therefore, the total number of standards rated equals 2,400 (400 x 6). Of the 2,400 standards rated, only 1,200 standards are passed (answered "Yes" ).
    The organizational quality score is 50 = (1,200 / 2,400 X 100)
    Example 2:
    During the fiscal year, TEQMS samples 400 EP Merit cases. The EP Merit case survey has five quality elements. Therefore, the total number of elements rated equals 2,000 (400 x 5). Of the 2,000 elements rated, 1,800 elements are passed (answered "Yes" ).
    The organizational quality score is 90 = (1,800 / 2,000 X 100)

Consistency Reviews to Ensure Data Reliability

  1. To encourage greater consistency between reviewers in applying TEQMS criteria to cases, the Manager, EP Determinations QA, is responsible for doing consistency reviews as needed. The TEQMS analyst is responsible for scheduling and administering these reviews.

  2. The analyst selects at least one EP Determination case and at least one EP Merit case that each reviewer rates independently. Each reviewer prepares a checksheet similar to the survey forms. The analyst receives a copy of each reviewer's checksheet and prepares a summary of their answers.

  3. The analyst schedules a meeting with reviewers to discuss the summary document with an emphasis on the standards, elements and key aspects where reviewers differ. The goal is that the group will reach a consensus on the differences. If this is not possible, the analyst notes it in a report and recommends appropriate action for reviewers, such as additional training or clarification of a procedure.

  4. The analyst may prepare a report on the consistency review and forward it to the Manager, EP Determinations QA for approval. After approval, the analyst files the report and disseminates it as deemed appropriate. On a rotational basis, at least one reviewer will be the primary reviewer on cases selected for the review. The Manager, EP Determinations QA is responsible to act if the consistency review results indicate that additional training or improved procedures are required.

Case Return Criteria Overview

  1. This section describes:

    1. Criteria for returning cases to the specialist for further development.

    2. Instructions to the QA reviewers for returning TEQMS cases to specialists.

    3. Type of feedback prohibited from TEQMS.

Case Return Criteria

  1. QA reviewers may correct administrative-type errors on EDS closing documents but not closing letters. Cases are reviewed in a closed status and determination letters are mailed prior to cases being sent to reviewers. Superseding letters may not be issued on TEQMS cases.

  2. Reviewers will return cases to specialists for further review with approval from QA management, if there is clear evidence that an incorrect determination has been reached. This applies when the qualification of a retirement plan results in a significant negative impact on participants' benefits.

    Note:

    Cases that involve user fee issues will be returned.

Procedures for a Returned Case

  1. Cases will be re-opened under established guidelines, with QA managerial approval, if a reviewer disagrees with the group’s recommended case closure. The case file will be returned for correction and/or further development with a completed Form 5456, Reviewer's Memorandum - TE/GE, Employee Plans Determinations, approved by the Quality Assurance Manager.

  2. If the specialist receives a case from QA, they should correct any issues and complete a Form 5457, Response to Reviewer's Memorandum - TE/GE, explaining corrections or disagreements with all of the reviewer listed issues. Attach the form to the case and forward it to the group manager for approval before returning the case to QA.

Feedback Prohibited from TEQMS

  1. Reviewers may no longer give feedback on TEQMS cases because it is an "organizational" measure, not an individual, group or area performance review.