2.5.14 Quality Assurance

Manual Transmittal

February 22, 2023

Purpose

(1) This transmits revised IRM 2.5.14, Systems Development, Quality Assurance (QA).

Material Changes

(1) The following revisions were made to align with current Quality Assurance practices and incorporates the revisions made in the Interim Guidance Memo IT-02-04220004. They are as follows:

  • IRM 2.5.14.3.1 - Auditing Process Roles and Skills - AD Quality Assurance Program Office (Quality Specialist) was changed to QA Program Manager, Domain Director was changed to QA Program Staff, Domain POC was changed to Project/Program Manager, Project Manager/Branch Chief was changed to Project/Program Team, Project Team (Quality Analyst) was Project/Program Team

  • IRM 2.5.14.3.2 - Audit Process Control - Added 6 new Process Areas - Risk Management, Privacy, Section 508, Security, Quality Assurance, Process Management, Updated Process Audit to Process Area Audit and Changed Release Review Audit to Process Owner Audit.

  • IRM 2.5.14.3.3.1 - Plan to Audit - Updated QA steps to align with current QA Practices.

  • IRM 2.5.14.3.3.2 - Perform the Audit -Added The auditors coordinate their efforts to ensure consistency in auditing and The Audit Closing Meeting is conducted to communicate high level audit results, global findings, and next steps to the projects and programs.

  • IRM 2.5.14.3.3.3 - Manage Documents -Updated tasks to align with current QA Practices.

  • IRM 2.5.14.3.3.4 - Monitor and Control Audit Findings -Updated tasks to align with current QA Practices.

  • IRM 2.5.14.3.4 - Audit Process Review - Removed Number 2 - “At least quarterly, Program Management will report to senior management on the overall status of task covered by this process.”

Effect on Other Documents

IRM 2.5.14, dated June 30, 2010, is superseded. This IRM incorporates the revisions made in Interim Guidance Memo IT-02-0422-0004, which were made to align with current QA practices.

Audience

The audience for this IRM is all Applications Development (AD) personnel responsible for the development and maintenance of Agency software systems identified in the Enterprise Architecture. This IRM applies to all QA activities conducted on projects, including contractor QA functions. It establishes the policy for conducting QA activities and the responsibilities and authority for performing QA across the Applications Development organization.

Effective Date

(02-22-2023)

Nancy Seiger
Chief Information Officer

Program Scope and Objectives

  1. Purpose

    The purpose of this IRM is to provide the framework for conducting Quality Assurance Audit activities within Applications Development (AD). It establishes a standard context for Project Teams, including contractors working for the Applications Development organization, to participate in the QA audit process.

  2. Audience

    The audience for this IRM is all Applications Development personnel responsible for the development and maintenance of Agency software systems identified in the Enterprise Architecture. This IRM applies to all QA activities conducted on projects, including contractor QA functions. It establishes the policy for conducting QA activities and the responsibilities and authority for performing QA across the Applications Development organization.

  3. Policy Owner

    The Associate, Chief Information Officer (ACIO), Application Development establishes all Information Technology (IT) internal controls for this IRM.

  4. Program Owner

    The Application Development Director, Delivery Management and Quality Assurance is the program owner.

  5. Primary Stakeholders

    This policy applies to all IT projects and programs including contractors.

  6. Program Goals

    The objective of this IRM is to establish the overall approach to Quality Assurance, apply Quality Assurance standards, report and control requirements for the QA program as outlined by AD QA Directive and related processes and procedures.

Background

  1. GAO/AIMD/GGD-98-54, February 1998, and Clinger-Cohen Act of 1996

Authority

  1. GAO/AIMD/GGD-98-54, February 1998, and Clinger-Cohen Act of 1996.

Roles and Responsibilities

  1. Below is a list of QA Roles and Responsibilities:

    • QA Program Manager - Manages QA Program

    • QA Program Staff - Serves as liaison from QA Program to project/program personnel and Process Owners

    • Project/Program Manager - Manages Project quality

    • Project/Program Team - Produces quality software products, systems, and documentation

    • Process Owner - Owner of Specific Process Area (e.g., Configuration Management)

    • Enterprise Life Cycle (ELC) Coach - Provides assistance with ELC matters to Project/Program Teams

Program Management and Review

  1. Program Reports

    • Audit Report

    • Non-Compliance Summary Report

    • Corrective Action Plan (CAP)

    • Heat Chart

    • DMQA Weekly Status Report

    • Audit Team Daily Status Report

  2. Program Effectiveness

    Measurements show how well products and processes conform to organizational and industry standards, they also indicate how well projects follow documented processes. The Quality Assurance Program reports on process metrics, derived from the number of non-compliances identified during process audits and the number of projects audited to produce process measures. To keep senior management abreast of project status and process performance metrics, the Quality Assurance Program Office will generate, retain, and report process metrics in order to demonstrate Program effectiveness.

Program Controls

  1. The following list represents the IRS controls, and mandates applicable to AD projects:

    • IRM 2.5.1 Systems Development

    • IRM 2.5.14 Quality Assurance (QA)

    • IRM 2.16.1 Enterprise Life Cycle (ELC)

    • IRM 2.21 Introduction to Shopping Cart Processing for IT

    • IRM 2.22 Business Planning and Risk Management

    • IRM 2.25 Managed Service for IRS

    • IRM 2.100 Integrated Process Management

    • IRM 2.110 Requirements Engineering

    • IRM 2.120 Engineering

    • IRM 2.125 Change Management

    • IRM 2.126 Enterprise Organizational Readiness

    • IRM 2.127 Testing Standards and Procedures

    • IRM 2.144 Capacity Management

    • IRM 2.149 IT Asset Management

    • IRM 2.150 Configuration Management

    • IRM 2.152 Data Engineering

    • IRM 10.8.1 Information Technology Security Policy and Guidance

    • Federal Information Security Management Act (FISMA)

    • Privacy Act of 1974

    • Section 508 of the Rehabilitation Act of 1973

    • Risk, Issue, and Action Item Management Directive

    • Other internal standards are located on the Process owners' SharePoint sites

Terms/Definitions/Acronyms

  1. The following tables are a list of terms and acronyms used throughout this IRM Section.

    Defined Terms

    Term Definition
    Applications Development An organization that supports IT components of a system that utilizes IT resources to store, process, retrieve or transmit data or information using IT hardware and software.
    Clinger-Cohen Act The Clinger-Cohen Act of 1996 (40 U.S.C. 1401(3)), also known as the Information Technology Management Reform Act, was intended, to "reform acquisition laws and information technology management of the Federal Government.
    Configuration Management Establish and maintain the integrity of work products using configuration identification, configuration control, configuration status accounting, and configuration audits
    Corrective Action Plan Changes made to bring expected future performance of the project in line with the project plan
    Daily Metrics Daily status of Quality Assurance Program Staff
    Enterprise Life Cycle A framework that provides a workflow for projects to follow to move an IT solution from concept to production while making sure that they are in compliance with IRS guidelines and are compatible with the overall goals of the IRS
    Heat Chart Visual representation of compliance data that uses colors to illustrate project and Domain compliance
    Internal Revenue Manual Official communications that designate authorities and/or disseminate instructions to staff for IRS officials and employees
    Lessons Learned Practices for evaluating past performance of activities
    Non-Compliance Finding and/or weaknesses usually found in a quality audit
    Peer Review Defect and Resolution Report Used to document Peer Reviews of project artifacts.
     

    Acronyms

    Acronym Definition
    AD Applications Development
    ACIO Associate, Chief Information Officer
    CAP Corrective Action Plan
    CM Configuration Management
    DID Data Item Descriptions
    ELC Enterprise Life Cycle
    ESC Executive Steering Committee
    FISMA Federal Information Security Management Act
    IRM Internal Revenue Manual
    PRIV Privacy
    PRM Process Management
    PP Project Planning
    PMC Project Monitoring and Control
    QA Quality Assurance
    RSKM Risk Management
    RQEN Requirements Engineering
    SEC Security
    SWDEV Software Development
    SME Subject Matter Expert
    SM Supplier Management
    TEST Testing
    U.S.C. United States Code
     

Related Resources

  1. QA evaluation of the project processes throughout the life cycle is based on the processes defined by the following supporting documents:

    • AD Quality Assurance Directive

    • AD Quality Management Plan

    • AD Quality Assurance Plan

    • AD Quality Assurance Program (processes, procedures, etc.)

    • Internal Revenue Manual (IRM) 2.16 IRS Enterprise Life Cycle

    • IRM 2.5 Systems Development

Introduction

  1. This IRM provides the framework for conducting Quality Assurance Audit activities within Applications Development. It establishes a standard context for Project Teams, including contractors working for the Applications Development organization, to participate in the QA audit process. This IRM establishes:

    • the overall approach to Quality Assurance

    • the applicable Quality Assurance standards

    • the reporting and control requirements for the QA program as outlined by the AD QA Directive and related processes and procedures.

  2. The Quality Assurance (QA) Program Office supports the delivery of high-quality products and services by ensuring that projects implement a coordinated set of activities that conform to organizational policies, processes and procedures.

  3. Quality Assurance is a systematic, planned set of activities necessary to provide adequate confidence that the product conforms to stated customer requirements. The activities are designed to evaluate the processes (i.e. Project Planning, Project Monitoring and Control, Requirements Management, etc.) by which products are developed.

Quality Assurance Auditing Process

  1. QA Audit Process and Procedures are used to objectively and independently evaluate adherence of the process and work products to applicable directives, processes, standards, procedures, and guidelines. The objectives of the audit process are to:

    • identify and track noncompliance instances

    • communicate and facilitate the resolution of noncompliance issues

    • identify and communicate, to senior management, best practices and opportunities for improvement

    • document Quality Assurance activities; and

    • report quality issues to relevant stakeholders

  2. Benefits of the audit process are realized through:

    • consistency in assessing use of organizational processes

    • facilitation of improvements

    • enhanced planning and resource allocation capability

Auditing Process Roles and Skills

  1. To meet the objectives and realize the benefits of QA auditing, the following roles and skill sets are needed to perform QA auditing activities.

    Roles Responsibilities
    QA Program Manager Manages QA Program
     
    • Develops, updates, and maintains quality standards and procedures (e.g., plans, processes, activities, templates, checklists, and guidelines)

    • Approves or rejects changes to quality documents

    • Assigns tasks to QA Program Staff

    • Manages Audit Schedule

    • Evaluates auditor feedback and workload

    • Provides QA Program Staff with Audit Checklist

    • Reviews, resolves, or escalates issues to senior program management, if necessary

    • Responsible for the generation of Daily Metrics, Weekly Metrics, and Heat Charts

    • Coordinates quality training based on feedback

    QA Program Staff Serves as liaison from QA Program to project/program personnel and Process Owners
     
    • Functions as the Subject Matter Expert (SME) on QA program objectives and procedures

    • Updates quality documents as assigned by QA Program Manager

    • Attends necessary internal training

    • Tailors Audit checklist based on current standards and audit/project specifics

    • Ensures compliance with applicable IRS directives, processes, and organizational standards by conducting audits

    • Reports on quality assessments and the audit findings to QA Program Manager

    • Coordinates with other auditors to establish audit baseline.

    • Coordinates with Project/Program Team on non-compliance items and corrective actions, if necessary Escalates to QA Program Manager, if necessary

    • Receives and acts upon Daily Metrics

    • Schedules a QA Services Team meeting

    • Provides coaching, training, and mentoring to projects/programs

    • Participates in Lessons Learned and continuous improvement activities

    Project/Program Manager Manages Project quality
     
    • Commits to compliance with applicable IRS directives, processes, procedures, and organizational standards and their deliverables

    • Has working knowledge of the QA Program objectives and standards; and required deliverables

    • Reviews current QA Program standards and objectives to incorporate into Project Plan

    • Performs project level self assessments

    • Supports and participates in QA audits

    • Receives audit results from the QA Program and responsible for follow-up action, as appropriate

    • Develops strategy for resolution and ensures timely responses to deficiencies with a Corrective Action Plan (CAP)

    • Ensures that quality management activities are scheduled, documented, and performed in accordance with Project Plan.

    • Supports Peer Review activities by allocating personnel, facilities, and time for Peer Reviews

    • Identifies trends in peer reviews and responsible for follow up action, as appropriate

    • Tracks project progress

    • Attends QA Services Team Meeting to discuss non-compliance and resolution

    Project/Program Team Produces quality software products, systems, and documentation
     
    • Has working knowledge of QA objectives and standards

    • Submits request to change existing Quality documents

    • Executes quality related activities and produces deliverables

    • Utilizes external Quality documents for support

    • Participates and performs peer reviews, technical reviews, and quality assurance activities

    • Performs independent/self-assessments of project/program artifacts and work products

    • Conducts ELC and quality reviews within various stages of the project lifecycle

    • Supports and participates in audits

    • Reconciles Non-Compliance issues resulting from independent/self-assessments

    • Participates in Lessons Learned and continuous improvement activities

    • Attends quality related training and provides feedback

    Process Owner Owner of Specific Process Area (e.g., Configuration Management)
     
    • Participates in Lessons Learned and continuous improvement activities

    • Supports and participates in audits

    ELC Coach Provides assistance with ELC matters to Project/Program Teams
     
    • Provides assistance and feedback to projects/programs on organizational standards and actions

    • Collaborates with QA Program Staff to confirm non-compliance items

    • Attends QA Services Team Meeting to discuss non-compliance and resolution

Audit Process Control

  1. The audit process is controlled and driven by the Audit Checklists. The checklists are questionnaires used to gather data for the processes and products being audited and to evaluate the project’s level of compliance. The checklists are tailored based on the project or type of audit conducted. The Quality Assurance Program conducts audits on organizational process areas as outlined in the AD Process Framework contained in the IT Process list below.

  2. The process areas are:

    • Project Planning (PP)

    • Project Monitoring and Control (PMC)

    • Risk Management (RSKM)

    • Requirements Engineering (RQEN)

    • Software Development (SWDEV)

    • Testing (TEST)

    • Privacy (PRIV)

    • Section 508 (508)

    • Security (SEC)

    • Configuration Management (CM)

    • Supplier Management (SM)

    • Quality Assurance (QA)

    • Process Management (PRM)

    The Quality Assurance Program conducts three types of audits. The audit types are:

    • Process Area Audit - Evaluates adherence to the standards and procedures of a process area (e.g., Configuration Management) or a group of process areas (e.g., Engineering) within a program or project.

    • Work Product Audit - Reviews work products for conformance to the Enterprise Life Cycle (ELC), Data Item Descriptions (DID) and templates. Work Products can include any work products and deliverables, as well as the standards and/or procedures used to produce them.

    • Process Owner Audit- Evaluates Process Owner policies and procedures based on standards and requirements. This type of audit is performed upon request from the Process Owner or their management.

Auditing Process Tasks and Flow

  1. The following tasks constitute the flow of the QA auditing process:

    • Plan to Audit

    • Perform the Audit

    • Manage Documents

    • Manage and Control Audit Findings

Plan to Audit
  1. The QA Program Manager and QA Program Staff perform Audit Planning activities in conjunction with the overall planning activities. Project/Program Managers and Project/Program Teams support audit planning activities. The following steps describe the audit planning activities:

    • Establish QA Program goals using inputs from previous audits

    • Plan and develop annual QA Audit Plan including audit calendar

    • Plan and develop QA Audit Schedule for each audit

    • Assign projects/programs to auditors

    • Tailor Audit Checklist based on type of audit

    • Coordinate among auditors to ensure consistency in auditing

Perform the Audit
  1. This task is performed in accordance with the Audit Schedule. The following steps occur during this task:

    1. Projects and programs are notified of the audit and provide access to the project/program repository (if required).

    2. The auditor reviews the Health Assessment and ReadMe File and makes a preliminary assessment of the quality status of the project/program.

    3. The Audit Opening Meeting is conducted to communicate the scope and the objectives of the audit to the projects and programs.

    4. The auditor, guided by the appropriate Audit Checklist, will evaluate the level of compliance with standards and processes as well as evaluate associated work products by reviewing instructions and procedures, checking records and through observation.

    5. If necessary, project/program staff will be consulted to address questions that arise during the checks and observations.

    6. The auditors coordinate their efforts to ensure consistency in auditing.

    7. The auditor documents the findings and confirms accuracy with other auditors and (if necessary) ELC Coaches.

    8. The Audit Closing Meeting is conducted to communicate high level audit results, global findings, and next steps to the projects and programs.

    9. The auditor issues to the auditee a final report of findings and stores it in the appropriate repository based on type of audit.

    10. The auditor conducts a QA Services Team meeting with the auditee to discuss audit findings and recommend potential corrective actions. A QA Services Team is not conducted if no deficiencies are found by the auditor.

    11. When audit findings require corrective action, a CAP must be submitted by the project/program.

Manage Documents
  1. This task occurs after an audit is conducted or may occur anytime a document is established, or content/information is gathered relating to the audit process. The following activities occur during this task:

    • All documents generated and/or received as a result of an audit are collected and stored in soft and/or hard copy in the appropriate repository.

    • Changes to QA process assets (i.e., processes, procedures, templates) shall be controlled in accordance with the Document Management Procedure located in the QA Program’s shared repository.

  2. As part of this task, when new QA documents are developed and/or requests have been received to modify existing documents, the following activities are executed:

    • The QA Program Office reviews requested changes internally.

    • The QA Program Office implements requested changes, updates repositories, and notifies users of changes to document (if approved).

    • If the change is rejected, the QA Program Office informs the requestor that there will be no change.

Monitor and Control Audit Findings
  1. The auditor will use the information from the final report and any subsequent actions to update the QA Program database. Using this data, the auditor tracks audit findings to closure. If corrective actions are not completed by the resolution date as outlined in the approved CAP, the auditor escalates the unresolved findings to the QA Program Manager for resolution. This is not required for Process Owner audits.

  2. The auditor’s responsibilities are to:

    • Verify resolution of the corrective action.

    • Review updates to the QA Program Database to indicate the status of the audit’s corrective actions.

    • Identify and escalate corrective actions that remain unresolved five days after planned resolution date.

    • Perform trend analysis activities.

    • Prepare audit data and status reports for review activities.

    • Place reports in the appropriate repository.

  3. When all findings are resolved and verified, the auditor updates the data repositories, (i.e., shared drive, QA Program database).

Audit Process Management

  1. AD Quality Assurance Program Office will regularly maintain measurements on the status and progress of Quality Assurance tasks for the AD portfolio. Process trends shall be analyzed for efficiency and effectiveness.

  2. Data will be compiled to develop and report trends in performance and compliance. The reporting will occur through performance trends metrics and compliance metrics. Performance and compliance trends metrics will be reported:

    • At the AD portfolio, domain, and project levels; and

    • For all process areas (by AD portfolio, domain, project levels).

Audit Process Review

  1. The QA Program Management will report to Senior Management on the overall status of tasks covered by this process on a regular basis, but no less than annually.