Compliance Evaluation Report 121509

Compliance Enforcement, Registration, and
Certification Program
Process Evaluation Report
November 23, 2009
Submitted to:
North American Electric Reliability Corporation
116-390 Village Boulevard
Princeton, New Jersey 08540
Report prepared by:

Crowe Horwath LLP
70 West Madison Street, Suite 700
Chicago, Illinois 60602-4903
Compliance Enforcement, Registration and Certification
Table of Contents
Executive Summary ....................................................................................................................................... 3
Section 1: Overview ...................................................................................................................................... 8
Project Background ...................................................................................................................................... 8
Process Evaluation Methodology ............................................................................................................... 14
Purpose of Report ...................................................................................................................................... 16
Document Overview ................................................................................................................................... 17
Disclaimer of Confidentiality ....................................................................................................................... 17
Section 2: Observations and Recommendations Summary ................................................................... 18
Introduction................................................................................................................................................. 18
The Process-driven Organization............................................................................................................... 18
Process Governance and the Process Foundation Summary Observations ............................................ 20
Overarching Observations and Recommendations ................................................................................... 21
Categorization of Recommendations ......................................................................................................... 37
Section 3: Cross-Functional Areas Evaluation ........................................................................................ 42
Introduction................................................................................................................................................. 42
3.1. Compliance Program Confidentiality Requirements .......................................................................... 42
3.2. Developing and Overseeing the Compliance Training Program........................................................ 43
3.3. Developing and Disseminating Compliance Process Directives and Bulletins .................................. 44
3.4. Processing Reliability Standards Violations ....................................................................................... 45
Section 4: Functional Area Evaluation ...................................................................................................... 47
Introduction................................................................................................................................................. 47
4.1. Compliance Program Planning .......................................................................................................... 48
4.2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System ............................. 54
4.3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System ............................. 60
4.4. Overseeing Compliance Activities of Regional Entities (excluding CVIs) .......................................... 65
4.5. Overseeing Enforcement Activities of Regional Entities .................................................................... 76
4.6. Analyzing and Reporting Compliance Information ............................................................................. 83
4.7. Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs ........................ 88
4.8. NERC Involvement in Compliance Inquiries and Violation Investigations ......................................... 94
4.9. Handling Complaints ........................................................................................................................ 101
4.10. Executing Compliance Enforcement Authority Responsibilities .................................................... 105
Appendix I – Functional Area to Processes and Procedures Crosswalk ............................................. 114
Appendix II – Process Questionnaire ....................................................................................................... 117
Appendix III – Observations and Recommendations from Development of Agreed-Upon
Procedures .................................................................................................................................................. 118
Appendix IV – Excerpt from Management Letter to NERC ..................................................................... 127
AFFILIATES – Crowe Horwath LLP is a member of Crowe Horwath International, a Swiss association. Each member firm of Crowe Horwath
International is a separate and independent legal entity. Crowe Horwath LLP and its affiliates are not responsible or liable for any acts or
omissions of Crowe Horwath International or any other member of Crowe Horwath International and specifically disclaim any and all
responsibility or liability for acts or omissions of Crowe Horwath International or any other member of Crowe Horwath International. Crowe
Horwath International does not render any professional services and does not have an ownership or partnership interest in Crowe Horwath
LLP. Crowe Horwath International and its other member firms are not responsible or liable for any acts or omissions of Crowe Horwath LLP and
specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath LLP. © 2009 Crowe Horwath LLP
Compliance Enforcement, Registration and Certification
Table of Figures
TABLE 1 – PROJECT APPROACH PHASE 1................................................................................................................9
TABLE 2 – PROJECT APPROACH PHASE 2..............................................................................................................10
TABLE 3 – CERCP PROCESS EVALUATION FINAL SCOPE..........................................................................................12
TABLE 4 – CMEP PROCESSES AND PROCEDURES...................................................................................................13
FIGURE 1 – LEVEL OF EVALUATION......................................................................................................................14
TABLE 5 – POLICY, PROCESS, AND PROCEDURE DEFINED ........................................................................................15
TABLE 6 – THE INFRASTRUCTURE FOR PROCESS SUCCESS ........................................................................................19
TABLE 7 – RECOMMENDATION CATEGORIES .........................................................................................................37
TABLE 8 – RECOMMENDATIONS SUMMARY BY CATEGORY OF RECOMMENDATION .....................................................40
TABLE 9 – RECOMMENDATIONS COUNT BY SECTION, BY CATEGORY .........................................................................41
AFFILIATES – Crowe Horwath LLP is a member of Crowe Horwath International, a Swiss association. Each member firm of Crowe Horwath
International is a separate and independent legal entity. Crowe Horwath LLP and its affiliates are not responsible or liable for any acts or
omissions of Crowe Horwath International or any other member of Crowe Horwath International and specifically disclaim any and all
responsibility or liability for acts or omissions of Crowe Horwath International or any other member of Crowe Horwath International. Crowe
Horwath International does not render any professional services and does not have an ownership or partnership interest in Crowe Horwath
LLP. Crowe Horwath International and its other member firms are not responsible or liable for any acts or omissions of Crowe Horwath LLP and
specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath LLP. © 2009 Crowe Horwath LLP
Compliance Enforcement, Registration and Certification 3
Executive Summary
Project Objectives
North American Electric Reliability Corporation (“NERC”) determined the need for a project to
provide NERC with an evaluation of its Compliance Enforcement, Registration and Certification
Program (“CERCP”) processes and procedures. NERC engaged Crowe Horwath LLP to perform
this evaluation and Crowe completed this project between July and October, 2009.
The project was initiated to assist NERC’s Compliance area (“NERC Compliance” or the “NERC
Compliance Department”) in achieving its overall objectives for effective implementation of the
CERCP, including adequate management controls . The project objective, therefore, was to
identify and document whether the program has adequately implemented applicable CERCP
processes and procedures in accordance with the applicable law, FERC orders, and NERC Rules
of Procedure. Additionally, Crowe reviewed the internal processes and procedures used by the
Compliance Department in carrying out its duties for consistency with the Rules of Procedure
and for completeness and effectiveness.
Project Approach
For purposes of planning, tracking, and execution, the project was divided into two separate,
sequential phases where the outputs from Phase I became key inputs to Phase II activities.
Phase I of the project primarily involved (i) conducting necessary project initiation and planning
activities, and (ii) gathering information from NERC Compliance personnel concerning the
processes that NERC’s Compliance Department has in place over the compliance with and
enforcement of approved electric reliability standards. Phase II of the project involved (i)
performing analysis and review of process and procedure information and artifacts gathered in
Phase I, (ii) preparation of the public report and the confidential letter to management, (iii)
review and revisions to the reports based upon feedback, and (iv) final delivery of the reports
and project closeout.
Project Scope
Four cross-functional areas and ten functional areas comprise the final scope of the CERCP
process evaluation and, therefore, the scope of this report. Cross-functional areas are areas
that underlie all CERCP processes – for example, confidentiality requirements. Functional areas
represent groupings of related processes, frequently for purposes of mapping related processes
back to a unit or basic responsibility of the program – for example, registration, certification,
CVIs, and enforcement are all functional areas. The 37 processes defined by the NERC
Compliance Department’s CMEP Processes and Procedures Manual are all encompassed within
these 14 cross-functional and functional areas. The CMEP Processes and Procedures Manual is
an internal set of procedures developed and maintained by NERC’s Compliance department to
assist in the implementation of the compliance enforcement, registration and certification
program.
Cross-Functional Areas
1. Compliance Program Confidentiality Requirements
2. Developing and Overseeing the Compliance Training Program
3. Developing and Disseminating Compliance Process Directives and Bulletins
4. Processing Reliability Standards Violations

Functional Areas
1. Compliance Program Planning
2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System
3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System
4. Overseeing Compliance Activities of Regional Entities (excluding CVIs)
5. Overseeing Enforcement Activities of Regional Entities
6. Analyzing and Reporting Compliance Information
7. Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs
8. NERC Involvement in Compliance Inquiries and Compliance Violation Investigations
9. Handling Complaints
10. Executing Compliance Enforcement Authority Responsibilities
Purpose of Report
The purpose of this report is to provide NERC with an evaluation of its CERCP processes and
procedures across the 14 cross-functional and functional areas identified above. This report,
submitted by Crowe Horwath LLP, represents the culmination of activities performed on the
project.
The primary objective of the report is to provide observations as to whether the program has
adequately developed and implemented applicable CERCP processes and procedures, where
“adequacy” is defined by those criteria identified in the Process Evaluation Methodology section
of this document, and to make recommendations where the implementation of the CERCP
processes and procedures can be improved.
Process Governance and the Process Foundation
In summary, our observations regarding the governance and foundational layers of the NERC
Compliance process environment are as follows:
▪ As a regulatory entity, NERC is by its very nature compelled to maintain an environment
focused on the creation, compliance, and enforcement of its standards and rules. We
observed that the NERC CERCP program generally has the governance and “tone at the top”
to be successful with its processes. Our assessment of individual functional areas indicates
that process objectives are typically well known and well understood and that there is
clearly a culture of policy and process adherence.
▪ As part of our analysis we placed NERCs CERCP into appropriate context from the standpoint
that NERC’s Compliance organization and the purpose, roles, and scope of responsibilities
for that organization has existed in their current state only for a relatively very short period
of time. The relative immaturity of the organization certainly has a bearing on the
expectations for its level of process maturity. For example:
o We observed in our analysis that the organizational structure, and the resulting roles
and responsibilities within that structure, continue to mature and change fairly
frequently as the Compliance area has undergone numerous structural changes within
the past two to four years. Three years ago the Compliance organization shifted from a
“Service Organization” whose purpose was to provide technical assistance to a

“Regulatory Organization” whose purpose was to regulate (i.e. compliance
enforcement, in addition to the role of registration and certification). The changes in
scope of responsibilities and assignment of responsibilities within an organization
certainly create challenges when attempting to get to a level of process maturity.
o We observed that the NERC Compliance Director/Manager-level positions are staffed, in
most cases, by personnel that are relatively new to the NERC organization. Of the six (6)
Director/Manager-level positions reporting up through the Vice President of Compliance
the average length of tenure for the personnel is less than 40 months. If you filter out
the one Manager with significant tenure (i.e. greater than five years), we find that the
average Director/Manager in Compliance has been with the organization just over two
years.
o The newness of staff to their respective positions certainly impacts expectations with
respect to process documentation. Organizational and process problems and
inefficiencies are being addressed by NERC Compliance personnel (e.g. Compliance has
stood up 35+ processes in the past two years), but organizational and process best
practices emerge typically once some degree of longevity and critical mass has been
achieved. Procedurally, NERC’s Compliance area has achieved a great deal despite their
relatively short existence as an organization.
▪ We observed a number of areas (explained further in subsequent sections of this report)
where the NERC CERCP can improve its “process foundation”. It is our observation that a
number of these areas are a result of the NERC Compliance area’s relatively short duration
of existence and immature organizational infrastructure and, therefore, process
infrastructure. For example:
o Both the Rules of Procedures (ROP) and the NERC Compliance Monitoring and
Enforcement Program (“CMEP”) Processes and Procedures Manual can be significantly
upgraded to provide a more solid operational foundation. A number of enhancements
and changes are needed to the ROP and we outline those in this report. We also found
that the internal CMEP Processes and Procedures were substantially less mature than
the ROP and will require a great deal of attention to reach a point where they are
documented in a manner where the tieback to the ROP is more obvious, consistent
across the Processes and Procedures themselves, and adequate to provide the ultimate
level of management control needed. Generally, the CMEP Processes and Procedures
Manual needs better defined roles and responsibilities, timelines, and outcome-based
measurements.
o Tools, systems, and technologies can be leveraged to provide greater degrees of control
and security over both public and private/confidential assets, to enhance process
efficiency and effectiveness, and to assist with the creation of a continuous process
improvement environment. For example, we observed that the CERCP program
generally required a great deal of monitoring, in large part because there are a number
of reporting requirements that must be met and, therefore, require significant levels of
rigor in terms of tracking and measuring process execution. However, with that said, we
also observed that the systems and technologies available to CERCP personnel were
largely a collection of non-enterprise level solutions created by various means (e.g.
“grassroots”) to support the needs of the departments. Generally speaking, some of
these critical monitoring, measuring, and reporting systems were not structured as long
term solutions built on enterprise-level platforms with the foundation of IT controls

required of such systems.
Overarching Observations and Recommendations
As part of this project, Crowe identified observations in different functional areas and cross-
functional areas within the Compliance Department. In doing so, seven themes surfaced that
impact the Compliance Department as a whole, as opposed to a specific team, process, or
functional area. These seven themes are important to the NERC Compliance Department’s
maturity as a process-driven organization. We provide an overview of these themes below.
Each is addressed in further detail this report:
1. We recommend to NERC that a number of changes to the ROP (including its related
appendices). These changes should be implemented to ensure a solid foundation for NERC’s
compliance program. We observed a number of issues with the ROP whereby it could be
strengthened by adding to it (address areas of Regional Entity accountability – e.g.
Compliance Inquiry process), changing it (address areas where Regional Entities differ in
practice from the ROP as documented – e.g. terminology such as “guidelines” and notices of
violation), or deleting from it (removing redundancies).
2. We recommend to NERC that CMEP Process and Procedures documents should be
completed, reviewed, and approved, including incorporating more defined roles,
responsibilities, timelines, and outcomes where these were found to be lacking. We
observed that process documents lacked consistency in form and included some conflicting
information, and at times did not contain obvious tie-backs to the ROPs by virtue of the
process used to develop them. The individual documents requiring completion, review, and
approval are captured within the detailed recommendations of this report.
3. We observed that the Compliance Department was not consistently meeting a number of its
internal process goals for timeliness. NERC Compliance indicated to us that, with their
current staff resources, they often had to adjust timelines in order to ensure the quality of
their work. It is our observation, therefore, that staffing levels may not be appropriately
aligned for the workload required. However, it is also our observation that there are other
contributing factors (process inefficiency issues, deficiencies in the “process infrastructure,”
effort-based metrics) which may also contribute heavily towards NERC’s ability to meet its
goals in certain compliance enforcement, registration and certification process areas. The
lack of activity level, effort-based metrics impeded the ability to fully assess whether staffing
levels are adequate relative to workload and/or to assess the degree to which staff levels
are required to meet certain levels of desired timeliness and quality.
4. We observed that problems with the consistency of outputs from Regional Entities (in terms
of the level of quality of outputs and the timeliness of those outputs) and differences in
professional opinion between NERC, the Regional Entities, and FERC impacted the timelines
for the Compliance Department’s work and the quantity of work that can be accomplished
(i.e. as measured by the number of enforcement actions processed within established time
frames). For example, one manager noted that Regional Entities often submitted Notices of
Confirmed Violations that contained errors in dates and judgments that NERC did not find
appropriate, such as classifying an issue as a documentation error rather than a failure to
perform, when the standard required documentation of performance. Another manager
stated that NERC and FERC periodically had different opinions on applications of reliability
standards on Compliance Violations Investigations.
5. We observed that processes within some functional areas were not adequately monitored
because there were few interim checkpoints being taken during the overall duration of the
process. For example, the functional areas Analyzing and Reporting Compliance Violation
Information and NERC Compliance Enforcement Authority Responsibilities had no
monitoring in place or planned. We also observed that for those functional areas that were
monitored, there was often not adequate follow up when process deviations were found. In
the functional area Overseeing Compliance Activities of Regional Entities, for example, we
observed that staff was given reminders of the need to meet timeliness goals, but no other
actions were taken when these goals were not met.
6. We observed several processes that involve handling large amounts of information and
documentation. NERC had begun to address these issues through the development of new
technologies, but it was our observation that until these are fully implemented, the volume
of data and documentation will continue to be an impediment to accomplishing the
Compliance Department’s goals in a timely manner.
7. We identified some issues with the level of controls over data security, confidentiality, and
physical security. Confidential information has been removed from this public version and
has been provided under separate cover to NERC management.
Document Overview
This report takes a top-down approach towards presenting the detailed observations and
recommendations. The Overview section provides a more detailed look at the objectives, scope,
and approach of this Process Evaluation.
The subsequent section (Section 2) titled Observations and Recommendations Summary
provides a summary level view across all observations and recommendations. As part of this
project and the methodology used, Crowe Horwath LLP developed a “scorecard” for evaluating
the various functional and cross-functional areas. The summary contains the summarized level
view of that scorecard. The summary also contains a number of overarching recommendations.
These recommendations are summary-level findings that in many cases present macro-level
observations made across functional areas or within functional areas across multiple criteria.
The next section of the document, Section 3, Cross-Functional Areas Evaluation, contains the
observations and recommendations as they relate to the four cross-functional areas.
Finally, Section 4, Functional Area Evaluation, contains the observations and recommendations
as they relate to the ten functional areas evaluated. Especially relevant to the functional area
evaluations are appendices I and II. Appendix I contains a crosswalk of the functional areas back
to the actual CERCP processes and procedures as defined by the NERC CMEP Processes and
Procedures manual. As most analysis will be documented as the functional area level, it is
important to note which processes and procedures comprise each functional area.
Appendix II contains the criteria used to evaluation each functional area. Appendix III contains
detailed observations and recommendations regarding changes to the ROP. These observations
and recommendations were developed by Crowe as part of its development of the Agreed-Upon
Procedures. Appendix IV contains an excerpt from the Management Letter to NERC from the
results of recently completed Agreed-Upon Procedures project for a regional entity. The excerpt
contains key recommendations regarding the ROP and the CMEP Processes and Procedures.
Section 1: Overview
Project Background
Project Objectives
North American Electric Reliability Corporation (“NERC”) determined the need for a project to
provide NERC with an evaluation of its Compliance Enforcement, Registration and Certification
Program (“CERCP”) processes and procedures. NERC engaged Crowe Horwath LLP to perform
this evaluation and Crowe completed this project between July and October, 2009.
The project was initiated to assist NERC’s Compliance area in achieving its overall objectives for
effective implementation of the CERCP, including adequate management controls. The project
objective, therefore, was to identify and document whether the program meets the
requirements of the implementing rules established by FERC for the Energy Policy Act (i.e. the
NERC Rules of Procedure and subsequent FERC orders), and if the NERC implementation has
adequately implemented applicable CERCP processes and procedures.
More specifically, the intent of this engagement was to:
1. Assess the core internal processes of the NERC CERCP implementation through interviews of
NERC Compliance employees and inspection of documentary evidence, using criteria found in
the following program documents from the ROP, and applicable sections from 18 CFR Part 29 as
the primary basis for the evaluation:
a. Section 400 – Compliance Enforcement
b. Appendix 4B – Sanction Guidelines of the North American Electric Reliability
Corporation
c. Appendix 4C – Compliance Monitoring and Enforcement Program
d. Section 500 – Organization Registration and Certification
e. Appendix 5 – Organization Registration and Certification Manual
f. Section 1500 – Confidentiality of Information
2. Provide an independent Process Evaluation Report (i.e. this report) for public use to align
with NERC’s need to be transparent, stating process efficiency, resource, or other improvement
recommendations identified (if applicable) during the process evaluation.
3. Provide a Confidential Letter to Management (i.e. a separate letter from this report) for any
process efficiency, resource, or other improvement recommendations that for the purposes of
communicating such information must include the identification of confidential information,
including but not limited to company names, data, NERC confidential information or personnel
identification. NERC assisted Crowe Horwath LLP with identification of such information.
Project Approach
For purposes of planning, tracking, and effective execution, the project was divided into two
separate, sequential phases where the outputs from Phase I became key inputs to Phase II
activities. The purpose, scope, activities, and outcomes of the two phases are described below.
Phase I - Planning and Data Gathering
Purpose Phase I of the project involved (i) conducting necessary project initiation and planning
activities, and (ii) gathering information from Compliance personnel concerning the
processes that NERC’s Compliance Department has in place over the compliance with
and enforcement of approved electric reliability standards. The activities included a
review of the criteria contained in the applicable sections of the Rules of Procedure,
developing questionnaires for data gathering, scheduling and conducting interviews
with NERC Compliance staff, and reviewing information received from NERC
Compliance staff and other documentary evidence regarding the execution of the
CERCP processes.
Activities 1. Conduct project initiation activities including, but not limited to, project kickoff
meetings to coordinate all project stakeholders and to ensure that there is a
common understanding for the project objectives, scope, approach, schedule, and
responsibilities.
2. Plan and establish the operating model for the project. Planning included the
creation and coordination of the project schedule of activities, resource schedules
and availability, project communications and status reporting.
3. Create a crosswalk of NERC compliance processes and procedures back to
functional areas that effectively group and map the processes and procedures
back to areas of organizational responsibility (see Appendix I).
4. Conduct initial interviews with functional area owners (primarily CERCP Managers
and Directors) to confirm understanding of the scope of the functional area, key
interactions with other functional areas, and the processes and resources
implemented within the area. Identify key documents and information supporting
the implementation of the CERCP processes and procedures.
5. Request, collect, and review documents and information supporting the
implementation of the CERCP processes and procedures (received from functional
areas owners and key subject matter experts).
6. Conduct formal interviews with functional area owners and functional area staff
(primarily analysts, investigators, administrators, and auditors) using common
functional area evaluation criteria to determine the status of the CERCP
implementation with respect to the criteria (note: the interview criteria are
included as Appendix II to this report).
7. Conduct final functional area interviews to confirm understanding and answer
final questions regarding processes, procedures, documents, and process artifacts.
Interviews included, in some cases, observation of various supporting IT systems.
Outputs ▪ Project Operating Model and Project Schedule

▪ CMEP Process and Procedure-to-Functional Area Crosswalk (included as Appendix I
to this report)
▪ Process review criteria and interview template (included as Appendix II to this
report)
▪ Documents and Artifacts Log
Table 1 – Project Approach Phase 1
Phase II - Data Analysis and Reporting
Purpose Phase II of the project involved (i) performing analysis and review of information
gathered in Phase I, (ii) preparation of the public report and the confidential letter to
management, (iii) review and revisions to the reports based upon feedback, and (iv)
final delivery of the reports and project closeout.
Activities 1. Prepare preliminary process write-ups for functional areas and conduct follow-up
interviews and communications to confirm understanding and address open
questions.
2. Perform cross-process analysis to identify overarching findings (e.g. trends) and
recommendations and prepare draft report sections for cross-functional areas and
overarching items.
3. Prepare a draft of the report overview section and executive summary.
4. Combine report sections and prepare initial draft of the Confidential Letter to
Management, including the CERCP Process Evaluation Report.
5. Conduct an internal (that is, internal to Crowe Horwath LLP) quality assurance
review cycle to fully review and discuss content and revise as necessary for initial
external review.
6. Prepare and conduct a preliminary report presentation (deliver draft report,
communicate the preliminary evaluation results, explain and confirm the quality
review and report acceptance process). Discuss approach for the public report and
confidential management letter (e.g. identify confidential aspects of the draft
public report).
7. Facilitate external quality assurance review cycle (distribute draft report, collect
and vet feedback, make applicable changes to draft report and letter).
8. Issue final evaluation report (public) and confidential management letter (non-
public).
9. Conduct project closeout (turnover of project assets, final project assessment and
feedback, etc.)
Outputs ▪ CERCP Process Evaluation Report (public/non-confidential)

▪ CERCP Process Evaluation Confidential Letter to Management
Table 2 – Project Approach Phase 2
Project Scope
The Engagement Letter for this Process Evaluation project established that “The intent of this
engagement is to … assess the core processes of the CMEP [plus other compliance enforcement
areas+ … using criteria found in the following program documents as the primary basis for the
evaluation:
a. Section 400 - Compliance Enforcement
b. Appendix 4B - Sanction Guidelines of the North American Electric Reliability Corporation
c. Appendix 4C - Compliance Monitoring and Enforcement Program
d. Section 500 – Organization Registration and Certification
e. Appendix 5 - Organization Registration and Certification Manual
f. Section 1500 – Confidentiality of Information”
To that end, the Engagement Letter identified eleven internal “processes” related to NERC’s
compliance enforcement, registration and certification goals that we used as the initial basis for
the scope of this Process Evaluation:
1. Compliance program planning
2. Following compliance program confidentiality requirements
3. Registration of users, owners, and operators of the bulk power system
4. Certification of users, owners, and operators of the bulk power system
5. Overseeing the compliance activities of Regional Entities
6. Overseeing the enforcement actions of Regional Entities
7. Reporting to the Federal Energy Regulatory Commission (FERC) or other Applicable
Governmental Authorities
8. Conducting reviews of Regional Entities’ compliance and enforcement programs
9. Conducting Compliance Violation Investigations and other monitoring and oversight
methods
10. Processing reliability standard violations
11. Handling complaints received on the hotline and via the Web site and those
communicated by the Regional Entities appropriately
During the course of the project this list of eleven initial “processes” evolved to more accurately
reflect the scope of all CERCP responsibilities and the alignment of these processes to the CERCP
as functionally implemented by NERC’s Compliance organization. Crowe discovered that NERC
has defined and documented 37 different internal compliance enforcement, registration and
certification processes and procedures and that the initial list of eleven “processes” in fact
represents eleven different “groups of processes”. We termed these groups of processes
“functional areas” to avoid confusion on the project because we were using the term “process”
liberally whereby it could mean too many things – a policy or rule, a procedure, a group of
processes, etc.
In an effort to ensure that the scope of the assessment fully covered the applicable processes
and procedures, Crowe created a crosswalk of the 37 CMEP Processes and Procedures back to
the original process list of 11 items. The CMEP Processes and Procedures Manual is an internal
set of procedures developed and maintained by NERC’s Compliance department to assist in the
implementation of the compliance enforcement, registration and certification program. The
result of that crosswalk is contained in Appendix I of this report.
As the list of areas evolved, Crowe also recognized that some of these functional areas
represent responsibilities that are shared across processes – in essence these areas are core or
foundational elements across CERCP processes. Through reviews of NERC’s process
documentation and discussions with management in NERC’s Compliance Department, we
identified four such areas that are “cross-functional” in nature: Compliance Program
Confidentiality, Developing and Overseeing the Compliance Training Program, Developing and
Disseminating Compliance Process Directives and Bulletins, and Processing Reliability Standards
Violations. Because these cross-functional areas are not necessarily processes or groups of
processes in and of themselves, but rather requirements and policies with responsibilities
spread throughout the organization and across processes, we redefined the list of areas and
conducted project activities using the following breakout:
Cross-Functional Areas
1 Compliance Program Confidentiality Requirements
2 Developing and Overseeing the Compliance Training Program
3 Developing and Disseminating Compliance Process Directives and Bulletins
4 Processing Reliability Standards Violations
Functional Areas
1 Compliance Program Planning
2 Overseeing Registration of Owners/Users/Operators of the Bulk Power System
3 Overseeing Certification of Owners/Users/Operators of the Bulk Power System
4 Overseeing Compliance Activities of Regional Entities (excluding CVIs)
5 Overseeing Enforcement Activities of Regional Entities
6 Analyzing and Reporting Compliance Information
7 Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs
8 NERC Involvement in Compliance Inquiries and Compliance Violation Investigations
9 Handling Complaints
10 Executing Compliance Enforcement Authority Responsibilities
Table 3 – CERCP Process Evaluation Final Scope
These four cross-functional areas and ten functional areas comprise the final scope of the CERCP
process evaluation – that is, the areas assessed as part of the evaluation – and, therefore, the
scope of this report. The 37 processes defined by NERC CMEP Processes and Procedures manual
are all encompassed within these 14 areas. The list of processes is as follows:
NERC
Process NERC CMEP Processes and Procedures Manual
Identifier Process Name Relevant ROP Section
NPP-CME-101 Organization Certification Process Procedure ROP 500; ROP Appx 5
NPP-CME-102 Organization Registration Appeals Procedure ROP 500; ROP Appx 5
NPP-CME-103 Organization Certification Appeals Procedure ROP 500; ROP Appx 5
NPP-CME-200 CMEP Development and Maintenance Process ROP 401.1
NPP-CME-201 CMEP Implementation Plan Process ROP 402.1.1; CMEP 4.0
NPP-CME-202 Training Process ROP 402.9
Monitoring and Facilitating Effectiveness of the
NPP-CME-204 CMEP ROP 402; ROP 404
NPP-CME-205 Compliance Process Bulletins/Directives None
NPP-CME-300 Compliance Inquiry Process None
NPP-CME-301 Complaint Process CMEP 3.8
NPP-CME-302 Compliance Violation Investigation Process CMEP 3.4
NERC
Process NERC CMEP Processes and Procedures Manual
Identifier Process Name Relevant ROP Section
NPP-CME-303 Evidence Handling Process CMEP 3.4
NPP-CME-400 Observation of RE-led Compliance Audits CMEP 3.1.5
NPP-CME-401 Regional Entity-led Compliance Audit Process CMEP 3.1.6
Procedure for the Regions to Self-Certify Adherence
NPP-CME-402 to the ROP and CMEP during and Audit None
NPP-CME-403 Regional Entity Spot Check Process None
NERC Audit of Regional Entity Adherence to the
NPP-CME-404 CMEP ROP 402.1.3; ROP 404.3
NPP-CME-500 Remedial Action Process CMEP 7.0
Compliance Violation and Penalty Process - Regional
NPP-CME-501 Entity CEA CMEP 5.1, 5.2, 5.4, 5.6
NPP-CME-502 Settlement Process - Regional Entity CEA CMEP 5.4
NPP-CME-503 Mitigation Process - Regional Entity CEA CMEP 6.0
NPP-CME-504 Mitigation Process - NERC CEA CMEP 6.0
NPP-CME-505 Appeals and Hearing Process CMEP 5.3, 5.5
NPP-CME-506 Penalty Guidance Process Appx 4B
NPP-CME-602 Registered Entity Audit Process Procedure CMEP 3.1
NPP-CME-603 Self-Report Procedure CMEP 3.5
NPP-CME-604 Spot Check Procedure CMEP 3.3
NPP-CME-605 Mitigation Plan Procedure CMEP 6.0
NPP-CME-606 Self-Certification Procedure CMEP 3.2
NPP-CME-607 Data Reporting and Disclosure Procedure CMEP 8.0
NPP-CME-608 Exception Reporting Procedure CMEP 3.7
NPP-CME-609 Periodic Data Submittal Procedure CMEP 3.6
NPP-CME-610 Implementation and Tracking Procedure CMEP 5.1; CMEP 6.0; CMEP 7.0
NPP-CME-611 Remedial Action Directive Procedure - CEA CMEP 7.0
NPP-CME-700 Data Management, Evaluation, and Analysis Process ROP 408; CMEP 8.0
NPP-CME-701 Compliance Data Reporting Process CMEP 8.0
ROP 402.8; ROP 404.3; ROP
NPP-CME-800 Document Management and Control 1500; CMEP 9.0
Table 4 – CMEP Processes and Procedures
The evaluation and the results documented within this report are focused at the level of the
cross-functional and functional areas, as demonstrated below, because this was the level of
evaluation most closely tied to the scope and intent of the project as expressed by the
engagement letter. We used individual internal process documents and comparisons to the
Rules of Procedure and other policies for making our evaluations. We also rolled up
observations and recommendations at any individual process level to the relevant functional
area. To that end, this evaluation does not necessarily contain detailed observations and
recommendations for all 37 processes and procedures as these represent, in effect, a level of
detail lower than the focus of this evaluation.
ROP,
Policy FERC
Orders
4 Cross-Functional Areas and

10 Functional Areas
Level of Evaluation
Procedures 37 Individual Processes and Procedures

(from CMEP Processes and Procedures Manual)
Figure 1 – Level of Evaluation
Figure 1 – Level of Evaluation
Process Evaluation Methodology
In order to accomplish the project objective, namely, “to identify and document whether the
program has adequately implemented applicable CERCP processes and procedures”, Crowe first
contemplated those items that define and provide details regarding the applicable CERCP
processes and procedures. After doing so, we further assessed the role of these defining items
in order to determine the adequacy of the NERC CERCP’s implementation.
Applicable Processes and Procedures
We compared the applicable, defining items for NERC’s CERCP to the typical role of policies,
processes, and procedures within any organization as follows:
Definition Applicable
Artifacts
Policy Policies are concise, formal and mandatory statements Applicable Rules
of principles and rules formulated or adopted by or of Procedure
dictated to an organization to reach its objectives and (ROP) sections
perhaps its goals. They are designed to influence all
FERC Orders and
major decisions and actions and to set all boundaries for
related decisions
all activities that take place within the scope set by them.
Applicable laws
and regulations
Processes Defines what is to be done and describes how (that is, NERC CMEP
and the steps involved) the activities are to be performed. (internal)
Procedures Processes and
The mandatory steps and specific methods required to
implement and comply with a policy to meet its intent Procedures
and perform the operations of the organization. Manual
Processes and procedures must ensure (i.e. put controls NERC
in place) that a point of view held by the governing body Compliance
of an organization (that is, the policies) is translated into Directives and
steps that result in an outcome compatible with that Bulletins
view.
Note: while there are subtle, technical differences
between the terms process (typically refers only to the
“what is to be done”) and procedure (typically refers to
the “how it is to be done”), we do not attempt to
differentiate these terms or use them to infer specific
meaning by their usage – which is to say, they are used
interchangeably throughout this document per the
definition above.
Table 5 – Policy, Process, and Procedure Defined
Adequacy of Implementation
For each of the functional areas within the scope of the project, Crowe Horwath analyzed the
information obtained through interviews and review of documentation to assess the following
for each process within each functional area:
1. Whether the objective of the process is known and documented
2. Whether the process is accurately documented – that is, the process as documented
matches how the process is most commonly executed by practitioners
3. Whether the roles and responsibilities in executing the process are documented and
whether responsibilities in executing the process are understood
4. Whether necessary inputs are available and in place to support appropriate execution of the
process
5. Whether an appropriate “process environment” is in place to support appropriate execution

of the process (e.g. this would include, but not be limited to, governance, organizational
priorities, support resources like tools and technologies, etc.)
6. Whether the process appears to accomplish its desired objective within the time (duration),
cost, and resource/material usage limits (that is, within the control limits)
7. Whether the process is applied and/or executed consistently (i.e., it is “controlled” to the
extent that it consistently executes without significant deviations in procedures)
8. Whether the process is measured (observation and reporting of process execution results –
can be real-time or after-the-fact)
9. Whether the process is monitored (ongoing, real-time observation of in-process scenarios to
detect when execution is deviating from plan, requirements, or objectives)
10. Whether the process appears to be efficient, to the extent that unnecessary steps,
iterations, resources, and delays have been eliminated
11. Whether process exceptions are recorded and root causes are assessed for systematic
improvement of the process
12. Whether personnel responsible for executing the process have awareness and
understanding of the process (as documented), and capability to execute the process (i.e.
they are trained and possess appropriate levels of authority)
13. Whether process documentation and supporting tools, technologies, resources, and process
inputs are made readily available
14. Whether the process documentation is made available, as required, and is controlled.
Crowe Horwath performed additional analysis for functional areas that had deficiencies to
determine, where possible, the key factors (e.g. root causes) contributing to the noted
deficiencies. Crowe Horwath identified best practices and developed recommendations that if
implemented may correct any performance deficiencies noted. Crowe Horwath synthesized the
results of the evaluations across all functional areas into an overall summary and identified any
trends or overall issues common throughout functional areas. The results of these efforts are
included in this report.
As noted above, the cross-functional areas in many cases are not in and of themselves processes
as much as they are core or foundational elements across CERCP processes. As such, the
methodology used to assess those areas and make recommendations was limited to those
criteria from the above list that were deemed to be applicable. The methodology used for
cross-functional areas also contemplated the extent to which the area supports or is
implemented by the individual functional areas.
Purpose of Report
The purpose of this report is to provide NERC with an evaluation of its CERCP processes and
procedures. This report, submitted by Crowe Horwath LLP, represents the culmination of
activities performed on the project per the Project Approach and methodology described above.
The primary objective of the report is to document observations as to whether the program has
adequately implemented applicable CERCP processes and procedures, where “adequacy” is
defined by those criteria identified in the Process Evaluation Methodology section of this
document, and to make recommendations where the implementation of the CERCP processes
and procedures can be improved.
Document Overview
The following report takes a top-down approach towards presenting the observations and
recommendations. The subsequent section (Section 2) titled Observations and
Recommendations Summary provides a summary level view across all observations and
recommendations. As part of this project and the methodology used, Crowe Horwath LLP
developed a “scorecard” for evaluating the various functional and cross-functional areas. The
summary contains the summarized level view of that scorecard. The summary also contains a
number of overarching recommendations. These recommendations are summary-level findings
that in many cases present macro-level observations made across functional areas or within
functional areas across multiple criteria.
The next section of the document, Section 3, Cross-Functional Areas Evaluation, contains the
observations and recommendations as they relate to the four cross-functional areas.
Finally, Section 4, Functional Area Evaluation, contains the observations and recommendations
as they relate to the ten functional areas evaluated. Especially relevant to the functional area
evaluations are appendices I and II. Appendix I contains a crosswalk of the functional areas back
to the actual CERCP processes and procedures as defined by the NERC CMEP Processes and
Procedures manual. As most analysis will be documented at the functional area level, it is
important to note which processes and procedures comprise each functional area.
Appendix II contains the criteria used to evaluation each functional area. Appendix III contains
detailed observations and recommendations regarding changes to the ROP. These observations
and recommendations were developed by Crowe as part of its development of the Agreed-Upon
Procedures. Appendix IV contains an excerpt from the Management Letter to NERC from the
results of a recently completed Agreed-Upon Procedures project for a regional entity. The
excerpt contains key recommendations regarding the ROP and the CMEP Processes and
Procedures.
Disclaimer of Confidentiality
This report contains no confidential information. Confidential information gathered or shared

as part of Crowe’s process evaluation has been shared with NERC management in a separate
confidential letter.
Section 2: Observations and Recommendations

Summary
Introduction
During our data gathering process, we used a Process Questionnaire (Appendix II) and other
methods to identify observations in different functional areas and cross-functional areas within
the Compliance Department. This section presents a summary of our analysis conducted across
the functional and cross-functional areas.
The Process-driven Organization
Background
In the pre-ERO era of NERC as a “Council”, the predecessor department to NERC’s Compliance
Department could be characterized generally as a service provider organization that responded
predominantly to unique, frequently “one-off”, situations or requests by a constituency of
voluntary stakeholders, or to the Regions (now NERC’s delegated authorities the Regional
Entities) who themselves were also and similarly service providers to those same stakeholders.
However, beginning before and certainly since certification of NERC as the ERO in 2006 NERC
CMEP has been transformed into a regulatory and regulated organization that is significantly
dependant upon development and implementation of thorough and complete processes to
succeed in its primary task/goal, which is consistent monitoring and fair enforcement. NERC’s
CMEP implementation must do this in a significantly-prescribed, uniform manner, which is to
say the basis for NERC’s CMEP implementation has become significantly more process-driven.
Basis for Observations
Before we summarize the observations made across the various functional areas it is worthwhile
to understand the basis for the observations. In observing the process areas within NERC
Compliance we apply concepts from process engineering and classical process
improvement/process optimization techniques and theories such as Lean, Six Sigma, TQM, etc.
We assessed NERC Compliance processes and procedures across three “tiers” or “layers”
comprising the elements critical for organizations to be successful with their processes:
Process Organizational success with process starts at the top. Management must
Governance create and instill an environment whereby the organization will operate and
guide its decisions within the policies and processes set by management or
dictated externally by laws or regulations.
The Process In order for policies to be followed and processes to be successful in an
Foundation organization, management must, through whatever means available to it,
provide foundational elements that enable the organization to carry out its
mission and operate within the policies and processes. Organizations
frequently fail to achieve process efficiency and/or control process exceptions
(that is, process results outside of the results desired and/or considered
within tolerances set by policy) when they lack one or more foundational
elements that are required to enable processes. Such items include, but are
not limited to:

▪ Making process inputs available
▪ Supporting tools and technologies to operate the process
▪ Supporting tools and technologies to monitor and measure the process
▪ Appropriate resources needed to effectively execute the process (in
manufacturing this might include machinery, in many processes this
includes human resource – i.e. personnel – who are appropriately skilled
and trained to perform the process)
▪ A capability to control policies and processes and to communicate and
make these readily available to those expected to operate within them
Ultimately, the process foundation should create a platform for continual
process improvement. So, for example, using the process improvement
axiom “you can only improve what you can measure”, it becomes an
imperative of the organization to provide the foundation for monitoring and
measuring its processes and for analyzing and improving its processes based
on the results that it observes. Reporting and escalation of process variances
(i.e. exceptions or deviations) must exist with appropriate management
actions/reactions to process performance.
The Processes Once governance is in place and foundational elements are made available,
the policies and processes of the organization have an appropriate operating
environment. The observation at that point shifts to the processes
themselves and whether the processes are implemented in a manner
whereby they can be executed – for example:
▪ Are processes documented (the documented process can be followed as
written)
▪ Does the documented process define the role and responsibilities (the
resources involved and the responsibilities in executing the process are
defined by the process and can be understood by the resources executing
the process from the process documentation)
▪ Is the process monitored (in-process observations are made to ensure that
the process is executing per control guidelines and that exceptions are
recorded and escalated as appropriate for potential corrective action)
▪ Is the process measured (formally or informally reported observations
made about the results of process execution)
▪ Is the process consistently executed within the operational constraints
established
▪ Does the process consistently produce/achieve its desired result
▪ Are internal process changes implemented in a controlled manner such
that process deviations aren’t created merely as a result of the change to
the internal process
Table 6 – The Infrastructure for Process Success

Process Governance and the Process Foundation Summary Observations
Before we summarize the observations made across the various NERC CERCP functional process
areas it is worthwhile to note our observations regarding the governance and foundational
layers of the NERC process environment.
▪ As a regulatory entity, NERC by its very nature is compelled to maintain an environment
focused on the creation, compliance, and enforcement of its standards and rules. We
observe that the NERC CERCP program generally has the governance and “tone at the top”
to be successful with its processes. Our assessment of individual functional areas indicates
that process objectives are typically well known and well understood and that there is
clearly a culture of policy and process adherence.
▪ As part of our analysis we placed NERCs CERCP into appropriate context from the standpoint
that NERC’s Compliance organization and the purpose, roles, and scope of responsibilities
for that organization has existed in their current state only for a relatively very short period
of time. The relative immaturity of the organization certainly has a bearing on the
expectations for its level of process maturity. For example:
o We observed in our analysis that the organizational structure, and the resulting roles
and responsibilities within that structure, continue to mature and change fairly
frequently as the Compliance area has undergone numerous structural changes within
the past two to four years. Three years ago the Compliance organization shifted from a
“Service Organization” whose purpose was to provide technical assistance to a
“Regulatory Organization” whose purpose was to regulate (i.e. compliance
enforcement, in addition to the role of registration and certification). The changes in
scope of responsibilities and assignment of responsibilities within an organization
certainly create challenges when attempting to get to a level of process maturity.
o We observed that the NERC Compliance Director/Manager-level positions are staffed, in
most cases, by personnel that are relatively new to the NERC Compliance organization.
Of the six (6) Director/Manager-level positions reporting up through the Vice President
of Compliance the average length of tenure for the personnel is less than 40 months. If
you filter out the one Manager with significant tenure (i.e. greater than five years), we
find that the average Director/Manager in Compliance has been with the organization
just over two years (i.e. approximately 25 months).
o The newness of staff to their respective positions certainly impacts expectations with
respect to process documentation. Organizational and process problems and
inefficiencies are being addressed by NERC compliance personnel (e.g. Compliance has
stood up 35+ processes in the past two years), but organizational and process best
practices emerge typically once some degree of longevity and critical mass has been
achieved. Procedurally, NERC’s Compliance area has achieved a great deal despite their
relatively short existence as an organization.
▪ We observe a number of areas (explained further in subsequent sections of this report)
where the NERC CERCP can improve its “process foundation”. It is our observation that a
number of these areas are a result of the NERC Compliance area’s relatively short duration
of existence and immature organizational infrastructure and, therefore, process
infrastructure. For example:
o Both the Rules of Procedures (ROP) and the NERC CMEP Processes and Procedures
Manual can be significantly upgraded to provide a more solid operational foundation. A
number of enhancements and changes to the ROP are recommended and we outline
those in this report. We also find that the internal CMEP Processes and Procedures are
substantially less mature than the ROP and will require a great deal of attention to reach
a point where they are documented in a manner where the tieback to the ROP is more
obvious, consistent across the Processes and Procedures themselves, and adequate to
provide the ultimate level of management control needed. Generally, the CMEP
Processes and Procedures Manual needs better defined roles and responsibilities,
timelines, and outcome-based measurements.
o While existing systems/processes to measure some results and provide statistics, it is
our observation that tools, systems, and technologies can be leveraged to provide
greater degrees of control and security over both public and private/confidential assets,
to enhance process efficiency and effectiveness, and to assist with the creation of a
continuous process improvement environment. For example, we observe that the
CERCP program generally requires a great deal of monitoring, in large part because
there are a number of reporting requirements that must be met and, therefore, requires
significant levels of rigor in terms of tracking and measuring process execution.
However, with that said, we also observe that the systems and technologies available to
Compliance personnel are largely a collection of non-enterprise level solutions created
by various means (e.g. “grassroots”) to support the needs of the departments.
Generally speaking, some of these critical monitoring, measuring, reporting systems are
currently not structured as long term solutions built on enterprise-level platforms with
the foundation of IT controls required of such systems.
Overarching Observations and Recommendations
Introduction
During our data gathering process, we used a Process Questionnaire (Appendix II) and other
methods to identify observations in different functional areas and cross-functional areas within
the Compliance Department. In doing so, seven themes emerged that impact the Compliance
Department as a whole, as opposed to a specific team, process, or functional area. These seven
themes are important to the NERC Compliance Department’s maturity as a process-driven
organization. We provide an overview of these themes below and address each in further detail
in subsequent sub-sections:
1. We recommend to NERC that a number of changes to the ROP (including its related
appendices). These changes should be implemented to ensure a solid foundation for NERC’s
compliance program. We observed a number of issues with the ROP whereby it could be
strengthened by adding to it (address areas of Regional Entity accountability – e.g.
Compliance Inquiry process), changing it (address areas where Regional Entities differ in
practice from the ROP as documented – e.g. terminology such as “guidelines” and notices of
violation), or deleting from it (removing redundancies).
2. CMEP Process and Procedures documents should be completed, reviewed, and approved,
including incorporating more defined roles, responsibilities, timelines, and outcomes where
these were found to be lacking. We observed that process documents lacked consistency
and at times did not contain obvious tie-backs to the ROPs by virtue of the process used to
develop them. The individual documents requiring completion, review, and approval are
captured within the detailed recommendations of this report.
3. We observed that the Compliance Department was not consistently meeting a number of its
internal process goals for timeliness. NERC Compliance indicated to us that, with their
current staff resources, they often had to adjust timelines in order to ensure the quality of
their work. It is our observation, therefore, that staffing levels may not be appropriately
aligned for the workload required. However, it is also our observation that there are other
contributing factors (process inefficiency issues, deficiencies in the “process infrastructure,”
effort-based metrics) which may also contribute heavily towards NERC’s ability to meet its
goals in certain compliance enforcement, registration and certification process areas. The
lack of activity level, effort-based metrics impedes the ability to fully assess whether staffing
levels are adequate relative to workload and/or to assess the degree to which staff levels
are required to meet certain levels of desired timeliness and quality.
4. We observed that problems with the consistency of outputs from Regional Entities (in terms
of the level of quality of outputs and the timeliness of those outputs) and differences in
professional opinion between NERC, the Regional Entities, and FERC impacted the timelines
for the Compliance Department’s work and the quantity of work that could be accomplished
(i.e. as measured by the number of enforcement actions processed within establish time
frames). For example, one manager noted that Regional Entities often submitted Notices of
Confirmed Violations that contained errors in dates and judgments that NERC did not find
appropriate, such as classifying an issue as a documentation error rather than a failure to
perform, when the standard required documentation of performance. Another manager
stated that NERC and FERC periodically had different opinions on application of reliability
standards on Compliance Violations Investigations.
5. We observed that processes within some functional areas were not adequately monitored
because there were few interim checkpoints being taken during the overall duration of the
process. For example, the functional areas Analyzing and Reporting Compliance Violation
Information and NERC Compliance Enforcement Authority Responsibilities had no
monitoring in place or planned. We also observed that for those functional areas that were
monitored, there was often not adequate follow up when process deviations were found.
In the functional area Overseeing Compliance Activities of Regional Entities, for example, we
observed that staff was given reminders of the need to meet timeliness goals, but no other
actions were taken when these goals were not met.
6. We observed several processes that involved handling large amounts of information and
documentation. NERC had begun to address these issues through the development of new
technologies, but it was our observation that until these are fully implemented, the volume
of data and documentation will continue to be an impediment to accomplishing the
Compliance Department’s goals in a timely manner.
7. We identified some issues with the level of controls over data security, confidentiality and
physical security. Confidential information has been removed from this public version and
has been provided under separate cover to NERC management.
Underlying each of these themes are several overarching observations that we made during our
data gathering and analysis process. As appropriate, we also made recommendations to
address these observations. The following sub-sections provide our observations for each of the
seven key areas followed by our recommendations for each area.
Recommended Changes to the Rules of Procedure

Observations
The Rules of Procedure and its related appendices make up the foundation of NERC’s
compliance program. Without a solidly developed ROP1, NERC’s ability to oversee and
enforce compliance with reliability standards diminishes. For example, if the ROP does not
include a requirement for Regional Entities to submit draft spot check reports to NERC, then
NERC Compliance has no immediate visibility over whether those spot checks were carried
out as scheduled and in a consistent manner. See Overarching Recommendation ROP-01.
During the process of developing the agreed-upon procedures used as a part of NERC’s audit
procedures of Regional Entity compliance programs, Crowe identified almost 50 additions,
deletions, and revisions to the ROP that would improve NERC’s ability to carry out its
compliance and enforcement functions. These observations are listed and included as
Appendix III to this report. NERC should review these observations and consider the
applicable changes to the ROP. See Overarching Recommendation ROP-01.
While performing the agreed-upon procedures at one of three Regional Entities, we also
made a number of observations and recommendations related to improvements needed to
the ROP. These observations and recommendations are listed and included as Appendix IV
to this report. NERC should also review these observations and recommendations and
consider the related changes to the ROP. See Overarching Recommendation ROP-01.
Since developing the agreed-upon procedures, we found that NERC issued a number of
Compliance Directives, which NERC expected different parties, particularly Regional Entities,
to follow. Some of these were one-time directives that NERC did not expect to be
performed on an ongoing basis or that NERC expected to possibly change in the future.
However, others were permanent requirements, and not all of these permanent
requirements had been incorporated into the ROP. As a result, there is a higher risk that the
one-time directives and/or permanent requirements will not be followed, because they
were not in a single reference location and they may not have been viewed by the Regional
Entities as being required or as important as the ROP. Therefore, we recommend that NERC
consider a formal review of bulletins and Compliance Directives to determine those that
should be permanent requirements of the ROP. For those determined to be “permanent”
we recommend that NERC incorporate those changes into the ROP. See Overarching
Recommendation ROP-01.
1In this report, where we refer to the ROP, we are also referring to its appendices, including Appendix 4C (the Compliance
Management Enforcement Program or CMEP).
During this project, we recommended several other changes to the ROP, which are
described below. See Overarching Recommendation ROP-01.
o A section should be added to the CMEP to describe the rules governing the
Compliance Inquiry process. We observed that there was no reference to this
process in the ROP, although NERC expected Regional Entities to follow it. See
Recommendation CVI-01 in the Functional Area Evaluation “NERC Involvement in
Compliance Inquiries and Compliance Violation Investigations.”
o References to “Transitional Certification” in ROP Appendix 5 should be deleted,
because this process has never been implemented. It should be replaced with the
“Provisional Certification” process. Note at the time of our observations, a revision
of Appendix 5 was pending that would incorporate these changes, but it was not yet
approved. See Recommendation CER-01 in the Functional Area Evaluation
“Overseeing Certification of Owners, Operators, and Users of the Bulk Power
System.”
o NERC Compliance Staff have identified a gap in the RoP and CMEP concerning
violation dismissals. In order to exercise appropriate and expected oversight there
needs to be developed both an internal process for the review of dismissals prior to
approval and appropriate changes to RoP and CMEP to ensure due process for the
industry, regional entities and NERC. We observed that NERC must review Notices
of Confirmed Violations prior to filing a Notice of Penalty with FERC, but not before
this stage. As a result, NERC has spent a great deal of time working with Regional
Entities at this end phase after the Regional Entities had already presented their
findings and had significant points of contact with the violating Registered Entities.
See Recommendation ENF-03 in the Functional Area Evaluation “Overseeing
Enforcement Activities of Regional Entities.”
When revisions to the ROP are made, other documents, such as implementation plans,
delegation agreements, report templates, documents in the Compliance Department’s
Processes and Procedures Manual, training materials, and systems may need to be revised
as well. Once the ROP changes are implemented, NERC should undergo a process to ensure
that other updates are made to related documents and systems as well. See
Recommendation ROP-02.
Recommendations
ROP-01 Perform an assessment of ROP changes recommended as part of this evaluation
(along with changes that may by otherwise queued up within NERC’s own
assessment of the ROP) and then develop and implement a plan to incorporate the
following into the Rules of Procedure and related appendices (that is, where there
is concurrence on the need for the change):
Observations on the ROP that Crowe made while developing the Regional
Entity AUPs,
Observations on the ROP that Crowe made while performing the Regional
Entity AUPs,
Required Compliance Directives that are meant to be followed on an
ongoing basis and that have not already been incorporated into the ROP,
and
Recommended changes to the ROP that Crowe identified during the
process evaluation project.
As part of the plan, include a schedule for reviewing the ROP revisions internally,
drafting the revised ROP, obtaining necessary input from outside parties, obtaining
BOTCC approval, and issuing the revised ROP.
ROP-02 Based on the ROP changes that are made, determine what changes need to be
made to other documents, including implementation plans, templates used by
NERC and Regional Entities, the Compliance Department’s Policy and Procedure
Manual, and any internal systems (tracking, reporting, etc.) if applicable. We
recommend that NERC Compliance develop and implement a plan to incorporate
necessary changes.
ROP-03 Based upon observations made while executing recommendations ROP-01 and
ROP-02, we recommend that NERC Compliance should establish and implement a
formal “internal change control” process whereby changes to the ROP, delegation
agreements, implementation plans, templates, the Compliance Department’s
Policy and Procedure Manual, training materials, and any internal systems can be
fully managed, coordinated, and tracked to completion in a consistent manner.
Managing internal change in a consistent, methodical manner is critical towards
assuring consistency between all of these pieces that are ultimately critical
towards the effective implementation of the CERCP. The internal change process
would accommodate externally-driven changes (e.g. changes to the ROP and FERC
orders) and ensure that these changes appropriately permeate throughout the
organization and would also accommodate internal changes to ensure consistency
between the process assets (process documentation, training assets, templates,
etc.)
Process Documentation Development

Observations
The NERC Compliance Department underwent a concerted effort to document its internal
policies, processes, and procedures in a Processes and Procedures Manual. Each team within
the Department contributed to this effort, in addition to performing its regular duties, and a lot
was accomplished, with over 50 documents drafted. However, we observed that NERC
Compliance had a fairly substantial amount of progress to make before its process documents
could be considered mature and reflective of a process-driven organization.
Certain compliance-related internal processes that NERC performs had not yet been
documented. Specifically:
o No document had been drafted of the CMEP Development and Maintenance
Process, meaning that NERC Compliance did not have a documented tool to guide
the development, coordination, or management of changes to the ROP. (See the
Functional Area Evaluation “Compliance Program Planning,” Criterion 1.)
o No document had been drafted for Penalty Guidance beyond the Sanction
Guidelines contained in the ROP. As a result, NERC Compliance had no documented
practice for the review of penalties assessed by Regional Entities. In particular,
there was no formal process for ensuring consistent application of penalties across
Regional Entities. This is a key NERC responsibility under the CMEP and Appendix 4B
to the ROP. (See the Functional Area Evaluation “Overseeing the Enforcement
Activities of Regional Entities,” Criterion 1.)
Because the ROP did not specify how to carry out these processes, documented internal
processes are essential to assure consistent achievement of NERC’s compliance goals. See
Recommendation PPM-01.
Of the Processes and Procedures Manual documents that have been drafted, only five —-
NPP-CME-301 (Complaint Process); NPP-CME-303 (Evidence Handling Process); NPP-CME-
400 (Observation of RE-led Compliance Audits); NPP-CME-403 (RE Spot Check Process); NPP-
CME-404 (NERC Audit of RE Adherence to the CMEP)—have been finalized and reviewed by
the Vice President and Director of Compliance or his designee. We observed that several of
the documents were still in very early draft form, with unresolved details “blanked out” or
unanswered comments and questions. These included the “CMEP Implementation Plan
Process” (NPP-CME-201) in the functional area Compliance Program Planning; the “Training
Process,” (NPP-CME-202) in the cross-functional area Developing and Overseeing the
Compliance Training Program; and, several processes within the functional area Overseeing
Regional Entity Enforcement Programs. As a result, the Compliance Department may not
have been executing the processes in a manner consistent with management’s goals. See
We observed that the documents in the Processes and Procedures Manual did not clearly
distinguish between policies, processes, and procedures. Often the terms were used
interchangeably. For example, documents such as the “Auditor Training Process,” “Data
Management, Evaluation, and Analysis Process” and the “Evidence Handling Process” did
not really have a process flow, but were more like policy documents. As noted above,
policies form the underlying rules and principles of an organization, while processes provide
a general framework for implementing those policies (what is to be done), and procedures
provide the specific steps for executing the processes (how it is to be done). As a best
practice, NERC Compliance should ensure that it’s Processes and Procedures Manual follows
the appropriate hierarchy of policies, processes, and procedures. See Recommendation
PPM-03.
Several of the processes did not document well-defined roles and responsibilities (these are
detailed throughout the report). We observed that they often noted that steps were to be
performed by “NERC,” or they may have assigned general responsibility for a process to a
certain manager, without identifying what team members are responsible for what parts of
the process. Examples of processes where these types of issues were identified included the
“Regional Entity-led Compliance Audit Process” (NPP-CME-401), within the functional area
Overseeing Regional Entity Compliance Programs, and the “Data Management Evaluation
and Analysis Process” within the functional area Analyzing and Reporting Compliance
Information. (See Criterion 3 in the functional area evaluations.) Organizational flexibility is
critical, and generally it is not necessary to assign a specific individual to be responsible for a
specific process step. For example, a process could refer to “a designated member of the
Enforcement and Mitigation team,” or “a Regional Entity Compliance Auditor,” or “the
Manager or Organization Registration and Certification or his designee.” Essentially,
Compliance staff should be aware of what roles they have, or might have, within certain
processes. This is especially important as new staff are hired who would not be as familiar
with NERC’s policies, processes, and procedures as the current Compliance Department
staff, many of whom were involved in the actual development of these documents. See
We observed that some processes lacked adequate information on how they were to be
carried out. We found this to be especially true when the process involved reviewing or
observing the work of Regional Entities. For example, we observed that NERC’s role while
observing Regional Entity compliance audits and NERC’s role in reviewing compliance
violation investigations led by Regional Entities were not well defined. (See Criterion 3 in
the functional area evaluations Overseeing Compliance Activities of Regional Entities and
NERC Involvement in Compliance Inquiries and Compliance Violation Investigations.) In
addition, the enforcement process for when NERC is acting as the Compliance Enforcement
Authority was not fully documented. (See Criterion 1 in the functional area evaluation NERC
Compliance Enforcement Authority Responsibilities.) See Recommendation PPM-05.
We observed that a number of processes—such as the “Organization Registration Process”
(NPP-CME-100) and the “Compliance Violation and Penalty Process” (NPP-CME-501)—did
not include adequate timelines or other measurable outcomes, other than those required
by the ROP. (See Criterion 6 within the functional area evaluations.) Admittedly, this
timelines are often dependent on receiving information from outside parties who cannot be
held to deadlines not specified in the ROP or other policy directives. However, for purposes
of better measuring and monitoring of the processes, and for communicating process norms
to staff, key measurements should be built into the process documents. See
We observed that many of the processes that we reviewed were not developed with the
ROP as a starting point. Instead, Compliance staff related to us that they developed the
processes based on how they carried out their functions at the time or how the processes
had been historically executed. Staff noted that they kept the ROP requirements in mind
while drafting the documents. However, in instances we observed process documents that
were not based on ROP requirements, such as the process documents related to
Compliance Inquiries, and ROP requirements that did not have an associated process
document prepared, such as NERC’s reviews of penalties and sanctions. We did not observe
any obvious or direct conflicts between the process document contents and the ROP
requirements, largely because the ROP was generally non-specific on the way many of
NERC’s compliance duties are to be carried out. See Recommendation PPM-07.
As part of the review cycle of this process evaluation report it was noted that there were
inconsistent uses of the term CMEP (i.e. Compliance Monitoring and Enforcement Program).
It was NERC’s observation of our initial report draft that the scope of the processes
contained within this report, and likewise within NERC’s Compliance Department, was
broader than CMEP, using the ROPs definition of CMEP (which is identified and defined by
Appendix 4C of the ROP). As an example, NERC’s Compliance Department refers to its
processes and procedures as the CMEP Processes and Procedures Manual, when this
document contains items that map back to other sections of the ROP (e.g. registration,
certification, confidentiality). Similarly, the use of the term “RE” was noted to be ambiguous
to the extent that this can refer to both regional entities and registered entities. See
In this report, we made other recommendations to improve the quality of the process
documents themselves. These are specific to certain cross-functional and functional areas, and
for purposes of providing an easy cross reference to these related recommendations, these
consist of the following recommendations within the sections listed:
o Recommendations TRA-01 and TRA-02 within the Cross-Functional Area Evaluation
“Developing and Overseeing the Compliance Training Program,”
o Recommendation PRO-01 within the Cross-Functional Area Evaluation “Processing
Reliability Standards Violations,”
o Recommendations IMP-01 and IMP-02 in the Functional Area Evaluation
“Compliance Program Planning,”
o Recommendations REG-01 and REG-02 in the Functional Area Evaluation
“Overseeing Registration of Users, Owners, and Operators of the Bulk Power
System,”
o Recommendations CER-02, CER-04, and CER-05 in the Functional Area Evaluation
“Overseeing Certification of Users, Owners, and Operators of the Bulk Power
System,”
o Recommendations COM-01, COM-03, COM-04, COM-05, and COM-06 in the
Functional Area Evaluation “Overseeing Compliance Activities of Regional Entities,”
o Recommendations ENF-01 and ENF-02 in the Functional Area Evaluation
“Overseeing Enforcement Activities of Regional Entities,”
o Recommendation REP-03 in the Functional Area Evaluation “Analyzing and
Reporting Compliance Information,”
o Recommendations REV-01 and REV-03 in the Functional Area Evaluation
“Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs,”
o Recommendations CVI-02 and CVI-03 in the Functional Area Evaluation “NERC
Involvement in Compliance Inquiries and Compliance Violation Investigations,” and
o Recommendations CEA-01, CEA-02, and CEA-04 in the Functional Area Evaluation
“NERC Compliance Enforcement Authority Responsibilities.”
Recommendations
PPM-01 Develop internal process documents for the CMEP Development and Maintenance
Process and the Penalty Guidance Process. Include procedures for cross-regional
comparisons in the Penalty Guidance Process. Develop a due date for completion
of these drafts.
PPM-02 Finalize all internal process documents and have them reviewed by the
appropriate Compliance team manager and by the Vice President and Director of
Compliance or a designee. Reviewers of the process documents should ensure
that the Recommendations PPM-04, PPM-05, PPM-06, and all functional area-
specific recommendations made in this report to improve the quality of the
process documentation are incorporated. All processes should be finalized and
reviewed before FERC begins requesting information for its audit of NERC.
PPM-03 In the internal Processes and Procedures Manual documents, classify the policies,
processes, and procedures into a hierarchy. Note that for some purposes, policies
- and sometimes even processes - may be the underlying ROP or FERC orders,
which would not need to be repeated in their entirety within the documents.
PPM-04 We noted as a recommendation in many of the functional area evaluations, that

NERC should consider the definition of roles and responsibilities within its process
documents. As such, there are many references in the functional area evaluations
to this recommendation (i.e. Recommendation Id PPM-04). We recommend that
NERC should consider designating who is responsible for executing each step
within the related processes and that these designations should continue to be
tied to roles within the organization, as opposed to specific names of individuals.
As individuals are frequently added to the organization, leave the organization, or
change roles within the organization, best practices dictate that designating
responsibilities tied to roles eliminates the need to maintain process documents as
people change.
PPM-05 Where processes were found not to be clear or well-defined (see references to this
recommendation, that is, Recommendation Id PPM-05 in the functional area
evaluations), we recommend that NERC Compliance specify in greater detail what
steps are to be followed within the processes. In keeping with Recommendation
PPM-04, designate who (by role) is responsible for these process steps.
PPM-06 Where noted as an issue in the functional area evaluations (see references to this
Recommendation, i.e. PPM-06), we recommend that NERC Compliance consider
identifying key milestones (perhaps in many cases, more detailed milestones)
within the process documents and specifying a goal or outcome to be measured,
such as a due date, for each of those key milestones beyond those used in
reporting timeliness in Corporate Goal # 1.
PPM-07 We recommend that NERC Compliance more closely align its CMEP Process and
Procedure documents with the applicable ROP sections pertinent to the individual
process being documented. In a number of cases the ROP contains a “swim lane”
diagram indicating NERC role in executing a process. It is our observation that the
“NERC swim lane” can be utilized in many cases as the basic framework for the
CERCP process. For example, arrows on the ROP diagram flowing into the NERC
swim lane become process input requirements, arrows flowing out of the NERC
swim lane become process output requirements, etc.
PPM-08 We recommend that NERC Compliance consider changing in its process

documentation (including the name of the process and procedures manual itself)
the use of the term CMEP and use of the term RE. Although the CMEP Processes
and Procedures Manual contains solely “internal” processes, we noted elsewhere
in our recommendations the importance of tying internal processes to the ROP.
Using nomenclature consistent with the ROP will assist NERC in defining and
maintaining its processes in lockstep with the ROP.
Compliance Department Staffing Levels

Observations
We observed that the Compliance Department was not consistently meeting a number of its
internal process goals for timeliness. Management and staff cited current staffing levels as
one of the primary reasons for this. NERC Compliance indicated to us that, with their current
staff resources, they often have had to adjust timelines in order to ensure the quality of
their work. For example, the Regional Operations team had consistently not been
completing audit observation reports in a timely manner (i.e. within timelines indicated by
the internal NERC Compliance process document). The Director of Regional Operations
stated that, with the team’s current staff levels, they have missed these deadlines in order
to ensure that the reports were adequately reviewed for quality. Similarly, the Compliance
Violation Investigation (CVI) team stated that they had not been able to take on all the
investigations they had wanted in order to ensure that they could coduct quality
investigations with the staff resources they have available. It is our observation, therefore,
that staffing levels may not be appropriately aligned for the workload required. NERC
Compliance should give consideration to the proper balance of workload and staff resources
including the designation of staff within the Compliance Department.
It is worthwhile to note that there are other contributing factors (the “process
infrastructure”) which may impede NERC Compliance’s ability to achieve its goals in certain
process areas. Once these other contributing factors are addressed, NERC Compliance’s
staffing levels may be better aligned with its workload. For example, dealing with a large
volume of data and documentation slows NERC Compliance’s processes considerably, but
the new Compliance Reporting and Tracking System (CRATS) is expected to alleviate some of
those issues. See Recommendations STA-01 and STA-02. Another example is the quality of
work emanating from the Regional Entities. In this report Crowe has made observations
and recommendations that, if implemented, will provide NERC Compliance with a “process
environment” that better enables its personnel to simultaneously achieve quality and
timeliness goals and will provide a number of “quick wins” to alleviate issues impacting
timeliness and staff workload (e.g. quality issues).
We observed that NERC Compliance was in the processing of filling two to four positions on
each of several Compliance teams. However, it was not clear whether the appropriate
staffing balance would be achieved. For example, one of the teams that had recently hired
additional staff was the Compliance Reporting, Analysis, and Tracking team, but their
staffing needs may or may not be as great once CRATS is fully implemented – for example,
while CRATS may improve productivity and/or add efficiencies, it may also create additional
benefits to be gained by new types of analysis and reporting See Recommendations STA-01
and STA-02.
We generally observed across most processes that time/effort metrics were not captured,
reported, or analyzed. The lack of effort-based metrics impeded the ability to fully assess
whether staffing levels are adequate relative to workload and/or to assess the degree to
which staff levels were required to meet certain levels of desired timeliness and quality.
See Recommendations STA-01 and STA-02.
We observed that because of increasing requirements, Technical Feasibility Exception
reporting, Critical Infrastructure Protection and coordination between the Nuclear
Regulatory Commission and NERC, all with an undetermined impact and potentially
significant resource requirement, NERC’s Compliance Department requires a high degree of
flexibility in assigning staff. These programs will also undoubtedly evolve and mature over
the next couple of years.. For example, we observed that several staff on the Regional
Operations team were occupied with the Regional Entity audits, which are performed every
three years. Once this process is completed, that team may have more time to devote to
other responsibilities. In addition, an unexpected large disturbance in the bulk power
system (BPS) could require a sudden surge in the need for Compliance Violation
Investigation team resources.
We noted some sharing of staff among different teams, however, we also noted a trend
towards specializing staff (as opposed to cross training and/or cross-utilization) . For
example, we observed that NERC Compliance developed plans for members of the
Organization Registration to help perform the Regional Entity audits, however, most teams
were in the process of becoming highly specialized. NERC Compliance should consider
organizational alignment that encourages a certain amount of flexibility to react to sudden
major workload needs across functional areas. See Recommendations STA-02 and STA-03.
Recommendations
STA-01 We recommend that NERC Compliance consider enhancing its time tracking system
currently in place for tracking staff hours. The time tracking system could capture
more granular detail than it currently captures to track time needed to complete
individual processes in all functional areas and, in particular, steps or activities
within the processes. This would require staff to document time spent on
processes and activities within processes, including in some cases additional details
such as the violation or audit (for example) being worked on. This may be done on
a sample basis, especially for tasks that repeat often, such as reviewing mitigation
plans from Regional Entities. When a large internal process change occurs, such as
the implementation of CRATS, NERC can collect new hourly data on the process and
use that as a basis for measuring the organizational impact of change.
STA-02 Based on the data collected on project staff hours (recommendation STA-01), we
recommend that NERC Compliance consider development of benchmarks for the
completion of major processes and estimates for the total number of staff hours
spent on those projects within a given time period (monthly, quarterly, yearly).
NERC Compliance could re-run estimates periodically to account for “spikes” or
“lulls” in certain processes. NERC Compliance could then use this analysis to
determine if current staffing levels on each team are sufficient to meet the team’s
process needs, or if resources need to be re-aligned.
STA-03 We recommend that NERC Compliance consider developing a program to ensure

that Compliance staff, particularly newer staff, is cross-trained in a number of
functional areas and have an understanding of all Compliance processes.
Resolving Issues with Other Entities

Observations
Several Compliance teams noted difficulties obtaining consistent, quality outputs from
Regional Entities. In particular, this was noted in areas where NERC Compliance had the
ultimate responsibility for reviewing and assuring the quality of the information, such as
Compliance Violation Investigations and reviews of violations prior to issuing Notices of
Penalty. The issues identified ranged from repeated simple issues, such as dates not being
correctly documented, to differences in professional opinion or judgment. Managers noted
that processes were often delayed, because NERC Compliance had to spend a significant
amount of time working with the Regional Entities to improve the quality of their outputs
before they were approved. To save time in the long run, we recommend that NERC
Compliance consider developing and implementing a program to invest time in ensuring
better quality of work from Regional Entities. NERC Compliance should also consider
working to resolve differences in professional opinion in a timely manner. See
Recommendations ENT-01, ENT-02, and ENT-03.
Compliance teams also noted process delays due to differences in professional opinion with
FERC, or because FERC guidance changes were not clear. (For example, see Criterion 4 in
the functional area evaluation section Overseeing the Enforcement Activities of Regional
Entities and Criterion 6A.1 in the functional area evaluation section NERC Involvement in
Compliance Inquiries and Compliance Violation Investigations.) In these cases, NERC should
work with FERC to establish clear guidance in formulating reliability standard violations or
judgments. See Recommendation ENT-04.
We observed that process delays could be especially pronounced when obtaining consensus
from multiple organizations, both inside and outside NERC. While the development of
certain key documents, such as the ROP and implementation plans, required input from
multiple sources in order to achieve transparency, the process often became unnecessarily
cumbersome. See Recommendation ENT-05.
Recommendations
ENT-01 We recommend that NERC Compliance consider the development of checklists for
the proper content of documents that Regional Entities submit to NERC for review.
We recommend that NERC direct Regional Entities to utilize these checklists and, in
particular, the NERC work with Regional Entities that have repeated problems
submitting quality documentation to understand and use the checklists. For
example, we recommend that NERC Compliance utilize the guidance issued by Legal
after the July 3rd, 2009 order (titled “Lessons Learned: Initial Notice of Penalty
Filings”) into a working checklist where regional entities can literally check off the
necessary components as an indication of completeness.
ENT-02 We recommend that NERC Compliance consider the development of additional

training programs for Regional Entities in all compliance and enforcement functional
areas where issues with the quality or consistency of Regional Entity work have been
identified.
ENT-03 We recommend that NERC Compliance consider a review of the compliance and
enforcement staff structure of Regional Entities to ensure that they have the proper
mix of talent for carrying out all compliance and enforcement duties. For example,
NERC Compliance could ensure that they have adequate staff with a legal or
regulatory background. NERC Compliance could then direct Regional Entities that
are lacking in certain key skills to acquire those skills through added hiring, changes
in staff roles, or other means.
ENT-04 Identify key areas of differences with FERC in professional judgments. Work with
FERC to establish clear guidelines for handling these items, or make adjustments as
necessary (proposing to change a reliability standard, for example). We further
recommend that NERC track these agreements with NERC in a database for easy
query across NERC teams.
ENT-05 When implementing a process that requires obtaining multiple reviews from
different parties, we recommend that NERC Compliance consider establishing and
enforcing/reinforcing clear goals for obtaining comments, for entities with which this
can be done (Regional Entities and other NERC departments, primarily).
Monitoring and Measuring Compliance Processes

Observations
Monitoring and measuring are key components of a properly functioning process. A process
should be monitored and measured throughout its execution to ensure that the process is
on track toward achieving its desired final outcomes, and to take appropriate mitigating
actions if the process is not on track. Metrics taken throughout the process do not
necessarily have to be reported to upper management or any external parties, but internal
monitoring and measuring should take place on a regular basis for consistency, and staff
performing the process should be aware of how their actions are being tracked.
We observed that certain functional areas had well defined and implemented mechanisms
for monitoring and measuring processes, especially the following functional areas:
Overseeing Certification, NERC Involvement in Compliance Inquiries, and Compliance
Violation Investigations. However, most functional areas did not have adequate monitoring
of process outcomes or milestones, often because process norms had not yet been
developed. In particular, we noted inadequate monitoring of intermittent milestones within
certain processes, other than those required by the ROP. (For example, see Criterion 6 in
the functional area evaluation sections Compliance Program Planning and Overseeing
Regional Entity Enforcement Activities.) Other functional areas (Analyzing and Reporting
Compliance Information and NERC Compliance Enforcement Authority Responsibilities) had
no monitoring in place. See Recommendation MON-01.
Much of the process monitoring that we observed was found in the Compliance
Department’s monthly Goal 1 Update Report. We observed that this report contained a
number of informal metrics that the Compliance Department put together in order to
measure its progress in achieving NERC’s Corporate Priority #1, “on-time delivery of all NERC
and NERC Regional Entity delegated outputs.” While timeliness was identified as NERC’s top
corporate priority, it should also be noted that NERC identified nine other Corporate
Priorities that it monitored (although several of these do not apply to the Compliance
Department). However, for many of the functional areas, the Goal 1 Update Report was
referenced as the only method of monitoring and measuring underlying processes, which is
to say that other monitoring and measuring is not abundant (such as quality measurements,
interim process measurements, time/effort /productivity/efficiency measurements, process
exceptions or exception root causes, etc.) See Recommendation MON-01.
We observed that certain metrics that were monitored and measured, such as those in the
monthly Goal 1 Update Report, were measured as “yes/no” responses only. In other words,
there was no determination or measuring of control limits. For example, the Goal 1 Update
Report measured whether audit observation reports were completed on time, but it did not
measure by how many days a report was late. This made it difficult to get a complete
picture of how well process goals were or were not being met. It also made the tracking of
process improvement (or conversely, process degradation) over time more difficult. For
example, a team may have progressed from being several weeks late on average with a
report to being several days later on average, but this was not being captured. While in
some cases, such as deadlines required by the ROP, there would be no acceptable outer
control limit, in most other cases, upper and lower control limits should be established for
measuring process milestones and outcomes. See Recommendation MON-02.
We observed that within the Goal 1 Update report, most of the metrics were tracked
cumulatively over the year. As a result, those metrics did not show progress or decline from
month to month. Managers indicated to Crowe that this is typically because the level of
acceptability of a process deviation was determined based on the number of times that
deviation occurred within a given year. However, by focusing on measuring control limits,
as noted above, rather than the number of deviations, this cumulative tracking would not be
necessary, and NERC management would get a more complete picture of progress over
time. See Recommendation MON-02.
We observed that nearly all the processes that were monitored identified deviations from
the Processes and Procedures Manual. However, for certain processes in the functional
areas of Overseeing Regional Entity Compliance Programs and Overseeing Regional Entity
Enforcement Programs, there were not sufficient follow-up or corrective actions undertaken
when deviations were identified. For example, we observed that observation reports on
Regional Entity compliance audits were often not completed on time (see the Functional
Area Evaluation section Overseeing Compliance Activities of Regional Entities). However,
the only follow-up was to remind staff of the need to be timely. See Recommendation
MON-03.
Recommendations
MON-01 A number of the Functional Area Evaluations in this report will reference this
recommendation (that is, Recommendation Id MON-01). Our general, summary-
level recommendation referenced within the functional areas is that we
recommend that NERC Compliance consider the identification and documentation
of milestones and other goals (i.e. key performance indicators) to track and
monitor the progress of processes beyond those used for Corporate Goal #1.
Furthermore, we then recommend that NERC Compliance consider developing and
implementing a system for monitoring, measuring, and reporting on those key
performance indicators. NERC will need to allot the resources and infrastructure
necessary to implement the system.
MON-02 For all key goals that are established, and for those already being measured, we
recommend that NERC Compliance develop and implement a means for capturing
the level of deviation of each goal from its established norm (i.e. track exceptions).
For each performance metric NERC Compliance should determine how much
deviation is acceptable (in process terms this is typically referred to as “within
tolerance”) and capture those actual process results falling outside of tolerance.
MON-03 We recommend that NERC Compliance develop and implement a system of

corrective actions when process results deviate outside of acceptable tolerances.
NERC Compliance should communicate the intent of these corrective action plans
to Compliance staff. Note: the action of monitoring/measuring and
planning/implementing corrections is the basis for “continuous process
improvement”.
Handling Compliance Information

Observations
We observed that several functional areas within the Compliance Department, especially in
the enforcement area, processed large amounts of information and documentation on a
daily basis. We observed that NERC’s systems, which relied heavily on Excel spreadsheets
and manual tracking, were not able efficiently accommodate the volume of information
being produced and processed. NERC has recognized this as an issue and has been
developing and implementing a new system for tracking compliance information, known as
the Compliance Reporting and Tracking System (CRATS). Once fully implemented, this
system should save NERC considerable hours of work and should eliminate much of NERC’s
current issues with timeliness of processing and the workload placed upon the staff
members who manage the information. Per NERC Compliance management, after the
implementation of CRATS and as part of a second add-on phase NERC plans to develop a
tool for more effective and efficient document management, utilizing the Microsoft® Office
SharePoint Server.
As CRATS and SharePoint are implemented, many of NERC’s process documents will need to
be revised to account for the systems changes as the role of the system is reflected in the
documents. We encourage NERC Compliance to formulate a comprehensive plan for
organizational change (from updating process documents to training practitioners on
internal process changes resulting from the new system). See Recommendation INF-01.
Recommendations
INF-01 We recommend that NERC Compliance identify processes that will be affected by the
implementation of the Compliance Reporting and Tracking System (CRATS) and the
Microsoft® Office SharePoint Server. We recommend that NERC Compliance consider
updates required to process documents where internal process changes will occur
with the new systems. NERC Compliance will need to establish a plan whereby the
changes needed to the internal process documents as a result of the system changes
are made in concert with many of the recommendations in this report.
Maintaining Security and Confidentiality

Confidential information has been removed from this public version and has been provided
under separate cover to NERC management.
Categorization of Recommendations
Introduction
Throughout this report we have made a number of recommendations that we believe will
strengthen NERC’s execution of its compliance processes and decrease future risks of violating
the ROP, FERC Orders, and internal policies. To assist NERC Compliance in prioritizing these
recommendations, we have grouped them into five categories:
Category Description of the Recommendation Category

Identifier
1 Recommendations to address non-adherence with the Rules of Procedure
We noted one ROP non-adherence that was occurring at the time of our data
gathering. (See Recommendation CER-01.) However, note that our report also
identifies instances in which NERC Compliance, through its own monitoring
procedures, detected ROP non-adherence in its processes, and we commented on
whether NERC took appropriate action to follow up on those violations.
2 Recommendations to address non-adherence of internal processes and procedures
We noted one instance in which NERC was not following its processes and
procedures as documented, but was not out of adherence with ROP requirements.
(See recommendation REV-01.) Also, as with ROP non-adherence, the report
discusses process violations that NERC detected through its own monitoring
procedures, and we commented on whether NERC took appropriate follow-up
action.
3 Recommendations to address missing process documentation and issues within
existing process and procedure documents
These recommendations address findings within the process documents
themselves, such as processes that have not been documented and contradictions
within or between process documents. The recommendations also address any
processes which have not yet been implemented or are not accurately
documented to reflect the methods by which NERC intends to carry out these
processes.
4 General recommendations to strengthen or improve processes
These are typically overarching or structural recommendations that we believe
NERC Compliance should employ in order to strengthen its overall compliance
program. While these are not a lower priority, they typically would involve more
time, resources, and planning to address than other recommendations that can be
more quickly and easily implemented.
5 Other recommendations
These are recommendations that do not fit into one of the four categories above.
Again, these are not necessarily lower priorities, but may take additional time to
implement.
Table 7 – Recommendation Categories
Classification of Recommendations
All recommendations in this report have been summarized and classified into the
aforementioned “recommendation categories” (i.e. categories 1 through 5) as documented in
the tables below. Each specific recommendation is referenced by its recommendation identified
(or ID) and each category is referenced solely by its category number.
Report Section Topic or Functional Area ID Category
Overarching Recommended Changes to the Rules of ROP-01 5
Observations and Procedure ROP-02 4
Recommendations
ROP-03 4
Process Documentation Development PPM-01 3
PPM-02 3
PPM-03 3
PPM-04 4
PPM-05 4
PPM-06 4
PPM-07 3
PPM-08 4
Compliance Department Staffing Levels STA-01 4
STA-02 4
STA-03 4
Resolving Issues with Other Entities ENT-01 4
ENT-02 4
ENT-03 4
ENT-04 4
ENT-05 4
Monitoring and Measuring Compliance MON-01 4
Processes MON-02 4
MON-03 4
Handling Compliance Information INF-01 5

Cross-Functional Cross-Compliance Program Confidentiality CON-01 5
Areas Evaluations Requirements CON-02 5
CON-03 4
Developing and Overseeing the Compliance TRA-01 4
Training Program TRA-02 4
Processing Reliability Standards Violations PRO-01 4

Functional Areas Compliance Program Planning IMP-01 4
Evaluations
IMP-02 3
Overseeing Registration of Users/ REG-01 3
Owners/Operators of the BPS
REG-02 3
REG-03 4
Overseeing Certification of Users/ CER-01 1
Owners/Operators of the BPS
CER-02 3
CER-03 4
CER-04 3
CER-05 3
Overseeing Compliance Activities of COM-01 3
Regional Entities (excluding CVIs)
COM-02 4
COM-03 4
COM-04 4
COM-05 4
COM-06 3
Overseeing Enforcement Activities of ENF-01 3
Regional Entities
ENF-02 3
ENF-03 4
ENF-04 5
Analyzing and Reporting Compliance REP-01 5
Information
REP-02 4
REP-03 4
Conducting Reviews of Regional Entities’ REV-01 2
Compliance and Enforcement Programs
REV-02 4
REV-03 3
NERC Involvement in Compliance CVI-01 3
Inquiries and Compliance Violation
CVI-02 4
Investigations
CVI-03 3

Functional Areas Executing Compliance Enforcement CEA-01 3
Evaluations (continued) Authority Responsibilities CEA-02 3

CEA-03 4
CEA-04 4
CEA-05 4
Table 8 – Recommendations Summary by Category of Recommendation
Because there are numerous recommendations made by this report (as evidenced by the
preceding table) spread across multiple sections or “types” (overarching, cross-functional and
functional) with multiple classifications, we provide another table below that captures all
recommendations in this report by topic (of functional area) by section and by category with a
count of the recommendations found.
REPORT SECTION RECOMMENDATION ID, BY CATEGORY TOTAL

Over- Cross- Func NUMBER
TOPIC OR FUNCTIONAL AREA TOPIC ID arch func Area 1 2 3 4 5
Changes to Rules of Procedure ROP O 02, 03 01 3
Monitoring & Measuring Compliance MON O 01, 02, 03 3
Compliance Department Staffing STA O 01, 02, 03 3
Resolving Issues with other Entities ENT O 01, 02, 03, 04, 05 5
Handling Compliance Information INF O 01 1
Process Documentation Development PPM O 01, 02, 03, 07 04, 05, 06, 08 7
Processing Reliability Std Violations PRO X 01 1
Compliance Program Confidentiality Reqmts* CON X 03 01, 02 3
Develop & Oversee Training Program TRA X 01, 02 2
Compliance Program Planning IMP F 02, 01 2
Oversight of Registered REG F 01, 02 03 3
Oversight of Certified CER F 01 02, 04, 05 03 5
Oversight of Compliance (non-CVI) COM F 01, 06 02, 03, 04, 05 6
Oversight of Enforcement ENF F 01, 02 03 04 4
Reporting Compliance Information REP F 02, 03 01 3
Review of Regional Entities REV F 01 03 02 3
CVI Violations CVI F 01, 03 02 3
Compliance Enforcement Authority CEA F 01, 02 03, 04, 05 5
TOTAL NUMBER OF RECOMMENDATIONS 1 1 18 36 6 62
Table 9 – Recommendations Count by Section, by Category
* Confidential information has been removed from this public version and has been provided under separate cover to NERC management. .
Section 3: Cross-Functional Areas Evaluation
Introduction
NERC Compliance personnel have a number of responsibilities that are shared throughout the
Compliance Department, and in certain cases, with other departments as well. These cross-
functional areas have underlying policies or processes that are required to be followed in
performing the procedures underlying many, and in some cases, all, of the Compliance
functional areas. Through reviews of NERC Compliance’s process documentation and
discussions with management in NERC’s Compliance Department, we identified four such areas
that are cross-functional in nature:
▪ Compliance Program Confidentiality Requirements,
▪ Developing and Overseeing the Compliance Training Program,
▪ Developing and Disseminating Compliance Process Directives and Bulletins, and
▪ Processing Reliability Standards Violations.
Because these cross-functional areas are not necessarily processes, but often sets of
requirements and policies with responsibilities spread throughout the organization, we did not
evaluate them using the Process Questionnaire in Appendix II. Rather, we performed our
evaluations based on reviews of underlying policy documentation and from information
obtained during interviews with and observations of managers and staff throughout the
Compliance Department.
3.1. Compliance Program Confidentiality Requirements
Introduction and Scope

All NERC employees are required to follow strict policies governing the confidentiality of
information in all that they do. The Compliance Department, with its in-depth knowledge of and
access to sensitive information, such as early notifications of alleged violations, has its own
special requirements for maintaining confidentiality. Policies governing the requirement not to
publicly disclose confidential information are found throughout the ROP, but are primarily
found in ROP Section 1500 and CMEP Section 9.0.
As noted in Appendix I, the following process makes up the Compliance Program Confidentiality
cross-functional area: “Document Management and Control” (NPP-CME-800)
Observations and Recommendations

Confidential information has been removed from this public version and has been provided
under separate cover to NERC management.
3.2. Developing and Overseeing the Compliance Training Program

The ROP requires that individuals participating in compliance audits ( that is, “All industry
experts and regional entity members”), compliance violation investigation (CVIs), and
certification reviews must have completed NERC auditor training before beginning work in these
areas. Responsibility for developing and conducting the compliance training program is shared
between the Director of Regional Operations in the Compliance department and the Director of
Training, Education, and Personnel Certification in NERC’s Situation Awareness and
Infrastructure Security department. The Director of Regional Operations’ staff maintains copies
of the compliance training records for courses not taken online. Online course records are
recorded within the online training system.
As noted in Appendix I, the following process makes up the cross-functional area Developing
and Overseeing the Compliance Training Program: “Training Process” (NPP-CME-202)
Observations
While performing the agreed-upon procedures at one of the Regional Entities, we
discovered two issues with training on NERC’s end. Specifically, we found that NERC
Compliance did not ensure training had been provided to Certification Review Team
members by Regional Entities. We also found that NERC Compliance had not recorded (in a
system, a tracking sheet, or otherwise) the training that compliance auditors had taken
beyond what was recorded by the Training Department. Specifically, while performing
agreed-upon procedures at one Regional Entity, neither the NERC Training Department nor
the NERC Compliance Department could provide us with lead auditor training records for
certain auditors, although these auditors held training certifications on file at the Regional
Entity’s office. We made recommendations in those reports to provide direction regarding
corrections for these issues.
We observed that NERC’s compliance training process document (“Training Process,” NPP-
CME-202.R0) was in draft form and had not been reviewed by management. See
Overarching Recommendation PPM-02.
We observed that NERC’s compliance training process document did not fully spell out the
requirements of the training program, such as who is required to be trained and when, or
how training is to be recorded and monitored to assure that appropriate individuals have
completed their required training courses. See Recommendation TRA-01 below.
We observed that the roles as spelled out in the compliance training process document did
not always accurately reflect the division of responsibilities in practice. For example, the
document stated that the Director of Training maintains the compliance training records,
but in actuality certain records were maintained by the Compliance Department. In
addition, the document stated that the Director of Training was responsible for providing
the Compliance training. However, we have observed that some training presentations are
delivered by Compliance staff. See Recommendation TRA-02 below.
Recommendations
TRA-01 We recommend that NERC Compliance revise the “Training Process” document
to fully identify the requirements of NERC’s compliance training program,
including the basic requirements outlined in the ROP and the specific courses
that NERC has developed to satisfy those requirements. The document should
also identify who is required to be trained and when. It should also specify how
training is to be recorded and monitored to assure that appropriate individuals
have completed their required training courses.
TRA-02 With input from the Director of Training and all NERC Compliance teams that
have a stake in the training process, revise the current draft “Training Process”
document to accurately identify the roles and responsibilities for the different
aspects of the functional area, such as training development, training delivery,
and training records monitoring and maintenance. The changes should be
reviewed and approved along with the rest of the draft process document.
3.3. Developing and Disseminating Compliance Process Directives and Bulletins

Compliance Process Directives and Compliance Process Bulletins are developed by NERC as
policy or guidance to Regional Entities on the implementation of compliance monitoring and
enforcement goals. Compliance Process Directives are non-public and required to be followed,
whereas Compliance Process Bulletins are publicly posted as non-binding recommended
practices to the Registered Entities. A public Bulletin may be issued concurrently with a
Directive. Any group within NERC’s Compliance Department may create and issue a Compliance
Process Directive or Compliance Process Bulletin, but the Director of Regional Operations is
responsible for ensuring that Regional Entities are given an opportunity to provide input on draft
Directives and Bulletins.
As noted in Appendix I, the following process makes up the cross-functional area Developing
and Disseminating Compliance Process Directives and Bulletins: “Compliance Process
Bulletins/Directives” (NPP-CME-205)
Observations
As documented in Crowe’s management letter to NERC following one of the Regional Entity
agreed-upon procedures (AUPs), Regional Entities did not have a clear understanding on
whether bulletins, letters, and other directives issued by NERC outside of the ROP were
binding on them. Therefore, we recommended that NERC Compliance develop a hierarchy
of documentation to clearly indicate required actions on the part of Regional Entities.
Accordingly, NERC drafted a new document (“Compliance Process Bulletins/Directives,”

NPP-CME-205.R0) related to the issuance of Compliance Process Directives and Compliance
Process Bulletins. Prior to this process, guiding documents issued to Regional Entities were
referred to as “Process Bulletins.” In the new process, the distinction was made between
Compliance Process Directives and Compliance Process Bulletins, and this distinction was
communicated to the Regional Entities. We believe this resolves the recommendation made
in the prior report.
We observed that the roles within the “Compliance Process Bulletins/Directives” process
document were not c learly defined. For example, the document mentioned the role of the
Compliance Department in general, but it did not provide information on who was
responsible for the process. NERC Compliance staff personnel informed us that any teams
within Compliance have the authority to develop Compliance Directives and Bulletins. In
addition, under the actual process, NERC Compliance told us that an Administrative
Assistant routinely tracked Directives and Bulletins by sequential number, and assigned a
new sequential number to any new Directive or Bulletin. However, we observed that this
procedure and responsibility was not fully documented. We recommend (as we generally
have in Overarching Recommendation PPM-04) that NERC Compliance more fully
document roles and responsibilities within compliance process documentation.
Recommendations
There are no recommendations specific to this cross-functional area. As noted above,
recommendations for this area are fully covered within overarching sections of this reports and
within observations identified in previous reports to NERC Compliance.
3.4. Processing Reliability Standards Violations

In order to fulfill its functional obligations of overseeing Regional Entity enforcement programs
and tracking and reporting compliance information, NERC Compliance has developed processes
for the receipt and dissemination of information on reliability standard violations. Responsibility
for processing reliability standards violations is shared between the Manager of Compliance
Analysis, Reporting, and Tracking and the Manager of Enforcement and Mitigation. Specifically,
the Compliance Analysis, Reporting, and Tracking team receives new or updated information on
reliability standard violations through submissions to the Compliance Tracking and Reporting
Workbook Tool, which is checked daily by the Compliance Reporting Technical Analyst. The
Compliance Reporting Technical Analyst then informs the Manager of Enforcement and
Mitigation, and other Compliance managers as deemed necessary, of new information in the
Workbook Tool. Other NERC Compliance Managers and analysts are notified depending if it is a
new violation, or a particular update, notice, or document. New violations are reviewed,
analyzed and additional information is requested by the CART. Dismissals are reviewed
internally by CART, and additional supporting data is requested from the region if necessary.
NERC approval or rejection of such dismissal is recommended. The Manager of Enforcement
and Mitigation is responsible for ensuring that information on violations is reviewed and
handled as appropriate. The Compliance Analysis, Reporting, and Tracking team is also
responsible for monitoring and tracking the status of violations, which it does in considerable
detail.
As noted in Appendix I, the following processes make up the cross-functional area Processing
Reliability Standards Violations:
o “Compliance Data Reporting Process” (NPP-CME-701)

o “Compliance Violation and Penalty Proces – RE CEA” (NPP-CME-501)
These process documents were evaluated in the Functional Area Evaluations Overseeing
Enforcement Activities of Regional Entities and Analyzing and Reporting Compliance
Information.
Observations
We observed that each Compliance team responsible for this cross functional area has
drafted its own process document covering the processing of reliability standards violations.
The Compliance Analysis, Reporting, and Tracking team documented this within the
“Compliance Data Reporting Process,” and the Enforcement and Mitigation team
documented this within its “Compliance Violations and Penalty Process.” While the
documents did not appear to have any conflicts per our observations, they did overlap and
go into varying levels of detail. To avoid potential conflicts in future iterations of the
documents, we recommend that the teams should merge the processes into one document
on which they reach mutual agreement. See Recommendation PRO-01.
The implementation of the new Compliance Reporting and Tracking System (CRATS) will
greatly affect how reliability standards violations are received, processed, and tracked. The
process documents related to this functional area should be re-examined and updated for
changes that will occur with the new system. See Overarching Recommendation INF-01.
Recommendations
PRO-01 To eliminate confusion and future conflicts, we recommend that NERC Compliance
consolidate the processes related to the processing of reliability standards
violations into a single process document. The Manager of Enforcement and
Mitigation and the Manager of Compliance Analysis, Reporting, and Tracking
should coordinate the development of this document to ensure that each team’s
processes and requirements are accurately documented, and that the teams have
a mutual understanding of how the overall process should function. The
document needs to clearly delineate the roles and responsibilities of the
respective functions.
Section 4: Functional Area Evaluation
Introduction
In addition to the four cross-functional areas, we have identified ten functional areas that are
more limited in scope, in that they each fall under the direction of one manager within the
Compliance Department. However, these functional areas encompass most of the core
processes that NERC Compliance must carry out in order to accomplish its objectives of
achieving compliance with and enforcing the reliability standards. These ten functional areas
are:
1. Compliance Program Planning

2. Overseeing Registration of Users, Owners, and Operators of the Bulk Power System
3. Overseeing Certification of Users, Owners, and Operators of the BPS
4. Overseeing Compliance Activities of Regional Entities (excluding CVIs)
5. Overseeing Enforcement Activities of Regional Entities
6. Analyzing and Reporting Compliance Information
7. Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs
8. NERC Involvement in Compliance Inquiries and Compliance Violation Investigations
9. Handling Complaints Received on the Hotline and via the Website Appropriately, and
10. Executing Compliance Enforcement Authority Responsibilities
In the following ten evaluation sections we will briefly explain each of these functional areas and
we will provide observations on the execution and documentation of the processes underlying
each area. Where appropriate, we provide recommendations for improving the execution or
documentation of the processes, including any warranted changes to the Rules of Procedure
(ROP).
Our evaluations were based on our review of NERC’s process documentation and on interviews
with NERC personnel. Primarily, we posed a list of questions to the managers in charge of each
functional area, and at least one staff person on each manager’s team. The complete set of
questions asked in each interview is captured in Appendix II – Process Questionnaire.
Each evaluation section will use the following format:

▪ Introduction and Scope – provides introductory text regarding the functional area, including
the basic intent and scope of the functional area and a cross-reference back to the
processes contained within the functional area
▪ Functional Area Criteria and Observations – provides, in tabular format, our observations
and assessment for the functional area across the questionnaire criteria observed, along
with additional observations noted outside of the questionnaire criteria (if applicable)
▪ Recommendations – provides a listing of the recommendations made by Crowe Horwath
based upon observed areas for improvement for the processes within the functional area
Our observations for each functional area in many cases provide an indication of whether we
observed the criteria generally to be met, partially met, not met, or not applicable. As we
noted in the Composite Evaluation section of this report, the meaning of these relative
indicators is as follows:
Met The criterion assessed for the functional area was generally observed to
meet the expectations with respect to sufficiency, quality, and
completeness. Flaws or shortcomings that may exist do not significantly
impact the processes within the functional area to the extent that the
processes would generally fail to operate per their objective or
performance requirements.
Partially Met The criterion assessed for the functional area was generally observed to
not fully meet the expectations with respect to sufficiency, quality, and
completeness. Flaws or shortcomings that exist may impact the processes
within the functional area to the extent that the processes could fail to
fully or consistently achieve their objective or operate within performance
requirements.
Not Met The criterion assessed for the functional area was observed to not meet
expectations with respect to sufficiency, quality, and completeness.
Significant flaws or shortcomings exist which we believe impacts the
processes within the functional area to the extent that the processes have
a significant likelihood to fail and/or to fully and consistently achieve their
objective, operate within performance requirements, or operate
efficiently. With that said, it is worthwhile to note that a “not met” rating
does not mean the process or procedure is in conflict with the rules of
procedure and this rating should not be interpreted as such. The rating is
an indication that there is a higher likelihood of potential problems or
inefficiencies, not an absolute indicator that there were observations of
non-compliance.
Not Applicable The criteria for the functional area were not observed or not applicable
and therefore not assessed.
4.1. Compliance Program Planning

Compliance program planning encompasses NERC’s development of its annual compliance
implementation plan and NERC’s review of the annual compliance implementation plans
submitted by each of the Regional Entities. NERC’s implementation plan explains which
standards are to be actively monitored by the Regional Entities during the year and the methods
by which those standards are to be monitored for compliance. The plan is to be made public by
October 1 of the year prior to implementation. The Regional Entities’ plans, which are due to
NERC around November 1 of each year, explain specifically how each Regional Entity will
implement NERC’s plan within their own region during the year. The Regional Entities’
implementation plans also include a compliance audit schedule for the year. Responsibility for
carrying out the compliance program planning function at NERC lies with the Director of
Regional Operations, who is assisted by a member of his team. They also solicit and receive
input on the annual compliance implementation plan from other people within NERC and the
Regional Entities.
As noted in Appendix I, the following processes make up the Compliance Program Planning
functional area:
o “CMEP Development and Maintenance Process” (NPP-CME-200)

o “CMEP Implementation Plan Process” (NPP-CME-201)
o “Monitoring and Facilitating Effectiveness of the CMEP” (NPP-CME-204)
Compliance Program Planning – Functional Area Criteria and Observations
Criteria Evaluation and Supporting Observations

1 Is the process objective documented for all Partially met. At the time of our
processes within the functional area and are all observations, one of the processes within this
tied to underlying rules and policies? functional area, “CMEP Development and
If Yes and the crosswalk to the ROP is Maintenance Process” (NPP-CME-200.R0)
not apparent, ask which ones had not yet been drafted. (See Overarching
Recommendation PPM-01.) As a result,
NERC Compliance had no documented
internal process to guide changes to the
internal CERCP processes and procedures.
For each of the processes that had been
drafted, an objective had been documented,
and the processes tie to underlying ROP.
2A For this functional area, how well do the Met. The processes that had been
processes currently documented within the documented mapped to the ROP
Processes and Procedures Manual map to or requirements. The process document related
match the Rules of Procedure? to compliance program planning was
developed to document the process as it had
been carried out, keeping the ROP
requirements in mind during development.
The ROP provided significant leeway in how
to execute compliance implementation
planning.
2B For this functional area, how well do the Met. The processes as executed matched the
processes currently executed map to or match ROP requirements. For example, the
the Rules of Procedure? Are there areas deadlines specified in the ROP were met
where there are known exceptions or consistently.
discrepancies?

2C For this functional area, how well do the Partially met. Although the personnel we
processes currently executed map to or match interviewed did not identify any deviations
the Processes and Procedures Manual? Are from the Processes and Procedures Manual,
there areas where there are known exceptions we observed no formal tracking of process
or discrepancies? milestones, other than assuring that the final
due dates specified in the ROP were met via
formalized project management practices
that are utilized to guide the process. See
Overarching Recommendations PPM-06 and
MON-01.
3 How do people know their roles and Not met. The process documentation did not
responsibilities in executing the processes assign key roles and responsibilities within
within this functional area? Are roles and the process itself (meaning, for example, the
responsibilities documented? responsibilities for individual activities or
“steps” within the process), such as who
decides which standards should be actively
monitored and who will be responsible for
drafting the plan. Further, there were no
specific roles for managing comments or
updates from various parties, or for reviewing
Regional Entity implementation plans. We
observed that the responsibilities for
Compliance processes are reflected in the
position descriptions for NERC Compliance
management; however, this observation
addresses the documenting and identification
of responsibilities at a more granular level
(that is, who does what within a process,
versus who is (at an overarching level)
responsible for a process area. See
3A Do resources responsible for executing the Met. Although individual roles were not
process have awareness and understanding of documented in the process, we observed
the process (as documented), and capability to that the persons responsible were aware of
execute the process (e.g. they are trained, the process and demonstrated the capability
have the right skill sets, have the right tools at to execute the process fully and on time.
their disposal, etc.)?

4 Do you and your team have the resources Partially met. we observed that objectives
(personnel, systems, budget, etc.) necessary to were consistently accomplished on time, but
accomplish the objectives of the processes NERC Compliance noted that this is done with
within the functional area? If not, what is constrained resources in terms of time and
lacking? personnel. Members of this team were
responsible for two other major functional
areas as well—Overseeing Compliance
Activities of Regional Entities and Conducting
Reviews of Regional Entities’ Compliance and
Enforcement Programs. See Overarching
Recommendations STA-01, STA-02, and STA-
03.
5 Are the processes within this functional area Partially met. We observed that several
monitored to see that they are consistently parties within and outside NERC reviewed the
performed in accordance with the implementation plans for quality. In addition,
documented process? If so, how? the final deadlines for submitting
implementation plans were monitored for
completion, but intermittent milestones were
not formally monitored. See Overarching
Recommendations PPM-06 and MON-01.
5A If the processes within this functional Partially met. Final implementation plans
area are monitored (i.e. the answer to were measured for whether or not they were
#5 was “yes”), are they also issued by the deadline specified in the ROP.
measured? If so, what measurements However, interim milestones were not
are taken (and how) and what is done monitored or measured.
with these measurements (reported,
etc.)?
5B If the processes within this functional Not applicable. There have been no
area are monitored (i.e. the answer to identified deviations from the process goals
#5 was “yes”), do you do anything with that are monitored.
the cases where the actual execution
deviates from the norm – that is, the
so-called process exceptions?

6 Do the processes within this functional area Partially met. The primary goals and
have specific, measurable goals or objectives? objectives of this process were stated as to 1)
If so, what are those? develop a NERC implementation plan by
October 1 of each year that specifies the plan
for monitoring compliance with reliability
standards in the next year, including
determining which standards are to be
actively monitored; and 2) to review annual
implementation plans submitted by the
Regional Entities for conformance to the
goals in NERC’s implementation plan. NERC
also set informal milestones necessary to
meet these primary goals and objectives.
However, the process document itself did not
formally specify a timeline for completing key
interim milestones, such as development of
the initial implementation plan draft. In
addition, although a deadline of December 1
was noted in the process document as the
date for NERC to complete its review of
Regional Entities’ implementation plans, we
observed a question mark and note next to
this date, indicating that it was not final.
NERC Compliance uses formal project
management practices over the development
of the annual plan each year, including a
project plan with interim milestones used for
tracking progress. See Overarching
Recommendations PPM-02, PPM-06, and
MON-01.
6A If the processes within this functional area Met. The final ROP-mandated deadlines
do have specific, measurable goals (that were being consistently met each year. NERC
is, the answer to #6 was yes), how Compliance uses formal project management
frequently do the processes meet their practices over the development of the annual
goals or objectives? plan each year, including a project plan with
interim milestones used for tracking
progress. However, is worth noting that
there was no observed, consistent
monitoring of intermediate milestones to
determine if they had been met and,
therefore, we could not determine how often
these interim milestones were met. See
MON-01.

6A.1 If the answer to #6A is anything Primary goals and objectives had consistently
other than 100% of the time, then a been met, and we could not determine how
follow-on question: What are the often the intermediate goals and objectives
primary barriers for the processes are met.
to achieve their goals or objectives?
7 Do you consider the processes within this Partially met. We observed that
functional area to be efficient to the extent implementation plans must go through
that unnecessary steps, iterations, resources, several levels of review and input, both
and delays have been eliminated? If not, what within NERC and outside NERC, from the
changes can be made? Regional Entities, the BOTCC, and FERC.
Although it is important that the process is
open and that parties impacted by the
compliance plans have the opportunity to
comment on them, the process of soliciting
input and waiting on impacted parties to
provide the input has shown itself to be time
consuming. See Overarching
Recommendation ENT-05.
8 How is process documentation made available Met. We observed that process
to personnel? documentation was available to personnel on
a shared network drive.
9 How is process documentation controlled for Met. We observed that only the Compliance
changes? Do you have a process for formally Process Administrator had rights to edit the
responding to and dealing with changes that process documentation.
would come down from FERC (for example)?
We also noted the following additional observations:

The process documents that were developed for this functional area were in draft form
at the time of our data gathering and had not been reviewed by management. See
We observed that the “CMEP Implementation Plan Process” document did not specify
the need for an internal review of NERC’s draft implementation plan by a manager
before submittal to outside parties for comment. Although in practice the
implementation plans were drafted with input from a number of individuals within
NERC, an overall review of the implementation plan by a manager prior to submittal to
FERC for approval will help to assure the overall quality of the draft and avoid
unnecessary errors or misunderstandings. See Recommendation IMP-01.
The “CMEP Implementation Plan Process” document did not specify any minimum
criteria that Regional Entities’ compliance implementation plans must meet in order to
receive NERC approval. Such developed criteria would provide a guide to NERC
Compliance personnel and help achieve uniformity across the regions. See Overarching
The “CMEP Implementation Plan Process” document also did not address updating
implementation plans for significant changes to the compliance program that may occur
during the year. For example, if a new standard is approved by FERC during the year,
NERC Compliance must decide whether or not the standard should be actively
monitored during the year, and whether that in turn warrants a modification to the
NERC and regional implementation plans. We observed that NERC Compliance had no
formal documented process for implementing such changes. See Recommendation
IMP-02.
Recommendations
Rec Id Recommendation
IMP-01 Modify the “CMEP Implementation Plan Process” document (NPP-CME-201.R0) to
include a requirement for management review of NERC’s implementation plan draft
prior to submitting the plan to outside entities for review.
IMP-02 Develop, document, and approve a formal process for addressing significant changes
to the compliance program that occur during the year. Include roles and
responsibilities for deciding what modifications need to be made to implementation
plans, Reliability Standard Audit Worksheets, and other impacted NERC Compliance
documents and templates, and for actually making those modifications. Include
provisions for internal management review and obtaining input from FERC, Regional
Entities, and other stakeholders.
4.2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System

As mandated by the Federal Power Act, NERC is responsible for registering owners, users, and
operators of the bulk power system (BPS), and NERC maintains the official compliance registry.
However, it is primarily the delegated responsibility of Regional Entities to identify the owners,
users, and operators of the BPS that must register within their respective regions. At NERC, the
Manager of Organization Registration and Certification oversees this process and has one staff
person to receive and process registry updates from the Regional Entities. NERC Compliance
also hears appeals from entities that do not believe they should be registered for certain
functions. The Manager of Organization Registration and Certification manages this process as
well, although the NERC Board of Trustees Compliance Committee (BOTCC) actually hears the
appeals, and NERC’s Legal department is also involved.
As noted in Appendix I, the following processes make up the functional area for Overseeing
Registration of Users, Owners, and Operators of the Bulk Power System:
o “Organization Registration Process” (NPP-CME-100)

o “Organization Registration Appeals Procedure” (NPP-CME-102)
Overseeing Registration of Owners/Users/Operators of the Bulk Power System – Functional

Area Criteria and Observations

1 Is the process objective documented for all Partially met. We observed that the
processes within the functional area and are Registration process document specified a
all tied to underlying rules and policies? “Purpose” that is noted as “To describe
If Yes and the crosswalk to the ROP is
processes and procedures related to NERC's
not apparent, ask which ones compliance registration,” but it did not state
what the objective of the compliance
registration process is. See
Recommendation REG-01.
2A For this functional area, how well do the Met. No discrepancies were noted between
processes currently documented within the the ROP and the Registration Policies and
Processes and Procedures Manual map to or Procedures as documented.
match the Rules of Procedure?
2B For this functional area, how well do the Partially met. In 2009, one discrepancy was
processes currently executed map to or match noted between the ROP and the Registration
the Rules of Procedure? Are there areas Policies and Procedures as executed.
where there are known exceptions or Specifically, one update of the compliance
discrepancies? registry was not sent to NERC Compliance
within one month as specified in the ROP.
The update was for February 2009 and was
made within 30 days, but not within the
month of February. No other discrepancies
were noted.

2C For this functional area, how well do the Partially met. In 2009, one discrepancy was
processes currently executed map to or match noted between the Registration Policies and
the Processes and Procedures Manual? Are Procedures as documented and the Policies
there areas where there are known exceptions and Procedures as executed. Specifically,
or discrepancies? one update of the compliance registry was
not sent to NERC within one month as
specified in the ROP and process document.
The update was for February 2009 and was
made within 30 days, but not within the
month of February. No other discrepancies
were noted.
However, we observed that not all process

points were being formally tracked. For
example, the Registration Appeals Procedure
included a number of time-sensitive goals.
However, because there was no tracking of
these items, we could not determine
whether or how frequently they were being
met.
In addition, NERC Compliance stated that it

had a practice of reviewing changes
submitted to the compliance registry. In
particular, NERC Compliance noted that it
looked for proper justification of significant
changes, such as removals from the
compliance registry and changes to the
functions for which an entity is registered.
However, this process was not documented.
See Recommendation REG-02.
3 How do people know their roles and Partially met. Specified individual roles and
responsibilities in executing the processes responsibilities were documented in the
within this functional area? Are roles and “Organization Registration Process” (NPP-
responsibilities documented? CME-100.R0), but not in the “Organization
Registration Appeals Procedure” (NPP-CME-
102.R0). In this document, responsibilities
were listed as belonging to “NERC,” and it did
not specify who or what department within
NERC was responsible for each item in the
process. See Overarching Recommendation
PPM-04.

3A Do resources responsible for executing the Partially met. While the personnel and
process have awareness and understanding of managers at NERC Compliance that we spoke
the process (as documented), and capability to to demonstrated an understanding of the
execute the process (e.g. they are trained, Registration processes and policies, we
have the right skill sets, have the right tools at observed that they did not have the tools to
their disposal, etc.)? carry out the processes in the most efficient
and effective manner. Specifically, they had
to rely on manually processing information
from Regional Entities and other parties. This
left the process open to errors in recording
compliance registry updates and other
information. The new Compliance Reporting
and Tracking System (CRATS) will automate
the process by allowing for direct entry of
information by Registered Entities, with
review by Regional Entities and NERC.
However, at the time of our observations,
this system was not expected to be
functional for several months.
4 Do you and your team have the resources Partially met. As noted above, the
(personnel, systems, budget, etc.) necessary to Registration Process relied on manual
accomplish the objectives of the processes updates, which increased the risk of errors
within the functional area? If not, what is occurring.
lacking?
5 Are the processes within this functional area Partially met. We observed that two
monitored to see that they are consistently Registration metrics were tracked in the
performed in accordance with the monthly Goal 1 Update report: Notification
documented process? If so, how? of Registered Entities within 10 business days
of inclusion on registration list, and Monthly
posting of the NERC Compliance Registry per
ROP requirements. However, the primary
staff person responsible for carrying out the
Registration process was not aware of any
formal monitoring of the process. In
addition, there were no metrics formally
monitored related to the registration appeals
process. See Overarching
5A If the processes within this functional Partially met. The process points that were
area are monitored (i.e. the answer to being monitored were being measured in
#5 was “yes”), are they also terms of length of time and were reported
measured? If so, what measurements monthly on the Goal 1 Update report.
are taken (and how) and what is done However, as noted under the response to
with these measurements (reported, Criterion 5 above, other process points were
etc.)? not being monitored and thus were not
measured.

5B If the processes within this functional Met. In 2009, one update of the compliance
area are monitored (i.e. the answer to registry was not sent to NERC within one
#5 was “yes”), do you do anything month as specified in the ROP and process
with the cases where the actual document. The update was for February
execution deviates from the norm – 2009 and was made within 30 days, but not
that is, the so-called process within the month of February. Management
exceptions? brought the issue to the analyst’s attention,
and no other errors have occurred.
6 Do the processes within this functional area Partially met. We observed that the
have specific, measurable goals or objectives? Registration and Registration Appeals
If so, what are those? processes had some specific, measurable
goals, such as a requirement to provide
monthly reports to FERC. However,
measurable goals related to several key
milestones were not provided. For example,
there was no established or documented
timeline for NERC Compliance to review
proposed updates to the compliance registry.
See Overarching Recommendations PPM-06
and MON-01.
6A If the processes within this functional Partially met. For the goals that we observed
area do have specific, measurable goals were being measured and tracked, only one
(that is, the answer to #6 was yes), how process exception had been noted.
frequently do the processes meet their However, not all goals and objectives were
goals or objectives? formally tracked. For example, the
Registration Appeals Procedure included a
number of time-sensitive goals. However,
because there was no tracking of these goals,
we could not determine whether or how
frequently they were met. See Overarching
6A.1 If the answer to #6A is anything For the one process exception that was
other than 100% of the time, then a noted, the primary barrier was a
follow-on question: What are the misunderstanding of the technical nature of
primary barriers for the processes the requirement. One update of the
to achieve their goals or objectives? compliance registry was not sent to NERC
within one month as specified in the ROP and
process document. The update was for
February 2009 and was made within 30 days,
but not within the month of February.
7 Do you consider the processes within this Not met. We observed that the Registration
functional area to be efficient to the extent process relied heavily on manual input and
that unnecessary steps, iterations, resources, processing and was therefore inefficient. The
and delays have been eliminated? If not, what new Compliance Reporting and Tracking
changes can be made? System (CRATS) will make the process more
efficient once implemented. However, at the
time of our data gathering, this
implementation was months away.

8 How is process documentation made available Met. The Registration process was written
to personnel? by the employee who was responsible for
carrying it out. The Registration Appeals
process was available on the shared network
drive.
changes? Do you have a process for formally Process Administrator had rights to edit the
responding to and dealing with changes that process documentation.
We also noted the following additional observations:

At the time of our data gathering, the process documents that had been developed for
this functional area were in draft form and had not been reviewed by management. See
At the time of our data gathering, the process for registering entities was observed to be
in a state of flux. The registration process document reflected the requirements for
updating information in the manual registration system. However, NERC Compliance
was in the process of implementing the new Compliance Reporting and Tracking System
(CRATS) that will automate the process and allow regional entities to update their
information directly into the system. Once the new system is implemented, it will be
crucial that the process document is updated in order to assure that personnel are
performing the proper steps to register entities in an accurate, timely manner. See
overarching Recommendation INF-01.
Once the Registration process becomes more automated, process monitoring will be
especially critical. On one hand, the capability for running system reports will enhance
the monitoring process and make it easier to determine when key milestones have been
met. However, NERC Compliance will need to institute controls within the system to
assure that updates are accurately and completely recorded and that proper
justifications are made for significant registration changes. NERC Compliance told us
they were considering such process controls in the system, but had not yet decided
what controls will be implemented. For example, the new system will include a
comment field in which entities will record justifications for any significant changes to its
compliance status, such as changing the functions for which it is registered or removing
itself from the compliance registry. NERC Compliance stated that they were intending
to either 1) block the submission of such changes until these justifications can be
reviewed by NERC Compliance, or 2) produce a daily report to NERC showing what
changes have been made. Because the new process will give significant control to
registered entities for directly changing their compliance status, we strongly
recommend that NERC Compliance implement the first option. See Recommendation
REG-03.
Recommendations
REG-01 In creating a new registration process document, we recommend that NERC
Compliance includes a Purpose or Objective section that describes not only the
purpose of the document itself but also the purpose of the registration process.
REG-02 We recommend that NERC Compliance include its procedures for assuring that
compliance registry changes are properly reviewed and justified within the new
registration process document.
REG-03 In implementing the new registration software system, we recommend that NERC
Compliance include functionality for blocking submission of changes to the
compliance registry until NERC has reviewed each change and recorded its approval
within the system. This option is preferable to producing a daily report after the fact,
because a risk exists that this report may not be reviewed thoroughly enough or at all.
4.3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System

NERC has determined that certain functions—Reliability Coordinators (RC), Balancing Authorities
(BA), and Transmission Operators (TOP)—are so critical to the reliability of the bulk power
system that a process is needed to ensure that entities performing these functions have the
ability to do so. The process devised to ensure this is certification. Under this process, NERC
and/or Regional Entities perform a formal review of the functional and compliance capabilities
of these entities. NERC then determines whether or not the entity should be certified. At NERC,
this process is managed by the Manager of Organization Registration and Certification.
Certification of Users, Owners, and Operators of the Bulk Power System:
o “Organization Certification Process Procedure” (NPP-CME-101)

o “Organization Certification Appeals Procedure” (NPP-CME-103)
Overseeing Certification of Owners/Users/Operators of the Bulk Power System – Functional


1 Is the process objective documented for all Met. Process objectives related to
processes within the functional area and are all Certification were documented and tied to
tied to underlying rules and policies? underlying ROP.
not apparent, ask which ones

2A For this functional area, how well do the Partially met. The certification process for
processes currently documented within the entities that have never been certified
Processes and Procedures Manual map to or matched the ROP. However, the
match the Rules of Procedure? documented process for certifying entities
that have already been known to be
operating as a BA, TOP, or RC was not
accurately documented within the ROP. We
observed that the ROP outlined a
“Transitional Certification” procedure for
these entities, which expired January 1, 2009,
and was never implemented. Instead, NERC
told us that a different process that was in
effect,known as “Provisional Certification.”
Under this process, if an entity that had been
operating as a BA, TOP, or RC underwent a
readiness evaluation and a compliance audit
by June 18, 2009, and fixed any required
issues, the entity was deemed certified. The
Provisional Certification process was not
documented in the ROP, but was included in
a version of the ROP that was pending review
by the BOTCC at the time of our information
gathering. See Recommendation CER-01.
However, by the time this review was
scheduled to be completed, the Provisional
Certification process will no longer be
applicable. NERC Compliance noted that
these changes have been drafted for some
time, but the changes had to go through
multiple levels of review, which took a
significant amount of time. As a result, these
changes were not included in prior versions
of the ROP. See Overarching
Recommendation ENT-05.
2B For this functional area, how well do the Partially met. NERC Compliance noted one
processes currently executed map to or match instance in which a certification review team
the Rules of Procedure? Are there areas member who was not a NERC employee
where there are known exceptions or breached confidentiality rules. (See
discrepancies? Recommendation CON-03 in the cross-
functional area Compliance Program
Confidentiality Requirements.) The issue
was resolved to the satisfaction of the entity
undergoing the review. In addition, as noted
above, the Provisional Certification process as
currently implemented was not included in
the ROP.

2C For this functional area, how well do the Partially met. As stated above, NERC
processes currently executed map to or match Compliance noted one instance in which a
the Processes and Procedures Manual? Are certification review team member who was
there areas where there are known exceptions not a NERC employee breached
or discrepancies? confidentiality rules.
3 How do people know their roles and Partially met. Individual roles and
responsibilities in executing the processes responsibilities were identified within the
within this functional area? Are roles and process documents. However, NERC
responsibilities documented? informed us that NERC’s Legal Department
was heavily involved in the Certification
Appeals process, but we observed that no
mention of their role was provided in the
related process document. See
Recommendation CER-02.
3A Do resources responsible for executing the Partially met. We observed that NERC’s
process have awareness and understanding of Certification team was led by an individual
the process (as documented), and capability to with in-depth knowledge and experience in
execute the process (e.g. they are trained, the Certification process. NERC noted that to
have the right skill sets, have the right tools at fulfill the training requirements in the ROP,
their disposal, etc.)? Certification team members must read the
ROP, interview with NERC on conflicts of
interest, and attend NERC’s Quality of
Evidence training. However, as noted in our
management letter to NERC following a
recently completed agreed-upon procedures
engagement for a regional entity, NERC was
not able to provide evidence that the
members of a Certification team received
training as required under the ROP. Further,
because NERC stated that Certification teams
were primarily made up of outside industry
experts and that the team intended to hire
new NERC staff, a more comprehensive
training program than the one described
should be developed. See Recommendation
CER-03.

4 Do you and your team have the resources Partially met. During the time of our
(personnel, systems, budget, etc.) necessary to information gathering, we observed that only
accomplish the objectives of the processes one full time staff at NERC worked on
within the functional area? If not, what is Certification reviews. NERC Compliance
lacking? noted that personnel constraints had been an
issue, but should be resolved, as they were
looking to fill two open positions. NERC
Compliance also noted that the Certification
team used outside industry experts to work
on Certification reviews, but they would like
to move to primarily using NERC and Regional
Entity staff. See Overarching
03.
5 Are the processes within this functional area Met. We observed that the Certification
monitored to see that they are consistently process was primarily monitored through the
performed in accordance with the use of feedback questionnaires provided to
documented process? If so, how? entities undergoing a Certification review.
One regional entity observed also monitored
its adherence to Certification review
timelines. In addition, the Manager of
Registration and Certification told us that he
occasionally spot checks Certification reviews
against the ROP requirements. NERC also
noted that FERC was heavily involved in
directly monitoring Certification reviews.
5A If the processes within this functional Met. We observed that the process was
area are monitored (i.e. the answer to measured in terms of timelines for
#5 was “yes”), are they also Certification review milestones. Feedback
measured? If so, what measurements questionnaires provided to entities
are taken (and how) and what is done undergoing a Certification review were also
with these measurements (reported, scored, and the goal for one regional entity
etc.)? observed was an average monthly score of
greater than 85 percent. Both adherence to
timelines and feedback results were reported
in NERC’s monthly Goal 1 Update Report.
5B If the processes within this functional Met. NERC Compliance has identified cases
area are monitored (i.e. the answer to in which execution deviates from the norm in
#5 was “yes”), do you do anything with a “lessons learned” document, which was
the cases where the actual execution shared with Certification review team
deviates from the norm – that is, the members. One item noted was a
so-called process exceptions? confidentiality breach committed during a
Certification review. The breach was
addressed with the appropriate parties, and
the entity agreed with the actions taken by
NERC Compliance.

6 Do the processes within this functional area Met. Certification process goals and
have specific, measurable goals or objectives? objectives were outlined in prepared
If so, what are those? timelines and in the questionnaires provided
to entities undergoing Certification reviews.
6A If the processes within this functional area Partially met. We observed that process
do have specific, measurable goals (that goals and objectives were usually met, but a
is, the answer to #6 was yes), how few deviations have been noted. For
frequently do the processes meet their example, the feedback scores have been
goals or objectives? around 90 to 98 percent.
6A.1 If the answer to #6A is anything We observed that the primary barrier was
other than 100% of the time, then a NERC’s heavy reliance on outside experts for
follow-on question: What are the performing Certification reviews. For
primary barriers for the processes example, the “lessons learned” document
to achieve their goals or objectives? noted that some team members were
spending too much time taking care of
outside work and were not fully focused on
performing the Certification reviews while
onsite at entities’ facilities. In addition, one
outside expert breached NERC’s
confidentiality requirements on a
Certification review. During our information
gathering, NERC Compliance noted that they
were hiring additional staff to work on
Certification reviews and were planning to
rely primarily on NERC Compliance and
Regional Entity staff in the future.
7 Do you consider the processes within this Met. No inefficiencies have been identified
functional area to be efficient to the extent in the Certification processes, but NERC’s
that unnecessary steps, iterations, resources, Certification staff noted substantial delays in
and delays have been eliminated? If not, what revising Appendix 5 to the ROP. See
changes can be made? Overarching Recommendation ENT-05.
8 How is process documentation made available Met. The process document was available
to personnel? for viewing on a secure network drive.
9 How is process documentation controlled for Met. Only one employee had access to revise
changes? Do you have a process for formally the process document.
responding to and dealing with changes that
We also noted additional observations related to the Certification process documents:

We observed that the process documents that had been developed for this functional
area were in draft form and had not been reviewed by management. See Overarching
We observed that the process diagram within the Certification process document
showed that Regional Entities send schedule and information requests to an entity
being certified, but the document text stated this is the NERC Certification Team Lead
responsibility. NERC Compliance confirmed that the document text was correct. See
Recommendation CER-04.
The Certification process diagram implied that all parties must agree on a
recommendation to certify an entity, but the document text implied that only a majority
opinion is needed. NERC Compliance confirmed that the text is correct, but noted that if
a Certification review involves multiple regions, the regions must agree unanimously.
See Recommendation CER-05.
Recommendations
CER-01 Expedite the revision of Appendix 5 of the ROP to reflect the deletion of “Transitional
Certification” and the implementation of “Provisional Certification.”
CER-02 Document the Legal Department’s role in the “Organization Certification Appeals
Procedure” (NPP-CME-103.R0). Although for purposes of a Compliance Department
document, it is not necessary to include all the steps in the Legal Department’s own
procedure, it would be helpful for staff to have a resource for understanding Legal’s
role in the certification appeals process.
CER-03 Develop a more comprehensive training program for the Certification review process.
Topics to cover should include: the Certification policies and processes as spelled out
in the ROP and the Certification process document (NPP-ORC-002); confidentiality
and conflict of interest rules; evidence reviewed during the Certification process; the
Certification feedback process; and, lessons learned and best practices identified
during prior Certification reviews.
CER-04 Revise the diagram in the Certification process document to show that sending
schedule and information requests to an entity being certified is the responsibility of
the NERC Certification Team Lead.
CER-05 Revise the diagram and text of the Certification process document to show that only a
majority opinion is need to certify an entity, but that if a Certification review involves
multiple regions, the regions must agree unanimously.
4.4. Overseeing Compliance Activities of Regional Entities (excluding CVIs)

Through their delegation agreements with NERC and the ROPs, Regional Entities have most of
the responsibility for conducting the activities that monitor Registered Entity compliance with
the reliability standards. The primary monitoring methods used are compliance audits, spot
checks, self-certifications, self-reporting of violations, periodic data submittals, exception
reporting, compliance violation investigations (CVIs), and complaints of potential violations. The
ROP requires that Regional Entities notify NERC of compliance audits, CVIs, and complaints early
in these processes, and provides NERC the option of being involved in direct oversight of
Regional Entities’ conduct of these monitoring methods. “Day-to-day” oversight of CVIs and
complaints is handled by NERC’s Manager of Compliance Violation Investigations, and NERC’s
roles in these areas are discussed in different individual functional area evaluations. The
Director of Regional Operations is responsible for “day-to-day” oversight of the Regional
Entities’ compliance audit programs. The Director of Regional Operations and his staff oversee
Regional Entity performance of compliance audits through three processes: direct observations
of the performance of certain compliance audits, self certifications of compliance from Regional
Entities on audits that NERC did not directly observe, and reviews of information provided by
Regional Entities pursuant to their performance of compliance audits.
Compliance Activities of Regional Entities (excluding CVIs):
o “Observation of Regional Entity-led Compliance Audits” (NPP-CME-400)

o “Regional Entity-led Compliance Audit Process” (NPP-CME-401)
o “Procedure for the Regions to Self-Certify Adherence to the ROP and CMEP during and
Audit” (NPP-CME-402)
Overseeing Compliance Activities of Regional Entities – Functional Area Criteria and

Observations

1 Is the process objective documented for all Met. we observed that all process objectives
processes within the functional area and are all were documented and tied to underlying
tied to underlying rules and policies? Rules of Procedure.
2A For this functional area, how well do the Met. We observed that the processes
processes currently documented within the matched the ROP requirements. However, it
Processes and Procedures Manual map to or should be noted that, as with most of the
match the Rules of Procedure? processes we observed, the documents were
developed based on the way the process is
done, keeping the ROP in mind. They were
not developed starting from the ROP.
2B For this functional area, how well do the Partially met. NERC Compliance did not know
processes currently executed map to or match of any discrepancies between the ROP and
the Rules of Procedure? Are there areas the way the processes were executed.
where there are known exceptions or However, as noted under Criterion 5 below, a
discrepancies? number of key process components were not
being monitored; therefore, NERC
Compliance could not know of any deviations
of these process points.

2C For this functional area, how well do the Not met. We observed that NERC
processes currently executed map to or match Compliance did not consistently meet the
the Processes and Procedures Manual? Are deadlines outlined in their processes, such as
there areas where there are known exceptions the 30 day deadline for issuing a compliance
or discrepancies? audit observation report. For instance, as of
May 8, 2009, only 2 of 11 observation reports
had been completed by the 30 day deadline.
The Director of Regional Operations
attributed this mainly to the team’s heavy
workload and that the priority of meeting
these goals had not been emphasized clearly
enough. See Overarching Recommendations
MON-03, STA-01, STA-02, and STA-03.

3 How do people know their roles and Partially met. While we observed that
responsibilities in executing the processes specific roles and responsibilities were
within this functional area? Are roles and identified in the audit observation process
responsibilities documented? before and after an observation occurs, the
role of a NERC observer on a Regional Entity
compliance audit had not been fully defined.
NERC management and staff told us that over
time, NERC’s role evolved from direct
participation in audits, to providing guidance
during audits, to simply observing the audit
process onsite and providing feedback after
the audit in the form of an observation
report. However, the role of only observing,
with no direct participation, was not
documented within the process and had not
been formalized. Indeed, the process
document was silent on what NERC’s role
would be while an audit is being performed.
Although NERC Compliance has
communicated its role as observers to team
members, Regional Entities, and FERC
through a presentation and other less formal
discussions, NERC Compliance did not have
any written policy, process, or formal
directive to outline what activities were
allowed or prohibited by NERC observers
while onsite during an audit. See
Recommendation COM-01. In addition, we
observed that roles and responsibilities were
not clearly identified in the “Regional Entity-
led Compliance Audit Process” (NPP-CME-
401). The document referred to the
responsibilities of “NERC” but not to any
specific individuals or functions within NERC.
Further, although the “Procedure for the
Regions to Self-Certify Adherence to the ROP
and CMEP during and Audit” (NPP-CME-402)
included Regional Entities’ responsibilities for
self-certifying to requirements on audits it
conducts that NERC does not attend, there
was no documented process for NERC
Compliance to follow to track or review these
self-certifications. See Overarching

3A Do resources responsible for executing the Met. Staff demonstrated awareness of the
process have awareness and understanding of process and received compliance audit
the process (as documented), and capability to training.
execute the process (e.g. they are trained,
have the right skill sets, have the right tools at
their disposal, etc.)?
4 Do you and your team have the resources Not met. We observed that the team had not
(personnel, systems, budget, etc.) necessary to been consistently meeting process deadlines.
accomplish the objectives of the processes This occurred even though the number of
within the functional area? If not, what is audits the Regional Operations team
lacking? observed decreased significantly from 50% of
the 388 compliance audits performed in 2008
to 10% of the 467 audits scheduled to be
completed in 2009. Management attributed
this to the need to perform quality work with
staffing level limitations, which constrained
timeliness. In addition to the Oversight of
Regional Entity Compliance, staff members
on this team were responsible for a number
of other functional areas, including
Compliance Program Planning, Conducting
Reviews of Regional Entities’ Compliance
Audit Programs, and significant portions of
Compliance Training. We observed that the
process of auditing Regional Entities was
taking up a significant amount of the team’s
time. Importantly, the Manager of
Registration and Certification was working
with the Director of Regional Operations to
provide staff to help with future Regional
Entity audits, which should alleviate some of
the difficulties the Regional Operations team
has had with meeting deliverables in other
areas. See Overarching Recommendations
STA-01, STA-02, and STA-03.

5 Are the processes within this functional area Partially met. Certain points in the
monitored to see that they are consistently compliance audit observation process were
performed in accordance with the monitored, most notably the development of
documented process? If so, how? the audit observation report. In addition,
quality reviews were performed on the
process outputs, such as the audit
observation reports. However, we observed
that other significant milestones in this
process were not being formally monitored
that are worthy of consideration for formal
monitoring, such as the deadline for
developing the audit observation schedule,
notifications of audit observations (governed
by the CMEP which generally - but not always
- requires 60 day notice to the registered
entity), and completion of the audit checklist
(audit checklists are prepared on site and
serve as the basis for the audit observation
reports). In addition, the oversight process
for compliance audits that were not directly
observed was not being monitored at all. See
MON-01.
area are monitored (i.e. the answer to being monitored were also being measured,
#5 was “yes”), are they also specifically the number of audit observations
measured? If so, what measurements performed and the timeline for reporting on
are taken (and how) and what is done audit observations. These were reported in
with these measurements (reported, the monthly Goal 1 Update Report. However,
etc.)? as noted in the response to Criterion 5 above,
most significant points in the processes were
not monitored or measured at all.
5B If the processes within this functional Not met. We observed that the Compliance
area are monitored (i.e. the answer to Audit Observation team had not been
#5 was “yes”), do you do anything with consistently meeting its measured internal
the cases where the actual execution deadlines. The Director of Regional
deviates from the norm – that is, the Operations had reminded the team of the
so-called process exceptions? need to be timely, but no other actions had
been taken. As noted in the response to
Criterion 6A.1 below, this team has
undergone significant management changes,
which affected the communication of
priorities to staff. See Overarching
Recommendation MON-03.

6 Do the processes within this functional area Partially met. We observed that the audit
have specific, measurable goals or objectives? observation process had a number of
If so, what are those? measurable goals and objectives, such as
determining the audits to attend in the
following program year by the end of the
fourth quarter, and developing the audit
observation reports within 30 days of the end
of the audit. However, we observed that the
processes for reviewing regional entity audit
reports and receiving self-certifications from
Regional Entities on audit compliance lacked
measurable goals, such as timelines. See
MON-01.
6A If the processes within this functional area Not met. As noted above, only one process
do have specific, measurable goals (that had measurable goals and objectives. Of
is, the answer to #6 was yes), how those goals that were being measured, the
frequently do the processes meet their goals are often not met. For example, as of
goals or objectives? May 8, 2009, only 2 of 11 observation reports
had been completed by the 30 day deadline.
In addition, as noted above, several of the
measurable objectives within the audit
observation process were not being formally
monitored or measured, so how frequently
these goals were met was not known.

6A.1 If the answer to #6A is anything Inadequate staffing and scheduling
other than 100% of the time, then a inflexibility are the primary barriers to
follow-on question: What are the achieving the goals and objectives. In
primary barriers for the processes addition to the Oversight of Regional Entity
to achieve their goals or objectives? Compliance, staff on this team was
responsible for a number of other functional
areas, including Compliance Program
Planning, Conducting Reviews of Regional
Entities’ Compliance Audit Programs, and
significant portions of Compliance Training.
We observed that the process of auditing the
Regional Entities was taking up a significant
amount of the team’s time. In addition,
Regional Entity audits that were being
observed took place on Regional Entities’
schedules, and the resulting deadlines flow
from that schedule (e.g., the audit end
dates), regardless of competing priorities that
NERC staff may have. NERC’s only recourse
was to not perform certain audit
observations. As a result, with competing
schedule priorities, NERC Compliance has
performed fewer observations of Regional
Entity’s audits.
In addition, this team has undergone several

management changes within the last year
prior to our data gathering process, which
resulted in not having a consistent
managerial presence. As a result,
management noted that there was not an
ongoing emphasis on the need to achieve
timely results and meet other process
objectives.

7 Do you consider the processes within this Not met. We observed that the process of
functional area to be efficient to the extent observing regional entity audits itself has not
that unnecessary steps, iterations, resources, been the most efficient use of NERC’s time.
and delays have been eliminated? If not, what The actual on-site observations have taken
changes can be made? several days and required a significant
amount of follow-up work, including
documenting final scores and writing a
report. NERC Compliance often did not meet
its goals for completing the observation
process within its internally established
timelines. In addition, the dates of the audits
were determined by the Regional Entities and
left NERC Compliance with little flexibility in
terms of scheduling around competing
priorities. Although NERC Compliance had
the flexibility to determine which audits it
attended, this generally has resulted in fewer
audits being covered by observations or any
other means of oversight.
While observing audits has been a process

that FERC has endorsed and expected NERC
Compliance to perform, there was no actual
requirement to perform audit observations.
NERC Compliance should consider under
which circumstances observations should be
made and should consider other methods for
monitoring Regional Entity compliance
audits. NERC Compliance has developed such
processes, such as conducting spot checks,
but these methods had not yet been
employed, and at the time of our
observations, it was not clear when they
would be implemented to provide some
flexibility or alternatives.
NERC Compliance may also need to consider

whether its audit observation processes
might be amended where potentially only
parts of an audit would be observed versus
the full audit. Another consideration might
be an alternative whereby NERC Compliance
would perform post-audit desk reviews
instead of on-site observation. See
Recommendation COM-02.

8 How is process documentation made available Met. We observed that process
to personnel? documentation was available on a shared
network drive. Staff members have
demonstrated familiarity with the
documents.
changes? Do you have a process for formally Process Administrator has edit rights to the
responding to and dealing with changes that process documents. All others had read-only
would come down from FERC (for example)? access.
In addition, we identified the following observations:

We observed that the process documents that had been developed for this functional area
were in draft form and had not been reviewed by management. See Overarching
NERC’s ROP required that Regional Entities notify NERC about their performance of five of
the eight monitoring methods (self-certifications, self-reports, periodic data submittals,
exception reports, and spot checks) only if the Regional Entity has identified a possible
violation. At that point the process would be in the enforcement phase.2 We observed that
the Director of Regional Operations had the ability to review Regional Entities’ performance
of its responsibilities related to these methods “after the fact” (discussed in the functional
area evaluation “Review Of RE Compliance And Enforcement Programs”), but NERC
Compliance had no process for directly overseeing these methods prior to the enforcement
phase and lacked real-time visibility of the Regional Entities’ performance in these areas. In
a prior consulting report, we recommended that NERC Compliance revise the ROP to allow
for more direct oversight in these areas. See Overarching Recommendation ROP-01. If
NERC Compliance changes its ROP to reflect this, it will also need to establish processes and
responsibilities for accomplishing these oversight objectives. See Recommendation COM-
03.
2
NERC had been heavily involved in monitoring self certifications and spot checks related to the critical
infrastructure protection (CIP) standards. However, NERC management was not sure how long this would be a
continuing effort on NERC’s part, because the process had not been fully negotiated between NERC and the
Regional Entities.
We observed that NERC Compliance only reviewed the compliance audit reports for
conformance to the Audit Report Template, which we observed to be primarily a document
indicating what types of information needs to be in an audit report. Unless the audit was
one that NERC Compliance was directly observing, or if it was one of the eight audits that
NERC Compliance was reviewing as part of its three-year assessment, NERC Compliance
stated that they did not evaluate the evidence presented in the audit report for support for
the validity of the Regional Entity’s findings. See Recommendation COM-04.
We observed no method for tracking the status of Regional Entity compliance audits; as a
result, we observed NERC Compliance did not know at what point Regional Entities were at
in their auditing process until NERC Compliance received a final audit report. In addition,
NERC Compliance did not track whether scheduled audits took place or whether the
Regional Entity’s audit schedules met the requirements of the ROP that Balancing
Authorities, Transmission Operators, and Reliability Coordinators receive an audit at least
every three years. See Recommendation COM-05.
We observed that NERC Compliance had no formalized process for implementing changes to
reliability standards or other NERC Compliance directives within the Reliability Standard
Audit Worksheets (RSAWs), the standard working papers that were being used to record
findings and determinations of compliance with reliability standards on an audit.
Stakeholders have addressed with NERC Compliance the need to update the RSAWs in a
timely manner. See Recommendation COM-06.
NERC Compliance noted that their view of the audit observer role differed somewhat from
FERC’s. This was because NERC Compliance believed the observer should be watching and
recording how Regional Entities conduct an audit, and should not participate in the auditing
process or provide real-time advice to the Regional Entity. FERC audit observers, however,
tended to take a more active role in the process. This has led to some potentially confusing
situations for Regional Entities, especially when both NERC and FERC were observing an
audit. NERC Compliance stated that they were working with FERC to provide better
explanations of the roles of NERC and FERC observers on a Regional Entity’s audit.
Recommendations
COM-01 Document the role of a NERC observer on a Regional Entity audit. Specify what
activities the observer must perform while onsite and what activities must not be
performed onsite. Document this role within the process document “Observation of
Regional Entity-led Compliance Audits” (NPP-CME-400.R0).
COM-02 Decrease the reliance on direct observations as the primary method for monitoring
Regional Entity compliance audits. Develop a risk-based approach for assessing which
audits should be monitored by observation and which should be monitored by other
methods, such as spot checking, self certifications, or more in-depth reviews of
compliance audit reports and the evidence presented within them.
Consider whether audit observation processes might be amended where potentially

only parts of an audit would be observed versus the full audit. Consider an
alternative whereby NERC Compliance would perform post-audit desk reviews
instead of on-site observation.
COM-03 Implement and document processes to allow for more direct oversight of the
following Regional Entity monitoring methods: spot checks, self certifications,
periodic data submittals, self reports, and exception reports. Include as part of the
process a requirement for Regional Entities to periodically report the status of these
methods even when no violation has been identified. In addition, include monitoring
procedures, such as spot checks, for NERC Compliance to perform to assess Regional
Entity compliance with the ROP in these areas.
COM-04 Develop and implement procedures for reviews of compliance audit reports that
include an evaluation of the evidence and conclusions within the report as well as
conformance to the Audit Report template. Include a risk-based process for
determining which audit reports will undergo this more in-depth review and how
often a Regional Entity reports will undergo such a review (e.g., yearly spot checks
plus one major review every three years during the Regional Entity audit). In keeping
with Recommendation COM-03, consider substituting some audit observations with
this review.
COM-05 Develop and implement a process to monitor the status of planned and ongoing
Regional Entity compliance audits, through the point that a final non-public audit
report is issued. Include requirements for Regional Entities to periodically report the
status of their compliance audits to NERC and to certify that audits were scheduled
according to the requirements of the ROP.
COM-06 Develop a formal process for implementing changes to reliability standards and new
NERC directives within the Reliability Standard Audit Worksheets (RSAWs). Include
timelines within this process that will be monitored to ensure that Regional Entities
are using the most up-to-date information to perform their compliance audits.
4.5. Overseeing Enforcement Activities of Regional Entities
When Regional Entities identify possible violations of reliability standards through their eight
compliance monitoring methods, they must notify NERC of this and keep NERC informed of the
progress of the enforcement and mitigation of the violation. NERC in turn is delegated various
oversight responsibilities during this process. For example, NERC is responsible for reviewing
and approving mitigation plans, enforcement actions and settlement agreements from Regional
Entities. The Manager of Enforcement and Mitigation at NERC and his staff have primary
responsibility for carrying out these functions.
Enforcement Activities of Regional Entities:

o “Remedial Action Process” (NPP-CME-500)
o “Compliance Violation and Penalty Process - RE CEA” (NPP-CME-501)
o “Settlement Process - RE CEA” (NPP-CME-502)
o “Mitigation Process - RE CEA” (NPP-CME-503)
o “Appeals and Hearing Process” (NPP-CME-505)
o “Penalty Guidance Process” (NPP-CME-506)
Overseeing Enforcement Activities of Regional Entities – Functional Area Criteria and

Observations

1 Is the process objective documented for all Not met. Objectives were documented
processes within the functional area and are all explaining NERC’s role in the related process;
tied to underlying rules and policies? the objectives did not describe the ultimate
If Yes and the crosswalk to the ROP is goal or objective of the process itself. See
not apparent, ask which ones Recommendation ENF-01. In addition, no
draft had been developed for NERC’s role in
overseeing the application of penalties and
sanctions by the Regional Entities, although
the ROP noted this as a key responsibility:
“NERC shall oversee the regional entities’
application of the guidelines to ensure that
acceptable levels of consistency are
achieved.” See Overarching
2A For this functional area, how well do the Partially met. As noted in the response to
processes currently documented within the Criterion 1 above, NERC’s role in overseeing
Processes and Procedures Manual map to or the application of penalties and sanctions by
match the Rules of Procedure? Regional Entities, a NERC responsibility under
the ROP, was not captured in a process
document; instead reliance is made directly
upon the language in the ROP and the
relevant FERC Orders such as those issued
during the course of Commission’s approval
of Appendix 4C (“Uniform Compliance
Monitoring and Enforcement Program”) of
the ROP. The linkage and relationship
between the internal process documentation
and the ROP can be better documented in
the internal process documentation.
However, we noted no other differences
between ROP requirements and process
documentation in this functional area.
2B For this functional area, how well do the Partially met. NERC Compliance has not
processes currently executed map to or match always completed its review of mitigation
the Rules of Procedure? Are there areas plans within 30 days as required by the ROP.
where there are known exceptions or As of May 8, 2009, NERC Compliance had
discrepancies? completed 252 of 261 mitigation plan reviews
within 30 days. We found no other areas in
which the processes as currently executed
did not match the ROP policies.

2C For this functional area, how well do the Partially met. We observed that although
processes currently executed map to or matchNERC Compliance performed substantive
the Processes and Procedures Manual? Are reviews of Regional Entities’ applications of
there areas where there are known exceptionspenalties and sanctions, this was not
or discrepancies? documented in any process document;
instead reliance is made directly upon the
language in the ROP and the relevant FERC
Orders such as those issued during the course
of Commission’s approval of Appendix 4C
(“Uniform Compliance Monitoring and
Enforcement Program”) of the ROP. The
linkage and relationship between the internal
process documentation and the ROP can be
better documented in the internal process
documentation. Further, there was an item
of conflict between the documents
“Compliance Violation and Penalty Process”
and “Review and Approval of Settlements.”
Both documents addressed the review of
settlement agreements, but the former
document made no mention of the BOTCC’s
role, while the latter document stated that
the BOTCC had the sole discretion in
approving settlements. These steps should
be consolidated into one document which
accurately and fully reflects the settlement
process. See Recommendation ENF-02.
3 How do people know their roles and Partially met. Most of the process
responsibilities in executing the processes documents in this functional area specified
within this functional area? Are roles and who was responsible for carrying out
responsibilities documented? individual tasks. However, specific roles and
responsibilities were not always outlined in
the process documents, particularly within
the “Compliance Violation and Penalty
Process - RE CEA” (NPP-CME-501). In some
instances, responsibilities for carrying out
tasks were documented as NERC’s, but did
not specify who at NERC is responsible for
them. See Overarching Recommendation
PPM-04.

3A Do resources responsible for executing the Partially met. We observed that the NERC
process have awareness and understanding of Compliance team responsible for
the process (as documented), and capability to enforcement and mitigation oversight had
execute the process (e.g. they are trained, the capabilities and knowledge to do the job
have the right skill sets, have the right tools at well. However, the team lacked an
their disposal, etc.)? automated system for cohesively tracking
mitigation plans, alleged violations,
settlements, and confirmed violations.
Tracking was all done offline and was
complicated, because mitigation plans,
alleged violations, settlements, and
confirmed violations did not necessarily
corresponded one-to-one with each other.
For example, a settlement could have been
made for multiple violations, and those
violations could have had different mitigation
plans for each violation or for multiple
violations within the settlement.
Implementation of the new Compliance
Reporting and Tracking System (CRATS)
should be able to capture this information,
but at the time of our observations, the
process required substantial staff resources.

4 Do you and your team have the resources Not met. We observed that resources in this
(personnel, systems, budget, etc.) necessary to functional area were seriously constrained, as
accomplish the objectives of the processes evidenced in the Goal 1 Update report, which
within the functional area? If not, what is showed that most goals in enforcement and
lacking? mitigation oversight were in “red” or
“yellow” status. We observed several
reasons for this. First, as noted in the
response to Criterion 3A above, NERC’s lack
of an automated violation tracking system
required significant staff time and effort. In
addition, staff faced a considerable volume of
work, all of which required careful review.
This was further complicated because NERC
Compliance often identified significant
quality issues while reviewing enforcement
and mitigation documents from Regional
Entities, which it had to work with Regional
Entities to resolve. See overarching
Recommendations ENT-01, ENT-02, and ENT-
03. The Manager of Enforcement and
Mitigation told us the Enforcement and
Mitigation team was planning to add four
new staff members, to help manage the
workload. See Overarching
03.
5 Are the processes within this functional area Met. We observed that Enforcement and
monitored to see that they are consistently Mitigation processes were tracked in
performed in accordance with the significant detail by the NERC Compliance
documented process? If so, how? Reporting, Analysis, and Tracking team.
5A If the processes within this functional Met. We observed that each stage in the
area are monitored (i.e. the answer to Enforcement and Mitigation process was
#5 was “yes”), are they also tracked, and the number of violations in each
measured? If so, what measurements stage was reported regularly to FERC and the
are taken (and how) and what is done BOTCC.
etc.)?
5B If the processes within this functional Partially met. We observed that the team
area are monitored (i.e. the answer to was in the process of hiring new staff and
#5 was “yes”), do you do anything with was working closely with Regional Entities to
the cases where the actual execution improve the quality of data they were
deviates from the norm – that is, the sending to NERC. Both of these actions
so-called process exceptions? would improve the process overall. However,
we observed that no actions have been taken
to specifically target the individual process
exceptions.

6 Do the processes within this functional area Partially met. We observed that the
have specific, measurable goals or objectives? processes had certain specific goals, which
If so, what are those? were recorded in the monthly Goal 1 Update
report, but not all of these goals were in the
process documents themselves. For
example, NERC Compliance reported that
Notices of Confirmed Violations were to be
reviewed and either remanded to the Region
or draft Notices of Penalty prepared within
30 days of receipt. However, this goal was
not outlined in any process documentation.
In general, we observed that the processes
lacked specific timelines for completing
significant process milestones. Further,
several key processes were not adequately
addressed in terms of specific criteria to be
used or steps to be followed. For example,
the “Mitigation Process” did not mention or
include the Mitigation Plan Approval Criteria,
which NERC has developed to aid in
mitigation plan review. In addition, the
process document “Review and Approval of
Settlements” stated that NERC must
“appropriately monitor” settlement
proceedings, but it did not provide any
guidance as to how appropriate monitoring
should be carried out. See Overarching
6A If the processes within this functional area Partially met. We observed that different
do have specific, measurable goals (that goals had different frequencies with regard
is, the answer to #6 was yes), how to meeting them. For example, as of May 8,
frequently do the processes meet their 2009, only 66.7% of Notices of Confirmed
goals or objectives? Violations were reviewed by NERC within the
stated 30 day goal. However, 100% of
mitigation plans were submitted to FERC
within the goal of 7 days after NERC approval.
6A.1 If the answer to #6A is anything We observed that the primary barriers to
other than 100% of the time, then a achieving process goals and objectives are a
follow-on question: What are the high volume of work requiring careful
primary barriers for the processes attention to detail; a lack of an automated
to achieve their goals or objectives? system for tracking enforcement actions;
and, recurring issues with quality of data
submitted by Regional Entities. These are
discussed in greater detail under the
evaluation of Criterion 4.

7 Do you consider the processes within this Not met. We observed that the process was
functional area to be efficient to the extent not efficient, because it lacked an automated
that unnecessary steps, iterations, resources, system for tracking mitigation actions.
and delays have been eliminated? If not, what Tracking was all being done offline and was
changes can be made? complicated because mitigation plans,
alleged violations, settlements, and
confirmed violations did not correspond one-
to-one. Implementation of the Compliance
Reporting and Tracking System (CRATS) will
be able to capture this information, but at
the time of our observations, the process for
doing required substantial staff resources.
8 How is process documentation made available Met. Enforcement and Mitigation personnel
to personnel? contributed to the development of the
process documents, and they were able to
access them from the Compliance Process
Administrator.
9 How is process documentation controlled for Met. The Compliance Process Administrator
changes? Do you have a process for formally had control of the official process
responding to and dealing with changes that documentation and was the only person with
would come down from FERC (for example)? edit rights.
In addition, we made the following observations concerning Oversight of Regional Entity

Enforcement and Mitigation:
We observed that the process documents that had been developed for this functional area
were in draft form and had not been reviewed by management. See Overarching
Although the ROP required that NERC be provided with copies of Notices of Alleged
Violation and Penalty and Sanction (NAVAPS), the ROP did not require that NERC review the
NAVAPS. The ROP also did not give NERC sign-off authority at this stage. We further
observed that the ROP did not require NERC review of alleged violations and penalties until
the violation has gone through the Regional Entity’s full due process and the violation has
either been confirmed or settled on. We observed that this was often months after a
violation was first identified and came after considerable negotiations and information
exchanges had already occurred between the Regional Entity and the Registered Entity that
violated the standard. Often, once they received a Notice of Confirmed Violation (NOCV) or
settlement agreement, NERC would find significant issues with the documents that must be
resolved before they can approve them and file a Notice of Penalty with FERC. Many of the
same issues, such as inconsistent dates and improper categorization of a violation as a
documentation issue rather than a failure to perform, were repeating among the Regional
Entities. The process then involved a lot of back and forth between the Regional Entity to
resolve. Much of this could be prevented if the ROP allowed for NERC involvement earlier in
the process. See Recommendation ENF-03.
The document “Compliance Violation and Penalty Process” (NPP-CME-501) included

requirements of NERC’s Compliance Analysis, Reporting, and Tracking team. However, the
document was developed independently by members of the Enforcement and Mitigation
team. Although the document was developed to reflect what the two teams actually do,
the Enforcement and Mitigation team should seek assurance that their process documents,
to which the Compliance Analysis, Reporting, and Tracking team could be held accountable,
accurately reflect the roles and obligations of both teams. See Recommendation ENF-04.
Recommendations
ENF-01 Revise the process documents related to NERC’s oversight of Regional Entities’
enforcement programs to specify the objectives of the processes themselves.
ENF-02 Consolidate the review and approval of settlements process noted in the documents
“Compliance Violation and Penalty Process” (NPP-CME-501.R0) and “Review and
Approval of Settlements” (NPP-CME-502.R0) into one document. Ensure that the
document accurately and fully reflects the settlement review process, including the
role of the NERC BOTCC.
ENF-03 Revise the ROP to allow for NERC involvement in reviewing violations, penalties, and
sanctions prior to the Notice of Penalty stage.
ENF-04 Provide a copy of the “Compliance Violation and Penalty Process” (NPP-CME-501)
document to the Compliance Analysis, Reporting, and Tracking team for review, to
ensure that their role in the process is captured accurately and clearly.
4.6. Analyzing and Reporting Compliance Information
NERC Compliance has a responsibility to gather, analyze, and report compliance-related

information to the bodies that oversee its operations, such as FERC and the BOTCC. NERC
provides monthly reports to the BOTCC and quarterly reports to FERC, which are required under
the ROP. At NERC, the Manager of Compliance Analysis, Reporting, and Tracking has the
primary responsibility for these functions, although it is currently the responsibility of the
Manager of Enforcement and Mitigation to send reports to FERC, and reporting to the BOTCC is
a shared function between the two groups. Quarterly violation and mitigation plan status
reports are developed and sent to FERC independently by the Compliance Analysis, Reporting
and Tracking group. BOTCC has three types of meetings: quarterly open, monthly closed, and
monthly closed-closed. The Compliance Analysis, Reporting and Tracking Group primarily
handles the open and closed meetings, while the Enforcement and Mitigation Group handles
the closed-closed meetings.
As noted in Appendix I, the following processes make up the functional area for Analyzing and
Reporting Compliance Information:
o “Compliance Data Reporting Process” (NPP-CME-701)

o “Data Management, Evaluation, and Analysis Process” (NPP-CME-700)
Analyzing and Reporting Compliance Information – Functional Area Criteria and Observations

1 Is the process objective documented for all Met. Process objectives were documented
processes within the functional area and are all and tied to the ROP.
tied to underlying rules and policies?
2A For this functional area, how well do the Met. The processes as documented mapped
processes currently documented within the to the ROP. However, we noted that not all
Processes and Procedures Manual map to or required reports to FERC were included in
match the Rules of Procedure? these particular process documents. Most of
them, such as reporting updates to the
compliance registry and reporting new
alleged violations, were embedded in other
NERC processes, as they were the
responsibility of other teams within NERC. In
addition, NERC Compliance maintained a
separate list of all required reports to FERC.
2B For this functional area, how well do the Partially met. Although NERC Compliance
processes currently executed map to or match noted no instances in which the processes
the Rules of Procedure? Are there areas were not followed as documented, we
where there are known exceptions or observed that there were no monitoring
discrepancies? mechanisms in place that covered NERC’s
adherence to processes related to Analyzing
and Reporting Compliance Information. As a
result, we were not able to determine how
well the processes as executed matched the
ROP. See Overarching Recommendations
PPM-06 and MON-01.
2C For this functional area, how well do the Partially met. Although NERC Compliance
processes currently executed map to or match noted no instances in which the processes
the Processes and Procedures Manual? Are were not followed as documented, we
there areas where there are known exceptions observed that there were no monitoring
or discrepancies? mechanisms in place that cover NERC’s
and Reporting Compliance Information. See
MON-01.

3 How do people know their roles and Not met. We observed that personnel
responsibilities in executing the processes primarily understand their functions through
within this functional area? Are roles and information shared by word of mouth and, at
responsibilities documented? a high level, by their position descriptions
which generally indicate responsibilities for
whole processes (versus activities or steps
within processes). However, specific roles
and responsibilities were not documented
within the process documents. For example,
although the document “Data Management
Evaluation and Analysis Process” lists several
individuals that have a role in the process, all
responsibility for actual execution is assigned
to the Manager of Compliance Reporting,
Analysis, and Tracking. No specific roles were
documented for the staff members. (See
Overarching Recommendation PPM-04.)
Even if roles and responsibilities had been
identified within the process documents, the
documents had not been shared with
Compliance Reporting, Analysis, and Tracking
staff at the time of our information gathering.
See Recommendation REP-01.
3A Do resources responsible for executing the Partially met. We observed that several of
process have awareness and understanding of the Compliance Reporting, Analysis, and
the process (as documented), and capability to Tracking personnel were learning the
execute the process (e.g. they are trained, processes. However, they have been trained
have the right skill sets, have the right tools at on their roles and have been provided with
their disposal, etc.)? copies of the ROP and the CMEP Processes
and Procedures Manual (as well as applicable
FERC orders, etc.) to read. As noted in the
response to Criterion 3 above, however,
copies of the process documents have not
been provided to staff.
4 Do you and your team have the resources Partially met. New staff had been hired to
(personnel, systems, budget, etc.) necessary to handle the analysis and reporting tasks,
accomplish the objectives of the processes freeing up additional time for the team to
within the functional area? If not, what is focus on new analyses that will be helpful to
lacking? NERC and other parties. However, the new
Compliance Reporting and Tracking System
(CRATS), which will be largely designed to
track violations, had not yet been
implemented. At the time of our
observations, the processes related to this
functional area involved a great deal of
manual tracking and analysis, primarily
through Excel spreadsheets.

5 Are the processes within this functional area Not met. Although the processes for
monitored to see that they are consistently Analyzing and Reporting Compliance
performed in accordance with the Information were themselves a function for
documented process? If so, how? monitoring other team’s progress, there was
no internal department monitoring of
milestones related to this functional area.
For example, we observed that there was no
tracking of the development of reports to
assure that they are provided on time. See
Recommendation REP-02.
5A If the processes within this functional Not applicable. Processes were not
area are monitored (i.e. the answer to monitored.
#5 was “yes”), are they also
measured? If so, what measurements
are taken (and how) and what is done
etc.)?
5B If the processes within this functional Not applicable. Processes were not
area are monitored (i.e. the answer to monitored.
#5 was “yes”), do you do anything with
the cases where the actual execution
deviates from the norm – that is, the
so-called process exceptions?
6 Do the processes within this functional area Partially met. The “Compliance Data
have specific, measurable goals or objectives? Reporting Process” (NPP-CME-701.R0)
If so, what are those? included timelines for reviewing reports and
providing them to FERC, the BOTCC, or other
entities as applicable. However, the “Data
Management, Evaluation, and Analysis
Process” (NPP-CME-700.R0) was not really a
process document, but rather a statement of
generic requirements of the reporting and
analysis processes. Therefore, it did not have
specific, measurable goals or objectives. See
Recommendation REP-03.
6A If the processes within this functional area Partially met. NERC Compliance noted no
do have specific, measurable goals (that instances in which the processes were not
is, the answer to #6 was yes), how followed as documented. However, we
frequently do the processes meet their observed that there were no monitoring
goals or objectives? mechanisms in place that covered NERC’s
and Reporting Compliance Information.
Therefore, we could not verify how
frequently goals and objectives were being
met. See Overarching Recommendations
PPM-06 and MON-01.

6A.1 If the answer to #6A is anything We observed that there were no monitoring
other than 100% of the time, then a mechanisms in place that covered NERC’s
follow-on question: What are the adherence to processes related to Analyzing
primary barriers for the processes and Reporting Compliance Information.
to achieve their goals or objectives? Therefore, we could not determine how
frequently goals and objectives are met.
7 Do you consider the processes within this Partially met. At the time of our
functional area to be efficient to the extent observations, much of the compliance
that unnecessary steps, iterations, resources, tracking and reporting was done through
and delays have been eliminated? If not, what Excel spreadsheets, which were a laborious
changes can be made? process. The new Compliance Reporting and
Tracking System (CRATS) will automate much
of this process, but it had not yet been
implemented as of our information
gathering.
8 How is process documentation made available Partially met. Process documentation had
to personnel? not been made available to all Compliance
Reporting, Analysis, and Tracking personnel.
Some staff had seen process documentation,
but not all staff. See Recommendation REP-
01.
9 How is process documentation controlled for Met. The Compliance Process Administrator
changes? Do you have a process for formally had control of the official process
responding to and dealing with changes that documentation and was responsible for
would come down from FERC (for example)? making any changes.
Recommendations
REP-01 Provide Compliance Reporting, Analysis, and Tracking staff with copies of the process
documents related to Analyzing and Reporting Compliance Information. Obtain input
from staff members on the processes as documented.
REP-02 Identify key milestones and deadlines within the processes related to Analyzing and
Reporting Compliance Information. Develop a mechanism for measuring and tracking
these key milestones and deadlines, and develop goals and acceptable ranges for
meeting them. Report results to NERC management. In addition, develop a plan for
addressing goals that are not met within the acceptable range.
REP-03 Revise the “Data Management, Evaluation, and Analysis Process” (NPP-CME-700.R0)
to be more process-oriented. Establish specific roles and responsibilities and key
milestones, such as deadlines, within the process to help track progress and ensure
that the process is followed.
4.7. Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs

Under the ROP and FERC Order 672, NERC is responsible for auditing Regional Entities’
compliance monitoring and enforcement programs every three years. This engagement could
potentially cover all functions that Regional Entities perform pursuant to compliance and
enforcement, as well as any other functions specified in their Delegation Agreements with
NERC. Because the BOTCC required that this audit be performed by an outside auditing firm,
NERC hired Crowe Horwath to perform the work at the Regional Entities. The method chosen
was an agreed-upon procedures engagement. This first agreed-upon procedures engagement is
currently ongoing, with procedures having been completed at three of the eight Regional
Entities. The Director of Regional Operations has responsibility for this process.
As noted in Appendix I, the following processes make up the functional area for Conducting
Reviews of Regional Entities Compliance and Enforcement Programs:
o “NERC Audit of Regional Entity Adherence to the CMEP” (NPP-CME-404)

o “RE Spot Check Process” (NPP-CME-403)
Conducting Reviews of Regional Entities’ Compliance and Enforcement Programs – Functional


1 Is the process objective documented for all Met. We observed that an objective that tied
processes within the functional area and are to the ROP was documented for the process
all tied to underlying rules and policies? “NERC Audit of Regional Entity Adherence to
the CMEP” (NPP-CME-404). The objective of
not apparent, ask which ones the “RE Spot Check Process” (NPP-CME-403),
was not specifically authorized by the ROP
(the spot check process stems from ROP 402,
1.), but NERC presented its plans to conduct
spot checks to FERC, and FERC agreed with
this approach.
2A For this functional area, how well do the Met. The draft processes that have been
processes currently documented within the completed matched the ROP or were
Processes and Procedures Manual map to or otherwise agreed to with FERC.
2B For this functional area, how well do the Met. Only the process “NERC Audit of
processes currently executed map to or match Regional Entity Adherence to the CMEP”
the Rules of Procedure? Are there areas (NPP-CME-404) was required by the ROP, and
where there are known exceptions or the primary ROP requirement was that an
discrepancies? audit be conducted every three years, which
we observed was being done. No
discrepancies were identified.

2C For this functional area, how well do the Partially met. We observed that most of the
processes currently executed map to or match process “NERC Audit of Regional Entity
the Processes and Procedures Manual? Are Adherence to the CMEP” (NPP-CME-404) had
there areas where there are known exceptions been executed as documented. However,
or discrepancies? there were some notable discrepancies.
First, the document did not specify that
NERC, not the external auditor, performs the
agreed-upon procedures in the area of
Information Technology. Further, the
process did not allow changes to the external
auditor’s team, although in certain cases this
has been necessary. In addition, the process
called for a formal team meeting fourteen
days prior to performing the agreed-upon
procedures to discuss the completeness of
information provided by the Regional Entity.
However, this was not practical, because the
information was typically not received
fourteen days before the procedures
commenced, and the sheer volume of
information made it very difficult to
determine if the information was complete
before actually performing the related
procedures. See Recommendation REV-01.
The “RE Spot Check Process” (NPP-CME-403)

has not yet been performed, and therefore
this criterion was not applicable to it.
3 How do people know their roles and Partially met. Roles and responsibilities were
responsibilities in executing the processes well defined within the process “NERC Audit
within this functional area? Are roles and of Regional Entity Adherence to the CMEP”
responsibilities documented? (NPP-CME-404). However, the “RE Spot
Check Process” (NPP-CME-403) did not note
specific roles and responsibilities, but only
listed “NERC” as the responsible party. See

3A Do resources responsible for executing the Partially met. The NERC Senior Regional
process have awareness and understanding of Entity Compliance Program Auditor who has
the process (as documented), and capability to been involved in the Regional Entity audit
execute the process (e.g. they are trained, processes from the onset had a very detailed
have the right skill sets, have the right tools at understanding of how the processes are to
their disposal, etc.)? be executed. However, given the newness
and unique nature of the process, which was
not like any process NERC had performed
before, it has been challenging to bring on
new staff in the middle of the process. With
the tight time frames for completing the
Regional Entity audits, NERC has relied on on-
the-job training to bring new staff on board.
This has been an excellent way to immerse
members of the Regional Audit team in the
process, but staff members have cited a lack
of understanding of the details of the
process, which made it challenging to start
their work with sufficient knowledge to fully
understand their duties. However, they
noted that reading the “NERC Audit of
Regional Entity Adherence to the CMEP”
(NPP-CME-404) process helped. As this
process matures, however, NERC Compliance
should consider a more formal training
process for staff. See Recommendation REV-
02.
4 Do you and your team have the resources Partially met. Although NERC Compliance
(personnel, systems, budget, etc.) necessary to was consistently accomplishing the measured
accomplish the objectives of the processes objectives of the Regional Entity audit
within the functional area? If not, what is process, NERC has had to push back its
lacking? planned completion dates a number of times.
The primary causes cited for this were a lack
of NERC Compliance personnel devoted to
the AUPs and that NERC Compliance changed
its original plans from performing multiple
Regional Entity audits at one time to only
performing one audit at a time. . At the time
we were completing the information
gathering for this report, NERC was working
with FERC to allow the agreed-upon
procedures to be completed by December
31, 2010. Further, NERC Compliance planned
to provide additional personnel from the
Organization and Registration team to help
with Regional Entity audits in 2010.

5 Are the processes within this functional area Partially met. For the “NERC Audit of
monitored to see that they are consistently Regional Entity Adherence to the CMEP”
performed in accordance with the process, certain major milestones were being
documented process? If so, how? monitored, such as the final draft report date
and the date the request for information is
sent to a Regional Entity. However, other
significant milestones were not being
tracked, such as obtaining audit team
members’ confidentiality, non-disclosure
agreements, conflict of interest and work
history statements; the holding of required
meetings; and, the interim deadlines for
drafting the report. See Overarching
area are monitored (i.e. the answer to being monitored were also measured, in
#5 was “yes”), are they also terms of the number of regional entity audits
measured? If so, what measurements completed versus the audit schedule, and
are taken (and how) and what is done whether the request for information and
with these measurements (reported, final reporting timelines were met. The
etc.)? measurements were reported in the monthly
Goal 1 Update report. As noted in the
response to Criterion 5 above, however, only
these certain portions of the functional area
were monitored.
5B If the processes within this functional Not applicable. For the process points that
area are monitored (i.e. the answer to were being monitored, no deviations from
#5 was “yes”), do you do anything the process had been identified.
with the cases where the actual
execution deviates from the norm –
that is, the so-called process
exceptions?
6 Do the processes within this functional area Met. The process “NERC Audit of Regional
have specific, measurable goals or objectives? Entity Adherence to the CMEP” had a
If so, what are those? number of specific goals throughout the
process, such as sending a notification letter
to the Regional Entity at least 60 days before
the audit; obtaining confidentiality, non-
disclosure agreements, conflict of interest
and work history statements from audit team
members within 45 days of the audit, and the
scheduling of daily status meetings during
the audit. The “RE Spot Check Process” also
included specific goals, such as notifying the
Regional Entity of the spot check at 30 days
prior and developing a spot check report.

6A If the processes within this functional Partially met. Those objectives that were
area do have specific, measurable goals monitored had been met 100% of the time.
(that is, the answer to #6 was yes), how However, as noted in the evaluation of
frequently do the processes meet their Criterion 5, not all process milestones were
goals or objectives? tracked. Therefore, we could not determine
the frequency at which the unmonitored
objectives were met.
6A.1 If the answer to #6A is anything Although all objectives that were monitored
other than 100% of the time, then a in this functional area had been met 100% of
follow-on question: What are the the time, we observed that in other
primary barriers for the processes functional areas under this team’s
to achieve their goals or objectives? responsibility, internally established
timelines were not being sufficiently met.
Staff members on this team were responsible
for a number of other functional areas.
Importantly, the Manager of Registration and
Certification had been working with the
Director of Regional Operations to provide
help with Regional Entity audits, which
should alleviate some of the difficulties the
Regional Operations team has had with
meeting deliverables in other areas.
7 Do you consider the processes within this Met. We did not identify inefficiency in this
functional area to be efficient to the extent functional area.
that unnecessary steps, iterations, resources,
and delays have been eliminated? If not, what
changes can be made?
8 How is process documentation made available Met. We observed that the process
to personnel? documents were maintained by the Process
Administrator on a secured share drive to
which NERC staff working in this functional
area had read-onlu access. Managers and
the Process Administrator can access
editable, draft process documents for
change.
9 How is process documentation controlled for Met. We observed that the Compliance
changes? Do you have a process for formally Process Administrator had control of the
responding to and dealing with changes that official process documentation and was
would come down from FERC (for example)? responsible for making any changes.
In addition, we noted the following issues in this functional area:

We observed that the process documents that had been developed for this functional were
in draft form and had not been reviewed by management. See Overarching
We observed that the document Attachment 1 of the “RE Spot Check Process,” called the
“Primary Elements of a Regional Entity Spot Check” includes procedures for reviewing
Registered Entities’ mitigation plans that were approved by Regional Entities. Because of
how these procedures were written, they appeared to overlap with the normal duties of the
Enforcement and Mitigation team, which NERC Compliance has charged with reviewing
mitigation plans for conformance to the CMEP, including determining whether the actions
specified in those plans would actually mitigate the alleged violation. However, the spot
check process was not designed to review the mitigation plans for adequacy, but rather it
was designed to examine whether the Registered Entity actually followed through on the
actions in its mitigation plan. To provide a more accurate guide to staff performing spot
checks, NERC Compliance should revise Attachment 1 of the “RE Spot Check Process.” See
Recommendation REV-03.
Recommendations
REV-01 Revise the document “NERC Audit of Regional Entity Adherence to the CMEP” (NPP-
CME-404) to accurately reflect how the process is performed in practice, including:
Specify that some of the agreed-upon procedures may be performed by NERC
Compliance staff as agreed to with the BOTCC and FERC.
Include a provision to allow for changes to the external auditor team, as long
as NERC and the Regional Entity approve the change and the auditors have
submitted the requisite non-disclosure agreements, work histories, and
conflict of interest statements prior to working on the engagement.
Include a provision for the external audit team to prepare management
letters to the Regional Entity and NERC, when items are identified that are
not exceptions to the agreed-upon procedures, but should be changed in
either entity’s compliance program.
Delete the requirement for a formal team meeting fourteen days prior to the
performing the agreed-upon procedures to discuss the completeness of
information provided by the Regional Entity. Instead, specify that NERC and
the external auditor will review the items requested pursuant to the sampling
procedures for completeness within a certain time after the information is
provided by the Regional Entity.
REV-02 Develop a training program on NERC’s process for auditing Regional Entities. Include
details on how the process is carried out, such as the definition of agreed-upon
procedures, and which agreed-upon procedures are typically performed at Regional
Entities. Also, include any best practices and lessons learned from the first round of
Regional Entity audits.
REV-03 Revise the procedures from Attachment 1 of the “RE Spot Check Process” that
requires the spot check team to “review Mitigation Plans for appropriate content as
required by the CMEP” to more accurately reflect how the Compliance Audit Group
reviews completed mitigation plans as part of a spot check.
4.8. NERC Involvement in Compliance Inquiries and Violation Investigations

CMEP 3.4 states that although compliance violation investigations (CVIs) will generally be led by
Regional Entities, NERC may assume the lead of a CVI at its discretion. NERC Compliance could
choose to lead a CVI for several reasons, such when a Regional Entity may have a potential
conflict of interest, or when the allegations under investigation affect multiple Regional Entities.
When a Regional Entity does lead a CVI, NERC Compliance does not actively participate in the
investigation, but does maintain contact with the Regional Entity and requires updates on the
status of the CVI. In addition, FERC may also lead compliance investigations. NERC Compliance
participates in these investigations, although the investigations follow FERC’s processes, not the
ROP or NERC’s internal process. Responsibility for leading CVIs, overseeing Regional Entity CVIs,
and participating on FERC CVIs is vested with the Manager of Compliance Violation
Investigations and his staff.
Before initiating a CVI, NERC or a Regional Entity may perform a Compliance Inquiry, which is a
smaller scale review to determine if enough evidence exists to warrant a full CVI.
As noted in Appendix I, the following processes make up the functional area for NERC
Involvement in Compliance Inquiries and Compliance Violation Investigations:
o “Compliance Inquiry Process” (NPP-CME-300)

o “Compliance Violation Investigation Process” (NPP-CME-302)
o “Evidence Handling Process” (NPP-CME-303)
NERC Involvement in Compliance Inquiries and Compliance Violation Investigations –

Functional Area Criteria and Observations

1 Is the process objective documented for all Partially met. We observed that the
processes within the functional area and are all objectives of all processes were documented,
tied to underlying rules and policies? and the objectives of the “Compliance
If Yes and the crosswalk to the ROP is Violation Investigation Process” (NPP-CME-
not apparent, ask which ones 302) and the “Evidence Handling Process”
(NPP-CME-303) tied to the ROP. However,
the at the time of our observations, the ROP
did not include any provision for conducting
Compliance Inquiries. Because NERC’s
process in this area was not only internal to
NERC but also held Regional Entities
accountable to following the process through
Compliance Process Bulletin #2009-CVI-002,
it is important that Compliance Inquiries be
incorporated into the ROP. See
Recommendation CVI-01.

2A For this functional area, how well do the Partially met. As noted in the response to
processes currently documented within the Criterion 1 above, the “Compliance Inquiry
Processes and Procedures Manual map to or Process” was not referenced in the ROP,
match the Rules of Procedure? although it included procedures that NERC
expected Regional Entities to follow. The
other processes as documented mapped very
well to the ROP.
2B For this functional area, how well do the Partially met. NERC identified only one
processes currently executed map to or match exception to the ROP in carrying out CVI-
the Rules of Procedure? Are there areas related processes. Specifically, a new
where there are known exceptions or investigator sent a document to a company
discrepancies? that should not have been sent under the
confidentiality and evidence handling rules.
In response, the Manager of Compliance
Violation Investigations met with this
individual, helped create a corrective action
plan, and assigned a more senior investigator
to serve as a mentor to this person.
2C For this functional area, how well do the Partially met. NERC identified only one
processes currently executed map to or match exception to the Processes and Procedures
the Processes and Procedures Manual? Are Manual in carrying out CVI-related processes.
there areas where there are known exceptions Specifically, a new investigator sent a
or discrepancies? document to a company that should not have
been sent under the confidentiality and
evidence handling rules. In response, the
Manager of Compliance Violation
Investigations met with this individual,
helped create a corrective action plan, and
assigned a more senior investigator to serve
as a mentor to this person.

3 How do people know their roles and Partially met. We observed that process
responsibilities in executing the processes documents tended to use broad language for
within this functional area? Are roles and defining roles and responsibilities, such as
responsibilities documented? “NERC” or “Regional Entities.” See
Regarding NERC’s role on a Regional Entity-
led CVI, the “Compliance Violation
Investigation Process” document (NPP-CME-
302) stated, “The assigned NERC Compliance
Investigator will maintain regular contact
with the Regional Entity CVI Team Leader
during the Regional Led CVI regarding status
and will request copies of all correspondence
for quality review,” but it did not go into
further detail of NERC’s role on the CVI. See
However, the CVI team has documented

roles and responsibilities in other ways, such
as a “Compliance Violation Investigator List of
Expected Duties” (February 2009), and
various checklists and templates that staff
must use to document and perform their
duties. We observed that these documents
were available to all CVI team members on a
restricted network drive.
3A Do resources responsible for executing the Met. Several CVI team members participated
process have awareness and understanding of in developing the process documents. In
the process (as documented), and capability to addition, team members underwent training
execute the process (e.g. they are trained, specific to CVI-related processes. We
have the right skill sets, have the right tools at observed that they also had access to various
their disposal, etc.)? checklists and templates used in
documenting and performing the processes.
Further, the team used a database to track
the status of all active CVIs and Compliance
Inquiries in which NERC staff were involved.
4 Do you and your team have the resources Not met. Although the team had been able
(personnel, systems, budget, etc.) necessary to to accomplish most of the objectives of
accomplish the objectives of the processes individual CVIs and Compliance Inquiries, the
within the functional area? If not, what is team cited concerns that they lack enough
lacking? personnel to adequately cover the
appropriate volume of work. (See
Overarching Recommendations STA-01, STA-
02, and STA-03.) They noted that NERC’s CVI
team consisted of 11 individuals, who were
leading 11 investigations, overseeing 11
Regional Entity-led investigations, and
actively participating as team members on 5

FERC-led investigations. In addition, they
stated that the team handled between 12 –
15 active Compliance Inquiries at a time. The
team had received an unexpected volume of
complaints which needed to be addressed
and often led to additional CVI work. CVI
staff worked significant overtime, and the
Investigations put a limit on how much work
the team could take on, so as not to hurt the
quality of work that was performed. As a
result, certain CVIs were not being
performed. At the time of our observations,
NERC expected to add two more CVI staff
persons, which would alleviate some of the
staffing level issues.
Regarding this area, NERC provided the

following as background:
“During Q2 and Q3, the Manager of

Compliance Violation Investigations
conducted benchmarking of staffing for
similar investigation groups at the Federal
Energy Regulatory Commission (FERC) and
the Nuclear Regulatory Commission (NRC). It
became apparent that staffing for the NERC
CVI group which is responsible for conducting
investigations across North America
(Approximately 1800 Registered Entities and
4500 functional registrations) is deficient
even with the additional allotment of two
FTE’s in 2010.
The following are some of the metrics

captured during the benchmarking and
reported in the CVI groups Q2/Q3 report:
• The FERC OER staff currently has a FTE

count of 30 Engineers (who actively
participate on all NERC CVIs). This is 19 FTE
above the NERC CVI group’s resources. The
FERC OE staff also is a resource in that they
typically provide two or three attorneys for
leadership of any FERC 1b investigations
being conducted.
• The NRC currently has a FTE count of 48 for


the purpose of conducting Investigations on
approximately 90 nuclear stations. This is 37
FTE above the NERC CVI group’s resources.”
Further, we identified issues with resources

related to the evidence handling process.
These are specified within the separate
confidential letter to management in the
third observation under Section 3.1.
.
5 Are the processes within this functional area Met. We observed that on a weekly basis,
monitored to see that they are consistently compliance investigators were required to
performed in accordance with the submit a status report on their activities to
documented process? If so, how? the Manager of Compliance Violation
Investigations. The status was then logged
and tracked in an internal database. The
Investigations stated that he also performed
a quality review of all key steps in the
processes. For example, he would review any
information requests before they were sent
to Registered Entities under investigation. He
also signed off on the results of all
Compliance Inquiries. Further, the team
performed peer reviews at the end of CVIs to
assess the evidence and conclusions of the
investigator before a report was issued. In
addition, the team performed peer reviews at
least every six months to assess the status of
all ongoing work.
5A If the processes within this functional Partially met. We observed that the
area are monitored (i.e. the answer to measuring of processes for this functional
#5 was “yes”), are they also area was limited. NERC measured two
measured? If so, what measurements benchmarks and recorded them in the
are taken (and how) and what is done monthly Goal 1 Update Report: (i) the
with these measurements (reported, amount of time for performing a CVI from
etc.)? initiation through close and (ii) the
notifications of CVI initiations within two
business days; and in quarterly reports off of
the database. However, no other
benchmarks had been fully developed. See
Overarching Recommendations PPM-06,
MON-01, and MON-02.

5B If the processes within this functional Met. NERC adequately addressed the one
area are monitored (i.e. the answer to process deviation that was noted.
#5 was “yes”), do you do anything with Specifically, a new investigator sent a
the cases where the actual execution document to a company that should not have
deviates from the norm – that is, the been sent under the confidentiality and
so-called process exceptions? evidence handling rules. In response, the
Investigations met with this individual,
helped create a corrective action plan to
remand the document, and assigned a more
senior investigator to serve as a mentor to
this person. To address the underlying
control issue that led to the process
discrepancy, we recommend that NERC
develop a specific practice guide to aid in the
understanding of confidentiality rules. See
Recommendation CON-03.
6 Do the processes within this functional area Partially met. As noted in the response to
have specific, measurable goals or objectives? Criterion 5A above, NERC’s creation of
If so, what are those? benchmarked goals had been limited. The
goals were primarily those required by the
ROP (such as the required timeline for
notifying an entity of a CVI) and the length of
time for completing a CVI or Compliance
Inquiry in total. Some additional
benchmarks in stages in the middle of the
processes are taken through use of a project
management template tool. See
Overarching Recommendations PPM-06,
MON-01, and MON-02.
6A If the processes within this functional area Partially met. The Goal 1 Update Report at
do have specific, measurable goals (that May 8, 2009, showed no exceptions to the
is, the answer to #6 was yes), how requirements that applicable governmental
frequently do the processes meet their authorities and the Registered Entities under
goals or objectives? investigation should be notified of a CVI
within two business days of initiation.
However, the Goal 1 Update Report noted
that one of two CVIs performed entirely by
NERC (the only CVIs measured) was not
completed within the minimum time frame.
In addition, as noted in the response to
Criterion 5A, some Compliance Inquiry
benchmarks were formally tracked.

6A.1 If the answer to #6A is anything The primary barrier to achieving process
other than 100% of the time, then a goals was cited as a lack of adequate
follow-on question: What are the personnel resources, discussed in the
primary barriers for the processes evaluation of Criterion 4. In addition, the CVI
to achieve their goals or objectives? team told us that FERC required significant
amounts of contact and information for
NERC-led CVIs, and NERC spent a great deal
of time responding to FERC reviews and
interactive communications regarding
violations They noted that FERC reviewed all
the steps within an investigation to the
extent that it had a parallel team reviewing
data on investigations. See Overarching
Recommendation ENT-04. In addition, NERC
had been involved with large cross-border
investigations, where FERC and the Canadian
governments were prohibited by agreement
from sharing information on one another’s
jurisdictions. As a result, for example, NERC
had to conduct separate meetings with FERC
to discuss U.S. entities and with Canadian
provincial governments to discuss Canadian
entities. NERC estimated that this slowed
one CVI by three months.
7 Do you consider the processes within this Not met. As noted above, a significant level
functional area to be efficient to the extent of FERC involvement on CVIs and the
that unnecessary steps, iterations, resources, inefficiency of conducting cross-border
and delays have been eliminated? If not, what investigations have led to unnecessary delays
changes can be made? in conducting CVIs.
8 How is process documentation made available Met. We observed that CVI team personnel
to personnel? had access to the process documentation on
a secure web portal and on a limited-access
network drive.
9 How is process documentation controlled for Met. The CVI team reviewed the process
changes? Do you have a process for formally documents about every six months, and the
responding to and dealing with changes that Manager of Compliance Violation
would come down from FERC (for example)? Investigations assigned a team member to
update the documentation. We obtained a
copy of a document maintenance schedule
used to track this.
In addition to the observations noted in the evaluations above, we observed the following:
Except for the document “Evidence Handling,” the process documents that had been
developed for this functional area were in draft form and had not been reviewed by
management. See Overarching Recommendation PPM-02.
NERC noted significant concerns with the quality of CVIs performed by Regional Entities,
stating that they have found a majority of Regional Entity-led CVI to be deficient in some
way. NERC noted that the Regional Entities do have in place regional level processes
and methodology for conducting CVIs, but the Regional Entities are not consistent in
their method for performing CVIs. It is NERC’s observation that the Regional Entities
often assign investigations lower priority than other items such as compliance audits
and spot checks. They also noted that the Regional Entities’ staff lack experience in
investigation methodology and project management and they do not have the legal
resources to support CVIs. NERC provided CVI training to the Regional Entities, but the
Manager of Compliance Violation Investigations has noted that the Regional Entities are
so far from having a solid CVI program that he has recommended that all CVIs be
centralized under NERC. See Overarching Recommendations ENT-01, ENT-02, and ENT-
03.
In the “Compliance Violation Investigation Process” document (NPP-CME-302), the steps

in the CVI process methodology were called “suggested guidelines.” Similarly, the steps
in the Compliance Inquiry methodology were labeled “guidelines” in the “Compliance
Inquiry Process” (NPP-CME-300). This language could confuse NERC and Regional Entity
staff and cause them to believe the steps are not required to be followed. See
Recommendations
CVI-01 Add a section on Compliance Inquiries to the ROP / CMEP. Ensure that the
Compliance Inquiry Rules of Procedure align with the approved “Compliance Inquiry
Process” document (NPP-CME-300).
CVI-02 Add language to the “Compliance Violation Investigation Process” document (NPP-
CME-302) to more specifically describe a new role for a NERC single point of contact
when a Regional Entity-led CVI is in progress. Specify what the NERC staff’s duties are
(such as obtaining weekly updates from the Regional Entity, and reviewing all
Regional Entity correspondence) and what NERC is not required to do (attend site
visits, e.g.). Also, specify what obligations the Regional Entities have for providing
documentation and other materials to the NERC single point of contact.
CVI-03 Delete language in the “Compliance Violation Investigation Process” document (NPP-
CME-302) and the “Compliance Inquiry Process” document (NPP-CME-300) which
refers to the processes as “suggested guidelines” or “guidelines.” Replace this
language with “process requirements” or similar phrasing.
4.9. Handling Complaints
NERC maintains a telephone hotline and a website for receiving complaints from the public
related to the reliability of the bulk power system. Regional Entities may also forward
complaints that they have received to NERC for processing. The telephone hotline (and
corresponding voice mail box) is monitored at least daily by an Administrative staff member,
and the website is monitored at least daily for complaints by the Manager of Compliance
Violation Investigations or his designee. NERC, at its discretion, will forward a complaint to the
appropriate Regional Entity for review, unless the complaint is related to a Regional Entity or its
affiliates, the Regional Entity determines it cannot conduct the review, or the complainant
requests anonymity or specifically requests NERC to conduct the review of the complaint. In
such cases, NERC’s Manager of Compliance Violation Investigations is responsible for ensuring
that the complaint is reviewed by NERC.
As noted in Appendix I, the following process makes up the functional area for Handling
Complaints: “Complaint Process” (NPP-CME-301)
Handling Complaints – Functional Area Criteria and Observations

1 Is the process objective documented for all Met. The process objective was documented
processes within the functional area and are in the “Purpose” and “Scope” sections of the
all tied to underlying rules and policies? process document, and tied to the ROP.
2A For this functional area, how well do the Met. The complaints process matched the
processes currently documented within the ROP requirements.
Processes and Procedures Manual map to or
2B For this functional area, how well do the Partially met. NERC Compliance did not
processes currently executed map to or match identify any violations of the ROP related to
the Rules of Procedure? Are there areas the complaints process. However, we
where there are known exceptions or observed that NERC Compliance did not have
discrepancies? formal mechanisms in place for monitoring
the process, so there was not a way to know
whether exceptions to the ROP occurred.
See Overarching Recommendations PPM-06
and MON-01.
2C For this functional area, how well do the Partially met. NERC Compliance did not
processes currently executed map to or match identify any exceptions to the complaints
the Processes and Procedures Manual? Are process. However, we observed that NERC
there areas where there are known exceptions Compliance did not have formal mechanisms
or discrepancies? in place for monitoring the process, so there
was not a way to know whether deviations
from the documented process occurred. See
MON-01.

3 How do people know their roles and Partially met. Roles and responsibilities were
responsibilities in executing the processes outlined within the complaints process
within this functional area? Are roles and documentation. However, certain specific
responsibilities documented? functions to be performed were not
adequately addressed in the process
documentation. For example, the process
document specified that complaints made via
the website were to be logged, but it did not
specify that this was to be done for
telephone complaints, although the Manager
of Compliance Violation Investigations told us
the Administrative Assistant maintained a log
of telephone complaints. Further, the
document stated that the identity of
anonymous complainants was to be
protected, but it did not identify how
anonymous complaints were to be flagged as
such. It also did not specify what procedures
must be followed to protect an anonymous
complainant’s identity, nor did it specify that
the identity is not to be shared with any
entity or individual (even FERC). See
3A Do resources responsible for executing the Met. Individuals involved in the process have
process have awareness and understanding of been trained in their duties and were aware
the process (as documented), and capability to of the process. For example, monitoring
execute the process (e.g. they are trained, complaint mailboxes (voice mail, web, email)
have the right skill sets, have the right tools at was listed as a duty of senior compliance
their disposal, etc.)? violation investigators in the document
“Compliance Investigator Checklist of
Expected Duties.”
4 Do you and your team have the resources Partially met. The CVI team, which has been
(personnel, systems, budget, etc.) necessary to responsible for handling complaints, stated
accomplish the objectives of the processes that they were experiencing significant staff
within the functional area? If not, what is shortages (that is, staff shortages significant
lacking? enough to impact the team’s ability to meet
all of its responsibilities). However, because
the complaints handling process was much
less time consuming than other processes
the team performs, this has not been a major
hindrance to the appropriate handling of
complaints. See Overarching
03.

5 Are the processes within this functional area Partially met. We observed that some
monitored to see that they are consistently monitoring of the process occurred when the
performed in accordance with the Manager of Compliance Violation
documented process? If so, how? Investigations was alerted to new complaints
and when he reviewed complaints to
determine if they have merit. The on-duty
Senior Investigators monitor this process (as
described in the list of expected duties) in
that they check the mailbox on a daily basis
and open a complaint tracking number in the
CVI database for any complaints that come
through. The Admin monitors the phone,
logs any calls, and sends information to the
manager of CVI’s. A tracking number is
issued in the CVI database. As such, some
formal monitoring of the complaint process
was taking place. See Overarching
5A If the processes within this functional Not met. We observed that complaints were
area are monitored (i.e. the answer to reviewed by the Manager of Compliance
#5 was “yes”), are they also Violation Investigations for merit, but no
measured? If so, what measurements measurements were made of the complaints
are taken (and how) and what is done handling process. For example, there was no
with these measurements (reported, maximum time limit for making a
etc.)? determination on a complaint. See
MON-01.
5B If the processes within this functional Not applicable. NERC Compliance did not
area are monitored (i.e. the answer to note any process exceptions related to
#5 was “yes”), do you do anything handling complaints. However, as there
with the cases where the actual were no formally structured monitoring
execution deviates from the norm – procedures, this could not be determined
that is, the so-called process with certainty.
exceptions?
6 Do the processes within this functional area Partially met. We observed that the process
have specific, measurable goals or objectives? had certain specific goals. For example,
If so, what are those? complaints hotlines were to be checked
every day, and all complaints were to be
logged. However, some goals were lacking.
For example, there was no maximum time
limit for making a determination on a
complaint.
6A If the processes within this functional Partially met. NERC Compliance did not note
area do have specific, measurable goals any process objectives or goals that were not
(that is, the answer to #6 was yes), how met. However, as there were no formally
frequently do the processes meet their structured monitoring procedures, this could
goals or objectives? not be determined with certainty.

6A.1 If the answer to #6A is anything Not applicable. No unachieved process goals
other than 100% of the time, then a were identified.
follow-on question: What are the
primary barriers for the processes
to achieve their goals or objectives?
7 Do you consider the processes within this Met. No inefficiency was identified in the
functional area to be efficient to the extent complaints handling process.
8 How is process documentation made available Met. We observed that CVI team personnel
to personnel? had access to the process documentation on
a secure web portal and on a limited-access
network drive.
9 How is process documentation controlled for Met. The CVI team reviewed the process
changes? Do you have a process for formally documents about every six months, and the
responding to and dealing with changes that Manager of Compliance Violation
would come down from FERC (for example)? Investigations assigned a team member to
update the documentation. We obtained a
copy of a document maintenance schedule
used to track this.
In addition, we made the following observation about the complaint handling process:
The “Complaint Process” document (NPP-CME-301.R0) has been reviewed and

approved as final by the Vice President and Director of Compliance.
Recommendations
We have no recommendations specific to Handling Complaints. However, there are several

Overarching Recommendations as noted in the Criteria and Evaluation chart.
4.10. Executing Compliance Enforcement Authority Responsibilities

In most circumstances, Regional Entities are responsible for directly monitoring and enforcing
Registered Entities’ compliance with reliability standards. However, three of the eight Regional
Entities also function as Registered Entities in some capacity. These three Regional Entities used
to perform monitoring over their Registered Entity functions, in some cases essentially auditing
themselves. In 2008, FERC audited one of the Regional Entities and objected to this type of
relationship. Since that point, responsibility for monitoring the registered functions of these
Regional Entities began to be transitioned to NERC, who would perform monitoring and
enforcement as the Compliance Enforcement Authority (CEA). Management of NERC’s CEA
responsibilities has been delegated to the Manager of Organization Registration and

Certification.
At the time of our information gathering, NERC had not fully assumed its CEA responsibilities. At
that time, only one of the three registered Regional Entities had signed an agreement for NERC
to perform the applicable CEA functions. For the other two Regional Entities, no agreement had
been signed, so it was not yet clear which of the monitoring methods NERC would be obligated
to perform at those entities. However, NERC was moving forward with the CEA function, and
had begun processing some self-reported violations, had sent requests for self-certifications,
and was planning to perform its first reliability standard compliance audit of a registered
function of a Regional Entity in October 2009.
As noted in Appendix I, the following processes make up the functional area for NERC
Compliance Enforcement Authority Responsibilities:
o “Remedial Action Process” (NPP-CME-500)

o “Mitigation Process - NERC CEA” (NPP-CME-504)
o “Registered Entity Audit Process Procedure” (NPP-CME-602)
o “Self-Report Procedure” (NPP-CME-603)
o “Spot Check Procedure” (NPP-CME-604)
o “Mitigation Plan Procedure” (NPP-CME-605)
o “Self-Certification Procedure” (NPP-CME-606)
o “Data Reporting and Disclosure Procedure” (NPP-CME-607)
o “Exception Reporting Procedure” (NPP-CME-608)
o “Periodic Data Submittal Procedure” (NPP-CME-609)
o “Implementation and Tracking Procedure” (NPP-CME-610)
o “Remedial Action Directive Procedure – CEA” (NPP-CME-611)
NERC Compliance Enforcement Authority Responsibilities – Functional Area Criteria and

Observations


1 Is the process objective documented for all Partially met. We observed that all the
processes within the functional area and are processes that have been drafted had a
all tied to underlying rules and policies? documented objective that tied to the ROP.
If Yes and the crosswalk to the ROP is However, certain key processes that would
not apparent, ask which ones be required by the ROP had not been
documented, specifically those that related
to NERC’s enforcement duties when NERC
was to be the CEA. Although the Manager of
Organization Registration and Certification
and the Manager of Enforcement and
Mitigation agreed, in that they understood
that Registration and Certification would
develop the preliminary notice of alleged
violation, and would then turn the process
over to Enforcement and Mitigation.
However, these roles and responsibilities had
not been fully documented. For example,
there was no written process for the
development of a preliminary notice of
alleged violation. See Recommendation
CEA-01.
2A For this functional area, how well do the Partially met. The processes that had been
processes currently documented within the documented map to the Rules of Procedure,
Processes and Procedures Manual map to or but as noted in the response to Criterion 1
match the Rules of Procedure? above, key enforcement processes required
by the ROP had not yet been developed.
2B For this functional area, how well do the Not applicable. Because these processes
processes currently executed map to or match were in a very early implementation phase at
the Rules of Procedure? Are there areas the time of our observations, we could not
where there are known exceptions or make this determination.
discrepancies?
2C For this functional area, how well do the Not applicable. Because these processes
processes currently executed map to or match were in a very early implementation phase at
the Processes and Procedures Manual? Are the time of our observations, we could not
there areas where there are known exceptions make this determination.
or discrepancies?

3 How do people know their roles and Partially met. We observed that roles and
responsibilities in executing the processes responsibilities were documented within the
within this functional area? Are roles and processes. However, as noted in the
responsibilities documented? response to Criterion 1 above, roles and
responsibilities for enforcement of violations
when NERC was to act as the CEA, although
generally understood, had not been
documented. In addition, because the
process documents were in early draft form,
certain errors occurred in the documentation
of certain process roles. For example, in the
“Registered Entity Audit Process Procedure”
(NPP-CME-602), step 4.3.a stated: “The
MORC will forward the Public Audit Report to
the MORC for final processing.” In addition,
in the “Registered Entity Audit Process
Procedure” (NPP-CME-602), step 4.3.j stated:
“The MORC will endorse the report as being
the approved Public Audit Report and will
forward the approved report to the MORC.”
See Recommendation CEA-02.
3A Do resources responsible for executing the Met. At the time of our information
process have awareness and understanding of gathering, only one staff member, the one
the process (as documented), and capability to who developed the process documents, was
execute the process (e.g. they are trained, responsible for any execution of the
have the right skill sets, have the right tools at processes. However, as additional staff
their disposal, etc.)? persons come on board, they will have to
attend NERC compliance audit training
courses in order to participate in these
process activities.

4 Do you and your team have the resources Partially met. We observed that there was
(personnel, systems, budget, etc.) necessary to only one full time staff at NERC who was
accomplish the objectives of the processes assigned to work on NERC CEA
within the functional area? If not, what is responsibilities. This staff person was also
lacking? responsible for leading Certification reviews.
NERC noted that personnel constraints had
been an issue, but should be resolved, as
they were in the process of filling two open
positions.
This was complicated because it was still not

clear exactly what role NERC would have in
serving as the CEA at Regional Entities. One
regional entity signed an agreement with
NERC to have NERC perform all CEA
responsibilities. However, no agreement had
been signed with the two other Regional
Entities that NERC would oversee as the CEA.
Because of this, it was not known whether
NERC would perform all or a subset of the
CEA functions at these entities. For example,
NERC may only perform compliance audits
and leave the other monitoring methods to
the Regional Entities.
5 Are the processes within this functional area Not met. Although these processes were in a
monitored to see that they are consistently very early implementation phase, there did
performed in accordance with the not appear to be any plans for actively
documented process? If so, how? monitoring the NERC CEA functional area.
For example, NERC has not determined what
role the Regional Operations team would
have in monitoring the Organization
Registration and Certification team in the
performance of its duties as the CEA. At the
time of our observations, there were no
plans for involvement by the Director of
Regional Operations, such observation of
audits NERC conducts. NERC officials
expected that FERC would have the primary
oversight role in this capacity. See
Recommendation CEA-03.
5A If the processes within this functional Not applicable. This functional area was not
area are monitored (i.e. the answer to currently monitored.
#5 was “yes”), are they also
measured? If so, what measurements
are taken (and how) and what is done
etc.)?

5B If the processes within this functional Not applicable. This functional area was not
area are monitored (i.e. the answer to currently monitored.
#5 was “yes”), do you do anything
with the cases where the actual
execution deviates from the norm –
that is, the so-called process
exceptions?
6 Do the processes within this functional area Met. The processes documented were
have specific, measurable goals or objectives? detailed and included specific, measurable
If so, what are those? goals such as timelines for completing key
milestones.
6A If the processes within this functional Not applicable. Because at the time of our
area do have specific, measurable goals observations these processes were in a very
(that is, the answer to #6 was yes), how early implementation phase, there was no
frequently do the processes meet their information available to make a
goals or objectives? determination as to the frequency with
which the processes meet their intended
goals and objectives. For example, NERC has
not yet performed any audits of Regional
Entity compliance with reliability standards in
NERC’s capacity as the CEA.
6A.1 If the answer to #6A is anything Not applicable. Because at the time of our
other than 100% of the time, then a observations these processes were in a very
follow-on question: What are the early implementation phase, there was no
primary barriers for the processes information available to make this
to achieve their goals or objectives? determination.

7 Do you consider the processes within this Partially met. Most of the process
functional area to be efficient to the extent
documents in the NERC CEA functional area
avoided unnecessary steps and duplication.
However, we noted one redundancy.
changes can be made? Specifically, the document “Mitigation Plan
Procedure” (NPP-CME-605) outlined a
process for the receipt, review, and approval
of a mitigation plan submitted by a Regional
Entity for a violation it has been alleged to
have committed. Under this process, a
member of the Organization Registration and
Certification team would perform these
functions. However, a similar process
document, “Mitigation Process – NERC CEA”
(NPP-CME-504) was developed by the
Enforcement and Mitigation team and
assigns the same duties to that team.
Similarly, we observed two documents that
detailed NERC’s role as the Compliance
Enforcement Authority when issuing
Remedial Action Directives—“Remedial
Action Process” (NPP-CME-500) and
“Remedial Action Directive Procedure - CEA
(NPP-CME-611). See Recommendations
CEA-04 and CEA-05.
8 How is process documentation made available Met. The process document was available
to personnel? for viewing on a secure network drive.
9 How is process documentation controlled for Met. Only one employee had access to
changes? Do you have a process for formally revise the process document.
responding to and dealing with changes that
In addition, we noted the following issue in this functional area:
We observed that the process documents that had been developed for this functional
area were currently in draft form and had not been reviewed by management. See
Recommendations
CEA-01 Revise or create process documents to fully record the processes, roles, and
responsibilities for handling the enforcement of violations identified by NERC in the
course of its role as a CEA. Because both the Organization Registration and
Certification and Enforcement and Mitigation teams have roles in this process, the
Manager of Organization Registration and Certification and the Manager of
Enforcement and Mitigation should coordinate the drafting of these processes to
ensure agreement among the assignment of roles and responsibilities between the
teams.
CEA-02 Revise the following errors that were identified in the process documents in the
functional area NERC CEA Responsibilities:
In the “Registered Entity Audit Process Procedure” (NPP-CME-602),
step 4.3.a stated: “The MORC will forward the Public Audit Report to
the MORC for final processing.” The step should state: “The MORC
will forward the Public Audit Report to the Vice President and
Director of Compliance for final processing.”
step 4.3.h stated: “The AA *Administrative Assistant+ will process the
Non-Public Audit Report to ... Redact all confidential, privileged,
and/or critical energy infrastructure information...” The step should
state: “The MORC, or his designee, will process the Non-Public Audit
Report to...and will forward the report to the AA.”
step 4.3.j stated: “The MORC will endorse the report as being the
approved Public Audit Report and will forward the approved report to
the MORC.” The step should state: “The MORC will endorse the
report as being the approved Public Audit Report and will forward the
approved report to the Vice President and Director of Compliance.”
In the “Mitigation Plan Procedure” (NPP-CME-605), step 4.1.a stated:
“A registered entity found to be in violation of a reliability standard
shall file a mitigation plan with NERC.” The step should state: ““A
registered entity that NERC found to be in violation of a reliability
standard shall file a mitigation plan with NERC.”
In the “Mitigation Plan Procedure” (NPP-CME-605), the Note on page
8 stated: “If the mitigation plan was submitted via the portal, the
SPOC will contact the SPOC and request them to unlock the form to
allow editing by the entity.” This should state: “If the mitigation plan
was submitted via the portal, the NERC SPOC will contact the
Registered Entity’s SPOC and request them to unlock the form to
allow editing by the entity.”
CEA-03 Develop a comprehensive plan for monitoring NERC’s performance of its CEA duties in
compliance, enforcement, and mitigation. Establish key milestones to track and
report. Include oversight involvement by the Regional Operations team.
CEA-04 Consolidate the documents “Mitigation Plan Procedure” (NPP-CME-605) and
“Mitigation Process – NERC CEA” (NPP-CME-504) into one process document with
roles and responsibilities split appropriately between the Organization Registration
and Certification team and the Enforcement and Mitigation team. Assign the primary
role for interaction with the Regional Entity to the Organization Registration and
Certification team with oversight by the Enforcement and Mitigation team. Also,
assign interactions with FERC to the Enforcement and Mitigation team.
CEA-05 Consolidate the NERC Compliance Enforcement Authority functions noted in the
documents “Remedial Action Process” (NPP-CME-500) and “Remedial Action Directive
Procedure - CEA (NPP-CME-611) into one process document with roles and
responsibilities split appropriately between the Organization Registration and
Certification team and the Enforcement and Mitigation team. Assign the primary role
for interaction with the Regional Entity to the Organization Registration and
Certification team with oversight by the Enforcement and Mitigation team. Also,
assign interactions with FERC to the Enforcement and Mitigation team.
CMEP Process Evaluation Report 114
Appendix I – Functional Area to Processes and Procedures Crosswalk
NERC Process Person(s)

Crowe Processes for Evaluation Identifier NERC Processes and Procedures Name Responsible Relevant ROP Section
CROSS FUNCTIONAL AREAS:
1 - Following compliance program Mike DeLaura ROP 402.8; ROP 404.3; ROP
confidentiality requirements NPP-CME-800 Document Management and Control (and Kate Calla) 1500; CMEP 9.0
2 - Developing and overseeing the compliance
training program NPP-CME-202 Training Process Joel deJesus ROP 402.9
3 - Developing and Disseminating Compliance
Process Directives and Bulletins NPP-CME-205 Compliance Process Bulletins/Directives Joel deJesus None
NPP-CME-701 Compliance Data Reporting Process Mike DeLaura CMEP 8.0

4 - Processing Reliability Standard violations
Compliance Violation and Penalty Process - RE
NPP-CME-501 CEA Tim Kucey CMEP 5.1, 5.2, 5.4, 5.6
FUNCTIONAL AREAS:
1 - Compliance Program Planning NPP-CME-200 CMEP Development and Maintenance Process Dave Hilt ROP 401.1
NPP-CME-201 CMEP Implementation Plan Process Joel deJesus ROP 402.1.1; CMEP 4.0
Monitoring and Facilitating Effectiveness of
NPP-CME-204 the CMEP Dave Hilt ROP 402; ROP 404
2 - Overseeing registration of users, owners,
and operators of the BPS NPP-CME-100 Organization Registration Process Craig Lawrence ROP 500; ROP Appx 5
NPP-CME-102 Organization Registration Appeals Procedure Craig Lawrence ROP 500; ROP Appx 5
3 - Overseeing certification of users, owners,
and operators of the BPS NPP-CME-101 Organization Certification Process Procedure Craig Lawrence ROP 500; ROP Appx 5
NPP-CME-103 Organization Certification Appeals Procedure Craig Lawrence ROP 500; ROP Appx 5
4 - Overseeing the compliance activities of Observation of Regional Entity-led Compliance
Regional Entities (excluding CVIs) NPP-CME-400 Audits Joel deJesus CMEP 3.1.5
NPP-CME-401 Regional Entity-led Compliance Audit Process Joel deJesus CMEP 3.1.6
www.crowehorwath.com

Procedure for the Regions to Self-Certify
Adherence to the ROP and CMEP during and
NPP-CME-402 Audit Joel deJesus None
NPP-CME-204 the CMEP Dave Hilt ROP 402; ROP 404
5 - Overseeing the enforcement activities of NPP-CME-204 the CMEP Dave Hilt ROP 402; ROP 404
Regional Entities NPP-CME-500 Remedial Action Process Tim Kucey CMEP 7.0
Compliance Violation and Penalty Process - RE
NPP-CME-501 CEA Tim Kucey CMEP 5.1, 5.2, 5.4, 5.6
NPP-CME-502 Settlement Process - RE CEA Tim Kucey CMEP 5.4
NPP-CME-503 Mitigation Process - RE CEA Tim Kucey CMEP 6.0
NPP-CME-505 Appeals and Hearing Process Tim Kucey CMEP 5.3, 5.5
NPP-CME-506 Penalty Guidance Process Tim Kucey Appx 4B
6 - Analyzing and reporting compliance

NPP-CME-701 Compliance Data Reporting Process Mike DeLaura CMEP 8.0
information
Data Management, Evaluation, and Analysis
NPP-CME-700 Process Mike DeLaura ROP 408; CMEP 8.0
7 - Conducting review of Regional Entities'
compliance and enforcement programs NERC Audit of Regional Entity Adherence to
NPP-CME-404 the CMEP Joel deJesus ROP 402.1.3; ROP 404.3
NPP-CME-403 RE Spot Check Process Joel deJesus None
8 - NERC involvement in Compliance Inquiries

and Compliance Violation Investigations NPP-CME-300 Compliance Inquiry Process Earl Shockley None
NPP-CME-302 Compliance Violation Investigation Process Earl Shockley CMEP 3.4

NPP-CME-303 Evidence Handling Process Earl Shockley CMEP 3.4

9 - Handling complaints received on the hotline

and via the Web site appropriately NPP-CME-301 Complaint Process Earl Shockley CMEP 3.8
NPP-CME-500 Remedial Action Process Tim Kucey CMEP 7.0
NPP-CME-504 Mitigation Process - NERC CEA Tim Kucey CMEP 6.0
10 - NERC Compliance Enforcement NPP-CME-602 Registered Entity Audit Process Procedure Craig Lawrence CMEP 3.1
Authority responsibilities NPP-CME-603 Self-Report Procedure Craig Lawrence CMEP 3.5
(excluding conducting CVIs)
NPP-CME-604 Spot Check Procedure Craig Lawrence CMEP 3.3
NPP-CME-605 Mitigation Plan Procedure Craig Lawrence CMEP 6.0
NPP-CME-606 Self-Certification Procedure Craig Lawrence CMEP 3.2
NPP-CME-607 Data Reporting and Disclosure Procedure Craig Lawrence CMEP 8.0
NPP-CME-608 Exception Reporting Procedure Craig Lawrence CMEP 3.7
NPP-CME-609 Periodic Data Submittal Procedure Craig Lawrence CMEP 3.6
CMEP 5.1; CMEP 6.0; CMEP
NPP-CME-610 Implementation and Tracking Procedure Craig Lawrence 7.0
NPP-CME-611 Remedial Action Directive Procedure - CEA Craig Lawrence CMEP 7.0
Appendix II – Process Questionnaire
Functional Area Process Evaluation Criteria

O Overview Question: What does the NERC Processes and Procedures Manual mean to you –
that is, how would you envision or interpret it to be used?
1 Is the process objective documented for all processes within the functional area and are all
tied to underlying rules and policies?
If Yes and the crosswalk to the ROP is not apparent, ask which ones
2A For this functional area, how well do the processes currently documented within the
Processes and Procedures Manual map to or match the Rules of Procedure?
2B For this functional area, how well do the processes currently executed map to or match the
Rules of Procedure? Are there areas where there are known exceptions or discrepancies?
2C For this functional area, how well do the processes currently executed map to or match the
Processes and Procedures Manual? Are there areas where there are known exceptions or
discrepancies?
3 How do people know their roles and responsibilities in executing the processes within this
functional area? Are roles and responsibilities documented?
3A Do resources responsible for executing the process have awareness and understanding of
the process (as documented), and capability to execute the process (e.g. they are trained,
have the right skill sets, have the right tools at their disposal, etc.)?
4 Do you and your team have the resources (personnel, systems, budget, etc.) necessary to
accomplish the objectives of the processes within the functional area? If not, what is lacking?
5 Are the processes within this functional area monitored to see that they are consistently
performed in accordance with the documented process? If so, how?
5A If the processes within this functional area are monitored (i.e. the answer to #5 was “yes”),
are they also measured? If so, what measurements are taken (and how) and what is done
with these measurements (reported, etc.)?
5B If the processes within this functional area are monitored (i.e. the answer to #5 was “yes”),
do you do anything with the cases where the actual execution deviates from the norm –
that is, the so-called process exceptions?
6 Do the processes within this functional area have specific, measurable goals or objectives? If
so, what are those?
6A If the processes within this functional area do have specific, measurable goals (that is, the
answer to #6 was yes), how frequently do the processes meet their goals or objectives?
6A.1 If the answer to #6A is anything other than 100% of the time, then a follow-on
question: What are the primary barriers for the processes to achieve their goals or
objectives?
7 Do you consider the processes within this functional area to be efficient to the extent that
unnecessary steps, iterations, resources, and delays have been eliminated? If not, what
8 How is process documentation made available to personnel?
9 How is process documentation controlled for changes? Do you have a process for formally
responding to and dealing with changes that would come down from FERC (for example)?
Appendix III – Observations and Recommendations from

Development of Agreed-Upon Procedures
III. AREAS OF IMPROVEMENT AND RECOMMENDATIONS
In developing the agreed-upon procedures, we identified several key areas in which NERC’s
policies and processes for monitoring Regional Entity compliance could be improved. We
developed recommendations to address these areas of improvement. The details of each
condition identified and related recommendations are listed below.
A. Condition: The NERC Rules of Procedure and related appendices lack policies to
address certain key objectives.
In developing the agreed-upon procedures for Regional Entity compliance, we identified

several areas, outlined below, for which there are no specific policies within the Rules of
Procedure or its associated appendices. As a result, the Regional Entities cannot be held
accountable to a specific practice, and consistency of application of the Rules of
Procedure across regions could be impaired.
o Compliance Timelines - The Rules of Procedure and related appendices do not

specify deadlines for Regional Entities to complete and report to NERC the results or
status of various compliance and enforcement program duties, including compliance
audits, spot checks, compliance violation investigations, reviews of self-certifications
and other information submittals, reviews of mitigation plans, issuances of Notices of
Alleged Violations and Notices of Confirmed Violations, negotiation of settlements.
The Rules of Procedure often do specify timelines within which a given activity
“normally complete,” but these are not requirements.
o Notification of Compliance Activity Results - The Rules of Procedure and related

appendices do not include requirements for the Regional Entities to notify NERC of
the results of various compliance activities, such as compliance audits, self-
certifications, other registered entity data submittals, and spot checks, unless those
activities result in a determination of an alleged violation.
o Review of Conflict of Interest (COI) and Confidentiality Statements - There is no

requirement in the Rules of Procedure or related appendices for Regional Entity
management to perform a review of conflict of interest and confidentiality statements
signed by employees and others performing compliance program duties.
o Update of COI and Confidentiality Statements - There is no requirement for

individuals involved in compliance and enforcement to periodically update or re-
affirm conflict of interest and confidentiality statements.
o Reporting Confidentiality Violations - There is no requirement for Regional Entities

to report violations of confidentiality rules to NERC.
o Background Checks - There are no requirements for Regional Entities to conduct

background checks of contractors, employees, or other individuals working in
compliance and enforcement.
o Work History Restriction - Although Section 403.6.5 of the Rules of Procedure

specifies that an independent consultant cannot work in a compliance enforcement
program if he or she received compensation from a monitored BPS owner, operator or
user within the past 6 months, there is no such requirement specific to Regional
Entity staff.
o Data Security - NERC has not provided the Regional Entities any specific minimum
standards for maintaining data security.
o Timelines for Notifying NERC - Although the Rules of Procedure specify that
Regional Entities must notify NERC of the following situations, the Rules of
Procedures and related appendices do not specify time requirements for notifying
NERC of the following
 The initiation and outcome of a hearing
 Conclusion of a settlement proceeding
 Receipt of certification applications
 Initiation of an unscheduled compliance audit
 Results of a compliance violation investigation, if the investigation did not
substantiate an alleged violation
 Results of complaints to Regional Entities from third parties, if the Regional
Entity did not initiate a compliance violation investigation as a result
 Receipt of anonymous complaints
 Verification that a registered entity completed its mitigation plan
o CVI Initiation – Section 3.4.1 of Appendix 4C of the Rules of Procedure states,

“Within two (2) business days of the decision to initiate a Compliance Violation
Investigation, the Compliance Enforcement Authority: (i) notifies the Registered
Entity of the initiation and initial scope of the Compliance Violation Investigation,
the requirements to preserve all records and information relevant to the Compliance
Violation Investigation and, where appropriate, the reasons for the Compliance
Violation Investigation, and (ii) notifies NERC of the initiation of and the reasons for
the Compliance Violation Investigation.” However, NERC’s policies do not specify
the point at which a CVI is considered “initiated” and do not require regional entities
to document when a CVI is initiated.
Recommendation: NERC should review the instances we identified above in which no

specific policies exist and consider whether developing policies in these areas would
benefit NERC’s compliance program goals.
B. Condition: Certain policies in the NERC Rules of Procedure and related

appendices differ from NERC’s requirements of Regional Entities in practice.
We also identified a number of instances in which policies in the NERC Rules of

Procedure and related appendices are inconsistent with NERC’s actual practice. Such
inconsistencies could create confusion in the application of the Rules of Procedure and
the policies to which the Regional Entities are held accountable.
o Use of Terms Such as “Guidelines” - The Rules of Procedure note several instances
that refer to NERC “guidance,” “guidelines,” or “procedures,” implying that these are
not required actions. However, in practice, these are policies that NERC requires
Regional Entities to follow. For example, Rule of Procedure 403.10.2 states that
“When requested, the RE shall report promptly to NERC in accordance with NERC
procedures.” Also, Appendix 4B to the Rules of Procedure is titled “ERO Sanctions
Guidelines.” Further, Section 3.1.1 of Appendix 4C to the Rules of Procedure states,
“The audit team follows NERC audit guidelines in the implementation of the
Compliance Audit.”
o COI Statements - The Rules of Procedure and associated appendices do not specify a
requirement for compliance program participants to sign conflict of interest
statements, but NERC does require this in practice (ROP 400 line 52)
o Exception Reporting - Section 3.7 of Appendix 4C to the Rules of Procedure state

that Regional Entities must require Registered Entities to submit Exception Reports
“to the extent required by any Reliability Standard” and confirm the number of
exceptions to compliance with the reliability standards that have occurred within a
given time period identified by NERC, even if the number of exceptions reported is
zero. However, NERC in practice does not require periodic exception reporting, but
rather expects that Registered Entities will report all exceptions they identify
immediately through the Self-Reporting procedures in Section 3.5 of Appendix 4C to
the Rules of Procedure.
o 48 Hour Reporting - Section 408.1.1 of the Rules of Procedure and Section 8.0 of
Appendix 4C to the Rules of Procedure specify that Regional Entities must report
violations of certain specifically identified reliability standards with 48 hours.
However, NERC’s 2008 practice allowed for reporting of such violations within 2
business days. Per its implementation plan, in 2009 NERC decided not to specify any
standards that require 48 hour reporting.
o Notices of Violation – NERC’s use of terms regarding notices of violations are not
consistent with NERC’s use of the terms in practice. For example, NERC’s
definition of an “initial notice of violation” differs between its policy and its practice.
Section 5.1 of Appendix 4C to the Rules of Procedure describes an “initial notice of
Alleged Violation” as an optional notice that NERC or the Regional Entities can send
to Registered Entities informing them that they may have committed a violation.
However, in NERC’s actual practice, an “initial notice of Alleged Violation” (also
known as a “preliminary notice of alleged violation”) is a notification that NERC

sends to FERC after a Regional Entity has notified NERC of an alleged violation by a
Registered Entity. In addition, several areas of the Rules of Procedure, such as
Section 3.1.1 of Appendix 4C, refer to “Alleged Violations” that are what NERC in
practice considers the preliminary violation stage, before a Notice of Alleged
Violation and Penalty and Sanction has been issued.
o Deadline to Challenge NERC Decisions - Regarding the filing of challenges to NERC

decisions, Rule of Procedure 409.1 states: “A registered entity or a regional entity
wishing to challenge a finding of noncompliance and the imposition of a penalty for a
compliance measure directly administered by NERC, or a regional entity wishing to
challenge a regional compliance program audit finding, may do so by filing a notice
of the challenge with NERC’s director of compliance no later than 21 days after
issuance of the notice of finding of violation or audit finding.” In practice, however,
the 21 day timeline starts with the rendering of the decision by NERC.
o Use of Term “Appeals” - Regarding the entity registration and certification processes,
Section 504 of the Rules of Procedure states, Each regional entity with delegated
responsibilities shall establish and maintain a fair, independent, and
nondiscriminatory appeals process.” In addition, Section 7.0 of Appendix 4C to the
Rules of Procedure states: “Notice to contest the Remedial Action Directive and
participation in the hearing process set forth in Section 1.9 of Attachment 2, Hearing
Process shall constitute the Registered Entity’s right to appeal the Remedial Action
Directive.” However, NERC only intends to use the term “appeal” to refer to a
challenge brought before NERC. The term NERC uses to refer to a challenge
brought before a Regional Entity is a “hearing.”
o Compliance Audit Leads - Section 3.1.5 of Appendix 4C to the Rules of Procedure

states that only regional entity staff may serve as compliance audit team leaders.
However, in practice, NERC also allows individuals who are not employees of the
regional entity but directly contracted with the regional entity for at least one year to
lead compliance audits.
o Monthly and Quarterly Registered Entity Reporting - The regional entity compliance
schedule in NERC’s implementation plan requires regional entities to review monthly
and quarterly reports from registered entities. These types of reports are separate
from and not included with the data required to be submitted by registered entities
and reviewed by regional entities under Section 3.0 of Appendix 4C to the Rules of
Procedure.
o Mitigation Plan Review by NERC - Currently, Section 6.5 of Appendix 4C to the

Rules of Procedure states that the “Compliance Enforcement Authority” is
responsible for reviewing and approving mitigation plans and that NERC will review
the accepted Mitigation Plan and notify the Regional Entity and the Registered Entity
as to whether the mitigation plan is approved or disapproved by NERC in concert
with forwarding a confidential non-public copy of the mitigation plan, if approved by
NERC, to FERC. However, in practice, regional entities first review and approve a
mitigation plan, then submit it to NERC for further review.
o Difference in NERC Tool and Appendix to the ROP - Confidential information has
been removed from this public version and has been provided under separate cover to
NERC management.
o Timeline for Responding to Violation Notices - Section 5.2 of Appendix 4C of the

Rules of Procedure states: “If the Registered Entity does not contest or does not
respond to the notice of Alleged Violation within thirty (30) days, it shall be deemed
to have accepted the Compliance Enforcement Authority’s determination of violation
and sanction (if applicable), in which case the Compliance Enforcement Authority
shall issue to the Registered Entity and NERC a final report of Confirmed Violation.
A Registered Entity may provide a written explanatory statement to accompany the
final report.” However, in practice NERC allows regional entities to extend this
deadline.
o Timeline for Reporting Confirmed Violations – Section 8.0 of Appendix 4C to the

Rules of Procedure states: “Regional Entities shall report to NERC all Confirmed
Violations of Reliability Standards by Registered Entities including all penalties,
sanctions, Mitigation Plans and schedules, and settlements, within ten (10) business
days of each determination.” However, in practice NERC does not enforce any
deadline for regional entities to report on violations that have been confirmed.
Recommendation: We recommend that NERC consider revising its Rules of Procedure

to reflect its actual practices in the areas outlined above, or revise its actual practice to
conform to the applicable Rule of Procedure.
C. Condition: Redundant information and policies exist in the NERC Rules of

Procedure and related appendices.
The NERC Rules of Procedure and its associated appendices contain several instances of
information that is repeated in different sections of the document. Having redundant
information within the Rules of Procedure is not necessarily a shortcoming of the
document. For example, certain policies, such as the requirement for Regional Entities to
notify NERC of alleged violations, apply in various compliance situations, and it may be
easier for a reader to have that policy within the section of interest to him or her, rather
than having to follow a reference to a different part of the document. However, an excess
of repeated information can make the document difficult for NERC to manage. In
particular, if NERC chooses to amend a certain policy, it will have to ensure that all
references to that policy within the Rules of Procedure and its associated appendices are
updated.
During the process of developing the agreed-upon procedures, we identified the

following areas in NERC’s compliance enforcement program where policies are
mentioned in multiple sections of the Rules of Procedure or its related appendices.
Penalties, Sanctions, and Remedial Action Directives - Policies related to the

assessment of penalties, sanctions, and remedial action directives are provided in
Sections 401.7, 402.5, 403.17, and throughout Section 407 of the Rules of Procedure
(ROP), as well as Appendix 4B to the Rules of Procedure and Sections 5.0 and 7.0 of
Appendix 4C to the Rules of Procedure.
Data Confidentiality - The following sections in the Rules of Procedure all specify
policies for maintaining the confidentiality of information, such as requirements for
compliance program participants to have confidentiality agreements and policies for
redacting critical infrastructure information: sections 402.3, 402.8, 403.6.4, 403.7.4,
403.14 (which is specific to compliance audits and compliance violation
investigations), 403.16, 408.3 (which is specific to NERC), 408.6.2, and 1500. In
addition, Sections 3.1.5 and 3.1.6 of Appendix 4C to the Rules of Procedure specify
requirements in the area of confidentiality, related to the performance of compliance
audits. Section 9.0 of Appendix 4C to the Rules of Procedure also encompasses data
confidentiality requirements.
Independence - Section 403.6 of the Rules of Procedure addresses requirements for

the independence of compliance program staff. Section 403.7.2 of the Rules of
Procedure addresses such requirements for industry experts and Regional Entity
members. Similarly, Section 3.1.5 of Appendix 4C to the Rules of Procedure
addresses independence requirements for individuals participating in compliance
audits.
Audit Training - Requirements for the completion of compliance audit training are
addressed in Section 402.9 of the Rules of Procedure and in Section 3.1.5 of
Appendix 4C to the Rules of Procedure. Further requirements for industry experts
and Regional Entity members to attend compliance audit training are noted in section
403.7.5 of the Rules of Procedure.
Notification of Violations - Requirements to notify NERC of alleged violations are

found in Sections 403.10.3, 403.15, and 408.1, of the Rules of Procedure and in
Sections 5.1, 3.1.1, 3.2.1, 3.3.1, 3.4.1, 3.5.1, and 3.6.1 of Appendix 4C to the Rules of
Procedure. Section 8.0 of Appendix 4C to the Rules of Procedure includes this and
other situations that Regional Entities are required to report to NERC.
Mitigation Plans - Rules of Procedure Sections 403.10.4 and 403.18, and Section 6.0
of Appendix 4C to the Rules of Procedure list policies for the submission and review
of mitigation plans.
Settlements - Section 403.19 of the Rules of Procedure, Sections 3.2 through 3.4 of
Appendix 4B to the Rules of Procedure, and Section 5.4 Appendix 4C to the Rules of
Procedure all provide policies related to the conduct of settlements.
Hearings - Sections 403.4, 403.20, and 407.3 of the Rules of Procedure, along with
Sections 5.2 and 5.3 of and Attachment 2 to Appendix 4C to the Rules of Procedure
provide policies related to the hearing process.
Registration and Certification - Section 500 of the Rules of Procedure, Section 2.0 of
Appendix 4C to the Rules of Procedure, and Appendix 5 to the Rules of Procedure all
provide policies for the registration and certification of bulk power system entities.
Penalty Calculations - Within Appendix 4B to the Rules of Procedure, Sections 3.10

and 4.3.1 both address consideration of multiple and repetitive violations in
calculating the penalty amount; Sections 3.11 and 4.4.1 both address policies for
consideration of the registered entity’s ability to pay in determining a penalty;
Sections 3.13 and 4.3.8 both address consideration of extenuating circumstances in
determining a penalty; Sections 3.14 and 4.3.6 both address consideration of
concealment of a violation in calculating a penalty amount; and, Sections 3.14
through 3.16 and Section 4.3.7 both address consideration of intentional violation of a
reliability standard, including the economic choice to violate.
Recommendation: We recommend that NERC review its Rules of Procedure and

associated appendices for redundancies and consider consolidating, deleting, or
otherwise revising those policies that it deems duplicative.
D. Condition: Certain NERC policies are not adequately defined, specific, or

measurable enough to assure accountability.
In developing the agreed-upon procedures for Regional Entity compliance, we identified

a number of instances in which policies in the Rules of Procedures and related appendices
are non-specific, not adequately defined, or not measurable. In such cases, it becomes
difficult to hold the Regional Entities accountable for particular actions. We identified
the following policies that could be considered for further specificity, definition or
measurability:
o The “CMEP goals,” which the ROP states Regional Entities are required to meet, are
not specified or described. (ROP 401.4)
o The “appropriate codes of conduct” which compliance program participants should

follow are not defined. (ROP 402.8.1)
o What it means to “promote excellence in the enforcement of reliability standards” Is

not specified. (ROP 403)
o There is no guidance on what constitutes prohibited sub-delegation of compliance

activities and what specific activities are covered. (ROP 403.3)
o There is no guidance on the measurement of “sufficient resources” to meet

compliance enforcement responsibilities. (ROP 403.5)
o There is no guidance on how to measure whether compliance enforcement staff

participants are “capable.” (ROP 403.6)
o There is no guidance on how to measure whether compliance audits are “proactive.”

(ROP 403.11)
o There are no standards on how to assess whether bulk power system owners,
operators, and users are given a “reasonable opportunity” to demonstrate that
information is confidential before a report becomes public. (ROP 408.3.1)
o What constitutes the “entire record” surrounding a notice of appeal is not defined.
(ROP 410.3)
o The NERC requirements for “quality, thoroughness, timeliness, accuracy, efficiency,

cost-effectiveness, and participation” for Regional Entities participating in the
registration and certification program are not identified. (ROP 502.1.3)
o The “appropriate training” certification audit participants should receive is not

specified, nor does the policy specify whether the training is the same as that which
compliance audit participants must receive. (ROP 503.3.3.5.2)
o No specific timeline for “promptly” reporting to NERC is given. (ROP 403.10.2,

ROP 403.15, ROP 408.1, and ROP 507.6)
o The specific “NERC requirements” for maintaining work papers and other
documentation associated with a compliance audit are not identified. (ROP Appendix
4C, section 3.1.6)
Recommendation: We recommend that NERC review the wording of the policies

outlined above and consider taking the action to develop definitions or criteria that are
more measurable and specific.
E. Condition: Instances of inconsistent wording of policies exist within the NERC

Rules of Procedure and related appendices.
We identified two instances of inconsistent wording within the Rules of Procedure and
related appendices themselves. The inconsistent wording could complicate NERC’s
ability to hold Regional Entities accountable to the required policy. These instances also
illustrate the potential pitfalls that could occur when a policy is included in more than one
section of the Rules of Procedure.
o Data Retention Requirement - Section 9.1 of Appendix 4C to the Rules of Procedure

states that a compliance program records management policy must "at a minimum
conform to the Reliability Standards data retention requirements of the Reliability
Standards." However, data retention requirements in Section 9.2 of Appendix 4C are
stricter, specifying that compliance program records must be retained for the longer
of 5 years or the requirements of the Reliability Standard or Applicable Governmental
Authority.
o Implementation Plan Deadline - Section 403.21 of the Rules of procedure requires the
Regional Entities to submit an annual compliance enforcement implementation plan
“generally on or about November 1 of the preceding year.” However, Section 4.2 of
Appendix 4C to the Rules of Procedure requires submission of the annual
implementation plan, “By November 1 of each year.”
Recommendation: We recommend that NERC revise its Rules of Procedure to 1) bring

Section 9.1 of Appendix 4C of the Rules of Procedure into conformance with Section 9.2
of Appendix 4C, as noted above, and 2) bring Section 403.21 of the Rules of Procedure
into conformance with Section 4.2 of Appendix 4C to the Rules of Procedure, as noted
above.
Appendix IV – Excerpt from Management Letter to NERC
The following excerpt includes those observations and recommendations made to NERC to
revise the Rules of Procedure from a recently completed Agreed-Upon Procedures project for
one of the regional entities.
Note: The following excerpt includes only those observations and recommendations made to
NERC to revise the Rules of Procedure. Recommendations to NERC also resulted from the
agreed-upon procedures performed at other regional entities; however, none of these
recommendations involved changes to the ROP.
VIOLATIONS LANGUAGE
Observation: Language related to "violations" in the ROP and CMEP is not clear and
often not consistent. For example, throughout the ROP, NERC uses the
term "alleged" violation to refer to potential violations regardless of
whether a Notice of Alleged Violation and Penalty and Sanction has been
issued.
Recommendation: We recommend that NERC review the ROP and CMEP to ensure that
language related to violations is applied consistently and refers distinctly
to the levels of violations (e.g., possible, alleged, confirmed) that NERC has
recognized.
DATA RETENTION
Section 9.2 of the CMEP states: “The Compliance Enforcement Authority

records management policy will require that information and data
generated or received pursuant to Compliance Program activities, including
Compliance Audits, Self-Certifications, Spot Checking, Compliance
Violation Investigations, Self-Reporting, Periodic Data Submittals,
Exception Reporting, and Complaints, as well as a hearing process, will be
retained for the longer of (i) five (5) years or (ii) any retention period
specified in a Reliability Standard or by FERC or another Applicable
Governmental Authority. The obligation to retain information and data
Observation: commences upon the initiation of the Compliance
Program activity that produces the data or information.”
This requirement is not specific as to what the “the initiation of the

Compliance Program activity” means. For example, there is no specific
point a Compliance Violation Investigation is initiated. In addition, it is not
clear whether a self-certification is initiated on the due date of the
certification or the date it is received from the Registered Entity.
We recommend that NERC specify when each compliance activity is

Recommendation:
initiated in the context of the data retention requirement.
REFERENCE IN CVI PROCESS
On page 6 of the NERC Compliance Violation Investigation Process, a

Observation: reference is made to the NERC ROP on training. The current reference is to
section 400.7.5 of the ROP, when, in fact, it should reference section 403.7.5.
Recommendation: We recommend that NERC change the reference from 400.7.5 to 403.7.5.
NON-SUBMITTAL OF REQUESTED DATA
CMEP Attachment 1, "Process for Non-Submittal of Requested Data,”

states:
“If data, information, or other reports (including Mitigation Plans)

requested from a Registered Entity are not received by the Required Date,
the Compliance Enforcement Authority may sequentially execute the
following steps for each Reliability Standard for which the Compliance
Observation:
Enforcement Authority has requested data, information, or other reports.
The Compliance Enforcement Authority however will afford the Registered
Entity reasonable opportunity to resolve a difficulty submitting data due to
time or format issues."
Currently, the CMEP uses the term “may” which may imply that these
steps are optional.
We recommend that NERC update CMEP Attachment 1 to "require" the

Recommendation:
stated steps.
FORMAT FOR REPORTING
NERC’s policies do not specify the means by which Regional Entities are to
report information to NERC and other entities. For example, NERC’s
policies do not require that Regional Entities use the NERC Workbook for
reporting violations, or even require Regional Entities to report all the
Observation: information that is captured in the Workbook. In addition, NERC uses the
term “Initial Notice of Alleged Violation” to refer to the notice it sends to
FERC when a determination of an alleged violation is made. NERC’s
policies do not reference the INAV letter that Regional Entities send to
Registered Entities when a violation is identified.
We recommend that NERC clarify in its policies whether and when specific
Recommendation:
reporting templates are required to be followed.

Compliance Evaluation Report 121509

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Compliance Evaluation Report 121509

Uploaded by

Copyright:

Available Formats

Compliance Enforcement, Registration, and

Process Evaluation Report

November 23, 2009

Report prepared by:

4. Processing Reliability Standards Violations

“Service Organization” whose purpose was to provide technical assistance to a

term solutions built on enterprise-level platforms with the foundation of IT controls

Phase I - Planning and Data Gathering

Outputs ▪ Project Operating Model and Project Schedule

Table 1 – Project Approach Phase 1

Phase II - Data Analysis and Reporting

Outputs ▪ CERCP Process Evaluation Report (public/non-confidential)

Table 2 – Project Approach Phase 2

4 Cross-Functional Areas and

Procedures 37 Individual Processes and Procedures

Figure 1 – Level of Evaluation

Figure 1 – Level of Evaluation

Process Evaluation Methodology

5. Whether an appropriate “process environment” is in place to support appropriate execution

This report contains no confidential information. Confidential information gathered or shared

Section 2: Observations and Recommendations

The Process-driven Organization

not limited to:

Table 6 – The Infrastructure for Process Success

Process Governance and the Process Foundation Summary Observations

Overarching Observations and Recommendations

Recommended Changes to the Rules of Procedure

Process Documentation Development

PPM-04 We noted as a recommendation in many of the functional area evaluations, that

PPM-08 We recommend that NERC Compliance consider changing in its process

Compliance Department Staffing Levels

STA-03 We recommend that NERC Compliance consider developing a program to ensure

Resolving Issues with Other Entities

ENT-02 We recommend that NERC Compliance consider the development of additional

Monitoring and Measuring Compliance Processes

MON-03 We recommend that NERC Compliance develop and implement a system of

Handling Compliance Information

Maintaining Security and Confidentiality

Category Description of the Recommendation Category

Report Section Topic or Functional Area ID Category

Report Section Topic or Functional Area ID Category

Report Section Topic or Functional Area ID Category

Evaluations (continued) Authority Responsibilities CEA-02 3

Table 8 – Recommendations Summary by Category of Recommendation

REPORT SECTION RECOMMENDATION ID, BY CATEGORY TOTAL

Section 3: Cross-Functional Areas Evaluation

3.1. Compliance Program Confidentiality Requirements

Introduction and Scope

Observations and Recommendations

3.2. Developing and Overseeing the Compliance Training Program

Introduction and Scope

3.3. Developing and Disseminating Compliance Process Directives and Bulletins

Introduction and Scope

Accordingly, NERC drafted a new document (“Compliance Process Bulletins/Directives,”

3.4. Processing Reliability Standards Violations

Introduction and Scope

o “Compliance Data Reporting Process” (NPP-CME-701)

Section 4: Functional Area Evaluation

1. Compliance Program Planning

Each evaluation section will use the following format:

4.1. Compliance Program Planning

Introduction and Scope