You are on page 1of 36

Risk Analysis QC Plans: Principles and Priorities

James O. Westgard University of Wisconsin Medical School Westgard QC, Inc. Madison, WI www.westgard.com
1

Objectives

Review principles of industrial risk analysis Identify priorities when applying risk analysis for analytical quality management in medical labs Assess importance of method validation for dealing with safety characteristics Assess importance of Statistical QC for providing known detection capabilities Recognize the advantages of integrating SQC procedures and Risk Analysis control mechanisms in a QC Plan
2

Oct 25, 2011: CLSI EP23A Risk Analysis and The Right QC!

Whats the Right QC?

The Right QC is what I always do! Why would I do the wrong QC?

Whats The Right QC according to CLIA?

493.1256 Standard: Control procedures.

(a) Laboratory is responsible for having control procedures that monitor the accuracy and precision of the complete analytical process. (b) must establish the number, type, and frequency of testing control materials (c 1) The control procedures must detect immediate errors that occur due to test system failure, adverse environmental conditions, and operator performance; (c 2) Monitor over time the accuracy and precision of test performance
6

Where learn about Risk Analysis and QC?


ISO 14971

Manufacturers
ISO 15198 CLSI EP18

CLSI EP22

CLSI C24

CLSI EP23

ISO 22367

Laboratories
7

What is risk? Industrial Model

Risk - function of 3 factors:


OCCurrence probability or frequency of failure DETection probability that failure will be detected before harm SEVerity consequence or harm

How estimate risk?

Rank each factor


Scale of 1-10 common in industry Scales of 1-5 and 1-3 common in healthcare

Combine factors
Industry calculates Risk Priority Number Risk = OCC * DET * SEV = RPN Healthcare often use Risk Matrix

FMEA is the common industrial tool

Failure Modes and Effects Analysis


9

How mitigate risk in industry?

OCCURENCE - Design for safety to eliminate failure-modes

Disclose safety characteristics Provide alerts to identify problems

DETECTION - Build in Control Mechanisms

SEVERITY - Provide instructions for safe use and precautions for use errors

Inform laboratory of limitations and advise how to monitor performance (QC)


10

OCCURRENCE Key Guidance from ISO 14971

First, design for safety!

IVD medical devices have performance characteristics that determine the accuracy of examination results. Failure to meet the performance characteristics required for a specific medical use could result in a hazardous situation that should be evaluated for risk to patients.
11

Method Validation A prerequisite to risk analysis

Safety Characteristics performance characteristics that determine the accuracy of IVD devices - precision, trueness or bias, analytical specificity, quantitation limit, etc.

Laboratory must verify/validate performance is satisfactory for intended use

12

SEVERITY Key guidance from ISO 14971


H2.5.2

Estimating the severity of harm requires an understanding of the medical use of the IVD examination results, the analytical performance requirements for each application and the extent to which medical decisions are based on IVD examination results. For this reason, qualified medical input to the risk estimation process is essential.
13

What should a laboratory do about severity?

Detect immediate errors that occur due to test system failure, adverse environmental conditions, and operator performance (CLIA 493.1256) Perform corrective actions to recover before reporting of test results Provide information for safe use
14

DETECTION Key Guidance from ISO 15198

For existing IVD medical devices, conventional statistical quality control procedures (e.g., as described in CLSI C24) are considered adequate unless evidence from risk-monitoring activities indicates [other] quality control procedures are essential for maintaining risk at an acceptable level. In that case, the quality control procedures shall be validated.
15

Statistical Quality Control for Quantitative Measurement Procedures


CLSI C24-A3 (2006)

Describes QC planning process Provides Sigma-metric QC selection tool


Calculate Sigma = (TEa Bias)/SD Where TEa is quality required for test, Bias represents inaccuracy of method SD represents imprecision of method

16

CLSI C24A3 QC Planning Tool Relates Sigma to QC


Sigma Scale Probability for Rejection (P)
1.65 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.0
Desirable False Rejection Desirable Error Detection

3
3.65

4
4.65

5
1

6
5.65 P fr P ed N R /2of3 /R /3 /6 3s 2s 4s 1s x 0.07 0.07 6 1 1 /2 /R /4 3s 2s 4s 1s 0.03 0.03 4 1 1 2.5s 0.04 0.04 4 1 12.5s 0.03 0.03 13s /22s /R4s 0.01 13s 0.00 1 3.5s 0.00 1 3s 0.00 0.01 0.00 0.00 0.00 2 2 2 2 1 1 1 1 1 1

2.65

1.0

2.0

3.0

4.0

Systematic Error (SE, multiples of s)


17

How should a medical laboratory address Detection?

Right SQC is essential in any QC Plan


Define the quality needed for intended medical use or clinical application Account for the observed precision and bias of the measurement procedure Account for the known performance characteristics (sensitivity, error detection) of the QC procedure Account for Expected and Unexpected Events

18

Expected and Unexpected Events are Key to Risk Analysis QC Plans

Expected Events
What changes or failures might occur? What is the probability of failure? What is the severity of harm from a failure? What control mechanisms can be implemented to detect failures?

Unexpected Events

How verify the attainment of the intended quality of test results?


19

CLN November 2011 Risk Management for QC

20

CLSI EP23A Laboratory Quality Control based on Risk Management

Quality Control Plan a document that describes the practices, resources, and sequences of specified activities to control the quality of a particular measuring system or test process to ensure requirements for its intended use are met

21

Example QC Plan
QCPlan
Analyst/operatorcontrols
StandardOperatingProcedure YearlySOPreview Operatortraining Operatorchecklists Systemmaintenance Operatorcompetency Everyoperator Daily Manuf.Schedule Yearly Manuf. Manuf. Manuf. Manuf./Reg. Startup+Monitor Calibration 3/year Eachsample Directorreview Supervisorreview Supervisorreview Manuf.Repair Retrain Manuf.Instructions Manuf.Instructions Supervisorreview TSguidelines TSguidelines CAplan Repeattest No No No No No No No No No No No Yes 22

Frequency

Recovery

Disclosure

Builtinanalyzercontrols
Electronicchecks Functiontests Processtests Calibrationchecks Manuf.Instructions Samplecondition

Stablecontrolmaterials
StatisticalQC Truenesscontrol PeriodicEQA,PT

Patientdataanalysis
Implausiblevalues

TS = Trouble-Shooting; CA = Corrective Action

CLSI EP23A Laboratory Quality Control based on Risk Management

Risk combination of the probability of occurrence of harm and the severity of that harm

Risk = OCC * SEV

OCC - Probability of occurrence of harm is not the same as probability of occurrence of a failure

23

CLSI EP23A Occurrence Rating


EP23 Rating Frequent Probable Occasional Remote Improbable Definition Once per week Once per month Once per year Once every few years Once in lifetime of system Ratio 1/7 1/30 1/350 1/1000 1/2000 Defect rate 0.1429 0.0333 0.0029 0.0010 0.0005

What is Occurrence when observe 1 hemolyzed sample per day? Does it make any difference if the lab workload/day is 10 or 1000 samples?
24

1 failure with a sample 1 failure with an analytic run 1 failure per shift 1 failure per day 1 failure per bottle of reagent 1 failure per lot of reagent 1 failure per calibration 1 failure per calibration cycle 1 failure per proficiency testing event
25

Occurrence 1/What?

Occurrence - 1/What? Problem with ranking scales


Time is not the right reference for estimating occurrence Number of patient test results provides a better reference to account for workload Defect rate provides a better perspective for estimating occurrence

26

Estimating Occurrence 1. Describe Lab Process


Lab Process Description Samples/run Runs/day Workdays/week Weeks/year Months/year Workdays/year Samples/year 3 year factor 5 year factor Parameters 50 2 6 52 12 312 31200 0.33 0.20

27

Estimating Occurrence 2. Expand ranking scale


Ranking Very frequent Very frequent Frequent Frequent Probable Probable Occasional Remote Improbable Description 1 sample/day 1 run/day 1 sample/week 1 run/week 1 run/month 1 day/month 1 day/year 1 day/3 years 1 day/5 years

28

Estimating Occurrence 3. Calculate Defect Rate & DPM


Ranking 1 sample/day 1 run/day 1 sample/week 1 run/week 1 run/month 1 day/month 1 day/year 1 day/3 years 1 day/5 years Defects/ Year 312 15,600 52 2,600 600 1,200 100 33 20 Defect Rate 0.0100 0.5000 0.0017 0.0833 0.0192 0.0385 0.0032 0.0011 0.0006 Defects/ Million 10,000 500,000 1,667 83,333 19,231 38,462 3,205 1,058 641

29

Whats the point?

Risk analysis is more complicated than it first appears!

CLSI guidelines simplify risk analysis, but also make it qualitative, subjective, even arbitrary

Laboratories must prioritize activities for defining quality requirements, validating method performance, and selecting SQC procedures to provide a safety net for catching errors
30

Whats the point?

Risk analysis should be integrated into existing error framework for managing analytical quality of testing processes Development of Analytic QC Plan should complement Statistical QC by adding control mechanisms for specific failure modes

Should identify control mechanisms for preanalytic and post-analytic failure-modes


31

SQC vs Risk Analysis


(1) Specimen collection

QC1 QC2 QC3 QC4 QC5 QC6 QC7


32

(2) Sample processing

Traditional Statistical QC

(3) Amplification (4) Calibration

Risk Analysis QC Plan

(5) Detection (6) Readout (7) Result Report

SQC and Risk Analysis


(1) Specimen collection

QC1 QC2

(2) Sample processing

(3) Amplification (4) Calibration

SQC

(5) Detection (6) Readout (7) Result Report

QC7
33

Potential Benefits of Risk Analysis QC Plans

Address pre-analytic and post-analytic problems

Laboratory may change these processes more readily than changing analytic processes

Address specific problems in highly complex analytic systems

Need to monitor critical control points

Address QC for Lab Developed Tests


34

Thats my diagnosis!

Dr. Parvin, what do you think? Whats the probability of occurrence of harm?

You might also like