Chapter 4 PPT Updated

Chapter 4: Method Evaluation and Quality Management
By Christopher R. McCudden, Mike Rogers, Jordan Erickson, Ronald Erickson, Monte S. Willis
Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins
Let me hit you with some knowledge!

http://www.youtube.com/watch?v=C29zdg3F_GA
Let me know if I am going to fast through these slides after this short video.
REAL WORLD: Why is method evaluation and quality management important?

The majority of medical decisions are made using laboratory data. Physicians will base care off of your results!
It is critical that results generated by the laboratory be ACCURATE! These are real people you are providing results for! These results from the tests you run can be on me, you, or our family and friends!! Treat every sample as if you knew the patient personally!
Basic Concepts
Descriptive Statistics: Measures of Center, Spread, and Shape
Measures of Center Mean: average Median: middle point Mode: most frequently occurring value Spread: how data are distributed
Shape: distribution
Gaussian: mean, median and mode are identical; distribution is symmetrical (bell curve)
REAL WORLD: what do we normally use?
Normal distribution contains A) 68% within 1 SD B) 95% within 2 SD C) 99% within 3 SD REAL WORLD: Most labs use the 2 SD rule for quality control(QC) and the 95th percentile for reference intervals(normal ranges).
REAL WORLD: Calculations!

Although it is important to understand how these measures are calculated, many instruments, laboratory information systems and software packages determine these automatically.(ONLY APPLIES TO ME)
YOU need to pass the ASCP exam and these calculations of mean, SD, CV and more will be on your test, so review the statistics class you took and practice these with a basic scientific calculator! Practice problems in your book!
Notice that as concentration gets higher, the SD increases and the CV decreases. Vice versa. CV is a percentage, so higher numbers help reduce the CV(a CV <10% is acceptable in real world conditions). SDs can vary depending on analyte and concentration.
Basic Concepts (contd)

Descriptive Statistics of Groups of Paired Observations
Comparison-of-methods experiment: involves measuring patient specimens by both existing (reference) and new (test) methods for accuracy purposes Linear regression: indicates the strength of the relationship between tests Types of error: random and systemic Method comparison only used when validating a new test or correlating one instrument to another. Linear regression plotted using software and error limits are provided by CAP(College of American Pathologists). Will be covered some more with Method Validation section.
REAL WORLD:
Reference Interval Studies

Reference Interval
A pair of medical decision points that span limits of results expected for a given condition(doctors compare the measured test result with a reference interval) Established by scientific community or manufacturers of reagents(reference interval studies performed by the company who manufactures the test)
Types
Establishing a reference interval(performed by manufacturers and research labs) Verifying a reference interval(process that most labs perform or adopt reference interval that the manufacturer recommends or from diagnostic test literature)
Reference Interval Studies (contd)

Reference Intervals can be grouped into three main categories: Diagnosis of a disease or condition Monitoring of a physiologic condition Therapeutic management
Reference Interval for diagnosis of a disease or condition. Above reference interval is hypothyroidism Below reference interval is hyperthyroidism Notice interval based on patients age
Reference Interval for the monitoring of a physiological condition. Notice interval is dependent on number of weeks of pregnancy.
Reference Interval for therapeutic management Notice interval based on time of collection TDM very important! Too much gentamicin or vancomycin can cause organ failure!!
How do we get normal ranges?

A reference interval or normal range of results is between two medical decision points that correspond to the central(mean) and include 95%(2 SD) of results from healthy patient population.
This means that a 5% of HEALTHY patients will fall outside of the reference interval in the absence of a condition or disease.
REAL WORLD: Diagnosis is NEVER based on just lab data, physicians have to clinically correlate with patient physical condition!! Example
Real World Situation: True Story

Physicians inquires why lab is flagging his healthy nondiabetic patients as having a high Hemoglobin A1c Reference Interval used by lab was 4.5-5.5 and was adopted from the reagent manufacturer along with diagnostic literature. Pulled data from patient with normal glucose levels(70110) and ran Hgb A1cs
Found OUR local population normal range was 5.5-6.5

You will have to use this information in the real world!
Diagnostic Efficiency
Parameters used to determine how good a given test is at detecting and predicting presence of disease
Sensitivity Analytic: lower limit of detection for given analyte(test can detect 0.1, 0.01, 0.001 etc) Clinical: proportion of people with disease who test positive(rule in the disease) Specificity: proportion without disease who test negative(rule out the disease) Predictive values: positive and negative(False negative results and True Positive results)
Need a balance of sensitivity and specificity.
The Ideal Situation = test that is 100% specific and 100% sensitive, NOT REALITY, patients and disease do not read the book! The Reality = tests can approximately rule out 95%(2 SD) of healthy or sick people.
Labs do not have control of the overlap between healthy and non-healthy patients but does have control over test cutoff. Test cutoff, or medical decision limit, is the analyte concentration that seperates a positive test from a negative test. The best test with the wrong cutoff would be USELESS! To put this in its simplest form: High sensitivity is used for screening tests High specificity is used for confirmation tests Quantitative BHCG as example, cutoff is 5 mIU/mL, which means anything greater 5 is positive.
Test cutoff is 5 Sens = 80% Spec = 70% FN = 2 FP = 3
Cutoff = 8 Sens 40% Spec 80% FN 6 FP 2 Low false positive Specificity is ability of test to rule out disease or condition Confirmation test = more TN!
Cutoff = 2 Sens 90% Spec 40% FN 1 FP 6 Low false negative Sensitivity is ability to detect a disease or condition. Screening test = more TP!
REAL WORLD: Routine hospital labs use sensitive tests(screening)

Hospitals use sensitive tests because of the low False Negatives. In Real World FALSE NEGATIVES are worse than false positives. Example: Patient comes in with chest pain and a Troponin I is performed. If FALSE NEGATIVE = patient does not go to cath lab to see if there is any artery blockage = VERY BAD FOR PATIENT! If FALSE POSITIVE, patient goes to cath lab and they find no blockage BUT patient lives!!
REAL WORLD: Troponin I in perspective

In my lab, adopting new methodology for Troponin I. Current method was immunofluorescence on a point-ofcare meter, with a low end sensitivity of 0.05 ng/mL.
New method is on an immunoassay analyzer but the lower analytical sensitivity or LoD(limit of detection) is 0.3 ng/mL. This does not rule out the 99th percentile for AMI patients(which is the standard for Troponin I assays) Medical Director decided, after looking at CVs, specificity and sensitivity, that the immunoassay (chemiluminescence) was a better methodology despite the LoD.
Were half way throughand now for some Scrubs

http://www.youtube.com/watch?v=hZAgT8KOLF8&featur e=related
Method Evaluation
Regulatory Aspects of Method Evaluation
Center for Medicare and Medicaid Services (CMS) mostly for billing and what tests the lab will get paid for. Food and Drug Administration (FDA) approves all instrumentation and tests for use in the United States. Clinical Laboratory Improvement Amendments (CLIA) defines how method evaluations will be performed. College of American Pathologists (CAP) regulatory agency that provides services and materials to perform method evaluation. Joint Commission on Accreditation of Healthcare Organizations (JCAHO) mostly for nursing.
Method Selection
REAL WORLD: Most labs purchase commercially available instrumentation with methods that have been approved by the FDA. Only research labs develop their own methods and of course manufacturers.
This requires the lab to only validate the method by verifying the manufacturers performance claims. Accuracy via method comparison Precision via repeat testing of QC material
Reportable Range or Analytical Measurement Range(AMR) via the use of linearity material.
Reference range values are verified, adopted from manufacturer or approved diagnostic literature.
Method Evaluation (contd)

First Things First: Determine Imprecision and Inaccuracy
Imprecision: dispersion of repeated measurements about the mean due to analytic error Inaccuracy: difference between a measured value and its true value due to systemic error
Measurement of Imprecision
Estimates random error associated with test method
Detects any problems affecting its reproducibility

Should be performed over a 10- to 20-day period

Graphic representation of (A) imprecision and (B) inaccuracy on a dartboard

Measurement of Inaccuracy
Recovery Studies: show whether a method is able to accurately measure an analyte(use of linearity material that includes 3-7 specimens at varying levels, starting with the lowest concentration ending with the highest concentration) Interference Studies: determine if specific compounds affect accurate determination of analyte concentrations(specimens at different concentrations spiked with different levels of hemolysis, icteria and lipemia) Comparison-of-Methods Studies: examine patient samples by method being evaluated (test) with a reference method(run samples on a currently validated instrument and then on the new instrument and see if results are the same)

Allowable Analytic Error
Methodologies used in past to estimate medically allowable error Physiologic variation
Multiples of reference interval

Pathologist judgment CLIA have published error limits allowable by federally mandated proficiency testing.
REAL WORLD: Data is entered into software with a Total Error Limit to see if values pass within limits.
Real World: Can be done in <5 days as opposed to the 10 20 days recommended by the book.
Real World: Method Evaluations in Real Time

Method evaluations include calibration verification, linearity verification and correlation studies every six months per CAP standards. Method evaluations are required for all new tests on newly acquired instrumentation. Precision studies are required every 6 months. Interfering Substances study required with all new instrumentation implementation. Carryover studies required on all new instruments that do not utilize disposable sample tips.
Quality Control(after method validation)

Involves systematic monitoring of analytic processes to detect analytic errors that occur during analysis and prior to patient values being resulted. Ultimately prevents reporting of incorrect patient test results, when known values are not recovered by the test. Accomplished by assaying stable control materials and comparing determined vs. expected values. Expected values usually span the clinically important range at medical decision levels.
Quality Control (contd)

Quality Control (QC) Control Charts
Graphically represent observed values of a control material over time in context of upper and lower limit controls When values fall within control limits, method has performed adequately. When values fall outside control limits, problems are developing. Control limits expressed as mean +/- standard deviation Random errors are caused by variations in technique. Systemic errors arise from factors that contribute to constant differences between measurements (positive or negative).
+x and x are the 2 SD limits Good distribution of positive and negative control values.
Precicion problem due to too many values in the 3 SD area Accuracy problem due to center(mean) being in the -2 SD area.

Operation of a Quality Control System
Three-stage process: 1. Establishing allowable statistical limits of variation(2x2x10, 2 levels of controls run twice a day for 10 days to give mean and SD) 2. Using limits as criteria for evaluating quality control data(I use the standard 2 SD rule) 3. Taking action to remedy errors when indicated
Finding the causes of error(QC age, temp, reconstitution error, >4 SD = mechanical error)
Reanalyzing control(with same or new control material) Taking corrective action(calibration and not resulting out patients until issue is resolved)
Quality Control contd

In my lab, I have steps for QC that is out of range, they are as follows:
1. Rerun QC using the same control material

2. Rerun QC with new aliquot of control material 3. Calibrate test and rerun BOTH levels of QC
4. Open new vial of QC

5. Check lots of calibrators and check expiration dates of reagent and QC.
6. Is reagent pack nearly empty?

7. Call manufacturers technical support number.
Multi-rules RULE!
Multi-rule procedure developed to further judge whether control results indicate out-of-control situations
Learn the multi-rules, will be on your ASCP test, in the Real World I use Peer data which is my QC values compared to other people using my QC on my instrument, better way of detecting errors.
QC and Peer Data

Monthly reports of QC from lab are submitted to online software for comparison with peers QC data.
A standard deviation summary report is compiled that has my monthly means, my LTD(lot to date) means, montly SDs and LTD SDs. This summary is then compared to my peers monthly and LTD data to produce a SDI(standard deviation index) report. Same 2 SD rule applies.
SDI = [(my lab mean)-(my peers mean)]/peer SD

If I have SDIs >2.0 then I have to reevaluate my QC means and SDs for errors.
Real World: QC ranges

Quality control material is purchased with either ranges or without ranges(assayed or unassayed). Assayed controls are more expensive because the mean and SD data has already been calculated. A simple parallel study is performed to validate the range. Unassayed controls are cheaper because the lab has to determine its own means and SDs, which is sometimes better because data can vary even when instrument specific.

Proficiency Testing
A method used to validate a particular measurement process Required by CAP, CLIA, JCAHO to maintain lab accreditation
Most clinical labs use proficiency program provided by CAP.

Process A series of unknown samples is sent to lab from program. Samples are analyzed in same manner as patient specimens. Results are reported to CAP and compiled with results from all other labs participating in survey. Performance report is sent back to each participating lab.

Proficiency Testing
Strict requirements 1. Lab must incorporate proficiency testing into its routine workflow as much as possible. 2. Test values/samples must not be shared with other labs at any time during testing cycle. 3. Proficiency samples are tested by bench technical staff who normally do patient testing.
4. Testing should be completed within usual time.
CAP Surveys: Real World

Report sent back from CAP is peer oriented, which means my results are only compared to those labs using MY instruments.
Very good way of monitoring test performance. SDIs are used by CAP just like QC peer data. Acceptable limits usually 3 SDs. Results are graded, examples of CAP grades can include acceptable, unacceptable, imprecise, not linear and different. Any grade other than Acceptable must be investigated with a full report as to the cause of the problem.
Real World: CAP failures

My urine microalbumin proficiency testing results keep coming out consistently high(but are linear).
Used new reagent and new calibrator. Checked expiration dates.

No matter what I did, results came out the same.
Not in the 2 SD range but in the 3 SD range.

QC peer data passes. Conclusion: Sometimes one test runs slightly higher than peers. Dont worry, increase is not clinically significant! Statistically, values are variant, but are clinically the same. Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins
Quality Management
Quality Improvement: Lean Six Sigma
Methodology: waste elimination and variation reduction
Current and desired process Red segments indicate failures, or waste Waste are unnecessary steps that take up time and cause longer turn around times(which no lab wants!)
Quality Management (contd)

Improvement MethodologyDMAIC
(D) Define: end-users of services or products, their needs & expectations, project boundaries, process (M) Measure: collect data, determine defects, assess satisfaction (A) Analyze: examine data to identify root causes of error (I) Improve: fix problems and prevent future ones with creative solutions
(C) Control: continuous monitoring of new plans
Quality Management (contd)

MetricsLean and Six Sigma
Lean: eliminate non-value-adding steps and reduce cycle time Six Sigma: reduce variation and error
Perspective: Patient Safety, Lean Six Sigma, and the Laboratory

Medical errors are much more frequent than previously thought. Clinical labs have adopted Six Sigma and Lean processes to improve quality of lab testing.
Practical Application of Six Sigma Metrics

REAL WORLD: LEAN team is at my hospital right now trying to improve TATs due to increased work load!
In a perfect world, it would work just like this But in the real world you can only get close to this goal Improving Quality is an on-going process and never ends There can always be room for improvement!
Real World: Why is this important?...

Several months ago a pharmacist was discussing with a peer group the performance of his students in a class dealing with pharmaceutics(drugs). The class had just taken a mid-term exam, and the instructor mentioned that the class average was 76%.
His face showed concern. A professor within the peer group smiled, commenting, Thats not a bad average for a cumulutive exam. You should be pleased with that result, you teach a difficult course. He responded, Its not the 76% that worries me. Its the 24% that represents lack of knowledge about administration of drugs that troubles me.
End Note: Know your stuff!

Remember it could be one of us or someone we know that is getting administered drugs from people that dont pass with a B average! In our case, we are the ones providing lab results to doctors that will ultimately determine care, medicine and surgical procedures!! You must be knowledgeable of your field to provide good patient care!

Chapter 4 PPT Updated

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter 4 PPT Updated

Uploaded by

Copyright:

Available Formats

Chapter 4: Method Evaluation and Quality Management

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Let me hit you with some knowledge!

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

REAL WORLD: Why is method evaluation and quality management important?

REAL WORLD: what do we normally use?

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

REAL WORLD: Calculations!

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Basic Concepts (contd)

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Reference Interval Studies

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Reference Interval Studies (contd)

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

How do we get normal ranges?

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Real World Situation: True Story

Found OUR local population normal range was 5.5-6.5

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Need a balance of sensitivity and specificity.

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Test cutoff is 5 Sens = 80% Spec = 70% FN = 2 FP = 3

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

REAL WORLD: Routine hospital labs use sensitive tests(screening)

REAL WORLD: Troponin I in perspective

Were half way throughand now for some Scrubs

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Method Evaluation (contd)

Detects any problems affecting its reproducibility

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Method Evaluation (contd)

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Method Evaluation (contd)

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Method Evaluation (contd)

Multiples of reference interval

Real World: Method Evaluations in Real Time

Quality Control(after method validation)

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Quality Control (contd)

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Quality Control (contd)

Quality Control contd

1. Rerun QC using the same control material

4. Open new vial of QC

6. Is reagent pack nearly empty?

QC and Peer Data

SDI = [(my lab mean)-(my peers mean)]/peer SD

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Real World: QC ranges

Copyright 2010 Wolters Kluwer Health | Lippincott Williams & Wilkins

Quality Control (contd)