Appraising Diagnostic Study

Muhammad Ali Medical School University of Sumatera Utara

What are tests used for?

Log of reasons by several docs: Diagnosis most common but also Monitoring has it changed? Prognosis risk/stage within Dx Treatment planning

Basic principles (1)

Ideal diagnostic tests right answers: (+) results in everyone with the disease and ( - ) results in everyone else Usual clinical practice: The test be studied in the same way it would be used in the clinical setting Observational study, and consists of: Predictor variable (test result) Outcome variable (presence / absence of the disease)

Basic principles (2)

Sensitivity, specificity Prevalence, prior probability, predictive values Likelihood ratios Dichotomous scale, cutoff points (continuous scale) Positive (true and false), negative (true and false) ROC (receiver operator characteristic) curve

EBM Issues:
Is this evidence about the accuracy of a diagnostic test valid? Does this valid evidence demonstrate an important ability to accurately distinguish patients who do and dont have a specific disorder? Can I apply this valid, important diagnostic test to a specific patient?

Critical appraisal
Use worksheet Use supporting software
CAT Maker Save in CAT Banks

Is this evidence about a diagnostic test valid?

Was there an independent, blind comparison with a reference (gold) standard of diagnosis? Was the diagnostic test evaluated in an appropriate spectrum of patients (like those in whom we would use it in practice)? Was the reference standard applied regardless of the diagnostic test result?

Was the test (or cluster of tests) validated in a second, independent group of patients?

Does this valid evidence demonstrate an important ability to accurately distinguish patients who do and dont have a specific disorder?

Sensitivity Specificity Likelihood ratios

 SnNout
Diagnostic test with a very high sensitivity , a negative result effectively rules out the diagnosis

SpPin
Diagnostic test with a very high specificity , a positive result effectively rules in the diagnosis

Questions to answer in applying a valid diagnostic test to an individual patient Is the diagnostic test available, affordable, accurate, and precise in our setting? Can we generate a clinically sensible estimate of our patients pre-test probability? From personal experience, prevalence statistics, practice databases, or primary studies Are the study patients similar to our own? Is it unlikely that the disease possibilities or probabilities have changed since this evidence was gathered?

Guides for deciding whether a screening or early diagnostic maneuver does more good than harm:

Does early diagnosis really lead to improved survival, or quality of life, or both? Are the early diagnosed patients willing partners in the treatment strategy? Is the time and energy it will take us to confirm the diagnosis and provide (lifelong) care well spent? Do the frequency and severity of the target disorder warrant this degree of effort and expenditure?

Diagnostic tests Is not about finding absolute truth, but about limiting uncertainty establishes both the necessity and the logical base for introducing probabilities, pragmatic test-treatment thresholds ..
Start thinking about what youre going to do with the results of the diagnostic test, and whether doing the test will help your patients

DISEASE
Yes Yes No Total
a+b c+d

3
a b c D d

7 89 96
b+d

10 90 100

TEST

No Total

1 4
a+c

a+b+c+d

TRADITIONAL 2x2 TABLE

SENSITIVITY
DISEASE
Yes Yes No Total
a+b

3
a b c D d

7 89 96
b+d

10 90 100

FALSE NEGATIVES

TEST

No Total

1 4
a+c

a+b+c+d

SENSITIVITY
The proportion of people with the diagnosis (n=4) who are correctly identified (n=3) Sensitivity = a/(a+c) = 3/4 = 75%

c+d

SPECIFICITY
DISEASE
Yes Yes No Total
a+b

FALSE POSITIVES

3
a b c D d

7 89 96
b+d

10 90 100

TEST

No Total

1 4
a+c

a+b+c+d

SPECIFICITY
The proportion of people without the diagnosis (n=96) who are correctly identified (n=89) Specificity = d/(b+d) = 89/96 = 93%

c+d

PREDICTIVE VALUES (POS or NEG)

DISEASE
Yes Yes No Total
a+b c+d

3
a b c D d

7 89 96
b+d

10 90 100

TEST

No Total

1 4
a+c

a+b+c+d

Positife or Negative predictive values are the proportion of disease / non disease people to all who have positive/negative tests.

PPV = 3/10 = 30% NPV = 89/90=89%

PRE-TEST ODDS
DISEASE
Yes Yes No Total
a+b c+d

3
a b c D d

7 89 96
b+d

10 90 100

TEST

No Total

1 4
a+c

a+b+c+d

In the sample as a whole, the odds of having the disease are 4 to 96 or 4% (the PRE-TEST ODDS)

POST-TEST ODDS
DISEASE
Yes Yes No Total
a+b c+d

3
a b c D d

7 89 96
b+d

10 90 100

TEST

No Total

1 4
a+c

a+b+c+d

In the sample as a whole, the odds of having the disease are 4 to 96 or 4% (the PRE-TEST ODDS) In those who score positive on the test, the odds of having the disease are 3 to 7 or 43% (the POST-TEST ODDS)

POST-TEST ODDS
DISEASE
Yes Yes No Total
a+b c+d

3
a b c D d

7 89 96
b+d

10 90 100

TEST

No Total

1 4
a+c

a+b+c+d

In the sample as a whole, the odds of having the disease are 4 to 96 or 4% (the PRE-TEST ODDS)

In those who score positive on the test, the odds of having the disease are 3 to 7 or 43% (the POST-TEST ODDS)
In those who score negative on the test, the odds of having the disease are 1 to 89 or approximately 1%

DIAGNOSTIC ODDS RATIO

DISEASE
Yes Yes No Total
a+b c+d

3
a b c D d

7 89 96
b+d

10 90 100

Potentially useful as an overall summary measure, but only in conjunction with other measures (LR, sensitivity, specificity)

TEST

No Total

1 4
a+c

DOR

3 1

a+b+c+d

The Diagnostic Odds Ratio is the ratio of odds of having the diagnosis given a positive test to those of having the diagnosis given a negative test

89 0.429 38.2 0.011

BAYES THEOREM

POST-TEST ODDS = LIKELIHOOD RATIO x PRE-TEST ODDS

Interpreting Sensitivity & Specificity

Using natural frequencies to calculate Bayes theorem

Natural frequencies provide a more graphic, easy to understand way to portray probabilities for both physicians and patients

In small groups what is the likelihood that this patient has the disease?
A disease with a prevalence of 3% must be diagnosed There is a test for the disease It has a sensitivity of 50% and a specificity of 90%

For a prevalence of 3% - in every 100 patients

3 will have the disease

97

Will not have the disease

For a test with a sensitivity of 50%

For every 100 patients with the disease 1-2 will test positive

97

How many will not have the disease?

For a test with a specificity of 90%

False positive rate = 1-Specificity (10%)

87
For those who dont have the disease 10 patients will go on to test positive

Given a positive test what are my chances of disease doc?

11 people have a positive test Of which 1 have will have the disease

87
13% chance of having the disease given a positive test

Given a negative test what are my chances of not having the disease doc?

88 people will have a negative test of which 1 will have the disease False reassurance rate of 1.7%

87

Describe the result to your patient

For every 100 people who have a positive test 13 will have the disease

What happens if I have an negative test

Negative test pretty much rules out the disease but a small number 1-2 people in every 100 will still have the disease

Try it again what is the likelihood that this patient has the disease?

A disease with a prevalence of 30% must be diagnosed There is a test for this disease It has a sensitivity of 50% and a specificity of 90%

Prevalence of 30% Sensitivity of 50% Specificity of 90%

Disease +ve

30 100 Disease -ve

70

15

22 positive tests in total of which 15 have the disease

About 70%

Likelihood Ratio Sensitivity/1-specificity 50% / 10% =5

LR a b

Likelihood Ratio

LR a b
30 100
70 Prevalence of 30% Sensitivity of 50% Specificity of 90% 5 150

70

Now consider the FOB screening tests

You find out that your father has undertaken the test and has a positive result He ask you whether he has cancer?

Prevalence of disease is 0.3% must be diagnosed. Sensitivity of 50% False positive rate 3%.

APPLYING A DIAGNOSTIC TEST IN DIFFERENT SETTINGS

The Positive Predictive Value of a test will vary (according to the prevalence of the condition in the chosen setting)
Sensitivity and Specificity are usually considered properties of the test rather than the setting, and are therefore usually considered to remain constant However, sensitivity and specificity are likely to be influenced by complexity of differential diagnoses and a multitude of other factors

THANK YOU