20141110-Detty-IKK-EBM-Systematic Review and Meta-Analysis PDF

EVIDENCE-BASED MEDICINE WORKSHOP
SYSTEMATIC REVIEW
AND META-ANALYSIS
Dr. Detty S Nurdiati, MPH, PhD, SpOG(K)1,2
1Clinical
Epidemiology and Biostatistics Unit

2Dept of Obstetrics & Gynecology
Faculty of Medicine, Universitas Gadjah Mada
Yogyakarta, Indonesia
The 6th Board Meeting and Scientific Meeting of ICE-EBM Network

Faculty of Medicine, Universitas Maranatha, Bandung, December 13-15, 2013
Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia
What do you do?

For an patient with a painful sore throat, you wonder
whether corticosteroids will help with pain relief?
You do a search and find several studies:
some suggest that steroids reduce pain; some do not
What do you do?

Ask a consultant? Peer? Patient?
Ask research student to find all studies & select the
best?
How do you know which study to believe?

Mickan S. 2013. Systematic Reviews
Levels of Evidence,
in Context
Anderson, P.F. (2006).

Chain of Trust / Level of Evidence Vertical.
Available at: http://www-personal.umich.edu/~pfa/
pro/courses/ChainOfTrustLoEVert2.pdf.
Top pyramid is from:
Medical Research Library of Brooklyn.
Guide to Research Methods, The Evidence Pyramid. Available
at: http://library.downstate.edu/ebm/2100.htm
Anderson PF. 2011. Systematic Reviews: Context & Methodology
You find this review
How confident are you of the evidence?
Why systematic reviews?

efficient way to access the body of research
saves time required for searching
critical appraisal
interpretation of results
explore differences between studies

reliable basis for decision making
unbiased selection of relevant information
useful for health care, policy, future research
SYSTEMATIC REVIEW?
A review of a clearly formulated
question that uses systematic and
explicit methods to identify,
select and critically appraise
relevant research, and to collect
and analyse data from the studies
that are included in the review.
Narrative vs Systematic Review

Narrative
Many questions
No search methods
No inclusion criteria
No combining studies
Prone to random and
systematic error
Provide conflicting
summaries
Systematic
One question
Explicit search, reproducible
Explicit inclusion criteria
Combine study results
(with/without meta-analysis)
WHY do we need Systematic Reviews?

Purpose of systematic reviews

Provide up to date summary of all published
research literature
Allow large amounts of data to be assimilated
Provide an objective collation of results of
research
Provide reliable recommendations
Useful Resources
The Cochrane Collaboration www.thecochranelibrary.com/
Cochrane Handbook for Systematic Reviews of
Interventions (version 5 updated March 2011)
CRD www.crd.york.ac.uk/
The Centre for Reviews and Dissemination is a
department of the University of York and is part of
the National Institute for Health Research
EPPI-Centre www.eppi.ioe.ac.uk/
The Evidence for Policy and Practice Information and
Co-ordinating Centre, Social Science Research Unit,
Institute of Education, University of London.
Key features of a systematic review
clearly stated objectives

pre-defined eligibility criteria
explicit, reproducible methodology
systematic search
assessment of validity of included studies
systematic synthesis and presentation of
findings
The review team

review must be undertaken by more than one person
allows double-checking
eligibility of included studies
data collection and entry
risk of bias assessment
different areas of expertise

clinical (multidisciplinary)
systematic review methods (including statistics)
user perspective (consumer, professional, settings)
consider establishing an advisory group
Steps of a systematic review

1.
2.
3.
4.
5.
6.
7.
8.
Clear answerable question

Reproducible search strategy
Assessment of literature quality
Summary of the evidence
Statistical, sensitivity analyses
Interpretation
Conclusions, recommendations
Published protocol and review
Types of systematic review

Different research questions require different
study designs generate different types of
review
Variations occur in
Research questions asked
Primary study designs included
Methods for synthesis
Approaches to being systematic
Types of evidence included
Best evidence for different questions

Treatment
Prognosis
Particular
perspective
Systematic
Review of
Systematic
Review of
Systematic
Review of
Randomised
trials
Inception
Cohorts
Qualitative
studies
The most type of

SR published
Best evidence for different questions

Treatment
Prognosis
Particular
perspective
Systematic
Review of
Systematic
Review of
Systematic
Review of
Randomised
trials
Inception
Cohorts
Qualitative
studies
Risk of bias
use risk of bias
assessment
for studies that provided
data for sleep disruption:
what is the overall risk of
bias of these studies
does the risk of bias
reduce our confidence in
the effect?
Study level
Review level
Study A
Outcome data
Effect measure
Study B
Outcome data
Effect measure
Effect measure
Study C
Outcome data
Effect measure
Study D
Outcome data
Effect measure
Source: Jo McKenzie & Miranda Cumpston
What is a meta-analysis?
combines the results from two or more studies
estimates an average or common effect
optional part of a systematic review
Systematic
reviews
Source: Julian Higgins
Metaanalyses
Why perform a meta-analysis?

quantify treatment effects and their
uncertainty
increase power
increase precision
explore differences between studies
settle controversies from conflicting studies
generate new hypotheses
When not to do a meta-analysis

mixing apples with oranges
each included study must address same question
consider comparison and outcomes
requires your subjective judgement
combining a broad mix of studies answers broad

questions
answer may be meaningless and genuine effects
may be obscured if studies are too diverse
When not to do a meta-analysis

garbage in garbage out
a meta-analysis is only as good as the studies in it
if included studies are biased:
meta-analysis result will also be incorrect
will give more credibility and narrower confidence
interval
if serious reporting biases present:

unrepresentative set of studies may give misleading
result
When can you do a meta-analysis?

more than one study has measured an effect
the studies are sufficiently similar to produce
a meaningful and useful result
the outcome has been measured in similar
ways
data are available in a format we can use
Steps in a meta-analysis
identify comparisons to be made
identify outcomes to be reported and
statistics to be used
collect data from each relevant study
combine the results to obtain the summary
of effect
explore differences between the studies
interpret the results
Selecting comparisons
Hypothetical review: Caffeine for daytime drowsiness
caffeinated coffee vs
decaffeinated coffee
break your topic down into pair-wise comparisons

each review may have one or many
use your judgement to decide what to group
together, and what should be a separate
comparison
Selecting outcomes & effect measures

Hypothetical review: Caffeine for daytime drowsiness
caffeinated coffee vs
decaffeinated coffee
asleep at end of trial (RR)

irritability (MD/SMD)
headaches (RR)
for each comparison, select outcomes

for each outcome, select an effect measure
may depend on the available data from included

studies
Calculating the summary result

collect a summary statistic from each
contributing study
how do we bring them together?
treat as one big study add intervention &
control data?
breaks randomisation, will give the wrong answer
simple average?
weights all studies equally some studies closer to
the truth
weighted average
Weighting studies
more weight to the studies which give more
information
more participants, more events, narrower
confidence interval
calculated using the effect estimate and its variance
1
1
weight
2
varianceof estimate SE
inverse-variance method:
sum of (estimate weight)
pooled estimate
sum of weights
For example
Headache
Caffeine
Decaf
Amore-Coffea 2000
2/31
10/34
Deliciozza 2004
10/40
9/40
Mama-Kaffa 1999
12/53
9/61
Morrocona 1998
3/15
1/17
Norscafe 1998
19/68
9/64
Oohlahlazza 1998
4/35
2/37
Piazza-Allerta 2003
8/35
6/37
Weight
For example
Headache
Caffeine
Decaf
Weight
Amore-Coffea 2000
2/31
10/34
6.6%
Deliciozza 2004
10/40
9/40
21.9%
Mama-Kaffa 1999
12/53
9/61
22.2%
Morrocona 1998
3/15
1/17
2.9%
Norscafe 1998
19/68
9/64
26.4%
Oohlahlazza 1998
4/35
2/37
5.1%
Piazza-Allerta 2003
8/35
6/37
14.9%
Meta-analysis options
for dichotomous or continuous data
inverse-variance
straightforward, general method
for dichotomous data only

Mantel-Haenszel (default)
good with few events common in Cochrane reviews
weighting system depends on effect measure
Peto
for odds ratios only
good with few events and small effect sizes (OR close to
1)
A forest of lines
Trees Joyce Kilmer Forest by charlescleonard http://www.flickr.com/photos/charlescleonard/3754931947/
Forest plots
Headache at 24 hours
headings explain the comparison
Forest plots
list of included studies
Forest plots
raw data for each study
Forest plots
total data for all studies
Forest plots
weight given to each study
Forest plots
effect estimate for each study, with CI
Forest plots
effect estimate for each study, with CI
Forest plots
scale and direction of benefit
Forest plots
pooled effect estimate for all studies, with CI
Interpreting confidence intervals

always present estimate with a confidence interval
precision
point estimate is the best guess of the effect
CI expresses uncertainty range of values we can be
reasonably sure includes the true effect
significance
if the CI includes the null value
rarely means evidence of no effect
effect cannot be confirmed or refuted by the available evidence
consider what level of change is clinically important
Heterogeneity
consider heterogeneity in the meta- analysis
overlap in confidence intervals
I2 statistic
is there unexplained statistical heterogeneity?
Identifying heterogeneity
visual inspection of the forest plots
chi-squared (c2) test (Q test)
I2 statistic to quantify heterogeneity
Visual inspection
Forest plot A
Forest plot B
The chi-squared
2
(c )
test
tests the null hypothesis of homogeneity

low power with few studies
may detect clinically unimportant differences with
many studies
narrow question (yes/no) not useful if
heterogeneity is inevitable
calculated automatically by RevMan
The
2
I
statistic
I2 statistic describes the percentage of

variability due to heterogeneity rather than
chance (0% to 100%)
low values indicate no, or little, heterogeneity
high values indicate a lot of heterogeneity
calculated automatically by RevMan

be cautious in interpreting
What to do about heterogeneity

check that the data are correct
if heterogeneity is very high
interpret fixed-effect results with caution
consider sensitivity analysis would random-effects have
made an important difference?
may choose not to meta-analyse

average result may be meaningless in practice
consider clinical & methodological comparability of studies
avoid
changing your effect measure or analysis model
excluding outlying studies
explore heterogeneity
Reporting biases
consider whether the outcome is at risk
no large studies
industry sponsorship
few of your studies report this outcome
consider funnel plot results or statistical tests for
small study effects, if appropriate
External validity
how well, and how
completely, do the studies
address your review
question?
compare your inclusion
criteria to the Characteristics
of included studies table
PICO
The Results section of your review

a systematic, narrative summary of results
forest plots
key forest plots linked as figures
usually primary outcomes
all forest plots will be published as supplementary data

avoid forest plots with only one study
may also add other data tables

results of single studies
summary data for each group, effect estimates, confidence intervals
non-standard data
not helpful to report trivial outcomes or results at high risk of bias
RAPID CRITICAL APPRAISAL OF

SYSTEMATIC REVIEW
Projects Worked On
Rapid Critical Appraisal of SR
Step 1
What
question
did the
study ask?
Step 2
How well
was the
study
done?
Time Spent
Step 3
What do the
results
mean?
Step 4
How do the
results
apply to the
care of my
patients
Step 1
What question did the study ask?
P
I
C
O
:...................
:...................
:...................
:...................
Step 2
How well was the study done?
... Internal Validity...
QUESTION
Does the SR address a focused
question?
... And use it to direct the search
and select articles for inclusion?
Getting started
KEY = systematic, rigorous, transparent, reproducible
Define the research question
Clear background, scope, setting
Research question determines method of review (PICO)
Specify inclusion and exclusion criteria
FIND
Did the search find all the
relevant evidence?
Find the published research

Clear, comprehensive, reproducible search strategy
Search terms
Databases
Other strategies for grey literature
Manage the research evidence

Organise database, hand searching
Use of forward citation searching, reference lists
Manage references
Reference Management software eg Endnote
Screen studies to check fit

2 reviewers, process of agreement
Record decisions about whether studies meet
criteria
APPRAISE
Have the studies been critically
appraised?
... And was the overall quality

adequate?
Assess quality of the literature
Dual, independent assessment of design aspects

likely to cause bias depends on study designs
Resource: online www.cochrane.org/training/cochranehandbook or www.equator-network.org/home/
The Cochrane risk of bias tool

Risk of bias
Interpretation
Within a study
Across studies
Low risk of bias
Plausible bias unlikely to

seriously alter the
results.
Low risk of bias for all

key domains.
Most information is from

studies at low
risk of bias.
Unclear risk of bias
Plausible bias that raises

some doubt about the
results
Unclear risk of bias for

one or more key
domains.
Most information is from

studies at low or unclear
risk of bias.
High risk of bias
Plausible bias that

seriously weakens
confidence in the results.
High risk of bias for one

or more key
Domains.
The proportion of
information from studies
at high risk of bias is
sufficient to affect the
interpretation of the
results.
A visual representation - RCTs
Describe included studies

Design data extraction forms
General descriptive information
Research methods
Key results
2 reviewers, process of agreement
SYNTHESISE
Have the results been synthesised
with appropriate tables and plots?
... And were the results similar
between the studies?
Decide on process of synthesis

Factors to consider
Consistency of outcome measures
Sub groups
Heterogeneity
Common sense test
Details of data synthesis

Look for consistent measurement of data,
with 95% confidence intervals
Primary outcome/s
Basis for meta-analysis
Sub group analysis

Identify in protocol with justification
To enhance usefulness of research answers
Heterogeneity
Common sense test of study design, outcome
measurements, forest plot
Are syntheses meaningful (apples vs oranges)
Influences statistics within meta-analysis
Sensitivity analyses
determine whether the assumptions or decisions
made have a major effect on the results of the
review.
Step 3
What do the results mean?

What measure was used, how
large was the effect (could it
have been due to chance)?
Other comments?
Step 4
How do the results apply to the care

of my patients?
Is the review any good FAITH?

FINDING
Did they find most studies?
APPRAISAL
Did they use appropriate inclusion criteria?
INCLUDE
Did they include valid studies for question asked?
TOTAL UP
Did they synthesise similar outcomes?
HETEROGENEITY
The Cochrane Collaboration

International non-profit organisation that
prepares, maintains, and disseminates
systematic up-to-date reviews of
health care interventions

20141110-Detty-IKK-EBM-Systematic Review and Meta-Analysis PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

20141110-Detty-IKK-EBM-Systematic Review and Meta-Analysis PDF

Uploaded by

Copyright:

Available Formats

EVIDENCE-BASED MEDICINE WORKSHOP

Epidemiology and Biostatistics Unit

The 6th Board Meeting and Scientific Meeting of ICE-EBM Network

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

What do you do?

What do you do?

How do you know which study to believe?

Anderson, P.F. (2006).

Anderson PF. 2011. Systematic Reviews: Context & Methodology

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

You find this review

Mickan S. 2013. Systematic Reviews

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Mickan S. 2013. Systematic Reviews

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

How confident are you of the evidence?

Mickan S. 2013. Systematic Reviews

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Why systematic reviews?

explore differences between studies

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Narrative vs Systematic Review

WHY do we need Systematic Reviews?

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Purpose of systematic reviews

Mickan S. 2013. Systematic Reviews

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Key features of a systematic review

clearly stated objectives

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

The review team

different areas of expertise

consider establishing an advisory group

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Steps of a systematic review

Clear answerable question

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Types of systematic review

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Best evidence for different questions

Mickan S. 2013. Systematic Reviews

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

The most type of

Best evidence for different questions

Mickan S. 2013. Systematic Reviews

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Source: Jo McKenzie & Miranda Cumpston

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Source: Julian Higgins

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

Why perform a meta-analysis?

Source: Julian Higgins

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

When not to do a meta-analysis

combining a broad mix of studies answers broad

Source: Julian Higgins

Faculty of Medicine, Universitas Gadjah Mada, Yogyakarta, Indonesia

When not to do a meta-analysis

if serious reporting biases present:

Source: Julian Higgins