Critical Appraisal by John Vincent Pagaddu, M.D.

CRITICAL APPRAISAL
By: John Vincent A. Pagaddu, M.D.

Preface
I made this presentation to help us out and guide
us in doing our critical appraisal during
Residency.
I incorporated in this presentation the key
concepts of Critical appraisal, and I also included
the tools you can use for your presentation.
I hope that the things I have compiled in this
presentation could serve as a tool to ease you up
when critically appraising a Journal.
I wish you all the best guys! :D
- J.V.
Whats in it?
Critical Appraisal of a Journal Article
Critical Appraisal tools:
Critical Appraisal for Diagnostic Test Study
Critical Appraisal for Therapeutic Study
Critical Appraisal for Randomized Controlled Trial
(RCT)
Critical Appraisal for Systematic Review
Presenting a Journal
CRITICAL APPRAISAL OF
A JOURNAL ARTICLE
Sources:
UCL (2011) Retrieved from https://www.ucl.ac.uk/ich/support-
services/library/training-material/critical-appraisal
Acad Radiol 2012; 19:872888
Center for Evidence-based management
Introduction to critical appraisal
Critical appraisal is the process of carefully and
systematically examining research to judge its
trustworthiness, and its value and relevance in a
particular context. (Burls 2009)
Critical appraisal is an important element of
evidence-based medicine (EBM).
The 5 steps of evidence-based medicine
1. Asking a focused and answerable question that
translates uncertainty to an answerable question
2. Searching for the best available evidence
3. Critically appraising the evidence for validity
and clinical relevance
4. Applying the results to clinical practice
5. Evaluation of performance
Sources of Evidence
Primary literature is the first level of the evidence pyramid
(studies) and consists of original articles (eg, randomized
controlled trials, cohort, case-control, cross-sectional
studies) with variable quality published in journals.
These studies can found by searching medical search
engines including:
PubMed most used electronic database
MEDLINE
EMBASE
ISI Web of Knowledge
MD Consult
Google Scholar
Hierarchy of Evidence
How to read a research article?
Structure of an article:
Title
Abstract
Introduction
Background / review of literature
Research Setting
Methodology
Results
Discussion
Title
Not always a good indication of the content of the
article
Abstract
Sometimes unclear.
What should be in it: a summary of the research

question, key methods, results and conclusions of the
study.
Research Setting
Methodology
Should describe exactly how the research was carried
out.
Sample: characteristics, selection, number, non-response
Measures: description of tests / questionnaires (validated?), data,

outcome measures
Procedure: study design (qualitative, quantitative, controlled?)

Statistical Measures for Diagnostic and
Therapeutic Literature
Statistical Measures for Diagnostic and
Therapeutic Literature
Result
Should tell the reader what the findings were.
All outcome measures must be reported and confidence

intervals for effect sizes should be presented.
Discussion
Interpretation of the results / relation to theory
Comparison with the results of other studies
Weaknesses / limitations of the study
Implications
Recommendations
Study Design in Medical Literature
QUALITATIVE STUDIES explore and understand people's beliefs,
experiences, attitudes, behaviour and interactions. They generate
non-numerical data.
Document - study of documentary accounts of events, such as
meetings;
Passive observation - systematic watching of behaviour and talk
in natural occurring settings;
Participant observation - observation in which the researcher also
occupies a role or part in the setting, in addition to observing;
In depth interview - face to face conversation with the purpose of
exploring issues or topics in detail. Does not use preset questions, but
is shaped by a defined set of topics;
Focus group - method of group interview which explicitly includes
and uses the group interaction to generate data.
QUANTITATIVE STUDIES generate numerical data or data that can
be converted into numbers.
Case report - report on a single patient;
Case series - report on a series of patients (no control group);
Cross-sectional survey - the observation of a defined population
at a single point in time or time interval. Exposure and outcome are
determined simultaneously. Gold standard in diagnosis and screening
research;
Case control study - identifies patients with a particular outcome
(cases) and control patients without the outcome. Looks back and
explores exposures and possible links to outcome. Very useful in
causation research;
Cohort study - identifies two groups (cohorts) of patients one
which received the exposure of interest, and one which did not.
Follows these cohorts forward for the outcome of interest. Very useful
in causation as well as prognosis research.
Key quantitative studies:
Randomized Controlled Trial (RCT) - a clinical trial in which
participants are randomly allocated to a test treatment and a control;
involves concurrent enrolment and follow-up of both groups; gold
standard in testing the efficacy of an intervention (therapy/prevention);
Systematic review - identifies and critically appraises all research
on a specific topic, and combines valid studies; increasingly important
in evidence based medicine; different from review article (which is a
summary of more than one paper on a specific topic, and which may
or may not be comprehensive);
Meta-analysis - a systematic review that uses quantitative
methods to summarize the results
APPRAISING THE EVIDENCE
1. What is the research question?
What is the scientific rationale?
Is there a pre-defined hypothesis?
Where was the study conducted?
What was the study population?
What intervention(s) were compared?
What outcomes were considered?
2. Is the study ethical?
Was there a real need to conduct the trial?
Was the study approved by an ethics committee?
Did patients give informed consent?
Was the trial of sufficient size to reach a meaningful
conclusion?
3. Is the study valid?
The first step is to decide whether the study was
unbiased by evaluating its methodological quality.
Different criteria for validity of articles are used for
different types of questions on: treatment, diagnosis,
prognosis and economic evaluation.
Depending on the validity of an article we can classify it
within a scale of levels of evidence and degrees of
recommendation.
4. What are the results?
If we decide that the study is valid, we can go on to look
at the results.
At this step we consider whether the studys results are
clinically important.
For example, did the experimental group show a
significantly better outcome compared with the control
group?
We also consider how much uncertainty there is about
the results, as expressed in the form of p values,
confidence intervals and sensitivity analysis.
5. Are the results useful?
Once you have decided that your evidence is valid and
important, you need to think about how it applies to your
question.
It is likely, for example, that your patient or population
may have different characteristics to those in the study.
Critical appraisal of different study designs
Randomized Controlled Trials
Systematic Review
Randomized Controlled Trials (RCTs)
Critical Appraisal of RCTs:
Factors to look for:
allocation (randomisation, stratification, confounders);
blinding;
follow up of participants (intention to treat);
data collection (bias);
sample size (power calculation);
presentation of results (clear, precise);
applicability to local population.
Mechanisms to control bias in RCTs
Randomization indicates that participants are randomly
allocated to treatment or control group.
Acceptable methods of randomization include random
numbers, either from tables or computer-generated.
Unacceptable methods include last digit of date of birth,
date seen in clinic etc.
Stratified randomization is often used to avoid
confounding factors, i.e. to ensure equal distribution of
participants with a characteristic thought to affect
prognosis or response.
Mechanisms to control bias in RCTs
Blinding means masking who is getting treatment and
control.
Single blinding: participants do not know.
Double blinding: neither the participants nor those
giving the intervention know.
Triple blinding: statisticians doing the analysis also do
not know.
The sources of bias in RCTs
Presenting the results of RCTs
p-value - the p-value refers to the probability that any
particular outcome would have arisen by chance. A p-
value of less than 1 in 20 (p<0.05) is statistically
significant.
Presenting the results of RCTs
Confidence interval - the same trial repeated hundreds
of times would not yield the same results every time. But
on average the results would be within a certain range. A
95% confidence interval means that there is a 95%
chance that the true size of effect will lie within this range.
Systematic Reviews
Critical appraisal of systematic reviews
Factors to look for:
literature search (did it include published and unpublished
materials as well as non-English language studies? Was personal
contact with experts sought?);
quality-control of studies included (type of study; scoring system
used to rate studies; analysis performed by at least two experts);
homogeneity of studies;
presentation of results (clear, precise);
applicability to local population
Mechanisms to control bias in systematic
reviews
To avoid bias, systematic reviews must:
contain a statement of objectives, materials and
methods;
follow an explicit and reproducible methodology.
In a systematic review, all the primary studies identified are

critically appraised and only the best ones are selected. A
meta-analysis (i.e. a statistical analysis) of the results from
selected studies may be included.
Blobbogram/Forrest plot
A blobbogram or forest plot is a graphical display used to
present the result of a meta-analysis
Selected studies must be tested for homogeneity, which
should be >50%. A quick way to check for homogeneity is to
look at the confidence intervals for each study - if they dont
overlap, the studies are likely to be heterogeneous.
If studies are homogeneous, a fixed-effect model is normally
used in the meta-analysis. This means that results are only
interpreted within the populations/samples in the included
studies.
If studies are heterogeneous, a random-effects model is
used. This means that results are interpreted across the wider
population. A different underlying effect is assumed for each
study and an additional source of variation is added to the
model.
CRITICAL APPRAISAL
TOOLS
Source: The Joanna Briggs Institute (JBI)
Critical Appraisal Skills Programme (CASP)
College of Family Physicians of Canada (Pearls for
Residents)
Critical Appraisal for
Diagnostic Test Study
Source: Critical Appraisal Skills Programme (CASP)
(A) Are the results of the study valid?
1. Was there a clear question for the study?
Yes Cant tell No
HINT: A question should include information about

The population
The test
The setting
The outcomes
2. Was there a comparison with an appropriate
reference standard?
Yes Cant tell No
HINT: Is this reference test(s) the best available indicator in

the circumstances?
3. Did all patients get the diagnostic test and
reference standard?
Yes Cant tell No
HINT: Consider
Were both received regardless of the results of
the test of interest
Check the 2X2 table (verification bias)
4. Could the results of the test have been
influenced by the results of the reference
standard?
Yes Cant tell No
HINT: Consider
Was there blinding?
Were the tests performed independently
(Review bias)
5. Is the disease status of the tested population
clearly described?
Yes Cant tell No
HINT: Consider
Presenting symptoms
Disease stage or severity
Co-morbidity
Differential diagnoses (Spectrum Bias)
6. Were the methods for performing the test
described in sufficient detail?
Yes Cant tell No
HINT: Consider
Was a protocol followed?
(B) What are the results?
7. What are the results?
HINT: Consider
Are the sensitivity and specificity and/or likelihood
ratios presented?
Are the results presented in such a way that we
can work them out?
8. How sure are we about the results?
consequences and cost of alternatives
performed?
HINT: Consider
Could they have occurred by chance?
Are there confidence limits?
What are they?
(C) Will the results help me and my
patients?
9. Can the results be applied to your patients?
Yes Cant tell No
HINT: Do you think your patients are so different from

those in the study that the results cannot be applied?
Such as age, sex, ethnicity and spectrum bias.
patients?
10. Can the test be applied to your patients?
Yes Cant tell No
HINT: Consider
Resources and opportunity costs
Level and availability of expertise required to
Interpret the tests
Current practice and availability of services
patients?
11. Were all outcomes important to the individual
or population considered?
Yes Cant tell No
HINT: Consider
Will the knowledge of the test result improve
patient well-being?
Will the knowledge of the test result lead to a
change in patient management?
patients?
12. What would be the impact of using this test on
your patients/population?
Therapeutic Study
Source: College of Family Physicians of Canada (Pearls for Residents)
1. Was the assignment of patients to treatment
groups truly randomized?
Yes Cant tell No
2. Were all patients who entered the study
accounted for appropriately at the end?
Yes Cant tell No
HINT:
i) Is the number randomized the same as in results?
ii) Were patients analyzed in the groups they were
allocated to at the start of the study?
3. Were patients, physicians, and those doing the
assessments "blind" to treatment?
Yes Cant tell No
4. Was similarity between groups documented?
Yes Cant tell No
HINT:
i) Is there a table of baseline characteristics?
ii) Were differences controlled for in the analysis?
5. Aside from the intervention, were the groups
treated in the same way?
Yes Cant tell No
6. How large was the treatment effect?
7. How precise was the estimate of the treatment

effect?
patients?
8. Can the results be applied to your patients?
Yes Cant tell No
HINT: Are study patients similar to your patient?

patients?
9. Were all clinically important outcomes
reported?
Yes Cant tell No
HINT:
i) Were the most important outcomes used?
ii) Were quality of life outcomes used? Assessment of
adverse effects?
patients?
10. Are the likely treatment benefits greater than
the potential harms and costs?
Yes Cant tell No
Randomized Controlled
Trial (RCT)
Source: Critical Appraisal Skills Programme (CASP)
1. Did the trial address a clearly focused issue?
Yes Cant tell No
HINT: An issue can be focused In terms of

The population studied
The intervention given
The comparator given
The outcomes considered
2. Was the assignment of patients to treatments
randomized?
Yes Cant tell No
HINT: Consider
How was this carried out?
Was the allocation sequence concealed from
researchers and patients?
3. Were all of the patients who entered the trial
properly accounted for at its conclusion?
Yes Cant tell No
HINT: Consider
Was the trial stopped early?
Were patients analyzed in the groups to which
they were randomized?
4. Were patients, health workers and study
personnel blind to treatment?
Yes Cant tell No
HINT: Consider
Patients?
Health workers?
Study personnel?
5. Were the groups similar at the start of the trial?
Yes Cant tell No
HINT: Consider other factors that might affect the outcome

such as age, sex, social class
6. Aside from the experimental intervention, were
the groups treated equally?
Yes Cant tell No
7. How large was the treatment effect?
HINT: Consider
What outcomes were measured?
Is the primary outcome clearly specified?
What results were found for each outcome?
8. How precise was the estimate of the treatment
effect?
HINT: What are the confidence limits?

patients?
9. Can the results be applied in your context?
Yes Cant tell No
HINT: Do you think that the patients covered by the trial

are similar enough to the patients to whom you will apply
this?, if not how to they differ?
patients?
10. Were all clinically important outcomes
considered?
Yes Cant tell No
HINT: Consider
Is there other information you would like to have
seen?
If not, does this affect the decision?
patients?
11. Are the benefits worth the harms and costs?
Yes Cant tell No
HINT: Even if this is not addressed by the trial, what do

you think?
Systematic Review
Source: The Joanna Briggs Institute (JBI)
1. Is the review question clearly and
explicitly stated?
The review question is an essential step in the systematic
review process. A well-articulated question defines the
scope of the review and aids in the development of the
search strategy to locate the relevant evidence. An
explicitly stated question, formulated around its PICO
(Population, Intervention, Comparator, Outcome)
elements aids both the review team in the conduct of the
review and the reader in determining if the review has
achieved its objectives. Ideally the review question should
be articulated in a published protocol; however this will
not always be the case with many reviews that are
located.
2. Were the inclusion criteria
appropriate for the review question?
The inclusion criteria should be identifiable from, and match the
review question. The necessary elements of the PICO should
be explicit and clearly defined. The inclusion criteria should be
detailed and the included reviews should clearly be eligible
when matched against the stated inclusion criteria. Appraisers
of meta-analyses will find that inclusion criteria may include
criteria around the ability to conduct statistical analyses which
would not be the norm for a systematic review. The types of
included studies should be relevant to the review question, for
example, an umbrella review aiming to summarize a range of
effective non-pharmacological interventions for aggressive
behaviors amongst elderly patients with dementia will limit itself
to including systematic reviews and meta-analyses that
synthesize quantitative studies assessing the various
interventions; qualitative or economic reviews would not be
included.
3. Was the search strategy
appropriate?
A systematic review should provide evidence of the search strategy
that has been used to locate the evidence. This may be found in the
methods section of the review report in some cases, or as an
appendix that may be provided as supplementary information to the
review publication. A systematic review should present a clear search
strategy that addresses each of the identifiable PICO components of
the review question. Some reviews may also provide a description of
the approach to searching and how the terms that were ultimately
used were derived, though due to limits on word counts in journals
this may be more the norm in online only publications. There should
be evidence of logical and relevant keywords and terms and also
evidence that Subject Headings and Indexing terms have been used
in the conduct of the search. Limits on the search should also be
considered and their potential impact; for example, if a date limit was
used, was this appropriate and/or justified? If only English language
studies were included, will such a language bias have an impact on
the review? The response to these considerations will depend, in part,
on the review question.
4. Were the sources and resources
used to search for studies adequate?
A systematic review should attempt to identify all the available
evidence and as such there should be evidence of a
comprehensive search strategy. Multiple electronic databases
should be searched including major bibliographic citation
databases such as MEDLINE and CINAHL. Ideally, other
databases that are relevant to the review question should also
be searched, for example, a systematic review with a question
about a physical therapy intervention should also look to
search the PEDro database, whilst a review focusing on an
educational intervention should also search the ERIC. Reviews
of effectiveness should aim to search trial registries. A
comprehensive search is the ideal way to minimize publication
bias, as a result, a well conducted systematic review should
also attempt to search for grey literature, or unpublished
studies; this may involve searching websites relevant to the
review question, or thesis repositories.
5. Were the criteria for appraising
studies appropriate?
The systematic review should present a clear statement that
critical appraisal was conducted and provide the details of the
items that were used to assess the included studies. This may
be presented in the methods of the review, as an appendix of
supplementary information, or as a reference to a source that
can be located. The tools or instruments used should be
appropriate for the review question asked and the type of
research conducted. For example, a systematic review of
effectiveness should present a tool or instrument that
addresses aspects of validity for experimental studies and
randomized controlled trials such as randomization and
blinding if the review includes observational research to
answer the same question a different tool would be more
appropriate. Similarly, a review assessing diagnostic test
accuracy may refer to the recognized QUADAS1 tool.
6. Was critical appraisal conducted by
two or more reviewers independently?
Critical appraisal or some similar assessment of the
quality of the literature included in a systematic review is
essential. A key characteristic to minimize bias or
systematic error in the conduct of a systematic review is
to have the critical appraisal of the included studies
completed independently and in duplicate by members of
the review team. The systematic review should present a
clear statement that critical appraisal was conducted by at
least two reviewers working independently from each
other and conferring where necessary to reach decision
regarding study quality and eligibility on the basis of
quality.
7. Were there methods to minimize
errors in data extraction?
Efforts made by review authors during data extraction can
also minimize bias or systematic errors in the conduct of a
systematic review. Strategies to minimize bias may
include conducting all data extraction in duplicate and
independently, using specific tools or instruments to guide
data extraction and some evidence of piloting or training
around their use.
8. Were the methods used to combine
studies appropriate?
A synthesis of the evidence is a key feature of a systematic
review. The synthesis that is presented should be appropriate
for the review question and the stated type of systematic
review and evidence it refers to. If a meta-analysis has been
conducted this needs to be reviewed carefully. Was it
appropriate to combine the studies? Have the reviewers
assessed heterogeneity statistically and provided some
explanation for heterogeneity that may be present? Often,
where heterogeneous studies are included in the systematic
review, narrative synthesis will be an appropriate method for
presenting the results of multiple studies. If a qualitative review,
are the methods that have been used to synthesize findings
congruent with the stated methodology of the review? Is there
adequate descriptive and explanatory information to support
the final synthesized findings that have been constructed from
the findings sourced from the original research?
9. Was the likelihood of publication
bias assessed?
As mentioned, a comprehensive search strategy is the
best means by which a review author may alleviate the
impact of publication bias on the results of the review.
Reviews may also present statistical tests such as
Eggers test or funnel plots to also assess the potential
presence of publication bias and its potential impact on
the results of the review. This question will not be
applicable to systematic reviews of qualitative evidence.
10. Were recommendations for policy
and/or practice supported by the
reported data?
Whilst the first nine (9) questions specifically look to
identify potential bias in the conduct of a systematic
review, the final questions are more indictors of review
quality rather than validity. Ideally a review should present
recommendations for policy and practice. Where these
recommendations are made there should be a clear link
to the results of the review. Is there evidence that the
strength of the findings and the quality of the research
been considered in the formulation of review
recommendations?
11. Were the specific directives for new
research appropriate?
The systematic review process is recognized for its ability
to identify where gaps in the research, or knowledge
base, around a particular topic exist. Most systematic
review authors will provide some indication, often in the
discussion section of the report, of where future research
direction should lie. Where evidence is scarce or sample
sizes that support overall estimates of effect are small and
effect estimates are imprecise, repeating similar research
to those identified by the review may be necessary and
appropriate. In other instances, the case for new research
questions to investigate the topic may be warranted.
PRESENTING A JOURNAL
Source: British Journal of Hospital Medicine, October 2013, Vol 74, No 10
British Journal of Hospital Medicine, October 2013, Vol 74, No 10
STEP 1: INTRODUCTION
Explain the clinical question that prompted you to
consult the literature and what drew you to the article.
STEP 2: WHO WROTE THE PAPER?

Consider the following:
Title of the article
Author/s
Affiliations of the author/s
Are there any outstanding features?
What is the impact factor of the journal?
STEP 3: THE HYPOTHESIS
What is the research question?
Consider the 4 basic components of a good
research question- PICO.
Population: Who was studied?
Intervention: What was the intervention
tested?
Control: What was the alternative that the
intervention was compared to?
Outcome: What was the nature of the
outcome?
STEP 4: THE STUDY DESIGN
Consider the following:
The Study type: RCT, Cohort, Case-control, Descriptive
The Study population: Can the results of the study be
translated to the general population?
Randomization: How are the participants allocated in the
group?
Bias: Is the research method exposed to bias? Has
randomization been used to reduce bias? What form of
blinding or masking has been used to reduce bias?
Inclusion and Exclusion Criteria: Are these appropriate
and clearly stated?
STEP 5: IS THE METHOD THOROUGH?
Was the method and approach to the study
appropriately diligent?
Were processes consistent?
Was follow-up complete and consistent in each
group?
What outcome measures were used and were they
appropriate?
Are the statistical tools suitable and correctly
interpreted by the investigators?
What significance level has been used (p value)?
Does the study exhibit low variablity?
STEP 6: THE RESULTS
Are the results clearly stated?
Have any results been ignored and why?
Is the result statistically significant (p<0.05)?
Review graphs and tables and consider
whether they are accurate and represent the
data presented.
STEP 7: DISCUSSION AND INTERPRETATION
Discuss the strength and weaknesses of the
study
Do the results support the conclusion?
Consider the statistical significance vs the
clinical significance.
Pay attention to the limitations of the study.
Have the authors declared any conflicts of
interest?
STEP 8: CONCLUSION & CLINICAL CONTEXT
Generalization of population, setting, and
intervention.
Can the results be applied to my patient (care)?
Are study patients similar to your patient?
Are the likely benefits greater than the potential
harms and costs?
End your appraisal by assessing how the paper
might change clinical practice.
You may want to consider to refer back to the
clinical question that first drew you to the article.

Critical Appraisal by John Vincent Pagaddu, M.D.

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Critical Appraisal by John Vincent Pagaddu, M.D.

Uploaded by

Copyright:

Available Formats

CRITICAL APPRAISAL

By: John Vincent A. Pagaddu, M.D.

Background / review of literature

What should be in it: a summary of the research

Measures: description of tests / questionnaires (validated?), data,

Procedure: study design (qualitative, quantitative, controlled?)

All outcome measures must be reported and confidence

In a systematic review, all the primary studies identified are

HINT: A question should include information about

HINT: Is this reference test(s) the best available indicator in

HINT: Do you think your patients are so different from

7. How precise was the estimate of the treatment

HINT: Are study patients similar to your patient?

HINT: An issue can be focused In terms of

HINT: Consider other factors that might affect the outcome

HINT: What are the confidence limits?

HINT: Do you think that the patients covered by the trial

HINT: Even if this is not addressed by the trial, what do

STEP 2: WHO WROTE THE PAPER?

You might also like