Research Full Report

Examining the Reproducibility of 6 Published Studies in

Public Health Services and Systems Research
Jenine K. Harris, PhD; Sarah B. Wondmeneh, HBSc; Yiqiang Zhao, MPH; Jonathon P. Leider, PhD

ABSTRACT
Objective: Research replication, or repeating a study de novo, is the scientific standard for building evidence and identifying
spurious results. While replication is ideal, it is often expensive and time consuming. Reproducibility, or reanalysis of data
to verify published findings, is one proposed minimum alternative standard. While a lack of research reproducibility has
been identified as a serious and prevalent problem in biomedical research and a few other fields, little work has been done
to examine the reproducibility of public health research. We examined reproducibility in 6 studies from the public health
services and systems research subfield of public health research.
Design: Following the methods described in each of the 6 papers, we computed the descriptive and inferential statistics for
each study. We compared our results with the original study results and examined the percentage differences in descriptive
statistics and differences in effect size, significance, and precision of inferential statistics. All project work was completed
in 2017.
Results: We found consistency between original and reproduced results for each paper in at least 1 of the 4 areas examined.
However, we also found some inconsistency. We identified incorrect transcription of results and omitting detail about data
management and analyses as the primary contributors to the inconsistencies.
Recommendations: Increasing reproducibility, or reanalysis of data to verify published results, can improve the quality of
science. Researchers, journals, employers, and funders can all play a role in improving the reproducibility of science through
several strategies including publishing data and statistical code, using guidelines to write clear and complete methods
sections, conducting reproducibility reviews, and incentivizing reproducible science.

KEY WORDS: open science, public health services and systems research, reproducibility

I n recent decades, the academic culture of pub-

lish or perish has collided with technological
advances in publishing, helping to spur the
Author Affiliations: Brown School, Washington University in St Louis, St
Louis, Missouri (Dr Harris, Ms Wondmeneh, and Mr Zhao); and Johns
Hopkins Bloomberg School of Public Health, Baltimore, Maryland (Dr Leider).
The authors acknowledge Todd Combs for his helpful comments on drafts of
this manuscript and for his assistance in setting up the coding2share GitHub
account as a repository for their code. The research presented in this paper is
that of the authors and does not reflect the official policy of the Robert Wood
Johnson Foundation.
J.K.H. received a grant from the Robert Wood Johnson Foundation in March
2017 that supported a portion of her time completing the writing and editing
of this manuscript. S.W. and Y.Z. were paid hourly as part-time graduate
research assistants by Washington University in St Louis for their work on
this project.
The authors declare no conflicts of interest.
Supplemental digital content is available for this article. Direct URL citations
appear in the printed text and are provided in the HTML and PDF versions of
this article on the journal’s Web site (http:// www.JPHMP.com).
Correspondence: Jenine K. Harris, PhD, Brown School, Washington
University in St Louis, One Brookings Dr, Campus Box 1196, St Louis, MO
63130 (harrisj@wustl.edu).
Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.
DOI: 10.1097/PHH.0000000000000694
exponential growth of published research.1,2 Whether
by correlation or causation, challenges currently
facing science coincide with this growth, including in-
creasing rates of retraction,3 a lack of replication and
reproducibility,4 and the proliferation of predatory
journals.5
Research replication, or the process of repeating a
study de novo, is the scientific standard for building
evidence and identifying spurious results.6,7 While
replication is ideal, the process can be time consuming
and expensive.7 Reproducibility, or reanalysis of data
to verify published findings, is one proposed minimum
alternative standard.7,8 Reproducibility requires, at
a minimum, accessible data and clear instructions
for data management and analysis.8 A recent review
of 10 papers from each of 10 top scientific jour-
nals found that 20% to 80% of papers per journal
(median = 50%) included an unclear or unknown
sample size, and up to 40% of papers (median =
20%) included unclear or unknown statistical tests.9
Incorrect reporting of research results can also hinder
reproducibility. Two recent studies found approxi-
mately 6% of P values were reported incorrectly in a

00 2018 • Volume 00, Number 00 www.JPHMP.com 1

Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
2 Harris, et al • 00(00), 1–9
sample of psychology papers,10 and 11% of P values
were incorrect in a sample of medical papers.11
Lack of reproducibility is serious and prevalent,
with just 21% of 67 drug studies and 40% to
60% of 100 psychological studies being successfully
reproduced.4,6,12 While most of the work on repro-
ducibility in science has been in the biomedical or psy-
chology fields, 1 recent paper in the journal Science ex-
amined economics papers and found that 61% (11 of
18) papers were reproducible.13 Less has been done to
examine the reproducibility of public health research.
Public health services and systems research (PHSSR)
is a subfield of public health research concerned with
the “organization, financing, and delivery of public
health services, and the impact of these services on
public health.”14 Governmental public health practi-
tioners use research evidence, including PHSSR, to se-
lect programs and justify program selection to their
partners, evaluate programs, prepare funding appli-
cations, and conduct needs assessments.15,16
One of the major sources of data for PHSSR
research is the National Profile of Local Health
Departments Study (Profile), which is a comprehen-
sive survey of all local health departments (LHDs)
nationwide conducted regularly by the National
Association of County and City Health Officials
(NACCHO). Survey questions cover LHD funding,
workforce, programs, and partnerships. Profile data
are available for public use for free through the
Inter-university Consortium on Political and Social
Research or can be requested directly by completing
a data use agreement form and e-mailing it to NAC-
CHO. Given that it is publicly available at no cost
and widely used in PHSSR, we sought to examine
the reproducibility of published studies that used the
NACCHO Profile study data. We had 3 goals: (1) ex-
amine reproducibility in a sample of published PHSSR
studies using publicly available data; (2) demonstrate
one strategy for increasing research reproducibility;
and (3) encourage public health researchers to begin
adopting reproducible research strategies.
Methods
Goal 1: Examine reproducibility in PHSSR studies
Sample selection
In early 2017, we conducted a search to identify pub-
lished journal articles using the 2013 NACCHO Pro-
file data. We searched 10 sources: PubMed, Google
Scholar, Ebsco, Web of Knowledge, Scopus, EMBASE,
CINAHL plus, Academic Complete, Web of Science,
and CINAHL. Our search terms were as follows:
“National Association of County & City Health Of-
ficials,” “National Association of County and City
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Reproducing Public Health Services and Systems Research
Health Officials,” and “NACCHO.” We reviewed the
resulting papers and restricted the results to those us-
ing 2013 NACCHO Profile data where all or most
of the analyses were conducted without combining it
with other years of Profile data or other data sources
(n = 15). We reviewed all 15 papers and recorded the
following: variables examined, type of statistical anal-
yses conducted, inclusion of detail on missing data
handling, inclusion of detail on variable recoding and
combining, whether analyses used the NACCHO core
data and/or 1 or more of the modules, and detail on
inclusion of another data set, if any. We used this sum-
mary information to select 5 of the 15 articles (33.3%)
that (1) included clear descriptions of data manage-
ment and analyses, and (2) applied commonly used
statistical methods for PHSSR.17 Harris has also pub-
lished PHSSR using NACCHO data but has no publi-
cations using 2013 NACCHO data alone for analyses;
we reproduced the results of a study using 2008 Pro-
file data led by Harris to determine whether we would
be more successful in reproducing our own work.
Reproducing descriptive and inferential
statistics
The first step in reproducing the research was to
mimic the work described in each manuscript to re-
produce descriptive statistics. We examined tables and
methods sections in each paper to identify variables
used, recoding, whether weights were used, and types
of descriptive analyses conducted. All authors worked
together to manage and analyze the data in order
to reproduce the reported descriptive statistics. Once
we completed data management and were able to re-
produce the descriptive statistics as closely as possi-
ble to the original published manuscript, we repro-
duced bivariate and multivariate inferential model(s)
on the same managed data. We reproduced statistics
reported in all tables in each manuscript that used
only NACCHO 2013 Profile data (or 2008 Profile
data for Paper 6); we did not reproduce tables that
included other data sources nor statistics displayed in
figures.
In Paper 1, we were not able to closely reproduce
the tables of descriptive statistics without additional
information from the authors. For this paper, rurality
was coded on the basis of Rural-Urban Commut-
ing Area (RUCA) codes, which allows classification
of health department jurisdictions by how rural
or urban they are. The NACCHO data can be re-
quested with RUCA codes but we were unsure of
the exact categories used in Paper 1, which lacked
detail about the codes. We e-mailed the lead author
and were sent the RUCA codes used. The RUCA
codes for Paper 1 are included on GitHub with the
00 2018 • Volume 00, Number 00
SPSS, Stata, and R command files for all results re-
ported here (https://github.com/coding2share/phssr-
reproducibility/).
Comparing reproduced results with original
results
We examined how similar or different the reproduced
statistics were compared with the published results.
For descriptive statistics reported alone or as part of a
table of bivariate analyses, we determined the percent-
age difference between the original and reproduced
value. In computing the percentage difference, we sub-
tracted the original values from the reproduced values
and divided by the originals. The resulting percentages
were positive for reproduced values greater than the
originals, and negative for reproduced values less than
the originals. For each paper, we determined the mean
and standard deviation for the percentage difference
across all descriptive statistics reported in each table.
For multivariate inferential statistics, we compared
original and reproduced statistics in 3 areas: effect
size, statistical significance, and precision. For effect
size comparisons, we used a similar strategy to what
we used for descriptive statistics, computing the per-
centage difference between original and reproduced
odds ratios. For significance, we used the cutoff for
significance reported (or implied) in the published
manuscript and noted if reproduced statistical tests
matched the significance of original test results. For
those not matching, we recorded whether the result
became significant or nonsignificant compared with
the original. Statistical significance was also compared
for bivariate inferential analyses. Precision was exam-
ined by comparing the mean width of a confidence
interval in the original and reproduced results.
Examining manuscript features
Finally, after reproducing results from the 6 papers,
we reviewed each paper and noted whether the paper
included detail we felt would have aided us in more
accurately reproducing results: (1) the type of bivari-
ate analysis conducted; (2) test statistic values for bi-
variate analyses (eg, χ 2 , F); (3) the name or a detailed
description of the weight variable used (if any); (4) de-
tail on how missing data were handled; (5) detail on
variable combining and recoding; and (6) sample sizes
and raw frequencies for groups analyzed.
Goal 2: Demonstrate a strategy for research
reproducibility
During the process of reproducing the 6 studies, we
organized and annotated all of the SPSS and Stata
commands we used to get from the raw NACCHO
Profile data to our reproduced results. Summary
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
www.JPHMP.com 3
figures were produced in R. The SPSS, Stata, and
R command files are available on GitHub: https:
//github.com/coding2share/phssr-reproducibility/.
Each of our analyses reproducing the manuscript in
question used the statistical program noted by that pa-
per. Data and codebooks for the Profile study data are
available from NACCHO (http://nacchoprofilestudy.
org/).
Results
Sample characteristics
The 5 studies using 2013 NACCHO Profile study
data were published between 2015 and 2017 in 3 dis-
tinct journals with 4 distinct lead authors. The addi-
tional study by Harris was published in 2013 and used
2008 NACCHO Profile study data. Descriptive tables
reported frequencies, percentages, means, and stan-
dard deviations. Bivariate inferential statistics, such
as χ 2 or correlation analyses, were included in 4 of
the 6 papers. The 5 papers using 2013 data used
weights in analysis and conducted multivariate lo-
gistic regression for the primary analysis; Harris pa-
per neither weighted the data nor reported logistic
regression.
Reproducing descriptive statistics
Following the methods and results sections of the
6 papers, we computed descriptive statistics from
NACCHO Profile Study data sets obtained directly
from NACCHO. Figure 1 shows the mean percent-
age difference between the originally reported and re-
produced statistics. Most tables of descriptive statis-
tics were accurately reproduced with the exception of
Paper 1 Table 1, and Paper 6 Table 1 (Figure 1; see
Supplemental Digital Content Appendix A, available
at http://links.lww.com/JPHMP/A396). Tables show-
ing the original and reproduced values are shown in
Appendix A.
Most differences between original and reproduced
values in Paper 1 Table 1 were small, with a few ex-
ceptions (see Supplemental Digital Content Appendix
A, available at http://links.lww.com/JPHMP/A396).
One exception was the administrator highest degree
variable where the category “Associates degree” had
1.8% of the original study participants but 3.8% of
the reproduced participants, more than doubling the
percentage. Likewise, there was a large difference in
the percentage of LHDs a community health improve-
ment plan (CHIP) had, that developed an agency-wide
strategic plan in the original (47.8%) compared with
the reproduced (56.2%). These variables were created
by combining variables (degree variable) or recoding
4 Harris, et al • 00(00), 1–9 Reproducing Public Health Services and Systems Research

descriptive odds ratios

statistics and correlations

Paper 1 Table 4
Paper 5 Table 4 Model 2
Paper 3 Table 2
Paper 6 Table 3
Paper 5 Table 4 Model 1
Paper 5 Table 3
Manuscript & table number

Paper 1 Table 3
Paper 2 Table 1
Paper 4 Table 1
Paper 6 Table 2
Paper 3 Table 1
Paper 1 Table 2
Paper 3 Table 3 Model 2
Paper 3 Table 3 Model 3
Paper 3 Table 3 Model 1
Paper 1 Table 1
Paper 2 Table 3
Paper 4 Table 2
Paper 6 Table 1

−50% −25% 0% 25% 50%

Mean percent difference from original to reproduced
with standard deviation

FIGURE 1 Difference Between Original and Reproduced Descriptive and Inferential Statistics Reported in 5 Published Public Health Services and
Systems Research (PHSSR) Studies Using National Association of County & City Health Officials (NACCHO) 2013 Profile Data (Papers 1-5) and 1 PHSSR
Study by Harris Using NACCHO 2008 Profile Data (Paper 6)

a multicategory variable into a binary variable (CHIP
variable). The manuscript included very clear instruc-
tions for combining and recoding; we reviewed our
syntax in light of the instructions in the manuscript
but were not able to identify any obvious differences
in how we constructed the variables and how the
authors described variable construction. No sample
sizes or unweighted frequencies were reported, so
we did not have a way to compare the number of
LHDs recoded inconsistently from the original to the
reproduced.
The differences between original and reproduced
in Paper 6 Table 1 (see Supplemental Digital Con-
tent Appendix A, available at http://links.lww.com/
JPHMP/A396) were due to an error in transcription
by the paper authors or the journal staff. Specifically,
percentages in the land use planning, tobacco preven-
tion and control, influenza, and obesity rows in Pa-
per 6 Table 1 appear to have been swapped between
rows, suggesting the row labels were reordered but the
values were not. Other than this, the reproduced table
matched the original nearly exactly.
Reproducing inferential models
Comparing effect size
On average, the reproduced effect sizes were accurate
for most papers, with mean differences between
original and reproduced being less than 5% for all
but 1 paper. For Paper 1 Table 4, the mean differ-
ence between original and reproduced effect sizes
was 11.1%. We examined the differences between
original and reproduced odds ratios and found 4
odds ratios in Paper 1 that were very different from
the original, including both categories of rurality,
whether or not the health department had a board
of health, and whether the health department was in
the state governance category (Figure 2). We thought
that perhaps the large differences seen between
original and reproduced descriptive statistics for

Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
00 2018 • Volume 00, Number 00 www.JPHMP.com 5

Original Reproduced

Urban

State governance
Local Health Department Characteristic

Public health physician (ref=no)

Per capita revenue

Local governance

Local board(s) of health (ref=no)

Information systems specialist (ref=no)

Agency−wide strategic plan (ref=no)

*Micropolitan

**Epidemiologist (ref=no)

1.0 10.0
Odds ratio & 95% CI (log scale)

FIGURE 2 Original and Reproduced Odds Ratios and Confidence Intervals From Paper 4 Model 1
Abbreviation: CI indicates confidence interval.

this paper may have resulted in the low accuracy Comparing precision
of the regression model, but the variables with the
In addition to relatively high reproducibility in statis-
least accuracy in the descriptive analyses were not
tical significance, the precision of effect size estimates
used in the regression model. We were unable to
was relatively stable across all of the studies except
identify the source of the sizeable differences between
for Paper 1. The reproduced results of Paper 1 were
observed and reproduced regression results for this
more precise than the original results (Figure 2),
model.
which suggests that smaller standard deviations
Comparing statistical significance and/or larger sample sizes were used to compute the
reproduced confidence intervals. Figure 4 shows the
All but 1 of the bivariate tables reproduced were com- mean difference in width between original and re-
pletely accurate with respect to statistical significance produced confidence intervals for odds ratios across
status (Figure 3). The multivariate models each had studies. Supplemental Digital Content Appendix
1 or more changes to statistical significance status B, available at http://links.lww.com/JPHMP/A397,
with the exception of Paper 4. Paper 6 had by far the shows original and reproduced odds ratios for Papers
most changes in statistical significance; 10 of the 40 1 through 5.
correlations changed from statistically significant to
nonsignificant when reproduced. In an examination Examining manuscript features
of documents submitted by Harris for publication of
Paper 6, we found that 4 of the 10 correlations that Three of the 4 papers that included bivariate analy-
went from significant in the original to nonsignificant sis included the type of bivariate analysis conducted,
in the reproduced results were not marked as signif- but none of the tables of bivariate analyses included
icant in the submitted table but were marked as so test statistic value(s) such as χ 2 or t statistics. Of the
in the published table. At some point in the process 5 papers using weights, 2 of them were specific about
from submission to publication, 4 correlations were the weighting variable used. The other 3 papers stated
transcribed incorrectly as significant and not caught that “appropriate” or “proper” weights were used. All
during the proof review or any other stage before papers gave overall response rates of the study and 1
publication. paper (Paper 5) stated that all included variables had

Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
6 Harris, et al • 00(00), 1–9 Reproducing Public Health Services and Systems Research

Non−significant in original, Significant in original,

significant when non−significant when Stayed the same
reproduced reproduced

Paper 6 Table 3

Paper 6 Table 2

Paper 6 Table 1

bivariate
Paper 4 Table 1

Paper 3 Table 2
Manuscript & table number

Paper 1 Table 3

Paper 1 Table 2

Paper 5 Table 4 Model 2

Paper 5 Table 4 Model 1

Paper 4 Table 2

multivariate
Paper 3 Table 3 Model 3

Paper 3 Table 3 Model 2

Paper 3 Table 3 Model 1

Paper 2 Table 3

Paper 1 Table 4

0% 25% 50% 75% 100%

Percent of inferential statistics that maintained
or changed significance status when reproduced

FIGURE 3 Percentage of Bivariate and Multivariate Inferential Results That Maintained or Changed Statistical Significance Status When Reproduced
for 6 Public Health Services and Systems Research Studies

less than 5% missing data; no other detail on handling

Original Reproduced of missing data was included. Papers 2 and 4 included
unweighted frequencies, and Papers 5 and 6 included
sample sizes for analyses; other papers included nei-
Paper 5 Table 4 Model 1
ther of these.
Paper and table number

Paper 2 Table 3

Paper 5 Table 4 Model 2

Paper 4 Table 2 Discussion

Paper 3 Table 3 Model 3 We reproduced the tables of descriptive and infer-
Paper 3 Table 3 Model 1 ential statistics for 5 PHSSR studies using 2013
Paper 3 Table 3 Model 2
NACCHO Profile data and 1 study using 2008 Pro-
file data. All 6 used common statistical methods for
Paper 1 Table 4
PHSSR.17 Overall, the accuracy of the reproduced
0 3 6 9 results was high with a few exceptions. Consistent
Mean confidence interval width for with prior research in psychology and medicine,9-11 we
original and reproduced
logistic regression results identified 2 primary issues that hindered reproducibil-
ity: (1) typos and other inaccuracies in transcription;
FIGURE 4 Comparison of Confidence Interval Width (Precision) for Odds and (2) a lack of detail about steps taken and
Ratios From Original and Reproduced Logistic Regression Models in 5
Published Public Health Services and Systems Research Studies
choices made by the authors throughout the research
process.

Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
00 2018 • Volume 00, Number 00
First, typos and other errors when transferring text
during the writing and publishing process accounted
for numerous differences between original and repro-
duced results. The switching of 4 rows of data in Paper
6 Table 1 without editing the labels was the biggest in-
stance we found but not the only one. Paper 2 Table 1
had 4 frequency and percentage values switched (see
Supplemental Digital Content Appendix A, available
at http://links.lww.com/JPHMP/A396). In addition,
all of the significant P values in Paper 1 Table 2 were
shown as P = .001 while reproduced results for this
paper indicated that these P values were all P < .001.
Although it seems unlikely that all the P values would
be exactly equal to .001, without the test statistics
for this table, it is difficult to check whether this
difference is real or a set of typos. In Paper 6, 4 super-
scripts indicating significance were added mistakenly
to nonsignificant correlations at some point between
submission and publication and were not discovered
during the publishing process. Altogether, 3 of the 6
papers examined included transcription errors in re-
porting values. An additional paper, Paper 5, included
a labeling error that did not result in misreported
values but caused some initial confusion during data
management. Specifically, a less than symbol (<) was
printed in Tables 3 and 4 where there should have
been a greater than or equal to (≥) symbol.
The omission of detail throughout the research
process also hindered accuracy in reproducing these
papers. For example, our initial attempt to reproduce
the unweighted frequencies in Paper 2 Table 1 resulted
in lower values in the “No” category for 5 variables.
To reproduce the frequencies exactly, we tried assign-
ing missing data into the category for “No.” So, for
example, an LHD missing data on the variable for
strategic plan developed would be placed into the
“No” category indicating they were not developing
a strategic plan. This strategy was not intuitive to us
and was not described in the methods of the paper but
did result in accurate frequencies. We found a similar
recoding of missing values to “No” in Paper 3 for sev-
eral variables with no mention of this strategy in the
article text (see command file at https://github.com/
coding2share/phssr-reproducibility). In Paper 3, the
authors included unweighted percentages, which were
useful in determining how to treat the missing values.
Likewise, while 3 of the 4 papers including bivari-
ate statistics named the test they were using, none of
the 4 papers reporting bivariate analyses included the
values of test statistics for the bivariate analyses. Al-
though the values of test statistics, like χ 2 or U, are
often not inherently meaningful, they do allow the
determination or verification of the P value for a sta-
tistical test. As an example of how this may have
improved reproducibility, the percentages reported in
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
www.JPHMP.com 7
the original Paper 1 Table 2 were accurately repro-
duced, so we expected to also find accurately repro-
duced P values. However, the P values differed slightly
between original and reproduced results. We believe
that this was due to a transcription error; however,
including bivariate test statistics would have verified
our belief or alerted us to some analytic or reporting
choice the authors made that we did not include in
our process.
The differences found in effect size, precision, and
statistical significance in the reproduced analyses
likely resulted from not knowing data management
details, like the handling of missing data during re-
coding in Papers 2 and 3. If a small change in data
management results in significance changes or less
precise results, it suggests that findings from these
studies may not be very robust and should be inter-
preted with caution. For example, in Paper 5, the dif-
ferences between observed and reproduced descrip-
tive statistics were minor; however, in the inferential
results, the variable indicating whether the local board
of health had authority to approve the LHD bud-
get was a significant protective factor against LHD
budget cuts in the original analyses but nonsignifi-
cant in the reproduced analyses. The confidence in-
tervals in the original (0.50-0.98) and reproduced
(0.52-1.03) were close but just different enough to
change the statistical significance status. The authors
concluded that this significant protective factor was
one of a few indicators that board of health ac-
tions influence LHD financial stability, which may
or may not be the case depending on how the data
were managed. Instead of focusing on P values,18
a focus on effect sizes and practically relevant dif-
ferences is one possible strategy for addressing this
concern. Finally, as mentioned, we reproduced anal-
yses in the software used in the original manuscript.
This was at the recommendation of peer reviewers.
Initially, we conducted all analyses in R. However,
when comparing R results to the original results ob-
tained from Stata and SPSS, we noticed a number
of differences in statistical significance that gave us
pause. In examining possible reasons for these differ-
ences, we found slight differences in algorithms19,20
and more substantial differences in how weights are
applied across different statistical packages for some
statistical tests. Understanding and describing how the
program we use applies weights in our analyses could
increase the ability of others to reproduce our work.
Study limitations include the small number of
studies reproduced, the small selection of studies
based on data availability, and use of traditional
statistical methods. These studies were not likely to
be representative of the larger population of PHSSR
studies or public health research. However, given data
8 Harris, et al • 00(00), 1–9
Implications for Policy & Practice
■ Increasing reproducibility, or reanalysis of data to verify pub-
lished results, can improve the quality of science.
■ To increase reproducibility, include detail in publications
such as specific data management decisions, sample sizes
for all analyses, unweighted frequencies for weighted anal-
yses, and the name and test statistic for all statistical tests.
■ Checking proofs of publications closely, including all labels
and numbers in tables, can identify errors to correct in re-
porting or typesetting and increase reproducibility.
■ Researchers, journals, employers, and funders can improve
reproducibility by using existing publishing guidelines; pub-
lishing statistical code, output, and data when possible;
conducting reproducibility reviews; and incentivizing repro-
ducible science.
availability, the use of clear widely used methods, we
anticipate that these studies were more reproducible
than studies using nonpublic data and/or more
complex analytic methods.
Public health practitioners working in local and
state health departments nationwide use PHSSR to
select evidence-based programs15,16 that have been
demonstrated effective in publications like those we
examined. Classification of a program or strategy as
effective or evidence-based is often based on P val-
ues that reach a certain threshold and/or effect sizes,
like odds ratios, that appear large and precise. Error-
laden research wastes resources and puts people at
risk,21 and 73.4% of retractions are not fraud-related
but instead are due to error or something else.22 Al-
though a reproducible study is not necessarily a per-
fect study,7 reproducing research can aid in identifying
and correcting errors and improving the overall qual-
ity of research. Increasing reproducible research will
require a shift in the current scientific culture that has
researchers competing for scarce human and finan-
cial resources and often rewards publication quantity,
novel findings, and statistical significance over scien-
tific rigor and quality.4,23
To begin to shift the culture toward reproducibil-
ity, there are many strategies researchers, journals,
employers of researchers, funders, and others can
use.23,24 Specifically, reproducibility increases when re-
searchers (1) describe the methods used in enough de-
tail in publications so that they can be reproduced;25
(2) include a methodologist or biostatistician in all
stages of research;24 (3) publish data and statistical
source code;7,11,13,23 (4) follow existing guidelines for
reporting study results;23,25-27 and (5) take advan-
tage of new tools available for improving research
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
Reproducing Public Health Services and Systems Research
reproducibility.28 In addition, journals improve re-
producibility by (1) requiring authors to follow ex-
isting guidelines for reporting study results;23,25-27
(2) requiring authors to submit data and statistical
source code;7,11,13,23 (3) using reviewers with statistical
expertise;11 and (4) offering reproducibility reviews.7
Funders and employers of researchers could consider
incentivizing or funding reproducible research, repro-
duction efforts, and strategies for increasing repro-
ducibility such as sharing data and statistical code.6,29
While not all of these suggestions are appropriate for
all projects, especially for those with restricted use
data or data with private or confidential information,
all projects can employ some of these strategies. In-
creasing research reproducibility in public health can
help ensure that researchers are providing practition-
ers with the best possible science to use in determining
how to spend scarce resources and improve public
health.
References
1. Bornmann L, Mutz R. Growth rates of modern science: a biblio-
metric analysis based on the number of publications and cited ref-
erences. J Am Soc Inf Sci Technol. 2015;66(11):2215-2222.
2. MacRoberts MH, MacRoberts BR. Problems of citation analysis:
a study of uncited and seldom-cited influences. J Am Soc Inf Sci
Technol. 2010;61(1):1-12.
3. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the ma-
jority of retracted scientific publications. Proc Natl Acad Sci U S A.
2012;109(42):17028-17033.
4. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can
we rely on published data on potential drug targets? Nat Rev Drug
Discov. 2011;10(9):712-712.
5. Xia J, Harmon JL, Connolly KG, Donnelly RM, Anderson MR,
Howard HA. Who publishes in “predatory” journals? J Am Soc Inf
Sci Technol. 2015;66(7):1406-1417.
6. Open Science Collaboration. Estimating the reproducibility of psy-
chological science. Science. 2015;349(6251):aac4716.
7. Peng RD. Reproducible research in computational science. Sci-
ence. 2011;334(6060):1226-1227.
8. Peng RD, Dominici F, Zeger SL. Reproducible epidemiologic
research. Am J Epidemiol. 2006;163(9):783-789.
9. Gosselin RD. Open letter to scientific journals. http://www.
biotelligences.com/uploads/2/1/8/0/21806970/open_letter_to_
journals.pdf. Updated 2017. Accessed April 22, 2017.
10. Nuijten MB, Hartgerink CH, Assen MA, Epskamp S, Wicherts JM.
The prevalence of statistical reporting errors in psychology (1985-
2013). Behav Res Methods. 2016;48(4):1205-1226.
11. García-Berthou E, Alcaraz C. Incongruence between test statis-
tics and P values in medical papers. BMC Med Res Methodol.
2004;4(1):1.
12. Anderson CJ, Bahnik S, Barnett-Cowan M, et al. Response to com-
ment on “estimating the reproducibility of psychological science”.
Science. 2016;351(6277):1037.
13. Camerer CF, Dreber A, Forsell E, et al. Evaluating replicability of lab-
oratory experiments in economics. Science. 2016;351(6280):1433-
1436.
14. Mays GP, Halverson PK, Scutchfield FD. Making public health im-
provement real: the vital role of systems research. J Public Health
Manag Pract. 2004;10(3):183-185.
15. Jacob RR, Allen PM, Ahrendt LJ, Brownson RC. Learning about and
using research evidence among public health practitioners. Am J
Prev Med. 2017;52(3):S304-S308.
16. Brownson RC, Allen P, Duggan K, Stamatakis KA, Erwin PC. Fos-
tering more-effective public health by identifying administrative
00 2018 • Volume 00, Number 00
evidence-based practices: a review of the literature. Am J Prev
Med. 2012;43(3):309-319.
17. Harris JK, Beatty KE, Barbero C, et al. Methods in public health
services and systems research. Am J Prev Med. 2012;42(5):
S42-S57.
18. Nuzzo R. Statistical errors. Nature. 2014;506(7487):150.
19. Park HM. Comparing group means: t-tests and one-way ANOVA
using Stata, SAS, R, and SPSS. http://hdl.handle.net/2022/19735.
Published 2009. Accessed October 11, 2017.
20. Park HM. Linear regression models for panel data using SAS,
Stata, LIMDEP, and SPSS. http://www.indiana.edu/∼statmath/
stat/all/panel/panel.pdf. Published 2015. Accessed October 11,
2017.
21. Steen RG. Retractions in the medical literature: how many pa-
tients are put at risk by flawed research? J Med Ethics. 2011;37(11):
688-692.
22. Steen RG. Retractions in the scientific literature: is the inci-
dence of research fraud increasing? J Med Ethics. 2011;37(4):
249-253.
Copyright © 2018 Wolters Kluwer Health, Inc. Unauthorized reproduction of this article is prohibited.
www.JPHMP.com 9
23. Begley CG, Ellis LM. Raise standards for preclinical cancer re-
search. Nature. 2012;483(7391):531-533.
24. Ioannidis JP, Greenland S, Hlatky MA, et al. Increasing value and
reducing waste in research design, conduct, and analysis. Lancet.
2014;383(9912):166-175.
25. International Committee of Medical Journal Editors (ICMJE). Uni-
form requirements for manuscripts submitted to biomedical jour-
nals: writing and editing for biomedical publication. Haematologica.
2004;89(3):264.
26. Santori G. Journals should drive data reproducibility. Nature.
2016;535(7612):355-355.
27. Network E. Enhancing the quality and transparency of health re-
search. www.equator-network.org. Published October 4, 2015. Ac-
cessed 2009. Accessed October 11, 2017.
28. Lowndes JSS, Best BD, Scarborough C, et al. Our path to better
science in less time using open data science tools. Nat Ecol Evol.
2017;1:0160.
29. Ioannidis JP. How to make more published research true. PLoS
Med. 2014;11(10):e1001747.