System Dynamics at Sixty: The Path Forward

INTRODUCTION
System dynamics at sixty: the path forward

John D. Sterman*
Abstract
The late Jay Forrester founded the field of system dynamics 60 years ago. On this anniversary I
ask what lessons his remarkable life of innovation and impact hold for the field. From a
Nebraska ranch to MIT, servomechanisms, digital computing and system dynamics, Jay lived
his entire life on the frontier, innovating to solve critical problems. Today, more than ever,
humanity faces grave threats arising from our mismanagement of increasingly complex systems.
Fortunately, progress in system dynamics, the natural and social sciences, computing, data
availability, and other modeling disciplines have advanced the frontier in theory and methods
to build reliable, useful knowledge of complex systems. We must therefore ask: What would Jay
do if he were young today? How would he build on the knowledge, technology, data and
methods now available to improve the science and practice of system dynamics? What must we
now do to explore the next frontier?
© 2018 System Dynamics Society
Syst. Dyn. Rev. 34, 5–47 (2018)
The challenge
The field of system dynamics just marked the 60th anniversary of its found-
ing by the late Jay Forrester, who passed away at the age of 98 in November
2016. The 50th anniversary provided an occasion to look back and celebrate
the many accomplishments in the field (Sterman, 2007). There have been
many important contributions since then, some presented in this special
issue marking the 60th anniversary. These include methodological advances
expanding the system dynamics toolkit to more robust methods to capture
network dynamics (Lamberson, 2018), the interaction of dynamics at
multiple timescales (Ford, 2018), experimental studies of group modeling
(McCardle-Keurentjes et al., 2018) and insights arising from integration of
system dynamics and operations research (Ghaffarzadegan and Larson,
2018), along with applications in medicine (Rogers et al., 2018), epidemiol-
ogy (Tebbens and Thompson, 2018), health care (Minyard et al., 2018),
climate change (Woodruff et al., 2018), environmental policy (Kapmeier and
Gonçalves, 2018), and organizational transformation (Rydzak and
Monus, 2018).
Despite these advances, we live in a time of growing threats to society and
human welfare. On this anniversary we must therefore also look forward.
What must be done now to improve theory and practice, education and
* Correspondence to: John D. Sterman, MIT Sloan School of Management, System Dynamics Group, 100
Main Street, E62-436, Cambridge, Massachusetts 02142, U.S.A. E-mail: jsterman@mit.edu
Accepted by Yaman Barlas, Received 27 July 2018; Revised 31 July 2018; Accepted 1 August 2018
System Dynamics Review

System Dynamics Review vol 34, No 1-2 (January-June 2018): 5–47
Published online in Wiley Online Library
(wileyonlinelibrary.com) DOI: 10.1002/sdr.1601
5
6 System Dynamics Review
training, so that the next decades see even more progress and impact than
the last? Jay himself recognized the need for significant changes in system
dynamics to realize its full promise (Forrester, 2007a).
I begin with a brief review of Jay’s career and contributions in servomecha-
nisms, digital computation, the management of large-scale projects, and sys-
tem dynamics. I argue that his contributions are widely misunderstood. The
main lesson of Jay’s several careers does not lie in the particular tools or
methods he developed, but in the need for continual innovation to solve
important and difficult problems. Close examination of Jay’s life reveals a
relentless effort to make a difference on real and pressing problems. To do
so, in each of his careers, Jay studied the then-new advances in tools and
methods developed in any discipline relevant to the problem he sought to
address, mastering the state of the art—and then built on those advances.
The failure to appreciate Jay’s real contribution is a significant problem
today in the field of system dynamics. Despite many successes, too many in
the field continue to develop models the way Jay did in the 1950s, 60s and
70s, adhering to outdated methods for model development, testing and com-
munication, and closing themselves off from important developments in other
fields. The result is a gap between best practice and what is often done.
Sixty years after Jay brilliantly synthesized feedback control theory,
numerical methods, and digital computation to create the field of system
dynamics, we must close the “knowing–doing gap” (Pfeffer and Sutton,
2000) that has emerged, then innovate again. I outline developments in other
disciplines we must embrace to close the gap and improve the rigor, reliabil-
ity and impact of system dynamics research and practice.
A bold innovator
Jay’s remarkable life story has been told elsewhere, including by Jay himself
(Richardson, 1991; Forrester, 1992, 2007b; Green, 2010; Lane and Sterman,
2011, 2018). Born and raised on a ranch in western Nebraska where his par-
ents were the original homesteaders, Jay learned early on the value of educa-
tion, self-reliance and the need to solve difficult problems:
A ranch is a cross-roads of economic forces. Supply and demand, changing

prices and costs, and economic pressures of agriculture become a very personal,
powerful, and dominating part of life … [A]ctivities must be very practical. One
works to get results. It is full-time immersion in the real world. (Forrester, 1992).
To solve the pressing problems of life on a frontier homestead, Jay mas-

tered what was known and then innovated: after studying new develop-
ments in electricity he built a wind-driven generator that provided the first
electricity to the ranch and powered the lights, radio and shop tools:
DOI: 10.1002/sdr
J. D. Sterman: System Dynamics at Sixty 7
Fig. 1. Radar antenna stabilization servo designed by Jay Forrester. Photo: John Sterman [Color figure can be viewed at
wileyonlinelibrary.com]
“Building an electrical system from discarded automobile parts was a very

practical undertaking and another step in learning how to succeed in uncharted
territory” (Forrester, 1992).
After college, Jay joined MIT and the servomechanisms lab, then newly
founded by Gordon Brown. The war was underway; Jay’s master’s thesis was
to design and test a servomechanism to stabilize the radar antennae on naval
ships (Figure 1). To do so, Jay mastered the state of the art in feedback con-
trol systems, both theory and practice. Then he innovated: the servo would
have to generate large forces to keep the heavy antenna vertical despite high
winds and the pitch, roll and yaw of the ship beneath. Doing so required
very high gain in the control loop while maintaining stability. “Departing
from my training in electrical engineering, the work focused on designing
DOI: 10.1002/sdr
Fig. 2. At the insistence

of the captain, Forrester’s
prototype servo was
installed on the aircraft
carrier, USS Lexington,
CV-16, shown here after
launch in Boston, 17
February 1943. Photo:
http://www.navsource.
org/archives/02/16.htm.
mechanical hydraulic variable speed pumps, motors, and high-gain hydrau-

lic amplifiers because at that time the military mistrusted vacuum tubes in any-
thing except radios.” Jay built a positive feedback loop into the hydraulic
amplifier of the servo, finding that with sufficient phase lag the positive feed-
back would actually stabilize the system while generating the needed gain:
20,000 to 1. However, the patent office initially rejected Jay’s patent applica-
tion. As Jay told it, the patent examiner declared that everyone knew a positive
feedback loop was destabilizing so the thing would never work.
Meanwhile, Jay had built a prototype and had it running in the servo lab at
MIT. Clearly, as the old joke goes, it worked in practice, even if it didn’t work
in theory. But Jay was not satisfied. He knew that having it work in the lab
would not be enough: to survive wartime conditions, the servo had to be
robust in operation and survive what was likely to be inadequate mainte-
nance. As he related the story to me, “I knew the Navy boys would not always
be able to maintain it properly.” Jay went down to Revere beach, just north of
Boston, collected a large bucket of seawater, including sand and other debris,
then, back at the lab, replaced most of the hydraulic fluid with the brine and
“ran it for months that way” (Forrester, personal communication).
After demonstrating the prototype for the Captain of the aircraft carrier
USS Lexington (Figure 2), the Captain said “I want that; I mean that very

DOI: 10.1002/sdr
one; we can’t wait for the production equipment” (Forrester, 1992). After
working well for 9 months:
the experimental control units stopped working. In November 1943, I volun-

teered to go to Pearl Harbor to find the reason and repair the hydraulic controls.
Having discovered the problem, but not having time to fix it, I was
approached by the executive officer of the Lexington who said they were about
to leave Pearl Harbor. He asked me to come with them to finish my job. I
agreed, having no idea where that might lead.
We were off shore during the invasion of Tarawa and then took a turn down
between the Ratak and Ralik chains of the Marshall Islands. The islands on both
sides held enemy air bases, and the Japanese didn’t like having a U.S. Navy Task
Force wrecking their airports. They kept trying to sink our ships. Finally, twelve
hours later and after dark they dropped flares along one side of the task force and
came in with torpedo planes from the other side. About 11:00 pm, they succeeded
in hitting the Lexington, cutting through one of the four propeller shafts and setting
the rudder in a hard turn. Again, the experience gave a very concentrated immer-
sion in how research and theory are related to practical end uses. (Forrester, 1992).
Film of the Lexington on fire after the attack is available at https://youtu.

be/ePdR6CF1SD0.
Figure 3 shows Jay’s journey from Cambridge to the Marshalls and back,
from a scrapbook he kept.1 After returning to Pearl Harbor, Jay booked a
Fig. 3. Map from Jay’s

scrapbook showing the
route he took from
Cambridge to Pearl
Harbor to combat in the
Marshall Islands and
back. To enhance
visibility, the thick line
overlays Jay’s pencil
marks showing his route.
Source: Forrester
Scrapbook [Color figure
can be viewed at
1
I thank Judy, Nathan and Ned Forrester for permission to use the material from Jay’s scrapbook.

DOI: 10.1002/sdr
room at the Moana Hotel ($3 per night) for a little post-combat rest and
relaxation. Then, on 7 December 1943, he received an urgent message from
Gordon Brown (Figure 4):
YOUR PRESENCE URGENTLY NEEDED IN CAMBRIDGE MR STOUTS OFFICE

WILL ARRANGE PRIORITY IF YOUR RETURN IS BEING DELAYED PLEASE
REPLY GORDON S. BROWN.
Not wanting to interrupt his well-deserved R&R and a planned stop to visit
his family in Nebraska on the way back to MIT, Jay responded (Figure 5):
Am now in Honolulu address Moana Hotel. Performance and reputation of

equipment [on the Lexington] satisfactory. Received your cablegram tonight.
Plan to spend about 10 days around Pearl Harbor. Will be in Cambridge near
end of December. Is urgently needed presence normal or unusual situation.
Gordon responded:
FIELD SERVICE PROBLEM ON ALL OUR MATERIEL NEEDS CLARIFICATION

SERVICE PERSONNEL ATTITUDE INVARIABLY STUBBORN WILL RELY ON
YOU FOR PRACTICAL EVALUATION OF PROBLEM TO GUIDE LABORA-
TORY POLICY ON RAYTHEON SERVOS AND ARMY M3 AND HALL
AMPLIFIER
Fig. 4. Radiogram from

Gordon Brown,
7 December 1943. Source:
Forrester Scrapbook
[Color figure can be
viewed at

DOI: 10.1002/sdr
Fig. 5. Forrester’s
response to Gordon
Brown’s request to return
to MIT urgently,
11 December 1943.
Source: Forrester
Scrapbook [Color figure
can be viewed at
In characteristic fashion, Jay’s response did not lament the stubborn attitude
of the people using and maintaining the servos he had designed. He did not
blame the people in the system. Instead he sent pages of detailed instruc-
tions to improve the design of the servo so it would be more robust and even
easier to operate and maintain, including this telling passage:
Am convinced complete check of components must be possible without special

laboratory equipment or disassembly stop retain pressure indicator and pres-
sure relief at gear pump exit stop Have Kimball retain accessibility of both sides
of chassis while operating … although synchros are taper pinned initially stan-
dard nut and set screw must be retained for emergency replacement without
pin stop Believe new design overcomes all problems encountered here stop we
must plan mechanism and complete instructions to permit servicing and repair
in field.
Jay’s relentless focus on robustness, reliability, and designing the system

so that ordinary people could use and maintain it under realistic operating
conditions carried over into all his later work in computing, intercontinental
air defense, and system dynamics.
After the war Jay embarked on a project to develop a flight simulator
intended to test designs for aircraft before they were built. Initially envi-
sioned as an analog simulator, he concluded that analog computing could
DOI: 10.1002/sdr
never work in such a demanding real-time task. Instead, Jay decided to

develop the simulator “around the untested concept of a digital computer.”
The result was Whirlwind, the first reliable, high-speed digital computer
capable of real-time control (Green, 2010).2
The Whirlwind project began in 1947. Already by March 1949 Jay had for-
mulated a vision of the innovations needed to turn computers from mere tab-
ulation tools into devices that could control complex systems in real time
and carry out simulations (Figure 6). To do so Whirlwind required speed
and storage capabilities far beyond those of any existing machine. As he had
done in the servo lab, Jay built on state-of-the-art theory and applications.
Jay cast his net widely, attending important conferences where leaders in
computing such as John von Neumann and Alan Turing debated fundamen-
tal issues in the design of computing hardware and software, from how num-
bers should be represented, to basic architecture, programming, memory,
input/output devices and more. Jay spent considerable effort mastering prior
work, taking careful notes on books, papers and reports from many disci-
plines and sources, from a monograph on “Primitive Number Systems” to
the design of the proposed Harvard Mark IV (Figure 7).
Jay and the Whirlwind team, including Bob Everett, Jay’s right-hand man
from Whirlwind on, who later became the head of the MITRE Corporation
(MIT Research and Engineering), created dozens of innovations that proved
critical in the development of computing. Jay famously invented core mem-
ory, the first inexpensive, durable memory device for digital computers.
Core memory became the standard memory system for computers for
decades, remaining in use long after transistors and solid-state devices
were developed—for example, landing on the moon with the Apollo
missions.
Whirlwind became the basis for the SAGE air defense system, the first
transcontinental command, control and communication system. Deployed
during the Cold War, SAGE integrated radar and other data to provide real-
time early warning against possible Soviet attack. SAGE’s Whirlwind-based
computers integrated these data, then computed flight plans and targeting
information for interceptor aircraft and missiles and sending those data out
to the field. Once again, Jay and his team built on the rapidly evolving state
of the art in communication theory, computing, radar, manufacturing and
other fields to develop a system to meet the mission. The innovations Jay
and the SAGE team pioneered include “the modem, the mouse [light gun],
multi-tasking, array processing, computer learning, fault detection, magnetic
2
See https://youtu.be/5ZQP4G3Qwb4 for the famous segment of Edward R. Murrow’s See It Now program, in
which Forrester demonstrated Whirlwind, including a simulation model, on live television, 16 December
1951. See https://youtu.be/JZLpbhsE72I?t=1s for a 1980 presentation Jay gave on the history of Whirlwind.
MIT’s collection of Whirlwind material is described at https://libraries.mit.edu/archives/research/collections/
collections-mc/mc665.html.

DOI: 10.1002/sdr
Fig. 6. “Speed-Storage Diagram of Computer Rating” by Forrester, 18 March 1949, showing the capabilities of existing and
planned computers; log–log scale, with “Speed—complete arithmetic operations per second” (x-axis) and “High speed
storage—equivalent binary digits” (y-axis). Dotted lines show the frontier of speed and memory required for different tasks,
from “Scientific and accounting applications” (done by hand or using punch cards; lower left), through “Automatic machines
with program on external tape or plugboard” as in the Mark I, Mark II and ENIAC machines, through “Automatic machines
with program in high speed storage” as in the Mark III and EDVAC, to the “Control and simulation applications” Whirlwind I
was designed to do. Photo: J. Sterman, from Forrester files [Color figure can be viewed at wileyonlinelibrary.com]
memory, and interactive computer graphics” (http://history.sandiego.edu/

GEN/20th/sage.html).
The scale and scope of SAGE were immense. The program cost more
than the Manhattan Project (https://www.ll.mit.edu/about/History/
SAGEairdefensesystem.html). As project director, Jay oversaw all aspects of
design, development and manufacturing, including the work of the four
prime contractors who built the computers and other equipment: IBM, West-
ern Electric, Burroughs, and MIT’s Lincoln Lab Division Six (which became
the MITRE Corporation, headed by Robert Everett). Jay commented:
DOI: 10.1002/sdr
Fig. 7. Two of many hand-written notes Jay took summarizing the literature relevant to computing theory and technology for
the Whirlwind project [Color figure can be viewed at wileyonlinelibrary.com]
… we had control from basic research to military operational planning. We

wrote the contracts between the participating corporations and the Air Force.
We designed the computers with full control over what went into production.
We defended the Air Force’s budget before the Bureau of the Budget because
the technology was so new that it was outside the experience of the military
commands. (Forrester, 1992)
Managing the program provided first-hand insight into the operation and politics
of complex organizations. When Jay joined the MIT Sloan School of Manage-
ment in 1956 he used these insights in developing what became system dynam-
ics. At MIT Sloan, Jay spent the first year or so exploring the opportunities to
make a distinctive contribution. As he had done before in servomechanisms,
Whirlwind and SAGE, he explored new developments in operations research,
economics, management, organization theory, psychology and other disciplines.
He read widely, attended workshops and seminars, met new people, and learned
the latest work in management science. In the first chapter of Industrial Dynam-
ics (ID; Forrester, 1961, p. 14) he surveyed the state of the art, then identified.

DOI: 10.1002/sdr
Four foundations on which to construct an improved understanding of the

dynamics of social organizations …
• The theory of information-feedback systems
• A knowledge of decision-making processes
• The experimental model approach to complex systems
• The digital computer as a means to simulate realistic mathematical models.
Industrial Dynamics presented the results: a novel, innovative method to

develop realistic and useful models of complex organizations. He articulated
principles for model conceptualization, formulation, testing and policy anal-
ysis, and presented tools and models demonstrating how to do it. These
innovations included the DYNAMO simulation language developed by Jack
Pugh and diagramming conventions to represent stocks, flows, feedback pro-
cesses, time delays and other key concepts. They included standards for
model documentation and replicability so others could use and build on the
work. They included subtle and nuanced principles for model formulation
including the importance of “detecting the guiding policy” to capture the
behavioral decision processes actually used (ID, Ch. 10). They included
guidelines for model evaluation and testing. And, as he had done with ser-
vos, Whirlwind and SAGE, he designed system dynamics to be implementa-
ble and useful given the state of the computing technology of the day. He
did not try to build comprehensive models of a firm or economy. Instead, he
built models that could be simulated and analyzed fast enough to be useful
in solving specific problems. Nearly six decades after publication, Industrial
Dynamics remains a classic and essential text in the education of every
modeler.
To summarize: in each of his careers, “from the ranch to system dynam-
ics”, Jay focused on important problems. To do so he mastered the theory
and methods relevant to the problem at hand, then built upon them to create
new innovations needed to solve the problem. His greatest contribution was
that process, not the particular artifacts resulting from it. We do not celebrate
his servo, Whirlwind, core memory, and SAGE because they are the state of
the art today. We celebrate them because they were the result of a process
for problem solving and innovation that was extraordinarily fruitful, time
and time again. The same holds for system dynamics. Just as we would not
go back to the days of core memory and vacuum tubes, of punch cards and
DYNAMO, we should not hold fast to methods for the development and
analysis of system dynamics models that have outlived their useful life and
for which better methods now exist. If Jay were a young man entering the
field of system dynamics today, he would not defend as sufficient nor rely
on methods and tools that were once the state of the art but are no longer.
He would not isolate himself from what other disciplines and communities
offer. Rather, ever focused on solving important problems, he would master
the theories and methods now available, using them where appropriate and
DOI: 10.1002/sdr
building on them to create new tools where needed to propel the field of sys-
tem dynamics to new frontiers. Fortunately, many in the field are following
Jay’s process, connecting with other disciplines, forging links to other com-
munities, using the tools others have developed and then building
upon them.
Model architecture: compartments, agents and networks
Jay’s innovations in the Whirlwind project made it possible for computers to

carry out tasks in real time, including process control and engineering simu-
lations. These innovations also made simulation models of organizations
possible, freeing us from the narrow boundaries and unrealistic simplifying
assumptions needed to yield analytically tractable models. For the first time,
scholars and practitioners could build realistic models of markets, organiza-
tions and other complex systems—models far surpassing the complexity of
prior work.
However, the approach Jay developed must be understood as a product of
the context in which it emerged (Richardson, 1991, provides thoughtful dis-
cussion and contrasts Jay’s approach to other systems approaches). When Jay
created system dynamics, computers were scarce, expensive, hard to use and
snail-slow. Jay, Jack Pugh and others developed tools and guidelines for
model development consistent with the capabilities of computers of the day.
Principal among these was the use of compartment models, in which the enti-
ties and actors relevant to the problem at hand are aggregated into a relatively
small number of stocks. Compartment (or box) models are widely used in
fields from atmospheric chemistry to zoology and remain highly useful today.
However, compartment models cannot represent heterogeneity in members of
a compartment, nor is it possible to track the entry and exit of individual
items; the contents of every compartment are perfectly mixed, so the probabil-
ity that an item will exit the stock in the next instant is independent of how
long the item has been in the stock. Jay recognized the limitations of these
simplifying assumptions, and provided guidelines to help modelers know
when it is appropriate to disaggregate a stock into more categories (stocks) to
capture heterogeneity, and how to model situations in which an item’s exit
from a stock is not independent of its entry date (see, e.g., ID, Ch. 11, and Jay’s
formulations for material and information delays, including the Erlang family
and the pipeline delay, and his analysis of their significance and frequency
response in ID, Ch. 9 and Appendices E, G, H and N). Heterogeneity is com-
monly introduced into compartment models by disaggregating single stocks
into multiple compartments, as, for example, in aging chains or to represent
subsets of the total stock that differ along important dimensions. For example,
capital stock can be disaggregated into plant and equipment, customers can
DOI: 10.1002/sdr
be disaggregated by attributes including preferences, location, etc., with as

many categories as needed for the purpose (Sterman, 2000, ch. 6).
The exponential growth of computing power has led to the development
of new model architectures in which the aggregation assumptions of com-
partment models are relaxed, including agent-based (individual-based)
models (e.g., Epstein, 2007), patch models (e.g., Pickett and White, 1985; see
Struben and Sterman, 2008, for an application in system dynamics) and net-
work models (e.g., Strogatz, 2001). However, many introductory system
dynamics courses continue to teach compartment models only, defending
the high level of aggregation even when inappropriate for the problem at
hand. This is a residue of the time when computers were so slow that highly
disaggregated models of important problems could not be simulated. That
constraint is largely relaxed today, yet too many in the field continue to set
up a false opposition between “system dynamics” and “agent-based”
models. Worse, too many people, inside and outside the field, erroneously
equate “system dynamics” with deterministic compartment models with
continuous values for the state variables and formulated in continuous time,
in contrast to models that represent individual agents, use discrete time, or
include stochastic elements. This is a serious error and a false choice rooted
in a confusion about what system dynamics is.3
The persistence of the myth that system dynamics models are determinis-
tic is particularly puzzling since Forrester’s first system dynamics model
was stochastic, and Jay devoted considerable attention to the proper repre-
sentation of noise in dynamic models, including methods to specify the
power spectrum of random disturbances (ID, ch. 10.7; Appendix F). His dis-
cussion of this modeling choice is subtle and nuanced.
For example, in discussing continuous flows, Jay wrote: “… in formulating
a model of an industrial operation, we suggest that the system be treated, at
least initially, on the basis of continuous flows and interactions of the vari-
ables. Discreteness of events is entirely compatible with the concept of
information-feedback systems …” (Forrester, 1961, p. 64; emphasis added).
He then lists various justifications for continuous flows and aggregated com-
partment models. Some of these address modeling purpose and remain
important. For example, he notes that although “Orders arrive as separate
pieces of paper, … we represent them as a continuous order flow, and our
interest in the model, like the executive’s interest in the company, is from a
viewpoint above the separate individual transactions.” Continuous formula-
tions, he argues, also aid the process of learning from models: “As a starting
point, the dynamics of the continuous-flow model are usually easier to
understand and should be explored before complications of discontinuities
and noise are added” (p. 65). But Jay goes on to warn that:
3
This section is adapted and expanded, with permission, from Rahmandad and Sterman, https://www.
systemdynamics.org/assets/docs/sdorabm.pdf.

DOI: 10.1002/sdr
These comments must not be construed as suggesting that the model builder
should lack interest in the microscopic separate events that occur in a continuous-
flow channel … The study of individual events is one of our richest sources of
information about the way the flow channels of the model should be constructed
… The preceding comments do not imply that discreteness is difficult to represent,
nor that it should forever be excluded from a model. (ID, pp. 65–66).
Other guidelines Jay offers, however, were clearly compromises necessi-

tated by the limitations on computing power and data availability at that
time and are not appropriate today. Regarding delays, for example, he con-
ducted a structural sensitivity test on his supply chain model, varying the
order of the many time delays from first to third to infinite-order (pipeline)
delays. He concluded that for the purpose of the model—understanding the
origin of oscillation and amplification in the supply chain — “the gross over-
all behavior of the system is not greatly changed” but also cautions that.
The effect of the time response [impulse response distribution] of delays may
depend on where the delay is in the system. One should not draw the general
conclusion that the time response does not matter. However, for most of the
incidental delays within a large system we can expect to find that the time
response of the delay is not a critical factor. Third-order exponential delays are
usually a good compromise. (ID, pp. 420–421).
Jay clearly counsels against assuming that the order and specification of
delays does not matter; the third-order delay he used was instead a “compro-
mise” that provided the lowest-order (and thus least computationally demand-
ing) member of the Erlang family with reasonable impulse- and frequency-
response characteristics (specifically, the impulse response of the third-order
delay, like many real-life processes, shows no immediate response to a change
in its input). This compromise was a pragmatic decision dictated by limitations
on computer memory and speed, and on the availability of the data required to
identify and estimate the impulse response distribution. Today, both these con-
straints have been significantly relaxed, and it is often possible to specify the
order, mean delay time and other characteristics of delays that best represent
the process and fit the data (see Sterman, 2000, ch. 11, for examples).
System dynamics models can be implemented using a variety of different
simulation architectures. These vary in their representation of time (continuous
or discrete), state variables (continuous or discrete), and uncertainty (stochastic
or deterministic). Ordinary differential equations, stochastic differential equa-
tions, discrete event simulations, agent-based models and dynamic network
models are common computational architectures offering different choices on
these dimensions. Today, many software programs are available to implement
these architectures, and some allow hybrid models—for example, models that
have compartments (aggregated stocks) for some state variables and individual
DOI: 10.1002/sdr
agents for others. Both compartment and individual-level models can be formu-
lated in continuous or discrete time, with continuous or discrete quantities,
and either can be deterministic or stochastic.
Any mechanism can be specified at various levels of aggregation. For
example, to model the spread of infectious diseases one could use an aggre-
gated compartment model such as the classic SIR model and its variants
(Sterman, 2000, ch. 9), in which all individuals in a community are aggre-
gated into compartments representing the susceptible, infectious and
removed (recovered or deceased) populations. If needed, the model could be
disaggregated into multiple compartments to represent any relevant dimen-
sions of heterogeneity, such as age, gender, immune system status, location
(at multiple scales, from nation or province to postal code or even finer), or
by patterns of activity that determine contacts with others. High-impact
examples include the disaggregated compartment models developed by
Thompson, Tebbens and colleagues and used by WHO and CDC to design
policy to support the Global Polio Eradication Initiative (Thompson and Teb-
bens, 2007, 2008; Thompson et al., 2015; Tebbens and Thompson, 2018, this
issue). Alternatively, one could capture aspects of the contact networks
among individuals by considering their pair-wise interactions
(as demonstrated for system dynamics applications by Lamberson, 2018, this
issue, building on Keeling et al., 1997; see also Lamberson, 2016). Or one
could move to an individual-based model in which each individual is repre-
sented separately. Where the SIR compartment model has three states, the
individual-level SIR model portrays each individual as being either suscepti-
ble, infectious, or removed, and could potentially include heterogeneous
individual-level attributes including location, travel patterns, social net-
works, and others that determine the hazard rates of state transitions (see,
e.g., Waldrop, 2018, and the work of the MIDAS Group (Models of Infectious
Disease Agent Study), https://www.epimodels.org/). Obviously, individual-
level (agent) models include stocks, flows and feedback loops, just as com-
partment models do. Where the stock of susceptible individuals in the com-
partmental SIR model is a single state variable and is reduced by the aggregate
flow of new cases (infection), the stock of susceptible individuals in the agent-
based model is the sum of all those currently susceptible, and the aggregate
flow of new cases is the sum of those becoming infected. In the agent-based
model the hazard rate that each susceptible individual becomes infected is
determined by the rate at which each comes into contact with any infectious
individuals and the probability of infection given contact, closing the reinfor-
cing feedback by which the contagion spreads. In the compartment model the
hazard rate of infection is the same for all individuals, while it can differ
across individuals in the agent-based model. The feedback structure, how-
ever, is the same (see Rahmandad and Sterman, 2008, for an explicit compari-
son of individual and compartment models in epidemiology).

DOI: 10.1002/sdr
Similarly, recent developments now allow models to capture dynamics at

multiple timescales, something Forrester generally avoided. Forrester argued
that the purpose of a model would determine the timescale of interest, writ-
ing, “For a simple model to be used over a short time span, certain quantities
can be considered constants that for a model dealing with a longer time hori-
zon should be converted into variables. Those new variables would in turn
depend on more enduring parameters” (Forrester, 1980, p. 559). Likewise,
variables that change very quickly relative to the time horizon of the model
can be assumed to take their equilibrium values. For example, a model of
resource dynamics with a time horizon of a century or more need not con-
sider the rapid fluctuations in inventories of processed natural resources that
give rise to commodity price cycles. For many purposes the assumption of
weak couplings across timescales is acceptable, but for a number of important
problems fast and slow dynamics are inextricably coupled. Computational con-
straints when Forrester first formulated guidelines for system dynamics made it
difficult, if not impossible, to capture both fast and slow dynamics in a single
model: doing so increased simulation times unacceptably and potentially leads
to instability from errors in numerical integration; the “stiff system problem”
(e.g., Press et al., 2007, ch. 17.5). Today methods for multiscale modeling, where
submodels operating at different temporal (and spatial) scales are integrated, are
advancing rapidly and used in problems from quantum chemistry to climate
change (e.g., Hoekstra et al., 2014). Ford (2018, this issue), illustrates methods to
couple fast and slow dynamics in system dynamics models, with practical exam-
ples in the electric power industry where hourly, daily and seasonal variations
in load and generation (particularly with the rise of variable distributed renew-
ables such as wind and solar) interact importantly with slow dynamics around
the evolution of generation capacity and end-user capital stocks.
There is no “correct” level of aggregation independent of model purpose.
In the epidemiology example, it may seem obvious that the individual level
is more correct than the aggregated level. But there is nothing uniquely privi-
leged about the individual level: one could, in principle, further disaggregate
such a model to capture the different organs and systems within each indi-
vidual, or even further, to the cellular level. But why stop there? Cells and
the viruses that attack them consist of still smaller structures (organelles,
lipids, proteins and other molecules). Molecules consist of atoms; atoms of
protons, neutrons and electrons; protons and neutrons of quarks.... At the
other end of the spectrum, the purpose of the model may require a more
aggregated approach because the data needed to identify structure and esti-
mate parameters for a highly disaggregated model may not be available.
Pragmatic considerations also sometimes favor aggregated models. The com-
putational burden and long simulation times of highly disaggregated models
can limit the ability to carry out sensitivity analysis (both parametric and
structural), quickly change model structure and parameters when the situa-
tion is evolving rapidly, or enable key stakeholders to design and test
DOI: 10.1002/sdr
assumptions and policies for themselves to build their confidence in the

model and motivate the implementation of model-based recommendations.
Climate models provide a good illustration. To advance basic scientific
understanding of climate dynamics and the effects of greenhouse gases on
global warming, scientists develop models of immense complexity. These
General Circulation Models (GCMs) divide the atmosphere into a three-
dimensional lattice and then simulate, for each cell in the lattice, the equa-
tions of motion governing flows of energy, moisture and other physical quan-
tities across cells, including momentum, mass, heat and radiative energy
transfer, similar to the models used so successfully in short- and medium-
range weather forecasting. Modern climate models couple atmospheric
dynamics to the ocean, land, and biosphere and are known as coupled
AOGCMs (Atmosphere–Ocean GCMs). As computing power has grown, the
distance between lattice points and the time step for simulation have shrunk,
increasing model accuracy; current models use a grid size of the order of 1
of latitude and longitude and have 20 or more layers for the atmosphere; the
resulting number of cells is on the order of millions, each computed every
time step. Although these models run on some of the largest supercomputers
in the world it can take days, weeks or more to run a single simulation.4
Thus, while AOGCMs are essential for progress in basic scientific under-
standing, they are not useful for the rapid policy design, sensitivity analysis
and interactive learning that can support policymakers, negotiators, and edu-
cators. Therefore, Climate Interactive and MIT developed the C-ROADS cli-
mate policy model (Fiddaman, 2002, 2007; Sterman et al., 2012, 2013;
Sterman, 2015; Holz et al., 2018; http://climateinteractive.org). C-ROADS is
a member of the family of Simple Climate Models (SCMs), consisting of
low-dimensional compartment models of the carbon cycle and climate (see
also the ESCIMO model; Randers et al., 2016). C-ROADS captures the emis-
sions, other fluxes and atmospheric stocks of CO2, methane, nitrous oxide,
many species of fluorinated hydrocarbons and other greenhouse gases, along
with the fluxes of carbon and heat among the atmosphere, biosphere and a
four-layer ocean. C-ROADS also models climate change impacts including
ocean acidification and sea level rise. Without the fine detail of the
AOGCMs, SCMs must be parameterized to match both the historical data
and output of ensembles of simulations of multiple AOGCMs across a wide
range of scenarios for future greenhouse gas emissions; Sterman et al. (2013)
provides a quantitative assessment of model fit for CO2 and CH4 concentra-
tions, global mean surface temperature anomaly, and sea level rise, with low
measures of error (e.g., root mean square error, Theil inequality statistics). C-
ROADS thus matches the aggregate behavior of the AOGCMs but runs in less
than 1 second on ordinary laptops. Users can design policies, vary
4
Edwards (2010) provides a compelling history of weather and climate modeling and the coevolution of these
models with ever-finer and more extensive data collection.

DOI: 10.1002/sdr
parameters, and try any experiments they like and receive feedback immedi-
ately, thus enabling them to learn for themselves and assess new proposals
at the cadence required in negotiations and educational experiences (Rooney
Varga et al., 2018).
The capabilities and speed of computers today have greatly relaxed the
constraints on model size and level of aggregation compared to the 1950s,
when Forrester created system dynamics. But constraints still exist, and
always will. No matter how fast computers become, comprehensive models
of a system are impossible. As Mihailo Mesarovic, a developer of early global
simulations, noted: “No matter how many resources one has, one can envi-
sion a complex enough model to render resources insufficient to the task”
(Meadows et al., 1982, p. 197). Modelers should choose the simulation archi-
tecture and level of aggregation most appropriate for the purpose of the
model—that is, best suited to solve the problem. Typically, aggregate models
are easier to represent in differential equation frameworks (compartment
models, whether continuous or discrete, deterministic or stochastic), while
disaggregation to capture important degrees of heterogeneity across members
of the population may call for an individual-based architecture (again, deter-
ministic or stochastic). And just as one should carry out sensitivity analysis
to assess the impact of parameter uncertainty, structural sensitivity tests
assessing the similarities and differences across levels of aggregation and
model architectures are needed. For example, Rahmandad and Sterman
(2008) compared continuous, aggregated compartment models to individual-
based models in the context of epidemiology and the diffusion of infectious
diseases (see also Lamberson, 2018, this issue).
In sum, the goal of dynamic modeling is sometimes to build theoretical
understanding of complex dynamic systems, sometimes to implement policies
for improvement, and often both. To do so, system dynamics modelers seek to
include a broad model boundary that captures important feedbacks relevant to
the problem to be addressed; represent important structures in the system
including accumulations and state variables, delays and nonlinearities; use
behavioral decision rules for the actors and agents, grounded in first-hand
study of the relevant organizations and actors; and use the widest range of
empirical data to specify the model, estimate parameters, and build confidence
in the results. System dynamics models can be implemented using a wide
range of methods, including differential equations, difference equations or dis-
crete event simulation; aggregated or disaggregated compartment models or
individual-based models; deterministic or stochastic models; and so on.
Asking “Should I build a system dynamics or agent-based model?” is like
going to the hospital after an accident and having the doctor say “Would
you like to be treated or do you want surgery? The choice is not between
treatment in general and any particular treatment, but among the different
treatments available. You seek the best treatment for your condition, and
you expect doctors to learn and use the best options as new ones become
DOI: 10.1002/sdr
available. In the same way, computational capabilities today have expanded

the options for dynamic modeling. Good modelers choose the model archi-
tecture, level of aggregation and simulation method that best meet the pur-
pose of the study, taking account of computational requirements, data
availability, the time and resources available, the audience for the work, and
the ability to carry out sensitivity analysis, understand the behavior of the
model, and communicate the results and the reasons for them to the people
they seek to influence.
Evidence-based modeling: qualitative and quantitative data

sources
Forrester argued early and correctly that data are not only numerical data
and that “soft” (unquantified) variables should be included in our models if
they are important to the purpose. Jay noted that the quantified data are a
tiny fraction of the relevant data needed to develop a model and stressed the
importance of written material and especially the “mental data base” con-
sisting of the mental models, beliefs, perceptions and attitudes of the actors
in the system. His emphasis on the use of unquantified data was, at the time,
unusual, if not unique, among modelers in the management sciences and eco-
nomics, some of whom attacked his position as unscientific. Jay’s response:
“To omit such variables is equivalent to saying they have zero effect—
probably the only value that is known to be wrong!” (ID, p. 57). Omitting
structures or variables known to be important because numerical data are
unavailable to specify and estimate them is less scientific and less accurate
than using expert judgment and other qualitative data elicitation techniques
to estimate their values. Omitting important processes because we lack
numerical data to quantify them leads to narrow model boundaries and biased
results—and may lead to erroneous or harmful policy recommendations.
However, acknowledging the importance of qualitative data begs the ques-
tion of how these variables and effects can be identified and tested. Jay’s
approach emphasized observation and interviews to capture the “mental
data base” of the people in the system:
One can go into a corporation that has serious and widely known difficulties.
The symptom might be a substantial fluctuation of employment with peaks sev-
eral years apart. Or the symptom might be falling market share.... In the process
of finding valuable insights in the mental data store, one talks to a variety of
people in the company, maybe for many days, possibly spread over many
months. The discussion is filtered through one’s catalog of feedback structures
into which the behavioral symptoms and the discussion of structure and policy
might fit. The process converges toward an explicit simulation model. The poli-
cies in the model are those that people assert they are following; in fact, the

DOI: 10.1002/sdr
emphasis is often on the policies they are following in an effort to alleviate the
great difficulty. (Forrester, 1980, p. 560; emphasis added).
Jay’s approach is often oversimplified by newcomers to the field as just

“talking to people”. In fact, he recommended that modelers embed them-
selves in the system, similar to the methods of the ethnographer or cultural
anthropologist: “first-hand knowledge can be obtained only by living and
working where the decisions are made and by watching and talking with
those who run the economic [or any] system” (Forrester, 1980, p. 557).
Still, Jay was confident that he could “detect the guiding policy” through
observation and interviews, writing, “We usually start already equipped
with enough descriptive information to begin the construction of a highly
useful model” (ID, p. 57). However, Jay’s years of experience managing com-
plex programs such as Whirlwind and SAGE gave him extensive first-hand
knowledge and deep insight into organizations, enabling him to be an effec-
tive interviewer. Few of us have such knowledge. Far more important, today
we know that there are many pitfalls and biases in the elicitation of mental
models from subject matter experts or the actors in a system.
Jay recognized one important bias in information gained from interviews
and observation. He noted that the mental data base includes three types of
data: (i) observations about the structure of the system and the policies gov-
erning system flows; (ii) actual observed system behavior; and
(iii) expectations about the behavior of the whole system. He recognized that
people’s expectations about system behavior were not reliable: they “repre-
sent intuitive solutions to the nonlinear, high-order systems of integral equa-
tions that reflect the structure and policies of real systems. Such intuitive
solutions to complicated dynamic systems are usually wrong” (Forrester,
1980, p. 556). Subsequent research revealed that Jay was correct, but wildly
overoptimistic: a large body of work shows not only that people do poorly
anticipating the behavior of complex systems, but that even highly educated
people with significant training in STEM fields (science, technology, engi-
neering or mathematics) cannot correctly identify the behavior of even a sin-
gle stock from knowledge of its flows, without any feedbacks, time delays,
nonlinearities or other attributes of complex systems (Booth Sweeney and
Sterman, 2000; Cronin et al., 2009).
Although he recognized people’s poor mental simulation abilities, Jay did
assert, however, that information gleaned from interviews relating to struc-
ture and policies, and about the actual behavior of the system, was reliable:
“… mental information—information about policy and structure—is directly

tapped for transfer into a system dynamics model. In general, the mental data
base relating to policy and structure is reliable. Of course, it must be cross-
checked. Exaggerations and oversimplifications can exist and must be cor-
rected. Interviewees must be pressed beyond quick first responses. Interrogation

DOI: 10.1002/sdr
must be guided by a knowledge of what different structures imply for dynamic

behavior. But from the mental data base, a consensus emerges that is useful and
sufficiently correct.” (Forrester, 1980, p. 556)
Research since Jay’s pioneering studies shows that mental models are
often unreliable and biased even regarding policy, structure and actual
behavior, and that verbal accounts, even carefully elicited and cross-
checked, often do not reveal the causal structure of a system, the decision
processes of the informant or other actors, or even accurate accounts of
events and behavior. Today we know that many judgments and decisions
arise rapidly, automatically, and unconsciously from so-called “system 1”
neural structures, in contrast to the slow, “system 2” structures underlying
effortful, conscious deliberation (e.g., Sloman, 1996; Kahneman, 2011. Lakeh
and Ghaffarzadegan, 2016, explore implications for system dynamics). Per-
ceptions, judgments and decisions arising from system 1 are often systemati-
cally wrong, with examples ranging from optical illusions to violations of basic
rules of logic to stereotyping and the fundamental attribution error to the dozens
of “heuristics and biases” documented in the behavioral decision-making and
behavioral economics literature (e.g., Gilovich et al., 2002).
Furthermore, when asked to provide explanations for our beliefs, judg-
ments and decisions, we provide accounts that suffer from post hoc rationali-
zation and align with what we consider to be plausible, based on prior, often
implicit, and often erroneous mental models. These errors apply not only to
accounts of unconscious, system 1 processes, but to deliberative processes as
well: we often have “little or no direct introspective access to higher order
cognitive processes”, “telling more than we can know” (Nisbett and Wilson,
1977). Memory is fallible and eyewitness testimony is often unreliable, as
documented in a comprehensive National Academies review, which found:
Gaps in sensory input are filled by expectations that are based on prior
experiences with the world. Prior experiences are capable of biasing the visual
perceptual experience and reinforcing an individual’s conception of what was
seen.... [P]erceptual experiences are stored by a system of memory that is highly
malleable and continuously evolving.... The fidelity of our memories to actual
events may be compromised by many factors at all stages of processing, from
encoding to storage to retrieval. Unknown to the individual, memories are forgot-
ten, reconstructed, updated, and distorted. (National Research Council,
2014, pp. 1–2).
Worse, we are all subject to implicit (unconscious) racial, gender and other
biases (Greenwald and Banaji, 1995). These and other errors and biases
afflict lay people and experts alike. Indeed, experts are often more vulnera-
ble to them. Overconfidence bias provides a telling example: people tend to
be overconfident in their judgments, significantly underestimating
DOI: 10.1002/sdr
uncertainty. In many settings, training and expertise do not improve accu-

racy (e.g., Sanchez and Dunning, 2018). Further, “Some epistemic vices or
cognitive biases, including overconfidence, are ‘stealthy’ in the sense that
they obstruct their own detection. Even if the barriers to self-knowledge can
be overcome, some problematic traits are so deeply entrenched that even
well-informed and motivated individuals might be unable to correct them.
One such trait is overconfidence” (Cassam, 2017, p. 1).
Interviewers are subject to the same errors and biases as their subjects.
Bias can arise because the sample of informants selected is not representa-
tive; social pressures often lead people to tell interviewers what they believe
the interviewer wants to hear or what they believe will enhance their status
in the eyes of the interviewer; talking to people in groups can lead to group-
think and silencing of individuals with minority views or views diverging
from those of superiors and authority figures (e.g., Janis, 1982; Perlow and
Repenning, 2009, develop a system dynamics model showing how silencing
can be self-reinforcing).
Psychologists, sociologists, ethnographers and other social scientists have
developed methods to identify and mitigate these biases, both in sample
selection and protocols for data elicitation. Sample selection techniques
include both probability and non-probability (purposive) methods, including
random, stratified and cluster sampling, maximum variation, typical case,
critical case, and snowball (respondent-based) sampling methods, among
others. Each has strengths and weaknesses. For example, stratified methods
are useful when a representative sample of the underlying population is
needed, and snowball sampling is particularly useful to identify social net-
works and “hidden” populations such as individuals who fear authority
(e.g., drug users). With any method one must work carefully to capture
important subpopulations whose omission would bias the results—for exam-
ple, interviewing current employees but not those who quit, or current cus-
tomers and not those who switched to competitors.
Protocols for eliciting people’s knowledge, beliefs and preferences include
interviews, ethnographic study, survey methods, focus groups, conjoint anal-
ysis, nominal group technique, concurrent verbal protocols, and many others
(the literature is vast; see, e.g., Ericsson and Simon, 1993; Miles et al., 2014;
Agarwal et al., 2015; Turco, 2016; Glaser and Strauss, 2017). Such methods
have been used in system dynamics to, among others, elicit customer prefer-
ences (e.g., Sterman, 2000, ch. 2; Barabba et al., 2002), complement statisti-
cal analysis of decision making in experimental studies (Kampmann and
Sterman, 2014), understand medical decision making (Rudolph et al., 2009)
and to elicit reliable estimates of nonlinear relationships from managers
(Ford and Sterman, 1998); Minyard et al. (2018, this issue) illustrate by
evaluating the effectiveness of the ReThink Health simulation model
(https://www.rethinkhealth.org) in catalyzing health system innovation at
the community level.
DOI: 10.1002/sdr
Formal methods to extract meaning and underlying themes and constructs

from data include factor analysis, sentiment analysis and other statistical
techniques. Formal coding of data is often important and should be done by
multiple coders who are blind to the hypotheses of the researcher, with
inter-rater reliability assessed and reported. Keith et al. (2017) provide a
good example in the context of gathering data for a system dynamics model,
using multiple coders to estimate the existence and length of wait lists for
the Toyota Prius from press reports, then using the resulting data to test
hypotheses about how wait lists affect product adoption. (Do wait lists frus-
trate potential buyers and slow diffusion, or create buzz and excitement that
increases demand?)
Triangulation—using multiple data sources and research methods, both
quantitative and qualitative—provides important checks on reliability and
helps identify gaps and biases that any one method cannot. System dynam-
ics is particularly well suited to the use of multiple methods; see, for exam-
ple, Barabba et al. (2002) for an application in business strategy, and
Sterman et al. (1997), who integrated quantitative time series and cross-
sectional data at several levels of aggregation with interviews, documents
and other qualitative data in a formal model exploring the unintended con-
sequences of a successful quality improvement program at semiconductor
firm Analog Devices.
Many of the techniques discussed above did not exist when Forrester
wrote about the importance of qualitative data. Progress since then in psy-
chology, ethnography, sociology, marketing and other fields has built
methods to elicit and analyze qualitative data that modelers in all fields can
use to generate reliable, useful knowledge. Convenience samples and just
“talking to people” are no longer acceptable.
As important as qualitative methods and data remain, it is important to
use proper statistical methods to estimate parameters and assess the ability
of a model to replicate historical data when numerical data are available—
and to find ways to measure when they aren’t. Unfortunately, some modelers
discount the role of numerical data and formal methods to estimate model
parameters and assess the ability of a model to replicate historical data. They
argue that qualitative insights are more important than numerical precision.
This practice is often justified as being consistent with Forrester’s emphasis
on the importance of qualitative data. It is, however, a serious error. Ignoring
numerical data and failing to use statistical tools when appropriate increases
the chance that the insights derived from a model will be wrong or harmful.
Rigorously defining constructs, attempting to measure them, and using the
most appropriate methods to estimate their magnitudes are important anti-
dotes to casual empiricism, muddled formulations, and the erroneous
insights we often draw from our mental models. Forrester’s comment,
quoted above, that omitting structure and processes because numerical data
to estimate their effects are unavailable assumes “they have zero
DOI: 10.1002/sdr
effect—probably the only value that is known to be wrong!” is widely cited

by those who fail to do the hard work of using data appropriately. Less
widely cited is his caution that:
These comments are not to discourage the proper use of the data that are avail-
able nor the making of measurements that are shown to be justified … Lord Kel-
vin’s famed quotation, that we do not really understand until we can measure,
still stands. (ID, p. 59).
The quotation from Lord Kelvin (the physicist William Thomson) to which
Jay refers is:
a first essential step in the direction of learning about any subject is to find prin-
ciples of numerical reckoning and methods for practicably measuring some
quantity connected with it.... when you can measure what you are speaking
about, and express it in numbers, you know something about it; but when you
cannot measure it, when you cannot express it in numbers, your knowledge is
of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but
you have scarcely, in your thoughts, advanced to the stage of science, whatever
the matter may be. (Thomson, 1883, p. 73; emphasis in original).
As an engineer ever focused on practicalities, Forrester then adds an

important note:
But before we measure, we should name the quantity, select a scale of measure-
ment, and in the interests of efficiency we should have a reason for wanting to
know. (ID, p. 59).
To develop reliable knowledge, we must bring the best available data to

bear for the problem we seek to address. Following Jay’s advice, we must
find a way to measure what we need to know. We should not accept the
availability of numerical data as given, outside the boundaries of our project
or research. Once we recognize the importance of a concept, we can almost
always find ways to measure it. Within living memory there were no
national income accounts, no survey methodologies to assess political senti-
ment, no psychological inventories for depression or subjective well-being,
no protocols for semi-structured interviews or coding criteria for ethno-
graphic data. Today many apparently soft variables such as customer per-
ceptions of quality, employee morale, investor optimism, and political
values are routinely quantified with tools such as surveys, conjoint analysis
and content and sentiment analysis. Of course, all measurements are imper-
fect. Metrics for so-called soft variables continue to be refined, just as metrics
for so-called hard variables are. Often the greatest benefit of a modeling
DOI: 10.1002/sdr
project is to identify the importance of and then measure important concepts

previously ignored or unquantified.
To illustrate, consider Homer’s (1987, 2012) model of the diffusion of new
medical technologies. The model has a broad boundary and significant struc-
ture far beyond the classic Bass diffusion model (Bass, 1969; Sterman, 2000,
ch. 9), including endogenous adoption and disadoption by doctors, market-
ing, use (including initial and repeat purchases), patient eligibility criteria,
technological improvement, patient outcomes both beneficial and harmful,
and evaluation of safety and efficacy. Homer tested the model with two case
studies exhibiting very different adoption/diffusion patterns, the cardiac
pacemaker and the broad-spectrum antibiotic clindamycin (a good example
of the contrasting case method). In each case Homer sought the widest range
of data, using triangulation to help specify the model and estimate parame-
ters and relationships. For the pacemaker, this meant extensive interviews
with clinicians, researchers, and representatives of pacing companies,
including many pacing pioneers. Homer even attended two pacemaker
implantation procedures, donning the required lead vest to avoid fluoro-
scope radiation exposure.
The pacemaker was and remains one of the most successful medical inno-
vations, expanding from the first implantation in 1960 to more than 350,000
per year by 1980 in the U.S.A. alone. Growth was steady and smooth
(Figure 8). As the patient base for pacemakers expanded to include milder
arrhythmias, the need for follow-up studies assessing benefits and risks
grew. However, at the time, no data were available reporting the number of
follow-up cases published over time. Given the smooth growth in patients
receiving pacemakers it seemed reasonable to assume that follow-up studies
grew smoothly as well and build that assumption into the model. None of
the experts Homer interviewed suggested anything else. Homer, however,
believed it was necessary to find out. At that time (the early 1980s) there was
Fig. 8. Pacemaker
recipients, U.S.A.,
1960–1980. Source:
Homer (1987, 2012)
[Color figure can be
viewed at

DOI: 10.1002/sdr
no Internet or electronic databases to automate the search of medical jour-

nals. To gather the data, Homer spent days in the Harvard Medical School
library stacks, examining, by hand, every issue of the relevant cardiology
journals from 1960 through 1980. He coded every follow-up study reported,
tabulating the number of patients reported in each, the medical conditions
making them eligible for pacing, and the outcomes. The work was painstak-
ing and slow. But it yielded data no one had previously compiled, including
time series showing the number of follow-up cases published per year and
how eligibility criteria for pacing had expanded over time.
Contrary to intuition and what might be expected from the fact that none
of the experts Homer interviewed suggested anything else, evaluations did
not exhibit smooth growth. Instead, there was a pronounced cycle in follow-
up reporting, with a period of about 5 years (Figure 9). This surprising out-
come required Homer to expand the model structure to include a delay in
the execution and publication of follow-up studies and to model the
behavioral decision rule governing the initiation of studies (Figure 10).
Researchers and clinicians initiate follow-up studies when they perceive that
the published evaluative data fall short of the amount needed to properly
assess the safety and efficacy of pacing for particular subsets of patients. But
it takes time to propose, gain approval and funding, initiate, complete and
publish these follow-up studies. Researchers and journal editors generally
do not know how many evaluations are in the “supply chain” of research
being done by all relevant investigators and journals when they decide
whether to initiate or publish a new study. The result, as in classic supply-
chain models (Forrester, 1961; Sterman, 1989), is overshoot and oscillation,
as the initiation of new studies continues even after sufficient studies are in
the pipeline to provide the needed data.
Armed with the quantitative data, Homer could estimate the relationship
between the size of the pacing-eligible population and the number of
Fig. 9. Publication rate of

evaluative studies of
cardiac pacing (patients/
year), U.S.A., 1960–1980.
Source: Homer (1987,
2012) [Color figure can be
viewed at

DOI: 10.1002/sdr
Fig. 10. Structure for Mean Follow-Up

initiation and publication Publication Delay
of follow-up studies, with
lag between study Potential
initiation and publication. Pacemaker
Follow Up Follow Up Recipients
Based on Homer (1987, Studies in
Study Progress Reports
2012) [Color figure can be Study +
viewed at Initiation Publication
Required Sample
wileyonlinelibrary.com] + Size for Reliable
B Assessment
Follow-Up
+
Desired Follow
Adequacy of Up Reports
-
Reports
+
follow-up cases required for reliable inferences about safety and efficacy.
Basic sampling theory suggests that the required sample size for follow-up
studies would scale with the square root of the pacing-eligible population.
Homer found an excellent fit to the data (Figure 11). To estimate the nature
and length of the publication delay, note that the multi-stage nature of the
follow-up supply chain (initiation, approval, funding, execution, and so on)
suggests a high-order delay, but Homer did not merely assume this to be the
case; instead, he experimented with different delay types, finding that a
third-order delay with a mean delay time of 1.25 years fit the data best
(Figure 12).
Fig. 11. Data and

estimated relationship
between population of
pacing-eligible
arrhythmia patients and
the number of follow-up
reports published to date.
Reports to date (cases)
Based on Homer (1987,

viewed at
Best fit:
Reports = 100 + 700log2(EF/IEF)
Eligibility Fraction/Initial Eligibility Fraction

DOI: 10.1002/sdr
Fig. 12. Simulated and

actual publication of
follow-up reports, using
third-order publication
delay with mean delay
time of 1.25 years.
Source: Homer (1987,
viewed at
The oscillations in evaluations Homer found are not merely a curiosity but
have significant policy implications. During troughs in publications, follow-
up data became increasingly dated, providing a poor guide to the benefits
and risks as pacemakers improve and new patient populations become eligi-
ble for pacing. During such times, clinicians would be flying blind, perhaps
being too aggressive and placing at risk certain new patients who would ben-
efit little; or perhaps being too conservative and failing use pacing for others
who could benefit substantially.
Policy analysis with the model led Homer to endorse the use of large clini-
cal registries for new medical technologies as a supplement to randomized
clinical trials and evaluative studies. Registries, typically overseen by federal
agencies such as the National Institutes of Health, require clinicians to con-
tinuously report outcomes for their patients. Registries generate a steady flow
of information and shorten the delay in the evaluation process, thus keeping
up with changes in the technology and its application. They are widely used
today, having proven helpful in the early identification of harmful side
effects and unexpected benefits for certain patient subsets.
The lesson is clear: it would have been easy for Homer to rely on “com-
mon sense” and assume that the publication of follow up studies followed
the smooth growth of pacing. Instead, his careful empirical work revealed a
surprising phenomenon with important implications (see also Homer, 1996).
Parameter estimation and model analysis
When Forrester first formulated principles for system dynamics practice,

numerical data were scarce and many of the tools described above for quali-
tative data elicitation and quantified measurement did not exist.
DOI: 10.1002/sdr
Furthermore, the tools for statistical estimation of parameters were not well
developed compared to today and often inappropriate for use in complex
dynamic systems. Given the limitations of computing at the time, multiple
linear regression was the most widely used econometric tool. However, lin-
ear regression and related methods (e.g., ANOVA) impose assumptions that
are routinely violated in complex dynamic systems. These include perfect
specification of the model, no feedback between the dependent and indepen-
dent variables, no correlations among the independent variables, no mea-
surement error, and error terms that are i.i.d. normally distributed. And, of
course, classical multiple regression and similar statistical methods only
revealed correlations among variables and could not identify the genuine
causal relationships modelers sought to capture.
Given these limitations, Jay and other early modelers, including econo-
mists, sociologists and others, faced a dilemma: they could estimate parame-
ters and relationships in their models using econometric methods and
provide quantitative measures of model fit to the data, but at the cost of con-
straining the structure of their models to the few constructs for which
numerical data existed and using estimation methods that imposed dubious
assumptions; or they could use qualitative and quantitative data to build
models with broad boundaries, important feedbacks, and nonlinearities, at
the cost of estimating parameters by judgment and expert opinion when sta-
tistical methods were impossible or inappropriate. Forrester and early sys-
tem dynamics practitioners opted for the latter. Much controversy in the
early years surrounded the different paths modelers trained in different tra-
ditions chose when faced with these strong tradeoffs.
Today, data have never been more abundant. Methods to reliably measure
previously unquantified concepts, unavailable to Jay, are now essential. Rig-
orous data collection, both qualitative and quantitative, opens up new
opportunities for formal estimation of model parameters, relationships and
structure, to assess the ability of models to replicate the historical data, and
to characterize the sensitivity of results to uncertainty, providing more reli-
able and useful insights and policy recommendations. Rahmandad et al.
(2015) provide an excellent survey of these methods, with examples and
tutorials.
System dynamics modelers have long sought methods to test models and
assess their ability to replicate historical data (e.g., Barlas, 1989, 1996; Ster-
man, 2000, ch. 21; Saysel and Barlas, 2006; Yücel and Barlas, 2015, provide
an overview of many tests to build confidence in dynamic models, including
their ability to fit historical data, robustness under extreme conditions, and
generalizability). But system dynamics modelers were not the only ones con-
cerned with the limitations of early statistical methods. Engineers, statisti-
cians and econometricians worked to develop new methods to overcome the
limitations of traditional methods. By the late 1970s, control theory methods
designed to handle feedback-rich models with measurement error and
DOI: 10.1002/sdr
multicollinearity, such as the Kalman filter, were introduced into system

dynamics (Peterson, 1980). Faster and more capable computing spurred the
development of methods to estimate parameters in nonlinear systems, avoid
getting stuck on local optima in parameter space, deal with autocorrelation
and heteroscedasticity, and especially model misspecification and causal
identification (see below). These methods include indirect inference
(Gourieroux et al., 1993; Hosseinichimeh et al., 2016), the simulated method
of moments (McFadden, 1989; Jalali et al., 2015), Generalized Model Aggre-
gation (Rahmandad et al., 2017) and extended Kalman and particle filters
(Fernández-Villaverde and Rubio-Ramírez, 2005). Bootstrapping and sub-
sampling (see, e.g., Politis and Romano, 1994; Dogan, 2008; Struben et al.,
2015) and Markov chain Monte Carlo and related Bayesian methods (Ter
Braak, 2006; Vrugt et al., 2009; Osgood and Liu, 2015) became feasible for
models of realistic size and complexity, enabling modelers to estimate
parameters and the confidence intervals around them without making unre-
alistic assumptions about the properties of the models and error terms.
Methods for sensitivity analysis that were difficult or impossible to use when
Forrester first formulated system dynamics can now be run quickly even on
large models. These methods include standard multivariate Monte Carlo,
Latin hypercube, and others. Methods for formal analysis of model behavior
have also been developed so that understanding the origin of the behavior in
a complex model is no longer a matter of intuition or trial and error
(Mojtahedzadeh et al., 2004; Kampmann and Oliva, 2009; Saleh et al., 2010;
Kampmann, 2012; Oliva, 2015, 2016; Rahmandad et al., 2015, Part II). New
methods for policy optimization have also been developed (see Rahmandad
et al., 2015; Vierhaus et al., 2017, Part III).
System dynamics models seek to capture the causal structure of a system
because they are used to explore counterfactuals such as the response of a
system to new policies. Distinguishing genuine causal relationships from
mere correlations remains a critical issue. The gold standard for causal attri-
bution is experiment, particularly the RCT (Randomized Controlled Trial).
However, in many settings, particularly in the human and social systems
where system dynamics is often used, RCTs and experiments are often pro-
hibitively expensive, time consuming, unethical or simply impossible. We
have only one Earth and cannot compare our warming world to a control
world in which fossil fuels were never used; we cannot randomly assign half
the citizens of the U.S.A. to a world where those convicted of first-degree
murder are executed and half to a world without executions to determine
whether capital punishment deters murder. Nevertheless, there has been
dramatic progress in tools and methods to carry out rigorous RCTs in social
systems. These include methods to enhance replicability and avoid “p-hack-
ing” (cherry picking samples and models to get statistically significant
results; see Simmons et al., 2011) and the “replication project” (Open Sci-
ence Collaboration, 2015). RCTs are also no longer restricted to the
DOI: 10.1002/sdr
laboratory but include many large-scale field experiments (Duflo and Bane-
rjee, 2017). Many RCTs in social systems today are carried out in difficult
circumstances and provide important insights into policies to reduce poverty
and improve human welfare (see, e.g., the work of MIT’s Poverty Action Lab,
https://www.povertyactionlab.org).
System dynamics has long used experimental methods (Sterman, 1987,
1989), and experimental research in dynamic systems is robust (e.g., Booth
Sweeney and Sterman, 2000; Moxnes, 2001, 2004; Croson and Donohue,
2005; Kopainsky and Sawicka, 2011; Arango et al., 2012; Gonzalez and
Wong, 2012; Kampmann and Sterman, 2014; Lakeh and Ghaffarzadegan,
2015; Özgün and Barlas, 2015; Sterman and Dogan, 2015; Villa et al., 2015;
Gary and Wood, 2016). Many of these studies involve individuals or small
groups and have provided important insights into how people understand,
perform in and learn from experience in dynamic systems. Still, many set-
tings involve group decision-making. Experimental work involving groups
offers important opportunities; for example, McCardle-Keurentjes et al.
(2018, this issue), who report an experiment evaluating whether system
dynamics tools such as causal diagrams improved performance of individ-
uals in a group model building workshop. Extending experimental methods
to group behavior and large-scale field settings is a major opportunity to con-
duct real-world tests of the theories and policies emerging in the broad
boundary, feedback-rich models typical in system dynamics.
Despite progress in methods for and the scope of experimental methods,
experimentation remains impossible in many important contexts. Absent rig-
orous RCTs, researchers often seek to estimate causal structure and relation-
ships via econometric methods. Although we cannot randomly assign
murderers to execution versus prison to assess the deterrent effect of capital
punishment, we might estimate the effect by comparing states or countries
where the death penalty is legal to those where it is not. Of course, states dif-
fer from one another on multiple dimensions, including socio-demographic
characteristics of the population (age, race, education, religion, urban
vs. rural, family structure, etc.), economic conditions (income, unemploy-
ment, etc.), climate, cultural attitudes, political orientation, and a host of
others. Such studies are extremely common, typically using panel data and
specifying regression models using fixed effects to control for those differ-
ences deemed to be important. The hope is that controlling for these differ-
ences allows the true impact of capital punishment on the murder rate
(or any other effect of interest) to be estimated. The problem is that one can-
not measure or even enumerate all the possible differences among the states
that could potentially influence the murder rate. If any factors that affect the
murder rate are omitted, the results will be biased, particularly if, as is com-
mon, the omitted factors are correlated with those factors that are included
(omitted variable bias), and especially if a state’s decision to use the death

DOI: 10.1002/sdr
penalty was affected by any of these conditions, including the murder rate
itself (endogeneity bias).
These are more than theoretical concerns. Leamer (1983), in a provoca-
tively titled article, “Let’s take the ‘con’ out of econometrics”, showed that
choosing different control variables in panel data regressions led to statisti-
cally significant results indicating that a single execution either prevented
many murders or actually increased them. He concluded “that any inference
from these data about the deterrent effect of capital punishment is too fragile
to be believed” (Leamer, 1983, p. 42). The results challenged the economet-
ric community to take identification seriously: if econometric methods can-
not identify causal relationships then the results of such models cannot
provide reliable advice to policymakers.
The result was the “identification revolution”, an explosion in methods
for causal inference using formal estimation (see Angrist and Pischke, 2010,
whose paper is titled “The Credibility Revolution in Empirical Economics:
How Better Research Design Is Taking the Con out of Econometrics”). These
methods include natural experiments (e.g., Taubman et al., 2014) and, when
true random assignment is not possible, quasi-experimental methods,
including regression discontinuity, difference-in-difference and instrumental
variables (see Angrist and Pischke, 2009).5
Robust methods for parameter estimation, identification, assessment of good-
ness of fit, and parametric and structural sensitivity analysis are now available,
not only in econometrics but also in engineering, artificial intelligence and
machine learning (e.g. Pearl, 2009; Pearl and Mackenzie, 2018. Abdelbari and
Shafi, 2017, offer an application in system dynamics). Following Jay’s example,
system dynamics modelers should master the state of the art and use these
tools, follow new developments as the tools continue to evolve, and innovate to
develop new methods appropriate for the models we build.
The advent of “big data” and analytics including machine learning, vari-
ous artificial intelligence methods and the ability to process data sets of pre-
viously unimaginable size also create unprecedented opportunities for
dynamic modelers. These detailed data sources provide far greater temporal
and spatial resolution, generating insight into empirical issues relevant to
important questions. For example, Rydzak and Monus (2018, this issue) pro-
vide detailed empirical evidence on networks of collaboration among
workers from different departments in industrial facilities and model how
these evolved over time to explain why one facility was successful in
improving maintenance and reliability while another struggled. In the social
realm, the integration of big data sets describing population density, the
location of homes, schools, businesses and other buildings, road networks,
5
Angrist and Pischke (2010, p. 24) acknowledge a fear “that the experimentalist paradigm leads researchers
to look for good experiments, regardless of whether the questions they address are important” and note that
“There is no shortage of academic triviality.” The debate continues, but also spurs innovation that improves
methods for study design, data generation and estimation of impacts that shed light on important questions.

DOI: 10.1002/sdr
social media and cell phone activity provide the granular data needed to
specify patterns of commuting, communication, travel and social interac-
tions. These are essential in developing individual-level models to build
understanding of and design policies to respond to, for example, natural
disasters, terrorist attacks, and outbreaks of infectious diseases (Eubank
et al., 2004; Venkatramanan et al., 2018; Waldrop, 2018).
Big data also enable important theoretical developments in dynamics. For
example, most work in networks has focused on static networks, while many
important real-world networks are dynamic, with new nodes created and
lost, and links among them forged and broken, over time, both exogenously
and endogenously. Consider two examples. First, the famous “preferential
attachment” algorithm (Barabási and Albert, 1999) generates so-called
“scale-free” networks that closely resemble many real networks because it
embodies a powerful positive feedback loop in which new nodes arising in
the network are more likely to link to existing nodes with more links than to
nodes with few links, further increasing the probability that other nodes will
link to the popular ones. Second, conventional wisdom has held that the
presence of transient (“temporal”) links compromises information diffusion,
exploration, synchronization and other aspects of network performance,
including controllability. In contrast, Li et al. (2017) develop formal models
and simulations based on a range of real dynamic networks, from the yeast
proteome to cell phones, showing that “temporal networks can, compared to
their static counterparts, reach controllability faster, demand orders of mag-
nitude less control energy, and have control trajectories that are considerably
more compact than those characterizing static networks.” System dynamics
modelers who embrace big data, analytics and modern dynamical systems
theory will find tremendous opportunities for rigorous work that can address
critical challenges facing humanity.
Rigor or rigor mortis?
In Industrial Dynamics, Appendix O, entitled “Beginners’ Difficulties”, For-

rester wrote “It seems that a discussion of what should be done in a new
activity does not constitute adequate instruction. The positive approach
alone does not provide forwarning [sic] against the pitfalls that lie along the
road” (Forrester, 1961, p. 449). In that spirit, it seems necessary now to ask
why system dynamics modelers should learn and use the modeling, empiri-
cal and analytic methods described above. While the best work in the field
follows the procedures outlined above, some argue that the methods
described above are neither necessary nor helpful. Many descriptions of the
system dynamics method fail to note or encourage the use of new methods
to enhance the rigor, reliability, relevance and impact of dynamic modeling.
Many continue to hold fast to tools and methods for system dynamics
DOI: 10.1002/sdr
practice dating to the 1960s even though better choices now exist and some
old methods and tools are now neither effective nor acceptable for research
or practice. Some fail to emphasize, use, or teach basic principles to gather
evidence, test hypotheses and build confidence in models. To illustrate, for
many years the website of the System Dynamics Society, on a prominent
page titled “Introduction to System Dynamics”, described how system
dynamics ought to be done:6
The System Dynamics approach involves:
• Defining problems dynamically, in terms of graphs over time.

• Striving for an endogenous, behavioral view of the significant dynamics of a
system, a focus inward on the characteristics of a system that themselves gener-
ate or exacerbate the perceived problem.
• Thinking of all concepts in the real system as continuous quantities intercon-
nected in loops of information feedback and circular causality.
• Identifying independent stocks or accumulations (levels) in the system and
their inflows and outflows (rates).
• Formulating a behavioral model capable of reproducing, by itself, the dynamic
problem of concern. The model is usually a computer simulation model
expressed in nonlinear equations, but is occasionally left unquantified as a dia-
gram capturing the stock-and-flow/causal feedback structure of the system.
• Deriving understandings and applicable policy insights from the resulting
model.
• Implementing changes resulting from model-based understandings and
insights.
The description violates fundamental principles for good modeling,

including that one must always model a problem, never the system; that one
should involve the people one hopes to influence in the modeling process
from the beginning; and that modeling is iterative, with the need to revise
and improve the model arising from testing (see Forrester, 1961, Appendix
O and many other passages; also Sterman, 2000, ch. 3; Repenning et al.,
2017). The procedure recommended in the quoted passage jumps directly
from model formulation to “deriving understanding and … policy insights”
and then “implementing changes resulting from these understandings and
insights”. Nowhere is there any call for the use of evidence of any kind,
qualitative or quantitative, nor the importance of multiple methods, estimat-
ing and justifying parameters, assessing and testing the model and the sensi-
tivity of results to uncertainty in assumptions, or any other core elements of
the scientific method. At best, such descriptions, which are not limited to
this passage, will result in poor-quality work that will have no impact. At
6
http://lm.systemdynamics.org/what-is-s/ [accessed 10 July 2018]. As a result of this paper, the description
above has been replaced with a more accurate description, available at https://www.systemdynamics.org/
what-is-sd.

DOI: 10.1002/sdr
worst, policies based on such weakly grounded and poorly tested “insights”
put people at risk of harm.
Despite the rapid progress in methods for parameter estimation, statistical
inference, and causal attribution in complex dynamic systems, some con-
tinue to argue, wrongly, that formal estimation of model parameters is not
possible or not necessary. Today, best practice leverages the rapid progress
in data availability and methods for parameter estimation and causal infer-
ence in complex dynamic systems. Discussing policy based on model simu-
lations without presenting any formal measures of goodness of fit or
evidence to justify the parameters and assess the uncertainty around them
and in results is simply not acceptable.
Some go farther and argue that “quick and dirty” models are good enough,
that expert judgment—often their own—is a sufficient basis for the choice of
model boundary, formulations, and parameter values, that subjective
judgment—again, often their own—of model fit to historical data is suffi-
cient. After all, the alternative is that managers and policymakers will con-
tinue to use their mental models to make consequential decisions. Surely, it
is argued, even a simple model, or even a causal diagram or system arche-
type is a better basis for decision making. Such claims must be tested rigor-
ously. So far, the evidence does not support them (e.g. McCardle-Keurentjes
et al., 2018, this issue). Learning in and about complex systems requires con-
stant iteration between experiments in the virtual world of models, where
costs and risks are low, and interventions in the real world, where experi-
ments are often costly and risky (Sterman, 1994). Failing to test models
against evidence and through experiments in the real world cuts these criti-
cal feedbacks. Replacing a poor mental model with a diagram, archetype, or
simulation that is not grounded in evidence and is poorly tested may create
more harm by providing false confidence and more deeply embedding
flawed mental models.
Even more pernicious is the claim that the methods outlined here are only
relevant for academic work, or that those who advocate for greater scientific
rigor are motivated only to have their work published in academic journals,
instead of working on the grand challenges, as Jay called upon us to
do. Publishing for the sake of publishing is indeed a waste of resources and
talent. Publishing is a means, not an end. We should strive to publish our
work in the best academic journals because peer review provides valuable
feedback, helping us uncover errors and improve our work (Repenning,
2004). We should strive to publish our work because publication enables
others to learn about, use and build on it. We should strive to publish
because, yes, publication in respected journals is how academics gain ten-
ure. Promotion and tenure are, like publishing, not an end in themselves.
They are the means that provide the academic freedom that enables scholars
to work on the important issues, whether they are in vogue or not. Promo-
tion and tenure are the means through which the next generation of
DOI: 10.1002/sdr
researchers and practitioners are trained. The growth of any technical,

research-based field including system dynamics depends on the gain of the
fundamental reinforcing feedback in which scholars train the next genera-
tion of doctoral students, who then go on to their own academic posts,
advance the state of the art and train still more students (see Ghaffarzadegan
and Larson, 2018, this issue).7 Those who continue to use outdated, inferior
and unacceptable methods will not—and should not—succeed in academia.
The reinforcing feedback will falter, severely constraining the growth and
impact of the field.
Far more important, whether we are academics or practitioners, consul-
tants or K-12 educators, whether we are qualitative or quantitative, whether
our goal is understanding or impact, we must use the scientific method
because it is the most effective method yet devised to avoid fooling our-
selves, to overcome superstitious learning and the cognitive traps, errors and
biases afflicting us all. We must use the best available methods and evidence
because that is how we find error, improve our understanding, and discover
robust and reliable insights that can make lasting contributions to the critical
challenges we face.
The path ahead
Throughout his remarkable life, Jay Forrester always worked on problems

that matter, time and again risking his reputation, career, and, as his work
on the USS Lexington showed, even his life. To do so he mastered the theory
and methods relevant to each new problem he tackled and created new
innovations when needed to solve them. He always held himself and others
to uncompromising standards of excellence. His enduring legacy is that pro-
cess, not the particular artifacts resulting from it, whether core memory,
Whirlwind, SAGE—or the particular system dynamics models he built. Jay
always dared us to have courage, be bold, and work on the most important
challenges facing humanity. We must follow his example. We must master
the state of the art and modern methods to develop, test, communicate, and
implement rigorous, reliable and effective insights into the dynamics of com-
plex systems, wherever they originated. Where needed, we must innovate to
develop new methods to address the pressing challenges we face. We must
hold ourselves to the highest standards for rigorous inquiry. The need is
great. The path may be difficult. Will we take it?
7
System dynamics practitioners and consultants play a vital role in applications and real-world impact, but
they do not have the resources, infrastructure or incentives to educate and train people to the level required
to advance the state of the art or grow the capacity of the field, and, with very few exceptions, have not
done so.

DOI: 10.1002/sdr
Acknowledgements
I thank the Forrester family for permission to use images from Jay’s personal
files, and Yaman Barlas, Jack Homer, David Keith, Hazhir Rahmandad, Nel-
son Repenning, Rogelio Oliva, Jørgen Randers, George Richardson and many
other colleagues for helpful comments and suggestions. All errors are mine.
Biography
John D. Sterman is the Jay W. Forrester Professor of Management at the MIT

Sloan School of Management, and Director of the MIT System Dynamics
Group.
References
Abdelbari H, Shafi K. 2017. A computational intelligence-based method to “learn”

causal loop diagram-like structures from observed data. System Dynamics Review
33(1): 3–33.
Agarwal J, DeSarbo WS, Malhotra NK, Rao VR. 2015. An interdisciplinary review of
research in conjoint analysis: recent developments and directions for future
research. Customer Needs and Solutions 2(1): 19–40.
Angrist J, Pischke J. 2009. Mostly harmless econometrics: an empiricist’s companion.
Princeton University Press: Princeton, NJ.
Angrist J, Pischke J. 2010. The credibility revolution in empirical economics: how
better research design is taking the con out of econometrics. Journal of Economic
Perspectives 24(2): 3–30.
Arango S, Castañeda J, Olaya Y. 2012. Laboratory experiments in the system dynam-
ics field. System Dynamics Review 28(1): 94–106.
Barabási L, Albert R. 1999. Emergence of scaling in random networks. Science 286:
509–512.
Barabba V, Huber C, Cooke F, Pudar N, Smith J, Paich M. 2002. A multimethod
approach for creating new business models: the general motors OnStar project.
Interfaces 32(1): 20–34.
Barlas Y. 1989. Multiple tests for validation of system dynamics type of simulation
models. European Journal of Operational Research 42(1): 59–87.
Barlas Y. 1996. Formal aspects of model validity and validation in system dynamics.
System Dynamics Review 12(3): 183–210.
Bass F. 1969. A new product growth model for consumer durables. Management Sci-
ence 15: 215–227.
Booth Sweeney L, Sterman J. 2000. Bathtub dynamics: initial results of a systems
thinking inventory. System Dynamics Review 16(4): 249–294.
Cassam Q. 2017. Diagnostic error, overconfidence and self-knowledge. Palgrave Com-
munications 3. https://doi.org/10.1057/palcomms.2017.25.

DOI: 10.1002/sdr
Cronin M, Gonzalez C, Sterman J. 2009. Why don’t well-educated adults understand

accumulation? A challenge to researchers, educators, and citizens. Organizational
Behavior and Human Decision Processes 108(1): 116–130.
Croson R, Donohue K. 2005. Upstream versus downstream information and its impact
on the bullwhip effect. System Dynamics Review 21(3): 249–260.
Dogan G. 2008. Bootstrapping for confidence interval estimation and hypothesis test-
ing for parameters of system dynamics models. System Dynamics Review 23(4):
415–436.
Duflo E, Banerjee A. 2017. Handbook of field experiments, Vol. 2. North-Holland:
Amsterdam.
Edwards P. 2010. A vast machine: computer models, climate data, and the politics of
global warming. MIT Press: Cambridge, MA.
Epstein J. 2007. Generative social science: studies in agent-based computational
modeling. Princeton University Press: Princeton, NJ.
Ericsson K, Simon H. 1993. Protocol analysis: verbal reports as data, (2nd ed.)
ed. MIT Press: Cambridge, MA.
Eubank S, Guclu H, Kumar V, Marathe M, Srinivasan A, Toroczkai Z, Wang N. 2004.
Modeling disease outbreaks in realistic urban social networks. Nature 429(6988):
180–184.
Fernández-Villaverde J, Rubio-Ramírez J. 2005. Estimating dynamic equilibrium econ-
omies: linear versus nonlinear likelihood. Journal of Applied Econometrics 20(7):
891–910.
Fiddaman T. 2002. Exploring policy options with a behavioral climate–economy
model. System Dynamics Review 18(2): 243–267.
Fiddaman T. 2007. Dynamics of climate policy. System Dynamics Review 23(1): 21–34.
Ford A. 2018. Simulating systems with fast and slow dynamics: lessons from the elec-
tric power industry. System Dynamics Review 34(1–2): 222–254.
Ford D, Sterman J. 1998. Expert knowledge elicitation for improving mental and for-
mal models. System Dynamics Review 14(4): 309–340.
Forrester J. 1961. Industrial dynamics. MIT Press: Cambridge, MA.
Forrester J. 1980. Information sources for modeling the national economy. Journal of
the American Statistical Association 75(371): 555–574.
Forrester J. 1992. From the ranch to system dynamics: an autobiography. In Manage-
ment laureates: a collection of autobiographical essays, Bedeian A (ed). JAI Press:
Stamford, CT. Available: http://gu.friends-partners.org/Peace%20Gaming/System
%20Dynamics/Forrester%27s%20papers/Forrester-Ranch.html#anchor1604851
[18 august 2018].
Forrester J. 2007a. System dynamics: the next fifty years. System Dynamics Review
23(2–3): 359–370.
Forrester J. 2007b. System dynamics: a personal view of the first fifty years. System
Dynamics Review 23(2–3): 345–358.
Gary M, Wood R. 2016. Unpacking mental models through laboratory experiments.
System Dynamics Review 32(2): 101–129.
Ghaffarzadegan N, Larson R. 2018. SD meets OR: a new synergy to address policy
problems. System Dynamics Review 34(1–2): 327–353.
Gilovich T, Griffin D, Kahneman D. 2002. Heuristics and biases: the psychology of
intuitive judgment. Cambridge University Press: Cambridge, U.K.

DOI: 10.1002/sdr
Glaser B, Strauss A. 2017. The discovery of grounded theory: strategies for qualitative
research. Routledge: Abingdon, U.K.
Gonzalez C, Wong H. 2012. Understanding stocks and flows through analogy. System
Dynamics Review 28(1): 3–27.
Gourieroux C, Monfort A, Renault E. 1993. Indirect inference. Journal of Applied
Econometrics 8(S1): S85–S118.
Green T. 2010. Bright boys: the making of information technology. Taylor & Francis:
Milton Park, U.K.
Greenwald A, Banaji M. 1995. Implicit social cognition: attitudes, self-esteem, and
stereotypes. Psychological Review 102(1): 4–27.
Hoekstra A, Chopard B, Coveney P. 2014. Multiscale modelling and simulation: a
position paper. Philosophical Transactions Series A 372(2021). https://doi.org/10.
1098/rsta.2013.0377.
Holz C, Siegel L, Johnston E, Jones A, Sterman J. 2018. Ratcheting ambition to limit
warming to 1.5 C: trade-offs between emission reductions and carbon dioxide
removal. Environmental Research Letters 13(6): 1–11. https://doi.org/10.
1088/1748-9326/aac0c1.
Homer J. 1987. A diffusion model with application to evolving medical technologies.
Technological Forecasting and Social Change 31(3): 197–218.
Homer J. 1996. Why we iterate: scientific modeling in theory and practice. System
Homer J. 2012. Partial-model testing as a validation tool for system dynamics. System
Hosseinichimeh N, Rahmandad H, Jalali M, Wittenborn A. 2016. Estimating the
parameters of system dynamics models using indirect inference. System Dynamics
Review 32(2): 156–180.
Jalali M, Rahmandad H, Ghoddusi H. 2015. Using the method of simulated moments
for system identification. In Analytical Methods for Dynamic Modelers,
Rahmandad H, Oliva R, Osgood ND (eds). MIT Press: Cambridge, MA; 39–69.
Janis IL. 1982. Groupthink: psychological studies of policy decisions and fiascoes.
Houghton Mifflin: Boston, MA.
Kahneman D. 2011. Thinking, fast and slow. Farrar, Straus, and Giroux: New York.
Kampmann C. 2012. Feedback loop gains and system behavior. System Dynamics
Review 28(4): 370–395.
Kampmann C, Oliva R. 2009. Analytical methods for structural dominance analysis in sys-
tem dynamics. In Encyclopedia of complexity and systems science, Meyers R (ed).
Springer: Berlin.
Kampmann C, Sterman J. 2014. Do markets mitigate misperceptions of feedback? Sys-
tem Dynamics Review 30(3): 123–160.
Kapmeier F, Gonçalves P. 2018. Wasted paradise? Policies for Small Island States to
manage tourism-driven growth while controlling waste generation: the case of the
Maldives. System Dynamics Review 34(1–2): 172–221.
Keeling MJ, Rand DA, Morris AJ. 1997. Correlation models for childhood epidemics.
Proceedings of the Royal Society B Biological Sciences 264: 1149–1156.
Keith D, Sterman J, Struben J. 2017. Supply constraints and waitlists in new product
diffusion. System Dynamics Review 33(3–4): 254–279.

DOI: 10.1002/sdr
Kopainsky B, Sawicka A. 2011. Simulator-supported descriptions of complex

dynamic problems: experimental results on task performance and system under-
standing. System Dynamics Review 27(2): 142–172.
Lakeh A, Ghaffarzadegan N. 2015. Does analytical thinking improve understanding of
accumulation? System Dynamics Review 31(1–2): 46–65.
Lakeh A, Ghaffarzadegan N. 2016. The dual-process theory and understanding of
stocks and flows. System Dynamics Review 32(3–4): 309–331.
Lamberson P. 2016. Winner take all or long tail? A behavioral model of markets with
increasing returns. System Dynamics Review 32: 233–260.
Lamberson P. 2018. Approximating individual interactions in compartmental system
dynamics models. System Dynamics Review 34(1–2): 284–326.
Lane D, Sterman J. 2011. Jay Wright Forrester. In Profiles in operations research: pio-
neers and innovators, Assad AA, Gass SI (eds). Springer: Berlin; 363–386.
Lane D, Sterman J. 2018. A model simulator: the lives of Jay W Forrester. Journal of
Simulation 12(2): 90–97.
Leamer E. 1983. Let’s take the con out of econometrics. American Economic Review
73(1): 31–43.
Li A, Cornelius S, Liu Y, Wang L, Barabási A. 2017. The fundamental advantages of
temporal networks. Science 358: 1042–1046.
McCardle-Keurentjes M, Rouwette E, Vennix J, Jacobs E. 2018. Potential benefits of
model use in group model building: insights from an experimental investigation.
System Dynamics Review 34(1–2): 354–384.
McFadden D. 1989. A method of simulated moments for estimation of discrete
response models without numerical integration. Econometrica 57(5): 995–1026.
Meadows DH, Richardson J, Bruckmann G. 1982. Groping in the dark. Wiley:
Chichester, U.K.
Miles M, Huberman A, Saldaäna J. 2014. Qualitative data analysis: a methods source-
book, (3rd ed.) ed. SAGE: Thousand Oaks, CA.
Minyard K, Smith T, Turner R, Milstein B, Solomon L. 2018. Community and pro-
grammatic factors influencing effective use of system dynamic models. System
Dynamics Review 34(1–2): 154–171.
Mojtahedzadeh M, Andersen D, Richardson GP. 2004. Using digest to implement the
pathway participation method for detecting influential system structure. System
Moxnes E. 2001. Not only the tragedy of the commons: misperceptions of feedback and
policies for sustainable development. System Dynamics Review 16(4): 325–348.
Moxnes E. 2004. Misperceptions of basic dynamics: the case of renewable resource
management. System Dynamics Review 20(2): 39–162.
National Research Council. 2014. Identifying the culprit: assessing eyewitness identi-
fication. National Academies Press: Washington, DC.
Nisbett R, Wilson T. 1977. Telling more than we can know: verbal reports on mental
processes. Psychological Review 84(3): 231–259.
Oliva R. 2015. Linking structure to behavior using eigenvalue elasticity analysis. In
Analytical methods for dynamics modelers, Rahmandad H, Oliva R, Osgood ND
(eds). MIT Press: Cambridge, MA; 207–239.
Oliva R. 2016. Structural dominance analysis of large and stochastic models. System

DOI: 10.1002/sdr
Open Science Collaboration. 2015. Estimating the reproducibility of psychological

science. Science 349(6251): 1–8.
Osgood N, Liu J. 2015. Combining Markov chain Monte Carlo approaches and
dynamic modeling. In Analytical methods for dynamic modelers, Rahmandad H,
Oliva R, Osgood ND (eds). MIT Press: Cambridge, MA; 125–170.
Özgün O, Barlas Y. 2015. Effects of systemic complexity factors on task difficulty in a
stock management game. System Dynamics Review 31(3): 115–146.
Pearl J. 2009. Causality: models, reasoning and inference. Cambridge University
Press: Cambridge, U.K.
Pearl J, Mackenzie D. 2018. The book of why: the new science of cause and effect.
Basic Books: New York.
Perlow L, Repenning N. 2009. The dynamics of silencing conflict. Research in Orga-
nizational Behavior 29: 195–223.
Peterson D. 1980. Statistical tools for System Dynamics. In Elements of the System
Dynamics Method, Randers J (ed). MIT Press: Cambridge, MA; 224–245.
Pfeffer J, Sutton R. 2000. The knowing–doing gap: how smart companies turn knowl-
edge into action. Harvard Business School Press: Boston, MA.
Pickett S, White P. 1985. The ecology of natural disturbance and patch dynamics.
Academic Press: New York.
Politis D, Romano J. 1994. Large sample confidence regions based on subsamples
under minimal assumptions. Annals of Statistics 22(4): 2031–2050.
Press W, Teukolsky S, Vetterling W, Flannery B. 2007. Numerical recipes: the art of
scientific computing, (3rd ed.) Cambridge University Press: Cambridge, U.K.
Rahmandad H, Jalali M, Paynabar K. 2017. A flexible method for aggregation of prior
statistical findings. PLoS One 12(4): e0175111. https://doi.org/10.1371/journal.
pone.0175111.
Rahmandad H, Oliva R, Osgood N. 2015. Analytical methods for dynamic modelers.
MIT Press: Cambridge, MA.
Rahmandad H, Sterman J. 2008. Heterogeneity and network structure in the dynamics
of diffusion: comparing agent-based and differential equation models. Management
Science 54(5): 998–1014.
Randers J, Golüke U, Wenstøp F, Wenstøp S. 2016. A user-friendly earth system
model of low complexity: the ESCIMO system dynamics model of global warming
towards 2100. Earth System Dynamics 7: 831–850. https://doi.org/10.5194/esd-7-
831-2016.
Repenning N. 2004. Selling system dynamics to (other) social scientists. System
Repenning N, Kieffer D, Astor T. 2017. The most underrated skill in management.
MIT Sloan Management Review 58(3): 39–48. https://sloanreview.mit.edu/article/
the-most-underrated-skill-in-management/.
Richardson G. 1991. Feedback thought in social science and systems theory. Univer-
sity of Pennsylvania Press: Philadelphia, PA.
Rogers J, Gallaher E, Dingli D. 2018. Personalized ESA doses for anemia management
in hemodialysis patients with end-stage renal disease. System Dynamics Review
34(1–2): 121–153.
Rooney Varga J, Sterman J, Fracassi E, Franck T, Kapmeier F, Kurker V, Johnston E,
Jones A, Rath K. 2018. Combining role-play with interactive simulation to motivate

DOI: 10.1002/sdr
informed climate action: Evidence from the World Climate simulation. PLoS One
13(8): 1–28. https://doi.org/10.1371/journal.pone.0202877.
Rudolph J, Morrison J, Carroll J. 2009. The dynamics of action-oriented problem solv-
ing: linking interpretation and choice. Academy of Management Review 34(4):
733–758.
Rydzak F, Monus P. 2018. Shaping organizational network structure to enable sus-
tainable transformation. System Dynamics Review 34(1–2): 255–283.
Saleh M, Oliva R, Kampmann CE, Davidsen P. 2010. A comprehensive analytical
approach for policy analysis of system dynamics models. European Journal of
Operational Research 203(3): 673–683.
Sanchez C, Dunning D. 2018. Overconfidence among beginners: Is a little learning a
dangerous thing? Journal of Personality and Social Psychology 114(1): 10–28.
Saysel A, Barlas Y. 2006. Model simplification and validation with indirect structure
validity tests. System Dynamics Review 22(3): 241–262.
Simmons J, Nelson L, Simonsohn U. 2011. False-positive psychology: undisclosed
flexibility in data collection and analysis allows presenting anything as significant.
Psychological Science 22(11): 1359–1366.
Sloman S. 1996. The empirical case for two systems of reasoning. Psychological Bul-
letin 119(1): 3–22.
Sterman J. 1987. Testing behavioral simulation models by direct experiment. Man-
agement Science 3(12): 1572–1592.
Sterman J. 1989. Modeling managerial behavior: misperceptions of feedback in a
dynamic decision making experiment. Management Science 35(3): 321–339.
Sterman J. 1994. Learning in and about complex systems. System Dynamics Review
10(2–3): 291–330.
Sterman J. 2000. Business dynamics: systems thinking and modeling for a complex
world. Irwin/McGraw-Hill: New York.
Sterman J (ed). 2007. Exploring the next great frontier: system dynamics at fifty. Sys-
tem Dynamics Review 23(2–3): 89–93.
Sterman J. 2015. Learning for ourselves: interactive simulations to catalyze science-
based environmental activism. In Science based activism, Stoknes P, Eliassen K
(eds). Fagbokfolaget: Bergen, Norway; 253–279.
Sterman J, Dogan G. 2015. I’m not hoarding, I’m just stocking up before the hoarders
get here: behavioral causes of phantom ordering in supply chains. Journal of Oper-
ations Management 39–40: 6–22.
Sterman J, Fiddaman T, Franck T, Jones A, McCauley S, Rice P, Sawin E, Siegel L.
2012. Climate interactive: the C-ROADS climate policy model. System Dynamics
Review 28(3): 295–305.
Sterman J, Fiddaman T, Franck T, Jones A, McCauley S, Rice P, Sawin E, Siegel L.
2013. Management flight simulators to support climate negotiations. Environmen-
tal Modelling and Software 44: 122–135.
Sterman J, Repenning N, Kofman F. 1997. Unanticipated side effects of successful
quality programs: exploring a paradox of organizational improvement. Manage-
ment Science 43(4): 501–521.
Strogatz S. 2001. Exploring complex networks. Nature 410: 268–276.
Struben J, Sterman J. 2008. Transition challenges for alternative fuel vehicle and
transportation systems. Environment and Planning B 35: 1070–1097.

DOI: 10.1002/sdr
Struben J, Sterman J, Keith D. 2015. Parameter estimation through maximum likeli-

hood and bootstrapping methods. In Analytical methods for dynamic modelers,
Rahmandad H, Oliva R, Osgood N (eds). MIT Press: Cambridge, MA; 3–38.
Taubman S, Allen H, Wright B, Baicker K, Finkelstein A, Oregon Health Study
Group. 2014. Medicaid increases emergency department use: evidence from Ore-
gon’s health insurance experiment. Science 343(6168): 263–268.
Tebbens R, Thompson K. 2018. Using integrated modeling to support the global erad-
ication of vaccine-preventable diseases. System Dynamics Review 34(1–2): 78–120.
Ter Braak C. 2006. A Markov chain Monte Carlo version of the genetic algorithm dif-
ferential evolution: Easy Bayesian computing for real parameter spaces. Statistics
and Computing 16(3): 239–249.
Thompson K, Tebbens R. 2007. Eradication versus control for poliomyelitis: An eco-
nomic analysis. Lancet 369(9570): 1363–1371.
Thompson K, Tebbens R. 2008. Using system dynamics to develop policies that mat-
ter: global management of poliomyelitis and beyond. System Dynamics Review
24(4): 433–449.
Thompson K, Tebbens R, Pallansch M, Wassilak S, Cochi S. 2015. Polio eradicators
use integrated analytical models to make better decisions. Interfaces 45(1): 5–25.
Thomson W. 1883. Electrical units of measurement. Popular Lectures and Addresses
1: 73–136. https://archive.org/stream/popularlecturesa01kelvuoft#page/72/mode/2up.
Turco C. 2016. The conversational firm: rethinking bureaucracy in the age of social
media. Columbia University Press: New York.
Venkatramanan S, Lewis B, Chen J, Higdon D, Vullikanti A, Marathe M. 2018. Using
data-driven agent-based models for forecasting emerging infectious diseases. Epi-
demics 22: 43–49.
Vierhaus I, Fügenschuh A, Gottwald R, Grösser S. 2017. Using white box nonlinear
optimization methods in system dynamics policy improvement. System Dynamics
Review 33: 138–168.
Villa S, Gonçalves P, Arango S. 2015. Exploring retailers’ ordering decisions under
delays. System Dynamics Review 31(1–2): 1–27.
Vrugt J, ter Braak C, Diks C, Robinson B, Hyman J, Higdon D. 2009. Accelerating Mar-
kov chain Monte Carlo simulation by differential evolution with self-adaptive ran-
domized subspace sampling. International Journal of Nonlinear Sciences and
Numerical Simulation 10(3): 271–288.
Waldrop M. 2018. Free agents. Science 360: 144–147. https://doi.org/10.1126/
science.360.6385.144.
Woodruff S, BenDor T, Strong A. 2018. Fighting the inevitable: infrastructure investment
and coastal community adaptation to sea level rise. System Dynamics Review 34(1–2):
48–77.
Yücel G, Barlas Y. 2015. Pattern recognition for model testing, calibration and behav-
ior analysis. In Analytical methods for dynamic modelers, Rahmandad H, Oliva R,
Osgood N (eds). MIT Press: Cambridge, MA; 173–206.

DOI: 10.1002/sdr

System Dynamics at Sixty: The Path Forward

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

System Dynamics at Sixty: The Path Forward

Uploaded by

Copyright:

Available Formats

INTRODUCTION

System dynamics at sixty: the path forward

Syst. Dyn. Rev. 34, 5–47 (2018)

System Dynamics Review

A ranch is a cross-roads of economic forces. Supply and demand, changing

To solve the pressing problems of life on a frontier homestead, Jay mas-

“Building an electrical system from discarded automobile parts was a very

Fig. 2. At the insistence

mechanical hydraulic variable speed pumps, motors, and high-gain hydrau-

© 2018 System Dynamics Society

the experimental control units stopped working. In November 1943, I volun-

Film of the Lexington on ﬁre after the attack is available at https://youtu.

Fig. 3. Map from Jay’s

© 2018 System Dynamics Society

YOUR PRESENCE URGENTLY NEEDED IN CAMBRIDGE MR STOUTS OFFICE

Am now in Honolulu address Moana Hotel. Performance and reputation of

FIELD SERVICE PROBLEM ON ALL OUR MATERIEL NEEDS CLARIFICATION

Fig. 4. Radiogram from

© 2018 System Dynamics Society

Am convinced complete check of components must be possible without special

Jay’s relentless focus on robustness, reliability, and designing the system

never work in such a demanding real-time task. Instead, Jay decided to

© 2018 System Dynamics Society

memory, and interactive computer graphics” (http://history.sandiego.edu/

… we had control from basic research to military operational planning. We

© 2018 System Dynamics Society

Four foundations on which to construct an improved understanding of the

Industrial Dynamics presented the results: a novel, innovative method to

Model architecture: compartments, agents and networks

Jay’s innovations in the Whirlwind project made it possible for computers to

be disaggregated by attributes including preferences, location, etc., with as

© 2018 System Dynamics Society

Other guidelines Jay offers, however, were clearly compromises necessi-

© 2018 System Dynamics Society

Similarly, recent developments now allow models to capture dynamics at

assumptions and policies for themselves to build their conﬁdence in the

© 2018 System Dynamics Society

available. In the same way, computational capabilities today have expanded

Evidence-based modeling: qualitative and quantitative data

© 2018 System Dynamics Society

Jay’s approach is often oversimpliﬁed by newcomers to the ﬁeld as just

“… mental information—information about policy and structure—is directly

© 2018 System Dynamics Society

must be guided by a knowledge of what different structures imply for dynamic

uncertainty. In many settings, training and expertise do not improve accu-

Formal methods to extract meaning and underlying themes and constructs

effect—probably the only value that is known to be wrong!” is widely cited

As an engineer ever focused on practicalities, Forrester then adds an

To develop reliable knowledge, we must bring the best available data to

project is to identify the importance of and then measure important concepts

© 2018 System Dynamics Society

no Internet or electronic databases to automate the search of medical jour-

Fig. 9. Publication rate of

© 2018 System Dynamics Society

Fig. 10. Structure for Mean Follow-Up

Fig. 11. Data and

Based on Homer (1987,

Eligibility Fraction/Initial Eligibility Fraction

© 2018 System Dynamics Society

Fig. 12. Simulated and

Parameter estimation and model analysis

When Forrester ﬁrst formulated principles for system dynamics practice,

multicollinearity, such as the Kalman ﬁlter, were introduced into system