Validation and

Verification of
Simulation Models

Outline

Introduction
Definitions of validation and
verification
Techniques for verification of
simulation models
Techniques for validation of
simulation models
Statistical Methods for Comparing
real-world observations with
simulation output data
Inspection Approach
Confidence-Interval Approach
Summary

Introduction

One of the most difficult problems

facing the simulation analyst is
determining whether a simulation
model is an accurate representation
of the actual system being studied (
i.e., whether the model is valid).

If the simulation model is not valid,

then any conclusions derived from
it is of virtually no value.

Validation and verification are two

of the most important steps in any
simulation project.

What are Validation and

Verification?

Conceptual
Model

Va
l

id

at
io

Real
-World
System

Validation

at
io

Simulation
Program

ifi
c

Validation is the process of determining

whether the conceptual model is an accurate
representation of the actual system being
analyzed. Validation deals with building the
right model.
Verification is the process of determining
whether a simulation computer program
works as intended (i.e., debugging the
computer program). Verification deals with
building the model right.

Ve
r

Techniques for Verification

of Simulation Models

Use good programming practice:

Use structured walk-through:

Write and debug the computer program in

modules or subprograms.
In general, it is always better to start with
a moderately detailed model, and later
embellish, if needed.

Have more than one person to read

computer program.

the

Use a trace:

The analyst may use a trace to print out

some intermediate results and compare
them with hand calculations to see if
the program is operating as intended.

Techniques for Verification

of Simulation Models

Check simulation output for

reasonableness:

Run the simulation model for a variety

of input scenarios and check to see if
the output is reasonable.
In some instances, certain measures of
performance can be computed exactly
and used for comparison.

Animate:

Using animation, the users see

dynamic displays (moving pictures) of
the simulated system.
Since the users are familiar with the
real system, they can detect
programming and conceptual errors.

Techniques for Verification

of Simulation Models

Compare final simulation output with

analytical results:

May verify the simulation response by

running a simplified version of the
simulation program with a known
analytical result. If the results of the
simulation do not deviate significantly
from the known mean response, the
true distributions can then be used.
For example, for a queuing simulation
model, queuing theory can be used to
estimate steady state responses (e.g.,
mean time in queue, average
utilization). These formulas, however,
assume exponential interarrival and
service times with n servers (M/M/n).

Techniques for Validation of

Simulation Models

A three-step approach for developing a valid

and credible model:

1. Develop a model with high face validity:

The objective of this step is to develop
a model
that, on the surface, seems reasonable to people
who are familiar with the system under study.
This step can be achieved through discussions
with system experts, observing the system, or the
use of intuition.
It is important for the modeler to
interact with
the client on a regular basis throughout the
process.
It is important for the modeler to
perform a
structured walk-through
of the conceptual
model before key people to ensure the
correctness
of models
assumptions .

Techniques for Validation of

Simulation Models
2. Test the assumptions of the model
empirically:
In this step, the assumptions made in
the initial stages of model development
are tested quantitatively. For example,
if a theoretical distribution has been
fitted to some observed data, graphical
methods and goodness of fit tests are
used to test the adequacy of the fit.
Sensitivity analysis can be used to
determine if the output of the model
significantly changes when an input
distribution or when the value of an
input variable is changed. If the
output is sensitive to some aspect
of the model, that aspect of the model
must be modeled very carefully.

Techniques for Validation of

Simulation Models
3. Determine how representative the simulation
output data are:

The most definitive test of a models validity is

determining how closely the simulation output
resembles the output from the real system.

The Turing test can be used to compare the

simulation output with the output from the real
system. The output data from the simulation
can be presented to people knowledgeable about
the system in the same exact format as the
system data. If the experts can differentiate
between the simulation and the system outputs,
their explanation of how they did that should
improve the model.

Statistical methods are available for comparing

the output from the simulation model with those
from the real-world system .

Statistical Methods for Comparing

Real-World Observation With
Simulation Output Data

Suppose R1 , R2 , ..., Rk are observations

from a real-world system and
M1 , M 2 , ..., M l
are output data
from the simulation model.

Both, the real-world system outputs and

simulation outputs are almost always
non-stationary (the distribution of the
successive observations change over time)
and autocorrelated(the observations are
correlated with each other).

Therefore, because classical statistical

tests ( the t-test, chi-square test, K-S test,
etc.) assume I.I.D data, they can not
directly be used to compare the two data
sets to determine model validity.

Statistical Methods for Comparing

Real-World Observation With
Simulation Output Data

2 approaches for comparing

the outputs from the realworld system with the
simulation outputs are:

Inspection Approach
Confidence-Interval Approach

Inspection Approach

Run the simulation model with

historical system input data (e.g.,
actual observed interarrival and
service times) instead of sampling
from the input probability
distributions, and compare the
system and model output data.

The system and the model experience

exactly the same observations from
the input random variables.

This approach results in model and

system outputs being positively
correlated.

Inspection Approach

Historical system
input data

Historical system
input data

Actual system

Simulation
model

System output
data

Model output
data

Inspection Approach

If X is the output from the real-world

system and Y is the corresponding output
from the model, we are interested in
estimating X Y .

We make n experiments (using historical

data) and compute Xj - Yj (for j = 1, 2, , n)
as an estimate of X Y . Note that Xj and
Yj use exactly the same interarrival times
and service times).

Xj and Yj can then be plotted such that the

horizontal axis denotes time and the
vertical axis denotes the real and simulated
outputs. The user can then eyeball
timepaths to see if the model accurately
represents the real-world system.

Inspection Approach

Due to positive correlation between X

and Y, Var (X-Y) is much smaller than if
X and Y were independent which makes
Xj - Yj a much better estimate of
:

Reason:
-- In general,

X Y

Var (X-Y) = Var (X) + Var (Y) - 2Cov(X, Y)

-- If X and Y are independent, Cov (X, Y)=0
and Var (X-Y) = Var (X) + Var (Y)
-- But, if X and Y are positively correlated, Cov(X,
Y) > 0 leading to a smaller value for Var (X-Y).

Inspection Approach

Inspection approach (also called trace

driven method)may provide valuable
insight into the adequacy of the
simulation model for some simulation
studies.

In fact, this may be the only feasible

approach because of severe limitations
on the amount of data available on the
operation of some real-world systems
(e.g., military situation).

Care must, however, be taken in

interpreting the result of this
approach.

Confidence-Interval
Approach

A more reliable approach for

comparing a simulation model with the
real-world system.
Requires a large amount of data.
Suppose we collect m independent sets
of data from the system and n
independent sets of data from the
model (m and n can be equal).
Let Xj be the average of observations in
the jth set of system data with mean
and let Yj be the output from the
X E ( X j )
jth replication
of the simulation model
with
.
Y E (Yj )
The objective is to build a confidenceinterval for

X Y .

Confidence-Interval
Approach

In the case of correlated outputs where Xj is

correlated with Yj (e.g., using trace driven
simulation), let m = n and pair Xj s and Yjs. Let
Zj = Xj -Yj for j=1, 2 , n. Zjs are IID random
variables with E(Zj) = .

Let

and

Z ( n)

Z
j 1

Var Z (n)

j 1

Z ( n)

Then, the approximate 100(1C.I. is

n(n )percent
1)

If the confidence
notZinclude
Z (n) t ninterval
Var
(n) a zero,
1,1 2 does
then the observed difference between
and
is
statistically different at level .

Summary

Validation and verification are two

of the most important steps in any
simulation study.

Validation is not something to be

attempted after the simulation
model has already been developed,
and only if there is time and money
still remaining. Instead, model
development should be done
throughout the entire simulation
study.