2010 Sixth IEEE International Conference on e–Science
Scaling Benchmark of ESyS-Particle for Elastic Wave Propagation Simulations
D. K. Weatherley∗† , V. E. Boros∗ , W. R. Hancock† and Steffen Abe‡
∗
The University of Queensland
Earth Systems Science Computational Centre
Brisbane, Queensland, Australia
†
The University of Queensland
Sustainable Minerals Institute
W. H. Bryan Mining and Geology Research Centre
Brisbane, Queensland, Australia
‡
RWTH Aachen University
Aachen, Germany
Abstract
The Discrete Element Method (DEM) is a popular
particle-based numerical method for simulating geo-
physical processes including earthquakes, rock break-
age and granular flow. Often simulations consisting
of thousands of particles have insufficient resolution
to reproduce the micromechanics of many geophysical
processes, requiring millions of particles in some in-
stances. The high computational expense of the DEM
precludes execution of such problem sizes on desktop
PCs.
ESyS-Particle is a parallel implementation of the
DEM, designed for execution on cluster supercomput-
ers. Three-dimensional spatial domain decomposition
is implemented using the MPI for interprocess com-
munications. We present results of scaling benchmarks
in which problem size per worker remains constant.
As the number of workers increases from 27 to 1000,
execution time remains near-constant, permitting sim-
ulations of 8.7M particles in approximately the same
real time as simulations comprising 240K particles.
1. Introduction
The Discrete Element Method (DEM) is a gener-
alised numerical modelling method for the simulation
of processes that can be described using fundamental
interactions between grains or particles [1]. By using
a first principles approach, the aim of the DEM is to
model macroscopic processes such as rock breakage
and granular flow by solving Newton’s equations of
motion for a large assembly of microscopic particles
978-0-7695-4290-4/10 $26.00 © 2010 IEEE 277
DOI 10.1109/eScience.2010.40
undergoing particle-pair interactions. The success of
the DEM lies in its ability to be used as both a
fundamental research modelling tool and as a practical
engineering tool in industries ranging from pharmaceu-
ticals [2] to mining [3].
As explained in Section 2, the DEM is inherently
computationally expensive, with the total number of
particles being the largest constraint on execution
times. To replicate the macroscopic behaviour of many
geophysical processes, significant numbers of particles
are required. One such example is in rock fracture
simulations, where samples are created entirely of
bonded particles and fractures are resolved along bro-
ken bonds. The propagation of the crack tip is heavily
dependent on the resolution of the simulation where, in
the worst case if too few particles are used, cracks may
never coalesce to form macroscopic fractures. This
motivates the need for a parallel implementation of
the algorithm, one that is able to distribute workload
evenly across the compute space and maintain a rea-
sonable efficiency even at larger scales.
The aim of this paper is to benchmark the perfor-
mance of ESyS-Particle, a parallel implementation of
the DEM designed for cluster supercomputers. Ear-
lier work [4] has verified its scalability on a two-
dimensional elastic wave propagation problem. More
recent work [5] has showcased its three-dimensional
capability on the simulation of a fault gouge. Here we
verify its scalability on a three-dimensional version of
the elastic wave propagation problem. In the following
we first describe the basic features of DEM algorithms
then discuss the parallel implementation employed in
ESyS-Particle. Subsequently we describe a series of
 Initialisation
O(Np)
t=0
Yes
Neighbour Search
O(Np) x Nsearch
No Update
Force Summation
O(Np) x Nt Neighbour
t=t+1
List?
Position Update t<Nt
O(Np) x Nt
t=Nt
End
Figure 1. Flow diagram of the DEM algorithm.
benchmarking experiments and present results demon-
strating the scalability of the software. We find that
ESyS-Particle displays reasonable weak scalability i.e.
total execution time remains near-constant with in-
creasing numbers of processes for constant problem
size per processor. We conclude that ESyS-Particle
provides an effective platform for DEM simulations
involving tens of millions of particles.
2. Discrete Element Method
The DEM is a first principles approach to modelling
elastic solids or granular media in which Newton’s
equations of motion are used to compute the trajec-
tories of large assemblies of microscopic particles un-
dergoing particle-pair interactions. Spherical particles
described by their mass, radius, position and velocity
interact with neighbouring particles via a range of
simplified interactions including linear elastic repul-
sion and frictional forces. An explicit finite difference
time integration scheme is used to integrate Newton’s
equation of motion, solving for the velocity and new
position of each particle in response to the forces
it experiences at a given time. To ensure numerical
stability, positions are repeatedly updated at discrete
time intervals (∆t) which are typically of the order
of micro- or nano-seconds for real-world applications.
Since DEM simulations often comprise large numbers
of particles and very short time increments, the DEM
is a computationally expensive numerical method, with
computation time often the single most limiting factor
determining the resolution and accuracy of numerical
solutions.
Figure 1 illustrates the DEM solution algorithm. A
278
Figure 2. Example of a cubic volume packed with
spherical particles.
complete treatise on DEM algorithms is provided in
[6]. The DEM algorithm consists of four main com-
ponents: initialisation, neighbour search, force sum-
mation and position update. Initialisation of a DEM
simulation typically consists in specifying the initial
locations and velocities of all particles. Various meth-
ods have been devised for specifying initial particle
locations. For the simulations conducted here, we em-
ploy a geometrical particle-packing algorithm in which
particles ranging in size from Rmin to Rmax are closely
packed into a bounding volume (see Figure 2). For the
benchmarking results presented later, the details of this
packing algorithm are irrelevent because packing was
conducted in a separate stage prior to commencing
simulations and does not contribute to the execution
times reported.
The second component of any DEM algorithm is
an efficient neighbour search algorithm. This algorithm
generates a list of pairs of particles that are sufficiently
close to one another that they may interact. The
particle-pair list is used during the subsequent force
summation stage to avoid unnecessary calculations. A
number of efficient neighbour search algorithms are
described in [6], two of the most common being: main-
tain a Verlet list [7] to keep track of neighbouring pairs
to be considered for force calculations, and implement
a grid-neighbour search algorithm.
The Verlet list accelerates the force calculations by
storing a list of potentially interacting particle pairs.
Given that the force calculations in the DEM are based
on nearest-neighbour interactions only, the number of
particles which can potentially interact with a specific
particle is limited to a constant number which is
determined by the ratio of the radii of the largest and
smallest particles in the model. Therefore the total
number of potentially interacting particle pairs stored
in the Verlet list is O(Np ), where Np is the number
of particles. This results in a linear dependence of
the time needed for the force calculations on the total
number of particles for a given particle size range.
Similarly, the grid-based neigbour search algorithm
reduces the time for a rebuild of the list of potentially
interacting pairs of particle. Instead of checking every
particle in the domain against every other particle
in the domain for interactions, or Np (Np − 1)/2
particle-pairs (a search with O(Np2 ) complexity), the
algorithm looks for neighbouring particles within a
smaller, fixed space in which the number of particles
can be considered constant. So the number of particle-
pairs checked for interactions is now approximately
C1 Np (O(Np )), where C1 is a constant which mainly
depends on the range of particle sizes present in the
model. The neighbour search is only done on a needs
basis and is usually triggered when a particle has
moved a significant distance such that it may have new
neighbours.
Having obtained the list of potential particle-pairs
from the neighbour search, the force summation is
performed. This requires, for each particle-pair in the
list, computing the particle-pair forces and updating the
running sum of the net force acting on each particle.
At the end of the force summation, the algorithm
yields the total net force acting on each particle, from
which the current acceleration is obtained by dividing
by the mass of the particle. The force summation
requires approximately C2 Np calculations (O(Np )),
where C2 is the average coordination number (the
average number of particles interacting with a given
particle). After the net force acting on each particle
has been obtained, the velocities and positions of
each particle are individually updated via a finite-
difference time-integration scheme with an algorithmic
complexity of O(Np ).
As illustrated in Figure 1, the neighbour search,
force summation and position update stages are repeat-
edly executed until the desired number of simulation
timesteps are completed. Consequently, the total algo-
rithmic complexity of the DEM is at least O(Np × Nt )
for simulations with infrequent neighbour list updates.
To simulate some geophysical applications with the
DEM may require the use of both millions of particles
and millions of timesteps, a computational burden that
is beyond the scope of desktop computers.
279
3. Three-dimensional Parallel DEM Im-
plementation
ESyS-Particle is a parallel implementation of the
Discrete Element Method written in C++ with a Python
API [8], [9], [4]. ESyS-Particle is specifically designed
with geophysical applications in mind and is capable
of simulating large three-dimensional problems. The
software design permits distribution of both control
and data. It employs a distributed memory model
with communication implemented using the Message
Passing Interface (MPI) [10].
ESyS-Particle uses a master-worker model modified
to permit communication directly among workers. The
master process, known as the lattice master, provides a
high level of control, communicating requirements to
all worker processes in the lattice covering the domain
of the problem. Each worker process is comprised of a
sublattice controller and the sublattice itself. Each sub-
lattice controller communicates with the lattice master,
conveying instructions to its corresponding sublattice.
Each sublattice is responsible for the computations,
such as force calculations, in the simulation. In the
modified master-worker model, the sublattices also
communicate directly among themselves [8], [9].
Particles are packed into a closed geometrical space
with a power-law distribution of sizes. The problem of
how to assign particles within the domain to workers
is addressed with spatial domain decomposition. Each
worker is responsible for a subdomain of the problem
domain and thus all particles and their interactions
within that subdomain. The movement of particles be-
tween the subdomains is handled during the rebuilding
of the neighbour table described in Section 2.
In order to compute the net force on a particle near a
subdomain boundary, a worker process may require the
positions of particles residing in a neighbouring sub-
domain. To minimise communication between worker
processes, each worker stores the positions of particles
residing in a small region surrounding its own domain.
At the end of each position update stage, workers
then broadcast the positions of particles lying in their
subdomain boundary region. Neighbouring workers
then update their own particle position lists accord-
ingly. In addition to the memory and communication
overheads this approach entails, each worker also faces
an increased computation work-load to compute forces
in boundary regions, compared with that of an isolated
subdomain.
The output of simulation data to disk is also paral-
lelised in ESyS-Particle. There exist two types of data
structure that can be written from an ESyS-Particle ex-
periment: checkpointers and field saver. Checkpointers

Figure 3. Snapshot of particle velocities at
timestep 800 in a simulation using 27 worker pro-
cesses.
are used for verbose output of simulation information
including the positions, velocities and forces for each
particle. Fieldsavers are used to output specific quanti-
ties such as total potential or kinetic energy, forces
acting on walls or the number of remaining bonds
within a simulation. The communication that occurs
in a checkpoint operation involves the master request-
ing that all workers write information about particles
within their subdomain into a file that each worker
creates. The field saver approach is a scatter-gather
operation in which the master requests the workers
to calculate their individual totals. This information is
then sent back to the master which collates it into a sin-
gle datum which is then written to disk by the master
process. Both approaches have their advantages and
disadvantages. Checkpointers involve a large amount
of data output but use limited communication (as the
process is embarassingly parallel). Fieldsavers on the
other hand require a larger amount of communication
with usually small amounts of data output at the end
of the operation. One of the aims of the benchmarking
experiments below is to quantify the relative overheads
of checkpointer versus field saver data output options.
4. Benchmarking Experiment
In order to test the scalability of ESyS-Particle, a
simple problem is required which can be scaled up
with relative ease. A three-dimensional elastic wave
propagation problem is a suitable choice. The problem
consists of a cube of bonded particles excited into
movement by an impulse supplied by an external,
planar wall (see Figure 3). Wave propagation is a
fundamental process that occurs as kinetic energy is
280
Number of
Workers:
1 8 27
Figure 4. Subdomain decomposition for scaling
benchmark experiments.
propagated through a medium e.g. the transmission of
sound through air or earthquakes via ground motion.
A series of experiments are conducted in which the
size of the problem and the number of processor cores
are varied. As the problem size grows, the problem
domain is divided into subdomains of equal size, with
each worker process assigned one subdomain. The
experiment is scaled up by stacking the base problem
size in a cubic fashion as demonstrated in Figure 4. By
keeping the simulation model as a cubic volume, this
also ensures that the entire compute space has been
utilised.
The wave propagation experiment was selected be-
cause it is a small deformation problem where particles
do not have to move significant distances. Small de-
formation reduces the number of times the neighbour
search is triggered and limits the amount of communi-
cation to transfer particles from one worker process to
another due to migration across subdomain boundaries.
This simplifies the communication and computational
overheads for the simulations, thus easing interpreta-
tion of the timing benchmark results presented later.
The wave propagation model is initialised by pack-
ing a cubic domain with particles. A base cube mea-
suring 15 × 15 × 15 mm is packed using particles
with radii ranging between 0.2 and 1.0 mm. For larger
simulations, the same packing algorithm is used but the
side length of the cube is expanded according to the
number of subdomains in each coordinate direction.
In our simulations, each subdomain contains approxi-
mately 9000 particles, commencing with 8647 particles
in the smallest test (one worker process) and finishing
with 8 700 121 particles spread across 1000 workers in
the largest test. Table 2 shows the partitioning of the
problem for differing numbers of worker processes.
Having constructed the particle assembly, the ge-
ometries are stored to files that are read during ini-
tialisation of the actual simulations. The lattice master
reads the input geometry file and assigns each particle
to one of the workers according to the subdomain in
 Table 1. Simulation parameters for the wave Table 2. Partitioning of the domain.
propagation benchmark experiments.
Nodes Processes / Cores Subdomains Particles
Simulation Parameter Symbol Value xyz
Subdomain side length L 15.0 mm 1 2 111 8647
Minimum particle radius Rmin 0.2 mm 2 9 222 71615
Maximum particle radius Rmax 1.0 mm 4 28 333 241540
Number of timesteps Nt 10000 9 65 444 573027
Timestep increment ∆t 10−3 ms 16 126 555 1107566
Elastic stiffness K 1000.0 N/mm 28 217 666 1925620
Wave source amplitude AW 0.1 mm/ms 43 344 777 3020677
Wave source period τW 1.0 ms 65 513 888 4558458
92 730 999 6345300
126 1001 10 10 10 8700121

which the particle initially resides. Each particle is then

connected to its nearest neighbours via linear elastic
springs. Such a bonded particle assembly has the
macroscopic properties of an isotropic, linear elastic
solid [8]. In order to trigger elastic wave propagation,
a wall is created at the base of the cube which is moved
upwards to generate an input pulse. An overview
of simulation parameters is given in Table 1. Each
simulation is executed for a total of 10 000 timesteps
and an external timing function is employed to measure
execution time in real seconds.
The benchmarking experiments are conducted in
four stages, with ten simulations in each stage per the
list in Table 2. In the first stage, execution times are
recorded for simulations with no output data gener-
ated. This provides a base-line for computation time
without data I/O overheads. The second stage involves
simulations in which a field saver records a single
floating point number (the total kinetic energy) to a
file, once per timestep. This is considered the mini-
mum amount of data I/O for any practical simulation.
The third stage stores checkpoint files (containing the
entire simulation state) once every 100 timesteps. In
the final stage, both field saver and checkpoint files
are output. The hardware used for these experiments
is a newly commissioned SGI HPC cluster at The
University of Queensland containing 3144 cores on
384 nodes connected through InfiniBand and Gigabit
Ethernet networks. For the benchmarking tests, we
utilise between 1 and 126 nodes of that supercomputer,
each comprised of dual Intel Xeon L5520 2.26 GHz
quad-core CPUs. For each CPU the memory speed is
1066 MHz with 3 GB RAM available to each core.
5. Performance Results
Total execution times for each of the experiments
are provided in Figure 5. In all experiments, ESyS-
Particle shows evidence for weak scalability, even
for very large numbers of worker processes, although
scalability is degraded for simulations involving large
amounts of data output. For simulations involving no
data output, there is significant performance degrada-
tion when increasing from 1 to 27 worker processes,
however total execution time thereafter remains near-
constant. Simulating the interactions of 241 540 parti-
cles (in 27 subdomains) takes 1104.5 s of real time. By
comparison, simulations of 8 700 121 particles (in 1000
subdomains) takes 1617.5 s, an increase of 500 s over
10 000 timesteps. Although the number of particles for
1000 workers is 3600% more than for 27 workers, the
time involved increases only by 46%.
The initial increase in execution times arises due
to the increasing workload of one or more worker
processes as the number of subdomains increases. For
one worker process, there is no computation or com-
munication overhead from subdomain decomposition
(due to the lack of neighbouring subdomains). For
the eight-worker simulation (2 × 2 × 2 subdomains),
workers share at most three subdomain boundaries
with neighbouring subdomains, whereas for more than
27 workers, at least one worker shares the maximum
number of subdomain boundaries with its 6 neighbour-
ing workers. Since interprocess synchronisation occurs
only prior to the force summation and position update
stages, increasing the number of workers with a full
complement of shared subdomain boundaries causes
relatively little increase in the total time to execute a
single timestep of the simulation. The slight increase in
execution time as the number of workers increases is
likely due to a combination of communications latency
and variations in processor speeds.
For simulations in which field saver output occurs,
the total execution times closely track those of simula-
tions without data output. Field saver output amounts
to a constant time overhead (approximately 200 s over
10 000 timesteps), even for simulations with larger
numbers of worker processes. For field saver output
the increase in simulation time between 27 and 1000
workers is 37%. We also find that simulations of a

281
 Simulation Time for Number of Worker Processes
4500
4000
3500
3000
Time (s)
2500
2000
1500
1000
500
0
0 100 200 300 400
Workers
Figure 5. Simulation execution times for four experiments: No data output, field saver output, checkpoint
file output, and both field saver and checkpoint file output.
given number of workers and field saver output are
typically only between 10% and 20% slower than
counterparts without data output. These results testify
that inclusion of simple data output in the form of a
field saver is not a significant overhead for simulations
involving larger numbers of particles.
Simulations in which checkpoint files are saved also
display weak scalability, albeit with a more significant
linear increase in average execution time. Whereas the
field saver only requires approximately 10 B of data
to be stored per timestep, the amount of data saved
in checkpoint files scales with the number of particles
in the simulation. Approximately 2.3 MB of data are
stored per worker process which, for the largest simu-
lations, results in approximately 2.6 GB being stored to
disk every 100 timesteps. We observe that runtimes for
simulations involving 1000 workers are approximately
three times that of simulations with 27 workers. The
cause for this linear performance degradation is un-
doubtedly a filesystem bandwidth bottleneck, although
further analysis is required to determine the extent to
282
Output
None
FS
CP
FS & CP
500 600 700 800 900 1000
which such a bottleneck may be mitigated. It should
be noted that storage of checkpoint files every 100
timesteps is highly unusual for production simula-
tions. Typically checkpoint files would be stored every
10 000 timesteps for simulations comprising between
100 thousand and 1 million timesteps. In such cases,
the performance degradation from data output would
be negligible relative to the computational burden.
Writing both field savers and checkpointers does not
increase the total execution time significantly, com-
pared with writing checkpointers alone. This confirms
the previous observation that storing field saver output
is a relatively trivial overhead compared with the actual
computations and storage of simulation state data in
checkpoint files. For many production simulations,
storage of checkpoint files is mandatory for post-
analysis of simulation results and three-dimensional
visualisation. Storage of field saver data is typically
useful when post-analysis would be exceptionally time-
consuming to obtain the desired output data.
6. Conclusion [6] T. Pöschel and T. Schwager, Computational Granular
Dynamics: Models and Algorithms. Springer, 2005.

A scaling benchmark for ESyS-Particle, a paral-
lel implementation of the Discrete Element Method,
shows that ESyS-Particle scales well up to at least 1000
cores when simulating up to 8.7 million particles in a
three-dimensional elastic wave propagation problem.
The increase in computation time is as little as 46%
on 1000 cores compared with 27 cores. The trend for
simulation data output similarly shows a flattening of
the curve toward constant time for increasing worker
processes. On the basis of our analysis, ESyS-Particle
is demonstrably an efficient platform for parallel DEM
simulations, scaling to beyond 8.7 million particles
on cluster supercomputers. One would expect ESyS-
Particle to continue to display weak scalability well
beyond the limit of 1000 worker processes considered
in our analysis.
[7] L. Verlet, “Computer ‘experiments’ on classical flu-
ids. I. Thermodynamical properties of Lennard-Jones
molecules,” Phys. Rev., vol. 159, no. 1, p. 98, Jul 1967.
[8] S. Abe, “Investigation of the influence of different
microphysics on the dynamic behaviour of faults using
the lattice solid model,” Ph.D. thesis, The University of
Queensland, 2001.
[9] S. Abe, D. Place, and P. Mora, “A parallel implemen-
tation of the lattice solid model for the simulation of
rock mechanics and earthquake dynamics,” Pure. Appl.
Geophys., vol. 161, pp. 2265–2277, 2004.
[10] W. Gropp, E. Lusk, N. Doss, and A. Skjellum, “A high-
performance, portable implementation of the MPI mes-
sage passing interface standard,” Parallel Computing,
vol. 22, pp. 789–828, 1996.

Acknowledgments

This work was supported by AuScope, funded by

the Australian Federal Government National Research
Infrastructure Strategy. The authors wish to acknowl-
edge the useful contributions by colleagues at the
Earth Systems Science Computational Centre. ESyS-
Particle is Open Source Software, freely available from
https://launchpad.net/esys-particle. The SGI cluster su-
percomputer used in this work was provided by the
Queensland Cyber Infrastructure Foundation and The
University of Queensland.

References

[1] P. A. Cundall and O. D. L. Strack, “A discrete nu-

merical model for granular assemblies,” Geotechnique,
vol. 29, pp. 47–65, 1979.

[2] K. Yamane, “Discrete-element method application to

mixing and segregation model in industrial blending
system,” Journal of Materials Research, vol. 19, pp.
623–627, 2004.

[3] W. Hancock and D. Weatherley, “3D simulations of

block caving flow using ESyS-Particle,” in 1st Southern
Hemisphere International Rock Mechanics Symposium,
2008, pp. 221–229.

[4] S. Latham, S. Abe, and M. Davies, “Scaling evaluation

of the lattice solid model on the SGI Altix 3700,” in
Proc. HPCAsia’04, 2004.

[5] S. Latham, S. Abe, and P. Mora, “Parallel 3-D simu-

lation of a fault gouge using the lattice solid model,”
Pure. Appl. Geophys., vol. 163, pp. 1949–1964, 2006.

283