A COMPLETELY AUTONOMOUS SYSTEM THAT LEARNS ANOMALOUS MOVEMENTS
IN ADVANCED VIDEOSURVEILLANCE APPLICATIONS
Alessandro Mecocci, Massimo Pannozzo
University of Siena – Dept. Information Engineering
Via Roma , 56 – 53100 SIENA (Italy)
Phone: +39 0577 233604; email: alemecoc@tin.it
ABSTRACT
This paper describes an automatic real-time video
surveillance system, capable of autonomously learning
and signaling anomalous activities of moving objects. To
obtain these capabilities, an improved version of the
Altruistic Vector Quantization algorithm (AVQ) is
proposed. The modified AVQ automatically evaluates the
number of trajectory prototypes, and improves the
representativeness of the prototypes themselves, so the
visual events can be easily and accurately classified.
Anomalous behaviors are detected if visual trajectories
deviate from the self-learned representations of "typical"
behaviors. The system has been implemented by means of
standard PCs and TV cameras, and has been tested in
many real outdoor contexts in different conditions (night
and day). Currently it is used to monitor the storage areas
of British Airways at the airport of Peretola (Florence,
Italy), and some access gates of Autostrade per l’Italia
S.p.A. (the main Italian highways company). If the
camera field-of-view is changed, the system automatically
re-learns new “typical” behaviors and accurately detects
anomalous events.
1. INTRODUCTION
Nowadays, important research activities are dedicated to
automatic techniques that allow the characterisation of
moving objects. In fact, crimes, robberies, accidents and
terrorist attacks are highly increasing, and it is becoming
necessary to develop systems capable of autonomously
giving support for the early detection, characterization,
and real-time classification of visual events [1], [2], [3],
[4]. In general, the detection of anomalous events is based
on the description of specific motion patterns that can be
considered safe (acceptable motion behaviours). Once the
acceptable motion patterns have been defined, the current
movements of each tracked target are compared to the
acceptable patterns, searching for any significant
deviation. In the positive case, an alarm is given and
predetermined actions are performed. This paper
describes a system capable of automatically acquiring
“safe“ motion-patterns without any human intervention.
The self learning reliability and robustness is granted by a
modified version of the Altruistic Vector Quantization
technique and by a new methodology for automatically
selecting the optimum number of possible motion
patterns.
The proposed approach makes reference to the
original idea of Johnson and al. [5] that proposed to
describe motion trajectories by using the Altruistic Vector
Quantization (AVQ) algorithm, to filter out the intrinsic
variability of each specific trajectory, while preserving the
stable characteristics of each group of trajectories. In our
system two important improvements have been
introduced: 1) a check on motion vectors is performed to
increase the accuracy during the data learning phase; 2) an
automatic technique is used to define the number of
prototypes to be used during the training. Without such
improvements the original AVQ approach gives
unacceptable results in practical applications.
2. AVQ ALGORITHM OVERVIEW
To describe the behaviour of all objects that are moving in
a typical scene subjected to video-surveillance, it is
necessary to observe their movements for a certain time,
and to extract a suitable description of such movements.
For example, if the persons in a hall generally tend to
move up and then turn right to go out, this kind of general
visual pattern must be learned and synthesized in a model
that tolerates the intrinsic variations of each individual
trajectory while preserving enough information to decide
if each specific trajectory deviates from normality.
This important phase has been solved by means of a
Modified Altruistic Vector Quantization that is capable of
learning some Prototypes that approximate the pdf both of
the Instant Positions, and of the Generalized Trajectories
of multiple targets [6]. The two sets of calculated
prototypes have been called, respectively: Prototypes of
State and Prototypes of Trajectory.
2.1 Altruistic Vector Quantization (AVQ)
An Activation Function a i ( t ) is associated to each
G
Vector Quantization is an iterative procedure capable of sv p
State Prototype . The values of the Activation
learning the statistical arrangement of discreet sets of Function are used to define a Normalized Trajectory by
points (Instant Positions and Generalized Trajectories) by associating them to each Prototype:
means of a set of Prototypes that are allowed to displace
T j = {a0 (t ), a1 (t ),..., a NPS −1 (t )} (4)
themselves until they adequately describe the set of
training trajectories (see [7] for a more diffuse treatment).
The function i
a (t )
is updated each time a new
Nevertheless, such an approach shows some serious
component of the Generalized Trajectory is analyzed:
drawbacks in real operative conditions. In particular, the
algorithm tends to be trapped into local minima giving a i ( t + 1) = Oi (t ) if Oi ( t ) > ai ( t ) (5)
rise to not accurate models and even to gross errors. To
a i ( t + 1) = γ a i ( t ) if O i ( t ) ≤ a i ( t ) (6)
obtain the convergence to a global minimum, a Sensitivity
factor has been proposed in [1] [2]. The Sensitivity factor The index γ ∈ [0,1] is said Decadence Constant and
of the winning Prototype is decreased so the winner represents the memory of each Activation. The exit
cannot participate to the learning for a certain amount of
time. On the contrary, if the Prototype loses, its
Oi (t ) is computed in the following way:
G G
Sensitivity factor is increased (this improves its f k j − p si (7)
probability to be involved in subsequent learning). The O i (t ) = 1 −
d MAX
Sensitivity factor has the following form:
si (t + 1) = ς ⋅ si (t ) + Ai (1) The distance MAX d
is the maximum distance
detected in the learning set. Each component of the
where ς (the Shading Factor) and Ai are defined as
Normalized Trajectory comprises the activation functions
follows: of each State Prototype:
β T j = {a0 (d j − 1), a1 (d j − 1),..., aNPS −1 (d j − 1)} (8)
ς = 1− (2)
(k − 1) ⋅ n where NPS is the Number of State Prototypes, and
Ai = − β if i is the index of the winning prototype dj
is the duration of the Generalized Trajectory.
β According to the previous definition, it is evident
Ai = if i is not the winning prototype where:
k −1 Tj NPS
that the Normalized Trajectories belong to [0,1] .
β is a positive constant in the interval [0,1].
The exit factor, i
O (t )
, guarantees that each vector
Ai (called “Adjustment Factor”), has two purposes:
component is in the interval [0,1]. The activation of each
it allows the prototypes that lost the competition to State Prototype is used to preserve the temporal
β information of each Generalized Trajectory. In fact, the
increase their sensibility by a factor of . In this way,
k −1 Normalized Trajectory maintains memory of all the State
Prototypes that have been activated.
after k-1 steps, the highest increase-rate is equal to β .
Even if the proposed normalization step solves many
problems of the original AVQ algorithm, in real
2.2 Problems with the AVQ algorithm environments some major operative difficulties have
emerged. In particular, by looking at figure 1, two set of
Even if the classic AVQ algorithm can give good results, symbols can be distinguished. The first set comprises the
some problems occur when it is applied to real real object positions (little spots in figure 1); the second
trajectories. A first problem is due to the different set comprises the estimated Prototypes given by the AVQ
duration of each trajectory. In fact, the AVQ needs input algorithm (larger circular spot in figure 1). It is evident
vectors of the same length. To solve such problem we that the estimated Prototypes lie in completely wrong
have applied a normalization technique: the whole positions with respect to the real object position. This is
Generalized Trajectory is analyzed and for each an unacceptable result because the Prototypes position are
Gj
component f k the nearest State Prototype is obtained: used to describe the objects motion.
G G G G
f k j − p sv = min f k j − p si (3)
1≤ i ≤ NPS
 trajectories. This happens even if the targets follow
different motion directions. This is not acceptable for
practical visual inspection applications, where we want to
discriminate the spatio-temporal behaviour (that also
involves motion direction).
To solve this problem it is necessary to automatically
define the correct Number of Trajectory Prototypes. The
proposed approach extends the original AVQ formulation
Figure 1 Learned Prototypes by the AVQ algorithm. by adding directional information obtained during the
learning phase. For each Normalized Trajectory
3. IMPROVEMENTS TO THE AVQ ALGORITHM T j ∈ [0,1]NPS during the learning we define a set of
To solve this problem, we propose another modification Generalized Directions, DGj , as follows:
that changes the strategy used to update the decay factors
of the original AVQ algorithm. The original AVQ moves DGj = {RIj , DMj , RUj } (11)
G where:
the Winning Prototype ( mv (t ) ) towards the Observed
G RIj : represents the start region in the scene (where
Vector ( X j (t ) ) according to the following formula:
G Trajectory j begins);
G G G
mv (t + 1) = mv (t ) + α (t )[ X j (t ) − mv (t )] (20) DMj : is the mean direction of the Trajectory j in the
The decay factor α ( t ) of the winning prototype has scene;
the following form: RUj : is the exit region in the scene (where Trajectory j
2⋅t
α (t ) = 1 − 0.99 ⋅ ( ) 0 ≤ t ≤
N finishes).
N if 2 (9) The Generalized Directions are analyzed and
N
α (t ) = 0.01 if 2
≤ t ≤ N
(10) clustered to define the Number of different direction
where N is the maximum allowed number of iterations, classes in the set (called Number of Directions
and t is the current iteration number. Each Normalized Generalized, NDG). The NDG is used to automatically fix
Trajectory is a vector that has non null components only the number of prototypes to be used by the AVQ
in those positions interested by movements. All the other algorithm. In this way the number of Trajectory
positions (static areas in the scene) give rise to zero Prototypes automatically follows from the scene
values. If the original AVQ formulation is used, all the conformation and from the specific behaviour of the
components of the winning prototype are updated even if targets and can be automatically re-defined when the
they relate to areas not affected by any target movement field-of-view changes.
(components with zero or negligible values). As a
consequence, the winning prototype tends to move 4. ANOMALY DETECTION
towards erroneous zones in the scene. In the proposed
approach, the components of the Trajectory Prototypes are The Anomaly Detection is performed by associating a
updated only if they relate to non null motion components Convergence Region to each State and Trajectory
of the current Trajectory. With this modification the prototype. The Convergence Region is based on the Mean
results dramatically improve. distance and Variance of each Prototype from the true
target positions represented by the Prototype itself. For
Another problem of the original AVQ algorithm, is G
the fixed number of Prototypes that must be specified a each component of the Observed Vectors Vi , during the
priory during the learning phase. Even if this is not a G
Learning phase, the nearest Prototype pv is computed by
problem in the case of State Prototype (the number is
proportional to the image resolution), it turns out to be a an Euclidean metric:
G G G G
major problem in the case of Trajectory Prototypes. In Vi − pv = min Vi − p j (12)
1≤ i ≤ NPS
fact, the number of Trajectory Prototypes depends on the
the Mean Distance and S.D. from the winning prototype is
scenery and cannot be given a priory (for example if the
updated by:
camera field-of-view changes). On the other and, this Nv G G Nv G G
parameter is very important because if it is too large or too ∑V i − pv j
( ∑ Vi − pv j ) 2
(13)
small, the learning process gives poor performances. In µv = j =1
; σv =
j =1
− µv2
particular, when a too small number of prototypes is Nv Nv
specified, AVQ cannot distinguish between intersecting
 The index N v represent how many times the
considered prototype is the winner.
The Convergence Region of each Prototype is an
G represented in yellow or in red according to the gravity of
Hyper-sphere centred in
pv , with a radius of
µv + 3 ⋅ σ v .
After each Convergence Region has been defined, an
Anomaly Detection Module analyzes the Generalized
Trajectory of each target and its Normalized Trajectory.
For each Object the Instant Positions and the Normalized
Trajectory are considered: if the Instant Position lie
outside the Convergence Regions associated to the
corresponding nearest prototypes, an Anomaly State is
triggered. A similar process is used to verify the
Normalized Trajectory to decide if a Trajectory Anomaly
State must be raised.
5. EXPERIMENTAL RESULTS
The behaviour of the proposed system has been
extensively tested in different operating conditions (both
in inside and in outside environments). Figure 2, for
example, shows an indoor scene (left) of 200 trajectories
(right) used by the system during the Learning Phase. The
modified AVQ Algorithm has been used (1.000.000
iterations) giving 8 different Prototypes of Trajectory in a
fully automatic way (no manual parameter adjustment).
Two out of the 8 mentioned Prototypes are shown in
Figure 3.
Figure 2 Indoor environment: Peretola (Florence, Italy) airport.
The good performance can be easily assessed by
visually comparing figure 15 and figure 16.
Figure 3 2 out of the 8 Trajectory Prototypes automatically
learned by the Modified AVQ.
The Anomaly Detection results (obtained in real-
time) are shown in Figure 4, where “safe” trajectories are
represented in green, while anomaly trajectories are
the anomaly itself (evaluated by the system).
Figure 4 Results obtained in real-time by the system.
The described system is currently working in the
Peretola airport to monitor the cargo-boarding-areas of
British Airways. It must be emphasized that the detection
system works in complete autonomy: just decide the
scene, start the learning and then begin the anomaly
detection and logging.
8. REFERENCES
[1] Nils T. Siebel, Design and Implementation of People
Tracking Algorithms for Visual Surveillance Applications,
University of Reading, March 2003.
[2] V.A., “Special Section on Video Surveillance”, IEEE Trans.
on Pattern Analysis and Machine Intelligence, IEEE, August
2000.
[3] Wixson, L. “W4: real-time surveillance of people and their
activities” IEEE Trans. On Pattern Analysis and Machine
Intelligence, Vol. 22, pp. 774-780, Aug. 2000.
[4] F.M. Porikli, "Trajectory Pattern Detection by HMM
Parameter Space features and Eigenvector Clustering",
European Conference on Computer Vision (ECCV), May 2004.
[5] Johnson, N and Hogg, D C, “Learning the Distribution of
Object Trajectories for Event Recognition”, Proc. BMVC95,
Birmingham, England, 1995.
[6] Mecocci A., Pannozzo M. Fumarola A. “Automatic detection
of anomalous behavioural events for advanced real-time video
surveillance” IEEE CIMSA '03, Wien, pp. 187–192, 2003.
[7] Kohonen T. “The Self-Organizing Map”, Proceedings of The
IEEE, 78(9): pp. 1464-1480, 1990.