Proceedings of the First International Conference on Machine Learning and Cybernetics, Beijing, 4-5 November 2002

HMMS (HIDDEN MARKOV MODELS) BASED ON ANOMALY

INTRUSION DETECTION METHOD
BO GAO“’, HUI-YE MA‘b’,YU-HANG YANG“’

@’Departmentof Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China

‘b’Centerfor Space Science and Applied Research, Chinese Academy of Sciences
E-MAIL: gaobo@sjtu,edu.cn, pony@nc.poac.ac.cn

Abstract: “normal” database that contains all possible short sequences

In this paper we discuss our research in developing (e.g., of length 11) of system calls for each program (mail-
anomaly detecting method for intrusion detection. The key sending, [pr, etc.) that needs to be guarded. The normal
idea is to use HMMs (Hidden Markov models) to learn the database is then used to examine the behavior of a running
(normal and abnormal) patterns of U N x processes. These program (e.g., an instance of mil-sending). If the total
patterns can he used to detect anomalies and known intrusion. number (or percentage) of abnormal sequences, which are
Using experiments on the muiI-sending system call data, we
demonstrate that we can Constrnct coucise and accurate those that don’t match the normal sequences in the normal
classifiers to detect intrusion action. database, is above a threshold value, then the current run is
flagged as abnormal, i.e., a misuse or anomaly intrusion is
Keywords: detected.
Intrusion detection; Anomaly detecting; HMMS (Hidden There are many ways in which system call data could be
Markov Models); Machine learning; System call used to characterize normal behavior of programs, each of
which involves building or training a model using traces of
1 Introduction normal processes. A machine learning approach to this
problem would construct a finite state machine to recognize
Approaches to intrusion detection can be broadly divided the “language” of the program traces. A particularly
into anomaly detection and misuse detection. Misuse powerful finite state machine is the Hidden Markov models,
detection based approaches [‘.2,31 define and look for precise used widely in s eech recognition and also in DNA
sequences of events that damage the system. Misuse sequence modeling ? % ~ O ,
detection approaches are more precise and less prone to We get our experiment data, which are a set of traces of
false positives. However, since misuse detection the mail-sending program, from Stephanie Forrest. These
approaches require specification of damaging events, which traces were used io the experiments reported in (Forrest et
is usually manual and based on previously known attacks, What we want to know is whether a machine learning
they are less effective against newly discovered approach based on HMMs can be used to learn the normal
vulnerabilities and anacks. Anomaly detection based andor abnormal behavior of the process. Consequently, we
.approaches first create a profile that describes normal want to know whether our approach can detect known and
behaviors and then detects deviations from this profile [4’s’61. unknown intrusion more accurately.
Anomaly detection approaches possess the advantage that The rest of the paper is organized as follows: Section 2
learning to identify normal behavior can be automated, but presents the HMMs and how to use HMMs in our
they are prone to false positives, especially when experiments. Section 3 describes the details of our
permissible but previously unlearned behavior occurs. experiment. Section 4 presents the results of our
An alternative approach is taken by Fink, Levitt and KO experiments and analyses them in details. Section 5 outlines
[’I. hstead of trying to build up normal user profiles, they some open issues and avenues for future work. Section 6
focus on determining normal behavior for privileged makes our conclusions.
processes, those that run as root. They define normal
behavior using a program specification language, in which 2 Hidden Markov Models
the allowed operations (the system calls and their
garameters) of a process are formally specified. Forrest et al 2.1 Hidden Markov Models Description
I I introduced a novel and simpler method. They gathered
the traces of normal runs of a program and analyzed the The Hidden Markov Models is a finite set of states, each
“local (short) range ordering of system calls”. They of which is associated with a (generally multi-dimensional)
discovered that these local orderings “appears to be probability distribution. Transitions among the states are
remarkably consistent, and this suggests a simple definition governed by a set of probabilities called transition
of normal behavior”. The key idea here is to build a probabilities. In a particular state an outcome or observation

0-7803-7508-4/02/$17.00 02002 IEEE

381
Proceedings of the First Internadonal Conference on Machine Learning and Cybernetics,Beijing, 4 5 November 2002
can be generated, according to the associated probability
distribution. It is only the outcome, not the state visible to
an external observer and therefore states are “hidden” to the
outside. This is what the name Hidden Markov Models
comes from.
Traditionally, people have used Markov model to
successfully model a lot of real world processes. But for
some other processes, the strict assumption of Markov that
next state is dependent only upon the current state will not
hold, thus we need to find more general models to deal with
these processes while at the same time withhold some good
properties of Markov model. These principles motivate
people to generate the Hidden Markov Models. Hidden
Markov Models is a double embedded stochastic process
with two hierarch levels. The upper level is a Markov
process that the states are unobservable. Observation is a
probabilistic function of the upper level Markov states.
Different Markov states will have different observation
probabilistic functions.
The two hierarchy-level structure is the main idea and
advantage of HMMs. It can be used to model much more
complicated stochastic processes than traditional Markov
model. In speech recognition, HMMs have been widely
used for analysis human auditory signals as speech patterns
[’I. In modeling dynamic human control strategy, “‘I uses
HMMs to classify different human’s behavior patterns.
Transient sonar signals are analyzed with HMMs for ocean
surveillance [I2].
2.2 -
Usine HMMs model to characterize the behavior
of programs
Because Hidden Markov Models (HMMs) are statistical
recognition,speech recognition, and modeling of biological
sequences, and our traces are sequences of system calls
(“normal” and “abnormal”) from a privileged program, we
want to use HMMs to learn the pattern of normal andlor
abnormal behavior of the process. Standard HMMs have a
fixed number of states, so one must decide on the size of
the model before training. Preliminary experiments show us
that a good choice for our application is to choose a number
of states roughly corresponding to the number of unique
system calls used by the program. Our test programs use an
alphabet of about 60-sFte HMMs for mail-sending. The
states are fully connected transitions are allowed from state
to state. For each state then, we need store the probabilities
associated with transitions to another state, and the
probabilities associated with producing each system call.
For a program using S system calls, and hence a model of S
states, this means roughly 2S2 values. Transition and
symbol probabilities are initialized randomly, and then
trained using the Baum-Welch algorithm as described id9’.
3 Experiments
382
3.1 Profiling Normal and Abnormal Behavior
The first thing that we need to do is to profile normal and
abnormal behavior from the view of a specified process,
e.g., mail-sending. We have obtained two sets of mail-
sending system call data. The procedures of generating
these traces are described in (Forrest et al. 1996) [SI.Each
trace is the list of system calls issued by a single process
from the beginning of its execution to the end. Each trace
file lists pain of numbers, one pair per line. The first
number in a pair is the PID of the executing process, and
the second is a number representing the system call. The
mapping between system call numbers and actual system
call names is given in a separate file. For example, the
number 10 represents system call “execv”. Forks are traced
individually, and their traces are included as part of normal.
The set of traces includes:
Normal traces: a trace of the mail-sending daemon and
a concatenation of several invocations of mail-sending.
Abnormal traces: 2 traces of the syslog-remote
intrusion, 2 traces of the sscp intrusion, 2 traces of the
syslog-localintrusion, 2 traces of the decode intrusion,
I trace of the sm5x intrusion and 1 trace of the sm565a
intrusion. These traces come from the abnormal runs
of the mail-sending program which is exploited by
above intrusion tools. The mail-sending daemon deals
with incoming mails and all other processes deal with
outgoing mails.
Table 1. System Call Data. Each file has two columns, the
q i d s and the system call numbers.
lpID I 1381 1381 ... 1391 1391... 1
I
Systerncdh 5 2 3 4 5 4 2766 ...19 19 105 1 0 4 ...
The algorithm used to build the profile of normal and
abnormal behavior is extremely simple. Since short
sequences of system calls are important characteristics of a
program’s normal behavior. We can use a slide window to
cross the trace of mail-sending, and record which calls
follow which within the sliding window. By recording these
short sequences we can get the profile of normal behavior.
Following (Forrest et al 1996)@’,a sliding window of length
11 seemed to give the best predictive performance.
Therefore, we also use a sliding window of length 11 with
sliding (shift) step of 1 to create sequences of system calls
from the traces. We first use a sliding window to scan the
normal traces (of the mail-sending daemon and mail-
sending) and create a list of unique sequences of system
calls, 1,082 in total. We call this list the “normal” list. Next,
we check each of the intrusion traces against it using the
same method. We slide a window of size 11 across the
intrusion traces, determining if the sequence of system calls
differs from that recorded in the normal list. If a match can
he found then the sequence is labeled as “normal”.
Otherwise it is labeled as “abnormal”. See Table 2 for an
example of the labeled sequences. All sequences in the
Proceedings of the First International Conference on Machine Laruing and Cybernetics,Beijing, 4 5 November 2002
normal traces are labeled as “normal”. It should be noted
that for an intrusion trace not all the sequences are
“abnormal” since the illegal activities only occur in several
places within a trace.
Table 2. Classified System Call Data.
I System call (1 1) I ClaSsLabels
52345421665426666 ” n o d
3.2 Experimental Setup
We formulate our learning method as followings:
0 The training data is composed of all normal sequences,
plus the abnormal sequences from 1 traces of the sscp
intrusion, I trace of the syslog-local intrusion, and I
trace of the syslog-remote intrusion.
0 The testing data is all the sequences (normal and
abnormal) in the intrusion traces not included in the
training data.
The HMMs can be used to predict whether a sequence is
“abnormal” or “normal”. But what the intrusion detection
system needs to know is whether the trace being tested is an
intrusion or not. Can we say that whenever there is an
“abnormal” sequence in the trace, it is an intrusion? It
depends on the accuracy of our HMMs when classifying a
sequence as “abnormal”. Unless it is close to loo%, it is
unlikely that a predicted “abnormal” sequence is always
part of an intrusion rather than just an error.
3.3 Detecting Abnormal Behavior
We use the following algorithm to detect whether the
trace is an intrusion based on the HMMs predictions of its
constituent sequences:
1. Use a sliding window of length 2n+l, e.g., 7, 9, 11, 13,
etc., and a sliding (shift) step of n, to scan the predictions
made by HMMs.
2. For each of the regions (of HMMs predictions)
generated in Stepl, we use n to divide the sum of
“abnormal” predictions, and we notate the quotient as w. If
w is less than 1, the current region of predictions is a
“normal” region, else the current region is an “abnormal”
region and its weight is w.
3. We add all of the weights of “abnormal” regions, and
notate the sum as S. Then we use the number of all regions
to divide the S to get the percentage of “abnormal” regions,
and we notate it as a. If a is above a threshold value, say
lo%, then the trace is an intrusion.
This algorithm can filter out the false positives (i.e.,
classified a “normal” sequence to an “abnormal” sequence).
The principle behind this algorithm is that when an
intrusion actually occurs, it generates a number of abnormal
system calls, and as a result, the neighboring sequences of
system calls will not match the normal sequences. However,
the prediction errors tend to be isolated. We think the
weight of a region indicates its “abnormal” degree. A
number of abnormal system calls generated by an intrusion
action will lead to more abnormal sequences in a region
than a normal action, so we can get a greater weight. But a
prediction error only leads to very sparse “abnormal”
sequences, so the weight of this region is light. In (Forrest
I et al. 1996) [*I,the percentage of the mismatched sequences
(out of the total number of matches (lookups) performed for
the trace) is used to distinguish normal from abnormal. Our
algorithm is different in that ‘we look for “abnormal
regions” that contains more “abnormal” sequences than the
“normal” ones, and calculates the percentage of abnormal
regions (out of the total number of regions). Our algorithm
is more sensitive, and less false alarm.
4 Results
We now analyze the results of our experiments. We
demonstrate that our machine learning approach is useful in
detecting misuse and anomaly intrusions. We need include
all the unique normal sequences of mil-sending. The
abnormal sequences are taken from sscp-1, syslog-local-1,
and syslog-remote-l. We compare the results of the
following experiments that have different distributions of
“abnormal” versus “normal” sequences in the training data:
1. Experiment A: I copy of all unique normal sequences
and 1 copy of the abnormal sequences.
2. Experiment B: 4 copies of all unique abnormal
sequences.
3. Experiment C 4 copies of all unique normal
sequences.
4. Experiment D: 4 copies of all unique normal
sequences and 4 copies of the abnormal sequences.
Each copy of the abnormal sequences has 1,315
sequences. We test the performance of the classifiers
generated by HMMs on every intrusion trace by supplying
all the sequences (abnormal and normal) of the trace to the
classifiers. To detect abnormal behavior, with a sliding
window of length 9, is applied to the predictions of the
classifiers. Note that the window length here specifies the
size of the regions of predictions, which can be different
from the length of the sequences of system calls. Table 3
shows the anomaly detection results of these experiments
compared with the results from Forrest et al. (1996) [*I,
From Table 3, we can see that the classifier from
Experiment A is not acceptable because it classifies every
trace (including mil-sending) as an intrusion. This is due
to too few examples of “normal” and “abnormal” in the
training data (each unique “normal” sequence only appears
once). The classifier from Experiment B does better than
the classifier from Experiment A. It is important to note that
the classifiers from Experiment B perform quite well on
known intrusions, i.e., sscp-2, syslog-local-2, and syslog-
remote-2, because the training data includes the abnormal
sequences from the traces of the same types of intrusions.
But they perform relatively poorly on unknown intrusions
(the abnormal sequences of the traces of these types of
383
Proceedmgs of the First International Conference on Machme Learning and Cybernetics, Beijing, 4-5 November 2002
intrusions are not in the training data), i.e., decode- 1&2,
sm565a, and sm5x. This result shows ns that classifiers with
“abnormal” sequences are only good for detecting known
intrusions and hence don’t generalize to other “unseen”
intrusions. The classifier from Experiment C, which has the
4 copies of “normal” sequences, is an improvement. It
predicts correctly on all intrusions, including the known and
the unknown, but does worse than D on the known
intrusions i.e., sscp-2, syslog-local-2, and syslog-remote-2,
because in Experiment D, we have not only normal
sequences but also abnormal sequences to train our HMMs.
Our HMMs have knowledge about both normal and
abnormal sequences, and if the classifier trained with
normal sequences or the classifier trained with abnormal
5 Discussion and Future Work
These experiments teach us a very important lesson that
is in order to detect unknown anomalous behavion. it is
pivotal that a model of the normal behavior of the program
be used. This confirms the intuition of Forrest et al. (1996)
as well. Recall that we have chosen to define normal in
terms of short sequences of system calls. In the interests of
simplicity, we ignore the parameters passed to the system
calls, and look only at their temporal orderings. This
definition of normal behavior ignores many other
important aspects of process behavior, such as timing
information instruction sequences between system calls,
and interactions with other processes. Certain. intrusions
may only he detectable by examining other aspects of
process behavior, and so we may need to consider them
later.
6 Conclusions
We applied a machine leaming approach based on
HMMs to leam normal and abnormal patterns of program
behavior from its execution trace to generalize upon the
method introduced in (Forrest et al. 1996) ‘*], The resultant
normal patterns (classifiers) are shown to be able to
accurately detect anomalous intrusions. Our experiments
384
sequences detects an intrusion, we can say we detect this
intrusion. So the performance to recognize the known
intrusion is better thm C. Therefore, the classifier from
Experiment D has the hest performance. It correctly
classifies every tracc of the known and the unknown
intrusions. Experiment C and D also confirm our conjecture
that classifiers for the “normal” sequences can be used for
anomaly intrusion detection, thus generalizing the notion of
normalcy. The results from Forrest et al. (1996) showed
that their methods required a very low threshold in order to
correctly detect the decode and smS6Sa intrusions.
Comparing with it our results showed that our approach
generated much “stronger signals” of anomalies from the
intrnsiontraces.
demonstrate that machine learning can indeed play an
important role in intrusion detection of computer systems.
Much more research needs to he pursued in order to build
a system that can much more rapidly and correctly detect
intrusions.
References
P. Porras, R.Kernmerer. Penetration State Transition
Analysis-A Rule Based Intrnsion Detection Approach.
Computer Security Applications Conference, 1992.
K. Ilgun. A real-time Intrusion detection system for
UNIX. IEEE Symp. on Security and Privacy, 1993.
SKurmar, E. Spafford. A Pattern-Matching Model
for Intrusion Detection. National Computer Security
Conference, 1994.
K. Fox, R. Henning, J. Reed, R. Simonian. A Nueral
Network Approach Towards Intrusion Detection.
National Computer Security Conference, 1990.
T. Lunt. A Real-Time Intrusion Detection Expert
System (IDES)-Final Report. SRI-CSL-92-05, SRI
International, 1992.
D. Anderson, T. Lunt, H.Javitz, A. T a m m , A.
Valdes. Next-generation Intrusion Detection Expert
System(NIDES): A Summary, SRI-CSL-95-07, SRI
International, 1995.
Proceedings of the First International Conference on Machine Learning and Cybernetics, Beijing, 4 5 November 2002

[7] C KO, G Fink, K Levitt. Automated Detection of

Vulnerabilities in Privileged Programs by Execution
Monitoring. in Proceedings of the 10th Annual
computer Security Applications Conference pp.134-
144,1994.
[8] Stemphanie Forrest, Steven A. Hofmeyr, Ani1
Somayaji, Thomas A. Longtaff. A Sense of Self for
Unix Processes. In Proceedings of the 1996 IEEE
Symposium on Security and Privacy, IEEE Computer
Society Press,Los Alamitors, CA pp.120-128.
[9] L. R .Rabiner. A tutorial on Hidden Markov Models
and selected applications in speech recognition.
Proceedings of the IEEE, 77(2):257-286,1989.
[IO] L. R. Rabiner, B. H. Juang. An introduction to
Hidden Markov Models. IEEE ASSP Magazine,
pp.4-16, January 1986.
[ I l l M. C. Nechyba, Y. Xu. Stochastic Similarity for
Validating Human Control Strategy Models.
[I21 Kundu, G C. Chen, C. E. Persons. Transient Sonar
Signal Classification Using Hidden Markov Models
and Neural Nets. IEEE Journal of Oceanic
Engineering, 1911):87-99, 1994.

305