Introduction to Data Mining

Chapters 8 & 9

Cluster Analysis
What Is Cluster Analysis

 Cluster analysis groups data objects based only

on information found in the data that describes
the objects and their relationships. The goal is
that the objects within a group be similar to one
another and different from the objects in other
groups. The greater the similarity within a group
and the greater the difference between groups,
the better or more distinct the clustering.
 Segmentation and partitioning.

2
What is Cluster Analysis?

Inter-cluster
Intra-cluster distances are
distances are maximized
minimized

3
Applications of Cluster Analysis
 Understanding
– Classes, or conceptually
meaningful groups of objects
that share common
characteristics, play an
important role in how people
analyze and describe the
world.
– Group related documents for
browsing, group genes and
proteins that have similar
functionality, or group stocks
with similar price fluctuations
 Summarization
– Reduce the size of large data
sets
Discovered Clusters Industry Group
Applied-Matl-DOWN,Bay-Network-Down,3-COM-DOWN,
1 Cabletron-Sys-DOWN,CISCO-DOWN,HP-DOWN,
DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN,
Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down,
Technology1-DOWN
Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN,
Sun-DOWN
Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN,
2 ADV-Micro-Device-DOWN,Andrew-Corp-DOWN,
Computer-Assoc-DOWN,Circuit-City-DOWN,
Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN,
Technology2-DOWN
Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN
Fannie-Mae-DOWN,Fed-Home-Loan-DOWN,
3 MBNA-Corp-DOWN,Morgan-Stanley-DOWN Financial-DOWN
Baker-Hughes-UP,Dresser-Inds-UP,Halliburton-HLD-UP,
4 Louisiana-Land-UP,Phillips-Petro-UP,Unocal-UP,
Schlumberger-UP
Oil-UP
Clustering precipitation
in Australia
4
What is not Cluster Analysis?

 Supervised classification
– Have class label information

 Simple segmentation
– Dividing students into different registration groups
alphabetically, by last name

 Results of a query
– Groupings are a result of an external specification

 Graph partitioning
– If the purpose is to divide a graph into some sub-graphs

5
Notion of a Cluster can be Ambiguous

How many clusters? Six Clusters

Two Clusters Four Clusters

6
Types of Clusterings (1/2)

 A clustering is a set of clusters

 Important distinction between hierarchical and

partitional sets of clusters
– Partitional Clustering
 A division data objects into non-overlapping
subsets (clusters) such that each data object is in
exactly one subset
– Hierarchical clustering
 A set of nested clusters organized as a
hierarchical tree

7
Partitional Clustering

Original Points A Partitional Clustering

8
Hierarchical Clustering

p1
p3 p4
p2
p1 p2 p3 p4
Traditional Hierarchical Clustering Traditional Dendrogram

p1
p3 p4
p2
p1 p2 p3 p4
Non-traditional Hierarchical Clustering Non-traditional Dendrogram

9
 A hierarchical clustering can be viewed as a
sequence of partitional clusterings and a
partitional clustering can be obtained by taking
any member of that sequence; i.e., by cutting the
hierarchical tree at a particular level.

10
Types of Clusterings (2/2)

 Exclusive versus non-exclusive

– In non-exclusive clusterings, points may belong to multiple
clusters.
– Can represent multiple classes or ‘border’ points

 Fuzzy versus non-fuzzy

– In fuzzy clustering, a point belongs to every cluster with some
weights between 0 and 1 In real fuzzy sets theory, this point
does not exist.
– Weights must sum to 1
– Probabilistic clustering has similar characteristics

 Partial versus complete

– In some cases, we only want to cluster some of the data
1 A B C D
Degree
0 11
Types of Clusters

 Well-separated clusters

 Prototype or Center-based clusters

 Graph-based or Contiguous clusters

 Density-based clusters

 Shared-Property or Conceptual

 Described by an Objective Function

12
Types of Clusters: Well-Separated

 Well-Separated Clusters:
– A cluster is a set of points such that any point in a cluster is
closer (or more similar) to every other point in the cluster than
to any point not in the cluster.
– Sometimes a threshold is used to specify that all the objects in
a cluster must be sufficiently close to one another.

3 well-separated clusters

13
Types of Clusters: Center-Based (Prototype)

 Center-based
– A cluster is a set of objects such that an object in a cluster is
closer (more similar) to the “center” of a cluster, than to the
center of any other cluster Continuous data

– The center of a cluster is often a centroid, the average of all

the points in the cluster, or a medoid, the most “representative”
point of a cluster

4 center-based clusters

14
Types of Clusters: Contiguity-Based (Graph)

 Contiguous Cluster (Nearest neighbor or

Transitive)
– A cluster is a set of points such that a point in a cluster is
closer (or more similar) to one or more other points in the
cluster than to any point not in the cluster.

8 contiguous clusters

15
Types of Clusters: Density-Based

 Density-based
– A cluster is a dense region of points, which is separated by
low-density regions, from other regions of high density.
– Used when the clusters are irregular or intertwined, and when
noise and outliers are present.

6 density-based clusters

16
Types of Clusters: Conceptual Clusters
(Shared Property)

 Shared Property or Conceptual Clusters

– Finds clusters that share some common property or represent
a particular concept.
.

2 Overlapping Circles

17
Types of Clusters: Objective Function

 Clusters Defined by an Objective Function

– Finds clusters that minimize or maximize an objective
function.
– Enumerate all possible ways of dividing the points into
clusters and evaluate the `goodness' of each potential set of
clusters by using the given objective function. (NP Hard)
– Can have global or local objectives.
 Hierarchical clustering algorithms typically have local
objectives
 Partitional algorithms typically have global objectives
– A variation of the global objective function approach is to fit
the data to a parameterized model.
 Parameters for the model are determined from the data.
18
Types of Clusters: Objective Function …

 Map the clustering problem to a different domain

and solve a related problem in that domain
– Proximity matrix defines a weighted graph, where the
nodes are the points being clustered, and the
weighted edges represent the proximities between
points

– Clustering is equivalent to breaking the graph into

connected components, one for each cluster.

– Want to maximize the edge weight between clusters

and minimize the edge weight within clusters
19
Clustering Algorithms

 Self-organizing maps (SOM) neural network

– This is a clustering and data visualization technique
based on a neural network viewpoint.
 K-means and its variants
– This is a prototype-based, partitional clustering
technique that attempts to find a user-specified
number of clusters (K), which are represented by their
centroids.
 Agglomerative hierarchical clustering
– Produce a hierarchical clustering by starting with each
point as a singleton cluster and then repeatedly
merging the two closest clusters until a single, all-
encompassing cluster remains.
20
Self-organizing maps (SOM)
Neural Network

21
SOM Neural Network

 Kohonen proposed SOM neural network in 1980.

 A kind of unsupervised neural network.
 The concept is based on characteristic of “birds
of a feather flock together.”
 It can be used as the pre-processing mechanism
for supervised neural network.

Obtain the class label

for every record
Training SOM neural BP neural
records
without network network
class
labels

22
Network Structure

5X5 topology

Output Layer
Node (j, k)

Input Layer Node i

23
Network Structure

 Input layer: Input vector of training samples. Use

linear transfer function f (x) = x.
 Output layer: Clustering of samples with network
topology and neighborhood concepts.

24
Basic Concept (1/8)

 Network topology: The corresponding position

is meaningful for the network output units in
SOM. It is different from other neural networks
whose outputs do not have meaning. Most
used output layer topology is 2-dimensional
structure. But, other shapes or number of
layers are allowed.

25
Basic Concept (2/8)

 Topology coordinate: Represent the

coordinate of the output topology for the
output nodes. The number of dimensions can
be more than 2.
Yt

a
Topology coordinate
b a : (1,3)
Xt b : (3,1)

26
Basic Concept (3/8)

 Neighborhood area: The area using an output

node as the center, or winner. The nodes within
this area will interact with each other. The size of
area is decreased during training.
nth iteration
(n+1)th iteration

(n+2)th iteration

27
Basic Concept (4/8)

 Neighborhood center: The center controls the

neighborhood area. Basically, it is the coordinate
value for an output node which is the center, or
winner.

 Neighborhood radius: The parameter controls the

size of the neighborhood area.

28
Basic Concept (5/8)

 Neighborhood distance: The distance between

the output node and the neighborhood center in
output topology.

rj  (X j  Cx ) 2  (Yj  Cy ) 2
where
X j , Y j = The coordinate values for jth output node.
C x , C y = The coordinate values of the neighborhood center.

 Neighborhood coefficient (R): The parameter

controls the interaction of output nodes inside the
neighborhood area. 29
Basic Concept (6/8)

 Neighborhood function: Control neighborhood

coefficient, neighborhood radius, and
neighborhood distance.
R _ factor j = f (r j , R)

 Four most used neighborhood functions:

30
Basic Concept (7/8)

R
R

Smokestack-hat function Mexican-hat function

R R

Chef-hat function Leaf-hat function

31
Basic Concept (8/8)

 Neighborhood shrinking: The neighborhood

radius decreases gradually.

Rn=R_rate x Rn-1

where R_rate=Neighborhood radius shrinking

coefficient (<1.0)

32
Algorithm-Training (1/5)

(1) Set up network parameters.

(2) Set up connecting weight matrix W randomly.
Set up topology coordinate values for the output
layer nodes
X _ Node [ j ][ k ]=j
Y _ Node [ j ][ k ]=k
(3) Input a training Output Layer

sample’s input vector

X. Input Layer

33
Algorithm Training (2/5)

(4) Select the winning output node Node[ j* ][ k* ]

(a) net[ j ][ k ]=Σ (X[ i ] - W[ i ][ j ][ k ])2
where W[ i ][ j ][ k ] is the connecting
weight between ith input node and jkth
output node.
(b) Find the winning node
net[ j* ][ k* ]
=min{ net[ j ][ k ] }
j,k
Output Layer

Input Layer
34
Algorithm-Training (3/5)

(5) Calculate the output vector Y for each output

layer.

if j=j* ,k=k* then

Y[ j ][ k ] = 1
else Output Layer

Y[ j ][ k ] = 0
Input Layer

35
Algorithm-Training (4/5)

(6) Calculate the updating amount ΔW for each

connecting weight.

ΔW[ i ][ j ][ k ]=η‧(X[ i ]-W[i][ j ][ k ])‧neighborhood[j][k]

where
R = Neighborhood radius
neighborhood[ j ][ k ]=exp(-r[ j ][ k ] / R)
r[ j ][ k ]=[(X_Node[ j ][ k ] - X_Node[ j* ][ k* ])2
+(Y_Node[ j ][ k ] - Y_Node[ j* ][ k* ])2]1/2
36
Algorithm-Training (5/5)

(7) Update connecting weight matrix W

W[ i ][ j ][ k ]=W[ i ][ j ][ k ] + ΔW[ i ][ j ][ k ]

(8) (a) Repeat Steps 3-7 until all training

samples have been input.
(b) Shrink learning rate and neighborhood
radius
ηn = η_rate ‧ηn-1
Rn = R_rate‧Rn-1
(9) Repeat Steps 3-8 until the termination
criterion is satisfied.
37
Algorithm-Recall

(1) Set up network parameters.

(2) Input connecting weight matrix W and set up
topology coordinate for the output nodes.
(3) Input the input vector X for a test sample
(4) Find the wining node with the smallest net
value.
(5) Calculate output vector Y for output layer.

38
An Example of Output Topology

Frequency

Output- Y axis

Output- X axis

39
Error Measure

 Error is measured by using “total distance.”

Total distance=  ( min d jp,k )

j ,k
p

where d jp,k   i i , j ,k
(
i
X p
 W ) 2

P: Training sample
 The average of total
distances can be used as
the termination criterion. Output Layer

40
Input Layer
Homework 3 (1/2)

 Write a computer program for SOM neural

network and use IRIS data set to verify your
program with following requirements:
– Test different network topologies, like 5X5 and10X10,
and find the best network structure.
– Test different training rates.
– Use program to draw the learning curve as shown in
appendix 1.
– Use Excel or Matlab to draw output topology as
shown in appendix 2.
– Compare your result with that of Weka.
– Present your work on November 29, 2018 in class.
41
Homework 3 (2/2)

 Appendix 1 250
200 385N

Error
150

誤 差
585N
100
785N
50
0
1 10 20 30 40 50 60 70 80 90 100
學 習 循 環
Training iteration

 Appendix 2

Frequency Topology k

Topology j
42
K-Means

43
K-means Clustering

 A kind of partitional clustering approach

 Each cluster is associated with a centroid (center point)
 Each point is assigned to the cluster with the closest
centroid
 Number of clusters, K, must be specified
 The basic algorithm is very simple

The condition can

be replaced by a
weaker one.

44
K-Means Example

 Coordinates of the samples

Sample X coordinate Y coordinate

X1 -1 1
X2 3 8
X3 0 0
X4 4 9
X5 5 0
X6 3 6
X7 1 2
X8 6 2
45
K-Means Example

 Training samples

Training samples

10
x4
9
x2
8
7
6 x6
5
4
3
x7 x8
2

x1 1
x3 x5
0
-2 -1 0 1 2 3 4 5 6 7

46
K-Means Example

 Step 1: Determine the number of clusters k

＝＞From the coordinate figure, it is very
apparent that there are three clusters. Thus, set
up k ＝3.
 Step 2: Generate K cluster centers randomly.
＝＞ Cluster center X coordinate Y coordinate
Z1 -1 0
Z2 2 8
Z3 6 0

47
K-Means Example

 Thus, initial samples and the corresponding

cluster centers are as follows:

Samples and clustering centers

10 X4
9 Z2 X2
8
7
X6
6
5
4
X7 X8
3
X1
2
X3 X5 Z3
Z1 1
0
-2 -1 0 1 2 3 4 5 6 7

48
K-Means Example

 Step 3: Calculate the Euclidean distance

d(x i , z j )  x i  z j
between data sample xi and clustering
center zj＝＞ Distance between x  1,1 and z  1,0
1 1

  1   12  1  02  1
Distance between x1  1,1 and z2 2,8
  1  2  1  8  7.616
2 2

Distance between x1  1,1 and z3 6,0

  1  6  1  0  7.017
2 2

49
K-Means Example

Z1 Z2 Z3
X1 1.000 7.616 7.071
X2 8.944 1.000 8.544
X3 1.000 8.246 6.000
X4 10.296 2.236 9.220
X5 6.000 8.544 1.000
X6 7.211 2.236 6.708
X7 2.828 6.083 5.385
X8 7.280 7.211 2.000

50
K-Means Example

 Step 4: Assign every data point xi to the

cluster that the distance between cluster
center and the data point is the smallest.
Z1 Z2 Z3 Assigned
cluster
X1 1.000 7.616 7.071 Z1
X2 8.944 1.000 8.544 Z2
X3 1.000 8.246 6.000 Z1
X4 10.296 2.236 9.220 Z2
X5 6.000 8.544 1.000 Z3
X6 7.211 2.236 6.708 Z2
X7 2.828 6.083 5.385 Z1
X8 7.280 7.211 2.000 Z3 51
K-Means Example

 Step 5: Recalculate new cluster centers.

＝＞ z new  1  x
j i
nj
n j : The number of data points belonging to jth cluster.

Z1 X Y
  1,1  0,0  1,2 
new 1
cluster coordinate coordinate z1
3
X1 -1 1
 0,1
X3 0 0
X7 1 2

52
K-Means Example

Z2 X Y
cluster coordinate coordinate
 3,8  4,9  3,6
new 1
X2 3 8 z 2
3
 3.333,7.667
X4 4 9
X6 3 6

 5,0  6,2
Z3 X Y new 1
cluster coordinate coordinate z 3
2
 5.5,1
X5 5 0
X8 6 2

53
K-Means Example

 The new cluster centers

Cluster center1 X coordinate Y coordinate

Z1new 0 1

Z2new 3.333 7.667

Z3new 5.5 1

54
K-Means Example

 The coordinate figure for data points and

new cluster centers after one iteration
Samples and cluster centers
10
9 X4
8 X2
7
Z2NEW
6
X6
5
4
3
X7 X8
2
X1 1
Z3NEW
Z1 NEW
X5
0
-2 -1
X30 1 2 3 4 5 6 7

55
K-Means Example

 Satisfy the pre-specified termination criteria

or not（Assume that the maximal number of
iterations is 50)
＝＞If not satisfied, go back to Step 3 until the
termination criteria is met.

56
K-Means Example

 After 50 iterations, we have the final cluster

centers as follows.

Cluster center50 X coordinate Y coordinate

Z1new 0 1

Z2new 3.333 7.667

Z3new 5.5 1

57
K-Means Example

 The coordinate figure of samples and new

cluster centers after 50 iterations.：

Samples and cluster centers

10 X4
9 X2
Z2NEW
8
7 X6
6
5
4 X7 X8
3
X1 Z3NEW
2 Z1 NEW
X5
X1 3
0
-2 -1 0 1 2 3 4 5 6 7

58
K-means Clustering – Details

 Initial centroids are often chosen randomly.

– Clusters produced vary from one run to
another.
 The centroid is (typically) the mean of the
points in the cluster.
 ‘Closeness’ is measured by Euclidean
distance, cosine similarity, correlation, etc.
 Most of the convergence happens in the first
few iterations.
– Often the stopping condition is changed to
‘Until relatively few points change clusters’
59
Assigning Points to the Closest Centroid

 Euclidean distance (L2) is often used for data

points in Euclidean space, while cosine similarity
is more appropriate for documents.
 Manhattan (L1) distance can be used for
Euclidean data, while Jaccard measure is often
employed for documents.

60
Centroids and Objective Functions

 The goal of the clustering is typically expressed

by an objective function that depends on the
proximities of the points to one another or to the
cluster centroids; e.g., minimize the squared
distance of each point to its closest centroid.
 Once we have specified a proximity measure and
an objective function, the centroid that we should
choose can often be determined mathematically.

61
Assigning Points to the Closest Centroid

Proximity Function Centroid Objective Function

Manhattan (L1) Median Minimize sum of the L1 distance
of an object to its cluster centroid

Squared Euclidean Mean Minimize sum of the squared L2

(L22) distance of an object to its cluster
centroid
Cosine Mean Maximize sum of the cosine
similarity of an object to its cluster
centroid
Bregman Mean Minimize sum of the Bregman
divergence divergence of an object to its
cluster centroid

62
Evaluating K-means Clusters

 Most common measure is Sum of Squared Error (SSE)

– For each point, the error is the distance to the nearest cluster
– To get SSE, we square these errors and sum them.
K
SSE    dist 2 ( mi , x )
i 1 xCi

– x is a data point in cluster Ci and mi is the representative point for

cluster Ci
 can show that mi corresponds to the center (mean) of the cluster
– Given two clusters, we can choose the one with the smallest
error
– One easy way to reduce SSE is to increase K, the number of
clusters
 A good clustering with smaller K can have a lower SSE than a poor
clustering with higher K

63
Two different K-means Clusterings
3

2.5

2
Original Points
1.5

y
1

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Optimal Clustering Sub-optimal Clustering

64
Importance of Choosing Initial Centroids

Iteration 6
1
2
3
4
5
3

2.5

1.5
y

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

65
 Importance of Choosing Initial Centroids
Iteration 1 Iteration 2 Iteration 3
3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

Iteration 4 Iteration 5 Iteration 6

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

66
Importance of Choosing Initial Centroids …

Iteration 5
1
2
3
4
3

2.5

1.5
y

0.5

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x

67
 Importance of Choosing Initial Centroids …

Iteration 1 Iteration 2
3 3

2.5 2.5

2 2

1.5 1.5
y

y
1 1

0.5 0.5

0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2

x x

Iteration 3 Iteration 4 Iteration 5

3 3 3

2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

y

y
1 1 1

0.5 0.5 0.5

0 0 0

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2
x x x

68
Problems with Selecting Initial Points

 If there are K ‘real’ clusters then the chance of

selecting one centroid from each cluster is
small.
– Chance is relatively small when K is large
– Sometimes the initial centroids will readjust
themselves in ‘right’ way, and sometimes
they don’t
– Consider an example of five pairs of
clusters (see next page)

69
10 Clusters Example
Iteration 1 Iteration 2
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x
Iteration 3 Iteration 4
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x

Starting with two initial centroids in one cluster of each pair of clusters
70
10 Clusters Example
Iteration 4
1
2
3
8

2
y

-2

-4

-6

0 5 10 15 20
x
Starting with two initial centroids in one cluster of each pair of clusters
71
10 Clusters Example
Iteration 1 Iteration 2
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x
Iteration 3 x
Iteration 4
8 8

6 6

4 4

2 2
y

y
0 0

-2 -2

-4 -4

-6 -6

0 5 10 15 20 0 5 10 15 20
x x

Starting with some pairs of clusters having three initial centroids, while other have only one.

72
10 Clusters Example
Iteration 4
1
2
3
8

2
y

-2

-4

-6

0 5 10 15 20
x
Starting with some pairs of clusters having three initial centroids, while other have only one.

73
Solutions to Initial Centroids Problem

 Multiple runs
– Helps, but probability is not on your side
 Take a sample of points and cluster them using a
hierarchical clustering tech to determine initial
centroids
 Select more than k initial centroids and then
select among these initial centroids
– Select most widely separated
– Limitations
The sample is relatively small
K is relatively small compared to the sample size

74
 Select the first point at random and then select
the next point which is farthest from any of the
initial centroids already selected. (well separated)
– But, outliers may be selected.
– Expensive to compute the farthest point from the
current set of initial centroids.
 Post-processing (in page 78)
 Bisecting K-means
– Not as susceptible to initialization issues

75
Handling Empty Clusters

 Basic K-means algorithm can yield empty

clusters

 Several strategies to handle empty clusters

– Choose the point that is farthest away from any
current centroid as the new centroid.
– Choose the replacement centroid from the cluster that
has the highest SSE (divide the cluster with highest
SSE into two clusters).
– If there are several empty clusters, the above
procedure can be repeated several times.

76
Outliers

 When outliers are present, the resulting cluster

centroids may not be as representative as they
would be and thus, the SSE will be higher as
well.
 Thus, it is often useful to discover outliers and
eliminate them beforehand.

77
Reducing the SSE with Pre-processing
and Post-processing
 Pre-processing
– Normalize the data
– Eliminate outliers

 Post-processing
– Eliminate small clusters that may represent outliers
– Split ‘loose’ clusters, i.e., clusters with relatively high
SSE
– Merge clusters that are ‘close’ and that have relatively
low SSE

78
 Two strategies that decrease the total SSE
– Split a cluster: The cluster with the largest SSE is
usually chosen, but we could also split the cluster with
the largest standard deviation for one particular
attribute.
– Introduce a new cluster centroid: Often the point that
is farthest from any cluster is chosen. Random
selection is another approach.
 Two strategies that decrease the number of
clusters
– Disperse a cluster: Remove the centroid that
corresponds to the cluster and reassigning the points
to other clusters.
– Merge two clusters: The clusters with the closest
79
centroids are typically chosen.
Updating Centers Incrementally

 In the basic K-means algorithm, centroids are

updated after all points are assigned to a centroid

 An alternative is to update the centroids after

each assignment (incremental approach)
– Each assignment updates zero or two centroids
– Never get an empty cluster
– More expensive
– Introduces an order dependency

80
Bisecting K-means

 Bisecting K-means algorithm

– Variant of K-means that can produce a partitional or a
hierarchical clustering
– There are many ways to choose which cluster to split:
 The largest cluster at each step
 The one with the largest SSE
 Use a criterion based on both size and SSE
– We often refine the resulting clusters by using their centorids
as the initial centroids for the basic K-means algorithm.

81
Limitations of K-means

 K-means has problems when clusters are of

differing
– Sizes
– Densities
– Non-globular shapes

 K-means has problems when the data contains

outliers.

82
Limitations of K-means: Differing Sizes

Original Points K-means (3 Clusters)

83
Limitations of K-means: Differing Density

Original Points K-means (3 Clusters)

84
Limitations of K-means: Non-globular Shapes

Original Points K-means (2 Clusters)

85
Overcoming K-means Limitations

Original Points K-means Clusters

One solution is to use many clusters.

Find parts of clusters, but need to put together.
86
Overcoming K-means Limitations

Original Points K-means Clusters

87
Overcoming K-means Limitations

Original Points K-means Clusters

88
Hierarchical Clustering

89
Hierarchical Clustering

 Produces a set of nested clusters organized as a

hierarchical tree
 Can be visualized as a dendrogram
– A tree like diagram that records the sequences of
merges or splits

6 5
0.2
4
3 4
0.15 2
5
2
0.1

1
0.05
3 1

0
1 3 2 5 4 6

90
Strengths of Hierarchical Clustering

 Do not have to assume any particular number of

clusters
– Any desired number of clusters can be obtained by
‘cutting’ the dendogram at the proper level

 They may correspond to meaningful taxonomies

– Example in biological sciences (e.g., animal kingdom,
phylogeny reconstruction, …)

91
Hierarchical Clustering

 Two main types of hierarchical clustering

– Agglomerative:
 Start with the points as individual clusters
 At each step, merge the closest pair of clusters until only one cluster
(or k clusters) left

– Divisive:
 Start with one, all-inclusive cluster
 At each step, split a cluster until each cluster contains a point (or
there are k clusters)

 Traditional hierarchical algorithms use a similarity or

distance matrix
– Merge or split one cluster at a time
92
Agglomerative Clustering Algorithm

 More popular hierarchical clustering technique

 Basic algorithm is straightforward
1. Compute the proximity matrix
2. Let each data point be a cluster
3. Repeat
4. Merge the two closest clusters
5. Update the proximity matrix
6. Until only a single cluster remains

 Key operation is the computation of the proximity of

two clusters
– Different approaches to defining the distance between
clusters distinguish the different algorithms

93
Starting Situation

 Start with clusters of individual points and a

proximity matrix p1 p2 p3 p4 p5 ...
p1

p2
p3

p4
p5
.
.
. Proximity Matrix

...
p1 p2 p3 p4 p9 p10 p11 p12

94
Intermediate Situation

 After some merging steps, we have some clusters

C1 C2 C3 C4 C5
C1

C2
C3
C3
C4
C4
C5

Proximity Matrix
C1

C2 C5

...
p1 p2 p3 p4 p9 p10 p11 p12

95
Intermediate Situation

 We want to merge the two closest clusters (C2 and C5) and
update the proximity matrix. C1 C2 C3 C4 C5
C1

C2
C3
C3
C4
C4
C5

Proximity Matrix
C1

C2 C5

...
p1 p2 p3 p4 p9 p10 p11 p12

96
After Merging

 The question is “How do we update the proximity matrix?”

C2
U
C1 C5 C3 C4

C1 ?

C2 U C5 ? ? ? ?
C3
C3 ?
C4
C4 ?

Proximity Matrix
C1

C2 U C5

...
p1 p2 p3 p4 p9 p10 p11 p12
97
How to Define Inter-Cluster Similarity

p1 p2 p3 p4 p5 ...
p1
Similarity?
p2

p3

p4

p5
 MIN
.
 MAX
.
 Group Average .
Proximity Matrix
 Distance Between Centroids
 Other methods driven by an objective
function
– Ward’s Method uses squared error

98
How to Define Inter-Cluster Similarity

p1 p2 p3 p4 p5 ...
p1

p2

p3

p4
The proximity between the closest two
points that are in different clusters p5
 MIN
.
 MAX
.
 Group Average .
Proximity Matrix
 Distance Between Centroids
 Other methods driven by an objective
function
– Ward’s Method uses squared error

99
How to Define Inter-Cluster Similarity

p1 p2 p3 p4 p5 ...
p1

p2

p3

p4

p5
Proximity between the farthest two
 MIN points in different clusters
.
 MAX
.
 Group Average .
Proximity Matrix
 Distance Between Centroids
 Other methods driven by an objective
function
– Ward’s Method uses squared error

100
How to Define Inter-Cluster Similarity

p1 p2 p3 p4 p5 ...
p1

p2

p3

p4
The average pairwise
p5
 MIN proximities of all pairs of
points from different
.
 MAX clusters
.
 Group Average .
Proximity Matrix
 Distance Between Centroids
 Other methods driven by an objective
function
– Ward’s Method uses squared error

101
How to Define Inter-Cluster Similarity

p1 p2 p3 p4 p5 ...
p1

  p2

p3

p4

 MIN p5

 MAX .
.
 Group Average
.
 Distance Between Centroids Proximity Matrix
 Other methods driven by an objective function
– Ward’s Method uses squared error
 Like distance between centroids, but it measures
the proximity between two clusters in term of the
increase in the SSE that results from merging the
102
two clusters.
103
Cluster Similarity: MIN or Single Link

 Similarity of two clusters is based on the two

most similar (closest) points in the different
clusters
– Determined by one pair of points, i.e., by one link in
the proximity graph.

104
Hierarchical Clustering: MIN

Nested Clusters Dendrogram

105
Strength of MIN

Original Points Two Clusters

• Can handle non-elliptical shapes

106
Limitations of MIN

Original Points Two Clusters

• Sensitive to noise and outliers

107
Cluster Similarity: MAX or Complete Linkage

 Similarity of two clusters is based on the two least

similar (most distant) points in the different
clusters
– Determined by all pairs of points in the two clusters

108
Hierarchical Clustering: MAX

Nested Clusters Dendrogram

109
Strength of MAX

Original Points Two Clusters

• Less susceptible to noise and outliers

110
Limitations of MAX

Original Points Two Clusters

•Tends to break large clusters

111
Cluster Similarity: Group Average

 Proximity of two clusters is the average of pairwise proximity

between points in the two clusters.

proximity(p , p )
piClusteri
i j

p jClusterj
proximity(Clusteri , Clusterj ) 
|Clusteri ||Clusterj |

112
Hierarchical Clustering: Group Average

Nested Clusters Dendrogram

113
Hierarchical Clustering: Group Average

 Compromise between Single and Complete

Link

 Strengths
– Less susceptible to noise and outliers

114
Cluster Similarity: Ward’s Method

 Similarity of two clusters is based on the increase

in squared error when two clusters are merged
– Similar to group average if distance between points is
distance squared

 Less susceptible to noise and outliers

 Hierarchical analogue of K-means

– Can be used to initialize K-means

115
116
Hierarchical Clustering: Time and Space requirements

 O(N2) space since it uses the proximity matrix.

– N is the number of points.

 O(N3) time in many cases

– There are N steps and at each step the size, N2,
proximity matrix must be updated and searcheed

117
Hierarchical Clustering: Problems and Limitations

 Once a decision is made to combine two clusters,

it cannot be undone

 No objective function is directly minimized

 Different schemes have problems with one or

more of the following:
– Sensitivity to noise and outliers
– Difficulty handling different sized clusters and convex
shapes
– Breaking large clusters

118
Cluster Validity

119
Cluster Validity

 For supervised classification we have a variety of

measures to evaluate how good our model is
– Accuracy, precision, recall
 For cluster analysis, the analogous question is how to
evaluate the “goodness” of the resulting clusters?
 Each clustering algorithm defines its own type of cluster-
It may seem that each situation might require a different
evaluation measure.
 Then why do we want to evaluate them?
– To avoid finding patterns in noise
– To compare clustering algorithms
– To compare two sets of clusters
– To compare two clusters

120
 Clusters found in Random Data
1 1

0.9 0.9

0.8 0.8

0.7 0.7

Random 0.6 0.6 DBSCAN

Points 0.5 0.5
y

y
0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
x x
1 1

0.9 0.9

0.8 0.8
K-means Complete
0.7 0.7
Link
0.6 0.6

0.5 0.5
y

0.4 0.4

0.3 0.3

0.2 0.2

0.1 0.1

0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
x x
121
Different Aspects of Cluster Validation

1. Determining the clustering tendency of a set of data, i.e.,

distinguishing whether non-random structure actually exists in the
data.
2. Comparing the results of a cluster analysis to externally known
results, e.g., to externally given class labels.
3. Evaluating how well the results of a cluster analysis fit the data
without reference to external information.
- Use only the data
4. Comparing the results of two different sets of cluster analyses to
determine which is better.
5. Determining the ‘correct’ number of clusters.

For 2, 3, and 4, we can further distinguish whether we want to

evaluate the entire clustering or just individual clusters.

122
Some Challenges for Numerical Measure

 A measure of cluster validity may be quite limited

in the scope of its applicability
– Most work on measures of clustering tendency has
been done for two- or three-dimensional spatial data.
 A framework to interpret any measure is needed.
– Whether it is good or poor.
 If a measure is too complicated to apply or to
understand, then few will use it.

123
Measures of Cluster Validity

 Numerical measures that are applied to judge

various aspects of cluster validity, are classified into
the following three types.
– External Index (Supervised): Used to measure the
extent to which cluster labels match externally supplied
class labels.
 Entropy
– Internal Index (Unsupervised): Used to measure the
goodness of a clustering structure without respect to
external information.
 Sum of Squared Error (SSE) (see next two pages)
– Relative Index: Used to compare two different
clusterings or clusters.
 Often an external or internal index is used for this function, e.g., SSE
or entropy 124
Internal Measures: Cohesion and Separation

 Cluster Cohesion: Measures how closely related

are objects in a cluster
– Example: SSE
 Cluster Separation: Measure how distinct or well-
separated a cluster is from other clusters
 Example: Squared Error
– Cohesion is measured by the within cluster sum of squares (SSE)
WSS    ( x  mi )2
i xCi
– Separation is measured by the between cluster sum of squares

BSS   Ci (m  mi )2
i
– Where |Ci| is the size of cluster i
125
Internal Measures: Cohesion and Separation

 Example: SSE
– BSS + WSS = constant
m
  
1 m1 2 3 4 m2 5

K=1 cluster: WSS (1  3)2  (2  3)2  (4  3)2  (5  3)2  10

BSS 4  (3  3)2  0
Total  10  0  10

K=2 clusters: WSS (1  1.5)2  (2  1.5)2  (4  4.5)2  (5  4.5)2  1

BSS 2  (3  1.5)2  2  (4.5  3)2  9
Total  1  9  10
126
Internal Measures: SSE
 Clusters in more complicated figures aren’t well separated
 Internal Index: Used to measure the goodness of a clustering
structure without respect to external information
– SSE
 SSE is good for comparing two clusterings or two clusters
(average SSE).
 Can also be used to estimate the number of clusters
10

6 9

8
4
7

2 6

SSE
5
0
4
-2 3

2
-4
1
-6 0
2 5 10 15 20 25 30
5 10 15
K
127
Internal Measures: SSE

 SSE curve for a more complicated data set

1
2 6

3
4

SSE of clusters found using K-means

128
Unsupervised Cluster Evaluation Using
Cohesion and Separation

 Most are based on cohesion or separation.

 Two different kinds of cluster validity measure:
– Prototype-based clustering technique
– Graph-based clustering technique
 General form for measuring cluster validity
k
overall _ validity   wi validity(Ci )
i 1

129
Graph-Based View of Cohesion and
Separation
 The cohesion of a cluster can be defined as the
sum of the weights of the links in the proximity
graph that connect points within the cluster.
 The separation between two clusters can be
measured by the sum of the weights of the links
from points in one cluster to points in the other
cluster.

130
cohesion separation
cohesion(Ci )   proximity( x, y)
xCi
yCi

separation(Ci , C j )   proximity( x, y)
xCi
yC j

131
Prototype-Based View of Cohesion and
Separation
 The cohesion of a cluster can be defined as the sum of the
proximities with respect to the prototype (centroid or
medoid).
 The separation between two clusters can be measured by
the proximity of the two cluster prototypes.

cohesion(Ci )   proximity( x, ci )
xCi

separation(Ci , C j )  proximity (ci , c j )

132
 Both the cluster cohesion measure and
separation measure can be combined together
using the following equation:

k
overall _ validity   wi validity(Ci )
i 1

 This can refer to Table 8.6.

 A popular method of silhouette coefficients
combines both cohesion and separation (see next
page)
133
The Silhouette Coefficient

 Silhouette Coefficient combine ideas of both cohesion and separation,

but for individual points, as well as clusters and clusterings
 For an individual point, i
– Calculate a = average distance of i to the points in its cluster
– Calculate b = min (average distance of i to points in another cluster)
– The silhouette coefficient for a point is then given by
s = (b-a) / max(a,b) (*a<b is better)

– Typically between -1 and 1.

– A negative value is undesirable because this corresponds to a case in
which a is greater than b. b
– The closer to 1 the better. a

 Can calculate the Average Silhouette width for all the points in a
cluster.

134
Determining the Correct Number of
Clusters
 It is necessary to use some measures to
determine the number of clusters

135
  Wilk’s Lambdan value is the other choice for
determining the number of clusters.
SSwithin
Wilk ' s Lambda 
SStotal
ART2 ART2
Wilk’s Wilk’s
parameters Number of parameter Number of
Lambda Difference Lambda Difference
vigilance clusters vigilance clusters
Value Value
value value
0.96 80 0.49511 0.0486 0.9491 59 0.5899 0.0160

0.954 70 0.54374 0.0445

0.948 57 0.6059 0.0323
0.9494 60 0.58824 0.0655
0.9461 55 0.63821 0.0012
0.9422 50 0.65376 0.0383
0.9445 53 0.63936 0.0044
0.934 40 0.69201 0.0493

0.943 51 0.6438 136

0.91851 30 0.74127 136
Clustering Tendency

 Two ways for measuring clustering tendency\

– Using clustering algorithms- Use multiple algorithms
and again evaluate the quality of the resulting
clusters. If the clusters are uniformly poor, then this
may indeed indicate that there are no clusters in the
data.
– Using statistical tests- like Hopkins statistics

137
Supervised Measures of Cluster Validity

 Classification-orientated measure
 Similarity-orientated measure
(See next page)

138
Classification-Oriented Measure of
Cluster Validity
 Measure the degree to which predicted class
labels correspond to actual class labels.
– Entropy
– Purity
– Precision
– Recall
– F-measure (Combination of Precision and Recall)

139
External Measures of Cluster Validity: Entropy and Purity

140
Similarity-Oriented Measure of Cluster
Validity
 Compare two matrices:
– The ideal cluster similarity matrix which has a 1 in the
ijth entry if two objects, i and j, are in the same cluster
and 0, otherwise.
– The ideal class similarity matrix which has a 1 in the
ijth entry if two objects, i and j, belong to the same
class, a 0 otherwise.
Point Point
p1 p2 p3 p4 p5 p1 p2 p3 p4 p5
P1 1 1 1 0 0 P1 1 1 0 0 0
Correlation
P2 1 1 1 0 0 P2 1 1 0 0 0
=0.359
P3 1 1 1 0 0 P3 0 0 1 1 1
P4 0 0 0 1 1 P4 0 0 1 1 1
p5 0 0 0 1 1 p5 0 0 1 1 1
Clustered results 141
Actual results
Final Comment on Cluster Validity

“The validation of clustering structures is the

most difficult and frustrating part of cluster
analysis.
Without a strong effort in this direction, cluster
analysis will remain a black art accessible only to
those true believers who have experience and
great courage.”

Algorithms for Clustering Data, Jain and Dubes

142
THE END!

143