BRANCH AND BOUND

TECHNIQUE
(Feature Selection Algorithm)

Group Members:
Unais
K142222
Deepak Kumar Khatri

MOTIVATION
The optimal feature selection (subset selection) has
been very difficult because of its computational
complexity
All the subsets of given cardinality that have to be
evaluated to find the optimal set of features among
large set of measurements
Exhaustive search is impractical even for relatively
small size of problems
Finding 2 features from 10 feature set would generate 45
possible combinations.

This challenge has motivated over the years to

speeding up the search process in the arena of feature

INTRODUCTION
As a solution Branch and Bound algorithm was
developed by Narendra and Fukunaga in 1977
Introduced heuristic measures which can help to
identify parts of the search space that can be left
unexplored without missing the optimal solution
Guaranteed to find the optimal feature subset
without evaluating all possible subsets
B & B is an exponential search method
Assumes that feature selection criterion is monotonic

BRANCH AND BOUND TECHNIQUE

Solution Branch and bound methods solve a

discrete optimization problem by breaking
up its feasible set into successively
smaller subsets, calculating bounds on
the objective function value over each
subset, and using them to discard certain
subsets from further consideration.

GENERAL IDEA OF B&B

ALGORITHM
The procedure ends when each subset has either
produced a feasible solution, or was shown to contain
no better solution than the one already in hand. The
best solution found during the procedure is a global
optimum.
The general idea of B&B is a BFS-like search for the
optimal solution, but not all nodes get expanded (i.e.,
their children generated). Rather, a carefully selected
criterion determines which node to expand and when,
and another criterion tells the algorithm when an
optimal solution has been found.

STEPS OF ALGORITHM

1. Place the start node of zero path length on the queue.

2. Until the queue is empty or a goal node has been found do the
following:
(i) Determine if the s first path in the queue contains a good
node.
(ii) If the first path contains a goal node exit with success.
(iii) If the first path does not contain a good node remove the
path from the queue and form new paths by
extending the
removed paths by on step.
(iv)Compute the cost of the new paths and add them to the
queue.
(v) Sort the paths on the queue with lowest-cost path in front.

ANALYSIS
Start from the full set of features and remove features using
depth first strategy
Monotonicity property should be satisfied to apply the algorithm
Branching is the constructing process of tree
For each tree level, a limited number of sub-trees is generated by
deleting one feature from the set of features from the parent
node
Bounding is the process of finding optimal feature set by
traversing the constructed tree

ANALYSIS

Algorithm

1. Construct an ordered tree by satisfying the

Monotonicity
property
Let xj be the set of features obtained by removing j features
y1 , y2 yj from the set Y of all features
Xj = Y \ {y1 , y2 yj }
The monotonicity condition assumes that, for feature
subsets x1 , x2 xj where,
x1 x2 x3 . xj
The criterion function J fulfills,
J(x1) < J(x2) < J(x3) < < J(xj)

ANALYSIS

Algorithm

2. Traverse the tree from right to left in depth first search

pattern
. If the value of the criterion is less than the threshold (relevant to the
most recent best subset) at a given node,
All its successors will also have a value less than criterion value

3. Pruning
. Anytime the criterion value J(xm) in some internal node is found

to
be lower than the current bound, due to the Monotonicity
condition the
whole sub tree may be cut off and many computations may be omitted

. B&B creates tree with all possible combinations of r element subsets

from the n set, but searches only some of them

ANALYSIS

Tree Properties

Root of the tree represents the set of all features (n)

and leaves represent target subsets (r) of features
For each tree level, a limited number of sub-trees is
generated by deleting one feature from the set of
{ X1,X2,X
All features
features from the parent
node
Removed
feature

3}

X
1
{ X2,X3
}

(n)

X
2

X
3

{ X1,X3
}

{ X1,X2
}

Target subset
(r)

ANALYSIS

Tree Properties

In practice, we have only allowed variables to be

removed in increasing order. This removes
unnecessary repetitions in the calculation. Therefore tree
is not symmetrical
{ X1,X2,X3,
X4 }

X
1
{ X2,X3,X
4}
X
2
{ X3,X4
}

X
2
{ X1,X3,
X4 }

{ X3,X4
}
Repetition

X
1

Not in
increasing
order

ANALYSIS

Tree Properties

Number of leaf nodes in tree = nCr

Number of levels = n r
{ X1,X2,X
3}

Ex:

X
1

{ X2,X3
}

X
2

{ X1,X3
}

X
3

{ X1,X2
}

3 features reduced to 2
features

No of leaf nodes = 3C2

No of levels

=3

=32=1

EXAMPLE

How to reduce 5 features in to 2 features

using B & B Algorithm?
1, 2, 3,
4, 5

?,?

Finding best 2 features from full set of features

EXAMPLE

Branching Step 4

L
0

1,2,3,4,5
1

L1

2,3,4,5
2

L
2

4,5

4
2,4,5

3,5

3,4

3
1,2,4,5

1,3,4,5

3,4,5
3

L
3

3
1,4,5

2,3,5

2,5

2,4

5
2,3

1,3,5

1,5

1,4

1,2,5
5

1,3

1,2

ASSIGNMENT PROBLEM USING

B&B TECHNIQUE

ASSIGNMENT PROBLEN USING B&B

TECHNIQUE

Given n jobs and

n persons, and cost
of person I, doing
job j, find the best
job assignment.

TOTAL LEAVES = 24

We can use BRUTE FORCE to solve this problem but

the number of steps required to solve the problem
will be quite large as the number of jobs and persons
increases.
That is n!
In the given problem, total number of cases form is
4! = 24
(As we have 4 jobs and 4 Persons in our example)

APPLICATIONS OF B&B
ALGORITHM
1. Integer programming
2. Nonlinear programming
3. Travelling salesman problem (TSO) [3][10]
4. Quadratic assignment problem (QAP)
5. Maximum satisfiability problem (MAX-SAT)
6. Nearest neighbor search [11]
7. Cutting stock problem

THANK
YOU!