Nonlinear Programming (NLP)

Fundamentals

Dr. Ahmed Sayed AbdelSamea

Giza, Egypt, Spring 2017

aabdelsamea@zewailcity.edu.eg
Objectives
To learn fundamental algorithms and basic
concepts of continuous optimization.
To recognize the applicability of the optimization
algorithms with their requirements and
computational costs.
To discover the wide application of optimization
techniques in engineering areas such as control
systems, estimation and signal processing,
communications and networks, circuit design,
data analysis and modeling, statistics as well as
other areas such as medicine and finance.

2
References
S. Rao, Engineering Optimization: Theory and
Practice, 4th ed., 2009.

J. Nocedal and S.J. Wright, Numerical

Optimization, Springer, 1999.

S. Boyd and L. Vandenberghe, Convex

Optimization, Cambridge University Press, 2004.

3
Fundamentals
Optimization (Programming) is the science that is
concerned with the efficient use (allocation) of limited
resources to meet certain objectives and satisfy predefined
constraints (limitations).
Optimization is the act of obtaining the best result under
given circumstances.
In design, construction, and maintenance of any
engineering system, engineers have to take many
technological and managerial decisions at several stages.
The ultimate goal of all such decisions is either to minimize
the effort (cost) required or to maximize the desired benefit
(profit).
It is also called Mathematical Programming techniques
and are generally studied as a part of operations research. 4
Fundamentals
An amazing variety of practical problems involving
decision making can be cast in the form of a mathematical
optimization problem.
It is widely used in engineering, in
electronic design automation,
automatic control systems, and
network design and operation,
finance, supply chain management, scheduling.
civil, chemical, mechanical, and aerospace engineering.
With the proliferation of computers embedded in products,
we have seen a rapid growth in embedded optimization
where, optimization is used to automatically make real-
time choices, with no (or little) human intervention.
Fundamentals
Example
Device sizing in electronic design, which is the task of
choosing the width and length of each device in an
electronic circuit.
The variables represent the widths and lengths.
The constraints represent a variety of engineering
requirements, such as
limits on the device sizes imposed by the manufacturing process,
timing requirements that ensure that the circuit can operate
reliably at a specified speed, and
a limit on the total area of the circuit.
A common objective in a device sizing problem is the total
power consumed by the circuit. 6
Fundamentals
The general optimization problem can be given as:
min

. . = 0, = {1,2, . . , }
0, = {1,2, . . , }
.
where , and , 1 , 2 , , , 1 , 2 ,
are functions defined on .
The set of feasible solutions is defined by:

= { | = 0, and 0, }
7
Fundamentals
A hypothetical two-dimensional design space

8
Fundamentals
Contours of the objective function

9
Fundamentals
The constrained problem can be described also as:
min

. . () .
where () represents the set of whole constraints
In case of equality constraints:
= 0 0 and 0
In case of inequality constraints of the form
0 0

10
Fundamentals
The optimal value is defined by
= ()| = , ,
The optimal point (solution) is defined by:
, where
= | = , , , , =
Think about Maximization problem ?!

11
Fundamentals
Locally versus globally optimal points.
Feasibility versus optimality conditions.
If is feasible and = , we say the jth
inequality constraint is active at .
If < , we say is inactive at .
All the equality constraints = are active
at all the feasible points.

12
Fundamentals
A function (), of single variable, is said to have a local
(relative) minima at = if ( ) ( + ) for all
sufficiently small positive and negative values of (in
the neighborhood of , i.e., 0).

A function () has a global (absolute) min at if

( ) () for all dom().

13
Classification of Optimization Problems

Linear Programming
If all the functions , 1 , 2 , , , 1 , 2 , are
affine (linear) functions in :
min

s. . = .
Unconstrained Optimization
The indices I and J are empty and
min

14
Classification of Optimization Problems

Convex Optimization
If , 1 , 2 , are convex functions and 1 , 2 , ,
are affine (linear) functions in and dom() is convex.

Quadratic Optimization
If the objective function is a quadratic function and
1 , 2 , , , 1 , 2 , are affine (linear) functions.
the quadratic objective function is given by:
= + +
15
Classification of Optimization Problems

Geometric Programming

min

. . 1, = 1,2, . . ,
> .
where , 1 , 2 , are posynomial functions.

16
Classification of Optimization Problems

Posynomial function is defined as

1 2
= 1 2
=1
0;

Example: 1 , 2 , 3 = 12 21 31.2 + 223 30.5

GP arise in layout and circuits applications where are
physical dimensions and constraints involve areas,
weight and volumes.
17
Classification of Optimization Problems

Discrete (Integer) Programming

o It refers to optimization problems in which the solution is
one of a number of objects in a finite set.
o The mathematical formulation is changed by adding the
following constraint
, = 1,2, ,
where is the set of all integers.
o One technique is to solve the problem with real variables and
then rounding all the components to the nearest integer (No
guarantee of optimal solution !)
o Branch-and-bound method for solving linear programming
problems.
18
Classification of Optimization Problems

Stochastic Programming
o In many economic and financial planning models, which
often depends on the future movement of interest rates and
the future behavior of the economy, the optimization models
can not be fully specified (stochastic representation).
o In some techniques, modelers can predict or estimate the
unknown quantities with some degrees of confidence.
o Stochastic optimization algorithms use these uncertainties to
produce solutions that optimize the expected performance of
the model.

19
Classical Optimization Techniques
Single Variable Optimization
Theorem (Necessary condition)
If a function (), is defined in the interval
and has a local (relative) minima at = , where
and if ( ), exists, then = 0.
Proof
+ ( )
It is given that = lim exists, we need
0
to prove that it is zero!!
Since is a local minima, we have ( ) +
for all values of sufficiently close to zero.

20
Classical Optimization Techniques
Single Variable Optimization (cont.)
Hence,

+ ( )
lim 0 if > 0 0
0

+ ( )
lim 0 if < 0 0
0

which never happens except for = 0.

21
Classical Optimization Techniques
Single Variable Optimization (cont.)
Theorem (Sufficient Condition)
let = = (1) = 0, but () 0
Then ( ) is
a min value of if () > 0 and is even.
a max value of if () < 0 and is even.
Neither max nor min if is odd.

22
Classical Optimization Techniques
Multivariable Optimization
Theorem (Necessary condition)
If (), has a local min (max) at = , and if the first
partial derivatives of () exists at , then = 0.

Proof
+ = + ( , )
The 1st order terms of dominates the higher orders for
small . The sign of LHS depends on the sign of 1st
term (component-wise).

23
Classical Optimization Techniques
Multivariable Optimization
Theorem (Sufficient condition)
A sufficient condition for a stationary point to be a
local min is the second partial derivatives (Hessian
matrix) of (), at is positive definite matrix (PD)
= 2 > 0
Proof
Using 2nd order Taylor expansion:

2
+ = + +

For < < .
24
Unconstrained Optimization Algorithms
Any optimization algorithm generates a sequence
of iterates =
The user with knowledge of the application under
consideration is in a position to choose the starting
(initial) point to be a reasonable estimate of the
solution
The algorithm terminates when either no progress
can be made or when it seems that a solution point
has been approximated with sufficient accuracy

25
Unconstrained Optimization Algorithms
The optimization algorithm uses the information
about the function at to find the new iterate
+ with lower function value
There are two fundamental strategies for moving
from the current point to a new iterate + :

Line Search and Trust Region

26
Line Search Strategy
In Line Search (LS) strategy, the algorithm
chooses a search direction denoted by along
which and starting from the current iterate , the
algorithm searches for a new iterate with lower
function value
+ = +
At the k th iterate, the step length can be found
by solving the following one-dimensional
(univariate) minimization problem:
= +
>
27
Line Search Strategy

28
Unconstrained Optimization Techniques
Unconstrained optimization methods can be
classified into two categories:
Direct Search Methods (non-gradient methods):
Random search, Grid search, Univariate search
Pattern search (Hooke-Jeeves, Powell)
Simplex method (Nelder and Mead)
Descent Methods (gradient methods):
Steepest descent method
Fletcher-Reeves method
Newtons method, Marquardt method
Quasi-Newtons methods (SR1, DFP, BFGS)
29
Unconstrained Optimization Techniques
Descent Methods (derivative-based)
In Line Search (LS) strategy, the algorithm chooses
a search direction , through the current point ,
along which the function decreases
+ < ( )
Search directions which satisfy the above condition
include: descent directions, Newton direction,
Quasi-Newton directions

30
Unconstrained Optimization Techniques
Descent Methods (derivative-based)
Descent Search
Consider the Taylors expansion of the function at the
point with a sufficiently small step length () and search
direction :
+ = + + ( )
The direction is called descent (downhill) direction if the
angle between and has cos < , i.e.
= cos <
This yields that + < for each positive but
sufficiently small values of

31
Unconstrained Optimization Techniques
Descent Methods (derivative-based)
Descent Search
This graph shows the
ascent direction

32
Unconstrained Optimization Techniques
Descent Methods (derivative-based)
Steepest Descent (Cauchy) Method
Among all the descent directions, the steepest direction
gives the most rapid decrease of the function f . This
can be obtained by solving =

which gives = ; ( = , = )
The steepest descent direction is orthogonal to the function
contours
The advantage of steepest search direction, it requires only
the calculation of gradient not Hessian
However, since it is a local property, it is not effective in
most of practical problems.
33
Unconstrained Optimization Techniques
Descent Methods (derivative-based)
Steepest Descent (Cauchy) Method
Convergence (stopping) Criteria
The algorithm stops using one (or combined) of the
following conditions:

f (x k 1 ) f ( x k )
1. function value: 1
f (x k )
2. derivative: f (x k 1 ) 2
x k 1 x k
3. point: 3
xk
34
Unconstrained Optimization Techniques
Descent Methods (derivative-based)
Steepest Descent (Cauchy) Method
Example Starting from (0, 0) Find the minimum of the function:
1 , 2 = 1 2 + 212 + 21 2 + 22
Solution
start with =
Iteration = : evaluate = =
Find by minimizing + = = ,
= + = and = (Not optimal?)
Iteration = : evaluate = =
Find by minimizing + = = . ,
= + = . . and = . . (Not
optimal?) Continue .. 35