Introduction to Simulation - Lecture 10

Modified Newton Methods

Jacob White

Thanks to Deepak Ramaswamy, Jaime Peraire, Michal

Rewienski, and Karen Veroy
 Outline

Damped Newton Schemes

Globally Convergent if Jacobian is Nonsingular
Difficulty with Singular Jacobians
Introduce Continuation Schemes
Problem with Source/Load stepping
More General Continuation Scheme
Improving Continuation Efficiency
Better first guess for each continuation step
Arc Length Continuation
SMA-HPC 2003 MIT
 Multidimensional Newton Algorithm
Newton Method
Newton Algorithm for Solving F ( x ) = 0
x = Initial Guess, k = 0
0

Repeat {
( ) ( )
Compute F x k , J F x k

Solve J F ( x )( x x ) = F ( x )
k k +1 k k
for x k +1
k = k +1

} Until (
x k +1 x k , F x k +1 ) small enough

SMA-HPC 2003 MIT

 Multidimensional
Multidimensional Convergence Theorem
Newton Method
Theorem Statement
Main Theorem

If
a) ( )
J F1 x k ( Inverse is bounded )
b) JF ( x) JF ( y) A x y ( Derivative is Lipschitz Cont )

Then Newtons method converges given a sufficiently

close initial guess

SMA-HPC 2003 MIT

 Multidimensional
Multidimensional Convergence Theorem
Newton Method
Implications
If a functions first derivative never goes to zero, and its
second derivative is never too large

Then Newtons method can be used to find the zero

of the function provided you all ready know the
answer.

Need a way to develop Newton methods which

converge regardless of initial guess!

SMA-HPC 2003 MIT

 Non-converging 1-D Picture
Case
f(x)

x1
0
X
x

Limiting the changes in X might improve convergence

SMA-HPC 2003 MIT
 Newton Method Newton Algorithm
with Limiting
Newton Algorithm for Solving F ( x ) = 0
x = Initial Guess, k = 0
0

Repeat {
( ) ( )
Compute F x k , J F x k

Solve J F ( x ) x = F ( x )
k k +1 k
for x k +1
x k +1 = x k + limited ( x ) k +1

k = k +1
} Until (
x k +1 , F x k +1 ) small enough
SMA-HPC 2003 MIT
 Newton Method Damped Newton
with Limiting Scheme

General Damping Scheme

( )
Solve J F x k x k +1 = F x k ( ) for x k +1

x k +1 = x k + k x k +1

Key Idea: Line Search

( )
2
Pick to minimize F x + x
k k k k +1
2

( ) ( ) F (x )
2 T
F x + xk k k +1
F x + xk k k +1 k
+ k x k +1
2

Method Performs a one-dimensional search in

Newton Direction
SMA-HPC 2003 MIT
 Newton Method Damped Newton
with Limiting Convergence Theorem
If
a) J F1 ( x k ) ( Inverse is bounded )
b) JF ( x) JF ( y) A x y ( Derivative is Lipschitz Cont )
Then
There exists a set of k ' s ( 0,1] such that
F ( x k +1 ) = F ( x k + k x k +1 ) < F ( x k ) with <1

Every Step reduces F-- Global Convergence!

SMA-HPC 2003 MIT

 Newton Method Damped Newton
with Limiting Nested Iteration
x 0 = Initial Guess, k = 0
Repeat {
( ) ( )
Compute F x k , J F x k
Solve J ( x ) x = F ( x ) for x
F
k k +1 k k +1

Find ( 0,1] such that F ( x + x )

k k k k +1
is minimized
x k +1 = x k + k x k +1
k = k +1
} Until (
x k +1 , F x k +1 ) small enough

SMA-HPC 2003 MIT

Newton Method Damped Newton
with Limiting Example
v1 10 v2 1
I r Vr = 0
10
1v + + Vd
- Vd I d I s (e Vt
1) = 0
-
Nodal Equations with Numerical Values

f ( v2 ) =
( v2 1) 16
+ 10 (e
( v 0) 2
0.025
1) = 0
10
Newton Method Damped Newton
with Limiting Example cont.

f ( v2 ) =
( v2 1) 16
+ 10 (e
( v 0)
2
0.025
1) = 0
10
 Newton Method Damped Newton
with Limiting Nested Iteration
x 0 = Initial Guess, k = 0
Repeat {
( ) ( )
Compute F x k , J F x k
Solve J ( x ) x = F ( x ) for x
F
k k +1 k k +1

Find ( 0,1] such that F ( x + x )

k k k k +1
is minimized
x k +1 = x k + k x k +1
k = k +1
} Until (
x k +1 , F x k +1 ) small enough

How can one find the damping coefficients?

SMA-HPC 2003 MIT
 Newton Method Damped Newton
with Limiting Theorem Proof

By definition of the Newton Iteration

( ) ( )
1
x k +1
=x -
k k
JF x k
F xk
 

Newton Direction
Multidimensional Mean Value Lemma
A
F ( x ) F ( y ) J F ( y )( x y )
2
x y
2
Combining
2
A
( ) F (x )+ J (x ) ( )
k J x k ( )
F x k J F xk ( ) ( )
1 1
k +1 k k k k
F x F x
F F 2

SMA-HPC 2003 MIT

 Newton Method Damped Newton
with Limiting Theorem Proof-Cont
From the previous slide
2
A k
( ) F (x )+ J (x ) ( )
J x ( ) ( ) ( )
1 1
F x k +1 k k k k
F xk
J F xk F x k
F F 2

Combining terms and moving scalars out of norms

2
A
( ) (1 ) F ( x ) ( ) ( ) ( )
2 1
k +1 k k k
F x J F xk F x k

2
Using the Jacobian Bound and splitting the norm
2 A 2
2
F ( x ) (1 ) F ( x ) + ( )
k +1 k k k
F (x )
k

Yields a quadratic in the damping coefficient

SMA-HPC 2003 MIT
 Newton Method Damped Newton
with Limiting Theorem Proof-Cont-II
Simplifying quadratic from previous slide
2A
( ) ( ) ( ) ( )
2
F x k +1
1 k + k F x k
F x k

2
Two Cases:
2A 1
1)
2
F xk( ) <
2
Pick k = 1 (Standard Newton)

2 A 1
2
1 +
k k

2
F xk ( ) ( ) <
2
2A 1 1
2) F xk > ( )Pick k = 2
2 2 A F (x ) k

2A 1
( ) ( )
2
1 k + k F xk < 1
2 ( )
2 2A F x k
SMA-HPC 2003 MIT
 Newton Method Damped Newton
with Limiting Theorem Proof-Cont-III
Combining the results from the previous slide
( )
F x k +1 k F x k ( ) not good enough, need independent from k
The above result does imply
( )
F x k +1 F x 0 ( ) not yet a convergence theorem

For the case where 2

A 1
F ( xk ) >
2 2
1 1
1 1 0
( )
2 2A F x k ( )
2 2A F x0
Note the proof technique
First Show that the iterates do not increase
Second Use the non-increasing fact to prove convergence
SMA-HPC 2003 MIT
 Newton Method Damped Newton
with Limiting Nested Iteration
x 0 = Initial Guess, k = 0
Repeat {
( ) ( )
Compute F x k , J F x k
Solve J ( x ) x = F ( x ) for x
F
k k +1 k k +1

Find ( 0,1] such that F ( x + x )

k k k k +1
is minimized
x k +1 = x k + k x k +1
k = k +1
} Until (
x k +1 , F x k +1 ) small enough

Many approaches to finding k

SMA-HPC 2003 MIT

 Newton Method Damped Newton
with Limiting
Singular Jacobian Problem
f(x)

x2
1 0 X
1
x xD x

Damped Newton Methods push iterates to local minimums

Finds the points where Jacobian is Singular
SMA-HPC 2003 MIT
 Basic Concepts
Continuation Schemes
Source or Load-Stepping
Newton converges given a close initial guess
Generate a sequence of problems
Make sure previous problem generates guess for next problem
Heat-conducting bar example

1. Start with heat off, T= 0 is a very close initial guess

2. Increase the heat slightly, T=0 is a good initial guess
3. Increase heat again

SMA-HPC 2003 MIT

 Basic Concepts
Continuation Schemes
General Setting
Solve F ( x ( ) , ) = 0 where:
a) F ( x ( 0 ) , 0 ) = 0 is easy to solve Starts the continuation
b) F ( x (1) ,1) = F ( x ) Ends the continuation

c) x ( ) is sufficiently smooth Hard to insure!

x ( )

Dissallowed

0 1

SMA-HPC 2003 MIT
 Basic Concepts
Continuation Schemes
Template Algorithm
Solve F ( x ( 0 ) , 0 ) , x ( prev ) = x ( 0 )
=0.01, =
While < 1 {
x 0 ( ) = x ( prev )
Try to Solve F ( x ( ) , ) = 0 with Newton
If Newton Converged
x ( prev ) = x ( ) , = + , = 2
Else
1
= , = prev +
2
}
SMA-HPC 2003 MIT
 Basic Concepts
Continuation Schemes
Source/Load Stepping Examples
R
v 1
f ( v ( ) , ) = idiode ( v ) + ( v Vs ) = 0
R
Vs
+
-
Diode
f ( v, ) idiode ( v ) 1
= + Not dependent!
v v R

f G f x ( x, y ) = 0
F ( x, ) =

f y ( x, y ) + f l = 0
fL

Source/Load Stepping Does Not Alter Jacobian

SMA-HPC 2003 MIT
 Jacobian Altering Scheme
Continuation Schemes
Description

F ( x ( ) , ) = F ( x ( ) ) + (1 ) x ( )
Observations
=0 F ( x ( 0 ) , 0 ) = x ( 0 ) = 0 Problem is easy to solve and
F ( x ( 0 ) , 0 ) Jacobian definitely nonsingular.
=I
x

=1 F ( x (1) ,1) = F ( x (1) )

Back to the original problem
F ( x ( 0 ) , 0 ) F ( x (1) ) and original Jacobian
=
x x

SMA-HPC 2003 MIT

 Jacobian Altering Scheme
Continuation Schemes
Basic Algorithm
Solve F ( x ( 0 ) , 0 ) , x ( prev ) = x ( 0 )
=0.01, =
While < 1 {
x 0 ( ) = x ( prev ) + ?
Try to Solve F ( x ( ) , ) = 0 with Newton
If Newton Converged
x ( prev ) = x ( ) , = + , = 2
Else
1
= , = prev +
2
}
SMA-HPC 2003 MIT
 Jacobian Altering Scheme
Continuation Schemes
Initial Guess for each step.

x()

x ( + )
Initial Guess Error
x0 ( + ) = x ( )

0 + 1
SMA-HPC 2003 MIT
 Jacobian Altering Scheme
Continuation Schemes
Update Improvement
F ( x ( + ) , + ) F ( x ( 0) , ) +

F ( x ( ) , )
x
( x ( + ) x ( ) ) +

F ( x ( ) , )


F ( x ( ) , ) F ( x ( ) , )

x
( )
x 0 ( + ) x ( ) =


Better Guess
Have From last
for next steps
steps Newton
Newton
SMA-HPC 2003 MIT
 Jacobian Altering Scheme
Continuation Schemes
Update Improvement Cont.

If
F ( x ( ) , ) = F ( x ( ) ) + (1 ) x ( )

Then
F ( x, )
= F ( x) x ( )

Easily Computed

SMA-HPC 2003 MIT

 Jacobian Altering Scheme
Continuation Schemes
Update Improvement Cont. II.

 ( x ( ) , ) F ( x ( ) , )
1
F
x 0 ( + ) = x ( ) Graphically
x

x()

x0 ( + )

0 + 1
SMA-HPC 2003 MIT
 Jacobian Altering Scheme
Continuation Schemes
Still can have problems
x()

Must switch back to

increasing lambda

Must switch from

Arc-length increasing to
steps
decreasing lambda

0 1
lambda steps

SMA-HPC 2003 MIT

 Jacobian Altering Scheme
Continuation Schemes
Arc-length Steps?
x()

( x ) + ( )
2 2
Arc-length
arc-length
steps

0 1
Must Solve For Lambda
F ( x, ) = 0

( prev ) ( prev )
2 2
+ x x arc 2
=0
2
SMA-HPC 2003 MIT
 Jacobian Altering Scheme
Continuation Schemes
Arc-length steps by Newton

(
F x k , k )  (
F x ,
k k
)
x k +1 x k
x k +1 k
=

( x ( prev ) ) ( )
T
2 x k
2 prev
k


F x ,
k
(k
)

k arc 2
( ) x ( )
2 2

+ x k
prev prev
2

SMA-HPC 2003 MIT

 Jacobian Altering Scheme
Continuation Schemes
x( )
Arc-length Turning point

What happens here?

0 1

Upper left-hand (
F x k , k

) ( )
F x k , k

Block is singular x

( x ( prev ) ) ( )
T
2 x k
2 prev
k

SMA-HPC 2003 MIT
 Summary

Damped Newton Schemes

Globally Convergent if Jacobian is Nonsingular
Difficulty with Singular Jacobians
Introduce Continuation Schemes
Problem with Source/Load stepping
More General Continuation Scheme
Improving Efficiency
Better first guess for each continuation step
Arc-length Continuation
SMA-HPC 2003 MIT