Introduction to Simulation - Lecture 6

Krylov-Subspace Matrix Solution Methods

Jacob White

Thanks to Deepak Ramaswamy, Michal Rewienski,

and Karen Veroy

Outline
General Subspace Minimization Algorithm
Review orthogonalization and projection formulas

Generalized Conjugate Residual Algorithm

Krylov-subspace
Simplification in the symmetric case.
Convergence properties

Eigenvalue and Eigenvector Review

Norms and Spectral Radius
Spectral Mapping Theorem

Arbitrary Subspace
methods

Approach to Approximately Solving Mx=b

w01
wk 11

,...,
{w0 ,..., wk 1}
w

w
k 1N
0N

Pick a kdimensional
Subspace

Approximate x as a weighted sum of {w0 ,..., wk 1}

k

x =
k

k 1

w
i =0

SMA-HPC 2003 MIT

Residual Minimization

Arbitrary Subspace
methods

The residual is defined as r b Mx

k

If x =
k

k 1

w
i

i =0

k 1

r = b Mx = b i Mwi
k

i =0

Residual Minimizing idea: pick i ' s to minimize

r

k 2
2

( ) (r )

SMA-HPC 2003 MIT

k 1

= b i Mwi b i Mwi
i =0
i =0

k 1

Arbitrary Subspace
methods

Minimizing r

k 2
2

= b

Residual Minimization
Computational Approach
k 1

Mw
i =0

is easy if

i
2

( Mw ) = 0, i j or ( Mw ) orthogonal to ( Mw )
Create a set of vectors { p0 ,..., pk 1} such that
( Mwi )

span { p0 ,..., pk 1} = span {w0 ,..., wk 1}

and ( Mpi ) ( Mp j ) = 0, i j
T

SMA-HPC 2003 MIT

Residual Minimization

Arbitrary Subspace
methods

Algorithm Steps

Given M , b and a set of search directions {w0 ,..., wk 1}

1) Generate p j 's by orthogonalizing Mw j ' s
j 1

For j = 0 to k 1 p j = w j
i =0

( Mw ) ( Mp ) p
T

( Mpi ) ( Mpi )
T

2) compute the r minimizing solution x k

k 1

( r ) ( Mp )

i =0

( Mpi ) ( Mpi )

x =
k

SMA-HPC 2003 MIT

0 T

k 1

( r ) ( Mp )

i =0

( Mpi ) ( Mpi )

pi =

i T

pi

Arbitrary Subspace
methods
1) orthogonalize the Mwi ' s

w
p00

w11
p

Residual Minimization
Algorithm Steps by Picture

w
p22

w
p33

2) compute the r minimizing solution x k

M p1
r0

SMA-HPC 2003 MIT

M p0

Minimization Algorithm

Arbitrary Subspace
Solution Algorithm

r 0 = b Ax 0
For j = 0 to k-1
p j = wj
For i = 0 to j-1
T
p j p j ( Mp j ) ( Mpi ) pi
pj

( Mp ) ( Mp )
T

j +1

j +1

Normalize

= x + (r
j

=r

pj

) ( Mp ) p
( r ) ( Mp ) M p

SMA-HPC 2003 MIT

Orthogonalize
Search Direction

j T

Update Solution

j T

Update Residual

Arbitrary Subspace
methods

Subspace Selection
Criteria

Criteria for selecting w0 ,..., wk 1

All that matters is the span {w0 ,..., wk 1}
i ' s such that b Mx k = b

k 1

Mw
i =0

A b in the span {w0 ,..., wk 1} for k

1

is small

One choice, unit vectors, x k span {e1 ,..., ek }

Generates the QR algorithm if k=N
Can be terrible if k < N
SMA-HPC 2003 MIT

Arbitrary Subspace
methods

Subspace Selection
Historical Development

1 T
T
Consider minimizing f ( x ) = x Mx x b
2
T
T

Assume M = M (symmetric) and x Mx > 0 (pos. def)

x f ( x ) = Mx b x = M 1b minimizes f

( )

Pick span {w0 ,..., wk 1} = span x f x ,..., x f x

0

Steepest descent directions for f, but f is not residual

Does not extend to nonsymmetric, non pos def case
SMA-HPC 2003 MIT

k 1

)}

Arbitrary Subspace
methods

Subspace Selection
Krylov Subspace

Note: span x f x 0 ,..., x f x k 1

)}

( )

= span r 0 ,..., r k 1

If: span {w0 ,..., wk 1} = span r ,..., r

then r k = r 0

k 1

Mr
i
i =0

k 1

and span r ,..., r

k 1

} = span {r , Mr ,..., M
0

k 1 0

Krylov Subspace
SMA-HPC 2003 MIT

The Generalized Conjugate

Residual Algorithm

Krylov Methods

The kth step of GCR

( r ) ( Mp )
k T

k =

( Mpk ) ( Mpk )
T

Determine optimal stepsize in

kth search direction

x k +1 = x k + k pk

k +1

Update the solution

and the residual

= r k Mpk

pk +1 = r

k +1

Mr ) ( Mp )
(

p
( Mp ) ( Mp )

SMA-HPC 2003 MIT

j =0

k +1 T

Compute the new

orthogonalized
search direction

The Generalized Conjugate

Residual Algorithm

Krylov Methods

Algorithm Cost for iter k

( r ) ( Mp )
k T

k =

Vector inner products, O(n)

Matrix-vector product, O(n) if sparse

( Mpk ) ( Mpk )
T

x k +1 = x k + k pk

k +1

Vector Adds, O(n)

= r k Mpk

pk +1 = r

k +1

Mr ) ( Mp )
(

p
( Mp ) ( Mp )
k

j =0

k +1 T

O(k) inner products,

total cost O(nk)

If M is sparse, as k (# of iters) approaches n,

3
total cost = O (n ) + O (2n ) + .... + O ( kn ) = O (n )
SMA-HPC 2003 MIT

Better Converge Fast!

The Generalized Conjugate

Residual Algorithm

Krylov Methods

Symmetric Case

An Amazing fact that will not be derived

T
k +1
j
If M = M then r Mp j < k
Mr ) ( Mp )
Mr ) ( Mp )
(
(
p = r
p p =r
p
( Mp ) ( Mp )
( Mp ) ( Mp )
Orthogonalization in one step
If k (# of iters ) n, then symmetric,
sparse, GCR is O(n2 )
Better Converge Fast!
k +1

k +1

j =0

SMA-HPC 2003 MIT

k +1 T

k +1 T

k +1

k +1

Krylov Methods
Nodal Formulation

No-leak Example
Insulated bar and Matrix

Incoming Heat

T (1)

T (0)
Near End
Temperature

Discretization

m
SMA-HPC 2003 MIT

2 1

1 2
Nodal

1 Equation
Form

1 2

Far End
Temperature

Krylov Methods
Nodal Formulation
1

m
SMA-HPC 2003 MIT

No-leak Example
Circuit and Matrix

2 1

1 2

1 2

m 1

Nodal
Equation
Form

Krylov Methods
Nodal Formulation

leaky Example
Conducting bar and Matrix

T (1)

T (0)
Near End
Temperature

Discretization

2.01 1

1 2.01
Nodal

Equation

1 Form

1 2.01

m
SMA-HPC 2003 MIT

Far End
Temperature

leaky Example

Krylov Methods
Nodal Formulation
1

m
SMA-HPC 2003 MIT

Circuit and Matrix

m 1

2.01 1

1 2.01
Nodal

Equation

1
Form

1 2.01

GCR Performance(Random Rhs)

10

R
E
S
I
D
U
A
L

10

10

10

10

Insulating
-1

Leaky
-2

-3

-4

10

20

Iteration

30

40

Plot of log(residual) versus Iteration

SMA-HPC 2003 MIT

50

60

GCR Performance(Rhs = -1,+1,-1,+1.)

0

10

R
E
S
I
D
U
A
L

-1

10

-2

10

-3

10

Insulating

-4

10

Leaky
-5

10

10

15

20

25

Iteration

30

35

Plot of log(residual) versus Iteration

SMA-HPC 2003 MIT

40

45

50

Convergence Analysis

Krylov Subspace
Methods

Polynomial Approach

If span {w0 ,..., wk } = span r 0 , Mr 0 ,..., M k r 0

k +1

M r
i =0

k +1

i 0

= k ( M ) r

kth order polynomial

= r i M r = ( I M k ( M )) r
0

i +1 0

i =0

Note: for any 0 0

0
1
0
0
span r , r = r 0 Mr

SMA-HPC 2003 MIT

}=

span r , Mr

Krylov Methods

Convergence Analysis
Basic Properties

If j 0 for all j k in GCR, then

0
0
k 0
1) span { p0 , p1 ,..., pk } = span r , Mr , ..., M r

2) x

= k ( M )r , k is the k order
k +1
polynomial which minimizes r

k +1

th

2
2

= b Mx = r M k ( M )r
0
0
= ( I M k ( M ) ) r k +1 ( M ) r
th
0
where k +1 ( M ) r is the ( k + 1) order poly
k +1 2
minimizing r
subject to k +1 ( 0 ) =1

3) r

k +1

k +1

SMA-HPC 2003 MIT

Convergence Analysis

Krylov Methods

Optimality of GCR poly

GCR Optimality Property

r

k +1 2
2

k+1 ( M )r

0 2
2

where k+1 is any k order

th

polynomial such that k+1 ( 0 ) =1

Therefore
Any polynomial which satisfies
the zero constraint can be used
to get an upper bound on
SMA-HPC 2003 MIT

k +1 2
2

Eigenvalues and
Vectors Review

Basic Definitions

Eigenvalues and eigenvectors of a matrix M satisfy

eigenvalue

Mui = i ui
eigenvector

Or, i is an eigenvalue of M if

M i I is singular

ui is an eigenvector of M if

( M i I ) ui

SMA-HPC 2003 MIT

=0

Eigenvalues and
Vectors Review

1.1 1
1 1.1

M 11
M
21

M N 1

Examples

1 1 0 0
1 1 0 0

0 0 1 1

0 0 1 2

0
M 22

SMA-HPC 2003 MIT

Basic Definitions

M NN 1

M NN

Eigenvalues?
Eigenvectors?

What about a lower

triangular matrix

A Simplifying
Assumption

Eigenvalues and
Vectors Review

Almost all NxN matrices have N linearly

independent Eigenvectors

u1

u2

u3

uN

= 1u1

2u2

3u3

N u N

The set of all eigenvalues of M is known as

the Spectrum of M
SMA-HPC 2003 MIT

Eigenvalues and
Vectors Review

A Simplifying
Assumption Continued

Almost all NxN matrices have N linearly

independent Eigenvectors

MU =U

1
0

0
0

0
0
N

U MU = or M = U U
1

Does NOT imply distinct eigenvalues, i can equal j

Does NOT imply M is nonsingular
SMA-HPC 2003 MIT

Eigenvalues and
Vectors Review
Im ( )

Spectral Radius

Re ( )

The spectral Radius of M is the radius of the smallest

circle, centered at the origin, which encloses all of
Ms eigenvalues
SMA-HPC 2003 MIT

Eigenvalues and
Vectors Review

Heat Flow Example

Incoming Heat

T (1)

T (0)
Unit Length Rod

T1
+
-

vs = T(0)

SMA-HPC 2003 MIT

TN
+
-

vs = T(1)

Eigenvalues and
Vectors Review

Heat Flow Example

Continued

2 1 0 0
1 2

0
1

0
0
1
2

Eigenvalues N=20

SMA-HPC 2003 MIT

Eigenvalues and
Vectors Review

Heat Flow Example

Continued

Four Eigenvectors Which ones?

SMA-HPC 2003 MIT

Useful
Eigenproperties

Spectral Mapping
Theorem

Given a polynomial

f ( x ) = a0 + a1 x + + a p x

Apply the polynomial to a matrix

f ( M ) = a0 + a1M + + a p M
Then

spectrum ( f ( M ) ) = f ( spectrum ( M ) )

SMA-HPC 2003 MIT

Useful
Eigenproperties

Spectral Mapping
Theorem Proof

Note a property of matrix powers

MM = U U U U = U U
p
p 1
M = U U
1

Apply to the polynomial of the matrix

1
1
p 1
f ( M ) = a0UU + a1U U + + a pU U
Factoring

f ( M ) = U ( a0 I + a1 + + a p
Diagonal

f ( M ) U = U ( a0 I + a1 + + a p
SMA-HPC 2003 MIT

)U

Useful
Eigenproperties

Spectral
Decomposition

Decompose arbitrary x in eigencomponents

x = 1u1 + 2u2 + + N u N

1
Compute by solving U = x = U 1 x

N
Applying M to x yeilds
Mx = M (1u1 + 2u2 + + N u N )
= 11u1 + 2 2u2 + + N N u N
SMA-HPC 2003 MIT

Krylov Methods

Convergence Analysis
Important Observations

1) The GCR Algorithm converges to the exact solution

in at most n steps

Proof: Let n ( x ) = ( x 1 )( x 2 ) ... ( x n )

where i ( M ) .

n ( M ) r 0 = 0 and therefore r n = 0
2) If M has only q distinct eigenvalues, the GCR
Algorithm converges in at most q steps

Proof: Let q ( x ) = ( x 1 )( x 2 ) ... ( x q )

SMA-HPC 2003 MIT

Summary
Arbitrary Subspace Algorithm
Orthogonalization of Search Directions

Generalized Conjugate Residual Algorithm

Krylov-subspace
Simplification in the symmetric case.
Leaky and insulating examples

Eigenvalue and Eigenvector Review

Spectral Mapping Theorem

GCR limiting Cases

Q-step guaranteed convergence