Professional Documents
Culture Documents
2002/
page 3
Chapter 10
10.1
\dqbjV
2002/
page 3
304
Chapter 10.
tions.
xi =
1
b
aii i
n
X
j =1;j 6=i
aij xj ;
i = 1; 2 : : : ; n:
(10.1.1)
In a (minor) step of the iteration we pi
k one equation, say the ith, and then
adjust the ith
omponent of x(k) so that this equation be
omes exa
tly satised.
Hen
e, given x(k) we
ompute
x(ik+1) = x(ik) +
1 (k)
r ; ri(k) = bi
aii i
n
X
j =1
aij x(jk) :
(10.1.2)
In the days of \hand"
omputation one pi
ked an equation with a large residual
and went through the equations in an irregular manner. This is less e
ient
when using a
omputer, and here one usually perform these adjustments for i =
1; 2 : : : ; n, in a
y
li
fashion. The resulting iterative method is
alled the method
of simultaneous displa
ements or Ja
obi's method. Note that all
omponents of
x
an be updated simultaneously and the result does not depend on the sequen
ing
of the equations.
The method of su
essive displa
ements or Gauss{Seidel's method1 diers
from the Ja
obi method by using new values x(jk+1) as soon as they are available as
follows:
jri
x(ik+1) = x(ik) +
1 (k)
r ; ri(k) = bi
aii i
i 1
X
j =1
aij x(jk+1)
n
X
j =i
aij x(jk) ; i = 1; 2 : : : ; n:
(10.1.3)
Here the
omponents are su
essively updated and the sequen
ing of the equations
will in
uen
e the result.
Sin
e ea
h new value xi(k+1)
an immediately repla
e x(ik) in storage the Gauss{
Seidel method storage for unknowns is halved
ompared to Ja
obi's method. For
1
It was noted by G. Forsythe that Gauss nowhere mentioned this method and Seidel never
advo
ated using it!
\dqbjV
2002/
page 3
10.1.
305
x(1k)
x(2k)
x(3k)
x(4k)
1
2
3
4
5
0:25
0:375
0.4375
0.46875
0.48344
:::
0.5
0.5
0.625
0.6875
0.71875
0.73438
:::
0.75
0
0.125
0.1875
0.21875
0.23438
:::
0.25
0.25
0.375
0.4375
0.46875
0.48344
:::
0.5
0.25
0.40625
0.47656
0.49414
0.49854
:::
0.5625
0.70312
0.73828
0.74707
0.74927
:::
0.0625
0.20312
0.23828
0.24707
0.24927
:::
0.40625
0.47656
0.49414
0.49854
0.49963
:::
In both
ases the iteration
onverges, but rather slowly. Note that the
onvergen
e
with Gauss{Seidel's method is, in this example, about twi
e as fast as with the
Ja
obi method. This will be shown in Se
tion 10.3.1 to be true for an important
lass of matri
es, so
alled
onsistently ordered matri
es, see Def. 10.3.4. However,
there are examples for whi
h the Gauss{Seidel method diverges and the Ja
obi
method
onverges, and vi
e versa.
It was noted early that more rapid
onvergen
e
ould often be a
hieved by
making the
orre
tion
x(ik+1) = x(ik) + ri(k) ;
aii
\dqbjV
2002/
page 3
306
Chapter 10.
L. F. Ri hardson [1910
\dqbjV
2002/
page 3
10.1.
307
j=2
sin
ij
; j = 1; : : : ; n:
n+1
A = (I
T ) + (T
I );
\dqbjV
2002/
page 3
Chapter 10.
308
i.e., A is the Krone
ker sum of T and T . It follows that the (n 1)2 eigenvalues
of A are (i + j ), i; j = 1; : : : ; n 1, and hen
e the
ondition number of A is the
same as for T . The same
on
lusion
an be shown to hold for a three dimensional
problem.
10.2
The Ja
obi, Gauss{Seidel, and Ri
hardson methods are all spe
ial
ases of a
lass
of iterative methods, the general form of whi
h is
Mx(k+1) = Nx(k) + b;
Here
A=M
k = 0; 1; : : : :
(10.2.1)
(10.2.2)
x(k+1) = Bx(k) + ;
where
k = 0; 1; : : : ;
B = M 1N = I
M 1 A;
(10.2.3)
= M 1 b:
x(k+1)
x = B (x(k)
x)
(10.2.4)
A = DA LA UA ;
where DA = diag (a11 ; : : : ; ann ),
1
0
0
C
B a21
C ; UA =
LA = B
...
...
A
..
.
an1 an;n 1 0
0
(10.2.5)
0 a12 a1n 1
.. C
... ...
B
. C;
B
0 an 1;n A
0
0
\dqbjV
2002/
page 3
10.2.
309
and LA and UA are stri
tly lower and upper triangular, respe
tively. Assuming that
DA > 0, we
an also write
DA 1 A = I
L U;
L = DA 1 LA ;
U = DA 1 UA :
x(k+1) = (L + U )x(k) + ;
= DA 1 b:
x(k+1) = (I
L) 1 Ux(k) + ; = (I
(10.2.6)
LA )x(k+1) = UA x(k) + b, or
L) 1 DA 1 b
Hen
e these methods are spe
ial
ases of one-step stationary iterative methods, and
orrespond to the matrix splittings
Ja
obi:
M = DA,
N = LA + UA ,
Gauss{Seidel: M = DA LA, N = UA ,
The iteration matri
es for the Ja
obi and Gauss{Seidel methods are
BJ = DA 1 (LA + UA ) = L + U;
BGS = (DA LA ) 1 UA = (I L) 1 U:
The iterative method (10.1.6) is
alled
onvergent if the sequen
e fx(k) gk=1;2;:::
onverges for all initial ve
tors x(0) . Of fundamental importan
e in the study of
onvergen
e of stationary iterative methods is
onditions for a sequen
e of powers
of a matrix to
onverge to the null matrix. For this we need some results from the
theory of eigenvalues of matri
es.
In Se
tion 6.2.2 we introdu
ed the spe
tral radius of a matrix A as the nonnegative number
(A) = max ji (A)j:
1in
We have the following important result:
(i)
(ii)
lim B k = 0,
k!1
lim B k x = 0;
k!1
8x 2 Cn ,
\dqbjV
2002/
page 3
Chapter 10.
310
(iii)
(B ) < 1,
(iv)
kB k < 1
For any ve
tor x we have the inequality kB k xk kB k k kxk, whi
h shows that
(i) implies (ii).
If (B ) 1, then there is an eigenve
tor x 2 Cn su
h that Bx = x, with jj 1.
Then the sequen
e B k x = k x, k = 1; 2; : : :, is not
onvergent when k ! 1 and
hen
e (ii) implies (iii).
By Theorem 10.2.9, (see Se
tion 10.2.4) given a number > 0 there exists a
onsistent matrix norm k k, depending on B and , su
h that
kB k < (B ) + :
Therefore (iv) follows from (iii).
Finally, by applying the inequality kB k k kB kk , we see that (iv) implies (i).
The following ne
essary and su
ient
riterion for
onvergen
e of a stationary
iterative method follows from Theorem 10.2.1.
x(k)
x = B k (x(0)
x):
(10.2.8)
Hen
e x(k)
onverges for all initial ve
tors x(0) if and only if limk!1 B k = 0. The
theorem now follows from Theorem 10.2.1.
The problem of obtaining the spe
tral radius of B is usually no less di
ult
than solving the linear system. Hen
e the following upper bound is useful when
trying to prove
onvergen
e.
Lemma 10.2.3.
For any matrix A 2 Rnn and for any
onsistent matrix norm we have
(A) kAk:
Proof. Let be an eigenvalue of A su
h that jj = (A). Then Ax = x, x 6= 0,
and taking norms
kxk = (A)kxk = kAxk kAk kxk:
Sin
e kxk > 0, we
an divide the inequality by kxk and the theorem follows.
\dqbjV
2002/
page 3
10.2.
311
Usually, we are not only interested in
onvergen
e, but also in the rate of
onvergen
e. By (10.2.8) the error at step k, e(k) = x(k) x, satises e(k) = B k e(0) ,
we have for any
onsistent pair of norms
Rk (B ) =
(10.2.9)
Theorem 10.2.5.
Assume that the eigenvalues i of A are all real and satisfy
0 < a i b; i = 1; : : : ; n:
Then Ri
hardson's method is
onvergent for 0 < ! < 2=b.
\dqbjV
2002/
page 3
312
Chapter 10.
Assuming that a = min and b = max , whi
h value of ! will minimize the
spe
tral radius
(B ) = maxfj1 !aj; j1 !bjg ?
It is left as an exer
ise to show that the optimal ! satises 1 !a = !b 1. (Hint:
Plot the graphs of j1 !aj and j1 !bj for ! 2 (0; 2=b).) Hen
e !opt = 2=(b + a),
for whi
h
2
b a 1
=
=1
;
(B ) =
b+a +1
+1
where = b=a is the
ondition number of A. Note that for large values of the rate
of
onvergen
e with !opt is proportional to 1 . This illustrates a typi
al fa
t for
iterative methods: ill-
onditioned systems require in general more work to a
hieve
a
ertain a
ura
y!
Theorem 10.2.6. The Ja obi method is onvergent if A is stri tly row-wise diagonally dominant, i.e.,
jaii j >
n
X
j=1
j6=i
jaij j; i = 1; 2; : : : ; n:
Proof.
kBJ k1 = 1max
in
n
X
j=1
j6=i
jaii j
n
X
j=1
j6=i
jaij j; i = 1; 2; : : : ; n;
The
olumn sum
riterion
an be similarly improved. The
onditions in Theorem 10.2.6{10.2.7 are also su
ient for
onvergen
e of the Gauss{Seidel method
for whi
h (I L)BGS = U . Consider the stri
tly row-wise diagonally dominant and
hoose k so that
T k = kB T e k :
kBGS k1 = kBGS
1
GS k 1
\dqbjV
2002/
page 3
10.2.
313
T e = B T LT e + U T e , we get
Then from BGS
k
k
k
GS
Example 10.2.1. In Se
tion 10.1.3 it was shown that the (n 1)2 eigenvalues of
the matrix A = (I
T ) + (T
I ) arising from the model problem are (i + j ),
i; j = 1; : : : ; n 1, where i = 2(1 +
os (i=n)). It follows that the eigenvalues of
the
orresponding Ja
obi iteration matrix BJ = L + U = (1=4)(A 4I ) are
1
ij = (
os ih +
os jh); i; j = 1; 2; : : : ; n 1;
2
where h = 1=n is the grid size. The spe
tral radius is obtained for i = j = 1,
(BJ ) = os(h) 1
2
1
2 (h) :
This means that the low frequen
y modes of the error are damped most slowly,
whereas the high frequen
y modes are damped mu
h more qui
kly.3 The same is
true for the Gauss{Seidel method, for whi
h
\dqbjV
2002/
page 3
314
Chapter 10.
B = 0 ; 0 < < 1; 1;
(10.2.10)
<
(10.2.11)
B k = 0 kk ; k = if 6= ;
:
kk 1
if = .
Clearly the element k will grow initially. In the
ase that = the maximum of
jk j will o
ur when k =(1 ). (See also Computer Exer
ise 1.)
For matri
es of larger dimension the initial in
rease of kB k k
an be huge as
shown by the following example:
k
Example 10.2.2. Consider the iteration x(k+1) = Bx(k) , where B 2 R2020 is the
bidiagonal matrix
1
0
0 1
0:5 1
1
C
B
0
:
5
1
C
B1C
B
.
.
(0)
C
C;
.. ..
x =B
B=B
C
B
... A :
A
0:5 1
1
0:5
Here (B ) = 0:15, and hen
e the iteration should
onverge to the exa
t solution of
the equation (I B )x = 0, whi
h is x = 0. From Fig. 10.2.1 it is seen that kx(n) k2
in
reases by almost a fa
tor 1015 until it starts to de
rease after 25 iterations!
Although in the long run the norm is redu
ed by about a fa
tor of 0:5 at ea
h
iteration, large intermediate values of x(n) give rise to persistent rounding errors.
The
urve in Figure 10.2.1 shows a large hump. This is a typi
al phenomenon
in several other matrix problems and o
urs also, e.g., when
omputing the matrix
exponential eBt , when t ! 1.
For the
ase when the iteration pro
ess is
arried out in exa
t arithmeti
we
found a
omplete and simple mathemati
al theory of
onvergen
e for iterates x(k) of
stationary iterative methods. A
ording to Theorem 10.2.1 there is
onvergen
e for
any x(0) if and only if (B ) < 1, where (B ) is the spe
tral radius of B . The same
ondition is ne
essary and su
ient for limk!1 B k = 0 to hold. In nite pre
ision
arithmeti
the
onvergen
e behavior turns out to be more
omplex and less easy to
analyze.
It may be thought that iterative methods are less ae
ted by rounding errors
than dire
t solution methods, be
ause in iterative methods one
ontinues to work
\dqbjV
2002/
page 3
10.2.
315
10
10
10
10
-2
10
-4
10
-6
10
10
20
30
40
50
60
70
80
90
100
x1 = fl b1
n
X
j =1
a1j xj =a11 = b1 (1 + 1 )
n
X
j =1
a1j xj (1 + j ) =a11 ;
with the usual bounds for i ,
f. Se
tion 2.4.1. This
an be interpreted that we
have performed an exa
t Gauss{Seidel step for a perturbed problem with elements
b1 (1 + 1 ) and a1i (1 + i ), i = 2; : : : ; n. The bounds for these perturbations are of
the same order of magnitude that for the perturbations in Gaussian elimination.
The idea that we have worked with the original matrix is not
orre
t.
\dqbjV
2002/
page 3
316
Chapter 10.
where u is the ma
hine pre
ision. (Compare the dis
ussion of pseudospe
tra in
Se
tion 9.3.3.) The following rule of thumb has been suggested:
The iterative method with iteration matrix B
an be expe
ted to
onverge
in nite pre
ision arithmeti
if the spe
tral radius
omputed via a ba
kward
stable eigensolver is less than 1.
This is an instan
e when an inexa
t results is more useful than the exa
t result!
kx x(k) k
(10.2.12)
is satised for some suitable ve
tor norm k k. However, su
h a
riterion
annot
\dqbjV
2002/7
page 3
Review Questions
317
jri(k) j ;
(10.2.14)
(E jx(k) j + f )i
where E > 0 and f > 0, and stop when ! . It then follows from Theorem 6.6.4
! = max
i
Review Questions
1.
2.
3.
4.
The standard dis
retization of Lapla
e equation on a square with Diri
hlet boundary
onditions leads to a
ertain matrix A. Give this matrix in its blo
k triangular form.
What iterative method
an be derived from the splitting A = M N ? How is a
symmetrizable splitting dened?
Dene the average and asymptoti
rate of
onvergen
e for an iterative method
x(k+1) = Bx(k) +
. Does the
ondition (B ) < 1 imply that the error norm
kx x(k) k2 is monotoni
ally de
reasing? If not, give a
ounterexample.
Give at least two dierent
riteria whi
h are suitable for terminating an iterative
method.
Problems
1.
Let A 2 Rnn be a given nonsingular matrix, and X (0) 2 Rnn an arbitrary matrix.
Dene a sequen
e of matri
es by
X (k+1) = X (k) + X (k) (I AX (k) ); k = 0; 1; 2; : : : :
\dqbjV
2002/
page 3
Chapter 10.
318
2.
Iterative Methods
(a) Prove that limk!1 X (k) = A 1 if and only if (I AX (0) ) < 1.
Hint: First show that I AX (k+1) = (I AX (k) )2 .
(b) Use the iterations to
ompute the inverse A 1 , where
A = 11 12 ;
X (0) = 10:9:9 00:9:9 :
Verify that the rate of
onvergen
e is quadrati
!
Let A 2 Rmn be a given nonsingular matrix, Consider the stationary iterative
method
x(k+1) = x(k) + !AT (b Ax(k) );
where A 2 Rmn is a possibly rank de
ient matrix.
2
(a) Show that if rank (A) = n and 0 < ! < 2=max
(A) then then the iteration
onverges to the unique solution to the normal equations AT Ax = AT b.
(b) If rank(A) < n, then split the ve
tor x(k) into orthogonal
omponents,
3.
in this
ase the iteration
onverges to the unique solution of the normal equations
whi
h minimizes kx x(0) k2 .
Show that if for a stationary iterative method x(k+1) = Bx(k) +
it holds that
kB k < 1, and
kx(k) x(k 1) k (1 )=;
then the error estimate kx x(k)k holds.
10.3
\dqbjV
2002/
page 3
10.3.
319
1
b
aii i
i 1
X
j =1
aij x(jk+1)
n
X
j =i
where the expression in bra
kets is the
urrent residual of the ith equation. For
! = 1, the SOR method redu
es to the Gauss{Seidel method. The idea is now
to
hoose ! so that the asymptoti
rate of
onvergen
e is maximized. The SOR
method
an be written in matrix form as
(I
U )x(k) ;
B! = (I
(10.3.2)
We now
onsider the
onvergen
e of the SOR method and rst show that only
values of !, 0 < ! < 2 are of interest.
Theorem 10.3.1.
Let B = L+U be any matrix with zero diagonal and let B! be the
orresponding
iteration matrix in the SOR method. Then we have
(B! ) j! 1j;
(10.3.3)
Proof. Sin
e the determinant of a triangular matrix equals the produ
t of its
diagonal elements we have
det(B! ) = det(I
1j.
The following theorem asserts that if A is a positive denite matrix, then the
SOR method
onverges if 0 < ! < 2.
\dqbjV
2002/
page 3
320
Chapter 10.
Iterative Methods
For an important
lass of matri
es an expli
it expression for the optimal value
of !
an be given. We rst introdu
e the
lass of matri
es with property A.
\dqbjV
2002/
page 3
10.3.
321
Theorem 10.3.5.
Let A = DA(I L U ) be a
onsistently ordered matrix. Then if is an
eigenvalue of the Ja
obi matrix so is . Further, to any eigenvalue 6= 0 of the
SOR matrix B! , ! 6= 0, there
orresponds an eigenvalue of the Ja
obi matrix,
where
+! 1
(10.3.5)
=
!1=2
Proof. Sin
e A is
onsistently ordered the matrix J ( 1) =
L U = J (1) has
the same eigenvalues as J (1). Hen
e if is an eigenvalue so is . If is an
eigenvalue of B! , then det(I B! ) = 0, or sin
e det(I !L) = 1 for all !, using
(10.3.2)
det[(I
!L)(I
B! ) = det[(I
!L) (1 !)I
+! 1
I
!1=2
!U = 0:
(1=2 L + 1=2 U ) = 0
(L + U ) = 0, where
Theorem 10.3.6.
Let A be a
onsistently ordered matrix, and assume that the eigenvalues of
BJ = L + U are real and J = (BJ ) < 1. Then the optimal relaxation parameter
! in SOR is given by
2
p
!opt =
:
(10.3.6)
1 + 1 2J
For this optimal value we have
(B!opt ) = !opt
1:
(10.3.7)
(See also Young [21, Se
tion 6.2.) We
onsider, for a given value of in
the range 0 < (L + U ) < 1, the two fun
tions of ,
Proof.
f! () =
+! 1
;
!
g(; ) = 1=2 :
\dqbjV
2002/
page 3
Chapter 10.
322
Iterative Methods
=1.7, =0.99;
2
1.5
f(,), g(,)
0.5
0.5
1.5
0.2
0.4
0.6
0.8
1.2
1.4
1.6
1.8
2 + 2 (! 1)
1 2 2
! + (!
2
1)2 = 0:
(10.3.8)
1 2 = (! 1)2 :
From this it follows that j1 j = j2 j = ! 1, 1 < !~ < ! < 2, and hen
e the
minimum value of of maxi=1;2 ji j o
urs for !~ . Sin
e the parabola g(; (L + U )) is
the envelope of all the
urves g(; ) for 0 < (L + U ) < 1 the theorem follows.
Example 10.3.2. By (10.3.6) for SOR !opt = 2=(1 + sin h), giving
(B!opt ) = !opt
1=
1 sin h
1 + sin h
1
2h:
(10.3.9)
\dqbjV
2002/
page 3
10.3.
323
R1 (B!opt ) 2h;
whi
h shows that for the model problem the number of iterations is proportional to
n for the SOR method
In Table 10.1.1 we give the number of iterations required to redu
e the norm
of the initial error by a fa
tor of 10 3.
0.95
0.9
0.85
0.8
0.75
0.2
0.4
0.6
0.8
1.2
1.4
1.6
1.8
Figure 10.3.2. The spe
tral radius (B! ) as a fun
tion of ! ( = 0:99,
!b = 1:7527).
A simple s
heme for estimating !opt is to initially perform a xed number of
iterations using ! = 1, i.e., with the Gauss{Seidel method, and attempt to measure
the rate of
onvergen
e. The su
essive
orre
tions satisfy
(n) = x(n+1)
x(n) :
(BJ )2 = (BGS ) n ;
n = k(n+1) k1 =k(n)k1 ;
\dqbjV
2002/
page 3
Chapter 10.
324
Iterative Methods
An estimate of !opt is then obtained by substituting this value into (10.3.6). A
loser
analysis shows, however, that the number of iterations to obtain a good estimate of
!opt is
omparable to the number of iterations needed to solve the original problem
by SOR. The s
heme
an still be pra
ti
al if one wishes to solve a number of systems
involving the same matrix A. Several variations of this s
heme have been developed,
see Young [21, p. 210.
In more
ompli
ated
ases when J is not known, we have to estimate !opt in
the SOR method. From Fig. 10.2.2 we
on
lude that it is mu
h better to overestimate !opt than to underestimate it.
U )x(k) ;
(I
L)x(k+1=2) :
(I
This method is due to Sheldon [1955. The iteration matrix for SSOR is
S! = (I
(1 J )=2
1
2
p
;
(S!b )
:
1 + 2(1 J )
1 + (1 J )=2
In parti
ular, for the model problem in Se
tion 10.1.3 it follows that
1 sin h=2
1 h:
(B!b )
1 + sin h=2
This is half the rate of
onvergen
e for SOR with !opt .
!b =
\dqbjV
2002/
page 3
10.3.
325
A = DA LA UA ;
and where LA and UA are stri
tly lower and upper triangular. As before Ja
obi's
method
an be written DA x(k+1) = (LA + UA )x(k) + b, or with x is partitioned
onformally
Aii x(ik+1)
x(ik) = b
n
X
j =1
Aij x(jk) ; i = 1; : : : ; n:
Hen
e it is important that linear systems in the diagonal blo
ks Aii
an be solved
e
iently.
Example 10.3.3. For the model problem in Se
tion 10.1.3 the matrix A
an
naturally be written in the blo
k form where the diagonal blo
ks Aii = 2I + T
are tridiagonal and nonsingular, see (10.1.1). The resulting systems
an be solved
with little overhead. Note that here the partitioning is su
h that xi
orresponds to
the unknowns at the mesh points on the ith line. Hen
e blo
k methods are in this
ontext also known as \line" methods and the other methods as \point" methods.
Blo
k versions of the Gauss{Seidel, SOR, and SSOR methods are developed
similarly. For SOR we have
Aii x(ik+1)
xi(k) = ! b
i 1
X
j =1
Aij x(jk+1)
n
X
j =i
Aij x(jk) ; i = 1; : : : ; n:
B! = (I
!L) 1 (1 !)I + !U ;
\dqbjV
2002/
page 3
Chapter 10.
326
Iterative Methods
has spe
tral radius (BJ ) < 1. Then the optimal value of ! in the SOR method is
given by (10.3.6). Note that with the blo
k splitting any blo
k-tridiagonal matrix
0
1
D1 U1
B L2 D2 U2
C
B
C
... ...
B
C
A=B
L3
C;
B
C
.
.
. . . . Un 1 A
L n Dn
is
onsistently ordered; for the point methods this was true only in
ase the blo
k
diagonal matri
es Di , i = 1; : : : ; n were diagonal. In parti
ular we
on
lude that
with the blo
k splitting the matrix A in (10.1.1) is
onsistently ordered.
Review Questions
1.
2.
3.
When is the matrix A redu
ible? Illustrate this property using the dire
ted graph
of A.
Let A = DA (I L U ), where DA > 0. When is A said to have \property A". When
is A
onsistently ordered? How are these properties related to the SOR method?
For the model problem the asymptoti
rate of
onvergen
e for the
lassi
al iterative
methods is proportional to hp , where h is the mesh size. Give the value of p for
Ja
obi, Gauss{Seidel, SOR and SSOR. (For the last two methods it is assumed that
the optimal ! is used.
Problems
1.
2.
(a) Show that if A is redu
ible so is AT . Whi
h of the following matri
es are
irredu
ible?
1 0
0 1
1 1
1 1 :
0 1
1 0
0 1
1 0
(b) Is it true that a matrix A, in whi
h the elements take the values 0 and 1 only, is
irredu
ible if and only if the non-de
reasing matrix sequen
e (I + A)k , k = 1; 2; 3; : : :
be
omes a full matrix for some value of k?
The matrix A in (10.1.1) is blo
k-tridiagonal, but its diagonal blo
ks are not diagonal
matri
es. Show that in spite of this the matrix is
onsistently ordered.
Hint: Perform a similarity transformation with the diagonal matrix
D() = diag (D1 (); D2 (); : : : ; Dn ());
\dqbjV
2002/
page 3
10.4.
Convergen e A eleration
10.4
327
Convergen e A eleration
B = M 1N = I
k = 0; 1; : : : ;
(10.4.1)
M 1A:
In this se
tion we des
ribe an important method for a
elerating the
onvergen
e
of the method (10.4.1) provided it is symmetrizable.
Denition 10.4.1. The stationary iterative method (10:4:1) is said to be symmetrizable if there is a nonsingular matrix W su
h that the matrix W (I B )W 1
is symmetri
and positive denite.
For a symmetrizable method the matrix I B has real positive eigenvalues.
A su
ient
ondition for a method to be symmetrizable is that both A and the
splitting matrix M are symmetri
, positive denite., sin
e then there is a matrix W
su
h that M = W T W , and
W (I
B )W 1 = W M 1AW 1 = W W 1 W
T AW
1=W
T AW
1;
x~(k) =
k
X
i=0
ki x(i) ;
k
X
i=0
ki = 1; k = 0; 1; 2; : : : ;
(10.4.2)
Su
h methods, introdu
ed by Varga (1957), are non-stationary and
alled semiiterative methods. It follows from the error equation x(k) x = B k (x(0) x)
that
x~(k)
x=
k
X
i=0
ki (x(i)
x) =
k
X
i=0
ki B i (x(0)
x):
x~(k)
x = pk (B )(x(0)
x);
pk () =
k
X
i=0
ki i ;
(10.4.3)
\dqbjV
2002/
page 3
Chapter 10.
328
Iterative Methods
where pk () is a polynomial of degree k. Therefore this pro
edure is also known as
polynomial a
eleration. Note that from (10.4.2) it follows that pk (1) = 1. In
the spe
ial
ase that M = I we obtain for the residual r~(k) = b Ax~(k)
qk () = pk (1 ):
(10.4.4)
where we have used that A = I B . The polynomials qk () have the property that
qk (0) = 1 and are known as residual polynomials.
r(k) = qk (A)r(0) ;
qk () =
kY1
i=0
(I
!i ):
(10.4.5)
(10.4.6)
q21k 2[a;b
where 1k denotes the set of residual polynomials qk of degree k su
h that qk (0) =
1. By a similar argument as used in the proof of the minimax property of Chebyshev
polynomials, see Se
tion 9.3.4, the solution to the above minimization problem is
given by the shifted and normalized Chebyshev polynomials
(10.4.7)
\dqbjV
2002/
page 3
10.4.
Convergen e A eleration
329
where Tk (z ) is the Chebyshev polynomial of degree k and z () the linear transformation, whi h maps the interval 2 [a; b onto z 2 [ 1; 1. Hen e
z () =
where
b + a 2
=
b a
2
;
b a
(10.4.8)
b+a +1
b
=
> 1;
= :
(10.4.9)
b a 1
a
Note that if M 1A is symmetrizable is the
ondition number of M 1 A.
Sin
e jTk (z )j 1, z 2 [ 1; 1, and > 1, we have
= z (0) =
k )
= log
> 12 ek
;
1. We obtain using (10.4.9)
p + 1 2
p 1 > p :
p
(Verify the last inequality! See Problem 2.) Hen
e Tk () > 12 e2k= , and to redu
e
the error norm at least by a fa
tor of < 1 it su
es to perform k iterations, where
2
1p
(10.4.10)
log ;
2
k>
where
x(k+1)=x(k) + !k M 1 (b Ax(k) ); k = 0; 1; : : : ; N
!k=2 (b + a) (b a) os k + 21 =N
1;
(10.4.11)
; k = 0; 1; : : : ; N
1(10.4.12)
:
\dqbjV
2002/
page 3
Chapter 10.
330
Iterative Methods
T1 (z ) = zT0;
Tk+1 (z ) = 2zTk (z ) Tk 1 (z ); k 1:
(10.4.13)
x) Tk 1 ()~x(k 1) :
Further, we have
M 1A(~x(k)
x) = M 1 (Ax~(k)
b) = M 1r(k) :
Substituting Tk 1 () = 2Tk ()+ Tk+1 () and dividing with Tk+1 () we obtain
x~(k 1) );
k = 1; 2; : : : ;
!k = 2
Tk ()
;
Tk+1 ()
k = !k ; k 1:
A similar
al
ulation for k = 0 gives x~(1) = x~(0) + M 1 r(0) . Dropping the tilde
this leads to the following algorithm:
Algorithm 10.4.1
The Chebyshev Semi-Iterative Method
Assume that the eigenvalues fi gni=1 of M 1 A are real and satisfy 0 < a i < b.
Then
(0) M 1 r(0) ;
k = 0,
x(k+1) = xx(k +
(10.4.14)
1) + !k M 1 r(k) + x(k) x(k 1) ; k = 1; 2; : : : ;
where = (b + a)=(b a), = 2=(b + a), and
!0 = 2; !k = 1
!k 1
42
; k = 1; 2; : : : :
\dqbjV
2002/7
page 3
Review Questions
331
A disadvantage of Chebyshev
onvergen
e a
eleration is that it requires knowledge of an interval [a; b en
losing the (real) spe
trum of M 1A is needed. If this
the en
losure is too
rude, then the pro
ess loses e
ien
y.
The eigenvalues of the iteration matrix of the SOR-method B!opt are all
omplex and have modulus j!opt j. Therefore in this
ase
onvergen
e a
eleration is of
no use. (A pre
ise formulation is given in Young [21, p. 375.) However, Chebyshev
a
eleration
an be applied to the Ja
obi and SSOR methods, with
1
! 1
DA LA DA 1 DA UA ;
MJ = DA ;
M! =
2 ! !
!
respe
tively, as well as blo
k versions of these methods, often with a substantial
gain in
onvergen
e rate.
Review Questions
1.
Problems
1.
2.
3.
jbj
jbj
Use this to give estimates for the two spe
ial
ases (i) Symmetri
intervals with
respe
t to the origin. (ii) The
ase when jbj
.
\dqbjV
2002/
page 3
Chapter 10.
332
Iterative Methods
Let A be a matrix with real eigenvalues fi gni=1 , 0 < a i < b. Then the
Chebyshev semi-iterative method for solving Ax = b
an be implemented by the
re
ursion (10.4.11){(10.4.12). The instability of this s
heme
an be eliminated by
using an ordering of the iteration parameters !k given by Lebedev and Finogenov.
For N = 2p this permutation ordering is
onstru
ted by the following Matlab
program:
N = 2p ; int = 1;
kappa = ones(1; N );
for i = 1 : p
int = 2 int; ins = int + 1;
for j = int=2 : 1 : 1
kappa(2 j ) = ins kappa(j );
kappa(2 j 1) = kappa(j );
end;
end;
Implement and test this method using the system Ax = b from the Lapla
e equation
on the unit square, with A blo
k tridiagonal
2
2
A = tridiag( I; T + 2I; I ) 2 Rn n ;
T = tridiag( 1; 2 1) 2 Rnn :
Constru
t the right hand so that the exa
t solution be
omes x = (1; 1; : : : ; 1; 1)T .
Let x(0) = 0 as initial approximation and solve this problem using
The implementation based on the three term re
ursion of Chebyshev polynomials
Ri
hardson implementation with natural ordering of the parameters
Ri
hardson implementation with the Lebedev{Finogenov ordering of the parameters
Take n = 50 and N = 128. Use the same number of iterations in all three implementations. List in ea
h
ase the maximum norm of the error and the residual. Compare
the results and draw
on
lusions!
10.5
x^ 2 K;
b Ax^ ? L:
(10.5.1)
\dqbjV
2002/
page 3
10.5.
333
K = R(U );
L = R(V );
(10.5.2)
A = 01 11 :
Then A^ = 0. Note that the matrix A here is symmetri
, but not positive denite.
There are two important spe
ial
ases in whi
h the matrix A^
an be guaranteed
to be nonsingular.
1. Let A by symmetri
, positive denite (s.p.d.) and L = K. Then we
an take
V = U , and have A^ = U T AU . Clearly A^ is s.p.d., and hen
e also nonsingular.
2. Let A be nonsingular and L = AK. Then we
an take V = AU , and we get
A^ = U T AT AU . Here AT A is s.p.d. and hen
e A^ is nonsingular.
We now derive important optimality properties satised in these two spe
ial
ases. For this purpose we rst dene a new inner produ
t and norm related to a
s.p.d. matrix A.
Lemma 10.5.2.
Let A by symmetri
, positive denite (s.p.d.) and
onsider the
ase L = K,
(V = U ). Then x^ = U (U T AU ) 1 U T b minimizes the A-norm of the error over all
ve
tors x 2 K, i.e., x^ solves the problem
min kx x kA ; x = A 1 b:
(10.5.5)
x2K
\dqbjV
2002/
page 3
Chapter 10.
334
Iterative Methods
Proof.
But here the last term is zero be
ause vT Ae^ = vT (Ax^ b) = 0. It follows that kekA
is minimum if v = 0.
A related result is obtained for the se
ond
ase.
Lemma 10.5.3.
Let A be nonsingular and
onsider the
ase L = AK, (V = AU ). Then
x^ = U (U T AT AU ) 1 U T AT b minimizes the 2-norm of the residual over all ve
tors
x 2 K, i.e.,
min kb Axk2 :
(10.5.6)
x2K
Proof. Using x = Uz we have kb Axk2 = kb AUz k2, whi
h is minimized when z
satises the normal equations U T AT AUz = U T AT b. This gives the desired result.
In an iterative method often a sequen
e of proje
tion steps of the above form
is taken. Then we need to modify the above algorithms slightly so that they
an
start from a given approximation xk .4
If we let x = xk + z , then z satises the system Az = rk , where rk = b Axk .
In step k we now apply the above proje
tion method to this system. Hen
e we
require that z 2 K and that rk Az = b A(xk + z ) ? L. This gives the equations
rk = b Axk ;
z^ = (V T AU ) 1 V T rk ;
xk+1 = xk + Uz:
(10.5.7)
for
omputing the new approximation xk+1 . A generi
proje
tion algorithm is
obtained by starting from some x0 (e.g., x0 = 0), and repeatedly perform (10.5.7)
for a sequen
e of subspa
es L = Lk , K = Kk , k = 1; 2; : : :.
where we have to require that vkT Auk 6= 0. By
onstru
tion the new residual rk+1
is orthogonal to vk . Note that rk
an be
omputed re
ursively from
rk = rk 1
k 1 Auk 1 :
(10.5.9)
In the rest of this
hapter we will use ve
tor notations and xk will denote the kth approximation
and not the kth
omponent of x.
4
\dqbjV
2002/
page 3
10.5.
335
rk = b Axk ; k =
uTk rk
; xk+1 = xk + k uk ;
uTk Auk
(10.5.10)
(10.5.11)
1
(uTk rk )2
; (!) = !(2 !);
2
uTk Auk
(10.5.12)
(10.5.13)
x^ xk+1 = x^ xk
0 < ! < 2:
uT r
! Tk k uk = I
uk Auk
u uT
! Tk k A (^x xk ):
uk Auk
Example 10.5.2. For the Gauss{Seidel method in the ith minor step the ith
omponent of the
urrent approximation xk is
hanged so that the ith equation is
satised, i.e., we take
xk := xk
^ei ;
eTi (b A(xk
^ei )) = 0;
where ei is the ith unit ve
tor. Hen
e the Gauss{Seidel method is equivalent to a
sequen
e of one-dimensional modi
ations where the sear
h dire
tions are
hosen
equal to the unit ve
tors in
y
li
order e1 ; : : : ; en ; e1; : : : ; en ; : : :.
This interpretation
an be used to prove
onvergen
e for the Gauss{Seidel
(and more generally the SOR method) for the
ase when A is s.p.d..
\dqbjV
2002/
page 3
Chapter 10.
336
Iterative Methods
Theorem 10.5.4.
If A is symmetri
, positive denite then the SOR method
onverges for 0 <
! < 2, to the unique solution of Ax = b. In parti
ular the Gauss{Seidel method,
whi
h
orresponds to ! = 1,
onverges.
In a minor step using sear
h dire
tion ei the value of will de
rease unless
eTi (b Axk ) = 0, i.e., unless xk satises the ith equation. A major step
onsists
of a sequen
e of n minor steps using the sear
h dire
tions e1 ; : : : ; en . Sin
e ea
h
minor step ee
ts a linear transformation of the error yk = x^ xk , in a major
step it holds that yk+1 = Kyk , for some matrix K . Here kKyk kA < kyk kA unless
yk is un
hanged in all minor steps, i = 1; : : : ; n, whi
h would imply that yk = 0.
Therefore if yk 6= 0, then kKyk kA < kyk kA, and thus kK kA < 1. It follows that
kK ny0 kA kK knAky0kA ! 0 when n ! 1;
i.e., the iteration
onverges.
If we dene the minor step as x := !^ei , where ! is a x relaxation fa
tor,
the
onvergen
e proof also holds. (We may even let ! vary with i, although the
proof assumes that ! for the same i has the same value in all major steps.) This
shows that the SOR method is
onvergent and by Theorem 10.2.2 this is equivalent
to (B! ) < 1, 0 < ! < 2.
Proof.
We make two remarks about the
onvergen
e proof. First, it also holds if
for the basis ve
tors fei gni=1 we substitute an arbitrary set of linearly independent
ve
tors fpj gni=1 . Se
ond, if A is a positive diagonal matrix, then we obtain the
exa
t solution by the Gauss{Seidel method after n minor steps. Similarly, if A
assumes diagonal form after a
oordinate transformation with, P = (p1 ; : : : ; pn );
i.e., if P T AP = D, then the exa
t solution will be obtained in n steps using sear
h
dire
tions p1 ; : : : ; pn . Note that this
ondition is equivalent to the requirement that
the ve
tors fpj gni=1 should be A-orthogonal, pTi Apj = 0;
i 6= j .
k rkT rk ;
\dqbjV
2002/
page 3
10.5.
337
where we have used that rkT+1 rk = 0. Using the expression (10.5.14) for k we
obtain
Tr
rkT rk
k
:
(10.5.15)
kek+1 k2A = kek k2A 1 rrTkAr
T
1
k rk A rk
k
Assume that A is s.p.d. with eigenvalues equal to 0 < 1 2 : : : n . Then
from a well known inequality by Kantorovi
h it follows that (see, e.g., Axelsson and
Barker [1984, Se
. 1.2) that for the method of steepest des
ent
kx xk kA
1
+1
k
kx x0 kA ;
= 1 =n :
(10.5.16)
It
an also be shown that asymptoti
ally this bound is sharp. Hen
e, the asymptoti
rate of
onvergen
e only depends on the extreme eigenvalues of A.
If the matrix A is ill-
onditioned the level
urves of are very elongated
hyper-ellipsoids. Then the su
essive iterates xk , k = 0; 1; 2; : : : will zig-zag slowly
towards the minimum x = A 1 b as illustrated in Fig. 10.5.3 for a two dimensional
ase. Note that su
essive sear
h dire
tions are orthogonal.
pk+1 = rk+1 + k pk ; k = 0; 1; 2 : : : :
(10.5.17)
\dqbjV
2002/
page 3
Chapter 10.
338
Iterative Methods
regardless of the value of the parameter k , and (xk+1 ) < (xk ). From (10.5.18)
we also obtain the alternative expression
k = (rkT rk )=(pTk Apk ):
(10.5.19)
Review Questions
1.
2.
3.
10.6
The Lan
zos algorithm and the
onjugate gradient algorithm of Hestenes and Stiefel
are the most important examples of Krylov subspa
e methods. They were developed already in the early 1950s, but did not
ome into wide use until twenty years
later. Now these methods,
ombined with pre
onditioning te
hniques, have been
developed into sophisti
ated solvers for large s
ale linear systems. Today these are
the standard method for solving linear systems involving large, sparse, symmetri
(or Hermitian) systems.
We
onsider now proje
tion methods where the subspa
es K and L are
hosen to be
the sequen
e of Krylov subspa
es Kk+1 (r0 ; A), k = 0; 1; 2; : : :, where r0 = b Ax0 ,
and
Km (r0 ; A) = span fr0 ; Ar0 ; : : : ; Am 1 r0 g:
(10.6.1)
This
hoi
e leads to one of the most important iterative methods for solving large
symmetri
positive denite systems, the
onjugate gradient method.
If p0 = r0 , and the re
urren
e (10.5.17) is used to generate pk+1 then a simple
indu
tion argument shows that the ve
tors pk and rk both will lie in the Kk+1 (r0 ; A).
In the
onjugate gradient method the parameter k is
hosen to make pk+1 Aorthogonal or
onjugate to the previous sear
h dire
tion, i.e.,
pTk+1 Apk = 0:
(10.6.2)
(A motivation to this
hoi
e is given in the se
ond remark to Theorem 10.5.4.)
Multiplying (10.5.17) by pTk A and using (10.6.2) it follows that
k = (pTk Ark+1 )=(pTk Apk ):
(10.6.3)
\dqbjV
2002/
page 3
10.6.
339
We now prove the important result that this hoi e will in fa t make pk+1 A onjugate to all previous sear h dire tions!
Lemma 10.6.1.
In the
onjugate gradient algorithm the residual ve
tor rk is orthogonal to all
previous sear
h dire
tions and residual ve
tors
rkT pj = 0; j = 0; : : : ; k 1;
(10.6.4)
pTk Apj = 0; j = 0; : : : ; k 1:
(10.6.5)
We rst prove the relations (10.6.4) and (10.6.5) jointly by indu
tion.
Clearly rk is orthogonal to the previous sear
h dire
tion pk 1 , and (10.6.2) shows
that also (10.6.5) holds for j = k 1, Hen
e these relations are
ertainly true for
k = 1.
Assume now that the statements are true for some k 1. From pTk rk+1 = 0,
hanging the index, and taking the s
alar produ
t with pj , 0 j < k we get
Proof.
rkT+1 pj = rkT pj
k pTk Apj :
From the indu
tion hypothesis this is zero, and sin
e rkT+1 pk = 0 it follows that
(10.6.4) holds for k := k + 1. Using equation (10.5.17), the indu
tion hypothesis
and equation (10.5.9) and then (10.5.17) again we nd for 0 < j < k
Theorem 10.6.2.
The ve
tor xk in the
onjugate gradient method solves the minimization problem
1
kx x k2A ; x x0 2 Kk (r0 ; A)
(10.6.6)
min
(
x
)
=
x
2
From this property it follows dire
tly that the \energy" norm kx x kA in
the CG method is monotoni
ally de
reasing. It
an also be shown that the error
norm kx xk k2 is monotoni
ally de
reased (see Hestenes and Stiefel [9).
\dqbjV
2002/
page 3
Chapter 10.
340
Iterative Methods
Sin
e the ve
tors r0 ; : : : ; rk 1 span the Krylov subspa
e Kk (r0 ; A) the following
orthogonality relations also hold:
rkT rj = 0; j = 0; : : : ; k 1:
(10.6.7)
Equation (10.6.7) ensures that in exa
t arithmeti
the
onjugate gradient method
will terminate after at most n steps. For suppose the
ontrary is true. Then
rk 6= 0, k = 0; 1; : : : ; n and by (10.6.7) these n + 1 nonzero ve
tors in Rn are
mutually orthogonal and hen
e linearly independent, whi
h is impossible. Hen
e
the
onjugate gradient method is in ee
t a dire
t method! However, as is now
well know, round-o errors spoil the nite termination property and this aspe
t has
little pra
ti
al relevan
e.
From these relations, we
an
on
lude that the residuals r0 ; r1 ; : : : ; rk are
the same ve
tors as those obtained from the sequen
e r0 ; Ar0 ; : : : ; Ak r0 by GramS
hmidt orthogonalization. This gives a
onne
tion to the Lan
zos pro
ess des
ribed
in Se
tion 10.8.4, whi
h is further dis
ussed below in Se
tion 10.6.3. The ve
tors
p0 ; p1 ; p2; : : : may be
onstru
ted similarly from the
onjuga
y relation (10.6.5).
An alternative expression for k is obtained by multiplying the re
ursive expression for the residual rk+1 = rk k Apk by rkT+1 and using the orthogonality
(10.6.7) to get rkT+1 rk+1 = k rkT+1 Apk . Equations (10.5.19) and (10.6.3) then
yield
k = rkT+1 rk+1 =rkT rk :
We observe that in this expression for k the matrix A is not needed. This property
is important when the
onjugate gradient method is extended to non-quadrati
fun
tionals.
We now summarize the
onjugate gradient method. We have seen that there
are alternative, mathemati
ally equivalent formulas for
omputing rk , k and k .
However, these are not equivalent with respe
t to a
ura
y, storage and
omputational work. A
omparison tends to favor the following version:
Algorithm 10.6.1
The Conjugate Gradient Method
r0 = b Ax0 ; p0 = r0 ;
for k = 0; 1; 2; : : : while krk k2 > do
k = (rk ; rk )=(pk ; Apk );
xk+1 = xk + k pk ;
rk+1 = rk k Apk ;
k = (rk+1 ; rk+1 )=(rk ; rk );
pk+1 = rk+1 + k pk ;
end
Here the inner produ
t used is (p; q) = pT q. Four ve
tors x; r; p and Ap need to
\dqbjV
2002/
page 3
10.6.
341
be stored. Ea
h iteration step requires one matrix by ve
tor produ
t when forming
Ap, two ve
tor inner produ
ts and three s
alar by ve
tor produ
ts.
By instead taking the inner produ
t in the above algorithm to be (p; q) = pT Aq
we obtain a related method that in ea
h step minimizes the Eu
lidian norm of
the residual over the same Krylov subspa
e. In this algorithm the ve
tors Api ,
i = 0; 1; : : : are orthogonal. In addition, the residual ve
tors are required to be
A-orthogonal, i.e.,
onjugate. Consequently this method is
alled the
onjugate
residual method. This algorithm requires one more ve
tor of storage and one more
ve
tor update than the
onjugate gradient method. Therefore, when appli
able the
onjugate gradient method is usually preferred over the
onjugate residual method.
rk = qk (A)r0 ;
where qk 2 ~ 1k , the set of polynomials qk of degree k with qk (0) = 1. Sin
e
(x) = 1 kx x k2 = 1 rT A 1 r = krk2 1 ;
2
krk k2A
(10.6.8)
r0 =
we have for any qk 2 ~ 1k
n
X
i=1
i vi ;
(10.6.9)
T
1 qk (A)r0 = X
2 1 qk (i )2 :
T
1 = r0 qk (A) A
i i
i=1
1
1
;
(10.6.10)
1
2
n
we get krn kA 1 = 0. This is an alternative proof that the CG method terminates
after at most n steps in exa
t arithmeti
.
If the eigenvalues of A are distin
t then qn in (10.6.10) is the minimal polynomial of A (see Se
tion 10.1.2). If A only has p distin
t eigenvalues then the minimal
qn () = 1
\dqbjV
2002/
page 3
Chapter 10.
342
Iterative Methods
n
X
i=1
kx xk kA Mk kx x0 kA :
(10.6.11)
We now sele
t a set S on the basis of some assumption regarding the eigenvalue
distribution of A and seek a polynomial q~k 2 ~ 1k su
h that Mk = max2S jq~k ()j is
small.
~ 1k whi
h
A simple
hoi
e is to take S = [1 ; n and seek the polynomial q~k 2
minimizes max1 n jqk ()j. The solution to this problem is known to be a shifted
and s
aled Chebyshev polynomial of degree k, see the analysis in Se
tion 10.4.2. It
follows that
k
p
(10.6.12)
kx xk kA < 2 p + 11 kx x0 kA:
where = n (A)=1 (A). This estimate is the same as (10.4.10) for Chebyshev
semi-iteration.
We note that the
onvergen
e of the
onjugate residual method
an be analyzed using a similar te
hnique.
Example 10.6.1. For the model problem in Se
tion 10.1.3 the extreme eigenvalues
of 41 A are max = 1 +
os h, min = 1
os h. It follows that
=
1 +
os h
1
os h
For h = 1=100 the number of iterations needed to redu
e the initial error by a fa
tor
of 10 3 is then bounded by
p
1
k log 2 103 242:
2
\dqbjV
2002/
page 3
10.6.
343
This is about the same number of iterations as needed with SOR using !opt to
redu
e the L2-norm by the same fa
tor. However, the
onjugate gradient method
is more general in that it does not require the matrix A to have \property A".
The error estimate above tends to be pessimisti
asymptoti
ally. One often
observes, in pra
ti
e, a superlinear
onvergen
e for the
onjugate gradient method.
This
an be theoreti
ally explained for the
ase when there are gaps in the spe
trum of A. Then, as the iterations pro
eeds, the ee
t of the smallest and largest
eigenvalues of A are eliminated and the
onvergen
e then behaves a
ording to
a smaller "ee
tive"
ondition number. This behavior,
alled superlinear
onvergen
e, is in
ontrast to the Chebyshev semi-iterative method, whi
h only takes the
extreme eigenvalues of the spe
trum into a
ount and for whi
h the error estimate
in Se
tion 10.4.2 tends to be sharp asymptoti
ally.
We have seen that, in exa
t arithmeti
, the
onjugate gradient algorithm will
produ
e the exa
t solution to a linear system Ax = b in at most n steps. However, in
the presen
e of rounding errors, the orthogonality relations in Theorem 10.5.4 will
no longer be satised exa
tly. Indeed, orthogonality between residuals ri and rj , for
ji j j is large, will usually be
ompletely lost. Be
ause of this, the nite termination
property does not hold in pra
ti
e. Its main use is instead as an iterative method
for solving large, sparse, well-
onditioned linear systems, using far fewer than n
iterations.
The behavior of the
onjugate gradient algorithm in nite pre
ision is mu
h
more
omplex than in exa
t arithmeti
. It has been observed that the bound
(10.6.12) still holds to good approximation in nite pre
ision. On the other hand a
good approximate solution may not be obtained after n iterations, even though a
large drop in the error sometimes o
urs after step n. It has been observed that the
onjugate gradient algorithm in nite pre
ision behaves like the exa
t algorithm
applied to a larger linear system A^x^ = ^b, where the matrix A^ has many eigenvalues distributed in tiny intervals about the eigenvalues of A. This means that
(A^) (A), whi
h explains why the bound (10.6.12) still applies. It
an also be
shown that even in nite pre
ision krk k2 ! 0, where rk is the re
ursively
omputed
residual in the algorithm. (Note that the norm of true residual kb Axk k2
annot
be expe
ted to approa
h zero.) This means that a termination
riterion krk k2
will eventually always be satised even if u, where u is the ma
hine pre
ision.
Table 10.6.1. Maximum error for Example 10.6.2 using Chebyshev iteration with optimal parameters and the
onjugate gradient algorithm.
Iteration Chebyshev Conj. gradient
1
1:6 10 2
1:6 10 2
4
2
7:1 10
6:5 10 4
5
3
1:1 10
1:0 10 5
7
4
2:7 10
1:0 10 7
9
5
4:3 10
8:1 10 10
6
1:2 10 10
5:7 10 12
\dqbjV
2002/
page 3
Chapter 10.
344
Iterative Methods
2u 2u
+ p(x; y)u = f (x; y);
+
x2 y2
p=
6(x2 + y2 )
;
1 + 12 (x4 + y4 )
1 2
B 2 2
B
...
Tk = B
B
3
C
C
...
...
C:
C
k 1 k 1 k A
k k
and a matrix Vk = (v1 ; : : : ; vk ) with orthogonal olumns spanning the Krylov subspa e Kk (v1 ; A) su h that
(10.6.13)
In the
ontext of solving the linear system Ax = b, using Krylov subspa
es the
appropriate
hoi
e of starting ve
tor is
1 v1 = r0 = b Ax0 ;
1 = kr0 k2:
We write the kth approximation as xk = x0 + Vk yk 2 Kk (r0 ; A). Here yk is determined by the
ondition that rk = b Axk is orthogonal to Kk (r0 ; A), i.e., VkT rk = 0.
Using (10.6.13) we have
rk = r0
AVk yk = 1 v1
Vk Tk yk
VkT rk = 0 = 1 e1
Tk yk :
(10.6.14)
\dqbjV
2002/
page 3
10.6.
345
Tk yk = 1 e1 ;
(10.6.15)
and then xk = x0 + Vk yk . Mathemati
ally this gives the same sequen
e of approximations as generated by the CG method. Moreover, the
olumns of Vk equal the
rst k residual ve
tors in the
onjugate gradient method, normalized to unit length.
The Lan
zos pro
ess stops if k+1 = krk k2 = 0 sin
e then vk+1 is not dened.
However, then we have AVk = Vk Tk and using (10.6.14)
0 = rk = 1 v1
Vk Tk yk = r0
AVk yk = r0
A(xk
x0 ):
x2Sk
Sk = x0 + Kk (r0 ; A):
In
ase A is symmetri
but indenite kkA is no longer a norm. Hen
e the standard
onjugate gradient method may break down. This is also true for the
onjugate
residual method.
A Krylov subspa
e method for symmetri
indenite systems was given by
Paige and Saunders [15, 1975. Using the Lan
zos basis Vk they seek approximations
xk = Vk yk 2 Kk (b; A), whi
h are stationary values of kx^ xk k2A . These are given
by the Galerkin
ondition
VkT (b AVk yk ) = 0:
This leads again to the tridiagonal system (10.6.15). However, when A is indenite,
although the Lan
zos pro
ess is still well dened, the Cholesky fa
torization of Tk
may not exist. Moreover, it may happen that Tk is singular at
ertain steps, and
then yk is not dened.
If the Lan
zos pro
ess stops for some k n then AVk = Vk Tk . It follows
that the eigenvalues of Tk are a subset of the eigenvalues of A, and thus if A is
\dqbjV
2002/7
page 3
346
Chapter 10.
Iterative Methods
B
C:
Tk G12 Gk 1;k = Lk = B 3 3 3
C
... ... ...
A
k k
k
The rotation Gk 1;k is dened by elements
k 1 and sk 1 . The bar on the element
k is used to indi
ate that L k diers from Lk , the k k leading part of L k+1 , in
the (k; k) element only. In the next step the elements in Gk;k+1 are given by
k =
k =
k ;
sk = k+1 =
k :
k = (
k2 + k2+1 )1=2 ;
Sin
e the solution yk of Tk yk = 1 e1 will
hange fully with ea
h in
rease in k
we write
k zk ;
xk = Vk yk = (Vk QTk )zk = W
and let
k = (w1 ; : : : ; wk 1 ; wk );
W
zk = (1 ; : : : ; k 1 ; k ) = Qk yk :
k
an be
Here quantities without bars will be un
hanged when k in
reases, and W
updated with Tk . The system (10.6.15) now be
omes
k zk :
L k zk = 1 e1 ;
x
k = W
This formulation allows the vi and wi to be formed and dis
arded one by one.
In implementing the algorithm we should note that x
k need not be updated
at ea
h step, and that if
k = 0, then zk is not dened. Instead we update
xLk = Wk zk = xLk 1 + k wk ;
where Lk is used rather than L k . We
an then always obtain x
k+1 when needed
from
x
k+1 = xLk + k+1 wk+1 :
This denes the SYMMLQ algorithm. In theory the algorithm will stop with k+1 =
0 and then x
k = xLk = x. In pra
ti
e it has been observed that k+1 will rarely be
small and some other stopping
riterion based on the size of the residual must be
used.
Paige and Saunders also derived an algorithm
alled MINRES, whi
h is based
on minimizing the Eu
lidian norm of the residual rk . It should be noted that
MINRES suers more from poorly
onditioned systems than SYMMLQ does.
\dqbjV
2002/
page 3
Review Questions
347
Review Questions
1.
2.
3.
Dene the Krylov spa
e Kj (b; A). Show that it is invariant under (i) s
aling A.
(ii) translation A sI . How is it ae
ted by an orthogonal similarity transformation
= V T AV ,
= V T b?
What minimization problems are solved by the
onjugate gradient method? How
an this property be used to derive an upper bound for the rate of
onvergen
e of
the
onjugate gradient method.
Let the symmetri
matrix A have eigenvalues i and orthonormal eigenve
tors vi ,
i = 1; : : : ; n. If only d < n eigenvalues are distin
t, what is the maximum dimension
of the Krylov spa
e Kj (b; A)?
Problems
1.
2.
3.
4.
10.7
Nonsymmetri Problems
An ideal
onjugate gradient-like method for nonsymmetri
systems would be
hara
terized by one of the properties (10.6.6) or (10.6.14). We would also like to be
able to base the implementation on a short ve
tor re
ursion. Unfortunately, it turns
out that su
h an ideal method essentially
an only exist for matri
es of very spe
ial
form. In parti
ular, a two term re
ursion like in the CG method is only possible in
ase A either has a minimal polynomial of degree 1, or is Hermitian, or is of the
form
A = ei (B + I );
B = BH ;
where and are real. Hen
e the
lass essentially
onsists of shifted and rotated
Hermitian matri
es.
\dqbjV
2002/
page 3
Chapter 10.
348
Iterative Methods
AT Ax = AT b
give the
onditions for the solution of the linear least squares problems
min
kAx bk2 ;
x
(10.7.1)
(10.7.2)
Similarly, When rank (A) = m, the normal equations of the se ond kind
AAT y = b;
x = AT y;
(10.7.3)
give the
onditions for the solution of the minimum norm problem
min kxk2 ;
Ax = b:
(10.7.4)
In parti
ular, if A is nonsingular, then both systems are symmetri
positive definite with solution x = A 1 b. Hen
e a natural extension of iterative methods for
symmetri
positive denite systems to general nonsingular, non-symmetri
linear
systems Ax = b is to to apply them to the normal equations of rst or se
ond kind.
Be
ause of the relation (AT A) = (AAT ) = 2 (A), the
ondition number is
squared
ompared to the original system Ax = b. From the estimate (10.6.12) we
note that this
an lead to a substantial de
rease in the rate of
onvergen
e.
8k:
k (G) = 1 k2 ; k = 1; : : : ; n;
where k are the singular values of A. From this it
an be shown that Ri
hardson's
method
onverges to the least squares solution x = Ay b if
x(0) 2 R(AT );
(10.7.5)
\dqbjV
2002/
page 3
10.7.
Nonsymmetri Problems
349
T
k = 1; 2; : : : ;
is omputed from
(10.7.6)
(10.7.7)
(10.7.8)
The Gauss{Seidel method is a spe
ial
ase of the following
lass of residual
redu
ing methods. Let pj 62 N (A), j = 1; 2; : : :, be a sequen
e of nonzero n-ve
tors
and
ompute a sequen
e of approximations of the form
x(j+1) = x(j) + j pj ;
(10.7.9)
r(j) = b Az (j) ;
(10.7.10)
j aj ;
(10.7.11)
Note that in the j th minor step only the j th
olumn of A is a
essed. and that it
an be implemented without forming the matrix ATA expli
itly. In
ontrast to the
\dqbjV
2002/
page 3
Chapter 10.
350
Iterative Methods
Ja
obi method the Gauss{Seidel method is not symmetrizable and the ordering of
the
olumns of A will in
uen
e the
onvergen
e.
The Ja
obi method has the advantage over Gauss{Seidel's method that it is
more easily adapted to parallel
omputation, sin
e (10.7.8) just requires a matrixve
tor multipli
ation. Further, it does not require A to be stored (or generated)
olumnwise, sin
e produ
ts of the form Ax and AT r
an
onveniently be
omputed
also if A
an only be a
essed by rows. In this
ase, if aT1 ; : : : ; aTm are the rows of
A, then we have
(Ax)i = aTi x; i = 1; : : : ; n; AT r =
m
X
i=1
ai r i :
That is, for Ax we use an inner produ
t formulation, and for AT r, an outer produ
t
formulation.
The su
essive overrelaxation (SOR) method for the normal equations
ATAx = AT b is obtained by introdu
ing an relaxation parameter ! in the Gauss{
Seidel method (10.7.12),
j aj ;
(10.7.12)
The SOR method always
onverges when ATA is positive denite and ! satises
0 < ! < 2. The SOR shares with the Gauss{Seidel the advantage of simpli
ity and
small storage requirements.
The Gauss{Seidel method for solving the normal equations of se
ond kind
an
also be implemented without forming AAT . We dene a
lass of error redu
ing
methods as follows: Let pi 62 N (A), i = 1; 2; : : :, be a sequen
e of nonzero m-ve
tors
and
ompute approximations of the form
x(j+1) = x(j) + j AT pj ;
(10.7.13)
(10.7.14)
\dqbjV
2002/
page 3
10.7.
Nonsymmetri Problems
351
and with x(j) = AT y(j) and x(j+1) = x(j) + AT y(j) we re
over (10.7.14). This
shows that if we take x(0) = Ay(0) , then for an arbitrary y(0) (10.7.14) is equivalent
to the Gauss{Seidel method for (10.7.3).
The SOR method applied to the normal equations of the se
ond kind
an be
obtained by introdu
ing an a
eleration parameter !, i.e.
(10.7.15)
Algorithm 10.7.1
CGLS.
r0 = b Ax0 ; p0 = s0 = AT r0 ;
for k = 0; 1; : : : while krk k2 > do
qk = Apk ;
k = ksk k22 =kqk k22 ;
xk+1 = xk + k pk ;
rk+1 = rk k qk ;
sk+1 = AT rk+1 ;
k = ksk+1 k22 =ksk k22;
pk+1 = sk+1 + k pk ;
end
Note that it is important for the stability that the residuals rk = b Axk
and not the residuals sk = AT (b Axk ) are re
urred. The method obtained by
applying CG to the normal equations of the se
ond kind is also known as Craig's
Method. This method
an only be used for
onsistent problems, i.e., when b 2
R(A). It
an also be used to
ompute the (unique) minimum norm solution of an
underdetermined system, min kxk2 , subje
t to Ax = b, where A 2 Rmn , m < n.
Craig's method (CGNE)
an be implemented as follows:
Algorithm 10.7.2
CGNE
r0 = b Ax0 ; p0 = AT r0 ;
for k = 0; 1; : : : while krk k2 > do
k = krk k22 =kpk k22 ;
\dqbjV
2002/
page 3
Chapter 10.
352
end
Iterative Methods
xk+1 = xk + k pk ;
rk+1 = rk k Apk ;
k = krk+1 k22 =krk k22 ;
pk+1 = AT rk+1 + k pk ;
Both CGLS and CGNE will generate iterates in the shifted Krylov subspa e,
xk 2 x0 + Kk (AT r0 ; AT A):
From the minimization property we have for the iterates in CGLS
k
kx xk kAT A = kr rk k2 < 2 + 11 kr0 k2;
where = (A). Similarly for CGNE we have
k
ky yk kAAT = kx xk k2 < 2 + 11 kx x0 k2:
1 u1 = b Ax0 ;
1 v1 = AT u1;
(10.7.16)
(10.7.17)
Vk = (v1 ; : : : ; vk );
\dqbjV
2002/
page 3
10.7.
Nonsymmetri Problems
353
1
B 2 2
B
.
Bk = B
3 . .
B
B
...
k
k+1
C
C
C
C
C
A
2 R(k+1)k :
(10.7.18)
1 Uk+1 e1 = b;
(10.7.19)
AVk = Uk+1 Bk ;
(10.7.20)
tk+1 = 1 e1 Bk yk :
(10.7.22)
Using the orthogonality of Uk+1 and Vk , whi
h holds in exa
t arithmeti
, it follows
that kb Axk k2 is minimized over all xk 2 span(Vk ) by taking yk to be the solution
to the least squares problem
min
kBk yk
y
k
1 e1k2 :
(10.7.23)
This forms the basis for the algorithm LSQR. Note the spe
ial form of the righthand side, whi
h holds be
ause the starting ve
tor was taken as b. Now xk = Vk yk
solves minxk 2Kk kAx bk2, where Kk = Kk (AT b; ATA). Thus mathemati
ally LSQR
generates the same sequen
e of approximations as Algorithm 10.7.3 CGLS.
To solve (10.7.23) stably we need the QR fa
torization QTk Bk = Rk . This
an
be
omputed by premultiplying Bk by a sequen
e Givens transformations, whi
h
are also applied to the right hand side e1 ,
Rk yk = e1 ;
kb Axk k2 = jk j:
\dqbjV
2002/
page 3
Chapter 10.
354
Iterative Methods
Note that the whole ve
tor yk diers from yk 1 . An updating formula for xk
an be derived using an idea due to Paige and Saunders. With Wk = Vk Rk 1 we
an write
xk = x0 + Vk yk = x0 + 1 Vk Rk 1 dk = x0 + 1 Wk dk
= x0 + 1 (Wk 1 ; wk ) dk 1 = xk 1 + 1 k wk :
k
(10.7.24)
(10.7.25)
AT Uk = Vk LTk :
(10.7.26)
Lk yk = 1 e1 ;
xk = Vk zk :
(10.7.27)
Using (10.7.26) and (10.7.16) it follows that the residual ve tor satises
6 0 then k =
6 0. Hen
e the
and hen
e UkT rk = 0. It
an be shown that if rk 1 =
ve
tors yk and xk
an re
ursively be formed using
k =
k
;
k k 1
xk = xk 1 + k vk :
\dqbjV
2002/
page 3
10.7.
Nonsymmetri Problems
355
and the ve
tors v1 ; : : : ; vk form an orthogonal basis in the Krylov subspa
e Kk (r0 ; A).
Further we have
AVk = Vk Hk + hk+1;k vk+1 eTk :
The Arnoldi pro
ess will break down at step j if and only if the ve
tor zj vanishes.
When this happens we have hk+1;k = 0 and hen
e AVk = Vk Hk , and Vk is a right
invariant subspa
e of A.
In the generalized minimum residual (GMRES) method we seek at step j = k
an approximate solution of the form
xk = x0 + Vk yk 2 x0 + Kk (r0 ; A):
If we write (10.7.29) as AVk = Vk+1 H k , where
0
h11 h12
B h21 h22
B
...
H k = B
B
...
hk;k 1
h 1k
h2k
..
.
hkk
(10.7.28)
1
C
C
C
C
A
2 R(k+1)k :
hk+1;k
then the residual rk = b Axk
an be expressed as
rk = r0 AVk yk = 1 v1 Vk+1 H k yk = Vk+1 (1 e1 H k yk ):
(10.7.29)
\dqbjV
2002/
page 3
Chapter 10.
356
Iterative Methods
Sin
e (in exa
t arithmeti
) Vk+1 has orthogonal
olumns, krk k2 is minimized by
taking yk to be the solution of the least squares problem
k1e1 H k yk k2 :
(10.7.30)
min
y
k
QTk (H k
(10.7.31)
where Gj+1;j is
hosen to zero the subdiagonal element hj+1;j . Then the solution
to (10.7.30) is obtained from the triangular system
Rk yk = 1 dk ;
Sin
e H k 1 determines the rst k 1 Givens rotations and H k is obtained
from H k 1 by adding the kth
olumn, it is possible to update the QR fa
torization
(10.7.31) at ea
h step of the Arnoldi pro
ess. To derive the updating formulas for
step j = k we write
T
k 1
R
Q
0
H
h
k
k
T
k
1
Qk Hk = Gk;k+1
0
hk+1;k = 0 ;
0
1
where, sin
e Gk;k+1 will not ae
t Rk 1 ,
0
h1k 1
hk = ... A :
hkk
The rotation Gk;k+1 and the kth
olumn in Rk is determined from
Rk = Rk0 1 rk
1 ;
k
(10.7.32)
rk 1
k 1
Gk;k+1 k A =
k A :
hk+1;k
0
(10.7.33)
krk k2 = 1 k :
(10.7.34)
(Note that QTk , whi
h is a full upper Hessenberg matrix, should not be expli
itly
formed.)
Pro
eeding similarly with the right hand side, we have
QTk e1 = Gk;k+1
QTk
dk 1
dk 1
1 e1 = Gk;k+1 k 1 A = k A dk : (10.7.35)
0
k
0
k
(Note that the dierent dimensions of the unit ve
tors e1 above is not indi
ated in
the notation.) The rst k 1 elements in QTk e1 are not ae
ted.
\dqbjV
2002/
page 3
10.7.
Nonsymmetri Problems
357
The new approximate solution
ould be obtained from (10.7.28), but note
that the whole ve
tor yk diers from yk 1 . Sin
e krk k2 = jk j is available without
forming xk , this expensive operation
an be delayed until k is small enough.
An updating formula for xk
an be derived using the same tri
k as in (10.7.24).
There we have set Wk Rk = Vk , whi
h
an be written
wk = (vk
Wk 1 rk 1 )= k ;
(10.7.36)
whi
h determines wk . Note that if this formula is used we only need the last
olumn
of the matrix Rk . (We now need to save Wk but not Rk .)
The steps in the resulting GMRES algorithm
an now be summarized as follows:
1. Obtain last
olumn of H k from the Arnoldi pro
ess and apply old rotations
gk = Gk 1;k G12 hk .
2. Determine rotation Gk;k+1 and new
olumn in Rk , i.e.,
k 1 and
k a
ording
to (10.7.33). This also determines k and k = krk k2 .
3. Compute the new
olumn wk using (10.7.35) and xk from (10.7.24).
In exa
t arithmeti
GMRES
annot break down and will give the exa
t solution in at most n steps. If at some step we have v~k+1 = 0 then hk+1;k = 0, and
the ve
tor vk+1
annot be
onstru
ted. However, then it is seen that AVk = Vk Hk ,
where Hk is the matrix
onsisting of the rst k rows of H k . This means that
xk = x0 + Vk yk , with yk = Hk 1 1 e1 is the exa
t solution. It follows that the
GMRES terminates in at most n steps.
Suppose that the matrix A is diagonalizable, A = X X 1, where =
diag(i ). Then, using the property that the GMRES approximations minimize
the Eu
lidian norm of the residual rk = b Axk in the Krylov subspa
e Kk (r0 ; A),
it
an be shown that
jjrk jj2 (X ) min max jq ( )j;
(10.7.37)
jjr0 jj2 2 qk i=1;2;:::;n k i
where qk is a polynomial of degree k and qk (0) = 1. The proof is similar to the
onvergen
e proof for the
onjugate gradient method in Se
tion 10.6.2. This results
looks similar to that obtained for the symmetri
ase. However, be
ause of the
fa
tor 2 (X ) in (10.7.37) the rate of
onvergen
e
an no longer be dedu
ed from the
spe
trum fi g of the matrix A alone. In the spe
ial
ase that A is normal we have
2 (X ) = 1, and the
onvergen
e
an be determined via the
omplex approximation
problem
min
max jqk (i )j;
qk (0) = 1:
q
k i=1;2;:::;n
\dqbjV
2002/
page 3
Chapter 10.
358
Iterative Methods
Be
ause
omplex approximation problems are harder than real ones no simple results are available even for this spe
ial
ase.
If GMRES is applied to a real symmetri
system, it
an be implemented with
three-term re
urren
es, whi
h avoids the ne
essity to store all basis ve
tors vj . This
leads to the method MINRES by Paige and Saunders mentioned in Se
tion 10.7.5.
(However, a version of GMRES, whi
h uses the three-term re
urren
e but stores
all the Lan
zos ve
tors, has been shown to be less ae
ted by rounding errors than
MINRES.)
In pra
ti
e the number of steps taken by GMRES must be limited be
ause of
limited memory and
omputational
omplexity. This
an be a
hieved by restarting GMRES after ea
h m iterations, and we denote the
orresponding algorithm
GMRES(m). Note that storage requirements grow linearly with m, and the
omputational requirement quadrati
ally with m. In pra
ti
e typi
ally m 2 [5; 20.
GMRES(m)
an not break down in exa
t arithmeti
before the true solution has
been produ
ed. Unfortunately it may never
onverge for m < n.
H := (I
lj eTj )H (I + lj eTj );
j = 1; : : : ; n 1:
In this step row pivoting
an not be used, sin
e this would destroy the lower Hessenberg stru
ture. As a
onsequen
e, the redu
tion will fail if a zero pivot element
is en
ountered. Hen
e the stability of this pro
ess
annot be guaranteed, and this
redu
tion is in general not advisable.
Assume that A
an be redu
ed to tridiagonal form
1
1 2
...
C
B
C
B
2 2
C
B
T
.
.
.
C;
B
.. .. ..
W AV = Tn = B
C
C
B
... ...
n A
n n
0
(10.7.38)
\dqbjV
2002/
page 3
10.7.
Nonsymmetri Problems
359
Algorithm 10.7.4
The Lan
zos Bi-orthogonalization Pro
ess Let v1 and w1 be two ve
tors su
h
that w1T v1 = 1. The following algorithm
omputes in exa
t arithmeti
after k
steps a symmetri
tridiagonal matrix Tk = trid (
j ; j ; j+1 ) and two matri
es Wk
and Vk with bi-orthogonal
olumns spanning the Krylov subspa
es Kk (v1 ; A) and
Kk (w1 ; AT ):
w0 = v0 = 0;
1 =
1 = 0;
for j = 1; 2; : : :
j = wjT Avj ;
vj+1 = Avj j vj j vj 1 ;
wj+1 = AT wj j wj j wj 1 ;
j+1 = jwjT+1 vj+1 j1=2 ;
if j+1 = 0 then exit;
j+1 = (wjT+1 vj+1 )=j+1 ;
vj+1 = vj+1 =j+1 ;
wj+1 = wj+1 =j+1 ;
end
Note that if A = AT , w1 = v1 , and we take k =
k , then the two sequen
es
generated will be identi
al. The pro
ess then is equivalent to the symmetri
Lan
zos
pro
ess.
If we denote
Vk = (v1 ; : : : ; vk );
Wk = (w1 ; : : : ; wk );
\dqbjV
2002/
page 3
Chapter 10.
360
Iterative Methods
then we have WkT AVk = Tk , and the re
urren
es in this pro
ess
an be written in
matrix form as
(10.7.41)
(10.7.42)
There are two
ases when the above algorithm breaks down. The rst o
urs
when either v~k+1 or w~k+1 (or both) is null. In this
ase it follows from (10.7.41){
(10.7.42) that an invariant subspa
e has been found; if vk+1 = 0, then AVk = Vk Tk
and R(Vk ) is an A-invariant subspa
e. If wk+1 = 0, then AT Wk = Wk TkT and
R(Wk ) is an AT -invariant subspa
e. This is
alled regular termination. The se
ond
ase,
alled serious breakdown by Wilkinson, o
urs when w~kT v~k = 0, with neither
v~k+1 nor w~k+1 null.
xk
x0 = Vk yk 2 Kk (r0 ; A):
rk = b Axk = 1 v1 AVk yk = 1 v1
Vk Tk yk
AVk yk ) = 1 e1 Tk yk :
Hen
e, if the matrix Tk is nonsingular yk is determined by solving the tridiagonal system Tk yk = 1 e1 . If A is symmetri
, this method be
omes the SYMMLQ
method, see Se
tion 10.7.5. We remark again that this method
an break down
without produ
ing a good approximate solution to Ax = b. In
ase of a serious
breakdown, it is ne
essary to restart from the beginning with a new starting ve
tor
r0 . As in SYMMLQ the matrix Tk may be singular for some k.
The nonsymmetri
Lan
zos pro
ess is the basis for several iterative methods
for nonsymmetri
systems. The method
an be written in a form more like the
onjugate gradient algorithm, whi
h is
alled the bi-
onjugate gradient or BCG
method, see, e.g., Barret et al. [pp. 21{22, 1994. This form is obtained from the
Lan
zos formulation by
omputing the LU de
omposition of Tk . Hen
e the BCG
method
annot pro
eed when a singular Tk o
urs, and this is an additional
ause
for breakdown.
There are few results known about the
onvergen
e of the BCG and related
methods. In pra
ti
e it has been observed that sometimes
onvergen
e
an be as
fast as for GMRES. However, often the
onvergen
e behavior
an be very irregular,
\dqbjV
2002/7
page 3
10.7.
Nonsymmetri Problems
361
(10.7.43)
\dqbjV
2002/
page 3
Chapter 10.
362
Iterative Methods
Review Questions
1.
2.
3.
What optimality property does the residual ve
tors rk = b Axk in the GMRES
method satisfy. In what subspa
e does the ve
tor rk r0 lie?
In Lan
zos bi-orthogonalization bases for two dierent Krylov subspa
es are
omputed. Whi
h subspa
es and what property has these bases?
(a) The bi-
onjugate gradient (BCG) method is based on the redu
tion of A 2 Cnn
to tridiagonal form by a general similarity transformation.
(b) What are the main advantages and drawba
ks of the BCG method
ompared to
GMRES.
) How are the approximations xk dened in QMR?
Problems
1.
2.
Derive Algorithms CGLS and CGNE by applying the
onjugate gradient algorithm
to the normal equations AT Ax = AT b and AAT y = b, x = AT y, respe
tively.
Consider using GMRES to solve the system Ax = b, where
A=
0 1 ;
1 0
b = 11 ;
using x0 = 0. Show that x1 = 0, and that therefore GMRES(1) will never produ
e
a solution.
10.8
The term \pre
onditioning" dates ba
k to Turing in 1948, and is in general taken
to mean the transformation of a problem to a form that
an more e
iently be
solved. In order to be ee
tive iterative methods must usually be
ombined with a
(nonsingular) pre
onditioning matrix M , whi
h in some sense is an approximation
to A. The original linear system Ax = b is then transformed by
onsidering the
left-pre
onditioned system
M 1 Ax = M 1 b:
(10.8.1)
AM 1 u = b;
u = Mx:
(10.8.2)
The idea is to
hoose M so that the rate of
onvergen
e of the iterative method is
improved. Note that the produ
t M 1 A (or AM 1 ) should never be formed. The
pre
onditioned iteration is instead implemented by forming matrix ve
tor produ
ts
with A and M 1 separately. Sin
e forming u = M 1v for an arbitrary ve
tor v is
equivalent to solving a linear system Mu = v, the inverse M 1 is not needed either.
\dqbjV
2002/
page 3
10.8.
363
Often the rate of
onvergen
e depends on the spe
trum of the transformed
matrix. Sin
e the eigenvalues of M 1 A and AM 1 are the same, we see that the
main dieren
e between these two approa
hes is that the a
tual residual norm is
available in the right-pre
onditioned
ase.
If A is symmetri
, positive denite, the pre
onditioned system should also
have this property. In this
ase, it is natural to
onsider a split pre
onditioner.
Let M = LLT where L is the Cholesky fa
tor of M . Then we
onsider the pre
onditioned linear system
A~ = L 1AL T ;
x~ = LT x;
~b = L 1 b;
and the spe
trum of A~ is real. Note that the spe
trum of A = C 1 AC
same as for L T L 1A = M 1 A.
(10.8.3)
T
is the
M = LLT ;
(10.8.4)
Algorithm 10.8.1
Pre
onditioned Conjugate Gradient Method
r0 = b Ax0 ; p0 = z0 = M 1 r0 ;
for k = 0; 1; 2; : : : ; while krk k2 > do
w = Apk ;
k = (zk ; rk )=(pk ; Apk );
xk+1 = xk + k pk ;
\dqbjV
2002/
page 3
Chapter 10.
364
Iterative Methods
rk+1 = rk k Apk ;
zk+1 = M 1 rk+1 ;
k = (zk+1 ; rk+1 )=(zk ; rk );
pk+1 = zk+1 + k pk ;
end
A surprising and important feature of this version is that it depends only on
the symmetri
positive denite matrix M = LLT .
The rate of
onvergen
e in the A~-norm depends on (A~), see (10.6.12). Note,
however, that
T (~
x
x~k ) = kx xk k2A;
so the rate of
onvergen
e in A-norm of the error in x also depends on (A~). The
pre
onditioned
onjugate gradient method will have rapid
onvergen
e if one or
both of the following
onditions are satised:
i. M 1A to have small
ondition number, or
ii. M 1A to have only few distin
t eigenvalues.
For symmetri
indenite systems SYMMLQ
an be
ombined with a positive
denite pre
onditioner M . To solve the symmetri
indenite system Ax = b the
pre
onditioner is regarded to have the form M = LLT and SYMMLQ impli
itly
applied to the system
L 1 AL T w = L 1 b:
The algorithm a
umulates approximations to the solution x = L T w, without approximating w. A MATLAB implementation of this algorithm, whi
h only requires
solves with M , is given by Gill et al. [5.
bk2 ; Sx = y:
(Note that for a non
onsistent system Ax = b a left pre
onditioner would
hange
the problem.) If we apply CGLS to this problem and formulate the algorithm in
terms of the original variables x, we obtain the following algorithm:
\dqbjV
2002/
page 3
10.8.
365
Algorithm 10.8.2
Pre
onditioned CGLS.
r0 = b Ax0 ; p0 = s0 = S T (AT r0 );
for k = 0; 1; : : : while krk k2 > do
tk = S 1 pk ;
qk = Atk ;
k = ksk k22=kqk k22 ;
xk+1 = xk + k tk ;
rk+1 = rk k qk ;
sk+1 = S T (AT rk+1 );
k = ksk+1 k22=ksk k22 ;
pk+1 = sk+1 + k pk ;
end
For solving a
onsistent underdetermined systems we
an derive a pre
onditioned version of CGNE. Here it is natural to use a left pre
onditioner S , and apply
CGNE to the problem
min kxk2 ;
S 1 Ax = S 1 b;
i.e., the residual ve
tors are transformed. If the algorithm is formulated in terms of
the original residuals, the following algorithm results:
Algorithm 10.8.3
Pre
onditioned CGNE
r0 = b Ax0 ; z0 = S 1 r0 ; p0 = AT (S
for k = 0; 1; : : : while krk k2 > do
k = kzk k22 =kpk k22;
xk+1 = xk + k pk ;
rk+1 = rk k Apk ;
zk+1 = S 1 rk+1 ;
k = kzk+1 k22 =kzk k22 ;
pk+1 = AT (S T zk+1 ) + k pk ;
end
Tz
0 );
\dqbjV
2002/
page 3
Chapter 10.
366
Iterative Methods
r(k) k2 , where
x(k) = x(k) + Kk ;
Kk = (S 1 S T AT A; S 1 S T AT r0 ):
Algorithm PCCGNE minimizes the error fun
tional kx^ x(k) k2 , over the Krylov
subspa e
x(k) = x(k) + Kk ;
Kk = (AT S T S 1A; AT S T S 1 r0 ):
The rate of
onvergen
e for PCGTLS depends on (AS 1 ), and for PCCGNE on
(S 1 A) = (AT S T ).
r0 = M 1 (b Ax0 ); 1 = kr0 k2 ; v1 = r0 =1 ;
and dene
zj = M 1 Avj ; j = 1; 2; : : : ; k:
All
omputed residual ve
tors will be pre
onditioned residuals M 1 (b Axm). This
is a disadvantage sin
e most stopping
riteria depend are based on the a
tual residuals rm = b Axm . In this left pre
onditioned version the transformed residual
norm kM 1(b Ax)k2 will be minimized among all ve
tors of the form
x0 + Km (M 1 r0 ; M 1 A):
(10.8.5)
In the right pre
onditioned version of GMRES the a
tual residual ve
tors are
used, but the variables are transformed a
ording to u = Mx (x = M 1 u). The
right pre
onditioned algorithm
an easily be modied to give the untransformed
solution. We have
zj = AM 1 vj ; j = 1; 2; : : : ; k:
The kth approximation is xk = x0 + M 1Vk yk , where yk solves
min k1e1 H k yk k2 :
yk
xk = xk 1 + 1 k Mk 1 wk ; wk = Rk yk ;
\dqbjV
2002/
page 3
10.9.
Pre onditioners
367
see (10.7.24).
In the right pre
onditioned version the residual norm kb AM 1 uk2 will be
minimized among all ve
tors of the form u0 + Km (r0 ; AM 1 ). However, this is
equivalent to minimizing kb AM 1 uk2 among all ve
tors of the form
x0 + M 1 Km (r0 ; AM 1 ):
(10.8.6)
Somewhat surprisingly the two ane subspa
es (10.8.5) and (10.8.6) are the same!
The j th ve
tor in the two Krylov subspa
es are wj = (M 1 A)j M 1 r0 and w~j =
M 1 (AM 1 )j r0 . By a simple indu
tion proof it
an be shown that M 1 (AM 1 )j =
(M 1 A)j M 1 and so w~j = wj , j 0. Hen
e the left and right pre
onditioned
versions generate approximations in the same Krylov subspa
es, and they dier
only with respe
t to whi
h error norm is minimized.
For the
ase when A is diagonalizable, A = X X 1, where = diag(i ) we
proved the error estimate
in (10.8.7) the rate of
onvergen
e
an no longer be dedu
ed from the spe
trum
fi g of the matrix A alone. Sin
e the spe
trum of M 1 A and AM 1 are the same
we
an expe
t that the
onvergen
e behavior will be similar when if A is
lose to
normal.
10.9
Pre onditioners
\dqbjV
2002/
page 3
Chapter 10.
368
Iterative Methods
k = 0; 1; : : : ;
(10.9.1)
B = M 1N = I
M 1 A:
MJ = DA ; and MGS = DA LA :
If A is symmetri
positive denite then MJ = DA > 0 and symmetri
. However,
MGS is lower triangular and unsymmetri
.
The simplest
hoi
e related to this splitting is to take M = DA . This
orresponds to a diagonal s
aling of the rows of A, su
h that the s
aled matrix
M 1A = DA 1 A has a unit diagonal. For s.p.d. matri
es symmetry
an be preserved by using a split pre
onditioner with L = LT = DA1=2 . In this
ase it
an be
shown that this is
lose to the optimal diagonal pre
onditioning.
Although diagonal s
aling may give only a modest improvement in the rate of
onvergen
e it is trivial to implement and therefore re
ommended even if no other
pre
onditioning is
arried out.
In Se
tion 10.3.2 it was shown that for a symmetri
matrix A the SSOR iteration method
orresponds to a splitting with the matrix
1
D !LA DA 1 DA !UA :
MSSOR =
!(2 !) A
Sin
e MSSOR is given in the form of an LDLT fa
torization it is easy to solve linear
systems involving this pre
onditioner. It also has the same sparsity as the original
matrix A. For 0 < ! < 2, if A is s.p.d. so is MSSOR .
The performan
e of the SSOR splitting turns out to be fairly insensitive to
the
hoi
e of !. For systems arising from se
ond order boundary values problems,
\dqbjV
2002/
page 3
10.9.
Pre onditioners
369
e.g., the model problem studied previously, the original
ondition number (A) =
O(h 2 )
an be redu
ed to (M 1 A) = O(h 1 ). Taking ! = 1 is often
lose to
optimal. This
orresponds to the symmetri
Gauss{Seidel (SGS) pre
onditioner
MSGS = DA LA DA 1 DA UA :
All the above pre
onditioners satisfy
onditions (ii) and (iii). The appli
ation
of the pre
onditioners involves only triangular solves and multipli
ation with a
diagonal matrix. They are all dened in terms of elements of the original matrix
A, and hen
e do not require extra storage. However, they may not be very ee
tive
with respe
t to the
ondition (i).
A LU = DA LA UA
LA DA 1 DA UA = LADA 1 UA :
P Pn f(i; j ) j 1 i; j ng;
where the diagonal positions always are in
luded in P . For example, we
ould take
P = PA , the set of nonzero elements in A.
Algorithm 10.9.1
In
omplete LU Fa
torization
for k = 1; : : : ; n 1
for i = k + 1; : : : ; n
if (i; k) 2 P lik = aik =akk ;
for j = k + 1; : : : ; n
if (k; j ) 2 P aij = aij lik akj ;
\dqbjV
2002/
page 3
Chapter 10.
370
end
end
Iterative Methods
end
The elimination
onsists of n 1 steps. In the kth step we rst subtra
t from
the
urrent matrix elements with indi
es (i; k) and (k; i) 2= P and pla
e in a defe
t
matrix Rk . We then
arry out the kthe step of Gaussian elimination on the so
modied matrix. This pro
ess
an be expressed as follows. Let A0 = A and
Aek = Ak 1 + Rk ;
Ak = Lk Afk ; k = 1; : : : ; n 1:
An 1 = Ln 1Aen 1 = Ln 1 An 2 + Ln 1 Rn 1
= Ln 1Ln 2 An 3 + Ln 1 Ln 2Rn 2 + Ln 1 Rn 1
= Ln 1Ln 2 L1 A + Ln 1 Ln 2 L1 R1
+ + Ln 1 Ln 2 Rn 2 + Ln 1 Rn 1 :
We further noti
e that sin
e the rst m 1 rows of Rm are zero it follows that
Lk Rm = Rm , if k < m. Then by
ombining the above equations we nd LU = A+R,
where
U = An 1 ;
L = (Ln 1 Ln 2 L1) 1 ;
R = R1 + R2 + Rn 1 :
lik akj ;
In pra
ti
e the matrix A is sparse and the algorithm should be spe
ialized to take
this into a
ount. In parti
ular, a version where A is pro
essed a row at a time is
more
onvenient for general sparse matri
es. Su
h an algorithm
an be derived by
inter
hanging the k and i loops in Algorithm 10.9.2.
Example 10.9.1. For the model problem using a ve-point approximation the
non-zero stru
ture of the resulting matrix is given by
PA = f(i; j ) j ji j j = n; 1; 0; 1; ng:
Let us write A = LU + R, where
L=L
+ L 1 + L0 ;
U = U0 + U1 + Un ;
\dqbjV
2002/
page 3
10.9.
Pre onditioners
371
where L k (and Uk ) denote matri
es with nonzero elements only in the k-th lower
(upper) diagonal. By the rule for multipli
ation by diagonals (see Problem 6.1.6),
Ak Bl = Ck+l ; if k + l n 1;
we
an form the produ
t
0 for i 6= j , A is
The following result was proved by Meijerink and van der Vorst [13, 1977.
Theorem 10.9.3.
If A is an M -matrix, there exists for every set P su
h that (i; j ) 2 P for
i = j , uniquely dened lower and upper triangular matri
es L and U with lij = 0
or uij = 0 if (i; j ) 62 P , su
h that the splitting A = LU R is regular.
In
ase A is s.p.d., we dene similarly an in
omplete Cholesky fa
torization. Here the nonzero set P is assumed to be symmetri
, i.e., if (i; j ) 2 P then
also (j; i) 2 P . Positive deniteness of A alone is not su
ient to guarantee the
existen
e of an in
omplete Cholesky fa
torization. This is be
ause zero elements
may o
ur on the diagonal during the fa
torization.
For the
ase when A is a symmetri
M -matrix, a variant of the above theorem
guarantees the existen
e for ea
h symmetri
set P su
h that (i; j ) 2 P for i = j ,
a uniquely dened lower triangular matrix L, with lij = 0 if (i; j ) 62 P su
h that
the splitting A = LLT R is regular. In parti
ular the matrix arising from the
model problem is a symmetri
M -matrix. A symmetri
M -matrix is also
alled a
Stieltjes matrix.
An implementation of the in
omplete Cholesky fa
torization in the general
ase is given below.
\dqbjV
2002/
page 3
Chapter 10.
372
Iterative Methods
Algorithm 10.9.2
In
omplete Cholesky Fa
torization
for j = 1; 2; : : : ; n
j 1
X
ljj = ajj
k=1
2
ljk
1=2
for i = j + 1; : : : ; n
if (i; j ) 62 P then lij = 0 else
j 1
X
lij = aij
end
k=1
end
lik ljk
D1 AT2
B A2 D2 AT
3
B
...
B
A=B
A3
B
...
...
...
C
C
C
C
C
AT A
= DA
LA LTA ;
(10.9.2)
AN DN
with square diagonal blo
ks Di . For the model problem with the natural ordering
of mesh points we obtain this form with Ai = I , Di = tridiag( 1 4 1). If
systems with Di
an be solved e
iently a simple
hoi
e of pre
onditioner is the
blo
k diagonal pre
onditioner
M = diag (D1 ; D2 ; : : : ; DN ):
The
ase N = 2 is of spe
ial interest. For the system
D1 AT2
A 2 D2
x1 = b1 ;
x2
b2
(10.9.3)
the blo
k diagonal pre
onditioner gives a pre
onditioned matrix of the form
1 T
M 1A = D I1 A D1 I A2 :
2 2
\dqbjV
2002/
page 3
10.9.
Pre onditioners
373
Note that this matrix is of the form (10.3.4) and therefore has property A. Suppose the
onjugate gradient method is used with this pre
onditioner and a starting
approximation x(0)
1 . If we set
(0)
1
x(0)
2 = D2 (b2 A2 x1 );
(0)
then the
orresponding residual r2(0) = b2 D2 x(0)
1 A2 x2 = 0. It
an be shown that
in the following steps of the
onjugate gradient method we will alternately have
r2(2k) = 0;
r1(2k+1) = 0;
k = 0; 1; 2; : : : :
Sx2 = b2 A2 D1 1 b1 ;
S = D2
A2 D1 1 AT2 ;
(10.9.4)
r2 = (b2 D2 x2 ) A2 D1 1 (b1
AT2 x2 ):
A = ( + LA) 1 ( + LTA );
i = Di
1 = D 1 ;
i = Di
Ai i 11 ATi ; i = 2; : : : ; N:
Ai i 11 ATi ; i = 2; : : : ; N:
(10.9.5)
M = ( + LA) 1 ( + LTA);
= diag (1 ; : : : ; n ):
\dqbjV
2002/
page 3
374
Chapter 10.
Iterative Methods
C;
2
B
C
...
A
n 1
n 1
n
It
an be shown that the elements of the inverse T 1 = L T L de
rease stri
tly away
from the diagonal. This suggests that the matrix L 1, whi
h is lower triangular and
dense, is approximated by a banded lower triangular matrix L 1 (p), taking only
the rst p + 1 lower diagonals of the exa
t L 1. Note that elements of the matrix
L 1
0
1
1=
1
1=
2
B 1
C
B
C
.
B
C
..
L 1 = B 1
C;
2
B .
C
...
...
.
A
1=
n 1
.
n 2 n 1 1=
n
an be
omputed diagonal by diagonal. For example, we have
i = 2; : : : ; n 1:
i = i ;
i
i+1
For p = 0 we get a diagonal approximate inverse. For p = 1 the approximate
Cholesky fa
tor L 1(1) is lower bidiagonal, and the approximate inverse L T (1)L 1 (1)
a tridiagonal matrix. Sin
e we have assumed that Ai are diagonal matri
es, the approximations i generated by (10.9.5) will in this
ase be tridiagonal.
\dqbjV
2002/
page 3
10.9.
Pre onditioners
375
make strong assumptions about the regularity of the system. It applies only to dis
retizations of problems with
onstant
oe
ients on a re
tangular domain. However, if we have variable
oe
ients the fast solver may be used as a pre
onditioner
in the
onjugate gradient method. Similarly, problems on an irregular domain may
be embedded in a re
tangular domain, and again we may use a pre
onditioner based
on a fast solver.
Consider a linear system Ax = b where A has the blo
k-tridiagonal form
0
1
B T
0 1
0 1
b1
x1
BT B T
C
B
C
b2 C
x
C
B
B
2C
.
C
B . C;
A=B
(10.9.6)
T B ..
;
b
=
B
C; x = B
.
A
.. A
..
B
C
.
.
.
.
.
. TA
bq
xq
T B
where B; T 2 Rpp . We assume that the matri
es B and T
ommute, i.e., BT =
T B . Then it follows that B and T have a
ommon eigensystem, and we let
QT BQ = = diag (i );
QT T Q =
= diag (!i ):
B
C
B
C
...
B
C
C =B
C;
B
C
... ...
T
and xj = Qzj , yj = Q bj , j = 1; : : : ; q.
For the model problem with Diri
hlet boundary
onditions the eigenvalues and
eigenve
tors are known. Furthermore, the multipli
ation of a ve
tor by Q and QT
is e
iently obtained by the Fast Fourier Transform algorithm (see Se
tion 9.6.3).
One fast algorithm then is as follows:
1. Compute
yj = QT bj ;
j = 1; : : : ; q:
2. Rearrange taking one element from ea
h ve
tors yj ,
where
= y^i ;
i = 1; : : : ; p;
1
i !i
C
B !i i !i
C
B
...
C
B
!i i
C ; i = 1; : : : ; p:
i=B
C
B
... ...
!i A
!i i
0
\dqbjV
2002/7
page 3
Chapter 10.
376
Iterative Methods
xj = Qzj ;
j = 1; : : : ; q:
The fast Fourier transforms in steps 1 and 3 takes O(n2 log n) operations.
Solving the tridiagonal systems in step 2 only takes O(n2 ) operations, and hen
e
for this step Gaussian elimination is superior to FFT. In all this algorithm uses only
O(n2 log n) operations to
ompute the n2 unknown
omponents of x.
Review Questions
1.
2.
3.
What is meant by pre
onditioning of a linear system Ax = b. What are the properties
that a good pre
onditioner should have?
What is meant by an in
omplete fa
torization of a matrix? Des
ribe how in
omplete
fa
torizations
an be used as pre
onditioners.
Des
ribe two dierent transformations of a general nonsymmetri
linear system Ax =
b, that allows the transformed system to be solved by the standard
onjugate gradient
method.
Problems
1.
Consider square matri
es of order n, with nonzero elements only in the k-th upper
diagonal, i.e., of the form
0
B
t1
Tk = B
2.
...
tn k
C
C;
A
k 0:
3.
B1
C
CT
B2
\dqbjV
2002/
page 3
4.
377
2
3
expansion)
(I L) = I(I++LL+)(IL++LL2 )(I++ L 4 ) (Neumann
(Euler expansion)
1
Let A be a symmetri
positive denite matrix. An in
omplete Cholesky pre
onditioner for A is obtained by negle
ting elements in pla
es (i; j ) pres
ribed by a
symmetri
set
P Pn f(i; j ) j 1 i; j ng;
where (i; j ) 2 P , if i = j .
(a) The simplest
hoi
e is to take P equal to the sparsity pattern of A, whi
h for the
model problem is PA = f(i; j ) j ji j j = 0; 1; ng. This is
alled a level 0 in
omplete
fa
torization. A level 1 in
omplete fa
torization is obtained by using the union of
P0 and the pattern of the defe
t matrix R = LLT A. Higher level in
omplete
fa
torizations are dened in a similar way.
(b) Consider the model problem, where A is blo
k tridiagonal
2
2
A = tridiag( I; T + 2I; I ) 2 Rn n ;
T = tridiag( 1; 2 1) 2 Rnn :
\dqbjV
2002/
page 3
378
Chapter 10.
Iterative Methods
Notes
Two
lassi
al texts on iterative methods for linear systems are Varga [18, 1962 and
Young [21, 1971. Axelsson [1, 1994 is an ex
ellent sour
e of more modern material.
The book by Barret et al. [2, 1994 gives a
ompa
t survey of iterative methods and
their implementation. An up to date treatment of iterative methods and advan
ed
pre
onditioners are given by Saad [17, 1996. See also available software listed in
Chapter 15.
For the
ase of a square matrix A this method was originally devised by
Ka
zmarz [10, 1937. We remark that Ka
zmarz's method has been redis
overed
and used su
essfully in image re
onstru
tion. In this
ontext the method is known
as the un
onstrained ART algorithm (algebrai
re
onstru
tion te
hnique).
Krylov subspa
e iteration, whi
h originated with the
onjugate gradient method
has been named one of the Top 10 Algorithms of the 20th
entury. The
onjugate
gradient method was developed independently by E. Stiefel and M. R. Hestenes.
Further work was done at the Institute for Numeri
al Analysis, on the
ampus of
the University of California, in Los Angeles (UCLA). This work was published in
1952 in the seminal paper [9, whi
h has had a tremendous impa
t in s
ienti
omputing. In this paper the author a
knowledge
ooperation with J. B. Rosser, G. E.
Forsythe, and L. Paige, who were working at the institute during this period. They
also mention that C. Lan
zos has developed a
losely related method (see Chapter
9).
Referen
es
[1 O. Axelsson. Iterative Solution Methods. Cambridge University Press, Cambridge, UK, 1994.
[2 R. Barret, M. W. Berry, T. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. van der Vorst, editors. Templates for the
Solution of Linear Systems: Building Blo
ks for Iterative Methods. SIAM,
Philadelphia, PA, 1993.
[3 P. Con
us, G. H. Golub, and G. Meurant. Blo
k pre
onditioning for the
onjugate gradient method. SIAM J. S
i. Statist. Comput., 6:220{252, 1985.
[4 R. W. Freund, G. H. Golub, and N. M. Na
htigal. Iterative solution of linear
systems. A
ta Numeri
a, 1:57{100, 1991.
[5 P. E. Gill, W. Murray, D. B. Pon
eleon, and M. A. Saunders. Pre
onditioners
for indenite systems arising in optimization. SIAM J. Matrix. Anal. Appl.,
13:292{311, 1992.
[6 G. H. Golub and D. O'Leary. Some history of the
onjugate gradient and
Lan
zos algorithms: 1948{1976. SIAM Review, 31:50{102, 1989.
[7 G. H. Golub and H. A. van der Vorst. Closer to the solution: iterative linear
solvers. In I. S. Du and G. A. Watson, editors, The State of the Art in
Numeri
al Analysis, pages 63{92. Clarendon Press, Oxford, 1997.
\dqbjV
2002/
page 3
381
\dqbjV
2002/
page 3
382
Chapter 10.
Iterative Methods