Eigenvalues and Eigenvectors

Chapter 6
Eigenvalues and Eigenvectors

Po-Ning Chen, Professor
Department of Electrical Engineering
National Chiao Tung University
Hsin Chu, Taiwan 30010, R.O.C.
6.1 Introduction to eigenvalues 6-1
Motivations
The static system problem of Ax = b has now been solved, e.g., by Gauss-
Jordan method or Cramers rule.
However, a dynamic system problem such as
Ax = x
cannot be solved by the static system method.
To solve the dynamic system problem, we need to nd the static feature
of A that is unchanged with the mapping A. In other words, Ax maps to
itself with possibly some stretching ( > 1), shrinking (0 < < 1), or being
reversed ( < 0).
These invariant characteristics of A are the eigenvalues and eigenvectors.
Ax maps a vector x to its column space C(A). We are looking for a v C(A)
such that Av aligns with v. The collection of all such vectors is the set of eigen-
vectors.
6.1 Eigenvalues and eigenvectors 6-2
Conception (Eigenvalues and eigenvectors): The eigenvalue-eigenvector
pair (
i
, v
i
) of a square matrix A satises
Av
i
=
i
v
i
where v
i
,= 0 (but
i
can be zero).
Invariance of eigenvectors and eigenvalues.
Property 1: The eigenvectors stay the same for every power of A.

The eigenvalues equal the same power of the respective eigenvalues.
I.e., A
n
v
i
=
n
i
v
i
.
Av
i
=
i
v
i
= A
2
v
i
= A(
i
v
i
) =
i
Av
i
=
2
i
v
i
Property 2: If the null space of A

nn
consists of non-zero vectors, then 0 is
the eigenvalue of A.
Accordingly, there exists non-zero v
i
to satisfy Av
i
= 0 v
i
= 0.
6.1 Eigenvalues and eigenvectors 6-3
Property 3: Assume with no loss of generality [

1
[ > [
2
[ > > [
k
[. For
any vector x = a
1
v
1
+ a
2
v
2
+ + a
k
v
k
that is the linear combination of
all eigenvectors, the normalized mapping P =
1
1
A (namely, Pv
1
= v
1
)
(when being applied repeatedly) converges to the eigenvector with the largest
absolute eigenvalue.
I.e.,
lim
k
P
k
x = lim
k
1
k
1
A
k
x = lim
k
1
k
1
_
a
1
A
k
v
1
+ a
2
A
k
v
2
+ + a
k
A
k
v
k
_
= lim
k
1
k
1
_
a
1
k
1
v
1
. .
steady state
+a
2
k
2
v
2
+ +
k
k
A
k
v
k
. .
transient states
_
= a
1
v
1
.
6.1 How to determine the eigenvalues? 6-4
We wish to nd a non-zero vector v to satisfy Av = v; then
(A I)v = 0 det(A I) = 0.
So by solving det(A I) = 0, we can obtain all the eigenvalues of A.
Example. Find the eigenvalues of A =
_
0.5 0.5
0.5 0.5
_
.
Solution.
det(A I) = det
__
0.5 0.5
0.5 0.5
__
= (0.5 )
2
0.5
2
=
2
= ( 1) = 0
= 0 or 1.
Proposition: Projection matrix (dened in Section 4.2) has eigenvalues either 1
or 0.
Proof:
A projection matrix always satises P
2
= P. So P
2
v = Pv = v.
By denition of eigenvalues and eigenvectors, we have P
2
v =
2
v.
Hence, v =
2
v for non-zero vector v, which immediately implies =
2
.
Accordingly, is either 1 or 0. 2
Proposition: Permutation matrix has eigenvalues satisfying
k
= 1 for some
integer k.
Proof:
A purmutation matrix always satises P
k+1
= P for some integer k.
Example. P =
_
_
0 0 1
1 0 0
0 1 0
_
_
. Then, Pv =
_
_
v
3
v
1
v
2
_
_
and P
3
v = v. Hence, k = 3.
Accordingly,
k+1
v = v, which gives
k
= 1 since the eigenvalue of P cannot
be zero. 2
Proposition: Matrix
m
A
m
+
m1
A
m1
+ +
1
A +
0
I
has the same eigenvectors as A, but its eigenvalues become
m
+
m1
m1
+ +
1
+
0
,
where is the eigenvalue of A.
Proof:
Let v
i
and
i
be the eigenvector and eigenvalue of A. Then,
_
m
A
m
+
m1
A
m1
+ +
0
I
_
v
i
=
_
m
+
m1
m1
+ +
0
_
v
i
Hence, v
i
and
_
m
+
m1
m1
+ +
0
_
are the eigenvector and eigen-
value of the polynomial matrix. 2
Theorem (Cayley-Hamilton): (Suppose A has linearly independent eigen-
vectors.)
f(A) = A
n
+
n1
A
n1
+ +
0
I = all-zero matrix
if
f() = det(A I) =
n
+
n1
n1
+ +
0
.
Proof: The eigenvalues of f(A) are all zeros and the eigenvectors of A remain
the same as A. By denition of eigen-system, we have
f(A)
_
v
1
v
2
v
n
=
_
v
1
v
2
v
n
_
f(
1
) 0 0
0 f(
2
) 0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 f(
n
)
_
_
.
2
Corollary (Cayley-Hamilton): (Suppose A has linearly independent eigen-
vectors.)
(
1
I A)(
2
I A) (
n
I A) = all-zero matrix
Proof: f() can be re-written as f() = (
1
)(
2
) (
n
). 2
(Problem 11, Section 6.1) Here is a strange fact about 2 by 2 matrices with eigen-
values
1
,=
2
: The columns of A
1
I are multiples of the eigenvector x
2
. Any
idea why this should be?
Hint: (
1
I A)(
2
I A) =
_
0 0
0 0
_
implies
(
1
I A)w
1
= 0 and (
1
I A)w
2
= 0
where (
2
I A) =
_
w
1
w
2
. Hence,
the columns of (
2
I A) give the eigenvectors of
1
if they are non-zero vectors
and
the columns of (
1
I A) give the eigenvectors of
2
if they are non-zero vectors .
So, the (non-zero) columns of A
1
I are (multiples of ) the eigen-
vector x
2
.
6.1 Why Gauss-Jordan cannot solve Ax = x? 6-9
The forward elimination may change the eigenvalues and eigenvectors?
Example. Check eigenvalues and eigenvectors of A =
_
1 2
2 5
_
.
Solution.
The eigenvalues of A satisfy det(A I) = ( 3)
2
= 0.
A = LU =
_
1 0
2 1
_ _
1 2
0 9
_
. The eigenvalues of U apparently satisfy
det(U I) = (1 )(9 ) = 0.
Suppose u
1
and u
2
are the eigenvectors of U, respectively corresponding
to eigenvalues 1 and 9. Then, they cannot be the eigenvectors of A since if
they were,
_
_
3u
1
= Au
1
= LUu
1
= Lu
1
=
_
1 0
2 1
_
u
1
= u
1
= 0
3u
2
= Au
2
= LUu
2
= 9Lu
2
= 9
_
1 0
2 1
_
u
2
= u
2
= 0.
2
Hence, the eigenvalues are nothing to do with pivots (except for a triangular A).
6.1 How to determine the eigenvalues? (revisited) 6-10
Solve det(A I) = 0.
det(A I) is a polynomial of order n.
f() = det(A I) = det
_
_
_
_
_
_
_
_
_
a
1,1
a
1,2
a
1,3
a
1,n
a
2,1
a
2,2
a
2,3
a
2,n
a
3,1
a
3,2
a
3,3
a
3,n
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
a
n,1
a
n,2
a
n,3
a
n,n
_
_
_
_
_
_
_
_
= (a
1,1
)(a
2,2
) (a
n,n
) + (By Leibniz formula)
= (
1
)(
2
) (
n
)
Observations
The coecient of
n
is (1)
n
.
The coecient of
n1
is
n
i=1
i
=
n
i=1
a
i,i
= trace(A).
. . .
The coecient of
0
is
n
i=1
i
= f(0) = det(A).
6.1 How to determine the eigenvalues? (revisited) 6-11
These observations make easy the nding of the eigenvalues of 2 2 matrix.
_
1 1
4 1
_
.
Solution.
1
+
2
= 1 + 1 = 2
2
= 1 4 = 3
= (
1
2
)
2
= (
1
+
2
)
2
4
1
2
= 16
=
1
2
= 4
= = 3, 1. 2
_
1 1
2 2
_
.
Solution.
1
+
2
= 3
2
= 0
= = 3, 0. 2
6.1 Imaginary eigenvalues 6-12
In some cases, we have to allow imaginary eigenvalues.
In order to solve polynomial equations f() = 0, the mathematician was forced
to image that there exists a number x satisfying x
2
= 1 .
By this technique, a polynomial equations of order n have exactly n (possibly
complex, not real) solutions.
Example. Solve
2
+ 1 = 0. = = i. 2
Based on this, to solve the eigenvalues, we were forced to accept imaginary
eigenvalues.
_
0 1
1 0
_
.
Solution. det(A I) =
2
+ 1 = 0 = = i. 2
Proposition: The eigenvalues of a symmetric matrix A (with real entries) are
real, and the eigenvalues of a skew-symmetric (or antisymmetric) matrix B are
pure imaginary.
Proof:
Suppose Av = v. Then,
Av
(A real)
= (Av
)
T
v =
(v
)
T
v
= (v
)
T
A
T
v =
(v
)
T
v
= (v
)
T
Av =
(v
)
T
v (symmetry means A
T
= A)
= (v
)
T
v =
(v
)
T
v (Av = v)
= |v|
2
=
|v|
2
(eigenvector must be non-zero, i.e., |v|
2
,= 0)
= =
= real
Suppose Bv = v. Then,
Bv
(B real)
= (Bv
)
T
v = (
)
T
v
= (v
)
T
B
T
v =
(v
)
T
v
= (v
)
T
(B)v =
(v
)
T
v (skew-symmetry means B
T
= B)
= (v
)
T
()v =
(v
)
T
v (Bv = v)
= ()|v|
2
=
|v|
2
2
,= 0)
= =
= imaginary
2
6.1 Eigenvalues/eigenvectors of inverse 6-15
For invertible A, the relation between eigenvalues and eigenvectors of A and A
1
can be well-determined.
Proposition: The eigenvalues and eigenvectors of A
1
are
_
1
1
, v
1
_
,
_
1
2
, v
2
_
, . . . ,
_
1
n
, v
n
_
,
where (
i
, v
i
)
n
i=1
are the eigenvalues and eigenvectors of A.
Proof:
The eigenvalues of invertible A must be non-zero because det(A) ,= 0.
Suppose Av = v, where v ,= 0 and ,= 0. (I.e., and v are eigenvalue and
eigenvector of A.)
So, Av = v = A
1
(Av) = A
1
(v) = v = A
1
v =
1
v = A
1
v.
2
Note: The eigenvalues of A
k
and A
1
are
k
and
1
with the same eigenvectors
as A, where is the eigenvalue of A.
6.1 Eigenvalues/eigenvectors of inverse 6-16
Proposition: The eigenvalues of A
T
are the same as the eigenvalues of A. (But
they may have dierent eigenvectors.)
Proof: det(A I) = det ((A I)
T
) = det (A
T
I
T
) = det (A
T
I). 2
Corollary: The eigenvalues of an invertible matrix A satisfying A
1
= A
T
are
on the unit circle of the complex plain.
Proof: Suppose Av = v. Then,
Av
(A real)
= (Av
)
T
v =
(v
)
T
v
= (v
)
T
A
T
v =
(v
)
T
v
= (v
)
T
A
1
v =
(v
)
T
v (A
T
= A
1
)
= (v
)
T 1
v =
(v
)
T
v (A
1
v =
1
v)
=
1
|v|
2
=
|v|
2
2
,= 0)
=
= [[
2
= 1
2
_
0 1
1 0
_
that satises A
T
= A
1
.
Solution. det(A I) =
2
+ 1 = 0 = = i. 2
6.1 Determination of eigenvectors 6-17
After the identication of eigenvalues via det(AI) = 0, we can determine
the respective eigenvectors using the null space technique.
Recall (from slide 3-41) how we completely solve
B
nn
v
n1
= (A
nn
I
nn
)v
n1
= 0
n1
.
Answer:
R = rref(B) =
_
I
rr
F
r(nr)
0
(nr)r
0
(nr)(nr)
_
(with no row exchange)
N
n(nr)
=
_
F
r(nr)
I
(nr)(nr)
_
Then, every solution v for Bv = 0 is of the form
v
(null)
n1
= N
n(nr)
w
(nr)1
for any w 1
nr
.
Here, we usually nd the (n r) orthonormal bases for the null space
as the representative eigenvectors, which are exactly the (n r) columns of
N
n(nr)
(with proper normalization).
(Problem 12, Section 6.1) Find three eigenvectors for this matrix P (projection
matrices have = 1 and 0):
Projection matrix P =
_
_
0.2 0.4 0
0.4 0.8 0
0 0 1
_
_
.
If two eigenvectors share the same , so do all their linear combinations. Find an
eigenvector of P with no zero components.
Solution.
det(P I) = 0 gives (1
2
) = 0; so = 0, 1.
= 1: rref(P I) =
_
_
1
1
2
0
0 0 0
0 0 0
_
_
;
so F
12
=
_
1
2
0
, and N
32
=
_
_
1
2
0
1 0
0 1
_
_
, which implies eigenvectors =
_
_
1
2
1
0
_
_
and
_
_
0
0
1
_
_
.
= 0: rref(P) =
_
_
1 2 0
0 0 0
0 0 1
_
_
(with no row exchange);
so F =
_
_
2
0
_
_
, and N
31
=
_
_
2
1
0
_
_
, which implies eigenvector =
_
_
2
1
0
_
_
.
2
6.1 MATLAB revisited 6-20
In MATLAB:
We can nd the eigenvalues and eigenvectors of a matrix A by:
[V D]= eig(A); % Find the eigenvalues/vectors of A
The columns of V are the eigenvectors.
The diagonals of D are the eigenvalues.
Proposition:
The eigenvectors corresponding non-zero eigenvalues are in C(A).
The eigenvectors corresponding zero eigenvalues are in N(A).
Proof: The rst one can be seen from Av = v and the second one can be proved
by Av = 0. 2
6.1 Some discussions on problems 6-21
(Problem 25, Section 6.1) Suppose A and B have the same eigenvalues
1
, . . . ,
n
with the same independent eigenvectors x
1
, . . . , x
n
. Then A = B. Reason:
Any vector x is a combintion c
1
x
1
+ + c
n
x
n
. What is Ax? What is Bx?
Thinking over Problem 25: Suppose A and B have the same eigenvalues and
eigenvectors (not necessarily independent). Can we claim that A = B.
Answer to the thinking: Not necessarily.
As a counterexample, both A =
_
1 1
1 1
_
and B =
_
2 2
2 2
_
have eigenvalues 0, 0
and single eigenvector
_
1
1
_
but they are not equal.
If however the eigenvectors span the n-dimensional space (such as there are n of
them and they are linearly independent), then A = B.
Hint for Problem 25: A
_
x
1
x
n
=
_
x
1
x
n
1
0 0
0
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0
n
_
_
.
This important fact will be re-emphasized in Section 6.2.
(Problem 26, Section 6.1) The block B has eigenvalues 1, 2 and C has eigenvalues
3, 4 and D has eigenvalues 5, 7. Find the eigenvalues of the 4 by 4 matrix A:
A =
_
B C
0 D
_
=
_
_
0 1 3 0
2 3 0 4
0 0 6 1
0 0 1 6
_
_
.
Thinking over Problem 26: The eigenvalues of
_
B C
0 D
_
are the eigenvalues of B
and D because we can show
det
__
B C
0 D
__
= det(B) det(D).
(See Problems 23 and 25 in Section 5.2.)
(Problem 23, Section 5.2) With 2 by 2 blocks in 4 by 4 matrices, you cannot always
use block determinants:
A B
0 D
= [A[ [D[ but
A B
C D
= [A[ [D[ [C[ [B[.

(a) Why is the rst statement true? Somehow B doesnt enter.
Hint: Leibniz formula.
(b) Show by example that equality fails (as shown) when C enters.
(c) Show by example that the answer det(AD CB) is also wrong.
(Problem 25, Section 5.2) Block elimination subtracts CA
1
times the rst
row
_
A B
from the second row

_
C D
. This leaves the Schur complement

D CA
1
B in the corner:
_
I 0
CA
1
I
_ _
A B
C D
_
=
_
A B
0 D CA
1
B
_
.
Take determinants of these block matrices to prove correct rules if A
1
exists:
A B
C D
= [A[ [D CA
1
B[ = [AD CB[ provided AC = CA.
Hint: det(A) det(D CA
1
B) = det
_
A(D CA
1
B)
_
.
(Problem 37, Section 6.1)
(a) Find the eigenvalues and eigenvectors of A. They depend on c:
A =
_
.4 1 c
.6 c
_
.
(b) Show that A has just one line of eigenvectors when c = 1.6.
(c) This is a Markov matrix when c = .8. Then A
n
will approach what matrix
A
?
Denition (Markov matrix): A Markov matrix is a matrix with positive
entries, for which every column adds to one.
Note that some researchers dene the Markov matrix by replacing positive by
non-negative; thus, they will use the term positive Markov matrix. The below
observations hold for Markov matrices with non-negative entries.
1 must be one of the eigenvalues of a Markov matrix.
Proof: A and A
T
have the same eigenvalues, and A
T
_
_
1
.
.
.
1
_
_
=
_
_
1
.
.
.
1
_
_
. 2
The eigenvalues of a Markov matrix satisfy [[ 1.
Proof: A
T
v = v implies
n
i=1
a
i,j
v
i
= v
j
; hence, by letting v
k
satisfying
[v
k
[ = max
1in
[v
i
[, we have
[v
k
[ =
i=1
a
i,k
v
i
i=1
a
i,k
[v
i
[
n
i=1
a
i,k
[v
k
[ = [v
k
[
which implies the desired result. 2
(Problem 31, Section 6.1) If we exchange rows 1 and 2 and columns 1 and 2, the
eigenvalues dont change. Find eigenvectors of A and B for
1
= 11. Rank one
gives
2
=
3
= 0.
A =
_
_
1 2 1
3 6 3
4 8 4
_
_
and B = PAP
T
=
_
_
6 3 3
2 1 1
8 4 4
_
_
.
Thinking over Problem 31: This is sometimes useful in determining the eigenval-
ues and eigenvectors.
If Av = v and B = PAP
1
(for any invertible P), then and u = Pv are
the eigenvalue and eigenvector of B.
Proof: Bu = PAP
1
u = PAv = Pv = u. 2
For example, P =
_
_
0 1 0
1 0 0
0 0 1
_
_
. In such case, P
1
= P = P
T
.
With the fact above, we can further claim that
Proposition. The eigenvalues of AB and BA are equal if one of A and B is
invertible.
Proof: This can be proved by (AB) = A(BA)A
1
or (AB) = B
1
(BA)B. Note
that AB has eigenvector Av or B
1
v if v is the eigenvector of BA. 2
The eigenvectors of a diagonal matrix =
_
1
0 0
0
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0
n
_
_
are apparently
e
i
=
_
_
0
.
.
.
0
1
0
.
.
.
0
_
_
position i , i = 1, 2, . . . , n
Hence, A = SS
1
have the same eigenvalues as and have eigenvectors v
i
=
Se
i
. This implies that
S =
_
v
1
v
2
v
n
where v
i
are the eigenvectors of A.
What if S is not invertible? Then, we have the next theorem.
6.2 Diagnolizing a matrix 6-30
A convenience for the eigen-system analysis is that we can diagonalize a matrix.
Theorem. A matrix A can be written as
A
_
v
1
v
2
v
n
. .
=S
=
_
v
1
v
2
v
n
1
0 0
0
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0
n
_
_
. .
=
where (
i
, v
i
)
n
i=1
are the eigenvalue-eigenvector pairs of A.
Proof: The theorem holds by denition of eigenvalues and eigenvectors. 2
Corollary. If S is invertible, then
S
1
AS = equivalently A = SS
1
.
This makes easy the computation of polynomial with argument A.
Proposition (recall slide 6-6): Matrix
m
A
m
+
m1
A
m1
+ +
0
I
has the same eigenvectors as A, but its eigenvalues become
m
+
m1
m1
+ +
0
,
where is the eigenvalue of A.
Proposition:
m
A
m
+
m1
A
m1
+ +
0
I = S
_
m
+
m1
m1
+ +
0
_
S
1
Proposition:
A
m
= S
m
S
1
Exception
It is possible that an n n matrix does not have n eigenvectors.
Example. A =
_
1 1
1 1
_
.
Solution.

1
= 0 is apparently an eigenvalue for a non-invertible matrix.
The second eigenvalue is
2
= trace(A)
1
= [1 + (1)]
1
= 0.
This matrix however only has one eigenvector
_
1
1
_
(or some may say two
repeated eigenvectors).
In such case, we still have
_
1 1
1 1
_
. .
A
_
1 1
1 1
_
. .
S
=
_
1 1
1 1
_
. .
S
_
0 0
0 0
_
. .
but S has no inverse. 2

In such case, we cannot preform SAS
1
; so we say A cannot be diagonalized.
When will S be guaranteed to be invertible?
One possible answer: When all eigenvalues are distinct.
Proof: Suppose for a matrix A, the rst k eigenvectors v
1
, . . . , v
k
are linearly inde-
pendent, but the (k+1)th eigenvector is dependent on the previous k eigenvectors.
Then, for some unique a
1
, . . . , a
k
,
v
k+1
= a
1
v
1
+ + a
k
v
k
,
which implies
_
k+1
v
k+1
= Av
k+1
= a
1
Av
1
+ . . . + a
k
Av
k
= a
1
1
v
1
+ . . . + a
k
k
v
k
k+1
v
k+1
= a
1
k+1
v
1
+ . . . + a
k
k+1
v
k
Accordingly,
k+1
=
i
for 1 i k; i.e., that v
k+1
is linearly dependent on
v
1
, . . . , v
k
only occurs when they have the same eigenvalues. (The proof is not
yet completed here! See the discussion and Example below.)
The above proof said as long as we have the (k+1)th eigenvector, then it is linearly
dependent on the previous k eigenvectors only when all eigenvalues are equal! But
sometimes, we do not guarantee to have the (k + 1)th eigenvector.
Example. A =
_
_
5 4 2 1
0 1 1 1
1 1 3 0
1 1 1 2
_
_
.
Solution.
We have four eigenvalues 1, 2, 4, 4.
But we can only have three eigenvectors for 1, 2 and 4.
From the proof in the previous page, the eigenvectors for 1, 2 and 4 are linearly
indepedent because 1, 2 and 4 are not equal. 2
Proof (Continue): One condition that guarantees to have n eigenvectors is that
we have n distinct eigenvalues, which now complete the proof. 2
Denition (GM and AM): The number of linearly independent eigenvectors
for an eigenvalue is called its geometric multiplicity; the number of its
appearance in solving det(A I) is called its algebraic multiplicity.
Example. In the above example, GM=1 and AM=2 for = 4.
6.2 Fibonacci series 6-35
Eigen-decomposition is useful in solving Fibonacci series
Problem: Suppose F
k+2
= F
k+1
+F
k
with initially F
1
= 1 and F
0
= 0. Find F
100
.
Answer:
_
F
k+2
F
k+1
_
=
_
1 1
1 0
_ _
F
k+1
F
k
_
=
_
F
k+2
F
k+1
_
=
_
1 1
1 0
_
k+1
_
F
1
F
0
_
So,
_
F
100
F
99
_
=
_
1 1
1 0
_
99
_
1
0
_
= S
99
S
1
_
1
0
_
=
_
1+
5
2
1
5
2
1 1
_
_
_
_
1+
5
2
_
99
0
0
_
1
5
2
_
99
_
_
_
1+
5
2
1
5
2
1 1
_
1
_
1
0
_
=
_
1+
5
2
1
5
2
1 1
_
_
_
_
1+
5
2
_
99
0
0
_
1
5
2
_
99
_
_
1
_
1+
5
2
_
_
1
5
2
_
_
1
1
5
2
1
1+
5
2
_
_
1
0
_
6.2 Fibonacci series 6-36
=
_
_
_
1+
5
2
_
100
_
1
5
2
_
100
_
1+
5
2
_
99
_
1
5
2
_
99
_
_
1
_
1+
5
2
_
_
1
5
2
_
_
1
1
_
=
1
_
1+
5
2
_
_
1
5
2
_
_
_
_
1+
5
2
_
100
_
1
5
2
_
100
_
1+
5
2
_
99
_
1
5
2
_
99
_
_
.
2
6.2 Generalization to u
k
= A
k
u
0
6-37
A generalization of the solution to Fibonacci series is as follows.
Suppose u
0
= a
1
v
1
+ a
2
v
2
+ + a
n
v
n
.
Then u
k
= A
k
u
0
= a
1
A
k
v
1
+ a
2
A
k
v
2
+ + a
n
A
k
v
n
= a
1
k
1
v
1
+ a
2
k
2
v
2
+ . . . + a
n
k
n
v
n
.
Examination:
u
k
=
_
F
k+1
F
k
_
=
_
1 1
1 0
_
k
_
F
1
F
0
_
= A
k
u
0
and
u
0
=
_
1
0
_
=
1
_
1+
5
2
_
_
1
5
2
_
_
1+
5
2
1
_
1
_
1+
5
2
_
_
1
5
2
_
_
1
5
2
1
_
So
u
99
=
_
F
100
F
99
_
=
_
1+
5
2
_
99
_
1+
5
2
_
_
1
5
2
_
_
1+
5
2
1
_
_
1
5
2
_
99
_
1+
5
2
_
_
1
5
2
_
_
1
5
2
1
_
.
6.2 More applications of eigen-decomposition 6-38
(Problem 30, Section 6.2) Suppose the same S diagonalizes both A and B. They
have the same eigenvectors in A = S
1
S
1
and B = S
2
S
1
. Prove that AB =
BA.
Proposition: Suppose both A and B can be diagnosed, and one of A and B
have distinct eigenvalues. Then, A and B have the same eigenvectors if, and only
if, AB = BA.
Proof:
(See Problem 30 above) If A and B have the same eigenvectors, then
AB = (S
A
S
1
)(S
B
S
1
) = S
A
B
S
1
= S
B
A
S
1
= (S
B
S
1
)(S
A
S
1
) = BA.
Suppose without loss of generality that the eigenvalues of A are all distinct.
Then, if AB = BA, we have for given Av
and u
= Bv
,
Au
= A(Bv
) = (AB)v
= (BA)v
= B(Av
) = B(
) =
(Bv
) =
.
Hence, u
= Bv
and v
are both the eigenvectors of A corresponding to the

same eigenvalue
.
6.2 More applications of eigen-decomposition 6-39
Let u
n
i=1
a
i
v
i
, where v
i
n
i=1
are linearly independent eigenvectors of A.
_
Au
n
i=1
a
i
Av
i
=
n
i=1
a
i
i
v
i
Au
n
i=1
a
i
v
i
= a
i
(
i
) = 0 for 1 i n.
This implies
u
i:
i
=
a
i
v
i
= a
= Bv
.
Thus, v
is an eigenvector of B. 2
6.2 Problem discussions 6-40
(Problem 32, Section 6.2) Substitute A = SS
1
into the product (A
1
I)(A
2
I) (A
n
I) and explain why this produces the zero matrix. We are sub-
stituting the matrix A for the number in the polynomial p() = det(A I).
The Cayley-Hamilton Theorem says that this product is always p(A) =zero
matrix, even if A is not diagonalizable.
Thinking over Problem 32: The Cayley-Hamilton Theorem can be easily proved
if A is diagonalizable.
Corollary (Cayley-Hamilton): Suppose A has linearly independent eigen-
vectors.
(
1
I A)(
2
I A) (
n
I A) = all-zero matrix
Proof:
(
1
I A)(
2
I A) (
n
I A)
= S(
1
I )S
1
S(
2
I )S
1
S(
n
I )S
1
= S (
1
I )
. .
(1, 1)th entry= 0
(
2
I )
. .
(2, 2)th entry= 0
(
n
I )
. .
(n, n)th entry= 0
S
1
= S
_
all-zero entries
S
1
=
_
all-zero entries
2
6.3 Applications to dierential equations 6-41
We can use eigen-decomposition to solve
_
_
u
1
(t)
u
2
(t)
.
.
.
u
n
(t)
_
_
=
du
dt
= Au =
_
_
a
1,1
u
1
(t) + a
1,2
u
2
(t) + + a
1,n
u
n
(t)
a
2,1
u
1
(t) + a
2,2
u
2
(t) + + a
2,n
u
n
(t)
.
.
.
a
n,1
u
1
(t) + a
n,2
u
2
(t) + + a
n,n
u
n
(t)
_
_
where A is called the companion matrix.
By dierential equation technique, we know that the solution is of the form
u = e
t
v
for some constant and constant vector v.
Question: What are all and v that satisfy
du
dt
= Au?
Solution: Take u = e
t
v into the equation:
du
dt
_
= e
t
v
_
= Au
_
= Ae
t
v
_
= v = Av.
The answer are all eigenvalues and eigenvectors.
Since
du
dt
= Au is a linear system, a linear combination of solutions is still a
solution.
Hence, if there are n eigenvectors, the complete solution can be represented as
c
1
e
1
t
v
1
+ c
2
e
2
t
v
2
+ + c
n
e
n
t
v
n
where c
1
, c
2
, . . . , c
n
can be determined by the initial conditions.
Here, for convenience of discussion at this moment, we assume that A gives us
exactly n eigenvectors.
It is sometimes convenient to re-express the solution of
du
dt
= Au is e
At
u(0) for
a given initial condition u(0) as e
At
u(0), i.e.,
c
1
e
1
t
v
1
+ c
2
e
2
t
v
2
+ + c
n
e
n
t
v
n
=
_
v
1
v
2
v
n
. .
S
_
_
e
1
t
0 0
0 e
2
t
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 e
n
t
_
_
_
_
c
1
c
2
.
.
.
c
n
_
_
= e
At
u(0)
where we dene
e
At
k=0
1
k!
(At)
k
=
k=0
t
k
k!
A
k
=
k=0
t
k
k!
(S
k
S
1
) = S
_

k=0
t
k
k!
k
_
S
1
= S
_
k=0
t
k
k!
k
1
0 0
0
k=0
t
k
k!
k
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0

k=0
t
k
k!
k
n
_
_
S
1
= S
_
_
e
1
t
0 0
0 e
2
t
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 e
n
t
_
_
S
1
.
and hence u(0) = Sc.
Key to remember: Again, we dene by convention that
f(A) = S
_
_
f(
1
) 0 0
0 f(
2
) 0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 f(
n
)
_
_
S
1
.
So, e
At
= S
_
_
e
1
t
0 0
0 e
2
t
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 e
n
t
_
_
S
1
.
We can solve the second-order equation in the same manner.
Example. Solve my
//
+ by
/
+ ky = 0.
Answer:
Let z = y
/
. Then, the problem is reduced to
_
y
/
= z
z
/
=
k
m
y
b
m
z
=
du
dt
= Au
with u =
_
y
z
_
and A =
_
0 1
k
m

b
m
_
.
The complete solution for u is therefore
c
1
e
1
t
v
1
+ c
2
e
2
t
v
2
.
2
Example. Solve y
//
+ y = 0 with initial y(0) = 1 and y
/
(0) = 0.
Solution.
Let z = y
/
_
y
/
= z
z
/
= y
=
du
dt
= Au
with u =
_
y
z
_
and A =
_
0 1
1 0
_
.
The eigenvalues and eigenvectors of A are
_
,
_
1
__
and
_
,
_
1
__
.
The complete solution for u is therefore
_
y
y
/
_
= c
1
e
t
_
1
_
+ c
2
e
t
_
1
_
with initially
_
1
0
_
= c
1
_
1
_
+ c
2
_
1
_
.
So, c
1
=

2
and c
2
=
2
. Thus,
y(t) =

2
e
t
()

2
e
t
=
1
2
e
t
+
1
2
e
t
= cos(t).
2
In practice, we will use a discrete approximation to approximate a continuous
function. There are however three dierent discrete approximations.
For example, how to approximate y
//
(t) = y(t) by a discrete system?
Y
n1
Forward approximation
Y
n+1
2Y
n
+ Y
n1
(t)
2
=
Y
n+1
Y
n
t

Y
n
Y
n1
t
t
= Y
n
Centered approximation
Y
n+1
Backward approximation
Lets take forward approximation as an example, i.e.,
Y
n+1
Y
n
t
. .
Z
n
Y
n
Y
n1
t
. .
Z
n1
t
= Y
n1
.
Thus,
_
y
/
(t) = z(t)
z
/
(t) = y(t)
_
Y
n+1
Y
n
t
= Z
n
Z
n+1
Z
n
t
= Y
n
_
Y
n+1
Z
n+1
_
. .
u
n+1
=
_
1 t
t 1
_
. .
A
_
Y
n
Z
n
_
. .
u
n
Then, we obtain
u
n
= A
n
u
0
=
_
1 1

_ _
(1 + t)
n
0
0 (1 t)
n
_ _
1
2

2
1
2
2
_ _
1
0
_
and
Y
n
=
(1 + t)
n
+ (1 t)
n
2
=
_
1 + (t)
2
n/2
cos(n )
where = tan
1
(t).
Problem: Y
n
as n large.
Backward approximation will leave to Y
n
0 as n large.
_
y
/
(t) = z(t)
z
/
(t) = y(t)
_
Y
n+1
Y
n
t
= Z
n+1
Z
n+1
Z
n
t
= Y
n+1
_
1 t
t 1
_
. .
A
1
_
Y
n+1
Z
n+1
_
. .
u
n+1
=
_
Y
n
Z
n
_
. .
u
n
A solution to the problem: Interleave the forward approximation with
the backward approximation.
_
y
/
(t) = z(t)
z
/
(t) = y(t)
_
Y
n+1
Y
n
t
= Z
n
Z
n+1
Z
n
t
= Y
n+1
_
1 0
t 1
_ _
Y
n+1
Z
n+1
_
. .
u
n+1
=
_
1 t
0 1
_ _
Y
n
Z
n
_
. .
u
n
We then perform this so-called leapfrog method. (See Problems 28 and 29.)
_
Y
n+1
Z
n+1
_
. .
u
n+1
=
_
1 0
t 1
_
1
_
1 t
0 1
_ _
Y
n
Z
n
_
. .
u
n
=
_
1 t
t 1 (t)
2
_
. .
[eigenvalues[=1 if t2
_
Y
n
Z
n
_
. .
u
n
(Problem 28, Section 6.3) Centering y
//
= y in Example 3 will produce Y
n+1
2Y
n
+Y
n1
= (t)
2
Y
n
. This can be written as a one-step dierence equation for
U = (Y, Z):
Y
n+1
= Y
n
+ t Z
n
Z
n+1
= Z
n
t Y
n+1
_
1 0
t 1
_ _
Y
n+1
Z
n+1
_
=
_
1 t
0 1
_ _
Y
n
Z
n
_
.
Invert the matrix on the left side to write this as U
n+1
= AU
n
. Show that detA =
1. Choose the large time step t = 1 and nd the eigenvalues
1
and
2
=

1
of
A:
A =
_
1 1
1 0
_
has [
1
[ = [
2
[ = 1. Show that A
6
is exactly I.
After 6 steps to t = 6, U
6
equals U
0
. The exact y = cos(t) returns to 1 at t = 2.
(Problem 29, Section 6.3) That centered choice (leapfrog method) in Problem 28 is
very successful for small time steps t. But nd the eigenvalues of A for t =
2
and 2:
A =
_
1
2 1
_
and A =
_
1 2
2 3
_
.
Both matrices have [[ = 1. Compute A
4
in both cases and nd the eigenvectors
of A. That value t = 2 is at the border of instability. Time steps t > 2 will
lead to [[ > 1, and the powers in U
n
= A
n
U
0
will explode.
Note You might say that nobody would compute with t > 2. But if an atom
vibrates with y
//
= 1000000y, then t > .0002 will give instability. Leapfrog
has a very strict stability limit. Y
n+1
= Y
n
+ 3Z
n
and Z
n+1
= Z
n
3Y
n+1
will
explode because t = 3 is too large.
A better solution to the problem: Mix the forward approximation
with the backward approximation.
_
y
/
(t) = z(t)
z
/
(t) = y(t)
_
Y
n+1
Y
n
t
=
Z
n+1
+ Z
n
2
Z
n+1
Z
n
t
=
Y
n+1
+ Y
n
2
_
1
t
2
t
2
1
_ _
Y
n+1
Z
n+1
_
. .
u
n+1
=
_
1
t
2
t
2
1
_ _
Y
n
Z
n
_
. .
u
n
We then perform this so-called trapezoidal method. (See Problem 30.)
_
Y
n+1
Z
n+1
_
. .
u
n+1
=
_
1
t
2
t
2
1
_
1
_
1
t
2
t
2
1
_ _
Y
n
Z
n
_
. .
u
n
=
1
1 +
_
t
2
_
2
_
1
_
t
2
_
2
t
t 1
_
t
2
_
2
_
. .
[eigenvalues[=1 for all t>0
_
Y
n
Z
n
_
. .
u
n
(Problem 30, Section 6.3) Another good idea for y
//
= y is the trapezoidal method
(half forward/half back): This may be the best way to keep (Y
n
, Z
n
) exactly on a
circle.
Trapezoidal
_
1 t/2
t/2 1
_ _
Y
n+1
Z
n+1
_
=
_
1 t/2
t/2 1
_ _
Y
n
Z
n
_
.
(a) Invert the left matrix to write this equation as U
n+1
= AU
n
. Show that A is an
orthogonal matrix: A
T
A = I. These points U
n
never leave the circle.
A = (I B)
1
(I + B) is always an orthogonal matrix if B
T
= B (See the
proof on next page).
(b) (Optional MATLAB) Take 32 steps from U
0
= (1, 0) to U
32
with t = 2/32. Is
U
32
= U
0
? I think there is a small error.
Proof:
A = (I B)
1
(I + B)
(I B)A = (I + B)
A
T
(I B)
T
= (I + B)
T
A
T
(I B
T
) = (I + B
T
)
A
T
(I + B) = (I B)
A
T
= (I B)(I + B)
1
A
T
A = (I B)(I + B)
1
(I B)
1
(I + B)
= (I B)[(I B)(I + B)]
1
(I + B)
= (I B)[(I + B)(I B)]
1
(I + B)
= (I B)(I B)
1
(I + B)
1
(I + B)
= I
Question: What if the number of eigenvectors is smaller than n?
Recall that it is convenient to say that the solution of du/dt = Au is e
At
u(0)
for some constant vecor u(0).
Conveniently, we can presume that e
At
u(0) is the solution of du/dt = Au even if
A does not have n eigenvectors.
Example. Solve y
//
2y
/
+ y = 0 with initially y(0) = 1 and y
/
(0) = 0.
Answer.
Let z = y
/
_
y
/
= z
z
/
= y + 2z
=
du
dt
=
_
y
/
z
/
_
=
_
0 1
1 2
_
. .
A
_
y
z
_
= Au
The solution is still e
At
u(0) but
e
At
,= Se
t
S
1
because there is only one eigenvector for A(it doesnt matter whether we regard
this case as S does not exist or we regard this case as S exists but has no
inverse).
_
0 1
1 2
_
. .
A
_
1 1
1 1
_
. .
S
=
_
1 1
1 1
_
. .
S
_
1 0
0 1
_
. .
and e
At
= e
It
e
(AI)t
= e
It
e
(AI)t
e
At
= e
It
e
(AI)t
(since (It)((AI)t) = ((A I)t)(It). See below.)
=
_

k=0
1
k!
I
k
t
k
__

k=0
1
k!
(A I)
k
t
k
_
=
__

k=0
1
k!
t
k
_
I
__
1
k=0
1
k!
(A I)
k
t
k
_
where we know I
k
= I for every k 0 and (A I)
k
=all-zero matrix for
k 2. This gives
e
At
=
_
e
t
I
_
(I + (A I)t) = e
t
(I + (A I)t)
and
u(t) = e
At
u(0) = e
At
_
1
0
_
= e
t
(I + (A I)t)
_
1
0
_
= e
t
_
1 t
t
_
2
Note that e
A
e
B
, e
B
e
A
and e
A+B
may not be equal except AB = BA!
Properities of e
At
The eigenvalues of e
At
are e
t
for all eigenvalues of A.
For example, e
At
= e
t
(I + (A I)t) has repeated eigenvalues e
t
, e
t
.
The eigenvectors of e
At
remains the same as A.
For example, e
At
= e
t
(I + (A I)t) has only one eigenvector
_
1
1
_
.
The inverse of e
At
always exist (hint: e
t
,= 0), and is equal to e
At
.
For example, e
At
= e
It
e
(IA)t
= e
t
(I (A I)t).
The transpose of e
At
is
_
e
At
_
T
=
_

k=0
1
k!
A
k
t
k
_
T
=
_

k=0
1
k!
(A
T
)
k
t
k
_
= e
A
T
t
.
Hence, if A
T
= A (skew-symmetric), then
_
e
At
_
T
e
At
= e
A
T
t
e
At
= I; so, e
At
is an orthogonal matrix (Recall that a matrix Q is orthogonal if Q
T
Q = I).
6.3 Quick tip 6-58
For a 2 2 matrix A =
_
a
1,1
a
1,2
a
2,1
a
2,2
_
, the eigenvector corresponding to the
eigenvalue
1
is
_
a
1,2
1
a
1,1
_
.
(A
1
I)
_
a
1,2
1
a
1,1
_
=
_
a
1,1
1
a
1,2
a
2,1
a
2,2
1
_ _
a
1,2
1
a
1,1
_
=
_
0
?
_
where ?= 0 because the two row vectors of (A
1
I) are parallel.
This is especially useful when solving the dierential equation because a
1,1
= 0
and a
1,2
= 1; hence, the eigenvector v
1
corresponding to the eigenvalue
1
is
v
1
=
_
1
1
_
.
Solve y
//
2y
/
+ y = 0. Let z = y
/
_
y
/
= z
z
/
= y + 2z
=
du
dt
=
_
y
/
z
/
_
=
_
0 1
1 2
_
. .
A
_
y
z
_
= Au = v
1
=
_
1
1
_
Problem 6.3B and Problem 31: A convenient way to solve d
2
u/dt = Au.
(Problem 31, Section 6.3) The cosine of a matrix is dened like e
A
, by copying
the series for cos t:
cos t = 1
1
2!
t
2
+
1
4!
t
4
cos A = I
1
2!
A
2
+
1
4!
A
4

(a) If Ax = x, multiply each term times x to nd the eigenvalue of cos A.
(b) Find the eigenvalues of A =
_

_
with eigenvectors (1, 1) and (1, 1). From
the eigenvalues and eigenvectors of cos A, nd that matrix C = cos A.
(c) The second derivative of cos(At) is A
2
cos(At).
u(t) = cos(At)u(0) solve
d
2
u
dt
2
= A
2
u starting from u
/
(0) = 0.
Construct u(t) = cos(At)u(0) by the usual three steps for that specic A:
1. Expand u(0) = (4, 2) = c
1
x
1
+ c
2
x
2
in the eigenvectors.
2. Multiply those eigenvectors by and (instead of e
t
).
3. Add up the solution u(t) = c
1
x
1
+ c
2
x
2
. (Hint: See slide 6-45.)
(Problem 6.3B, Section 6.3) Find the eigenvalues and eigenvectors of A and write
u(0) = (0, 2
2, 0) as a combination of the eigenvectors. Solve both equations

u
/
= Au and u
//
= Au:
du
dt
=
_
_
2 1 0
1 2 1
0 1 2
_
_
u and
d
2
u
dt
2
=
_
_
2 1 0
1 2 1
0 1 2
_
_
u with
du
dt
(0) = 0.
The 1, 2, 1 diagonals make A into a second dierence matrix (like a second
derivative).
u
/
= Au is like the heat equation u/t =
2
u/x
2
.
Its solution u(t) will decay (negative eigenvalues).
u
//
= Au is like the wave equation
2
u/t
2
=
2
u/x
2
.
Its solution will oscillate (imaginary eigenvalues).
Thinking over Problem 6.3B and Problem 31: Solve
d
2
u
dt
= Au.
Solution. The answer should be e
A
1/2
t
u(0). Note that A
1/2
has the same eigenvec-
tors as A but its eigenvalues are the square root of A.
e
A
1/2
t
= S
_
_
e
1/2
1
t
0 0
0 e
1/2
2
t
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 e
1/2
n
t
_
_
S
1
.
As an example from Problem 6.3B, A =
_
_
2 1 0
1 2 1
0 1 2
_
_
.
The eigenvalues of A are 2 and 2
2.
So,
e
A
1/2
t
= S
_
_
e
2 t
0
0 e
2 t
0
0 0 e
2+
2 t
_
_
S
1
.
A more direct solution is u(t) = cos(At)u(0). See Problem 31.
Final reminder on the approach delivered in the textbook
In solving
du(t)
dt
= At with initial u(0), it is convenient to nd the solution as
_
u(0) = Sc = c
1
v
1
+ + c
n
v
n
u(t) = Se
t
c = c
1
e
1
v
1
+ + c
n
e
n
v
n
This is what the textbook always does in Problems.
You should practice these problems by yourself !
Problem 15: How to solve
du
dt
= Au b (for invertible A)?
(Problem 15, Section 6.3) A particular solution to du/dt = Aub is u
p
= A
1
b,
if A is invertible. The usual solutions to du/dt = Au give u
n
. Find the complete
solution u = u
p
+ u
n
:
(a)
du
dt
= u 4 (b)
du
dt
=
_
1 0
1 1
_
u
_
4
6
_
.
Problem 16: How to solve
du
dt
= Au e
ct
b (for non-eigenvalue c of A)?
(Problem 16, Section 6.3) If c is not an eigenvalue of A, substitute u = e
ct
v and
nd a particular solution to du/dt = Aue
ct
b. How does it break down when c is
an eigenvalue of A? The nullspace of du/dt = Au contains the usual solutions
e
i
t
x
i
.
Hint: The particular solution v
p
satises (A cI)v
p
= b.
Problem 23: e
A
e
B
, e
B
e
A
and e
A+B
are not necessarily equal. (They are equal
when AB = BA.)
(Problem 23, Section 6.3) Generally e
A
e
B
is dierent from e
B
e
A
. They are both
dierent from e
A+B
. Check this using Problems 21-2 and 19. (If AB = BA, all
these are the same.)
A =
_
1 4
0 0
_
B =
_
0 4
0 0
_
A + B =
_
1 0
0 0
_
Problem 27: An interesting brain-storming problem!
(Problem 27, Section 6.3) Find a solution x(t), y(t) that gets large as t . To
avoid this instability a scientist exchanges the two equations:
dx/dt = 0x 4y
dy/dt = 2x + 2y
becomes
dy/dt = 2x + 2y
dx/dt = 0x 4y
Now the matrix
_
2 2
0 4
_
is stable. It has negative eigenvalues. How can this be?
Hint: The matrix is not the right one to be used to describe the transformed linear
equations.
6.4 Symmetric matrices 6-65
Denition (Symmetric matrix): A matrix A is symmetric if A = A
T
.
When A is symmetric,
its eigenvalues are all reals;
it has n orthogonal eigenvectors (so it can be diagonalize). We can then
normalize these orthogonal eigenvectors to obtain an orthonormal basis.
Denition (Symmetric diagonalization): A symmetric matrix A can be
written as
A = QQ
1
= QQ
T
with Q
1
= Q
T
where is a diagonal matrix with real eigenvalues at the diagonal.
Theorem (Spectral theorem or principal axis theorem) A symmetric
matrix A with distinct eigenvalues can be written as
A = QQ
1
= QQ
T
with Q
1
= Q
T
Proof:
We have proved on slide 6-13 that a symmetric matrix has real eigenvalues
and a skew-symmetric matrix has pure imaginary eigenvalues.
Next, we prove that eigenvectors are orthogonal, i.e., v
T
1
v
2
= 0, for unequal
eigenvalues
1
,=
2
.
Av
1
=
1
v
1
and Av
2
=
2
v
2
= (
1
v
1
)
T
v
2
= (Av
1
)
T
v
2
= v
T
1
A
T
v
2
= v
T
1
Av
2
= v
T
1
2
v
2
which implies
(
1
2
)v
T
1
v
2
= 0.
Therefore, v
T
1
v
2
= 0 because
1
,=
2
. 2
A = QQ
T
implies that
A =
1
q
1
q
T
1
+ +
n
q
n
q
T
n
=
1
P
1
+ +
n
P
n
Recall from slide 4-37, the projection matrix P
i
onto a unit vector q
i
is
P
i
= q
i
(q
T
i
q
i
)
1
q
T
i
= q
i
q
T
i
So, Ax is the sum of projections of vector x onto each eigenspace.
Ax =
1
P
1
x + +
n
P
n
x.
The spectral theorem can be extended to a symmetric matrix with repeated
eigenvalues.
To prove this, we need to introduce the famous Schurs theorem.
Theorem (Schurs theorem): Every square matrix A can be factorized into
A = QTQ
1
with T upper triangular and Q
= Q
1
,
where denotes Hermisian transpose operation.
Further, if A has real eigenvalues (and hence has real eigenvectors), then Q and T
can be chosen real.
Proof: The existence of Q and T such that A = QTQ
and Q
= Q
1
can be
proved by induction.
The theorem trivially holds when A is a 1 1 matrix.
Suppose that Schurs Theorem is valid for all (n1) (n1) matrices. Then,
we claim that Schurs Theorem will hold true for all n n matrices.
This is because we can take t
1,1
and q
1
to be the eigenvalue and eigenvector
of A
nn
(as there must exist at least one pair of eigenvalue and eigenvector
for A
nn
). Then, choose any p
2
, . . ., p
n
such that they together with q
1
span the n-dimensional space, and
P =
_
q
1
p
2
p
n
is a orthonormal matrix.
We derive
P
AP =
_
_
q
1
p
2
.
.
.
p
n
_
_
_
Aq
1
Ap
2
Ap
n
=
_
_
q
1
p
2
.
.
.
p
n
_
_
_
t
1,1
q
1
Ap
2
Ap
n
(since A q
1
..
eigenvector
= t
1,1
..
eigenvalue
q
1
)
=
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
A
(n1)(n1)
_
_
where t
1,j
= q
1
Ap
j
and
A
(n1)(n1)
=
_
_
p
2
Ap
2
p
2
Ap
n
.
.
.
.
.
.
.
.
.
p
n
Ap
2
p
n
Ap
n
_
_
.
Since Schurs Theorem is true for any (n1) (n1) matrix, we can nd
Q
(n1)(n1)
and

T
(n1)(n1)
such that
A
(n1)(n1)
=

Q
(n1)(n1)

T
(n1)(n1)

Q
(n1)(n1)
and
(n1)(n1)
=

Q
1
(n1)(n1)
.
Finally, dene
Q
nn
= P
_
1 0
T
0

Q
(n1)(n1)
_
and
T
nn
=
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
T
(n1)(n1)
_
_
satisfy that
Q
nn
Q
nn
=
_
1 0
T
0

Q
(n1)(n1)
_
P
P
_
1 0
T
0

Q
(n1)(n1)
_
= I
nn
and
Q
nn
T
nn
= P
nn
_
1 0
T
0

Q
(n1)(n1)
_
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
T
(n1)(n1)
_
_
= P
nn
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
Q
(n1)(n1)

T
(n1)(n1)
_
_
= P
nn
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
A
(n1)(n1)

Q
(n1)(n1)
_
_
= P
nn
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
A
(n1)(n1)
_
_
_
1 0
T
0

Q
(n1)(n1)
_
= P
nn
(P
nn
A
nn
P
nn
)
_
1 0
T
0

Q
(n1)(n1)
_
= A
nn
Q
nn
It remains to prove that if A has real eigenvalues, then Q and T can be chosen
real, which can be similarly proved by induction. Suppose If A has real
eigenvalues, then Q and T can be chosen real. is true for all (n1) (n1)
matrices. Then, the claim should be true for all n n matrices.
For a given A
nn
with real eigenvalues (and real eigenvectors), we can
certainly have the real t
1,1
and q
1
, and so are p
2
, . . . , p
n
. This makes real
the resultant

A
(n1)(n1)
and t
1,2
, . . . , t
1,n
.
The eigenvector associated with a real eigenvalue can be chosen real. For
complex v and real and A, by its denition,
Av = v
is equivalent to
A Rev = A Rev and A Imv = A Imv.
The proof is completed by noting that the eigenvalues of

A
(n1)(n1)
sat-
isfying
P
AP =
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
A
(n1)(n1)
_
_
are also the eigenvalues of A
nn
; hence, they are all reals. So, by the validity
of the claimed statement for (n 1) (n 1) matrices, the existence of
real

Q
(n1)(n1)
and real T
(n1)(n1)
satisfying
A
(n1)(n1)
=

Q
(n1)(n1)

T
(n1)(n1)

Q
(n1)(n1)
and
(n1)(n1)
=

Q
1
(n1)(n1)
is conrmed. 2
Two important facts:
P
AP and A have the same eigenvalues but possibly dierent eigenvectors. A

simple proof is that for v = Pv
/
,
(P
AP)v
/
= P
Av = P
(v) = P
v = v
/
.
(Section 6.1: Problem 26 or see slide 6-22)
det(A) = det
__
B C
0 D
__
= det(B) det(C)
Thus,
P
AP =
_
_
t
1,1
t
1,2
t
1,n
0
.
.
.
0
A
(n1)(n1)
_
_
So the eigenvalues of

A should be the eigenvalues of P
AP.
Theorem (Spectral theorem or principal axis theorem) A symmetric
matrix A (not necessarily with distinct eigenvalues) can be written as
A = QQ
1
= QQ
T
with Q
1
= Q
T
Proof:
A symmetric matrix certainly satises Schurs Theorem.
A = QTQ
= Q
1
.
A has real eigenvalues. So, both T and Q are reals.
By A
T
= A, we have
A
T
= QT
T
Q
T
= QTQ
T
= A.
This immediately gives
T
T
= T
which implies the o-diagonals are zeros.
By AQ = QT, equivalently,
_
_
Aq
1
= t
1,1
q
1
Aq
2
= t
2,2
q
2
.
.
.
Aq
n
= t
n,n
q
n
we know that Q is the matrix of eigenvectors (and there are n of them) and T
is the matrix of eigenvalues. 2
This result immediately indicates that a symmetric A can always be diagonalized.
Summary
A symmetric matrix has real eigenvalues and n real orthogonal eigenvec-
tors.
(Problem 21, Section 6.4)(Recommended) This matrix M is skew-symmetric and
also . Then all its eigenvalues are pure imaginary and they also have
[[ = 1. (|Mx| = |x| for every x so |x| = |x| for eigenvectors.) Find all
four eigenvalues from the trace of M:
M =
1
3
_
_
0 1 1 1
1 0 1 1
1 1 0 1
1 1 1 0
_
_
can only have eigenvalues or .
Thinking over Problem 14: The eigenvalues of an orthogonal matrix satises
[[ = 1.
Proof:
|Qv|
2
= (Qv)
Qv = v
Qv = v
v = |v|
2
implies
|v| = |v|.
2
[[ = 1 and pure imaginary implies = .
(Problem 15, Section 6.4) Show that A (symmetric but complex) has only
one line of eigenvectors:
A =
_
1
1
_
is not even diagonalizable: eigenvalues = 0, 0.
A
T
= A is not such a special property for complex matrices. The good property
is

A
T
= A (Section 10.2). Then all s are real and eigenvectors are orthogonal.
Thinking over Problem 15: That a symmetric matrix A satisfying A
T
= A has
real eigenvalues and n orthogonal eigenvectors is only true for real symmetric
matrices.
For a complex matrix A, we need to rephrase it as A Hermitian symmetric
matrix A satisfying A
= A has real eigenvalues and n orthogonal eigenvectors.

Inner product and norm for complex vectors:
v w = v
w and |v|
2
= v
v
Proof of the red-color claim: Suppose Av = v. Then,
Av = v
= (Av)
v = (v)
v (i.e., (

A v)
T
v = (
v)
T
v)
= v
v =
v
= v
Av =
v
= v
v =
v
= |v|
2
=
|v|
2
= =
2
(Problem 28, Section 6.4) For complex matrices, the symmetry A
T
= A that
produces real eigenvalues changes to

A
T
= A. From det(A I) = 0, nd the
eigenvalues of the 2 by 2 Hermitian matrix A =
_
4 2 + ; 2 0
=

A
T
. To
see why eigenvalues are real when

A
T
= A, adjust equation (1) of the text to
A x =

x. (See the green box above.)
(Problem 27, Section 6.4) (MATLAB) Take two symmetric matrices with dierent
eigenvectors, say A =
_
1 0
0 2
_
and B =
_
8 1
1 0
_
. Graph the eigenvalues
1
(A+tB)
and
2
(A + tB) for 8 < t < 8. Peter Lax says on page 113 of Linear Algebra
that
1
and
2
appear to be on a collision course at certain values of t. Yet at
the last minute they turn aside. How close do they come?
Correction for Problem 27: The problem should be . . . Graph the eigenvalues
1
and
2
of A + tB for 8 < t < 8. . . ..
Hint: Draw the pictures of
1
(t) and
2
(t) with respect to t and check
1
(t)
2
(t).
(Problem 29, Section 6.4) Normal matrices have

A
T
A = A

A
T
. For real matrices, A
T
A = AA
T
includes symmetric, skew symmetric, and orthogonal matrices. Those have real , imaginary
and [[ = 1. Other normal matrices can have any complex eigenvalues .
Key point: Normal matrices have n orthonormal eigenvectors. Those vectors x
i
probably will have complex components. In that complex case orthogonality means x
T
i
x
j
= 0
as Chapter 10 explains. Inner products (dot products) become x
T
y.
The test for n orthonormal columns in Q becomes

Q
T
Q = I instead of Q
T
Q = I.
A has n orthonormal eigenvectors (A = Q

Q
T
) if and only if A is normal.
(a) Start from A = Q

Q
T
with

Q
T
Q = I. Show that

A
T
A = A

A
T
.
(b) Now start from

A
T
A = A

A
T
. Schur found A = QT

Q
T
for every matrix A, with a triangular
T. For normal matrices we must show (in 3 steps) that this T will actually be diagonal.
Then T = .
Step 1. Put A = QT

Q
T
into

A
T
A = A

A
T
to nd

T
T
T = T

T
T
.
Step 2: Suppose T =
_
a b
0 d
_
has

T
T
T = T

T
T
. Prove that b = 0.
Step 3: Extend Step 2 to size n. A normal triangular T must be diagonal.
Important conclusion from Problem 29: A matrix A has n orthogonal eigenvec-
tors if, and only if, A is normal.
Denition (Normal matrix): A matrix A is normal if A
A = A
A.
Proof:
Schurs theorem: A = QTQ
= Q
1
.
A
A = AA
= T
T = TT
= T diagonal = Qmatrix of eigenvectors

by AQ = QT. 2
6.5 Positive denite matrices 6-83
Denition (Positive denite): A symmetric matrix A is positive de-
nite if its eigenvalues are all positive.
The above denition only applies to a symmetric matrix because a non-symmetric
matrix may have complex eigenvalues (which cannot be compared with zero)!
Properties of positive denite (symmetric) matrices
Equivalent Denition: A is positive denite if, and only if, x

T
Ax > 0 for
all non-zero x.
x
T
Ax is usually referred to as the quadratic form!
The proofs can be found in Problems 18 and 19.
(Problem 18, Section 6.5) If Ax = x then x
T
Ax = . If x
T
Ax > 0,
prove that > 0.
(Problem 19, Section 6.5) Reverse Problem 18 to show that if all > 0 then
x
T
Ax > 0. We must do this for every nonzero x, not just the eigenvectors.
So write x as a combination of the eigenvectors and explain why all cross
terms are x
T
i
x
j
= 0. Then x
T
Ax is
(c
1
x
1
+ +c
n
x
n
)
T
(c
1
1
x
1
+ +c
n
n
x
n
) = c
2
1
1
x
T
1
x
1
+ +c
2
n
n
x
T
n
x
n
> 0.
Proof (Problems 18):
(only if part : Problem 19) Av
i
=
i
v
i
implies v
T
i
Av
i
= v
T
i
v
i
> 0 for all
eigenvalues
i
n
i=1
and eigenvectors v
i
n
i=1
. The proof is completed by
noting that with x =
n
i=1
c
i
v
i
and v
i
n
i=1
orthogonal,
x
T
(Ax) =
_
n
i=1
c
i
v
T
i
_
_
_
n
j=1
c
j
j
v
j
_
_
=
n
i=1
c
2
i
i
|v
i
|
2
> 0.
(if part : Problem 18) Taking x = v
i
, we obtain that v
T
i
(Av
i
) = v
T
i
(v
i
) =
i
|v
i
|
2
> 0, which implies
i
> 0. 2
Based on the above proposition, we can conclude similar to two positive scalars
(as if a and b are both positive, so is a + b) that
Proposition: If A and B are both positive denite, so is A + B.
The next property provides an easy way to construct positive denite matrices.
Proposition: If A
nn
= R
T
nm
R
mn
and R
mn
has linearly independent
columns, then A is positive denite.
Proof:
x non-zero = Rx non-zero.
Then, x
T
(R
T
R)x = (Rx)
T
(Rx) = |Rx|
2
> 0. 2
Since (symmetric) A = QQ
T
, we can choose R = Q
1/2
Q
T
, which requires
R to be a square matrix. This observation gives another equivalent denition
of positive denite matrices.
Equivalent Denition: A
nn
is positive denite if, and only if, there exists
R
mn
with independent columns such that A = R
T
R.
nn
is positive denite if, and only if, all pivots
are positive.
Proof:
(By LDU decomposition,) A = LDL
T
, where D is a diagonal matrix with
pivots as diagonals, and L is a lower triangular matrix with 1 as diagonals.
Then,
x
T
Ax = x
T
(LDL
T
)x = (L
T
x)
T
D(L
T
x)
= y
T
Dy where y = L
T
x =
_
_
l
T
1
x
.
.
.
l
T
n
x
_
_
= d
1
y
2
1
+ + d
n
y
2
n
= d
1
_
l
T
1
x
_
2
+ + d
n
_
l
T
n
x
_
2
.
So, if all pivots are positive, x
T
Ax > 0 for all non-zero x. Conversely, if
x
T
Ax > 0 for all non-zero x, which in turns implies y
T
Dy > 0 for all
non-zero y (as L is invertible), then pivot d
i
must be positive for the choice
of y
j
= 0 except for j = i. 2
An extension of the previous proposition is:
Suppose A = BCB
T
with B invertible and C diagonal. Then, A is positive
denite if, and only if, diagonals of C are all positive! See Problem 35.
(Problem 35, Section 6.5) Suppose C is positive denite (so y
T
Cy > 0 when-
ever y ,= 0) and A has independent columns (so Ax ,= 0 whenever x ,= 0).
Apply the energy test to x
T
A
T
CAx to show that A
T
CA is positive denite:
the crucial matrix in engineering.
nn
is positive denite if, and only if, the n upper
left determinants (i.e., the n leading principle minors) are all positive.
Denition (Minors, Principle minors and leading principle mi-
nors):
A minor of a matrix A is the determinant of some smaller square matrix,
obtained by removing one or more of its rows or columns.
The rst-order (respectively, second-order, etc) minor is a minor, obtained
by removing (n 1) (respectively, (n 2), etc) rows or columns.
A principle minor of a matrix is a minor, obtained by removing the same
rows and columns.
A leading principle minor of a matrix is a minor, obtaining by removing
the last few rows and columns.
The below example gives three upper left determinants (or three leading
principle minors), det(A
11
), det(A
22
) and det(A
33
).
a
1,1
a
1,2
a
2,1
a
2,2
a
1,3
a
2,3
a
3,1
a
3,2
a
3,3
Proof of the equivalent denition: This can be proved based on slide 5-18:
By LU decomposition,
A =
_
_
1 0 0 0
l
2,1
1 0 0
l
3,1
l
3,2
1 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
l
n,1
l
n,2
l
n,3
1
_
_
_
_
d
1
u
1,2
u
1,3
u
1,n
0 d
2
u
2,3
d
2,n
0 0 d
3
d
3,n
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 0 0 d
n
_
_
= d
k
=
det(A
kk
)
det(A
(k1)(k1)
)
2
nn
is positive denite if, and only if, all (2
n
2)
principle minors are positive.
Based on the above equivalent denitions, a positive denite matrix cannot
have either zero or negative value in its main diagonals. See Problem 16.
(Problem 16, Section 6.5) A positive denite matrix cannot have a zero (or even
worse, a negative number) on its diagonal. Show that this matrix fails to have
x
T
Ax > 0:
_
x
1
x
2
x
3
_
_
4 1 1
1 0 2
1 2 5
_
_
_
_
x
1
x
2
x
3
_
_
is not positive when (x
1
, x
2
, x
3
) = ( , , ).
With the above discussion of positive denite matrices, we can proceed to de-
ne similar notions like negative denite, positive semidenite, negative
semidefnite matrices.
6.5 Positive semidenite (or nonnegative denite) 6-92
Denition (Positive semidenite): A symmetric matrix A is positive
semidenite if its eigenvalues are all nonnegative.
Equivalent Denition: A is positive semidenite if, and only if, x
T
Ax0 for
all non-zero x.
nn
is positive semidenite if, and only if, there exists
R
mn
(perhaps with dependent columns) such that A = R
T
R.
nn
is positive semidenite if, and only if, all pivots
are nonnegative.
nn
is positive semidenite if, and only if, all principle
minors are non-negative.
Example. Non-positive semidenite A =
_
_
1 0 0
0 0 0
0 0 1
_
_
has (2
3
2) principle minors.
_
1 0
0 0
_
,
_
1 0
0 1
_
,
_
0 0
0 1
_
,
_
1
,
_
0
,
_
1
.
Note: It is not sucient to dene the positive semidenite based on the non-
negativity of leading principle minors. Check the above example.
6.5 Negative denite, negative semidenite, indenite 6-93
We can similarly dene negative denite.
Denition (Negative denite): A symmetric matrix A is negative def-
inite if its eigenvalues are all negative.
Equivalent Denition: A is negative denite if, and only if, x
T
Ax<0 for all
non-zero x.
nn
is negative denite if, and only if, there exists
R
mn
with independent columns such that A = R
T
R.
nn
is negative denite if, and only if, all pivots are
negative.
nn
is negative denite if, and only if, all odd-order
leading principle minors are negative and all even-order leading principle minors
are positive.
nn
principle minors are negative and all even-order principle minors are positive.
6.5 Negative denite, negative semidenite, indenite 6-94
We can also similarly dene negative semidenite.
Denition (Negative denite): A symmetric matrix A is negative
semidenite if its eigenvalues are all non-positive.
Equivalent Denition: A is negative semidenite if, and only if, x
T
Ax0 for
all non-zero x.
nn
is negative semidenite if, and only if, there
exists R
mn
(possibly with dependent columns) such that A = R
T
R.
nn
is negative semidenite if, and only if, all pivots
are non-positive.
nn
principle minors are non-positive and all even-order principle minors are non-
negative.
Note: It is not sucient to dene the negative semidenite based on the non-
positivity of leading principle minors.
Finally, if some of the eigenvalues of a symmetric matrix are positive, and some are
negative, the matrix will be referred to as indenite.
6.5 Quadratic form 6-95
For a positive denite matrix A
22
,
x
T
Ax = x
T
(QQ
T
)x = (Q
T
x)
T
(Q
T
x)
= y
T
y where y = Q
T
x =
_
q
T
1
x
q
T
2
x
_
=
1
_
q
T
1
x
_
2
+
2
_
q
T
2
x
_
2
So, x
T
Ax = c gives an ellipse if c > 0.
Example. A =
_
5 4
4 5
_

1
= 9 and
2
= 1, and q
1
=
1
2
_
1
1
_
and q
2
=
1
2
_
1
1
_
.
x
T
Ax =
1
(q
T
1
x)
2
+
2
(q
T
2
x)
2
= 9
_
x
1
+ x
2
2
_
2
+
_
x
1
x
2
2
_
2
_
q
T
1
x is an axis perpendicular to q
1
. (Not along q
1
as the textbook said.)
q
T
2
x is an axis perpendicular to q
2
.
Tip: A = QQ
T
is called the principal axis theorem (cf. slide 6-66) because
x
T
Ax = y
T
y = c is an ellipse with axes along the eigenvectors.
(Problem 13, Section 6.5) Find a matrix with a > 0 and c > 0 and a + c > 2b
that has a negative eigenvalue.
Missing point in Problem 13: The matrix to be determined is of the shape
_
a b
b c
_
.
6.6 Similar matrices 6-97
Denition (Similarity): Two matrices A and B are similar if B = M
1
AM
for some invertible M.
Theorem: A and M
1
AM have the same eigenvalues.
Proof: For eigenvalue and eigenvector v of A, dene v
/
= M
1
v (hence, v =
Mv
/
). We then derive
(M
1
AM)v
/
= M
1
A(Mv
/
) = M
1
Av = M
1
v = v
/
So, is also the eigenvalue of M
1
AM (associated with eigenvector v
/
= M
1
v).
2
Notes:
The LU decomposition over a symmetric matrix A gives A = LDL
T
but A and
D (pivot matrix) apparently may have dierent eigenvalues. Why? Because
(L
T
)
1
,= L. Similarity is dened based on M
1
, not M
T
.
The converse to the above theorem is wrong!. In other words, we cannot
say that two matrices with the same eigenvalues are similar.
Example. A =
_
0 1
0 0
_
and B =
_
0 0
0 0
_
have the same eigenvalues 0, 0, but
they are not similar.
6.6 Jordan form 6-98
Why introducing matrix similarity?
Answer: We can then extend the diagonalization of a matrix with less than
n eigenvectors to the Jordan form.
For an un-diagonalizable matrix A, we can nd invertible M such that
A = MJM
1
,
where
J =
_
_
J
1
0 0
0 J
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 J
s
_
_
and s is the number of distinct eigenvalues, and the size of J
i
is equal to the
multiplicity of eigenvalue
i
, and J
i
is of the form
J
i
=
_
i
1 0 0
0
i
1 0
0 0
i
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 0 0
i
_
_
6.6 Jordan form 6-99
The idea behind the Jordan form is that A is similar to J.
Based on this, we can now compute
A
100
= MJ
100
M
1
= M
_
_
J
100
1
0 0
0 J
100
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 J
100
s
_
_
M
1
.
How to nd M
i
corresponding to J
i
?
Answer: By AM = MJ with M =
_
M
1
M
2
M
s
, we know
AM
i
= M
i
J
i
for 1 i s.
Specically, assume that the multiplicity of
i
is two. Then,
A
_
v
(1)
i
v
(2)
i
_
=
_
v
(1)
i
v
(2)
i
_
_
i
1
0
i
_
which is equivalent to
_
Av
(1)
i
=
i
v
(1)
i
Av
(2)
i
= v
(1)
i
+
i
v
(2)
i
=
_
(A
i
I)v
(1)
i
= 0
(A
i
I)v
(2)
i
= v
(1)
i
(Problem 14, Section 6.6) Prove that A
T
is always similar to A (we know the s
are the same):
1. For one Jordan block J
i
: Find M
i
so that M
1
i
J
i
M
i
= J
T
i
.
2. For any J with blocks J
i
: Build M
0
from blocks so that M
1
0
JM
0
= J
T
.
3. For any A = MJM
1
: Show that A
T
is similar to J
T
and so to J and to A.
Thinking over Problem 14: A
T
and A are always similar.
Answer:
It can be easily checked that
_
_
u
1,1
u
1,2
u
1,n
u
2,1
u
2,2
u
2,n
.
.
.
.
.
.
.
.
.
.
.
.
u
n,1
u
n,2
u
n,n
_
_
=
_
_
0 0 1
.
.
. 1 0
.
.
.
.
.
.
.
.
.
1 0 0
_
_
_
_
u
n,n
u
n,2
u
n,1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. u
2,2
u
2,1
u
1,n
u
1,2
u
1,1
_
_
_
_
0 0 1
.
.
. 1 0
.
.
.
.
.
.
.
.
.
1 0 0
_
_
= V
1
_
_
u
n,n
u
n,2
u
n,1
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. u
2,2
u
2,1
u
1,n
u
1,2
u
1,1
_
_
V
where here V represents a matrix with zero entries except v
i,n+1i
= 1 for
1 i n. We note that V
1
= V
T
= V .
So, we can use the proper size of M
i
= V to obtain
J
T
i
= M
1
i
J
i
M
i
.
Dene
M =
_
_
M
1
0 0
0 M
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 M
s
_
_
We have
J
T
= MJM
1
,
where
J =
_
_
J
1
0 0
0 J
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 J
s
_
_
and s is the number of distinct eigenvalues, and the size of J
i
is equal to the
multiplicity of eigenvalue
i
, and J
i
is of the form
J
i
=
_
i
1 0 0
0
i
1 0
0 0
i
0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 0 0
i
_
_
Finally, we know
A is similar to J;
A
T
is similar to J
T
;
J
T
is similar to J.
So, A
T
is similar to A. 2
(Problem 19, Section 6.6) If A is 6 by 4 and B is 4 by 6, AB and BA have dierent
sizes. But with blocks,
M
1
FM =
_
I A
0 I
_ _
AB 0
B 0
_ _
I A
0 I
_
=
_
0 0
B BA
_
= G.
(a) What sizes are the four blocks (the same four sizes in each matrix)?
(b) This equation is M
1
FM = G, so F and G have the same eigenvalues. F
has the 6 eigenvalues of AB plus 4 zeros; G has the 4 eigenvalues of BA plus
6 zeros. AB has the same eigenvalues as BA plus zeros.
Thinking over Problem 19: A
mn
B
nm
and B
nm
A
mn
have the same eigenval-
ues except for additional (mn) zeros.
Solution: The example shows the usefulness of the similarity.
_
I
mm
A
mn
0
nm
I
nn
_ _
A
mn
B
nm
0
mn
B
nm
0
nn
_ _
I
mm
A
mn
0
nm
I
nn
_
=
_
0
mm
0
mn
B
nm
B
nm
A
mn
_
Hence,
_
A
mn
B
nm
0
mn
B
nm
0
nn
_
and
_
0
mm
0
mn
B
nm
B
nm
A
mn
_
are similar and have the
same eigenvalues.
From Problem 26 in Section 6.1 (see slide 6-22), the desired claim is proved.2
(Problem 21, Section 6.6) If J is the 5 by 5 Jordan block with = 0, nd J
2
and
count its eigenvectors (are these the eigenvectors?) and nd its Jordan form (there
will be two blocks).
Problem 21: Find the Jordan form of A = J
2
, where
J =
_
_
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
0 0 0 0 0
_
_
.
Solution.
A = J
2
=
_
_
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
0 0 0 0 0
0 0 0 0 0
_
_
, and the eigenvalues of A are 0, 0, 0, 0, 0.
Av
1
= A
_
_
v
1,1
v
2,1
v
3,1
v
4,1
v
5,1
_
_
=
_
_
v
3,1
v
4,1
v
5,1
0
0
_
_
= 0 = v
3,1
= v
4,1
= v
5,1
= 0 = v
1
=
_
_
v
1,1
v
2,1
0
0
0
_
_
.
Av
2
= A
_
_
v
1,2
v
2,2
v
3,2
v
4,2
v
5,2
_
_
=
_
_
v
3,2
v
4,2
v
5,2
0
0
_
_
= v
1
=
_
_
v
3,2
= v
1,1
v
4,2
= v
2,1
v
5,2
= 0
= v
2
=
_
_
v
1,2
v
2,2
v
1,1
v
2,1
0
_
_
Av
3
= A
_
_
v
1,3
v
2,3
v
3,3
v
4,3
v
5,3
_
_
=
_
_
v
3,3
v
4,3
v
5,3
0
0
_
_
= v
2
=
_
_
v
3,3
= v
1,2
v
4,3
= v
2,2
v
5,3
= v
1,1
v
2,1
= 0
= v
3
=
_
_
v
1,3
v
2,3
v
1,2
v
2,2
v
1,1
_
_
To summarize,
v
1
=
_
_
v
1,1
v
2,1
0
0
0
_
_
= v
2
=
_
_
v
1,2
v
2,2
v
1,1
v
2,1
0
_
_
= If v
2,1
= 0, then v
3
=
_
_
v
1,3
v
2,3
v
1,2
v
2,2
v
1,1
_
_
_
_
v
(1)
1
=
_
_
1
0
0
0
0
_
_
= v
(1)
2
=
_
_
v
(1)
1,2
v
(1)
2,2
1
0
0
_
_
= v
(1)
3
=
_
_
v
(1)
1,3
v
(1)
2,3
v
(1)
1,2
v
(1)
2,2
1
_
_
v
(2)
1
=
_
_
0
1
0
0
0
_
_
= v
(2)
2
=
_
_
v
(2)
1,2
v
(2)
2,2
0
1
0
_
_
Since we wish to choose each of v
(1)
2
and v
(2)
2
to be orthogonal to both v
(1)
1
and
v
(2)
1
,
v
(1)
1,2
= v
(1)
2,2
= v
(2)
1,2
= v
(2)
2,2
= 0,
i.e.,
_
_
v
(1)
1
=
_
_
1
0
0
0
0
_
_
= v
(1)
2
=
_
_
0
0
1
0
0
_
_
= v
(1)
3
=
_
_
v
(1)
1,3
v
(1)
2,3
0
0
1
_
_
v
(2)
1
=
_
_
0
1
0
0
0
_
_
= v
(2)
2
=
_
_
0
0
0
1
0
_
_
Since we wish to choose v
(1)
3
to be orthogonal to all of v
(1)
1
, v
(2)
1
, v
(1)
2
and v
(2)
2
,
v
(1)
1,3
= v
(1)
2,3
= 0.
As a result,
A
_
v
(1)
1
v
(1)
2
v
(1)
3
v
(2)
1
v
(2)
2
_
=
_
v
(1)
1
v
(1)
2
v
(1)
3
v
(2)
1
v
(2)
2
_
_
_
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 1
0 0 0 0 0
_
_
2
Are all v
(1)
1
, v
(1)
2
, v
(1)
3
, v
(2)
1
and v
(2)
2
eigenvectors of A (satisfying Av = v)?
Hint: A does not have 5 eigenvectors. More specically, A only has 2 eigen-
vectors? Which two? Think of it.
Hence, the problem statement may not be accurate as I mark (are these the
eigenvectors?).
6.7 Singular value decomposition (SVD) 6-110
Now we can represent all square matrix in the Jordan form:
A = M
1
JM.
What if the matrix A
mn
is not a square one?
Problem: Av = v is not possible! Specically, v cannot have two dierent
dimensionalities.
A
mn
v
n1
= v
n1
infeasible if n ,= m.
So, we can only have
A
mn
v
n1
= u
m1
.
If we can nd enough numbers of orthogonal u and v such that
A
mn
_
v
1
v
2
v
n
nn
=
_
u
1
u
2
u
m
mm
_
1
0 0 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0
r
0 0
0 0 0 0
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
0 0 0 0
_
_
mn
.
where r is the rank of A, then we can perform the so-called singular value
decomposition (SVD)
A = UV
1
.
Note again that the required enough number of orthogonal u and v may be
impossible when A has repeated eigenvalues.
If V is chosen to be an orthogonal matrix satisfying V
1
= V
T
, then we have
the so-called reduced SVD
A = UV
T
=
r
i=1
i
u
i
v
T
i
=
_
u
1
u
r
mr
_
_
1
0
.
.
.
.
.
.
.
.
.
0
r
_
_
rr
_
_
v
T
1
.
.
.
v
T
r
_
_
rn
Usually, we prefer to choose an orthogonal V (as well as orthogonal U).
In the sequel, we will assume the found U and V are orthogonal matrices in
the rst place; later, we will conrm that orthogonal U and V can always be
found.
6.7 How to determine U, V and
i
? 6-113
A = UV
T
= A
T
A =
_
UV
T
_
T
_
UV
T
_
= V
T
U
T
UV
T
= V
2
V
T
.
So, V is the (orthogonal) matrix of n eigenvectors of symmetric A
T
A.
A = UV
T
= AA
T
=
_
UV
T
_ _
UV
T
_
T
= UV
T
V
T
U
T
= U
2
U
T
.
So, U is the (orthogonal) matrix of m eigenvectors of symmetric AA
T
.
Remember that A
T
A and AA
T
have the same eigenvalues except for additional
(mn) zeros.
Section 6.6, Problem 19 (see slide 6-104): A
mn
B
nm
and B
nm
A
mn
have
the same eigenvalues except for additional (mn) zeros.
In fact, there are only r non-zero eigenvalues for A
T
A and AA
T
, which satisfy
i
=
2
i
, where 1 i r.
i
? 6-114
Example (Problem 21 in Section 6.6): Find the SVD of A = J
2
, where
J =
_
_
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
0 0 0 0 0
_
_
.
Solution.
A
T
A =
_
_
0 0 0 0 0
0 0 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
_
_
and AA
T
=
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
_
_
.
So V =
_
_
0 0 0 1 0
0 0 0 0 1
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
_
_
and U =
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
_
_
for =
2
=
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
_
_
.
i
? 6-115
Hence,
_
_
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
0 0 0 0 0
0 0 0 0 0
_
_
. .
A
=
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
_
_
. .
U
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
_
_
. .
_
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
1 0 0 0 0
0 1 0 0 0
_
_
. .
V
T
2
We may compare this with the Jordan form.
_
_
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
0 0 0 0 0
0 0 0 0 0
_
_
. .
A
=
_
_
1 0 0 0 0
0 0 0 1 0
0 1 0 0 0
0 0 0 0 1
0 0 1 0 0
_
_
. .
M
_
_
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 1
0 0 0 0 0
_
_
. .
J
_
_
1 0 0 0 0
0 0 1 0 0
0 0 0 0 1
0 1 0 0 0
0 0 0 1 0
_
_
. .
M
1
i
? 6-116
Remarks on the previous example
In the above example, we only know =
2
=
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
_
_
.
Hence, =
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
_
_
.
However, by Av
i
=
i
u
i
= (
i
)(u
i
), we can always choose positive
i
by
adjusting the sign of u
i
.
6.6 SVD 6-117
Important notes:
In terminology,
i
, where 1 i r, is called the singular value.
Hence, the name of singular value decomposition is used.
The singular value is always non-zero (even though the eigenvalues of A can
be zeros).
Example. A =
_
_
0 0 1 0 0
0 0 0 1 0
0 0 0 0 1
0 0 0 0 0
0 0 0 0 0
_
_
but =
_
_
1 0 0 0 0
0 1 0 0 0
0 0 1 0 0
0 0 0 0 0
0 0 0 0 0
_
_
.
It is good to know that:
The rst r columns of V row space R(A) and are bases of R(A)
The last (n r) columns of V null space N(A) and are bases of N(A)
The rst r columns of U column space C(A) and are bases of C(A)
The last (mr) columns of U left null space N(A
T
) and are bases of N(A
T
)
How useful the above facts are can be seen from the next example.
6.6 SVD 6-118
Example. Find the SVD of A
43
= x
41
y
T
13
.
Solution.
A is a rank-1 matrix. So, r = 1 and =
_
1
0 0
0 0 0
0 0 0
0 0 0
_
_
, where
1
,= 0. (Actually,
1
= 1.)
The base of the row space is v
1
=
y
|y|
; pick up perpendicular v
2
and v
3
that
span the null space.
The base of the column space is u
1
=
x
|x|
; pick up perpendicular u
2
, u
3
, u
4
that span the left null space.
The SVD is then
A
43
=
_
x
|x|
..
column space
u
2
u
3
u
4
. .
left null space
_
44
43
_
_
y
T
|y|
v
T
2
v
T
3
_
_
33
row space
_
null space
2

Eigenvalues and Eigenvectors

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Eigenvalues and Eigenvectors

Uploaded by

Copyright:

Available Formats

Chapter 6

Eigenvalues and Eigenvectors

Property 1: The eigenvectors stay the same for every power of A.

Property 2: If the null space of A

Property 3: Assume with no loss of generality [

= [A[ [D[ but

= [A[ [D[ [C[ [B[.

from the second row

. This leaves the Schur complement

but S has no inverse. 2

are both the eigenvectors of A corresponding to the

2, 0) as a combination of the eigenvectors. Solve both equations

AP and A have the same eigenvalues but possibly dierent eigenvectors. A

with T upper triangular and Q

= A has real eigenvalues and n orthogonal eigenvectors.

with T upper triangular and Q

= T diagonal = Qmatrix of eigenvectors

Equivalent Denition: A is positive denite if, and only if, x

You might also like