=

-1) 1

[ 3 1 1J
2

v'6 +
v'6 _1
v'2

-1

-1 DJ

VS VS

= VU and A2 = v1O, we find that the singular-value decomposition of

102 Chapter 2 Matrix Algebra and Random Vectors

Ais

Taking Al

bijf = tr[(A - B)(A - B)']

2: AiDi v;

k and singular

The equality may be checked by carrying out the operations on the right-hand side.
The singular-value decomposition is closely connected to a result concerning
the approximation of a rectangular matrix by a lower-dimensional matrix, due to
Eckart and Young ([2]). If a m X k matrix A is approximated by B, having the same
dimension but lower rank, the sum of squared differences

2: 2: (aij -

i=1 j=1

Result 2A.16. Let A be an m X k matrix of real numbers with m

value decomposition VAV'. Lets < k = rank (A). Then
B
i=1

is the rank-s least squares approximation to A. It minimizes

tr[(A - B)(A - B)')

;=s+1

2:

AT.

Cij? =

2:2:
CTj
i"j

to write the sum of

Cii)2

i=1

2: Ai Di vi

'* j and cns = Ai for

2: (Ai = Ofor i

i=1

over all m X k matrices B having rank no greater than s. The minimum value, or
error of approximation, is

= Im and VV' = Ik

= tr[UV'(A - B)VV'(A - B)')

To establish this result, we use vV'

squares as
tr[(A - B)(A - B)'j

i=1 j=1

2: 2: (Aij -

= tr[V'(A - B)VV'(A - B)'V)

- C)(A - C)') =

= V'BV. Clearly, the minimum occurs when Cij

= tr[(A
where C

the s largest singular values. The other Cu = O. That is, UBV' = As or B =

Letx' = [5, 1, 3] andy' = [-1,

. (a) Graph the two vectors.

Exercises
2.1.

3,

1].

= B'A' .

J U J
(kxt)

and B , (AB)'

and

Exercises

103

(b)
(i)
length of x, (ii) the angle between x and y, and (iii) the projection of y on x.
(c) Smce x = 3 and y = 1, graph [5 - 3,1 - 3,3 - 3] = [2 -2 DJ and
[-1-1,3-1,1-1J=[-2,2,OJ.
'
,

2.2. Given the matrices

perform the indicated multiplications.

(a) 5A
(b) BA
(c) A'B'
(d) C'B
(e) Is AB defined?

2.3. Verify the following properties of the transpose when

A

(a) (A')' = A
(b) (C,)-l = (C- I )'
(c) (AB)' = B' A'
(mXk)

(d) For general A

2,4. When A-I and B- 1 exist, prove each of the following.

.
(a) (A,)-l = (A-I),
(b) (AB)-I = B-IA- I

Q =

5
12J
IT
IT
12
5
-IT IT

Hint: Part a can be proved br noting that AA-I = I, I'; 1', and (AA-i)' = (A-I),A'.
Part b follows from (B- 1A- )AB = B-I(A-IA)B = B-IB = I.

2.5. Check that

is an orthogonal matrix.

2.6. Let

(a) Is A symmetric?
(b) Show that A is positive definite.

104

Chapter 2 Matrix Algebra and Random Vectors

(d) Find the eigenvaiues and eigenvectors of A-I.

2.7. Let A be as given in Exercise 2.6.

(a) Determine the eigenvalues and eigenvectors of A.
(b) Write the spectral decomposition of A.
(c) Find A-I.
2.8. Given the matrix
A =

find the eigenvalues Al and A2 and the associated nonnalized eigenvectors el and e2.
Determine the spectral decomposition (2-16) of A.
2.9. Let A be as in Exercise 2.8.
(a) Find A-I.

4.001J
4.002
and

4
B = [ 4.001

4.001
4.002001

(b) Compute the eigenvalues and eigenvectors of A-I.

(c) Write the spectral decomposition of A-I, and compare it with that of A from
Exercise 2.8.
2.10. Consider the matrices

A = [:.001

These matrices are identical except for a small difference in the (2,2) position.
Moreover, the columns of A (and B) are nearly linearly dependent. Show that
A-I ='= (-3)B- I. Consequently, small changes-perhaps caused by rounding-can give
substantially different inverses.
2.11. Show that the determinant of the p X P diagonal matrix A = {aij} with aij = 0, i *- j,
is given by the product of the diagonal elements; thus, 1A 1 = a" a22 ... a p p.
Hint: By Definition 2A24, I A I = a" A" + 0 + ... + O. Repeat for the submatrix
All obtained by deleting the first row and first column of A.

rr;=1

= II I. Now

2.12. Show that the determinant of a square symmetric p x p matrix A can be expressed as
the product of its eigenvalues AI, A2, ... , Ap; that is, IA I =
Ai.
Hint: From (2-16) and (2-20), A = PAP' with P'P = I. From Result 2A.1I(e),
lA I = IPAP' I = IP IIAP' I = IP 11 A liP' I = I A 1111, since III = IP'PI = IP'IIPI. Apply
Exercise 2.11.

(pXp)

Q and A have the same eigenvalues if Q is orthogonal.

2.13. Show that I Q I = + 1 or -1 if Q is a p X P orthogonal matrix.

Hint: I QQ' I = II I. Also, from Result 2A.11, IQ" Q' I = IQ 12. Thus, IQ 12
use Exercise 2.11.

(pXp)(pXp)(pxp)

2.14. Show that Q'

Hint: Let A be an eigenvalue of A. Then 0 = 1A - AI I. By Exercise 2.13 and Result

2A.11(e), we can write 0 = IQ' 11 A - AlII Q I = IQ' AQ - All, since Q'Q = I.

X P

matrix. Show

2.1 S. A quadratic form x' A x is said to be positive definite if the matrix A is positive definite.
- 2XIX2 positive definite?
.
Is the quadratic form 3xt +
2.16. Consider an arbitrary n X p matrix A. Then A' A is a symmetric p
that A' A is necessarily nonnegative definite.
Hint: Set y = A x so that y'y = x' A' A x.

Exercises

105

2.17. Prove that every eigenvalue of a k x k positive definite matrix A is positive.

Hint: Consider the definition of an eigenvalue, where Ae = Ae. Multiply on the left by
e' so that e' Ae = Ae' e.
2

2.18. Consider the sets of points (XI, x2) whose "distances" from the origin are given by

= ;=1

VA;eie; = PA J/ 2P',wherePP'

= P'P

'

I. (The A.'s and the e.'s are

c = 4xt +
- 2v'2XIX2
2
2
for c = 1 and for c = 4. Determine the major and minor axes of the ellipses of constant distances and their associated lengths. Sketch the ellipses of constant distances and
comment on their pOSitions. What will happen as c2 increases?

(mXm)

2.19. Let AI/2

the eigenvalues and associated normalized eigenvectors of the matrix A.) Show Properties
(1)-(4) of the square-root matrix in (2-22).

2.20. Determine the square-root matrix AI/2, using the matrix A in Exercise 2.3. Also, deter. mine A-I/2, and show that A I/ 2A- I/2 = A- 1f2A1/ 2 = I.
2.21. (See Result 2AIS) Using the matrix

= [; 86 -98J

(a) Calculate A' A and obtain its eigenvalues and eigenvectors.

(b) Calculate AA' and obtain its eigenvalues and eigenvectors. Check that the nonzero
eigenvalues are the same as those in part a.
(c) Obtain the singular-value decomposition of A.
2.22. (See Result 2A1S) Using the matrix

(a) Calculate AA' and obtain its eigenvalues and eigenvectors.

(b) Calculate A' A and obtain its eigenvalues and eigenvectors. Check that the nonzero
eigenvalues are the same as those in part a.
(c) Obtain the
decomposition of A.

2.23. Verify the relationships V I/ 2pV I!2 = I and p = (Vlf2rII(VI/2rl, where I is the
p X .P
matrix
(2-32)], p is the p X P population correlatIOn matnx [EquatIOn (2-34)], and V /2 is the population standard deviation matrix
[Equation (2-35)].
2.24. Let X have covariance matrix

Find
(a) I-I
(b) The eigenvalues and eigenvectors of I.
(c) The eigenvalues and eigenvectors of I-I.

106 Chapter 2 Matrix Algebra and Random Vectors

2.25. Let X have covariance matrix

25
I = -2
[
4

-2 4]
4 1
1 9

(a) Determine p
V 1/2.
(b) Multiply your matrices to check the relation VI/2pVI/2 =
2.26. Use I as given in Exercise 2.25.

(a) Findpl3'
(b) Find the correlation between XI and

I.

2.27. Derive expressions for the mean and variances of the following linear combinations in
terms of the means and covariances of the random variables XI, X 2, and X 3.
(a) XI - 2X2
(b) -XI + 3X2
(c) XI + X 2 + X3
(e) XI + 2X2 - X3
(f) 3XI - 4X2 if XI and X 2 are independent random variables.
2.28. Show that

where Cl = [CJl, cl2, ... , Cl PJ and ci = [C2l> C22,' .. , C2 pJ. This verifies the off-diagonal
elements CIxC' in (2-45) or diagonal elements if Cl = C2'
Hint: By (2-43),ZI - E(ZI) = Cl1(XI - ILl) + '" + Clp(Xp - ILp) and
Z2 - E(Z2) = C21(XI - ILl) + ... + C2p(Xp - ILp).SOCov(ZI,Zz) =
E[(ZI - E(Zd)(Z2 - E(Z2J = E[(cll(XI - ILl) +
'" + CIP(Xp - ILp(C21(XI - ILd + C22(X2 - IL2) + ... + C2p(Xp - ILpJ.

C2m(Xm - ILm)

CJ(C2 m(Xe - ILe) (Xm - ILm)

cu(Xe - ILe)
p

(=1 m=1

2: 2:

+ Clp(Xp - IL p(C21(XI - ILl) + C22(X2 - IL2) + ... + C2p(Xp - ILp

(Cu(XI - ILl) + CdX2 - IL2) + .. ,

The product

has expected value

Verify the last step by the definition of matrix multiplication. The same steps hold for all
elements.

= [Xl> X 2, X 3, X 4, X5J

X (2)

[;;] .nd X'"

X =

2.29. Consider the arbitrary random vector X'

,.,: = [ILl> IL2. IL3, IL4, Jl.sJ Partition X into

where

xl"

Exercises

107

with mean vector

Let I be the covariance matrix of X with general element (Tik' Partition I into the
covariance matrices of X(l) and X(2) and the covariance matrix of an element of X(1)
and an element of X (2).

Ix =

2J

3 0

2 1

2 0

and

B =

C=n

2.30. You are given the random vector X' = [XI' X 2, X 3, X 4 J with mean vector
Jl.x = [4,3,2, 1J and variance-covariance matrix

Partition X as

Let
A = (1

B =

and consider the linear combinations AX(!) and BX(2). Find

(a) E(X(J)
(b) E(AX(l)
(c) Cov(X(l)
(d) COY (AX(!)
(e) E(X(2)
(f) E(BX(2)
(g) COY (X(2)
(h) Cov (BX(2)
(i) COY (X(l), X (2)
(j) COY (AX(J), BX(2)

-1 J and

2 .31. Repeat Exercise 2.30, but with A and B replaced by

A = [1

108

Chapter 2 Matrix Algebra and Random Vectors

=D

Ix =

-1

I
-2:

-1

-1

6
0

1
1

-1
1.

-1

and

B=

I
-2 -1

I
2:

2.32. You are given the random vector X' = [XI, X 2 , ... , Xs] with mean vector
IJ.'x = [2,4, -1,3,0] and variance-covariance matrix

Partition X as

Let
A

and

B =

[11 -12J

and consider the linear combinations AX(I) and BX(2). Find

(a) E(X(l)
(b) E(AX(I)
(c) Cov(X(1)
(d) COV(AX(l)
(e) E(X(2)
(f) E(BX(2)
(g) COy (X(2)
(h) Cov (BX(2)
(i) COy (X(l), X(2)
(j) COy (AX(I), BX(2)

-11 0J
3

2.33. Repeat Exercise 2.32, but with X partitioned as

and with A and B replaced by

A =

2.34. Consider the vectorsb' = [2, -1,4,0] and d' = [-1,3, -2, 1]. Verify the Cauchy-Schwan
inequality (b'd)2 s (b'b)(d'd).

Exercises

109

+ 6XIX2 for

2.3S. Using the

b' = [-4,3] and d' = [1,1]' verify the extended Cauchy-Schwarz
inequality (b'd) s (b'Bb)(d'B-1d) if

B= [ -22 -2J5

2.36. Fmd the maximum and minimum values of the quadratic form
all points x' = [x I , X2] such that x' x = 1.

2.37. With A as given in Exercise 2.6, fmd the maximum value of x' A x for x' x = 1.

A =

2 -2

C has (i,j)th entry

10

aicbckCkj

bCkCkj = dCj' So A(BC) has (i, j)th element

2.38. Find the maximum and minimum values of the ratio x' Ax/x'x for any nonzero vectors
x' = [Xl> X2, X3] if

2.39. Show that

(rXs)(sXt)(tXV)

Hint: BC has (e, j)th entry

aj{E(XCk)bkj

aieXCkbkj, and

2.40. Verify (2-24): E(X + Y) = E(X) + E(Y) and E(AXB) = AE(X)B.

Hint: X. + has Xij + Yij as its
element. Now,E(Xij + Yij ) = E(X ij ) + E(Yi)
by a umvanate property of expectation, and this last quantity is the (i, j)th element of

+ E(Y). Next (see Exercise 2.39),AXB has (i,j)th entry

by the additive property of expectation,
E(X)

aiCXCkbkj) =

e k e

which is the (i, j)th element of AE(X)B.

2.41. You are given the random vector X' = [Xl, X 2, X 3 , X 4 ] with mean vector
IJ.x = [3,2, -2,0] and variance-covariance matrix

[30

[1 -1

0
o 3 0
Ix = 0 0 3
o 0 0
Let
0
A = 1
1 -2
1
1
1
(a) Find E (AX), the mean of AX.
(b) Find Cov (AX), the variances and covariances ofAX.
(c) Which pairs of linear combinations have zero covariances?