CSCI 5304 Fall 2014

COMPUTATIONAL ASPECTS OF MATRIX THEORY

Class time : 2:30 3:45 pm MW
Room : MechE 212
Instructor : Daniel Boley
URL : www-users.itlabs.umn.edu/classes/Fall-2014/csci5304/
September 3, 2014
Thanks to Yousef Saad for the use of his class notes.
CSci 5304: header 0-1
GENERAL INTRODUCTION
Linear algebra and numerical linear algebra
Types of problems to be seen in this course
Mathematical background - matrices, eigenvalues, ...
Quick review of Matlab
Vector norms, matrix norms
CSci 5304: BG 1-1
Introduction
This course is about Matrix algorithms or matrix com-
putations
It involves: algorithms for standard matrix computations
(e.g. solving linear systems) - and their analysis (e.g., their
cost, numerical behavior, ..)
Matrix algorithms pervade most areas of science and
engineering.
In computer science: recent increase of interest in matrix
algorithms for data mining, information retrieval, search
engines, pattern recognition, graphics, ...
CSci 5304: BG 1-2
Examples
Ancient Chinese Problem (3rd cent BC) [Babylonians
had similar problem]:
There are three types of corn, of which three bundles of
the rst, two of the second, and one of the third make 38
measures. Two of the rst, three of the second and one of
the third make 33 measures. And one of the rst, two of
the second and three of the third make 26 measures. How
many measures of corn are contained in one bundle of each
type?
CSci 5304: BG 1-3
Pagerank of Webpages (21st cent AD)
If one were to do a random walk from web page to web
page, following each link on a given web page at random
with equal likelihood, which are the pages to be encoun-
tered this way most often?
CSci 5304: BG 1-3a
Vibrations in mechanical systems. See:
www.cs.umn.edu/
~
saad/eig_book_2ndEd.pdf
Problem: Determine the vibration
modes of the mechanical system [to
avoid resonance]. See details in
Chapter 10 (sec. 10.2) of above
reference.
m
1
m
2
l
1
l
2
k
1
k
2
CSci 5304: BG 1-4
Examples (cont.)
Method of least-squares (inspired by rst use of least
squares ever, by Gauss around 1801)
A planet follows an elliptical orbit according to ay
2
+bxy+
cx + dy + e = x
2
in cartesian coordinates. Given a set of
noisy observations of (x, y) positions, compute a, b, c, d, e,
and use to predict future positions of the planet. This
least squares problem is nearly rank-decient and hence
very sensitive to perturbations in the observations.
Read Wikipedias article on planet ceres:
http://en.wikipedia.org/wiki/Ceres_(dwarf_planet)
CSci 5304: BG 1-5
Background in linear algebra
Review vector spaces. Read section 1.1 of text on vector
notation.
A vector subspace of R
n
is a subset of R
n
that is also
a real vector space. The set of all linear combinations of
a set of vectors G = {a
1
, a
2
, . . . , a
q
} of R
n
is a vector
subspace called the linear span of G,
If the a
i
s are linearly independent, then each vector of
span{G} admits a unique expression as a linear combina-
tion of the a
i
s. The set G is then called a basis.
Recommended reading: Sections 1.1 1.6 of
www.cs.umn.edu/
~
saad/eig_book_2ndEd.pdf
CSci 5304: BG 1-6
Matrices
A real mn matrix A is an mn array of real numbers
a
ij
, i = 1, . . . , m, j = 1, . . . , n.
The set of all mn matrices is a real vector space denoted
by R
mn
.
Complex matrices dened similarly.
A matrix represents a linear mapping between two vector
spaces of nite dimension n and m.
CSci 5304: BG 1-7
Operations:
Addition: C = A + B, where A, B, C R
mn
and
c
ij
= a
ij
+ b
ij
, i = 1, 2, . . . m, j = 1, 2, . . . n.
Multiplication by a scalar: C = A, where
c
ij
= a
ij
, i = 1, 2, . . . m, j = 1, 2, . . . n.
CSci 5304: BG 1-7a
Multiplication by another matrix:
C = AB,
where A R
mn
, B R
np
, C R
mp
, and
c
ij
=
n

k=1
a
ik
b
kj
.
CSci 5304: BG 1-8
Transposition: If A R
mn
then its transpose is a matrix
C R
nm
with entries
c
ij
= a
ji
, i = 1, . . . , n, j = 1, . . . , m
Notation : A
T
.
CSci 5304: BG 1-8a
Transpose Conjugate: for complex matrices, the trans-
pose conjugate matrix denoted by A
H
is more relevant:
A
H
=

A
T
= A
T
.
CSci 5304: BG 1-9
Review: Matrix-matrix and Matrix-vector producs
Recall denition of C = AB.
Recall what C represents [in terms of mappings]..
Can do the product column-wise [Matlab notation used]:
C
:,j
=
n

k=1
b
kj
A
:,k
Can do it row-wise:
C
i,:
=
n

k=1
a
ik
B
k,:
CSci 5304: BG 1-10
Can do it as a sum of outer-product matrices:
C =
n

k=1
A
:,k
B
k,:
Verify (prove) all 3 formulas above..
Complexity? [number of multiplications and additions]
What happen to these 3 dierent approaches when B
has one column (p = 1)?
CSci 5304: BG 1-11
Square matrices, matrix inversion, eigenvalues
Square matrix: n = m - so A R
nn
Identity matrix: square matrix with
a
ij
=
_
1 if i = j
0 otherwise
Notation: I.
Property: AI = IA = A
Inverse of A (when it exists) is a matrix C such that
AC = CA = I
Notation: A
1
CSci 5304: BG 1-12
Eigenvalues and eigenvectors
A complex scalar is an eigenvalue of the square matrix
A if a nonzero vector u of C
n
exists such that
Au = u.
The vector u is an eigenvector of A associated with .
The set of all the eigenvalues of A is the spectrum of A.
Notation: (A).
is an eigenvalue of A if and only if det(AI) = 0
p
A
() = det(AI) is a polynomial of degree n in
= characteristic polynomial of A.
(A) if and only if is a root of the characteristic
polynomial p
A
().
CSci 5304: BG 1-13
Spectral radius = The maximum modulus of the eigen-
values
(A) = max
(A)
||.
Trace of A = sum of diagonal elements of A.
tr(A) =

n
i=1
a
ii
.
tr(A) = sum of all the eigenvalues of A counted with
their multiplicities.
CSci 5304: BG 1-13a
Recall that det(A) = product of all the eigenvalues of
A counted with their multiplicities.
Example: Trace, spectral radius, and determinant of
A =
_
2 1
3 0
_
.
CSci 5304: BG 1-14
For two n n matrices A and B are the eigenvalues
of AB and BA the same?
If Ais nonsingular what are the eigenvalues/eigenvectors
of A
1
?
What are the eigenvalues/eigenvectors of A
k
for a given
integer power k?
What are the eigenvalues/eigenvectors of p(A) for a
polynomial p?
Review the Jordan canonical form. Dene the eigen-
values, and eigenvectors from the Jordan form.
CSci 5304: BG 1-15
Range and null space
Range: Ran(A) = {Ax | x R
n
} R
m
Null Space: Null(A) = {x R
n
| Ax = 0 } R
n
Range = linear span of the columns of A
Rank of a matrix rank(A) = dim(Ran(A)) n
CSci 5304: BG 1-15a
Ran(A) R
m
rank (A) m
rank (A) min{m, n}
rank (A) = number of linearly independent columns of
A = number of linearly independent rows of A
A is of full rank if rank(A) = min{m, n}. Otherwise
it is rank-decient.
CSci 5304: BG 1-16
Rank+Nullity theorem for an mn matrix:
dim(Ran(A)) + dim(Null(A)) = n
Apply to A
T
: dim(Ran(A
T
))+dim(Null(A
T
)) = m
rank(A) + dim(Null(A
T
)) = m
Terminology:
dim(Null(A)) is the Nullity of A
n rank(A) is called the co-rank of A
so: co-rank of A equals nullity of A.
CSci 5304: BG 1-17
Show that A R
nn
is of rank one i [if and only if]
there exist two nonzero vectors u and v such that
A = uv
T
.
What are the eigenvalues and eigenvectors of A?
Is it true that
rank(A) = rank(

A) = rank(A
T
) = rank(A
H
) ?
Matlab exercise: explore the matlab function rank.
Find the range and null space of the matrix
_
_
_
_
_
1 1 0
1 2 3
1 2 1
2 1 1
_
_
_
_
_
Verify your result with matlab.
CSci 5304: BG 1-18
CSci 5304: BG 1-19
Types of matrices (square)
Symmetric A
T
= A. Skew-symmetric A
T
= A.
Hermitian A
H
= A. Skew-Hermitian A
H
= A.
Normal A
H
A = AA
H
.
Nonnegative a
ij
0, i, j = 1, . . . , n
Similarly for nonpositive, positive, and negative matrices
Unitary Q
H
Q = I.
CSci 5304: matrices 1-20 GvL (2.1);Heath 2.1; TB 1-2
[Note: Common useage restricts this denintion to com-
plex matrices an orthogonal matrix is a unitary real matrix
not very natural ]
Orthogonal Q
T
Q = D [orthogonal columns]
The term orthonormal matrix is never used.
CSci 5304: matrices 1-21 GvL (2.1);Heath 2.1; TB 1-2
What is the inverse of a unitary matrix?
What can you say about the diagonal entries of a skew-
symmetric (real) matrix?
What can you say about the diagonal entries of a
Hermitian (complex) matrix?
What can you say about the diagonal entries of a skew-
Hermitian (complex) matrix?
The following types of matrices are normal [true-false]:
real symmetric, real skew-symmetric, complex Hermitian,
complex skew-Hermitian.
Find all real 2 2 matrices that are normal.
Show that any triangular matrix that is normal is diag-
onal
CSci 5304: matrices 1-22 GvL (2.1);Heath 2.1; TB 1-2
Matrices with structure
Diagonal a
ij
= 0 for j = i. Notation :
A = diag (a
11
, a
22
, . . . , a
nn
) .
Upper triangular a
ij
= 0 for i > j.
Lower triangular a
ij
= 0 for i < j.
Upper bidiagonal a
ij
= 0 for j = i or j = i + 1.
Lower bidiagonal a
ij
= 0 for j = i or j = i 1.
Tridiagonal a
ij
= 0 when |i j| > 1.
CSci 5304: matrices 1-23 GvL (2.1);Heath 2.1; TB 1-2
Banded a
ij
= 0 only when i m
l
j i + m
u
,
Bandwidth = m
l
+ m
u
+ 1.
Upper Hessenberg a
ij
= 0 when i > j + 1. Lower
Hessenberg matrices can be dened similarly.
Outer product A = uv
T
, where both u and v are vectors.
Block tridiagonal generalizes tridiagonal matrices by
replacing each nonzero entry by a square matrix.
CSci 5304: matrices 1-24 GvL (2.1);Heath 2.1; TB 1-2
Special matrices
Vandermonde :
Given a column of entries [x
0
, x
1
, , x
n
]
T
put its (com-
ponent-wise) powers into the columns of a matrix V :
V =
_
_
_
_
_
1 x
0
x
2
0
x
n
0
1 x
1
x
2
1
x
2
1
.
.
.
.
.
.
.
.
.
.
.
.
1 x
n
x
2
n
x
n
n
_
_
_
_
_
Try the matlab function vander
What does the matrix-vector product V a represent?
Interpret the solution of the linear system V a = y
where a is the unknown. Sketch a fast solution method
based on this.
CSci 5304: matrices 1-25 GvL (2.1);Heath 2.1; TB 1-2
CSci 5304: matrices 1-26 GvL (2.1);Heath 2.1; TB 1-2
Toeplitz :
Entries are constant along diagonals, i.e., a
ij
= r
ji
.
Determined by m + n 1 values r
ji
.
T =
_
_
_
_
_
_
_
_
r
0
r
1
r
2
r
3
r
4
r
1
r
0
r
1
r
2
r
3
r
2
r
1
r
0
r
1
r
2
r
3
r
2
r
1
r
0
r
1
r
4
r
3
r
2
r
1
r
0
_
_
_
_
_
_
_
_
. .
Toeplitz
CSci 5304: matrices 1-26a GvL (2.1);Heath 2.1; TB 1-2
Toeplitz systems (m = n) can be solver in O(n
2
) ops.
The whole inverse (!) can be determined in O(n
2
) ops.
Explore toeplitz(c,r) in matlab.
CSci 5304: matrices 1-27 GvL (2.1);Heath 2.1; TB 1-2
Hankel : Entries are
constant along anti-
diagonals, i.e., a
ij
=
h
j+i1
.
Determined by m+n
1 values h
j+i1
.
H =
_
_
_
_
_
_
_
_
h
1
h
2
h
3
h
4
h
5
h
2
h
3
h
4
h
5
h
6
h
3
h
4
h
5
h
6
h
7
h
4
h
5
h
6
h
7
h
8
h
5
h
6
h
7
h
8
h
9
_
_
_
_
_
_
_
_
. .
Hankel
CSci 5304: matrices 1-27a GvL (2.1);Heath 2.1; TB 1-2
Circulant : Entries
in a row are cyclically
right-shifted to form
next row. Determined
by n values.
C =
_
_
_
_
_
_
_
_
v
1
v
2
v
3
v
4
v
5
v
5
v
1
v
2
v
3
v
4
v
4
v
5
v
1
v
2
v
3
v
3
v
4
v
5
v
1
v
2
v
2
v
3
v
4
v
5
v
1
_
_
_
_
_
_
_
_
. .
Circulant
CSci 5304: matrices 1-27a GvL (2.1);Heath 2.1; TB 1-2
Explore hankel(c,r) in matlab.
How can you generate a circulant matrix in matlab?
CSci 5304: matrices 1-28 GvL (2.1);Heath 2.1; TB 1-2
Sparse matrices
Matrices with very few nonzero entries so few that
this can be exploited.
Many of the large matrices encountered in applications
are sparse.
Main idea of sparse matrix techniques is not to rep-
resent the zeros.
This will be covered in some detail at the end of the
course.
CSci 5304: matrices 1-29 GvL (2.1);Heath 2.1; TB 1-2