Lecture 2: Matrix Algebra

By Tilak Abeysinghe ©

* Notation

 a11 a12 . . a1n 

a a 22 . . a 2 n 
 21
Matrix A =  . . . . .  , ij th element is aij; A=[aij].
 
 . . . . . 
a m1 am2 . . a mn 

* Transpose: A’ or AT

* Addition and subtraction (if orders are conformable): A+B, A-B

* Multiplication

Amxn, Bnxk, then (AB)mxk

In general AB≠BA.

A(B+C) = AB+AC
(A+B)C = AC+BC
A(BC) = (AB)C
(AB)′ = B′A′

* Sums

i = (1,1,...,1)′ , ∑x j = i ′x

* Trace of square matrix

tr(A) = a11+a22+…+ann = sum of diagonal elements = constant

tr(A′) = tr(A)
tr(kA) = ktr(A)
tr(In) = n
tr(A+B) = tr(A) +tr(B)
tr(ABC) = tr(BCA) =tr(CAB), if orders are conformable.
* Determinant of a square matrix

det( A) =| A | =constant

 a11 a12 a13 

Minor of a11 of A = a 21 a 23  is 22
a a 23
* a 22 etc.
a32 a33
 a31 a32 a33 

Minor of aij is denoted by |Aij|.

a 22 a 23
* Cofactor of a11 = (−1)1+1
a32 a33

Cofactor of aij is cij = (−1) i + j | Aij | .

Cofactor matrix = C = [cij].

Read a text for the properties of determinants.

* Rank of a matrix

The maximum number of linearly independent columns or rows is defined as the

rank. If rank of An is n, A is called a non-singular matrix. If rank(A)<n then A is
called a singular matrix.

Rank can be determined by calculating determinants of all square sub-matrices. A

non-zero determinant indicates the rank. Alternatively work out the eigen values.
Number of non-zero eigen values indicates the rank.

* Inverse of a matrix

adj ( A)
Inverse of An is A −1 = where adj(A) = adjoint of A = C ′ = [cij ]′ .
A

If |A| = 0, then the inverse of A is not unique.

I −1 = I
( A −1 ) −1 = A
( A′) −1 = ( A −1 )′
( AB) −1 = B −1 A −1
* Solution for a system of simultaneous equations

Ax = b , A is (n×n), x is (n×1) and b is (n×1).

x = A −1b

Using Cramer’s rule

| Ai |
xi =
| A|

where Ai is A with ith column replaced by b.

* Partitioned Matrices

A A12 
A =  11
A22 
.
 A21

Read a text for the inverse of a partitioned matrix.

* Characteristic roots and vectors of a square matrix

Characteristic roots are also known as eigen values or latent roots. The
characteristic vectors are also known a eigne vectors.

Linear transformation: y = A x transforms x in n space to y in m space.

m×1 m×n n×1

Does there exist a vector x (n×1) such that A (n×n) transform x into a multiple of
itself. Such a vector is called an invariant vector. This leads to

Ax = λx

where λ is called a characteristic root of A which yields the above transformation.

This equation can be written as

( A − λI ) x = 0 .

Note that x=0 is always a solution. A non-trivial solution exists only if

rank(A-λI)<n, i.e. only if |A-λI|=0.
 adj ( A − λI )
[ x = ( A − λI ) −1 .0 = .0 , or | A − λI | x = 0 ].
| A − λI |
|A-λI|=0 is a polynomial in λ. Solutions for λ are called characteristic roots or eigen
values. Corresponding to each eigen value there will be a solution of (A-λI)x=0 of
the form kixi, xi a non-zero vector and ki and arbitrary const. These solutions are
called eigen vectors. The solution kixi is not unique because the system is a
homogenous set of equations. To achieve a unique solution impose a normalizing
condition xi' xi = 1 .

Example:

 5 −3
A= .
 4 −2 
Form | A − λI |= 0 . This leads to λ1 = 2, or λ 2 = 1 . These are eigen values.

For λ = 1, ( A − λI ) x = 0 gives

5 − 1 − 3   x1  0
=
 4 − 2 − 1  x 2  0
. This yields


4 x1 − 3 x 2 = 0 , repeated twice. To get a unique solution for x1 and x2 impose the

normalizing condition x12 + x 22 = 1 . Solving the two equation together gives

x1 = ±3 / 5 and x 2 = ±4 / 5 .

Therefore, the eigen vector corresponding to λ = 1 is

 x1   3 / 5   x   − 3/ 5
  =   or  1  =   .
 x2   4 / 5   x2   − 4 / 5 

Similarly for λ = 2

 x1   1 / 2   x   − 1/ 2 
  =   or  1  =  .
  
 x2   1 / 2   x2   − 1 / 2 

* Orthogonal Matrices

If A is symmetric, then all its characteristic roots are real and the characteristic
vectors corresponding to different c-roots are orthogonal, i.e. xi, x j = 0 for λi ≠ λ j .
 If the n characteristic roots are distinct, form a matrix of characteristic vectors C
(n×n). The matrix C is an orthogonal matrix. Again note xi' xi = 1, xi' x j = 0 .

Further note that

C ′C = CC ′ = I ⇒ C ′ = C −1
| C |= ±1 .

* Diagonalization of symmetric matrix

Λ = C ′AC is diagonal and the diagonal elements are c-roots:

Λ = diag (λ1 , λ 2 ,..., λ n ) .

1 1
Define another diagonal matrix W = diag ( ,..., ).
λ1 λn

Then WΛW = I .

Some properties:

1. Number of non-zero characteristic roots = rank of a matrix

2. | Λ |=| C ′AC |=| C ′ || C || A |=| A | . Therefore, |A| is the product of the c-
roots.

* Factoring Matrix

Some applications require a matrix P such that P′P=A-1.

One choice is: P = Λ−1 / 2 C ′ .

Another choice is Cholesky decomposition. Any symmetric positive definite matrix

can be written as A = LU where L is lower triangular and U is upper triangular.

* Idempotent Matrix

If A.A = A then A is idempotnet.

Most well-known symmetric, idempotent matrix in econometrics is

M = I n − X ( X ′X ) −1 X ′.

Check rank(M) = n-k

 M′ = M
M2 = M
c-roots of M are all ones and zeros. Therefore, rank(M)=trace(M).

* Quadratic Form

If An is symmetric, then the scalar (constant)

n n
x ′Ax = ∑∑ aij xi x j is called the quadratic form in x.
i =1 j =1

 1 2   x1 
( x1 x2 )   x  = x12 + 4 x1 x2 + 4 x22 .
 2 3 2 

* Definite Matrices

An is symmetric.

A is positive definite iff x ′Ax > 0 , for all xi ≠ 0.

A is positive semi-definite iff x ′Ax ≥ 0 , for all xi ≠ 0.
A is negative definite iff x ′Ax < 0 , for all xi ≠ 0.
A is negative semi-definite iff x ′Ax ≤ 0 , for all xi ≠ 0.

A will be positive definite iff its all c-roots are positive.

* Kronecker Product (direct Product)

Am×n , Bk ×l

 a11 B . a1n B 
 
( A ⊗ B) mk ×nl = . . . 
a B . a B
 m1 mn 

A⊗ B ≠ B ⊗ A
( A ⊗ B)′ = A′ ⊗ B ′
( A ⊗ B) −1 = A −1 ⊗ B −1
( A ⊗ B)(C ⊗ D) = AC ⊗ BD
A ⊗ (B ⊗ C) = A ⊗ B + A ⊗ C
*Vector and matrix differentiation

Let x be a vector.

y = f ( x1 , x 2 ,..., x n )

 ∂y / ∂x1 
 ∂y / ∂x 
∂y 
2

= . .
∂x  
 . 
∂y / ∂xn 

The second derivative of y w.r.t. the vector x:

∂ 2 y / ∂x∂x ′ = ∂ 2 y / ∂xi ∂x j

 ∂ 2 y / ∂x1∂x1 . . . ∂ 2 y / ∂x1∂x n 
 
 . . 
=  . . 
 
 . . 
∂ 2 y / ∂x ∂x ∂ y / ∂x n ∂x n 
2
 n 1 . . .

which is symmetric. This is called the Hessian matrix.

Let a and x be vectors,

n
=
y a=
'x ∑a x
i =1
i i is a scalar. Then

 ∂y / ∂x1   a1 
∂y / ∂x  a 
∂y 
2  2
=  . =  .  = a.
∂x    
 .  .
∂y / ∂x 2   an 

Let y and x be an (nx1) vectors such that

 y = Ax .

n
Note each element of yi is=
yi a=′
i x ∑ a x , where a ′ is the ith row of A. Therefore,
i =1
i i i

∂yi
= ai =transpose of the ith row of A.
∂x

∂Ax
Thus, = A′ .
∂x

Quadratic form: y = x ′Ax . For example,

a a12  x1 
y = (x1 x 2 ) 11   = a11 x12 + a 21 x1 x 2 + a12 x1 x 2 + a 22 x 22
 a 21 a 22  x 2 

If A is symmetric then
y = a11 x12 + 2a12 x1 x 2 + a 22 x 22 .

∂y / ∂x1 = 2a11 x1 + 2a12 x 2

∂y / ∂x 2 = 2a12 x1 + 2a 22 x 2

a a12  x1 
∂y / ∂x = 2 11  
 a12 a 22  x 2 

= 2Ax in general.