Solutions of Linear Algebraic

Equations
 Introduction
a11 x1 + a12 x2+…a1nxn = b1
a21 x1 + a22 x2+…a2nxn = b2
……………………………. system of linear algebraic
……………………………. equations
…………………………….
an1 x1 + an2 x2+…annxn = bn

In a matrix form
é a11 a12 a1n ù é x1 ù é b1 ù
êa a22 a2 n úú êê x2 úú êêb2 úú
ê 21
ê a31 a32 a3n ú ê x3 ú = êb3 ú Ax= b
ê úê ú ê ú
ê úê ú ê ú
êëan1 an 2 ann úû êë xn úû êëbn úû
 Number of Solutions
• When do we have
– An unique solution?
– More than one solution?
– No solution?

Recall Cramer's rule

a11 a12 a1n
If D ≠ 0, unique solution
a21 a22 a2 n
xj=Dj / D D = det A =
If D = 0, many solution
an1 an 2 ann or no solution
 Example 1: D ≠ 0

é2 1ù é x1 ù é b1 ù
ê 1 2 ú ê x ú = êb ú D=3 unique solution
ë ûë 2 û ë 2 û

From Cramer’s rule:

2b1 - b2
x1 =
3 the solution is unique, regardless
2b - b of b1 and b2
x2 = 2 1
3
 Example 2: D = 0
é1 1ù é x1 ù é0ù
ê1 1ú ê x ú = ê0ú D=0
ë ûë 2 û ë û

à Cramer’s rule fails

x1 + x2 = 0 2 equations are identical à many solutions

if we let x2= a, then x1= -a

x1 = -a é - 1ù α is an arbitrary number:
x2 = a
or x = a ê ú
ë1û many solutions exist
 Example 3: D = 0

é1 1ù é x1 ù é2ù
ê1 1ú ê x ú = ê3ú D=0
ë ûë 2 û ë û

à Cramer’s rule fails

x1 + x2 = 2
2 equations are inconsistent
x1 + x2 = 3

No solution
Homogenous or Nonhomogenous?
é2 1ù é x1 ù é0ù é2 1ù é x1 ù é2ù
ê 1 2 ú ê x ú = ê 0ú ê1 2 ú ê x ú = ê 0ú
ë ûë 2 û ë û ë ûë 2 û ë û

Ax = 0 Ax = b
homogenous nonhomogenous

x = 0 is always a solution

Since D = 3, both systems have an unique solution

 When Many Solution Exist…
é1 1ù é x1 ù é0ù é1 1ù é x1 ù é2ù
ê1 1ú ê x ú = ê0ú ê1 1ú ê x ú = ê2ú
ë ûë 2 û ë û ë ûë 2 û ë û

é - 1ù x1 + x2 = 2
x = aê ú
ë1û
é2ù é1ù é0ù é 3 ù é 100 ù
homogenous solution ê0ú ê1ú ê2ú ê- 1ú...ê- 98ú
ë û ëû ë û ë û ë û
particular solution

à How do you find a particular solution?

 é1 1ù é x1 ù é2ù
To solve ê1 1ú ê x ú = ê2ú
ë ûë 2 û ë û
First, let x1 + x2 = 0, then x1 = a , x2 = – a

é- 1ù
xh = a ê ú homogenous solution
ë1û
Second, find a particular solution

é1ù
xp = ê ú
ë1û
General solution = homogenous solution + particular solution

é 1 ù é1ù
x = aê ú + ê ú How does this work?
ë- 1û ë1û
General Solution for Ax = b
(when many solutions exist)

Ax h = 0 homogenous solution

Ax p = b particular solution

If x = xh + xp
0
A x= A (xh + xp)= Axh+Axp= b

so x = xh+xp is also a solution for A x = b

 Necessary Conditions to Have an
Unique Solution
• The number of equations must be equal to that
of the unknowns
• Each equation is linearly independent

why linearly independent?

Importance of Linear Independence
Example 1
2 x1 + 3 x2 = 1 two variables,
4 x1 + 6 x2 = 2 but only 1 linearly independent eq.

Example 2

2 x1 + 3 x2 + x3 = 1
three variables,
x1 + 2 x2 - 2 x3 = 2 but how many linearly independent eq?
4 x1 + 7 x2 - 3 x3 = 5

concept of rank
 Rank of a Matrix
The rank of an (m ´ n) matrix, A, is the largest integer, r,
such that A contains an (r ´ r), square matrix with
determinant | Sr | ≠ 0

é0 1 0 1 2 0 3 ù
ê ú rows: # of eqs.
A = ê0 2 0 2 4 0 6ú columns: # of unknowns
êë0 1 0 2 3 - 1 4úû 3´7

r = 4 is impossible because we cannot form such a matrix

r=3?
Rank of a Matrix (Cont'd)

1 1 2 1 1 0 1 2 3
2 2 4 =0; 2 2 0 =0; 2 4 6 =0
1 2 3 1 2 -1 1 3 4

Thus, the rank of the matrix is not 3

1 1
S2 = ¹0
1 2
Therefore, the rank of the matrix is 2
 How to Determine the Rank of a
Matrix?
|S1| = 5 nonzero
é1 1 1 1 ù
ê
A = ê1 2 3 4 úú the rank of A is at least 1
1 1
êë2 3 4 5úû 3´4 | S 2 |= =2
3 5
the rank of A is at least 2

1 1 1 1 1 1
| S 3 |= 2 3 4 = 0 | S 3 |= 1 2 4 = 0
3 4 5 2 3 5
We only need to look at those S3 that incorporate S2
 Example
é2 3 1 ù
Find the rank of A = ê1 2 - 2 ú
ê ú
êë4 7 - 3úû

1¹0 rank is at least 1

3 1
¹0 rank is at least 2
2 -2
2 3 1
1 2 -2 =0 à rank is 2
4 7 -3
 Rank Property
• Rank of A = Rank of AT

• If A B = C, then the rank of C = smaller of the

rank of A and B

• If An´n is nonsingular à |A| ≠ 0

then the rank of A is n, and the rank of A-1 is also n
proof A A-1 = I rank n

rank n
 Rank Property
• The rank of a matrix An×p is invariant after
multiplication by a non-singular matrix Bn´n

Let rank(A) = r

If n > r rank (B n´n A n´ p ) = r

rank (A n´ p ) = n

rank (B n´n A n´ p ) = n
n n
Elementary Operations of Matrices
• Elementary operation involving I
An´n In´n = In´n An´n = An´n

Define Iij as a matrix in which the ith and jth rows have been
interchanged
é1 0 0 0ù
ê0 0 1 0úú
ê
I 23 = ê0 1 0 0ú
ê ú
ê 0ú
êë0 0 0 0 1úû
 Iij as a Multiplier
pre-multiplier
é1 0 0 0ù é 2 3 1 0 4ù é2 3 1 0 4ù
ê0 0 1 0úú êê6 2 5 1 3úú ê9 1 3 2 4úú
ê =ê
ê0 1 0 0 ú ê9 1 3 2 4ú ê6 2 5 1 3ú
ê ú ê ú ê ú
ë0 0 0 1û 4´4 ë6 3 0 1 5û 4´5 ë6 3 0 1 5û 4´5

post-multiplier
é2 1 4 - 1 ù é1 0 0 0ù é2 4 1 - 1ù
ê3 5 2 0 úú êê0 0 1 0úú ê3 2 5 0 úú
ê =ê
ê1 1 2 - 2 ú ê0 1 0 0ú ê1 2 1 - 2ú
ê ú ê ú ê ú
ë4 0 1 3 û 4´4 ë0 0 0 1û 4´4 ë4 1 0 3 û 4´4
 Jij as a Multiplier
J23
é1 0 0 0ù é a11 a12 a13 a14 ù é a11 a12 a13 a14 ù
ê0
ê 1 k 0úú êêa21 a22 a23 a24 úú êêa21 + ka31 a22 + ka32 a23 + ka33 a24 + ka34 úú
=
ê0 0 1 0ú êa31 a32 a33 a34 ú ê a31 a32 a33 a34 ú
ê úê ú ê ú
ë0 0 0 1û ëa41 a42 a43 a44 û ë a41 a42 a43 a44 û

é a11 a12 a13 a14 ù é1 0 0 0ù é a11 a12 a13 + ka12 a14 ù

êa a22 a23 a24 úú ê0 1 k 0úú êa a22 a23 + ka22 a24 úú
ê 21 ê = ê 21
êa31 a32 a33 a34 ú ê0 0 1 0ú êa31 a32 a33 + ka32 a34 ú
ê ú ê ú ê ú
ëa41 a42 a43 a44 û 4´4 ë0 0 0 1û 4´4 ëa41 a42 a43 + ka42 a44 û 4´4

what if we choose k = -a21 / a31 ?

 Gauss Elimination
We can find J21 as a pre-multiplier with k = -a21/a11 such that:
é a11 a12 a13 a14 a15 ù éa11 a12 a13 a14 a15 ù
ê0 b25 úú ê0
ê b22 b23 b24
ê b22 b23 b24 b25 úú
A1 = ê a31 a32 a33 a34 a35 ú A4 = ê 0 b32 b33 b34 b35 ú
ê ú ê ú
êa41 a42 a43 a44 a45 ú
ê0 b42 b43 b44 b45 ú
êë a51 a52 a53 a54 a55 úû êë 0 b52 b53 b54 b55 úû
éa11 a12 a13 a14 a15 ù éa11 a12 a13 a14 a15 ù
ê0 b b23 b24 b25 ú ê0 b b23 b24 b25 ú
ê 22 ú ê 22 ú
A7 = ê 0 0 c33 c34 c35 ú A10 = ê 0 0 c33 c34 c35 ú
ê ú ê ú
ê0 0 c43 c44 c45 ú ê0 0 0 d 44 d 45 ú
êë 0 0 c53 c54 c55 úû êë 0 0 0 0 e55 úû

Remember: the rank of the matrix is still unchanged!

 Rank of Upper Triangular Matrix
If we eliminate only the element "below" the diagonal terms,
we get an upper triangular matrix (Gauss elimination)

é2 1 4 - 1ù é2 1 4 - 1ù
ê0 5 2 0 úú ê0
ê ê 5 2 0 úú
ê0 0 2 - 2ú ê0 0 2 - 2ú
ê ú ê ú
ë0 0 0 3û ë0 0 0 0û

upper triangular one row is missing

rank = 4 rank = 3
 Example
é2 3 1 ù é1 2 - 2 ù
Find the rank of A = ê1 2 - 2ú pivot ® ê2 3 1 ú
ê ú ê ú
êë4 7 - 3úû êë4 7 - 3úû

Use Gauss elimination

é1 2 - 2ù é1 2 - 2ù
A¢ = êê0 - 1 5 úú A¢¢ = êê0 - 1 5 úú upper triangular matrix
êë0 - 1 5 úû êë0 0 0 úû

rank( A) = rank( A¢) = rank( A¢¢) = 2

 Matrix and Vector
In a coefficient matrix, each row vector represent an equation

é2 3 1 ù é x1 ù é0ù
ê 1 2 - 2 ú ê x ú = ê 0ú
ê úê 2 ú ê ú
êë4 7 - 3úû êë x3 úû êë0úû

é2ù é1ù é4ù

v1 = ê 3ú , v 2 = ê 2 ú , v 3 = ê 7 ú
ê ú ê ú ê ú
êë1úû êë- 2úû êë- 3úû

v 3 = v1 + 2 v 2 or v1 + 2 v 2 - v 3 = 0

v3 is a linear combination of v1 and v2

 Rank and Linear Independence

é2ù é1ù é4ù é2 1 4 ù éa1 ù

a1 ê 3ú + a 2 ê 2 ú + a 3 ê 7 ú = 0 ê3 2 7 ú êa 2 ú = 0
ê ú ê ú ê ú ê úê ú
êë1úû êë- 2úû êë- 3úû êë1 - 2 - 3úû êëa 3 úû

What is the value of A ?

If A ¹0 rank(A) = 3 a1 = a 2 = a 3 = 0

If A =0 ?
 Linear Independence

For a1v1 + a 2 v 2 + an vn = 0

If the only solution is a1 = a 2 = an = 0

é ù éa1 ù
êv v2 v 3 ú êa 2 ú = 0 |A| ≠ 0
then ê 1 úê ú
êë úû êëa 3 úû

In this case, rank(A) = n, and vectors v1, v2,.. vn are

linearly independent
 Example 1
é1 ù é 3ù
Consider ê ú and ê ú , are they linearly independent?
ë 2û ë 4û

é1 ù é 3ù é1 3ù éa1 ù é0ù
a1 ê ú + a 2 ê 4ú = 0 ê2 4ú êa ú = ê0ú A ¹0
ë 2û ë û ë ûë 2 û ë û

a1 = a 2 = 0 Yes. They are linearly independent

How about the rank?

 Example 2
é1 ù é- 1ù é1 ù
Consider v1 = ê2ú v 2 = ê 1 ú and v 3 = ê8ú
ê ú ê ú ê ú
êë1úû êë 3 úû êë9úû

Are they linearly independent?

é1 ù é- 1ù é1 ù é1 - 1 1ù éa1 ù
a1 ê2ú + a 2 ê 1 ú + a 3 ê8ú = 0 ê2 1 8ú êa ú = 0
ê ú ê ú ê ú ê úê 2 ú
êë1úû êë 3 úû êë9úû êë1 3 9úû êëa 3 úû

A =0

No. They are not linearly independent

 Basis Vectors
For a n-dimensional linear space, we can only obtain
n linearly independent vectors maximum.

In 2 dimensional space → 2 linearly independent vectors

In 3 dimensional space → 3 linearly independent vectors

These n linearly independent vectors are called basis vector

or complete set of vectors

All other vectors are linear combination of these basis vectors

 Example
é1 ù é2ù é5ù
Are v1 = ê ú v 2 = ê ú and v 3 = ê ú linearly independent or not?
ë2û ë1 û ë3û

One of them must be linearly dependent.

If we assume that v3 is dependent

é5ù é1 ù é2ù
ê3ú = a1 ê2ú + a 2 ê1 ú
ë û ë û ë û
v3 v1 v2

1 7
a1 = , a 2 =
3 3
 Basis Vectors

v1 v4
v3
v3
v1

v2 v2

2 basis vectors 3 basis vectors

 Homogeneous Equations
Ax= 0
If the coefficient matrix A is a (n ´ n) matrix
• When the rank of A = n,
(a) | A | ≠ 0 (b) only trivial solution, x = 0

• When the rank of A < n

(a) | A | = 0 (b) non-trivial solutions (many solutions)

The solution vector, x, is called the null space of A

 Dimension of Null Space
• Number of column = Number of unknowns
• Rank of matrix = number of linearly independent
equations
• Number of components of x that can be selected
arbitrarily for Ax = 0, is (number of unknowns –
rank): dimension of null space

rank of A = 2
number of unknown = 3
dimension of null space = 1
 Example 1

é x1 ù rank of A = 2
é1 2 3ù ê ú é0ù
ê1 9 5ú ê x2 ú = ê0ú number of unknown = 3
ë ûêx ú ë û dimension of null space = 1
ë 3û
If we select an arbitrary value, α, for x3
é- 17a ù é- 17 ù
é1 2ù é x1 ù é- 3a ù é x1 ù é- 17a 7 ù ê 7ú ê 7ú
x = ê - 2a ú = a ê - 2 ú
ê1 9 ú ê x ú = ê- 5a ú ê x ú = ê - 2a ú ê 7ú ê 7ú
ë ûë 2 û ë û ë 2 û êë ú
7û êë a úû êë 1 úû
 Example 2

é x1 ù
é1 2 3ù ê ú é0 ù
ê 2 4 6 ú ê x2 ú = ê0 ú
ë ûêx ú ë û
ë 3û
rank of A = 1
number of unknown = 3
dimension of null space = 2

x1 + 2 x2 + 3 x3 = 0

é- 2a - 3b ù é - 2ù é- 3ù Any other solutions

x = êê a ú = a ê 1 ú + b ê 0 ú will be a linear combination
ú ê ú ê ú
of these two vectors
êë b ú
û ê
ë0û ú ê
ë1û ú
basis vector
 Nonhomogeneous Equations
Ax= b
é a11 a12 a1n ù é a11 a12 a1n b1 ù
êa a22 a2 n úú êa a22 a2 n b2 úú
ê 21 ê 21
A = ê a31 a32 a3n ú B = ê a31 a32 a3n b3 ú
ê ú ê ú
ê ú ê ú
êëam1 am 2 amn úû êëam1 am 2 amn bm úû

rank = rA rank = rB

If rA ≠ rB inconsistent equations à no solution

If rA = rB = r, we need to compare r and n
 Example
é1 2 ù é b1 ù
Solve: ê2 4ú é x1 ù = êb ú
ê úêx ú ê 2 ú
êë3 6úû ë 2 û êëb3 úû
é1ù
ê ú
If b = 0 then, rB = 2 > rA = 1
ê ú no solution
êë0úû

é5ù
b = ê10ú then, r = 1 , r = 1
If ê ú B A 2 variables, 1 eq.
êë15úû many solutions
é5 ù é2ù
x = ê ú +a ê ú
xp ë 0û ë- 1û x
h
 Summary
If A is a n ´ n matrix, B is an augment matrix [A | b]

Det A ≠ 0 Det A = 0

Rank A = n Rank A < n

Dimension of null space = 0 Dimension of null space = n –

rank A
Trivial solution for Ax = 0 Non-trivial solution for Ax = 0

Unique solution for Ax = b No solution (rank A ≠ rank B)

or many solutions (rank A =
rank B) for Ax = b
 Methods for Solving System of
Linear Equations
• Cramer's rule
• Finding A-1, then x = A-1b
• Gauss elimination (GE) with partial pivoting
• Gauss-Jordon (GJ) method
• LU decomposition
• Iterative methods
 Cramer's Rule
Dj
xj =
D

Pros: Very simple and straightforward

Cons: Large computation time when n is large

- The total number of FLOPs (floating point operation)

required by Cramer's rule is (n2 ´ n!)

- When n = 100, total number of FLOPs = 10162

Cramer's rule is never used in engineering

 Inverse Matrix
-11
A = (adjA)
DA
x = A-1b
• Need to calculate determinant of one (n ´ n)
system and n (n-1)´(n -1) determinants in the
cofactor
• Number of operations required is very large!

A-1 is never used in large systems of equations

 Gauss Elimination
In Gauss elimination, we convert A x = b to U x = b
in an upper triangular matrix

é a11 a12 a13 b1 ù éa11 b12 c13 b ù

(n)
1
êa ê ú
ê 21 a22 a23 b2 úú ê 0 b22 c23 (n)
b ú
2

êëa31 a32 a33 b3 úû ê0 0 c33 b úû

(n)
ë 3

then we solve the system of equations with back

substitution
 Example 1

2× row 1 + row 2
é2 5 2 ù é2 5 2 ù
ê - 4 3 - 30ú ê0 13 - 26ú
ë û ë û

back substitution

x2 = -2, x1 = 6
 Example 2

é 1 -1 1 0ù Gauss
ê- 1 2 é1 - 1 1 0ù
6ú elimination ê0 1 6ú
3
ê ú 4
ê2 1 -4 0ú ê ú
ê ú ê0 3 - 6 0ú
ë0 1 4 6û ê ú
ë0 1 4 6û

é1 - 1 1 0 ù >> rref([A,b])
ê0 1 4 6 ú
ê ú row echelon ans =
1 0 0 1
ê0 0 - 18 - 18ú form 0 1 0 2
ê ú 0 0 1 1
ë 0 0 0 0 û 0 0 0 0
 Problems with Gauss Elimination

• When the first coefficient of the first row is zero

• When a diagonal coefficient becomes zero in the
process of elimination
• When the diagonal coefficient are much smaller than
the off-diagonal coefficient

Solution à partial pivoting

 Partial Pivoting
Presence of zero Find maximum |ai1|

é0 10 1 2ù é2 4 1 5ù switch
ê1 3 - 1 6ú pivoting ê1 3 - 1 6ú row 1
ê ú ê ú and 3
êë2 4 1 5úû êë0 10 1 2úû

é2 4 1 5ù é2 4 1 5ù
ê0 1 - 1.5 3.5ú pivoting ê ú
ê ú ê 0 10 1 2 ú
êë0 10 1 2 úû êë0 1 - 1.5 3.5úû

x3 = -2.0625, x2 = 0.4062, x1 = 2.7187

Partial Pivoting MUST be Used with
Gauss Elimination
é- 0.002 4.000 4.000 ù é 7.998 ù
A = êê- 2.000 2.906 - 5.387 úú b = êê - 4.481úú
êë 3.000 - 4.031 - 3.112úû êë- 4.413úû

x T = [1 1 1]

without pivoting with pivoting

x T = [- 1496 2.0 0] x T = [1 1 1]
round-off error
Therefore, just by rearranging the equations, we
can reduce the round-off errors
 Why Gauss Elimination is Popular?
• Number of mathematical operations is very
small
• We only need n3/3 operations to get U matrix
and n2/2 to get backward substitution
• For n = 100, total FLOPs = 338,333
 Gauss Jordan Method
In Gauss Jordan method, we convert A x = b to a diagonal
matrix

é a11 a12 a13 b1 ù éa11 b12 c13 b1( n ) ù

êa ú ê (n) ú
ê 21 a22 a23 b2 ú ê 0 b22 c23 b2 ú
êëa31 a32 a33 b3 úû ê0 0 c33 b3( n ) úû
ë

éa11 b12 c13 b (n)

ù é1 0 0 b1( 2 n ) ù
ê
1
ú ê (2n) ú

ê 0 b22 c23 b (n)

2 ú ê0 1 0 b2 ú
ê0 ú ê0 0 1 b3( 2 n ) ú
ë 0 1 ( n +1)
b
3 û ë û
unitary matrix
 Cost of Gauss Jordan Method
• Total number of operation: n3/2
• When n = 100, only 500000 FLOPs is
necessary

Gauss Jordan method Gauss elimination

n3 / 2 - n3 / 3
% increase =
n3 / 3
 Matrix Inversion by
Gauss-Jordan Method

A-1 A = I
A-1 I = A-1
[A | I ] [I | A ]
-1

é1 2 1 0ù é1 2 1 0ù
A=ê ê0 2
ë- 1 0 0 1úû ë 1 1úû

é1 0 0 - 1ù é1 0 0 - 1ù
ê0 2 ê0 1 0.5 0.5úû
ë 1 1 úû ë
 LU Decomposition
Convert matrix A into a product of lower triangular and
upper triangular matrix

é ù é ù é ù
ê ú ê ú ê ú
ê ú ê ú ê ú
ê ú ê ú ê ú
ê ú ê ú ê ú
ë û ë û ë û

Full matrix A L U

Ax= b à LU x=b

Ly= b solve y with forward substitution

Ux=y solve x with back substitution
 Remarks on LU Decomposition
• Any non-singular matrix can be factored into a
lower triangular matrix in an infinite number
of ways
• One can determine the determinant easily from
the L and U matrix

A=LU
Standard LU Decomposition

é 1 0 0 × 0 ù é* * * ×
*ù
êm 1 0 × 0úú ê0 * * *úú
×
ê 21 ê
A = êm31 m32 1 × 0 ú ê0 0 * *ú
×
ê úê ú
ê × × × × × úê × × × ×
×ú
êëmn1 mn 2 mn 3 × 1úû êë0 0 0 × *úû

L containing 1 on the U from Gauss

diagonal and mij below elimination
the diagonal
 é1 2 1ù Use m21 = 2/1 = 2, and m31 = –1/1 = –1
A = êê 2 2 3úú Row 2 = Row 2 – m21 × Row 1
Row 3 = Row 3 – m31 × Row 1
êë- 1 - 3 0úû

é1 2 1ù
Þ êê0 - 2 1úú
Define m32 = –1/–2 = ½
Row 3 = Row 3 – m32 × Row 2
êë0 - 1 1úû

é1 2 1 ù
Þ êê0 - 2 1 úú
êë0 0 1 / 2úû U from Gauss elimination

é1 2 1ù é 1 0 0ù é1 2 1 ù
A = êê 2 2 3úú = êê 2 1 0úú êê0 - 2 1 úú
êë- 1 - 3 0úû êë- 1 1 / 2 1úû êë0 0 1 / 2úû
 Crout's Method
é a11 a12 a13 a14 ù él11 0 0 0 ù é1 u12 u13 u14 ù
êa a24 úú êêl21 l22 0 úú ê0 1 u23 u24 úú
ê 21 a22 a23
=
0
ê
êa31 a32 a33 a34 ú êl31 l32 l33 0 ú ê0 0 1 u34 ú
ê ú ê úê ú
ëa41 a42 a43 a44 û ël41 l42 l43 l44 û ë0 0 0 1û

a11 = l11 a12 = l11u12 u12 = a12 / a11 a22 = l21u12 + l22
a21 = l21 a13 = l11u13 u13 = a13 / a11 a32 = l31u12 + l32
a31 = l31 a14 = l11u14 u14 = a14 / a11 a42 = l41u12 + l42
a41 = l41
and so on...
 Example
1 u12 u13
0 1 u23
0 0 1
l11 0 0 1 3 2
l21 l22 0 1 5 10
l31 l32 l33 1 6 15

5 = 1´ 3 +l 22 l22 = 2
l11 = 1, l21 = 1, l31 = 1
10 = 1´ 2 + 2 ´ u23 u23 = 4
u12 = 3, u13 = 2
6 = 1´ 3 + l32 l32 = 3
15 = 1´ 2 + 3 ´ 4 + l33 l33 = 1
Example of LU Decomposition
é2 - 1 - 1ù é x1 ù é1ù
ê 0 - 4 2 ú ê x ú = ê0 ú
ê úê 2 ú ê ú
êë6 - 3 0 úû êë x3 úû êë1úû

é2 - 1 - 1ù é2 0 0ù é1 - 0.5 - 0.5ù
ê ú ê L
ê 0 - 4 2 ú = ê 0 - 4 0 ú ê0
úê
êë6 - 3 0 úû êë6 0 3úû êë0
1
0
- 0.5ú
ú
1 úû
U
é2 0 0ù é y1 ù é1ù
solve for L y = b é 1 ù é
y 0.5 ù
ê 0 - 4 0 ú ê y ú = ê0 ú êy ú = ê 0 ú
ê úê 2 ú ê ú ê 2ú ê ú
êë6 0 3úû êë y3 úû êë1úû êë y3 úû êë- 2 / 3úû
 Example (Cont'd)

é1 - 0.5 - 0.5ù é x1 ù é 0.5 ù

ê0 1 - 0.5úêx ú = ê 0 ú
ê úê 2 ú ê ú
êë0 0 1 úû êë x3 úû êë- 2 / 3úû

é x1 ù é 0 ù
êx ú = ê -1/ 3ú
ê 2ú ê ú
êë x3 úû êë- 2 / 3úû
 Computation Time
For a computer performs 400M FLOP/s

Method Total # of FLOPs Total # of FLOPs Time

(For n = 100)
Cramer’s Rule n2 ´ n! 10162 8 ´ 10145 year

Gauss Elimination n3/3 + n2/2 338,333 8.46 ´ 10-4 s

Gauss Jordan n3/2 500,000 1.25 ´ 10-3 s

A-1 by Gauss 4n3/3 + n2 1.33 ´ 106 3.33 ´ 10-3 s

Jordan
LU decomposition n3/3 + n2 3.43 ´ 105 8.58 ´ 10-4 s
Iterative Methods for Solving
Linear Systems
 Vector and Matrix Norm
A norm (represented as || • ||), gives the size
of a vector or a matrix

• Positivity: || x || ≥0 with || x || = 0 if and only if x = 0

• Homogeneity || ax || = | a | || x ||
• Triangular inequality || x+y || ≤ || x ||+ || y ||

For matrix norm only

• || Ax || ≤ || A || || x ||
Different Types of Vector Norms
n
• 1-norm || x ||1 = å| x |
i =1
i

n
most common
• 2-norm || x ||2 =
åx
2
i
i =1

1/ p
æ n
pö
• p-norm || x ||p = ç å xi ÷ expensive
è i =1 ø
• ∞-norm || x ||∞ = max xi
 Example: Norms of a Vector
x T = [1.25 0.02 - 5.15 0]

x 1 = 1.25 + 0.02 + - 5.15 + 0 = 6.42

x2= (1.25) + (0.02) + (- 5.15) + (0)

2 2 2 2
= 5.2996

x ¥
= - 5.15 = 5.15
Graphical Representation
|| x ||2 =1 || x ||∞ =1

2-norm ∞-norm
 Matrix Norm
é n ù Maximum of
• 1-norm || A ||1 = max êå | aij |ú
column sum
1£ j £ n ë i =1 û

én ù Maximum of
• ∞-norm || A ||∞ = max êå | aij |ú
row sum
1£i £ n ë j =1 û
1
é n ù 2

ê å | aij
2
• F-norm || A ||F = | ú Frobenius norm
ëi , j =1 û

• 2-norm || A ||2 = max ( eig ( A T A) )

Note: The 2 norm for a matrix is not the same as the F-norm
How to Define Norms for Matrices?
A = max Ax
x =1

For example, A 2 = max Ax

x 2 =1

1 Ax for ||x||2=1

x ||A||2

-1 1

-1

The most x can be stretched when multiplied by A

 Example
é 5 - 5 - 7ù
é 5 9ù
A=ê ú B = êê- 4 2 - 4úú
ë - 2 1 û êë- 7 - 4 5 úû

|| A ||∞ = max row sum =|5| + |9| = 14

|| A ||1 = max column sum =|9| + |1| = 10
|| A ||F = 25 + 81 + 4 + 1 = 10.54
|| B ||1 = max column sum =|5| + |-4| + |-7| = 16
|| B ||F = 3 ´ 25 + 3 ´16 + 2 ´ 49 + 4 = 15
|| B ||2 = 12.0301
max{sqrt [eig(BTB)]}= ?
 How Do We Choose a Norm?
• Which norm is best?
– the one that provides the tightest measurement of
a matrix (2-norm)

• Which norm is preferred?

– the one that is less expensive to calculate
 Iterative Methods
• They will not give exact solutions even if all
calculations are performed correctly
• We need to estimate errors and convergence
by using norms
• Initial approximations are required
• Jacobi, Gauss-Seidel, and SOR method
 Advantages of Iterative Methods
• When the matrix is sparse, iterative method is
more efficient
• Reduces round-off error
• Self-correcting if error is made in any
iterations
• Iterative methods can be easily applied in non-
linear system of equations
 Jacobi Method
é 6 - 2 1 ù é x1 ù é11 ù
ê- 2 7 2 úêx ú = ê 5 ú
ê úê 2 ú ê ú
êë 1 2 - 5úû êë x3 úû êë- 1úû

11 2 1 11 2 old 1 old
x1 = + x2 - x3 x1new = + x2 - x3 initial guess
6 6 6 6 6 6
x old = [0 0 0]
T
5 2 2 5 2 2
x2 = + x1 - x3 x2new = + x1old - x3old
7 7 7 7 7 7
1 1 2 1 1 2
x3 = + x1 + x2 x3new = + x1old + x2old
5 5 5 5 5 5
 Jacobi Method in Action
initial guess
First iteration:
é 2 1ù é11ù
éx ù ê 0
new - ú é x old ù ê ú
ê
1
ú ê2 6 6 ê 1 ú 6
ú
2 ê old ú ê 5ú
ê x2 ú = ê
new
0 - ú x2 + ê ú
ê ú ê7 7 úê ú ê7ú
êx ú ê 1 2 ê ú 1
0 úú ë x3 û êê úú
new old
ë 3 û ê
ë5 5 û ë5û
xnew becomes xold in the new iteration
Second iteration:
é 2 1ù é11ù
éx ù ê 0
2nd - ú é x old ù ê ú
ê
1
ú ê2 6 6 ê 1 ú 6
ú
2 ê old ú ê 5ú
ê x2 ú = ê
2nd
0 - ú x2 + ê ú
ê ú ê7 7 úê ú ê7ú
ê x 2nd ú ê 1 2 ê old ú 1
ë 3 û ê 0 úú ë x3 û êê úú
ë5 5 û ë5û

check if error ε i = x - x
new old
< tolerance
 Convergence Condition for
Jacobi Method
The sufficient condition for Jacobi method is to have a
coefficient matrix in a diagonally dominant form
n
Definition of diagonally dominant matrix aii ³ å aij
j =1
i¹ j
é 6 -2 1 ù
1st row |6| > |2|+|1| = 3
ê- 2 7 2 ú
ê ú 2nd row |7| > |-2|+|2|= 4
êë 1 2 - 5úû 3rd row |5| > |1|+|2| =3
diagonally dominant
convergence is guaranteed
 Gauss-Seidel Iterative Method
In Jacobi iteration, the new x values are not used until we
complete calculation of all the component of x in that
iterative stageà but new x are usually better than old x

11 2 (k ) 1 (k )
x1(k +1) = + x2 - x3
6 6 6
5 2 2
x2(k +1) = + x1(k +1) - x3(k )
7 7 7
( k +1) 1 1 ( k +1) 2 ( k +1)
x3 = + x1 + x2
5 5 5

•Guaranteed convergence if the coefficient matrix is

diagonally dominant
•The rate of convergence is faster than Jacobi method
Successive Over Relaxation (SOR)
We can improve the convergence by using a combination
of the old value and the new guess of xi

xi( k ) = lxi(,kGS) + (1 - l )xi( k -1)

• l = 1: No relaxation
• 0 < l < 1: Under relaxation : e.g., l = 0.5
• 1 < l < 2: Over relaxation
 First iteration:
é 2 1ù é11ù
éx ù ê 0
new - ú
ê
1
ú ê2 6 6 é0 ù ê 6 ú
ê x2 ú = ê 2 úê ú ê 5 ú x1new = 1.8333
new
0 - ú ê0 ú + ê ú
ê ú ê7 7ú 7
ê0 ú ê 1 ú
ê x new ú ê 1 2
ë 3 û ê 0 úú ë û êê úú
ë5 5 û ë5û

x1sor = 0.5 ´1.8333 + 0.5 ´ 0 = 0.9167

Please note that the textbook says that you don’t need to update x1 using the SOR
equation, which is incorrect.

é 2 1ù é11ù
éx ù ê 0
new - ú
ê
1
ú ê2 6 6 é0.9167 ù ê 6 ú
2 úê ú ê5ú
ê x2 ú = ê
new
0 - úê 0 ú + ê ú x2new = 0.9762
ê ú ê7 7ú
ê 0 ú ê1ú
7
ê x new ú ê 1 2
ë 3 û ê 0 úú ë û ê ú
êë 5 úû
ë5 5 û

x2sor = 0.5 ´ 0.9762 + 0.5 ´ 0 = 0.4881

 Example
é 6 - 2 1 ù é x1 ù é11 ù é2.0000ù
ê- 2 7 úêx ú = ê 5 ú
exact solution ê1.0000 ú
ê 2 úê 2 ú ê ú ê ú
êë 1 2 - 5úû êë x3 úû êë- 1úû êë1.0000 úû

Jacobi n=1 2 3 4 5 6 7
0 1.8333 2.0381 2.0849 2.0044 1.9941 1.9965 2.0001
0 0.7143 1.1810 1.0531 1.0014 0.9903 0.9979 1.0005
0 0.2000 0.8524 1.0800 1.0382 1.0014 0.9950 0.9985
Gauss-Seidel
0 1.8333 2.0690 1.9982 1.9988 2.0001 2.0000 2.0000
0 1.2381 1.0020 0.9953 1.0003 1.0001 1.0000 1.0000
0 1.0619 1.0146 0.9978 0.9999 1.0001 1.0000 1.0000
SOR l = 0.5
0 0.9167 1.4322 1.7151 1.8651 1.9413 1.9778 1.9940
0 0.4881 0.7645 0.9071 0.9738 1.0010 1.0093 1.0099
0 0.2893 0.5408 0.7233 0.8429 0.9158 0.9575 0.9802
Error Bounds for Iterative Methods
• How do we know an approximate answer, x̂ is close
enough to the real answer x ?

(1) error x - xˆ = ?
(2) residual vector b - Axˆ = ?

Is it true to say that if (2) is small, (1) is also small?

 Example
é 1 2ù é x1 ù é 3 ù
ê1.0001 2ú ê x ú = ê3.0001ú
ë ûë 2 û ë û

x = (1, 1)
T

xˆ = (3, 0 )
T

é 3 ù é 1 2 ù é 3ù é 0 ù
b - Ax̂ = ê ú -ê ú ê ú =ê ú
ë3.0001û ë 1.0001 2 0
ûë û ë - 0.0002 û

An ill-conditioned matrix!
 Ill-Conditioned Matrix
A matrix is ill-conditioned if a small change in its
element causes significant changes in the solution

Are we all ill-conditioned?

Computation has to be very accurate, otherwise, we get

big errors in our solutions
 Ill-Conditioned Matrix - Example
é 1 - 11ù é x1 ù é- 1ù é x1 ù é10ù big change
case 1 ê- 9 100 ú ê x ú = ê10 ú, ê x ú = ê 1 ú
ë ûë 2 û ë û ë 2 û ë û -110%
100%
é 1 - 11ù é x1 ù é- 1ù é x1 ù é- 1ù
case 2 ê- 9 100 ú ê x ú = ê 9 ú, ê x ú = ê 0 ú
ë ûë 2 û ë û ë 2 û ë û
é 1 - 11.1ù é x1 ù é- 1ù é x1 ù é110ù
case 3 ê- 9 100 ú ê x ú = ê10 ú, ê x ú = ê 10 ú
ë ûë 2 û ë û ë 2 û ë û
é 1 - 11ù é x1 ù é- 1ù é x1 ù é?ù
case 4 ê- 9 99 ú ê x ú = ê10 ú, ê x ú = ê?ú no solution
ë ûë 2 û ë û ë 2 û ë û
D=0
 Condition Number
b = Ax
Ax = b
b £ A x
A(x + dx) = b + db
1 A
Adx = db £
x b
dx = A -1db
dx £ A -1 db
dx -1 db dx db
||x|| > 0 £ A £ A A -1

x x x b
Condition number
gA
Residual Vector Error Bounds
dx -1 db
£ A A
x b

A(x + dx) = b + db
dx = x - xˆ
db = b - Axˆ

x - xˆ -1 b - Axˆ
£ A A
x b
 Example
é7 10ù -1 é 7 - 10ù
A=ê ú ;A = ê ú
ë 5 7 û ë - 5 7 û

||A||1=17 ||A||∞=17 ||A-1||1=17 ||A-1||∞=17

||A||F=14.933 ||A-1||F=14.933

Condition number, γA = 289 based on the 1 norm

Condition number, γA = 222.9945 based on the F norm

for bT = [1.00 0.7], xT = [0 0.1] dx db

£ 289
for bT = [1.01 0.69], xT = [-0.17 0.22] x b
How to Detect Ill-Condition of a Matrix?

• Determinant A is very close to zero

• Very small number present at some pivot
locations
• Large value of condition number
• Perturb the matrix or b vector, and check
whether the solution x differs
solution:
Use double precision to reduce the round-off errors
Use iterative method to correct the solution
 Iterative Method to Improve the
Solution Error
error, ε = x - xˆ Therefore, we could
solve A e = r for e
residual, r = b - Axˆ
and apply this
r = b - Axˆ = A(x - xˆ ) = Aε to correct x̂

Procedure
1. Solve A x = b we get x̂ due to computational error
2. Calculate r = b - A x̂
3. Solve A e = r
4. New x = xˆ + ε
5. Check if e = 0 or r = 0
 Example
é3.3330 15920 - 10.333ù é x1 ù é 15913 ù exact solution
ê2.2220 16.710 9.6120 ú ê x ú = ê28.544ú x = (1, 1, 1)T
ê úê 2 ú ê ú
êë1.5611 5.7191 1.6852 úû êë x3 úû êë8.4254 úû

If you get an approximate solution x̂ = (1.2001, 0.99991, 0.92538)T

é 15913 ù é3.3330 15920 - 10.333ù é1.20001ù

r = b - Ax̂ = êê28.544úú - êê2.2220 16.710 9.6120 úú êê 0.99991úú
êë8.4254 úû êë1.5611 5.7191 1.6852 úû êë0.92538úû

Aε = r
ε = (-0.20008, 8.9987 ´10 -5 , 0.074607) T
xˆ + ε = (1.0000, 1.0000, 0.99999) T