Linear Algebra

Textbook: J. Hefferon, “Linear Algebra”

Reference: T.M.Apostol, “Linear Algebra”
Website: http://ckw.phys.ncku.edu.tw
Homework submission: class@ckw.phys.ncku.edu.tw
 Preface
Evolution of Linear Algebra:
• Linear simultaneous algebraic equations.
• Matrix formulation → matrix algebra.
• Vector space.
• Linear system of differential equations
→ Inner product space (Hilbert space).

Applications of Linear Algebra:

• Minimax (optimization) problems.
• Linear operators (eigen-problems).
• Linear approximations to non-linear problems.
 Computerization

• Procedural programming ( FORTRAN, C, … )

– Routine libraries: EISPACK, LINPACK, IMSL, …
• Symbolic programming
– Mathematica, Maple, Reduce, …
• Dedicated packages:
– MATLAB, …
 Being Computer Aided

• Leave drudgery to the C.

• Post-C learning process:
– Understand the subject:
• Reasoning.
• Applicability.
• Limitations / Error check.
• Comparison / relation with other subjects
– Apply by hand concept to simple cases.
– C.
– Make knowledge C. searchable.
 Goals of this Course

Concepts:
• Linear Systems of Equations
• Linear (Vector) Spaces
• Linear Transformations / Operators
• Matrix Algebra & Determinants
• Inner Product (Hilbert) Spaces

Techniques:
• Mathematical Proofs
• Computational Skills
 Hefferon

1. Linear Systems
2. Vector Spaces
3. Maps Between Spaces
4. Determinants
5. Similarity
 Chapter One: Linear Systems

I. Solving Linear Systems

II. Linear Geometry of n-Space
III. Reduced Echelon Form

Topic: Computer Algebra Systems

Topic: Input-Output Analysis
Topic: Accuracy of Computations
Topic: Analyzing Networks
 I. Solving Linear Systems

I.1. Gauss’ Method

I.2. Describing the Solution Set
I.3. General = Particular + Homogeneous
 Example of a Linear System: Balances

Find the unknown masses h & c balanced as follows:

→ 40h  15c  50  2 25c  25  2  50h

Example of a Linear System: Chemical Reaction

Toluene + Nitric acid → Trinitrotoluene (TNT) + Water

甲苯 + 硝酸 → 三硝基甲苯 (黄色炸藥) + 水
C6 H 2  NO2 3 CH3
x C7 H8  y HNO3  z C7 H 5O6 N 3  w H 2O

C : 7x  7z N : y  3z

H : 8 x  y  5z  2 w O : 3 y  6z  w
 Definition: An ordered set of n real numbers is called an n-tuple.
 a1 , a2 , , an  ai  R

Definition: The set of all n-tuples is called Rn.

Rn   a 1 , a2 , , an  ai  R 

Definition 1.1a: A linear equation in variables x1, x2, …, xn has the form

a1 x1  a2 x2   an xn  d (1)

where a1, a2, …, an R are the equation’s coefficients, and dR its constant.

Definition 1.1b: An n-tuple ( s1, s2, …, sn ) R is a solution of, or satisfies, eq(1) if

a1s1  a2 s2   an sn  d
Definition 1.1c: ( s1, s2, …, sn ) is a solution of the system of m linear equations

a11 x1  a12 x2   a1n xn  d1

a21 x1  a22 x2   a 2 n xn  d 2
(2)
am1 x1  am 2 x2   amn xn  d m

if it is a solution of all m equations in eq(2).

3x1  2 x2  7 3  1  2  5  7
Example 1.2: (1, 5 ) is a solution of since
 x1  x2  6   1  5  6

3  5  2  1  13  7
(5, 1) is not a solution because
  5   1  6  6
 I.1. Gauss’ Method
Gauss’method  Systematic elimination of variables.
3x3 9
Example 1.3: x1 5 x 2 2 x3 2
 
1
x1 2 x2 3
3
Forward elimination ( upper triangularization ) :
row 1  row 3 : 3 row 1 :  row 1 + row 2:
1
x1  2 x2 3 x1  6 x2 9 x1  6 x2 9
3
x1 5 x2 2 x3  2  x2 2 x3  7
x1 5 x2 2 x3 2
3x3 9 3x3 9
3x3 9
Backward substitution (diagonalization) with ρi = row i :
1/3ρ3 ; ρ2 : 2ρ3 + ρ2 : 6ρ2 +ρ1 :
x1  6 x2 9 x1  6 x2 9 x1 3
x2 2 x3 7 x2 1 x2 1
x3 3 x3 3 x3 3
Theorem 1.4 (Gauss’ Method) :
A linear system is invariant under the following operations :
1. ρi  ρj , where i  j.
2. ρi → c ρi , where c  0.
3. ρi → ρi + c ρj , where i  j and c  0.

Definition 1.5 (Elementary reduction / row / Gaussian operations):

1. Swapping.
2. Multiplying by a scalar.
3. Rescaling / pivoting.
Example 1.6:

x y 0
2 x  y 3z  3
x 2 y  z  3

x y 0 x y 0
21   2  2  3
  3 y 3z  3   3 y 3z  3
 1   3
3 y  z  3 4 z  0

1
 3 x y 0 1
 3 x 1
4
  3 y 3 3  y  1
3 3   2   2  1
z 0 z 0
Definition 1.9:
Leading variable = 1st variable with non-zero coefficient in a row.
A system is in echelon form if each leading variable is to the right of the one above it.

A linear system needs not have the same number of equations as unknowns.


Example 1.6:

x 3 y  1 x 3 y  1 

2 1   2
2 x  y  3   5 y  5 

2 1   3
2 x 2 y  3 4 y  4

1
 2 x 3 y  1 

5
  y 1


42  3
0 0
A linear system can have exactly one, none, or infinitely many solutions.

Example 1.12 ( System with no solution ) : 

x 3 y  1 x 3 y  1
2 1   2
2 x  y  3   5 y  5
2 1   3
2 x 2 y  0 4 y  2 

1
 2 x 3 y  1
5
  y 1
42  3 Inconsistent.
0 2

Example 1.14 ( System with infinitely many solutions ) :


x y  4 2 1   2 x y 4
  

2 x 2 y  8 0 0


 


Unique solution:

(Triangular) 


No solution: 

(Echelon) 


Many solution:

(Echelon) 


Exercise 1:
Four positive integers are given. Select any three of the integers, find their
arithmetic average, and add this result to the fourth integer. Thus the numbers
29, 23, 21, and 17 are obtained. One of the original integers is:
(a) 19 (b) 21 (c) 23 (d) 29 (e) 17

Exercise 2:
Laugh at this: AHAHA + TEHE = TEHAW.
It resulted from substituting a code letter for each digit of a simple example in
addition, and it is required to identify the letters and prove the solution unique.
 I.2. Describing the Solution Set
Example 2.1: System with many solutions.

1 2x z 3
2x z  3  1   2 3 1
2
  y  z  
x  y z  1 3 2 2
 1   3
3x  y 4 2 3 1
y  z  
2 2

1 3
x  z 
1 2 2
1 ; 2
3 1

2  y  z 
2  3 2 2
z = free variable
0 0

 3 1 1 3  
Solution set =   x, y , z     z ,  z , z  zR  z = parameter
 2 2 2 2  

Definition 2.2: The non-leading variables in an echelon form are free variables.
Variables used to describe a solution set are parameters of the solution set.
Example 2.3: System with 2 free variables.

x y z w 1 x y z w 1
y  z  w  1 31   3 y  z  w  1
 
3x 6 z 6w  6 3 y 3z 3w  3
 y  z w 1  y  z w  1
x  y  z w 1
3 2   3 y  z  w  1
 
2  4 0 0
0 0

x 2 z 2 w 2
  3  1 y z w  1
 
0 0
0 0
z, w = free variables

Solution set =   x, y , z , w    2  2 z  2 w ,  1  z  w , z , w z, w  R 
z, w = parameters
 Matrix

Definition 2.6 : Matrix

An mn matrix is a rectangular array of numbers with m rows and n columns.
Each number in the matrix is an entry.

Example: A 23 matrix

 1 2.2 5   a11 a12 a13 

A
    i j   a a
a
a23 
i = row index
 3 4 7   21 22 j = column index
 Example 2.7: Linear system

x1 2 x2 4
1 2 0 4 
x2  x3 0 can be abbreviated as  0 1 1 0  
 
x1 2 x3 4 1 0 2 4 

 

1 2 0 4 1 2 0 4 1 0 2 4
2 2 1
 13
   0 1 1 0 
2 2 3
  0 1 1 0     0 1 1 0 
  0 0 0 0
 0 2 2 0  0 0 0 0  
   

  4  2  
  0  z  1  z  R 
S    x1 , x2 , x3    4  2 z , z , z  z  R        
  0  1 
   

  
Definition 2.8: Vector  v1   1 
An m-dimensional (column) vector is an m1 matrix. v   2 
v v  2  
An n-dimensional row vector is an 1n matrix.  v3   7 
The entries of a vector are its components.    
 v4   1

Definition 2.9: Solution

A linear equation
a1x1+ a2x2+…+ an xn = d
of n unknowns x1 , x2 , …, xn is satisfied by a vector s
 a1s1+ a2s2+…+ an sn = d

A vector satisfies a linear system if it satisfies each equation in the system.

Definition 2.10: Vector Addition (Sum)

 u1   v1   u1  v1 
u  v         
  u  v i  ui  vi
u  v  u  v 
 n  n  n n
 Definition 2.10a: Matrix Addition (Sum)
 a11 a1n   b11 b1n   a11  b11 a1n  b1n 
A  B   
 
 
 


a amn   bm1 bmn   am1  bm1 amn  bmn 
 m1

 A  B i j  ai j  bi j i  1, ,m j  1, ,n

Definition 2.11: Scalar Multiplication

 a11 a1n   ca11 ca1n 

i  1,
cA  c     ,m
    cA i j  cai j
 ca j  1,
a  camn 
,n
 m1 amn   m1

 v1   cv1 
c v  c     
  cv i  cvi i  1, ,m
 v   cv 
 m  m

Example 2.12: 1  7 

 2  3   5  4   28 
 3    1    2  7  
        1   7 
 1  4   5    
       3   21
 Example 2.13: 2x  y w 4 

y  w u  4 


x  z 2w 0
 2 1 0 1 0 4  1
 1 3 2 1 0 1 0 4 
 0 1 0 1 1 4
  
2   0 1 0 1 1 4 
 1 0 1 2 0 0   0 1/ 2 1 5/ 2 0 2 
   

1
 2  3  2 1 0 1 0 4   2 0 0 2 1 0 
3

2   0 1 0 1 1 4     0 1 0 1 1 4 
 2 1
 0 0 1 3 1/ 2 0   0 0 1 3 1/ 2 0 
   
1
1  1 0 0 1 1/ 2 0  

2  0 1 0 1
 1 4  

  
 0 0 1 3 1/ 2 0 
 
  x 1  1/ 2   0  
  y  1  1   4  
 1 1            

S    w  u ,  w  u  4 , 3w  u  w, u  R     z   w  3   u  1/ 2    0  w, u  R 
 2 2            
w
  1
   0   0
 
 u 0  1   0 
       
Particular solution
Possible scenarios:
 

 


 

 

 

  

  

 
  

  

  

  

  

    

  

  
Exercise:
Solve the following using matrix notations &
express the solution sets in vector notations.
x y z 0
x  z w  4
y w  0
2x  y w  2
3 x 2 y 3 z  w  0
3x  y  z 7
y w  0
Exercise:
1. Describe all functions f(x) = a x2 + b x + c such that f(1) = 2 & f(1) = 6.
2. Describe all functions f(x) = a x2 + b x + c such that f(1) = 2

Exercise:
Consider the linear system { a x + y = a2 , x + a y = 1 }.
Find values of a so that it has
1) No solutions. 2) Infinitely many solutions.
 I.3. General = Particular + Homogeneous
Definition 3.2: Homogeneous Equations
A linear equation is homogeneous if it can be put in the form
a1x1 + a2x2 + … + anxn = 0
A linear system of equations is homogeneous if it can be put in the form

a11 x1  a12 x2   a1n xn  0

a21 x1  a22 x2   a2 n xn  0

am1 x1  am 2 x2   amn xn  0

Every linear system has an associated homogeneous system obtained by

setting all constant terms to zero.

Definition 3.4: Zero vector = 0 with 0i = 0  i.

A homogeneous system is always consistent since 0 is always a solution.

If s is a solution to a homogeneous system, so is c s  cR .
  
From Example 2.13:  
  x 1  1/ 2   0  
  y  1  1   4  
2x  y w 4  
       
y  w u  4 S  z   w  3   u  1/ 2    0  w, u  R 
         
x  z 2w 0 w
  1
   0   0
 
 u 0  1   0 
       
 Homogeneous Particular 
1  1/ 2   solution solution 
 1   1 
   
3 satisfy the assoicated 2 1   1  1 0
  &  1/ 2  homogeneous system:  1  1   0  0
1  0  1   3 2 1 0
0  1 
   
2 1/ 2    1   0 0
 1   0  1  0

 0
1/ 2   1/ 2  2  0  0
 4
  2  0   4    0 4
 0
 
satisfies the  4   0   0  4
 0 inhomogeneous system:  0   0 2  0 0
 0
 
Example 3.5: Homogeneous system with unique solution 0.

3x 2 y  z  0
6 x 4 y 0
y z  0

 3 2 1 0  3 2 1 0  3 2 1 0
3   2 
 6 4 0 0 2 12
   0 0 2 0     0 1 1 0 
  1  0 0 1 0
 0 1 1 0  0 1 1 0  3  
    2

 3 2 0 0 1 0 0 0
 3 2 2 2  1
  0 1 0 0    0 1 0 0 
 3 1  
 0 0 1 0 1
1  0 0 1 0
  3  
Lemma 3.7:
For any homogeneous linear system, there exists vectors β1 , …, βk s.t.


S  c1 β 1   ck βk c1 , , ck  R 
where k is the number of free variables in an echelon form of the system.
Proof :
Let the system be of m equations in n unknowns with m  n.
By means of the Gauss method, it can be reduced to an echelon form.
Next, we make sure the leading variable of every row is on the diagonal position
by moving any offending columns to the rightmost position.
Let there be p equations of the form 0 = 0 at the bottom:
 a11 a12 a13 a1n 0
 
 0 a22 a23 a2 n 0 Moving columns
 0  means re-ordering
 
 0 am  p , m  p am  p , m  p 1 am  p , n 0 the unknowns.
 0 0 0 0
 
 0 
 0 
 0
By meaning of backward substitution, one gets

1 0 0 b1, m  p 1 b1n 0
0 1 0 b2, m  p 1 b2 n 0 

 0 
 
 
 0 1 bm  p , m  p 1 bm  p , n 0
 
 0 0 0 0
 0 
 
 0 0

With suitable re-labeling of the unknowns, we get

xi  bi , m p 1 xm p 1   ai , n xn i  1, ,m  p

where xj for j = mp+1, …, n are free variables.

By definition: nm p  k
 Proof is completed by setting βj to be
the jth column of the nk matrix

 b1, m  p 1 b1n  z

 
 b2, m  p 1 b2 n 
 
 
 β1 βk    
 bm  p , m  p 1 bm  p , n 
 
 1 0 
 
 
 0 1 

The non-b part is a kk unit matrix.

Particular solution:
 Lemma 3.8:
S   p  h p = particular solution, h = homogeneous solution 

Proof:
1. Every solution s can be be written as s = p + h :
Let s be a solution, then sp satisfies the associated homogeneous system since

a i1  s1  p1    ai n  sn  pn    ai1s1   ai n sn    ai1 p1   ai n pn   di  di  0

Setting h = s  p completes the proof.

2. Every s = p + h is a solution :

a i1  p1  h1    ai n  pn  hn    ai1 p1   ai n pn    ai1h1   ai n hn   di  0  di

Theorem 3.1: For any linear system, there exists vectors β1 , …, βk s.t.


S  p  c1 β 1   ck βk c1 , , ck  R 
where p is a particular solution and k is the number of free variables.
Example 3.10: System with empty solution set

x  z w  1
2x  y w  3
x  y 3z 2w  1

1 0 1 1 0 1 0 1 1 0 1 0 1 1 0
  2 1  2    2  
 2 1 0 1 0     0 1 2 1 0    0 1 2 1 0
 1 3  2  3
1 1 3 2 0 0 1 2 1 0 0 0 0 0 0
     

  1   1 
     
  2   1 
Homogeneous solution set =  z w z, w  R 
 1 0 
     
  
0 1 

 1 0 1 1 1  1 0 1 1 1  1 0 1 1 1 
  2 1 2    2  
 2 1 0 1 3     0 1 2 1 5     0 1 2 1 5 
 1 3  2  3
1 1 3 2 1  0 1 2 1 2  0 0 0 0 7 
     

→ no particular solution → no general solution

 Corollary 3.11:
Solution sets of linear systems are either
• empty (system is inconsistent / example 3.10)
• have one element (solution is unique / example 3.5)
• have infinitely many elements (system is indeterminate / example 2.13)

Dim( h) = 1 Dim( h) > 1

Unique indeterminate
p exists
( 1 general solution ) (  general solutions )
inconsistent inconsistent
p does not exist
( 0 general solutions ) ( 0 general solutions )

Definition 3.12 : Singular and Nonsingular Matrices

A square matrix is nonsingular if it is the matrix of coefficients of a homogeneous
system with a unique solution.
It is singular otherwise, that is, if it is the matrix of coefficients of a homogeneous
system with infinitely many solutions.
Definition: Rank of a Matrix
The rank of a matrix is equal to the number of non-zero rows in its echelon form.

Let A be an mn matrix, then Rank A  min( m, n ).

Theorem
A square matrix is nonsingular iff its rank is equal to its dimension.
Exercises