Mathematics for Economists, 212.

214
Spring 2015
Part 1: Systems of Linear Equations∗

Øyvind Thomassen, Seoul National University

March 3, 2015

1 Gaussian and Gauss-Jordan Elimination

• A system of m linear equations in n unknowns can be written as

a11 x1 + a12 x2 + . . . + a1n xn = b1

a21 x1 + a22 x2 + . . . + a2n xn = b2

.. ..
. .

am1 x1 + am2 x2 + . . . + amn xn = bm

• An example:

2x1 + 3x2 = 7
(1)
x1 − x2 = 1.

• Substitution: A simple method for solving a system of linear equations is

∗
Based on Simon and Blume: Mathematics for Economists, ch. 7.1-7.4.

1
 to (a) solve one equation for the first unknown, (b) substitute the solution
for that variable in a second equation, (c) solve for the next unknown, etc.

• In the example (1), the second equation gives x1 = x2 + 1, which when used

in the first equation gives

2(x2 + 1) + 3x2 = 7

5x2 = 5

x2 = 1,

which, when put back in the solution for x1 gives x1 = 1 + 1 = 2.

• Elimination of variables: In the same example, starting from (1) again, if

we add − 21 times the first equation to the second, we get

2x1 + 3x2 = 7
   
2 3 7
1− x1 + −1 − x2 = 1 −
2 2 2

or

2x1 + 3x2 = 7
 
5 5
− x2 = − .
2 2

• The aim is to get a pattern where the first variable with a nonzero coefficient
in an equation has a zero coefficient in all the equations below it.

• From here we can take two routes, depending on which the entire procedure

2
 (including the steps up to here) will be called either Gaussian elimination
or Gauss-Jordan elimination.

• Gaussian elimination: The first is to solve this simplified system by sub-

stitution (called back substitution). From the last equation we get x2 = 1.

Using this in the first equation, we get

2x1 + 3 · 1 = 7

x2 = (7 − 3)/2 = 2.

• Gauss-Jordan elimination: The second way is to multiply both (all) the

equations by a scalar so that the first nonzero coefficient in each equation
is a one:

3 7
x1 + x2 =
2 2
x2 = 1.

• Next, we use elimination from the bottom up: we add − 23 times the second
equation to the first, to get

7 3
x1 = − =2
2 2
x2 = 1.

• We will use three types of transformations of linear systems of equations,

called elementary equation operations:

3
 – Adding a multiple of one equation to another equation.

– Multiplying both sides of an equation by a nonzero scalar.

– Interchanging two equations.

• Note that Gaussian elimination does not use the second elementary equation
operation, while Gauss-Jordan elimination does. Both may use the last
elementary equation operation in order to arrive at a pattern where the
first variable with a nonzero coefficient in an equation has a zero coefficient

in all equations below it.

• Performing any of the three elementary equation operations on a system of

linear equations does not affect the solutions of the system.

• We call two systems with the same set of solutions equivalent. So the
system before and after transformation by elementary equation operations

are equivalent.

• Another example:

2x + 2y − z = 2
x + y + z = −2 (2)

2x − 4y + 3z = 0.

• Adding − 21 times the first equation to the second, and −1 times the first

4
 equation to the third, gives

2x + 2y − z = 2
3
z = −3
2
−6y + 4z = −2.

• We need to interchange equation 2 and 3 to get the triangular pattern we

want:

2x + 2y − z = 2

−6y + 4z = −2
3
z = −3.
2

• For Gaussian elimination, we start by noting that z = −2, so that the

second equation is

−6y + 4(−2) = −2

y = (−2 + 8)/(−6) = −1.

• The first equation then gives

2x + 2(−1) − (−2) = 2

x = 2/2 = 1.

• If instead we proceed with Gauss-Jordan elimination, multiplying each

5
 equation with the inverse of the first coefficient, the system first becomes

1
x+y− z = 1
2
2 1
y− z =
3 3
z = −2.

2
• Eliminating from the bottom, first add 3
times the third equation to the
1
second and 2
times to the first equation:

1
x+y = 1−2 = 0
2
1 2
y = − 2 = −1
3 3
z = −2.

• And finally, add −1 times the second equation to the first:

x = 1

y = −1

z = −2.

2 Elementary row operations

• When working with a system of linear equations, it is inefficient to write

out the equations with variables, plus signs and equal signs. Instead, we

6
 can work with a coefficient matrix

 
a a12 ... a1n
 11 
 
 a21 a22 . . . a2n 
A=
 
.. .. .. .. 

 . . . . 

 
am1 am2 . . . amn

• and the augmented matrix of the system, which includes the right-hand side
constants (the vertical line has no other function than to remind us of the

distinction between coefficients and constants):

 
a11 a12 . . . a1n b1
 
 
 a21 a22 . . . a2n b2 
 = 
 
.. .. . . .. .. 

 . . . . .


 
am1 am2 . . . amn bm

• Corresponding to the three elementary equation operations, we have three

elementary row operations

– interchange two rows of a matrix,

– change a row by adding a multiple of another row, and

– multiply each element of a row by the same nonzero number.

• The goal of performing elementary row operations is to end up with a matrix

of a certain form, that helps us analyse the system. First some definitions:

• A row of a matrix has k leading zeros if the first k elements of the row are
all zeros and the (k + 1)th element of the row is not zero.

7
• A matrix is in row echelon form if each nonzero row has strictly more leading
zeros than the row preceding it.

• The first nonzero entry in each row of a matrix in row echelon form is called

a pivot.

• A matrix is in reduced row echelon form if it is in row echelon form, each

pivot is 1, and every column containing a pivot has no other nonzero entries.

• Consider the example (2), now on matrix form:

 
 2 2 −1 2 
 
 1 1 1 −2 
 
 
2 −4 3 0

• Adding − 21 times the first row to the second and −1 times to the third row,

 
 2 2 −1 2 
 
.
 0 3
 0 2
−3 
 
0 −6 4 −2

• Next we interchange the second and third rows to get the matrix on row
echelon form:
 
 2 2 −1 2 
 
.
 0 −6 4 −2 

 
3
0 0 2
−3

8
• From here we could follow the procedure for Gaussian elimination: rewrite
the system on equation form and solve it by substitution. This would be
exactly like in the previous section.

• Gauss-Jordan elimination can be done on matrix form. First divide each

row by its pivot
 
 1 1 − 21 1 
 
 0 1 −2 1 .
 3 3 
 
0 0 1 −2

2
• Then eliminate from the bottom, adding 3
times the third column to the
1
second, and 2
times to the first:

 
 1 1 0 0 
 
.
 0 1 0 −1 

 
0 0 1 −2

• Finally, to get the system in reduced row echelon form, add −1 times the
second row to the first:
 
 1 0 0 1 
 
.
 0 1 0 −1 

 
0 0 1 −2

• The solutions can now simply be read off the matrix.

9
3 Systems with many or no solutions

3.1 Examples

• So far we have looked at systems where the number of equations equals the
number of unknowns, and where there is exactly one solution.

• Here is a system of two equations in three unknowns:

 
 1 2 3 1 
 .
3 2 1 1

• Adding −3 times the first row to the second row gives the row echelon

matrix
 
 1 2 3 1 
 .
0 −4 −8 −2

• To get the reduced row echelon form, multiply the second row by − 41 .

 
 1 2 3 1 
 .
1
0 1 2 2

• Then add −2 times the second row to the first:

 
 1 0 −1 0 
 . (3)
1
0 1 2 2

10
• We can rewrite this system as

x = z
1
y = − 2z.
2

• If z = 1, x = 1 and y = − 23 . If z = −1, x = −1 and y = 52 . So the system

has more than one solution. In fact, it has infinitely many solutions.

• A system may not have a solution. Consider

 
 1 3 1 
 
.
 3 1 1 

 
2 3 1

• Adding −3 times the first row to the second, and −2 times the first row to

the third row,

 
 1 3 1 
 
.
 0 −8 −2 

 
0 −3 −1

• Adding − 83 times the second row to the third gives the row echelon form:

 
 1 3 1 
 
.
 0 −8 −2 

 
1
0 0 −4

11
• Dividing by the pivot in the second row, and adding −3 times the (new)
second row to the first row, gives the reduced row echelon form:
 
1
 1 0 4 
 
 0 1 1 .
 4 
 
0 0 − 14

1
• x=y= 4
is the only solution the first two equations, but the last equation
would then be

1 1 1
0· +0· =− ,
4 4 4

which is not true. In fact, no values of x and y could satisfy the third
equation. So the system does not have any solution.

• Note that a system with the slightly different reduced row echelon form
 
1
 1 0 4 
 
 0 1 1 ,
 4 
 
0 0 0

would have a solution, since

1 1
0· +0· =0
4 4

does hold.

12
3.2 Definitions

• Consider a system with coefficient matrix A, which has row echelon form

B. If the jth column of B contains a pivot, we call xj a basic variable.

Otherwise xj is called a free variable.

• The point of these definitions is that through Gauss-Jordan elimination, we

can find a solution of the system  in which all the basic variables are either

unambiguously determined or a linear expression of the free variables. The

free variables can take on any value. Therefore, the values we choose for
the free variables will determine values for some of the basic variables.1

• In the first example above, (3) expresses the system with x and y as basic

variables and z as free variable. (Column 1 and column 2 contain pivots,

so variables x1 and x2 are basic variables, while x3 is free.)

4 Rank - the fundamental criterion

4.1 Basic facts about the rank of a matrix

• We will develop criteria to decide whether a system has a solution, and if

so, how many. First the definition of an important concept:

• The rank of a matrix is the number of nonzero rows in its row echelon form.2

• For example, the following matrix is in row echelon form and has three
1
It should be noted that which variables are free and which are basic could depend on the
order of the equations and on the order in which the elementary row operations are performed.
Still, different outcomes reflect the same relationship between the variables.
2
A row is nonzero if some element in the row is not zero.

13
 nonzero rows. It therefore has rank 3:
 
7 4 2 0
 
 
 0 2 5 3 
. (4)
 

 0 0 0 6 
 
 
0 0 0 0

• Throughout this section, unless stated otherwise, let B and B̂ be the row
echelon forms of A and Â, respectively, where A is the coefficient matrix
for some system, and  the corresponding augmented matrix. ‘A system’

means ‘a system of linear equations’.

• Also, #rows means ‘the number of rows of’, and #columns means ‘the
number of columns of’.

• Fact 1. Consider a coefficient matrix A and the corresponding augmented

matrix Â:3

1. rankA ≤ rankÂ

2. rankA ≤ #rowsA

3. rankA ≤ #columnsA.

• Proof of 1.: Any nonzero row in B is also nonzero in B̂.

3
2. and 3. hold for any matrix, regardless of whether it forms part of a system of linear
equations.

14
• Examples of row echelon forms:
   
 7 4 3   7 4 3 1 
   
B=
 0 2 ,
1  B̂ = 
 0 2 1 2 

   
0 0 0 0 0 0 2
   
 7 4 3   7 4 3 1 
B= , B̂ =  
0 2 1 0 2 1 2
   
 7 4 3   7 4 3 1 
   
B=
 0 2 ,
1  B̂ = 
 0 2 1 .
2 
   
0 0 3 0 0 3 2

• Proof of 2.: There can be no more nonzero rows than the total number of
rows.

• Proof of 3.: Each nonzero row in the row echelon form contains exactly one
pivot, so the rank equals the number of pivots. Each column of B can have

at most one pivot. Therefore the rank cannot be greater than the number
of columns.

15
 • Examples:

 
7 4
 
 
 0 2 
rank   = 2 = #columns < #rows,
 
 0 0 
 
 
0 0
 
 7 4 
rank   = 2 = #columns = #rows,
0 2
 
 7 4 3 4 
rank   = 2 = #rows < #columns.
0 0 1 0

4.2 The number of solutions

• We will now look at what various features of A and  can tell us about how
many solutions the system has.

• Fact 2. A system has a solution if and only if

rank = rankA.

• Proof [⇒]. (We will prove that if rank 6= rankA, then the system has no

solution. From this it follows that, if it does have a solution, we must have
rank = rankA.)

If rank 6= rankA, rank > rankA, by Fact 1.1. Then there is at least one

16
 row that is zero in B but not in B̂. This row corresponds to the equation

0x1 + 0x2 + . . . + 0xn = b, b 6= 0. (5)

Since this equation has no solution, the system has no solution.

• Sketch of proof [⇐]. Suppose rank = rankA. Then there is no equation

like (5) that prevents the system from having a solution.

• Examples of row echelon forms of augmented matrices:

   
 7 4 1   7 4 1 
 
 7 4 3 1     
.
   0 2 2   0 2 2 
B̂ = , B̂ =  , B̂ = 
  
0 2 1 2  
0 0 2 0 0 0
infinite solutions
no solution unique solution

• Fact 3. A system with a solution has either one solution or infinitely many
solutions.

• Sketch of proof. If a system with a solution has free variables, these variables
can take on any value, and so there are infinitely many solutions. If it does
not have free variables, all variables are basic, and therefore unambiguously
determined, in which case there is a unique solution.

• Fact 4. If a system has exactly one solution, then A has at least as many
rows as columns.

[Or: a system with a unique solution must have at least as many equations
as unknowns.]

17
 • Proof. When the system has a unique solution, there can be no free vari-
ables, so every variable is basic. xj is basic if the jth column of the row
echelon matrix contains a pivot. So the number of pivots equals the num-

ber of columns4 . Furthermore, every nonzero row in B has a pivot, so their

number is the same as the number of columns. Since there are at least as
many rows as nonzero rows, there are at least as many rows as columns.

To sum this up:

#columnsA = #pivotsB = #nonzero rowsB ≤ #rowsA.

• From Fact 4 it follows that:

• Fact 5. If a system has more unknowns than equations, it has either no

solution or infinitely many solutions.

• A homogeneous system of linear equations is one where all the right-hand

side constants are zero.

• Since, x1 = x2 = . . . = xn = 0 is always a solution to a homogeneous

system, such a system always has a solution. From this, together with Fact
5, it follows that:

• Fact 6. A homogeneous system with more unknowns than equations has

infinitely many solutions.

• Fact 7. A system will have a solution for every choice of right-hand side
4
Since each variable corresponds to a column, and a column cannot have more than one
pivot

18
 (RHS) constants, b1 , . . . , bm if and only if

rankA = #rowsA. (6)

• Proof [⇐]. Suppose (6) holds. Consider some choice of RHS constants, and
let  be the corresponding augmented matrix. The rank of  (number of
nonzero rows in the row echelon form) can be no greater than the number
of rows, so

rank ≤ #rows = #rowsA = rankA.

Since Fact 1.1. tells us that rankA ≤ rankÂ, we must have rankA = rankÂ,
which by Fact 2 implies that the system has a solution.

• Proof [⇒]. (Again we will use a contrapositive proof: supposing that (6)
does not hold, we will show that there is some set of RHS constants for
which the system does not have a solution. Then, if it has a solution for

every choice, (6) must hold.)

Suppose (6) does not hold. Then the last row (and possibly more rows)

of B is all zeros. Let B̂ be the row echelon form of the augmented matrix
corresponding to certain RHS constants. Suppose the RHS constants have
been chosen so that the last row of B̂ is (0 . . . 0 | 1). This equation has no
solution. We can now perform on B̂ exactly the elementary row operations

that took us from A to B, but in reverse. This will give us an augmented

matrix  with RHS constants such that the system has no solution.

19
• The next fact follows quite easily from Fact 7.

• Fact 8. If a system has more equations than unknowns there is a set of

right-hand side constants such that the resulting system has no solution.

• Proof. Suppose A has fewer columns than rows. Since the rank is less than
or equal to the number of columns (Fact 1.3.), which is less than the number
of rows, the rank is less than the number of rows. By Fact 7, there is a
RHS such that the resulting system has no solution.

• Fact 9. A system has at most one solution for each choice of RHS constants
b1 , . . . , bm if and only if

rankA = #columns A. (7)

• Proof [⇐] Suppose (7) holds. The rank equals the number of nonzero rows
in the row echelon form B and therefore the number of pivots in B. Since
each column can contain at most one pivot, there is a pivot in each column.
Then every variable is basic, and - provided there is a solution - the solution
is unique, since all the variables are unambiguously determined.

• Proof [⇒] (Contrapositive) Suppose (7) does not hold, so the rank is less
than the number of columns (by Fact 1.3. it cannot be greater). Then there
must be some free variables (corresponding to columns with no pivot). If
the RHS constants are all zero, the system has a solution. But since there

are free variables, the number of solutions is infinite.

• A nonsingular coefficient matrix is a coefficient matrix such that for every

20
 set of RHS constants b1 , . . . , bm the corresponding system has a unique
solution.

• The final fact is a straightforward combination of Fact 7 and Fact 9.

• Fact 10. A coefficient matrix A is nonsingular if and only if

#rows A = #columns A = rankA. (8)

• Proof [⇒]. Suppose A is nonsingular, so the corresponding system has a

solution for every choice of RHS. Then by Fact 7,

rankA = #rows A.

Since the solution is unique for each RHS, by Fact 9,

rankA = #columns A.

• Proof [⇐]. Suppose (8) holds. Then by Fact 7 the system has a solution for
every choice of RHS constants, and by Fact 9 that solution will be unique.

21