LINEAR

ALGEBRA

.\ND ITS

Complementary
RICHARD

Pivot Theory

W. COTTLE

Stanford

l:nioersit)

Stanford,

California

103

APPLICATIONS

of Mathematical

AND GEOK(;E

Programming

B. D.4STZIG

1. FORMULATION

Linear
person,

programming,

nonzero-sum)

programming,

and bimatrix

games lead to the consideration

fundawaental problem1 :
M, find vectors

quadratic

(two-

of the following

Given a real p-v ctor 4 and a real J!Jx fi matrix

w and z which satisfy

w=g+AJz,

the conditionsa

W > 0,

z > 0,

(1)

ZW = 0.
The remainder
is so.

(2)

of this section is devoted

to an explanation

(There are other fields in which this fundamental

see, for example,

[6] and [13]-but

2 and 3 are concerned

damental

problem

1 The fundamental
pair of variables

we do not treat them here.)

with constructive

under

various

problem

procedures

assumptions

can be extended

only one of which can bc nonbasic

only one of which can be nonbasic.

of why this

problem

arisesSections

for solving the fun-

on the data

q and M.

from p sets each consisting

of a

to k sets of several variables

each,

To be specific,

consider

a system

w = q f

Nz,

zw> 0, z > 0, where N is a p x k matrix (k < p) and the variables wl,

, zu@are
partitioned into k nonempty sets SI, I = I,
., k. Let TI = SJ IJ {q}, I = 1,
, k.
We seek a solution
nonbasic.

of the system

(The fundamental

The underlying

idea of Lemkes

can be shown that

being prepared

this problem

for publication

* In general, capital
lower case italic letters.

in which exactly

problem

one member

of each

set T, is

is of this form where k = p and Tl = {q,

approach

(Sectlon

has a solution
in which

this

2) applies here.

when N > 0. A paper

extension

q}.)

For example,

is developed

it

is currently
in detail.

italic letters denote matrices while vectors are denoted by

Whether a vector is a row or a column will always be clear

from the context, and consequently we dispense with transpose signs on vectors.
In
(2), for example, zw represents the scalar product of z (row) and w (column).
The
superscript
T indicates the transpose of the matrix to which it is affixed.
Linear
Copyright

Algebra

and

1968 by American

Its Applications
Elsevier

1, 103-125

Publishing

Company,

(1968)
Inc.

104

R. W.

Consider
due to J.

first

linear

programs

von Neumann

Primal

in the

Find

symmetric

a vector

A-r: > b,
Dual linear program:

primal-dual

form

a vector

c,

_s 3 0,

when the primal and dual systems

programming

f such that

z = cx.

(3)

y and maximum

z such

_z= yb.

of linear programming

mathematical

G. B. DANTZIG

x and minimum

n >, 0,

Find

yA f

or -in

AND

[ZO].

linear program:

The duality theorem

COTTLE

that
(4)

[3] states that min 5 = max z

(3) and (4), respectively,

parlance - feasible.

are consistent
Since

z = yb < yAx < cx = 5

for all primal-feasible

x and dual-feasible

y, one seeks such solutions

for

which
yb = cx.
The inequality
converted

constraints

to equivalent

of the primal

systems

the introduction

systems

(3) and (4) are equivalent

of nonnegative

Ax-v==b,

in nonnegative

slack

variables.

can be
variables

Jointly,

the

to
u 3 0,

x 3 0,

242 0,

y > 0,

(6)

ATy+u=c,
linear

and dual problems

of equations

through

and the

@I

programming

problem

becomes

one of finding

vectors

U, v, x, y such that

(;)=(b)+(j:
-;i(;);;:I ;I;:;

(I

and, by (5),
xu + yv = 0.

(8)

The definitions

w=(I), q-lb), M_i; -oAT)>

z=(I)
establish
Linear

the correspondence

Algebra

and Its

between

A~@licaizons

(l),

1, 103-125

(2) and (3), (4).

(1968)

(9)

PIVOT

THEORY

OF MATHEMATICAL

The quadratic programming

manner:

Find

In this
ratic

form

x and minimum

Ax>.,

x > 0,
the matrix

f is a globally

is the case,
(3).
any

that

(10)

D may be assumed

D) is positive

to be symmetric.

of x if and only if the quadsemidefinite,

when D is the zero matrix,

and when this

programming

quadratic

problem

programming.
z -

(lo),

define

(11)

i only if there exists a vector y and vectors

in (10) are a direct consequence

of H. W. Kuhn and A. W. Tucker

[14].

to prove from first principles-that

It is well known(12), known as the

are also sufficient in the case of convex quadratic

conditions,

By direct substitution,
9 = c(x -

x0) + $xDx -

we have for any feasible

= 2cox $ VOV+ 4(x ~ xO)D(.r proves

convex

case.

Thus,

the sufficiency

the problem

for solution

of conditions

of solving

vector

X,

;xD.Y~

= u(x - x0) + y(v - v) + Q(x - x)D(x -

which

u and 71 by

by (11) for x = x0 satisfying

and not difficult

Kuhn-Tucker

It

is a special

~1= An: - b.

A=?, + c,

These necessary conditions for a minimum

of a theorem

problem

problem.

to the linear

problem.

programming

x0 yields minimum

u, v given

(10) reduces

In this sense, the linear programming

11 = D.Y A vector

,T?
= CM+ $x0x.

(10) is called the convex quadratic programming

case of the quadratic

For

stated in the following

f such that

convex function

XDX (or matrix

is immediate
program

problem is typically

a vector

formulation,

The minimand

106

PROGRAMMIKG

a quadratic

x0)

x0) 3 0,

(12) for a minimum

program

in the

leads to a search

of the system

zi=Dx-il=y+c,

x 3 0,

y > 0,

v=Ax-b,

143 0,

v 3 0,

(13)
X21+ yv = 0.
Linear

Algebra

and

Ifs

(14)
Applications

1. 103-125

(1968)

106

R. W.

COTTLE

AND

G. B. DANTZIG

The definitions

establish

(13),

(14) as a problem

Dual of a convex
to the consideration
is positive

of a matrix

semidefinite.

Primal

of the form

M =

(:

It is shown in
Find

quadratic jwogram:

Ax+Ey>b,

(l),

From

quadratic program.

(2).

(15) one is led naturally

wherein

2)
[l]

that

E, like D,

the

x and minimum

2 such that

f = cx + &(xDx + yEy),

x >, 0,

(16)

has the associated

program:

Dual quadratic

Find

-Dx+ATy<c,

y and maximum
by -

z =

Y >, 0,

All the results of duality in linear programming

and indeed they are jointly

solvable

_z such that
g(xDx

extend to these problems,

if either is solvable.

the primal problem is just (lo), for which W. S. Dorn

the duality
matrices,

theory

later extended

in [l].

(17)

- yEy).

When E = 0,

[5] first established

When both D and E are zero

this dual pair (16), (17) re d uces to the dual pair of linear programs

(3)l (4).
(a) The minimand

Remarks.
if the
convex
When

quadratic

form

quadratic

program

(or two-person

m pure strategies

are identified

which

called the columlz player,

to the columns

of B.

as ajj
and

strictly
x0.

(b)
quadratic

has n pure strategies

If the row player

Applications

B), is given by

called the row player,

with the rows of A.

and b,, respectively.

Its

game, r(A,

One party,

1, 103-125

which correspond

then their respective

Using

mixed strategies,

(1968)

has

The other

uses his ith pure strategy

the column player uses his jth pure strategy,

Algebra

solution

(the case of convex

nonzero-sum)

A and B.

Linear

if and only

so is

A bimatrix

are defined

convex

Any feasible

minimizing

semidefinite

a pair of m x n matrices
party,

definite.

has a unique

D and E are positive

programming),

in (10) is strictly

XDX is positive

and
losses

PIVOT

their

THEORY

expected

OF MATHEMATICAL

losses

in a mixed strategy

are

xAy

uses the corresponding

xB~,

respectively.

as the probability

(A component

with which the player

pure strategy.)

A pair (x0, y) of mixed

of r(A,

and

is interpreted

107

PROGRAMMING

strategies

is a Nash

[19] equilibrium

@o&t

B) if

It is evident
of r(A,

xOAy0 < xAy0,

all mixed

strategies

x,

xOBy0 < xOBy,

all mixed

strategies

y.

(see, for example,

[15]) that if (x0, y) is an equilibrium

I?), then it is also an equilibrium

point

point for the game r(AB)

in

which
A=

[@jfK],

where K and I, are arbitrary

in assuming

[bij + L],

I? =

Hence there is no loss of generality

scalars.

that A > 0 and B > 0, and we shall make this assumption

hereafter.
Next,
unity,

by letting

ek denote

it is easily shown that

the k-vector

all of whose components

(x0, y) is an equilibrium

are

point of T(A, B)

if and only if
(xAyo)e,

< Aye

(A > O),

(18)

(xByo)e,

< BTxo

(B > 0).

(19)

This characterization
relates
For

of an equilibrium

the equilibrium-point

point

problem

A > 0 and B > 0, if zt*, v*, x*, y*

2~ =

Ay -

e,,

leads to a theorem

to a system

of the form

is a solution

a43 0,

y 2 0,

212 0,

x b 0,

which
(l),

(2).

of the system

(20)
IJ= BTx ~ e,,
XM

yv = 0,

(21)

then

Linear

Algebra

ad

Its Applicatims

1, 103-125

(1968)

108

R. W.

is an equilibrium
point

of &4,

point of r(A)

COTTLE

AND

G. B. DANTZIG

B). Conversely, if (x0, y) is an equilibrium

B) then

is a solution of (ZO), (21).

The latter system is clearly of the form (l), (2),

where

Notice

that the assumption

the matrix

M above

The existence

A > 0, B > 0 precludes

belonging

to the positive

of an equilibrium

point

by J. Nash [19] whose proof employs

Recently,
Lemke
2.

an elementary

and J. T. Howson,

LEMKES

ITERATIVE

Jr.

SOLUTION

This section is concerned

Howson

for finding

proof

THE

first some terminology

Consider

fixed-point

theorem.

was discovered

with the iterative

common

the system

class.

B) was established

FUNDAMENTAL

points

by

of Lemke and

games

which

arbitrary.

Both

to the subject

of this section

and the

of linear equations

the p-vector

(22)
q and the p x ~5 matrix

For i = 1,. . . , p the corresponding

solution of (22) is a pair of vectors

variables

zi and wi are

satisfying

(22) and

Notice that a solution (ze);.z) of (l), (2) is

. a nonnegative
plementary

Finally,

a solution

called

A complementary

i = 1,. . .,p,

z+i = 0,

of (22).

M are

w and z are p-vectors.

com$dementary and each is the complement of the other.

solution

was

problem (l), (2). We introduce

w==q+Mz,
where, for the moment,

C. E.

PROBLEM

technique

of bimatrix

later extended by Lemke to the fundamental

next.

of

[15].
OF

equilibrium

semidefinite

for r(A,

the Brouwer

constructive

the possibility

(23)
complementary

of (22) will be called almost com-

if it satisfies (23) except for one value of i, say i = P.

That

is, zg # 0, wg # 0.
In general, the procedure
convex
Limar

assumes as given an extreme

set
Algebra and Its Applications

1, 103-125

(1968)

point of the

PIVOT

THEORY

OF MATHEMATICAL

PROGRAMMING

109

which also happens to be the end point of an almost complementary

ray (unbounded edge) of 2. Each point of this ray satisfies (23) but for
one value of i, say ,8. It is not always easy to find such a starting point
for an arbitrary M.
Yet there are two important realizations of the
fundamental problem which can be so initiated. The first is the bimatrix
game case to be discussed soon; the second is the case where an entire
column of M is positive. The latter property can always be artificially
induced by augmenting M with an additional positive column; as we
shall see, this turns out to be a useful device for initiating the procedure
with a general M.
Each iteration corresponds to motion from an extreme point Pi along
an edge of 2 all points of which are almost complementary solutions of
(22). If this edge is bounded, an adjacent extreme point P,+* is reached
which is either complementary or almost complementary.
The process
terminates if (i) the edge is unbounded (a ray), (ii) Pit, is a previously
generated extreme point, or (iii) Piti is a complementary extreme point.
Under the assumption of nondegeneracy, the extreme points of Z
are in one-to-one correspondence with the basic feasible solutions of (22)
(see 131). Still under this assumption, a ~o~~~~e~~~~ta~y
basic feasible
~olz~t~~~
is one in which the complement of each basic variable is nonbasic.
The goal is to obtain a basic feasible solution with such a property. In
an almost compIementary basic feasible of (23), there will be exactly
one index, say @, such that both We and zp are basic variables. Likewise,
there will be exactly one index, say Y, such that both zefyand z, are nonbasic variables.3
An almost complementary edge is generated by holding all nonbasic
variables at value zero and increasing either z, or W, of the nonbasic
pair z,, r0,. There are consequently exactly two almost complementary
edges associated with an almost complementary extreme point (corresponding to an almost complementary basic feasible solution).
Suppose that z, is the nonbasic variable to be increased. The values
of the basic variables will change linearly with the changes in z,. For
sufficiently small positive values of zy, the almost complementary solution
remains feasible. This is a consequence of the nondegeneracy assumption.
S C. van de Panne and A. Whinston
and nonbasic

pair

[21] have used the appropriate

terms basic

for {wb, zp} and {wv, zyl respectively.

Linear

Algebra

and

Its Appplications

1, 103-125

(1968)

110

K.

But in order to retain

be prevented

feasibility,

from becoming

W.

COTTLE

for a positive

is obtained

is either

A complementary

solution

complementary
occurs

uniqueness

case,

the

only if a member

of the blocking

The com@ementary

rule

nonbasic

variable

points

1.

THEOREM
com$dementary
Pvoof.

variable-or

nonbasic

consists

that

to the assertion
of almost

Algebra

in place

equivthe next

of the iteration
extreme

path.

path, the only almost

all basic

feasible

the ambiguities

of the theorem,

complementary

solutions

and Its

Applications

of (22) are non-

by any of the standard

lexicographic

of degeneracy.)

that

the procedure

basic feasible

term other than the first one (P, in the accompanying

Linear

pair-is

of almost complementary

an almost complementary

[3] for resolving

a sequence

blocking

an almost complementary

(This can be assured

techniques

nonbasic

the

basic feasible solution which can yeoccuuyis the initial one.

We assume

degenerate.

contrary

Along

In the almost
guarantees

of v.

The procedure

sequence

pair

case, we have the

solution.

It will become

of the new

to be increased.

and edges is called

of the basic

assumption

of the (now nonbasic)

The generated

complementary,

point solution is obtained

feasible

the new value

put, the other member

of these steps.

basic

variable.

becomes

alently

extreme

nondegeneracy

of z, and its index

The complement

or almost

In the complementary

a complementary

complementary

any

value of zy), then a new basic solution

occurs otherwise.

desired result:

forcing

In this event,

blocks the increase

if some basic variable

blocks z,. A new almost complementary

if the blocking

large without

then a ray is generated.

However,

of z, (i.e., vanishes
which

must

negative.

to become negative,

the process terminates.

G. B. DANTZIG

the values of the basic variables

If the value of z, can be made arbitrarily

basic variable

AND

1, 103-125 (1968)

solutions

Suppose,

generates
in which a

figure) is repeated

PIVOT

THEORY

(say P,).

OF MATHEMATICAL

By the nondegeneracy

are in one-to-one

Let P, denote the successor

to P,, namely
extreme

contradiction

state

It is easy to show by examples

satisfy
begins

yay, the procedure must terminate either

basic feasible solution.

that starting
can return

or nonexistence

from an almost complemen-

and bounded.

to the initial

of a solution

The set Z associated

Let the extreme

z2.

This

point regardless

to (l), (2).

no solution
point

w = (1, 0, 0), z = (1, 0, 2) be the initial

by increasing

com-

with

It is clear that

(2) since zrzq > 0.

solution
after

1.

The

path is initiated at the end

which is not the end point of an almost

solution

ray, the procedure

of the existence

to P,.

to P, along

the

ray OY in a complementary

tary basic feasible

of 2

of (22).

the proof.

point of an almost complementary

in a different

points

solutions

and each is adjacent

If the almost complementary

COKOLLARY.

is nonempty

feasible

But there are only two such edges at P,.

edge.

completes

\Ve can immediately

Example

the extreme

basic

of P, and let P, denote the second predecessor

P,, P,, P, are distinct

points

plementary

with

the one along the path just before the return

an almost complementary
This

assumption,

correspondence

111

PROGRAMMING

will return

of (1) can also

corresponding
point

of a path

to the initial

extreme

to the
which
point

4 iterations.

Example

is likewise
problem

2.

The

nonempty
(l),

(0, 1, 0, 1).

(2)
Yet

has

set Z associated

and

bounded.

a complementary

by starting

with

The corresponding
fundamental
solution w = (1, 0, 1, 0), z =

at w = (1, 2, 0, l), z = (3, 0, 0, 0) and in-

112

R. W. COTTLE

creasing
point

za, the method

after

generates

a path

AND

which

G. B. DANTZIG

returns

to its starting

4 iterations.

Furthermore,

even

if the

procedure

point at the end of an almost

is possible

whether

Example

3.

or not

Given

the

(1 + w2, 0, 0,O).

(1, w,, 4 +

ray, w =

so that

extreme

in a ray
a solution.

the process

is solved

wz, l), z =

by increasing

zs leads to

point at which the required increase

terminates

in a ray, and yet the

by
z = (0, l,O, 1).

w = (2, 0, 1, O),
4.

has

to w = (l,O, 4, l), z = (l,O, 0,O) is at the

complementary

problem

Example

problem

Moving along the edge generated

of z, is unblocked,

an extreme

the data

a new almost complementary

fundamental

from

ray, termination

fundamental

the point of 2 which corresponds

end of an almost

is initiated

complementary

In the problem

with

q-(-J7 M=(Y
-:)
the inequalities
point

(1) have solutions,

complementary
not blocked,

ray w = (1, ws), z = (ws, 0).

and the process

Consequelzces of termination
In this geometrical
to interpret
plementary
inequality

[ll,

(w * ; z*)

Algebra

The

When

zs is increased,

it is

in a ray.

to the fundamental

the meaning

of termination

problem,

it is useful

in an almost

com-

result in linear

31.
is

almost complementary

alz

basic feasible solution

to an almost complementary

ray,

there

wh, zh such that

wh = Mzh,

Linear

(2).

in a ray

(22), and (w *; z*) is incident

exist p-vectors

satisfies

is at the end of an almost

This can be achieved by use of a standard

theory
If

terminates

approach

algebraically
ray.

LEMMA.
of

but none of them

to (w ; z) = (1,0 ; 1,0)

corresponding

Wh 2 0,

and Its Applicatiom

1, 103-

Zh> 0,
125 (1968)

21# 0

(24)

PIVOT

THEORY

and points

OF MATHEMATIC_hL

113

PROGRAMMING

along the almost compdementav3, ray are of the form

(z&j*+ izah, z* + iz),

(25)

2 3 0,

and satisfy
for all ;Z3 0, and all i # /Ll.

(wi* + aW,h)(Z* + ;iZjh) = 0

2. If M > 0, (22) kas a complementary

THEOREM

(26)

basic feasible solution

for any vector q.

Proof.
assume

Select

problem.

A starting

generated

by taking

say zr.

Reduce

a unique

point

The

variable

procedure

corollary

ray of feasible

above,

lemma obtain
Hence

by

complementary

may

solutions

is

variable,

a value zi 3 0 at which

nondegeneracy)

initiated

and consequently

in the

becomes

zero.

An

manner

the procedure

described

must

by the

terminate

either

basic feasible solution or in an almost complementary

cannot

solution

happen.

(ZJ; z*) is reached.

For if it does, conditions

We now show
(24)-(26)

of the

with /3 = 1. Since M > 0 and zh > 0. this implies 7d'> 0.

(26),

zi* = zih = 0 for all i # 1.

which change with i are zi and the components

generated

We

solves the

been reached.

ray after some basic feasible

that the latter

almost

zero until it reaches

has been

in a complementary

in (22).

large value of any nonbasic

(assuming

has then

variables

(w; z) = (q; 0) immediately

a sufficiently

zi toward

basic

extreme

. , pip as the basic

or,

q 213 0 for otherwise

that

ray is the same

as the initiating

Hence

the

only variables

of w. Therefore
ray,

which

the final

contradicts

the

corollary.

THEOREM

3. A bimatrix

game IJA, B) has nw extreme equilibrhm

point.
Proof.

Initiate

the algorithm

by choosing

of x1, say xi, such that

where

B,

is the first

column

of R?.

With

the smallest

positive

value

114

K. W. COTTLE

it follows (assuming

nondegeneracy)

that no has exactly

nent, say the rth.

The ray is generated

and all the slack

variables

namely

yI, is chosen

For sufficiently
not equal 0.

U, u except

as the nonbasic

for v,.

Letting

yr decrease

If the procedure

toward

value

implies the existence

one zero compocomplement

to increase

except

x1

of II,,

indefinitely.

are all nonnegative

possibly

zero, the initial

xrztr might

extreme

point

of yr.

does not terminate

it terminates

G. B. DANTZIG

as basic variables

The

variable

is complementary

for some positive

by the corollary,
latter

by choosing

large values of y,, the basic variables

and the ray so generated

is obtained

AND

in an equilibrium

in an almost

point,

then

ray.

The

complementary

of a class of almost

complementary

solutions

of the form4

(2s)
(29)
(30)
Assume first that xh # 0.
for all j and all I 3
Assume

next

that

0.

Then y* = B*x > 0. By (30), yj* + ny;

But then u* + iluh = -

yh # 0 and xh = 0.

xi* = 0 for all i # 1;

e,<

By

uh = Ayh > 0.

Then

degeneracy

(.u*, o*, x*, y*) be an extreme-point

assumption,

(30) impliesy,*
terminating
game

7.

ray is the original ray.

tion. The algorithm

solution.

must terminate

value

By the non-

only v,* = 0, and vj* > 0 for all j # Y.

+ A_Yjh= 0 for all j f

(29),

Hence -oh= BTa? = 0 and

and xih = 0 for all i.

Y* is the same as v defined by (27) since x1 must be at the smallest

in order that

= 0

0, a contradiction.

Hence

It is now clear that the postulated

This furnishes

the desired contradic-

in an equilibrium

point of the bimatrix

r(A , B).

A ,modificatiolz of almost complementary

Consider

the system

basic sets

of equations
W = 4 + epzo + Mz,

where z. represents
It is clear that
is called

an artifical

and ep is a p-vector

(31) always has nonnegative

almost complementary

4 The notational
Linear Algebra

variable

solutions.

(1, . . . , 1).

A solution

of (31)

if ziwi = 0 for i = 1, . . . , p and is com-

analogy with the previously

and Its Applicatiovzs

(31)

1, 103-125

studied cast 121> 0 1s obvious.

(1968)

PIVOT

THEORY

elementary
but

OF MATHEMATICAL

if, in addition,

equivalent

z0 = 0.

definition

We consider
large z,.

the almost

The variables

are nonbasic

extreme

Ppz

Mz 3 0,

no algorithm

its lower bound 0.

The remaining

steps

After

is increased

its lower bound

is generated.

There

ray as just described;

(31), i.e.,

a solution

Interest

now centers

complementary

variable,

a basic variable

variable

to those

blocks

if

on the meaning

If

amplify

termination

process

reaches

of (31).

an extreme

point

of Z,,

of z, to the value 0

of termination

for every
(7u* +

feasible
then

of

in an almost

complementary
exists

q $- ep(z,* +

of this

statement.

solution

there

7ay if and

In the remainder

after

the

(w*; zo*, z*) corresponding

ray

to

a nonaero

occurs
vector

il > 0,
174

solution

classes of matrices, the

such that

Moreover

ray

One is in a

basic feasible

For certain

the preceding

in an almost
a basic

in
its

(2).

ray solution

we shall

in

the increase

two forms of termination.

the original system (1) has no sol&on.

section,

basic,

complementary

Process described above terminates in an almost complementary

only

say

of 2e/,,is to be

becomes

a basic variable

of a complementary

of (l),

basic

are now identical

a blocking

the other is in the reduction

and hence the attainment

An initial

value zoo for which

zr, the complement

0) or else an almost

are precisely

of zO, say zO&,

XV,decrease.

Then z. becomes

until either

(by attaining

by sufficiently

this is the trivial case for which

of the procedure

algorithm.

complement

value

If zoo > 0, some unique

place of Zen,and we have v = Y. Next,

increased.
the preceding

large

the minimum

0. If zoo = 0, then 4 > 0;

UJ~, has reached

z b 0).

b 0,

basic while z,,, zr, . . , zp

zero, the basic variables

is required.

let

ray generated

a sufficiently

point is reached when z0 attains

ZEI= 2 + epzo 3

. , wp are initially

For

toward

[16, p. 6851 where a different

In this case,

complementary

pi,

variables.

As z,, decreases

(See

is given.)

% = {(z,,, z) j w = q +

115

PROGRAMMING

AZ,,")+

M(z*

Izh)

(zN;z,;, z)

116

R. W.

COTTLE

AND

G. B. DANTZIG

and
(w,,* + izp)(zi*

i = 1,. . .,;h.

+ AZ,) = 0,

z,, > 0 and then wk > 0

The case 2 = 0 is ruled out, for otherwise

because

(& ; z,,, z) # 0.

(34)

Now if wk > 0, (34) implies z* + AZ= z* = 0.

This, in turn, implies that the ray is the original one, which is not possible.
Furthermore,

it follows from the almost complementarity

of solutions

along the ray that

zz*W,* = Zi*gelh= &&ji* = Q-Q
The individual

equations

_ 0

of the system

of (36) by zik leads,

i-

we conclude

THEOREM

nonnegative

,4t this juncture,

class,

i=

in a ray

(2) always
The

first

class

than

so that

regardless

mentioned

above

M will be considered.

in a ray implies

the in-

class, we will show that

for this

of what

class

of matrices,

g is.

was introduced

by

which we shall refer to as copositive plus,

Lemke

[lS].

are required

the two conditions

(M + Mr)z~ = 0

co$ositive

a nonzero

(38)

the second

occur,

UIVIU>, 0

Matrices

termination

For

(1).

cannot

has a solution

matrices,

to satisfy

there exists

l,...,).

two large classes of matrices

we show that

of the system

termination

These

(37)

i=l,...,P,

vector zh such that

For the first

(l),

to

in a ray implies

z,h(Mzh), < 0,

consistency

(36)

that

Termination

4.

1,...,p.

via (35),

0 = Z&h + z;k(Mzh),,
from which

(35)

(32) are of the form

m,k = z()Jt+ (A!fzh)t,

Multiplication

i = 1,. . .,/I.

satisfying

conditions

(see [18, 121).

[IS] on copositive

for all
if

uMz4

ZL3 0,
= 0

zt > 0.

(40)

(39) alone are known in the literature

To our knowledge,
matrices

the class of such matrices

and

(39)

satisfying

is large

there

is no reference

the condition

and includes

(40).

as

other

However,

PIVOT

THEORY

(i)

OF MATHEMATICAL

all strictly

copositive

117

PROGRAMMING

i.e.,

matrices,

those

for which

UMG > 0

when O#u>O;
(ii)

all positive

semidefinite

matrices,

i.e., those for which &Vfu > 0

for all u.
Positive

matrices

matrices

are both positive

more,

are obviously

it is possible

smaller ones.

strictly

to build

For example,

Moreover,

if M satisfies

satisfy

definite

copositive.

satisfying

Further-

(39) and (40) out of

if M, and M, are matrices

so is the block-diagonal

matrices

while positive

and strictly

matrices

(40) then

(of its order),

copositive

semidefinite

satisfying

(39) and

matrix

(39) and (40) and S is any skew-symmetric

then M + S satisfies

(39) and (40).

matrix

Consequently,

block

such as

(39) and (40) if and only if M, and M, do too.

[16, 171 has pointed

problem

with

A > 0 and

Howson iterative
section.

out, the matrices

However,

for bimatrix

as Lemke

in the bimatrix

B > 0 need not satisfy

procedure

If applied to bimatrix

terminates

encountered

(40).

The

game
Lemke-

games was given earlier in this

games, the modification

in a ray after just one iteration,

just given always

as can be verified

by taking

any example.
The
minors,

second

class,

consisting

has been studied

[2, 4, 8, 9, 10, 22, 241.

positive
breaks

principal

of matrices

by numerous

In the case of symmetric

minors

are

positive

down in the nonsymmetric

with positive principal

having

investigators;
definite.

situation.

positive

principal

see, for example,

matrices,

But

those with

the

equivalence

Nonsymmetric

minors need not be positive definite.

matrices

For example,

the matrix

has positive principal minors but is indefinite

positive

definite

minors.

(See,

matrices
e.g.,

are a subset

and not copositive.

of those

with positive

However,
principal

[2].)
Linear

9lgebra

and

Its

Applications

1, 103 - 125 (1968)

K. W.

11x
\Ve shall
no solution

make

use of the

if there

exists

fact

that

a vector

vh!! < 0,

COTTLE

AXD

w = 2 + Mz,

2 3 0

7vq< 0,

(41)

for otherwise

0 < VW = ~4 + uMz < 0, a contradiction.

of J. Farkas

exists

THEOREM

terminates

Let

5.

in a ray,

Indeed,

be copositive

$us.

If

the iterative

Procedure

(1) has no solution.

in a ray

(W* ; zo*, z*) will be reached

means

that

a basic

at which conditions

feasible

(32)-(34)

solution

hold and also

0 = zhwh= ziiepzgil+ zhAZz~~.

Since

M is copositive

are nonnegative,
The vanishing

But

and zir 3 0, both

hence both are zero.

of the quadratic

by (32), zok = 0 implies

what is the same thing,

() =
and we obtain

~*ig+

again

that

z*M,y

on the right

wR = Mz >, 0, whence

Next,

zz

z*(-

If

COROLLARY.

complementary

d$fz)

,$Mz*

by (35)

A+~
satisfying

copositivity
This
matrices

v = zh satisfies

= ~4 + ZkepZO*

M is strictly
feasible

copositive,

solution

(1) are therefore

(41).
the @ocess

terminates

in a

of (31).

If not, the proof of Theorem

Proof.
vector

basic

Mz < 0 or,

by (35),

It follows that zhq < 0 because zIZe

pz,,* > 0. The conditions
because

side of (42)

form zhMzh means

zM < 0.
z

terms

(42)

The scalar z~ = 0 because Zliep> 0.

0 = z%* = ~4 _I- 9epZ0* + @Mz*

inconsistent

it is

[7] that (1) has no solution if and only

of (41).

then

Termination

Proof.

theorem

a solution

(w; Z) > 0, has

u such that

a consequence
if there

G. B. DANTZIG

5 would imply the existence

zMzh = 0, 0 # zh > 0, which contradicts

the

of a
strict

of M.

corollary

clearly

M having

I>zi~earAlgebrn and

Its

generalizes

positive

principal

Ap~licatzons

Theorem

1.

minors.

1, 103-126

(196X)

We now turn

to the

PIVOT

THEORY

THEORE~I

OF X1THEMATICAL

If M has positizre principal

B.

minors, the @ocess

in a con@lementar~~ basic solution of (31) for

Proof.

\Ve have seen that termination

of a nonzero vector zh satisfying

Nikaido

[lo, Theorem

any

terminates

q.

in a ray implies the existence

the inequalities

(38).

21 have shown that matrices

minors are characterized

119

PROGRAMMING

by the impossibility

However,

Gale and

with positive principal

of this event.

Hence termina-

tion in a ray is not a possible outcome for problems in which A4 has positive
principal

minors.

We can even improve

THEOREM

7.

upon this.

If M has the property

that for each of its principal

sub-

matrices I@, the system

has no solution, then the process terminates in a complementary

basic solution

of (31) for any q.

Proof.

Suppose

the process terminates

in a ray.

system (32),
(wh; %Jh>zh) of the homogeneous
components of wh for which the corresponding
positive.

Then

components

by (34) Gh = 0.

Let

From

define

the

component

2 be the vector

the solution
vector

~5 of

of Z* + z? is

of corresponding

in zh. Clearly 0 # Zh > 0, since 0 # z > 0 and any positive

component

of zh is a positive

A? be the corresponding
of order k 3

component

principal

of Yh by definition

submatrix

of M.

of rZh. Let

Since i@ is a matrix

1 we may write
0 = Gh = efizOh+ ,llah

Hence
A9

< 0,

0 # zh> 0,

which is a contradiction.
3. THE

PRINCIPAL

PIVOTING

We shall now describe

predates
algorithm

that

METHOD

an algorithm

of Lemke.

It

proposed by the authors

evolved

from

a quadratic

[4] which

programming

of P. Wolfe [26], who was the first to use a type of complemenLinear

Algebra

and Its

.4pplicntions

1, 103-125

(1968)

120

R. W.

tary rule for pivot choice.

have positive principal
and, after

a minor

In Lemkes
troduced

Our method

modification,

procedure

problem

problem.

almost

(w; z) = (q; 0).

solution

If q 21s 0, we may
An almost

zr, the complement

along the path,
Step I:

assume

of the selected

Make the blocking

set.

It will be shown that

negative

of basic

variable

basic

Principal

basic variable.

variables,

properties

by a positive
nonbasic

Otherwise,

ForYpoints

solution

no more than p major

solution

of matrices

to zero.

one less

are at most

cycles

are required

The

under

At

However,

is at least
there

of (22).

invariant

I.

is obtained.

Since

its com-

if ze~rdrops

to Step

values

cycle.

variable

by pivoting

return

with negative

feasible

basic

to zero.

cycle or increases

basic

of the major

a complementary

on certain

that

by increasing

cycle is terminated

during a major

variables

at the beginning

negative

if necessary)

wr increasing

The major

a new complementary

the number

is immediately

path is generated

zr until it is blocked

into the basic

this point,

proof

principal

$
to

depends
pivoting.

pivot transform of a matrix

Consider

the

homogeneous
variables

system

or, .

ui basic.

B and the corresponding

the new basic

matrix

i@ is called

transformation
corresponding

variables

a principal

can be carried

Algebra

and

Its

is a square
in terms

the full set of basic

variables

d.

out only if the principal

zi and wi interchanged
the term

1, 103-125

Let

of the nonbasic

pivot transform of M.

whenever

AppZications

Relabel

nonbasic
fi in terms

to the set of variables

and this will be assumed

where

and expressed

, up. Let any subset of the vi be made

and the corresponding

express

u = Mu

vr, . . . , vp are basic

Here the variables

of the nonbasic

Linear

as well as non-

with the complementary

(relabeling

complementary

out of the basic set of variables.

variables

negative

is initiated

to zero or by the negative

Step II:
plement

nonbasic

for

of the original

ziww,= 0 for i # 1.

Increase

decreasing

matrix.

z,, is in-

solutions

only variables

If q 2 0, the procedure

terminated.

obtain

variable

values.

U+ = q1 < 0.

than

matrices.

complementary

In our approach,

A major cycle of the algorithm

basic

semidefinite

are used, but these can take on initially

negative

M that

to matrices

M, an artificial

for general

G. B. DANTZIG

to positive definite matrices)

to positive

feasible

AND

is applicable

minors (in particular

in order to obtain

the augmented

COTTLE

is used.

(1968)

0 = Wti

ones.

Of course,
submatrix

The
this
of M

is nonsingular,

PIVOT THEORY

OF MATHEMATICAL

THEOREM 8 (Tucker

If a square matrix M has Positive principal

[24]).

so does every principal

minors,

The proof of this theorem

pivot transform of M.

is easily obtained

the roles of one complementary

minors

in terms

of those

Proof.

The original

2;

inequality

variables.

>

with

(zbi, VJ except

U,V, > 0.

possibly

in reverse

The proof in the semidefinite

definite,
every pair

order.

Hence

case replaces

the

by 3.

of the algorithm

w2

m3

92 +

fn21Zl

m2222

i-

1n23z3

93

m3121

n23222

m33z3.

M has positive

that

efficients

for general $J. Consider

= q1 + f+~llzl + m12z2 + ~q3z3

Wl

principal

minors

so that

the diagonal

co-

are all positive :

furthermore

m33 > 0.

9%42
> 0,

ml1 > 0,
Suppose

We wish to

If M is positive

The proof given below for p = 3 goes through

Suppose

the following

After the principal pivot transformation,

is true if u # 0, and the former must hold because

?.zici =

Validity

was along the lines

P. Wolfe has suggested

theorem.

Consider v = Mu.

(z& fiJ is identical

ci

positive semidefinite

07

proof given by the authors

UazZ = ti$ > 0 if uMu = uv > 0.

the latter

principal

of M.

let B = A?S, where c is the new set of nonbasic

show that

by exchanging

the resulting

pivot transform of M.

of that for the preceding

elegant proof.

inductively

pair and evaluating

If a matrix M is positive definite

THEOREM 9.

so is every principal

121

PROGRAMMING

that some qj is negative,

say ql < 0. Then the solu-

tion (w; z) = (ql, q2, q3; 0, 0, 0) is complementary,

variable,

is negative.

We now initiate

the complement

but not feasible because

in this case wi, which we refer to as distinguished

a particular

an almost complementary

of the distinguished

call the driving variable.

Adjusting

variable,
the basic

(w ; 4 1 = bh + mllzl, q2 + m,,q

Linear

Algebra

and

Its

path by increasing

in this case .zl, which we

variables,

we have

q3 + fa313 ; O,O, 0).

Applications

1, 103- 125 (1968)

122

K.

Note that the distinguished

of the driving

variable

W.

COTTLE

AND

G. B. DANTZIG

variable or increases strictly with the increase

zr because

we can increase z1 by a positive

m,, > 0.

Assuming

nondegeneracy,

amount before it is blocked

reaching zero or by a basic variable that was positive

either by W,

and is now turning

negative.
In the former case, for some positive value zr* of the driving variable
zr, we have w1 = q1 + WZ~~Z~*
= 0.

The solution

(w ; z) 2 = (0, qz + ?12&*, q3 + ?2&* ; 0,

is complementary

and has one less negative

ml, replaces ZLV~

by zi as a basic variable.

0, 0)

component.

By Theorem

Pivoting

on

8, the matrix LY?

in the new canonical system relabeled ~8 = q + nZ has positive principal

minors,

allowing

the entire major

cycle

to be repeated.

In the latter case, we have some other basic variable,

msl.zl blocking

when

zi = zr* > 0.

Then

clearly

say Zeus= q2 +

msi < 0 and q2 > 0.

In this case,
@;2)2

THEOREM

10.

If

(vzllzl*

22
w3

system.

-t q3; zl*,o,

is blocked

by a basic
the blocking

ilzcrease of the driving

the further

41 +

qlzl

q2 $-

'ii,,Z, f

'G22ze2 +

"l,,Z,

q3 +

11231.zl+

cL32w2

?&,z,.

Therefore

setting

satisfy

9+p2

the above

variable
variable
variable.

system

(w; 2)s must

0).

pivot exchanging

on ma2 g enerates the canonical

Wl

The solution

variable

a principal

will permit

Pivoting

Proof.

q1,0,ra3,z,*

the driving

other than its com@ement,

with its com$lement

m13z3

since it is an equivalent

z1 = zi*, w2 = 0, Z, = 0 yields

(w; 4 = (41 + $,zl*,

i.e., the same almost complementary

0, q3 +

a3121*:

solution.

z1*, 0, O),

Increasing

z, beyond

z,*

yields
(9r + %zl,

0, q3 + m31zl; zl, 0, (3,

which is also almost complementary.

since +i,, = Linear

Algebra

m21/m2Z > 0.
and

Its

ApfiZications

Hence

The sign of %a1 is the reverse of m2i,

z2 increases

1, 103-

with increasing

125 (1968)

z1 > zl*;

PIVOT

THEORY

OF MATHEMXTICAL

i.e., the new basic variable

positive

principal

increasing

minors,

123

PROGRAMMING

replacing

is not blocking.

Since

has

Hence Zen,continues to increase with

YZ,, > 0.

z1 > q*.

THEOREM

11.

Proof.

There

be repeated

The number of iterations within a major cycle is finite.

are only finitely

with a larger

many

value of zr.

possible

bases.

To see this,

Ko basis can

suppose

it did for

This would imply that some component

of the solution
21** > zr*.
turns negative at zr = zr* and yet is nonnegative when zr = x1**. Since
the value

of a component

is linear

Paraph.rase of the principal

Along

the

freedom.

almost

pivoting

complementary

and zz was shown to increase.

path

each major

can be generated

cycle,

as follows.
variable

(a) replace the blocking

variable

is only one degree

the same

The

first

can be generated

variable

path

by using the

as the driving

the following

variable.

As

steps are iterated:

by the driving variable

variable

as adjusting.

complementary

edge is obtained

variable

is blocked,

cycle if the blocking

almost

of

zr was increasing

The same class of solutions

of the distinguished

soon as the driving

the major

there

of the algorithm,

z2 as the driving variable and the other variables

within

complement

a contradiction.

method

In the proof of the validity

by regarding
Hence

in zr we have

and terminate

is distinguished;

if the blocking

is not distinguished

(b) let the complement

variable

of the blocking

variable

be the new driving

and increase it until a new blocking

variable

is identified;

return

to (a).
The paraphrase

THEOREM

of (l),

form is used in practice.

12.

The jwincipal

pivoting

(2) if M has positive principal

method terminates in a solution

minors

(and, in particular,

if M is

positive definite).
Proof.
in a finite
variables

We have shown that the completion

number

of steps,

with negative

of a major

and each one reduces

values.

Hence

total is reduced to zero and a solution

cycle occurs

the total

in a finite number

of the fundamental

number

of steps,

problem

Linear Algebra ami Its Applications 1, 103-125

of
this

(l), (2)
(1968)

124

R. W. COTTLE

is obtained.

Since a positive definite matrix

the method

applies

As indicated

has positive

G. B. DANTZIG

principal minors,

to such matrices.

earlier,

the positive

by using the paraphrase

The reader

AND

semidefinite

form of the algorithm

will find details

in

case can be handled

with a minor modification.

[4].

ACKNOWLEDGMENTS
Richard W. Cottles research was partially supported by National Science Foundation

Grant

GP-3739;

U.S.

Army

Research

Research,

Contract

sion, Contract

George
Office

B. Dantzigs

Contract

No.

research

ONR-N-00014-67-A-0112-0011,

No. AT(04-3)-326

PA #lS,

was partially

DAHCOP677C-0028,
U.S. Atomic

and National

supported

Office

by

of Naval

Energy Commis-

Science Foundation

Grant

GP 6431.
REFERENCES

1 R. W. Cottle, Symmetric

dual quadratic

programs, Quart.

A&51.

Math.

21(1963),

237-243.
2 R. W. Cottle, Nonlinear

Appl.

Math.

14(1966),

3 G. B. Dantzig,

programs with positively

Linear

Progranzming

Princeton, New Jersey, 1963.

4 G. B. Dantzig and R. W. Cottle,
63-18

(RR),

bounded

Jacobians,

J, SIAM

147-158.

Positive

May 1963, Operations

Berkeley.

Revised

Amsterdam

I 1967 I pp. 55-73.

in Nonlinear

and Extensions,

Univ.

Press,

(semi-) definite programming,

ORC

Research

Princeton

Center, University

Programming

(J. Abadie,

5 W. S. Dorn, Duality in quadratic programming,

of California,

ed.), North-Holland,

Quart. Appl.

Math. 18(1960),

155-162.
6 P. Du Val, The unloading

problem

for plane curves. Amev. /. Math. 62(1940),

3077311.
7 J. Farkas, Theorie der einfachen Ungleichungen,

J. Reine Anger. Math. 124(1902),

l-27.
8 M. Fiedler

and V. Ptak,

and positive

principal

On matrices

minors,

with non-positive

Czech. Math. Journal

9 M. Fiedler and V. Ptak, Some generalizations

off-diagonal

12(1962),

elements

382-400.

of positive definiteness and mono-

tonicity, Numerische Math. 9(1966), 163-172.

10 D. Gale and H. Nikaido, The Jacobian matrix and global univalence of mappings,
Math. Ann. 169(1965), 81-93.
11 A. J. Goldman, Resolution and separation theorems for polyhedral convex sets,
in Linear Inequalities and Related Systems (H. W. Kuhn and A. W. Tucker, eds.),
Princeton Univ. Press, Princeton, New Jersey, 1956.
12 M. Hall,
Chapter

Jr., Combinatorial

Theory,

Blaisdell,

Waltham,

16.

Linear Algebra and

Its Applications

1, 103-125

(1968)

Massachusetts,

1967,

PIVOT

THEORY

OF MATHEMATICAL

13 C. W. Kilmister

and J. E. Reeve,

Rational

York, 1966, p 5.4.

14 H. W. Kuhn and A. W. Tucker,
Syw@osiurn 01%Mathematical
of California

Press, Berkeley,

15 C. E. Lemke

and

California,

programming,

Elsevier, New

in Second Berkeley

(J. Ncyman,

Probability

pomts

of bimatrix

J. Sot. Zndust. Appl. Math. 12(1964), 413-423.

16 C. E. Lemke, Bimatrix equilibrium points and mathematical
Managernext

Sci.

17 C. E. Lemke,

11(1965),

Private

ed.), Univ.

1951.

Jr., Equilibrium

and J. T. Howson,

American

Mechanics,

Sonlinear

Statistics

125

PROGRAMMING

games,

programming,

681-689.

communication.

18 T. S. Motzkin, Copositive quadratic forms,

Nat. BUY.Standards

1818(1952),

Report

11-12.
19 J. F. Sash,

Noncooperative

20 J. van Neumann,
H. Taub,

games, An+z. Math.

Discussion

ed.), Pergamon,

of a maximum

New York,

21 C. van de Panne and A. Whinston,

programming,
22 T. D. Parson,
Doctoral

Operations

A combinatorial

Dissertation,

A comparison

VI

(-4.

of two methods for quadratic

422-441.

approach

Department

28&-295.

CoZZec/ed Works

1963.

14(1966),

Res.

64(1951),

problem,

to convex

of Mathematics,

quadratic
Princeton

programming,
University,

May

1966.
23 A. W. Tucker, X combinatorial
ix Applied

1960.
24 -4. W. Tucker,
6(1963),

equivalence

of matrices, Proceedings

of Symposia

10 (R. Bellman and M. Hall, eds.). American Math. Sot.,

Mathematzcs

Principal

pivotal

transforms

of square matrices,

SZAM

Review

305.

25 A. W. Tucker,
of Mathematics,

Pivotal

Algebra,

Princeton

Lecture

Univ.,

notes (by T. D. Parsons),

Department

1965.

26 P. Wolfe, The simplex method for quadratic programming,

Econometrica

2i(1959),

382-398.

Received

Augwt

30,

1967

Lirzear

Algebra

and

Its

4pplications

1,

103-125

(1968)