Linear Programming

see ORF 307

Linear Programming What is it?

!Quintessential tool for optimal allocation of scarce resources,
among a number of competing activities.
!Powerful and general problem-solving method that encompasses:
– shortest path, network flow, MST, matching
– Ax = b, 2-person zero sum games

Why significant?
!Widely applicable.
!Fast commercial solvers: CPLEX, OSL.
!Powerful modeling languages: AMPL, GAMS.
!Ranked among most important scientific advances of 20th century.
!Dominates world of industry.

Ex: Delta claims saving $100 million per year using LP

Reference: The Allocation of Resources by Linear Programming, Scientific American, by Bob Bland.

Robert Sedgewick and Kevin Wayne • Copyright © 2005 • http://www.Princeton.EDU/~cos226 2

Applications Brewery Problem: A Toy LP Example

Agriculture. Diet problem. Small brewery produces ale and beer.

Computer science. Compiler register allocation, data mining. !Production limited by scarce resources: corn, hops, barley malt.
Electrical engineering. VLSI design, optimal clocking. !Recipes for ale and beer require different proportions of resources.
Energy. Blending petroleum products.
Economics. Equilibrium theory, two-person zero-sum games. Corn Hops Malt Profit
Beverage
(pounds) (ounces) (pounds) ($)
Environment. Water quality management.
Finance. Portfolio optimization. Ale (barrel) 5 4 35 13

Logistics. Supply-chain management. Beer (barrel) 15 4 20 23

Management. Hotel yield management. Limit 480 160 1190
Marketing. Direct mail advertising.
Manufacturing. Production line balancing, cutting stock. How can brewer maximize profits?
Medicine. Radioactive seed placement in cancer treatment. !Devote all resources to ale: 34 barrels of ale ! $442.
Operations research. Airline crew assignment, vehicle routing. !Devote all resources to beer: 32 barrels of beer ! $736.
Physics. Ground states of 3-D Ising spin glasses. !7.5 barrels of ale, 29.5 barrels of beer ! $776.
Plasma physics. Optimal stellarator design. !12 barrels of ale, 28 barrels of beer ! $800.
Telecommunication. Network design, Internet routing.
Sports. Scheduling ACC basketball, handicapping horse races.
3 4
 Brewery Problem Brewery Problem: Feasible Region

Hops Malt
Ale Beer 4A + 4B " 160 35A + 20B " 1190

max 13A + 23B Profit

s. t. 5A + 15B " 480 Corn
(0, 32)
4 A + 4B " 160 Hops
35A + 20B " 1190 Malt (12, 28)
A , B # 0
Corn
5A + 15B " 480
(26, 14)
!

Beer

(0, 0) Ale (34, 0)

5 6

Brewery Problem: Objective Function Brewery Problem: Geometry

Brewery problem observation. Regardless of objective function

coefficients, an optimal solution occurs at an extreme point.

Pr
of
it

(0, 32) (0, 32)

(12, 28) (12, 28)

13A + 23B = $1600

extreme point

(26, 14) (26, 14)

Beer 13A + 23B = $800 Beer

(0, 0) Ale (34, 0) (0, 0) Ale (34, 0)

13A + 23B = $442
7 8
 Standard Form LP Brewery Problem: Converting to Standard Form

"Standard form" LP. Original input.

! Input: real numbers aij, cj, bi. max 13A + 23B
! Output: real numbers xj. s. t. 5A + 15B " 480
! n = # nonnegative variables, m = # constraints. 4 A + 4B " 160
! Maximize linear objective function subject to linear inequalities. 35A + 20B " 1190
A , B # 0

n
(P) max c T x
(P) max "cj x j Standard form. !
s. t. Ax = b
Add slack variable for each inequality.
j=1
n !

s. t. " aij x j = bi 1 # i # m x " 0

j=1 ! Now a 5-dimensional problem.
xj $ 0 1# j #n

!
max 13A + 23B
! s. t. 5A + 15B + SC = 480
Linear. No x2, xy, arccos(x), etc. 4 A + 4B + SH = 160
Programming. Planning (term predates computer programming). 35A + 20B + SM = 1190
A , B , SC , SH , SM " 0

9 10
!

Geometry Geometry

Geometry. Extreme point property. If there exists an optimal solution to (P),

! Inequality: halfplane (2D), hyperplane (kD). then there exists one that is an extreme point.
! Bounded feasible region: convex polygon (2D), convex polytope (kD).
Consequence. Only need to consider finitely many possible solutions.
Convex set. If two points a and b are in the set, then so is !(a + b).
Challenge. Number of extreme points can be exponential!
Extreme point. A point in the set that can't be written as !(a + b),
n-dimensional hypercube
where a and b are two distinct points in the set.

extreme Greedy property. Extreme point is optimal

point iff no neighboring extreme point is better.

local optima are global optima

convex not convex

11 12
 Simplex Algorithm Simplex Algorithm: Basis

Simplex algorithm. [George Dantzig, 1947] Basis. Subset of m of the n variables.

! Developed shortly after WWII in response to logistical problems,
including Berlin airlift. Basic feasible solution (BFS). Set n - m nonbasic variables to 0,
! One of greatest and most successful algorithms of all time. solve for remaining m variables.
! Solve m equations in m unknowns.
Generic algorithm. never decreasing objective function ! If unique and feasible solution ! BFS.
! Start at some extreme point. ! BFS # extreme point.
! Pivot from one extreme point to a neighboring one.
Basis
! Repeat until optimal. {A, B, SM }
{B, SH, SM } (12, 28)
(0, 32) Infeasible
{A, B, SH }
How to implement? Linear algebra. (19.41, 25.53)

max 13A + 23B

Beer {A, B, SC }
s. t. 5A + 15B + SC = 480 (26, 14)
4 A + 4B + SH = 160
35A + 20B + SM = 1190
A , B , SC , SH , SM " 0

Ale {A, SH, SC }

{SH , SM, SC }
(34, 0)
(0, 0)
13 ! 14

Simplex Algorithm: Initialization Simplex Algorithm: Pivot 1

max Z subject to max Z subject to

Basis = {SC, SH, SM} Basis = {SC, SH, SM}
13A + 23B " Z = 0 A=B=0 13A + 23B " Z = 0 A=B=0
5A + 15B + SC = 480 Z=0 5A + 15B + SC = 480 Z=0
SC = 480 SC = 480
4A + 4B + SH = 160 4A + 4B + SH = 160
SH = 160 SH = 160
35A + 20B + SM = 1190 SM = 1190 35A + 20B + SM = 1190 SM = 1190
A , B , SC , SH , SM # 0 A , B , SC , SH , SM # 0

Substitute: B = 1/15 (480 – 5A – SC)

! !

max Z subject to
16 A " 23 SC " Z = "736 Basis = {B, SH, SM}
3 15 A = SC = 0
1 A + B + 1 SC = 32 Z = 736
3 15
8 4 B = 32
A " SC + SH = 32
3 15 SH = 32
85 4
3
A "
3
SC + SM = 550 SM = 550
A , B , SC , SH , SM # 0

15 16

!
 Simplex Algorithm: Pivot 1 Simplex Algorithm: Pivot 2

max Z subject to
max Z subject to
Basis = {SC, SH, SM} Basis = {B, SH, SM}
13A + 23B " Z = 0 A=B=0
16
3
A " 23
15
SC " Z = "736 A = SC = 0
Z=0
1
3
A + B + 1
15
SC = 32 Z = 736
5A + 15B + SC = 480 B = 32
SC = 480 8 A " 4 SC + SH = 32
4A + 4B + SH = 160
SH = 160
3
85
15
4
SH = 32
A " SC + SM = 550
35A + 20B + SM = 1190 SM = 1190
3 3 SM = 550
A , B , SC , SH , SM # 0
A , B , SC , SH , SM # 0

!
! Substitute: A = 3/8 (32 + 4/15 SC – SH)
Why pivot on column 2?
!Each unit increase in B increases objective value by $23.
max Z subject to
!Pivoting on column 1 also OK. Basis = {A, B, SM}
" SC " 2 SH " Z = "800 SC = SH = 0
B + 1 SC + 18 SH = 28 Z = 800
Why pivot on row 2? 10
1 3 B = 28
A " SC + SH = 12
!Preserves feasibility by ensuring RHS $ 0. 10
25
8
85
A = 12
" SC " SH + SM = 110 SM = 110
Minimum ratio rule: min { 480/15, 160/4, 1190/20 }.
6 8
!
A , B , SC , SH , SM # 0

17 18

Simplex Algorithm: Optimality Simplex Algorithm: Bare Bones Implementation

Q. When to stop pivoting? Construct the simplex tableaux.

A. When all coefficients in top row are non-positive. m A I b

1 c 0 0
Q. Why is resulting solution optimal? n n 1
A. Any feasible solution satisfies system of equations in tableaux.
! In particular: Z = 800 – SC – 2 SH public class Simplex {
! Thus, optimal objective value Z* " 800 since SC, SH $ 0. private double[][] a; // simplex tableaux
private int M, N;
! Current BFS has value 800 ! optimal.
public Simplex(double[][] A, double[] b, double[] c) {
M = b.length;
N = c.length;
max Z subject to a = new double[M+1][M+N+1];
Basis = {A, B, SM} for (int i = 0; i < M; i++)
" SC " 2 SH " Z = "800 SC = SH = 0 for (int j = 0; j < N; j++)
B + 1 SC + 18 SH = 28 Z = 800 a[i][j] = A[i][j];
10
A " 1 SC + 3 SH = 12 B = 28 for (int j = N; j < M + N; j++) a[j-N][j] = 1.0;
10
25
8
85
A = 12 for (int j = 0; j < N; j++) a[M][j] = c[j];
" SC " SH + SM = 110 SM = 110
6 8
for (int i = 0; i < M; i++) a[i][M+N] = b[i];
A , B , SC , SH , SM # 0 }

19 20

!
 Simplex Algorithm: Bare Bones Implementation Simplex Algorithm: Bare Bones Implementation

q q
Pivot on element (p, q). Simplex algorithm.
p p + +

public void solve() {

while (true) {
public void pivot(int p, int q) { int p, q;
for (int i = 0; i <= M; i++) for (q = 0; q < M + N; q++) find entering variable q
for (int j = 0; j <= M + N; j++) if (a[M][q] > 0) break; (positive objective function coefficient)

if (i != p && j != q) if (q >= M + N) break;

a[i][j] -= a[p][j] * a[i][q] / a[p][q];
for (p = 0; p < M; p++) find row p according to min ratio rule

for (int i = 0; i <= M; i++) if (a[p][q] > 0) break;

if (i != p) a[i][q] = 0.0; zero out column q for (int i = p+1; i < M; i++)
if (a[i][q] > 0)
for (int j = 0; j <= M + N; j++) if (a[i][M+N] / a[i][q] < a[p][M+N] / a[p][q])
if (j != q) a[p][j] /= a[p][q]; scale row p p = i; min ratio test
a[p][q] = 1.0; pivot(p, q);
} }
}
21 22

Simplex Algorithm: Degeneracy Simplex Algorithm: Running Time

Degeneracy. New basis, same extreme point. Remarkable property. In practice, simplex algorithm typically
terminates after at most 2(m+n) pivots.
"stalling" is common in practice
! No polynomial pivot rule known.
! Most pivot rules known to be exponential (or worse) in worst-case.

Pivoting rules. Carefully balance the cost of finding an entering

variable with the number of pivots needed.

Cycling. Get stuck by cycling through different bases that all

correspond to same extreme point.
! Doesn't occur in the wild.
! Bland's least index rule guarantees finite # of pivots.

23 24
 Simplex Algorithm: Implementation Issues LP Solvers

Implementation issues. AMPL. [Fourer, Gay, Kernighan] An algebraic modeling language.

!Avoid stalling. CPLEX solver. Industrial strength solver.
separate data from model
!Choosing the pivot.
!Numerical stability. requires fancy data structures

!Maintaining sparsity. set PROD := beer ale;

set INGR;
set PROD;
beer.mod
set INGR := corn hops malt;
!Detecting infeasiblity param profit {PROD};
param: profit := param supply {INGR};
!Detecting unboundedness. ale 13 param amt {INGR, PROD};
beer 23; var x {PROD} >= 0;
!Preprocessing to reduce problem size.
param: supply := maximize total_profit:
corn 480 sum {j in PROD} x[j] * profit[j];
hops 160
subject to constraints {i in INGR}:
malt 1190;
sum {j in PROD} amt[i,j] * x[j] <= supply[i];
Commercial solvers routinely solve LPs with millions of variables and param amt: ale beer :=
corn 5 15
tens of thousands of constraints. hops 4 4
malt 35 20; [cos226:tucson] ~> ampl
beer.dat AMPL Version 20010215 (SunOS 5.7)
ampl: model beer.mod;
ampl: data beer.dat;
ampl: solve;
CPLEX 7.1.0: optimal solution; objective 800
ampl: display x;
x [*] := ale 12 beer 28;

25 26

LP Duality: Economic Interpretation LP Duality

Brewer's problem. Find optimal mix of beer and ale to maximize profits. Primal and dual LPs. Given real numbers aij, bi, cj, find real numbers xj, yi
that optimize (P) and (D).

(P) max 13A + 23B

A* = 12
s. t. 5A + 15B " 480 n m
B* = 28 (P) max "cj x j (D) min " bi yi
4 A + 4B " 160 OPT = 800 j=1 i=1
35A + 20B " 1190 n m
s. t. " aij x j # bi 1 # i # m s. t. " aij yi # cj 1$ j $n
A , B # 0 j=1 i=1
xj $ 0 1# j #n yi # 0 1$i $m

!
Entrepreneur's problem. Buy resources from brewer at min cost. ! !
! C, H, M = unit price for corn, hops, malt. Duality Theorem. [Gale-Kuhn-Tucker 1951, Dantzig-von Neumann 1947]
! Brewer won't agree to sell resources if 5C + 4H + 35M < 13. If (P) and (D) have feasible solutions, then max = min.

(D) min 480C + 160H + 1190M C* = 1

s. t. 5C + 4H + 35M " 13 H* = 2
15C + 4H + 20M " 23 M* = 0
OPT = 800
C , H , M " 0

27 28

!
 LP Duality: Sensitivity Analysis History

Q. How much should brewer be willing to pay (marginal price) for 1939. Production, planning. [Kantorovich]
additional supplies of scarce resources? 1947. Simplex algorithm. [Dantzig]
A. Corn $1, hops $2, malt $0. 1950. Applications in many fields.
1975. Nobel prize in Economics. [Kantorovich and Koopmans]
Q. How do I compute marginal prices (dual variables)? 1979. Ellipsoid algorithm. [Khachian]
A. Simplex solves primal and dual simultaneously. 1984. Projective scaling algorithm. [Karmarkar]
1990. Interior point methods.
objective row of final simplex tableaux 200x. Approximation algorithms, large scale optimization.
provides optimal dual solution!

Q. New product "light beer" is proposed. It requires 2 corn, 5 hops,

24 malt. How much profit must be obtained from light beer to justify
diverting resources from production of beer and ale?
A. Breakeven: 2 ($1) + 5 ($2) + 24 ($0) = $12 / barrel.

29 30

Simplex vs. Interior Point Methods Ultimate Problem Solving Model

Ultimate problem-solving model?

! Shortest path.
! Maximum flow.
! Assignment problem. tractable
! Min cost flow.
! Multicommodity flow.
! Linear programming.
! Semidefinite programming.
! Nash equilibrium.
! ... complexity unknown

! TSP (or any NP-complete problem)

interior point faster when polyhedron simplex faster when polyhedron intractable (conjectured)
smooth like disco ball spiky like quartz crystal

Does P = NP? No universal problem-solving model exists unless P = NP.

31 32
 Assignment Problem Assignment Problem: Applications

Assignment problem. Assign n jobs to n machines to minimize total Natural applications.

cost, where cij = cost of assignment job j to machine i. !Match jobs to machines.
!Match personnel to tasks.
!Match PU students to writing seminars.

Non-obvious applications.
1' 2' 3' 4' 5' 1' 2' 3' 4' 5'
!Vehicle routing.
1 3 8 9 15 10 1 3 8 9 15 10 !Signal processing.
2 4 10 7 16 14 2 4 10 7 16 14
!Virtual output queueing.
!Multiple object tracking.
3 9 13 11 19 10 3 9 13 11 19 10 !Approximate string matching.
4 8 13 12 20 13 4 8 13 12 20 13 !Enhance accuracy of solving linear systems of equations.

5 1 7 5 11 9 5 1 7 5 11 9

cost = 3 + 10 + 11 + 20 + 9 =53 cost = 8 + 7 + 20 + 8 + 11 = 44

33 34

Assignment Problem: LP Formulation Perspective

LP is near the deep waters of NP-completeness.

min # # cij xij ! Solvable in polynomial time.
1"i " n 1" j " n
Interpretation: if xij = 1, then
! Known for % 25 years.
s. t. # xij = 1 1"i"n
1" j " n assign job j to machine i
# xij = 1
1"i " n
1 " j" n Integer linear programming.
xij $ 0 1 " i, j " n ! LP with integrality requirement.
! NP-hard.

! Theorem. [Birkhoff 1946, von Neumann 1953] All extreme points of

the above polyhedron are {0-1}-valued.

Corollary. Can solve assignment problem by solving LP since LP

algorithms return an optimal solution that is an extreme point.

An unsuspecting MBA student transitions from tractable LP to

intractable ILP in a single mouse click.

35 36