Subject Index
Acceptable point 27 fL 34 ff
Accumulation point 19
Active constraint 142, 199
Active set 303
Active set method 160, 240 fL
264 fL 26R, 303 , 316
primal 240 ff, 264 ff
dual 243
Actual reduction 96, 393 , 397
Ad hoc methods 16 ff, 142
Affine function 9
Affine transformation 57
Alternating variables method IR, 22
Angle criterion 30, 31
Applications 4
Arc 344
Artificial arc 352
Artificial constraint 167
Artificial variables 162 ff
Assignment problem 353
Augmented Lagrangian function 2R9,
314, 323
Barnes' formula 130
Barrier function 187, 286 , 326
Basic feasible solution (b.f.s.) 153,
227
initial 162
optimal 154
Basic NSO 383, 396
Basic variables 153
Bartels-Golub method 171
Basis 347
Basis matrix 153
Beale 's method 254
BFGS formula 59.66,78 , 310.316
BFGS method 55 ff, 67, 69 , ff, 114
B.f.s., see Basic feasible solution
Big M method 167
Binding constraint 142
Bisection 16 , 37
Bounds 143 , 151, 24R. 302
Boxstep method 99
Bracket 33 fL 39
Branch and bound method 331 ff
Branching 332
Brown-Brent method 132
Broyden family 56 , 62 ff
Broyden method 64
Broyden's formula 132
Bunch-Parlett factors 239
Bundle method 385
Canonical form 157
Capacities 349
Censored L approximation 359
Central difference 23
Chain rule R, 370
Chebyshev semi-iterataion 124
Choleski factors 15, 118.238, 244
Colemen- Conn method 316
Complementarity 201, 303
strict 201, 27 L 273, 294 , 376, 391
Complementary formula 55
Compleme ntary pivoting 250 ff
Composite function 359, 369
Composite NSO 359, 388
Concave function 216
Condition number 31, 118,236
Cone 205
Index 431

Conic function 73, 77 Cycle 344, 350, 354

Conjugate descent formula 1\3 Cycling 156, 177, 242
Conjugate direction method 24, 25,
80 ff
Dantzig two-phase method 332
Conjugate directions 25 , 54
Dantzig-Wolfe decomposition 175
Conjugate gradient method 26, 54,
Dantzig-Wolfe method 250
65, 80 ff
Data fitting Ill, 357
Conjugate subgradient method 385
Davidenko path method 126 ff
Constrained composite NSO 402
Davidon formula 72
Constrained NSO 398 ff
DC network 353
Constraint
Decomposition 357
equality 140, 195,207,230,259 ff,
Decomposition method
308
(Dantzig-Wolfe) 175
function 140
Degeneracy 156, 177 ff, 242, 353
inequality 140, 199,209,240,
Degenerate value 64
264 ff, 286, 292, 306, 371\
Degree of difficulty 342
linear 143, 162 ff, 229
Dennis-More theorem 25, 39, 125,
strict 146
315
Constrain qualification 202, 212, 4{)4
Depth-first ordering 351
Constraints 5
Derivatives 8, 14
Continuation method 126 ff
Descent direction 204, 405
Contour 7
Descent method 22, 26, 30
Contraction mapping 124
Descent property 22,31, 50, 94
Convergence
DFP formula 54,59, 310
first order 20,22, 113, 123
DFP method 54ff, 67, 69ff
global 19,22,30,54,96,272,21\1,
Differentiable arc 197
292, 321, 397
Difference approximation, see Finite
linear, see first order
differences
local 20,46, 120,391
Direction 6
order of 20
Direction set method 87
quadratic, see second order
Directional derivative 369, 391\
rate of 20, 115
Directional sequence 201
second order 20, 46, 98, 120, 308,
Dixon's theorem 66
388, 391
superlinear 20, 25, 54, 125, 275 Dog-leg method 131
Dog-leg trajectory 107
two-step super1inear 315
DSC method 19
Convergence test 22
Dual active set method 243
Convex class 68
Dual formula 55
Convex combination 213
Dual function 294
Convex cone 206, 406
Convex function 67, 214 ff, 340, 364, Dual LP 220
Dual norm 379
414
Dual problem 182, 219 ff
polyhedral 360, 365, 392
DualOP 222
strict 215
Dual transformation 219 ff, 249, 340,
Convex hull 213
416
Convex programming problem 216,
Duality 219 ff
218,219,340,416
Convex set 213
Convexity 213 ff Earth node 346
local 218 Edge 160
Coordinate vector 9 Elimination 145, 23(), 234, 319
Cost function 150, 181 generalized 231, 238, 260, 317, 354
Courant penalty function 279 ff nonlinear 317
Curvature 8, 209 Ellipsoid method 184
Cutting plane method 324, 386 EP, see Equality problem
432
Epigraph 214, 364
Equality constraint 140, 195, 207 ,
230 , 259 ff, 308
Equality constraint problem 278, 378
Equality problem (EP) 240, 264
Eta vector 169
Euclidean norm 73 , 76
Exact line search 21, 64 ff, 81
Exact pena lty function 296, 323, 357,
378 ff, 392
Exchange algorithm 146
Exterior penalty function 286
External flow 345
Extrapolation 37, 283
Extreme point 152,214, 227
Farkas' lemma 205, 407
Feasible arc 211, 317
Feasible direction 196 ff. 201, 210,
370, 403, 407
Feasible direction method 317 ff,
395
Feasible point 140, 162
initial 162, 242
Feasible region 140
Fletcher- Matthews method 172
Fibonnaci section search 39
Finite difference approximation 23 ,
129
Finite difference Newton method 49,
261
Finite differences 49, 56, 261, 262,
264
First order conditions 272, 354, 381,
389,399,406,412
First order convergence 20 , 22, 113,
123
First order necessary conditions 14,
96, 200 ff, 272. 298 fL 368, 370
Fixed point 122
Fletcher-Reeves method 65, 80 ff
Flow 345
Forward difference 23 , 129
Gauss-Jordan product form 169
Gauss-Newton method 112 fL 387
with line search 114
Gauss-Seidel method 123
Generalized elimination method 231 ,
238, 260, 317, 354
Genera lized gradient 369, 398
Generalized inverse 131, 199
Generalized least squares method
112
Index
Geometric programming problem
339
Global convergence 22, 30, 54, 96 ,
272 , 281 , 292 , 321,397
Global convergence proof 19
Global minimization 129, 217
Global minimizer 12
Global minimum 101
Golden section search 39
Goldfarb and Idnani method 243
Goldstein conditions 27 ff
GQT method 48, 100
Gradient projection method 262, 319
Gradient vector 8
Graph 344
GRG method 319
Groove 296, 313 , 360
Hereditary property 53, 64, 130
Hessian matrix 8, 207, 305
Hooke and Jeeves method 19
Horizontal step 316
Hoshino formula 68
Huang family 63
Hybrid method 117, 396
Hypercube method 99
Hypercube trajectory 108
Ill-conditioning 284, 287 , 288, 292
Incremental step 195
Indefinite matrix 16, 245
Indefinite QP 245
Inequality constraint 140, 199 , 209,
240, 264 ff. 286, 292
Inequality constraint problem 278,
306, 378
Inexact line search 22 , 30 , 71
Infeasible problem 151, 224 , 311
Integer LP problem 337
Integer programming 331 ff
Integer variable 5, 322
Interior point method 286
Interpolation 33, 37
Invariance 57 ff
Inverse barrier function 286, 326
Isolated local minimizerl3, 14 , 211,
289 , 372
Iterative method 6
Jacobi method 123
Jacobian matrix Ill, 141 , 318
Jump discontinuity 294
Karmarkar's method 185 ff
Kirchoffs laws 354
Index
Knot 303
Kt point 200,245, 282, 289, 301, 381
Kuhn- Tucker (KT) conditions 200 ,
217 , 272 , 283 , 299 , 310,314
see also First order necessary
conditions
L I exact penalty function 167, 296 ff,
311
L, norm Ill , 148 , 329, 363, 412
L,LP problem 302, 317, 330
L,QP problem 243 , 254 , 303 , 312,
393
Lx exact penalty function 323
L2 norm 9, 248
Lx norm 111,148,363,412
Lagrange multiplier 61, 101, 160,
195 ff, 219, 232, 263, 299, 341,
363
estima te 282, 321, 323, 325 , 326
method 145, 197,236,389
Lagrange-Newton method 125 , 305
Lagrangian function 61, 101, 198,
2]9 , 370, 406
Lagrangian matrix 236, 283, 308
Large residua l problem 115
LDLT factors 15,44,49,57 , 239
Least distance problem 248, 258
Least squares problem 323
Least squares solution 110, 248
Lemke's method 253
Level set 98
Levenberg- Marquardt method 48,
100 ff. 116 , 261 , 312
Levenberg- Marquardt trajectory 103
Lexicographic ordering 179
Line 6
Line search 26,47 , 113 , 160, 182,
382
exact 21. 64 ft, 81
Fibonnaci 39
golden section 39
inexact 22 , 30 , 71
no-derivitive 39
Line search method 21, 33 ff
Linear approximation 306, 312, 324,
387, 388, 395
Linear complementary problem 250,
304
Linear constraint 143, 162 ff. 229
Linear convergence 20 , 22
see also First order convergence
Linear function 9, 143, 216
Linear model 112
433
Linear network flow problem 349
Linear programming 150 ff. 220
dual 220
Linear programming problem 217
Linear transformation 231
Linearization 119, 143,215
LU factors 15, 48, 237
Local convergence 20, 46, 120, 391
Local convexity 218
Local minimizer 12 ff, 140,245
isolated 13,14,211 , 289,372
strict 13, 14, 208, 211, 301, 372,
409
Locally active constraints 313
Locally linear function 374, 391, 404,
410
Locally Lipschitz function 359
Logarithmic barrier function 187,
286, 326
LP, see Linear programming
Lu factors 119, 170 ff
MAP 324
Maratos effect 313, 393
Matrix 6
Max function 357
Maximization 12
Maximizer 16, 197
Maximum entropy problem 222
Maximum likelihood solution 112
Method of Lagrange multipliers 145 ,
197, 236, 389
Metric 59 , 99
Minimizer 12
see also Local and/or Global
minimizer
Min-max approximation 357
Mixed integer programming 331
Model 20
Monotonic norm 379, 414
Multiplier , see Lagrange multiplier
Multiplier penalty function 287 ff.
323
Necessary conditions 14
first order 14, 96 , 200 ff. 272 ,
298 ff, 368, 370
second order 14, 96, 208 , 210 ,
272, 301, 372, 408
Negative definite matrix 16
Neider and Mead method 18
Network flow problems 345 ff
Network programming 344 ff
Newton-like method 24, 38 , 58
434
Newton- Raphson method 119ff, 125 ,
318, 389
Newton's method 24, 44 ff, 58, 125,
260, 267, 273 , 291, 305, 319, 354
finite difference 49, 261
with line search 47
No-derivative line search 39
No-derivative methods 129 ff, 262
Node 333 , 344
earth 346
Node-arc incidence matrix 345
Non-differentiable function 296
Non-differentiable optimization
357 ff
Non-smooth function 13 , 296
Non-smooth optimization 357 ff
Non-unique solution 155
Nonbasic variables 153
Nonlinear elimination 317
Nonlinear equations 119 ff, 357
Nonlinear least squares 34. 110 ff
Nonlinear programming 143 , 277 ff
Nonlinear transformation 318, 321
Norm 9
Euclidean 73 , 76
LJ 111 , 148, 329 , 363,412
L2 9, 248
L, O Ill, 148,363,412
weighted 60
Normal equations 113, 187
Normal vector 141
NSO, see Non-smooth optimization
Null space 231
Null space matrix 354
Null space method 237
0(.), 0(.) notation Xlii
Objective function 5, 12, 140, 150
Optimal basic feasible solution 154
Optimal conditioning 73
Optimality 350
Optimality conditions 12 ff, 297 ff
Optimality test 155, 182
Order of convergence 20
Orthogonal factors 234
see also QR factors
Orthogonal factorization method 263
Out-of-kilter algorithm 353
Over determined systems 110 ff, 357
Parallel subspace property 87
Parametric programming 159. 165 ,
247
Partial derivatives 8
Index
Penalty function 278 ft, 322
augmented Lagrangian 289, 314, 323
Courant 279
exact 296, 323, 357, 378 ff, 392
exterior 286
LJ exact 167,296 ff, 311
Lx exact" 323
multiplier 287 ff, 323
sequential 278 ft
smooth exact 314
Penalty parameter 279, 288, 296, 304
Phase I1phase II method 163 ff , 181 ,
352
Piece 360
Pivot 157 , 251
Point 6
Polak-Ribiere formula 83
Polyhedral cone 205
Polyhedral convex function 360, 365 ,
392
Polynomial time algorithms 183 ff
Positive definite matrix 14, 31, 44,
46, 50, 54, 10 I , 216, 230 , 260
Positive semi-definite matrix 14, 42.
101 , 115 , 215 , 260
Posynomial 339
Potential 350
Potential difference 349
Powell-Hestenes method 291 ff
Powell's method 89 ff
Power 349, 353
Preconditioning 86
Predicted reduction 96, 393, 397
Primal active set method 240 ff,
264 ff
Primal dual method 244
Primal problem 219 ff
Principal pivoting method 250
Priorities 337
Product form 78
Product form method 169
Programming 3
Projection formula 83
Projective transformation 186
Prototype algorithm 21, 95, 398
Pseudoconstraints 99 , 160, 167 , 246
QL method 388
QP , see Quadratic programming
QR factors 118, 172, 233 , 238
Quadratic approximation 306, 312,
388
Quadratic convergence 20
see also Second order conve rgence
Index
Quadratic function 10, 24, 42, 54,
64, 95, 216, 229
Quadratic model 21,22,24,44,267
Quadratic programming 229 ff
dual 222
Quadratic programming problem
217, 221
Quadratic programming subproblem
307, 322, 324
Quadratic slack variable 146
Quadratic termination 25, 54, 266
Quasi-Newton condition 51, 54, 61,
130
Quasi-Newton method 24, 49 ff, 58,
125, 130,291,310,312.314.354
Quasi-triangular basis 347
Random search 16
Range space method 238
Rank one formula 51 ff. 62, 77
Rank one matrix 51
Rank one method 51 ff
Rank two formula 53 ff. 61
Rate of convergence 20, liS
Ratio test 156, 350
Recursive method 181
Reduced coordinates 232
Reduced costs 155
Reduced gradient method 235, 319
Reduced gradient vector 232, 260
Reduced Hessian matrix 232, 260,
266, 315, 320
Regularity assumption 204 , 372, 374,
406, 408
Reinversion 169
Relative interior 376
Residual 110. 111. 162
Restricted step method 21. 24. 95 ff,
261, 267, 312, 387. 397
Revised simplex method 158
p-Iine 29
Rooted spanning tree 346
Rosenbrock's function 7,313
Round-off error 181
Row operations 157
Saddle point 16, 197
Scalar product 6
Scaling 59, 148
Secant method 130 ff
Second order conditions 207 ff, 218,
412
435
Second order constraint qualification
212
Second order convergence 20. 46,
98, 120, 308. 388, 391
Second order correction step 313,
316, 394 ff, 401
Second order necessary conditions
14, 96, 208. 210, 272. 301, 372.
408
Second order sufficient conditions
98,211,273,282,289,301.306.
308, 372. 380, 391, 409
Sectioning 16. 33 ff
Self-dual formula 55. 76
Self-scaling method 74
Sensitivity 159
Separable programming 324. 354
Separating hyperplane lemma 206.
366, 406
Sequential LIQP method 312
Sequential non-smooth quadratic
programming method 125, 388 ff
Sequential penalty function 278 ff
Sequential quadratic programming
method 307 ff
see a/so SQP method
Sherman-Morrison formula 76. 158
Shortcut method 285
Simplex method 17. 153 ff. 349
Singular value decomposition 119
SLIQP method 312
Slack variable 146. 150, 163
Slope 8
Small residual problem 114
Smith's met hood 88
Smooth exact penalty function 314
SNQP method 125, 312, 388 ff. 401
SOC step 313.316.394 ff. 401
Software 139
SOLVER method 307.312
SOR method 124
Spanning tree 344 ff
Sparse matrix 73, 129. 168
Special ordered sets 331
SQP method 125. 244, 267. 307 ff.
354, 388, 391
Stack 334
Stack method 334
Standard form 150
Stationary point 16, 93 . 198. 305
Steepest ascent vector 11
Steepest descent method 22, 384
Steepest descent direction 22, 369
436
Strict complementarity 201, 271, 273,
294, 376, 391
Strict constraint 146
Strict local minimizer 13, 14, 208 ,
211, 301, 372, 409
Strictly active constraints 210
Strictly concave function 216
Strictly convex function 215
Structure functional 374, 376
Subdifferential 362, 364, 379 , 414
Subgradient 364
Subgradient inequality 364
Subgradient optimization 384
Successive substitution method 122
Sufficient conditions 14, 208
see also Second order sufficient
conditions
Sums of squares 110 ff
SUMT 326
Superlinear convergence 20 , 25, 54,
125 , 275
2-step 315
Supporting hyperplane 215, 364
Symmetric dual 221
Table of notation xiii
Tableau method 157 , 164 , 251
Tangent cone 202
Taylor series 10, 44, 305
Termination 156, 242, 274
Termination test 22
Test functions 20 , 69, 114
Transformations 144 ff
Index
Transportation problem 353
Tree 333, 344
Trust region 21, 95, 312, 397, 401
Unbounded problem 141, 151,223 ,
311
Unbounded variable 151
Updating formula 50 ff, 130, 244
Variable 5
integer 5, 332
unbounded 151
zero-one 331
Variable metric method 50, 60
Variance-covariance matrix 112
Variational method 73
Variational property 60 ff
Vector 6
Vertex 152 , 181, 274, 324
initial 162 , 242
Vertical step 316
Weighted least squares problem 187
Weighted norm 60
Well-determined system 119 ff
Wild point III
Wolfe dual 219
Wolfe- Powell conditions 28 ff, 37,
383
Zero-one variable 331
Zero residual problem 114
Zigzagging 268 ff, 321