DOVER BOOKS ON MATHEMATICS aggtoox o Nimauneat ene, Milton Abramowits and ene A ‘Stun (612729). “Tem dao or ro, lead Sakop apd Samuel L Gldbers ‘eas “Tor herr Nous, G Pe Bol. 602257) ‘Vesrou sto Teor hrs wrt Arron, A ovisenk and aap (88102) ‘pertieony op me Carats Mo hs Carrow, Deena, Ca B. oye. (tbo) ‘Tw Quam Taam of Onanasy Deve Eau AM srr, Ped ser and Soa A Nabe! (68465), conmeoe nuznn Wrnc ewan Richard Pret (AER) Paneetus oe Stree MG Buln (63750) “Te evo ros, Ee Cartan, (407) ose Nan Tass Harvey Can (623%) Satu Au, Edw L. Crom Fas Dal, and Margret Matick ‘on xn Sis Oxroct ieres Hay Fv (58739) Conny a Ueno, Marin Davis (LATS) ‘euro Ms Auch de Bra, (4215) Pan Gro Ts, oh D Dap. (1574%) “Tne Manes o aes Sra Min Dresher, (216%) done Pam Dremoran Bg, Past DuChateas spd Dad “achmann ise) Aspro Buns A Erdal. S18) Coxmax Vasutcs Hanne a Arte Poca Pangan ‘ssa ‘x Fomsty Uomomisi Promos oF Pancrik Manu 0 Reutes Sar, Kart Goel (68807) ‘A ror or Gros Mavs, Sir Tras Heath, (40738, 240746) "Teowohine st Gite eroruenn ro Nowe Ass, rane Hbrand (652699) enc or Arto Memes, Fra gered 672) “Toys John G. Hocking and Gall. Young (tT64), asc so Lace, Mark Kec and Sti M. Um (7085.5) iowa Furnes or brates Tb A 1 hich. (6BE4S) ‘amine rE, A Albee Kh 212416) Caco Rees, A Abert Ke (20700) roa Hoo Tanto ines Konrad Kapp (SS) Imarcreey RA Aas, ANKelmogoroy aS. Pom. (61268) omtinedon back Map)OPTIMAL CONTROL AND ESTIMATION DOVER PUBLICATIONS, INC. New YorkTo Pegi, Brooke, and Christopher Copyright Ccapigh © 1986, 1958 Robe Sea Sia eared nd en Aeron sn ners Copy Comer. Bibliographical Note “This De io, uid in 9%, ea mabe crmcted repblie ton fe nk ft pnb y itn ily & Sens CA, Wine Paton New Ys une ee Sth Optimal Cnel: Try end ‘ontario, ois sain thst nce aber of es dws Pres ad an sbogapbiea semen Library of Congress Cataloging Publication Data Sent, oer : {Sica pial on ‘pla cone! nd sntin Raber Sten ‘vil pli: Statins cnt New Yor Wil 16, With Ticonol theory. 2. Mathematical opinatization. 3. tochatle Guat 96 se ance in i Unit St of ein Dove Pano 31 Eat Stet M,N. 150 $$$, PREFACE TO THE DOVER EDITION 1 am paicly peste that the Dover eon of my book (rm ened ‘Sochaste Optinal Conro is being reprinted unde he ile Optimal Cond and ‘stimaton. The new le more lal eet the structure of ie text, which wets contol and estimation separately beloe combining them to form sochartic coal tes ants simpli weiforcsthe message tat his bok is sot racial cont system design and stu etna, ‘Since the original pulication ofthe book ia 1986, the Feld ss changed in two Important ways. Camper prog hat atte he dsign at sli of wal ‘arable con and estima lpi have bacome weslale. The technique ‘Serbo in th bok a ely implements using compe aided -desgn tol ‘Aoclly, new techniques for designing robust lear conto syste (6. contol stems tht prvi saison stub and performance nthe preseas of| plant uovertznty) have emerged. The book contains considerable materi om froqunsy-domainmoling of ies lara sytem, mn propre and stability margins of opsilly regulated, aking non-7e0 set poi choice ff eostinction weights (Q, MR) apd onl stucures (cluding mode li Ing, reqencyshpig, and ig compertion), Spt (lds) eu lens of continuous eatlers, lop treaer eee and sgl so Bi Italo intodies probabilistic evaluation of contolsystem busines, which Proves an atactve approach fr elting he iktinood of suis loop Prfonance to uncial parameters. ‘Opimel Coral and Esiation geste exp betacen case! metas of contol analy and modern methods of controls synthesis. The mete Aseribes are especialy valuable i obots, eco conl pros conta, ea Ines, and inliget onl syst dv lopment Roser Sr.PREFACE TO THE FIRST EDITION Stochaste Optimal Conta bas an intinidating sound about i, yet the term ‘escrito a body of techniques that may well be the most brody wef ia fivof mathematics, The notion ise i simple enough deteeie the bast, strategy for controling # dynamie sytem fm the presence of uncertainty. ‘This objective 8 common to many disciplines, including engineering, science, economics, sociology, and poles. Where there is such con tonality of purpose, tb vitally certain that the most fundamental Privciples ace staighttorward, The complenty antes when the principles fre reduced to practice, fr then the reais of implementing logle, storing information, and transerring data have to be adieesed. The symbology rust be learned, and are mist be exercised in stating each problem ad its Solution. The payot for understanding stochastic optimal control canbe a deep appreciation fr the unying preps of system dynamics and This book i intended to introduce stochastic optimal contol concepts for application to real problems, with suficienttheoretial background to isfy their use. As auch, the book spans the mill ground between pure ‘mathematics and application. Recognizing the diicalty in serving bot the beginner and the advanced reader, [have organized the book so that the flow of information ean be maintained for readers ia oh categories One objective to give the student with itl backyround in contol theory the tools (0 design practical contol sjstems and the confidence 10 tackle the ‘more advanced Ieratute in areas of particular interest A second is 19 provide ready reference for those who have atsined working knowledge St the subject, ‘Courset on optimal contol and estimation normally ate offered st the college senior or graduate level, although study could begin earlier. I is Important to understand something about the types of systems that may be subjected 10 coottel. Gathering such information takes tie, as wal 85 & Aogsoe of experiance and maturi). Nevertholss, the mathematics needed to get started i lly ba atthe level of second-year calelus. This bookprovides scent ineottory material 19 make it possible for college Juniors oF practicing engincers who may ave been away (com (orm lasses rcoently to study optimal control and estimation without sup plemental materials. It als includes bibliographic references t0 related publications to ai more detailed study. The fist chapter prevents qualitative iatroduetion to optinal conto: at fas end, the reader hs two choices. Chapter 2 provides review of the ‘mathematics of control and estimation, arranged s0 that each section Corresponds to a silly numbered chapter. (This rents in a minor problem of causality and slt-eference, as Section 21 provides background for Chapter I, and Section 22 provides background for Chapter 21) The Advanced reader might skip Chapter 2 entry, proceeding dzcely 10 (Chapter 3. However asthe ned arises, the propriate section of Chapt 2 is available for reference. Chapter 3 relies only on mathematics appeatng ‘through Section 23, and soon The remaining chapters Weat four topics. Chapter 3 addresses optinal control of systems that may he nonlinear and time-varying, but whose Inputs and parameters are known without error I ilutates how open-loop control polices generalize to clesedtoop control laws when stem “dynamice ave linear and the covt funtion ix quadratic, Chapter presents ‘lthods for estimating the dynamic sates of stem that i deven by Uncertain fores and observed with random measurement ertor. The ‘ultimate objective fo we sich estimates to hep control & system (vie feedback); hence the development stops shot of aterthe-fact optimal «estimation (ptinal smoothing), which has ite bearing on optimal contr For the moat pt i is asumed that random variables and processes have Gaussian. probability distributions, which ace tilly desctibed by their ‘means and covariances (or spectral denis) Italo avoids the more ‘complex theoretical sues of optimal estimation for highiy nonbinear sy ‘ems, which ony rarely can e put into practic with cuteat computational took. ‘Chapter $ discuss the general problem of stochastic optimal eontrel, where optimal canto! depends on optimal estimation of feedback in. formation. Conditions under which contol and estimation policies san be ‘erived separately are reviewed. In the linear-quadratic-Gaussian (LOG) ‘eas, the separation i seen to be certainty-equivalent the optimal contol policy i the same as it would be lathe detersnisie case, except that the ate estimate rather than the state isl i fed back, Chapter 6 focuses on linear, time-invariant systems, for which multivariable controllers can be ‘based on linear quadratic contol Is with inear-Gausian estimators. The emarkable bveadth and power of stochastic optimal contal becomes apparent, intrpreted by potions drawn frm casial control applied to ‘ltinputlmalt-ovtpat stems. Although many intresting developments inconial system analysis have been made recently, optimality emai the Pree Fir en ‘most important unifying criterion for control system symheis. The final ‘chapter presen bref comment onthe vale of stochartc optimal contol There is enough material in the book t0 support two-term couse sequence, and then some. At an advanced level, Chapters Te 3 4 5, and 6 ‘ould be presented inorder. At an introductory level it would be reas fable to intersperse sections or parts of sections fom Chapter 2. sample ‘one-term sequence would be Chapter 1, Section 2.1, Sections 22 and 2 Chapter 3, Section 24, and selected pars of Chapter 4. The second term ‘hen could be based onthe remainder of Chapter 4, Chapter 8, ann! Chapter 6; wi reference to Sections 25 an 26 a needed ‘The Book's examples nd problems are directed st confidence buiing. 1 would prefer that most readers be able to do most protiems rater sh Fesetving that secomplishment for oaly 4 few. One outcome Is thet many problems are meant to be solved with the ad of» personal computer. The Programming i It to the reader, who should Re! that even passing Acquaintance with an iatodgetory programming language like BASIC sleet forthe tasks at hand. The priaepal challenge i o understand the theory and algorithms of the book well enowgh to reduce them to practic, Capabilities gained should be readily transferable to more comples prob Jems requiring. more sophisticated computation. This iting, ey im plementation of stochaste optimal contol insist situations i entirely Sependent on the we of computers. ‘Writing this book has been a profound leasing experience forme, an hope that The Been able to convey a sense of my own excitement in completing this work Iam deeply indcbted to te indvkals than ean acknowledge here, most notably the many researchers whose work fas bee ny guide To the extent that this book interprets the ideat of othe, Task their forbearance it T have failed t capture the scope of thet intent Co-workers and studens provided an unceasing steam of night, inclad- ing all-important “mideourse guidance” on many occasions, T also am grateful 10 the supervinor, technic motors, and revicners who have helped my carer evolve; their support snd confidence was uci Several individuals have provided invaluable ssstnce nthe pre paraton of the book When T was formulating Meas for the book, ds ussons with Theodore Eadetbaum and Jason Speyer then at the Charles Stark Draper Laboratory; Michael Athass of MIT"; nd Charles Price, Bard Crawford, John Trousatd, Paul erry, Fred Maeve, and Michiel Safonov, then at The Analytic Seizes Corporation were most helpful ‘Others have provided suggestions and rts onthe tet isl. inehoing Profesor Bradley Dickinson and rudeats Mark Paki, Prakash Shrvaye fava, Stephen Lane, David Handelman, and Chien Muang of Princton University; Muthathamby Sti Jayanthe of TAM; and Robert Otto af the “Monsanto Corporation. While any fate in the book are wally my ow, 1 wish te share any compliments thatthe book may garner with these peo,‘And finally, warmest thanks go to my chief cheerleaders and cheer pro- ‘iets, Pegi, Brooke, and Chiistopher, whose continued understanding, Encouragement, and sense of hamor have been wuly optimum, Romer F, Srascr fea | CONTENTS 1. INTRopuCTION 1 LLL Framework for Optimal Control 1 1.2 Modeling Dynamic Systems. 5 13 Optimal Control Objectives 9 LA. Overviowof the Book 16 Problems 17 | References 8 MATHEMATICS OF CONTROL AND ESTIMATION 19 24. Seals, Vectors and Matrices 19 Scala 19 Vectors 20 Matrices 23 Inner and Outer Products 25 | Vector Leng, Norms and Weighted Norms 28, Stationary, Minimum, and Msimamy Pointe of « Salar ‘Variable (Ordinary Maxim an Minima) "2 Constrained Minima sod Lagrange Muliplers 36 2.2. Matrix Properties and Operations 4 Inverse Vector Relationship 41 Matrix Determinant 43 ‘Adjoiat Matai 45 Mates Inverse 45 Generated Inverses 49 Transformations 34 | Difleretiaton and Integration 59 i Some Mae Kdentties 60 Egenvales and Eigenvectors 62 Singular Value Decomposition 67 Some Determinant Identities "68PY 2s 26 Dynan System Medels and Solutions 69 [Noalinear System Equations 69 Toca Linearization | 74 Nemesia! Inceration of Nonlinear Equations 77 ‘Numeral Integration of Linear Equations 79 [Representation of Data 84 ‘Random Variables, Sequences, and Proceses 96 Scalar Random Variables 86 Groups of Random Variables. 90 Scalar Ranomn Sequencer and Processes 95 CCortlation and Covariance Functions 100, Fourier Serie and Tepes 106 Special Densty Functions of Random Processes 108 Spectral Functions of Random Sequences. 112 Maltvariate States 115 Properties of Dynamic Systems 119 Sta and Quasistatic Equilibrium | 121 Stability 126 ‘Moves of Motion for Linear, im Reachability, Control, and Stabiizabilty 139 Constrctabitty, Obsrvabls, ant Detectability 142 Discrete-Time Systems 144 Frequency Domain Modeling and Analysls 181 Root Locus. 155 Frequency. Response Function ad Bode Plot 159 Nygst Plot and Stability Criterion 165, fects of Sampling 168 Problems 171 Reterences 180 OPTIMAL TRAJECTORIES AND [NEIGHBORING-OPTIMAL SOLUTIONS 194 aa 2 3 Mt Statement ofthe Problem 185 ‘Cost Functions 188 Paranictrie Optimization 192 Conditions for Optimality 201 ‘evessry Conditions for Optimality 202 Suficrent Condition for Optimality) 213, ‘The Minimem Principle. 216 “The Hamilto-Sacab-Beliman Equation 218 one 35. Constraints and Singular Control 222 “Terminal State Equality Constraints 223 Equality Constraints on the State and Control 231 Inoqually Constraints onthe State and Control 287 Singur Coatol 247 36 Numedieal Optimization 254 Penalty Function Method 256 Dynamic Programming 257 [Neighboring Extremal Method 257 ‘Qusslincarzation Method 258 Grndiont Methods” 259 37. Nedghboring-Optimal Sotations 270 Continuous Neighboring-Optimat Control 271 Dynamic Programming Solution for Continuous Tineat-Quadetic Cont 274 Discrete Neighboring-Optinal Control 276 Dynamic Programming Solution for Discrete ‘ineat-Ouatratie Contol 281 Small Disturbances and Parameter Varitions 283 Problems 284 Reterences 298 OPTIMAL STATE ESTIMATION 41 Lenst-Squares Estimates of Constant Vectors 301 Leas-Squares Estimator 301 Weighted Least Squares Estimator 308 [Recursive Least Squares Estimator 312 42 Propagation of the Sate Kstinae and is Uncertainty 315, Discrete-Time Systems 318 Sampled-Data Representation of Coninuous-Tine ‘Systems 326 CContinuous-Time Systems 335 Simulating Cros Correlated White Noe 937 43. Discrete-Time Optinal Filters and Predictors 340 Kalenan Fitter 342 Lineat-Optimal Predictor 352 ‘Alterative Porm ofthe Linesr-Optinal Fiker 35244 Correlated Disturbance Inputs and Measurement Noise 361 (Cros-Coraton of Disturbance Inpat and Measurement ‘Noe 361 “ime-Correlated Measurement Noise 364 4.3. Contimmows-Time Optinal Fiters and Predictors 367 Kalman-Buey iter 367 Duality 372 Linear-Optimal Predictor 373, Alterative Form ofthe Lieat-Optinal Fiter 374 Coneltion in Disturbance Inputs and Measurement Noite 379 46 Optimal Nonlinear Estimation 382 ‘eighborng-Optimal Linear Estimator 385 Extended Kelman Bucy Filter 386 Quasiliear Fier 388 47 Adaptive Ftering 392 Parameter Adaptive tering 393 NNose- Adaptive Fitering 400 Maliple-Model Estimation 402 Problems 407 References 16 STOCHASTIC OPTIMAL CONTROL 0 ‘54 Noullene Systems with Random Inputs and Pesfect, Measurements 21 Stochatle Piaciple of Optimality for Nonlinear Systems 422 Stocharte Principle of Optimality fr Linear-Quadeatic Probleme 423 Neighbering-Optinal Control 426 Evaluation ofthe Vasitinal Cost Function 430 ‘52 Nonlieae Systems with Random Inputs and Imperfect Measerements 432 Stochartic Principle of Optimality 432 Daal Control 436 Neighboring- Optimal Control 443 5:3 The Certainty -Xquivalence Property of inear-Ouadratie-Gaussian Controllers 451 “The Continuous-Time Case 431 6. LINEAR MULTIVARIAB! ‘The Discrete-Time Case 455 Additional Cases Exhibiting Certainty Equivalence 459 4 Linea, Tlne-lavariant Systems with Rendon Ipets and Imperict Measurements 460 ‘Asymptotic Subilty ofthe Linear-Quadeatie Regulator 461 Asymptotic Stabity ofthe Kalman-Bucy Fier 74 ‘Asymptotic Stblity ofthe Stochare Regulator 480 Steady-State Performance ofthe Stochastic Regulator 483, The Dierete-Time Case 488 Probleme 489) References 493 conTROL 96 6.1. Solution ofthe Algebeae Riceatl Equation 497 ‘The Continuous-Time Case 497 ‘The Discete-Tine Cate 505 62. Steady-State Response to Commands 508 Open-Loop Equilibrium S10 Now-Zero'Set- Point Regulation S14 63. Cost Functions and Controle Strctores S17 Specication of Cost Function Weighting Matrices 518 Output Weiahting in the Cost Function 319 Implicit Model Following 520 Dynamic Compensation Through Sate Augmentation S22 Proportona-iaesral Compensation 822 tegral-Fite Compensation 528 it Model Flowing. 531 ‘Transformed and Partitioned Solutions $35 ‘State Estimation and the LOG Regulator | 539 64 Modal Properties of Optinal Comtrol Systems S44 ‘igenvaues of Optimally Regulated Systems $42 igenvectors of Linarly Repuated Systems 558 Eigenvectors of Optimally Regulated Systems 557 Eigenvalues and Eigenvectors of Opti Estimators 563 ‘Mol Properties forthe Stochaste-Optinal Regulator S64 Eigenvalues for the Diterete‘Time Linsar- Quadratic Regulator 56765. Robestnes of Linear Quadratic Regulators S71 Spectral Characersies of Optimally Regulated Sysiems 572 ‘Suatity Margins of Sealae Linear-Quadratic Regulators 576 Effects of Sytem Variations om Stability 584 Multivariable Nyquist Slaity Ceitron 589 Matrix Norms and Singular Value Analysis $91 Staiity Margins of Malivariable Lineat-Quadatic Regulators 595 ‘The Discrete-Time Case 599 6.6 Robustness of Stochstc-Optaal Reyulators 602 CCompensitor Stability 603 Stability Margins of LOG Regulators 60S Robustness Recovery 608 Probably of Instability 611 464 Footnote om Adaptive Control 614 Problems 615 Reterences 622 EPILOGUE, or INDEX oo 1. INTRODUCTION Designing control fogie that commands & dynanie system to a desired ‘utut of that augments the system's stability is a common objective in thany technical elds. ranging fom decision making for econome and ‘octal systems though tajetory contol fr robots and vehicles fo “knowl. Siige-bae engineering” in the application of “artifical inteligence” Concepts If the control objective. can be expressed as a. quanblaive eran then opimization of this ctrion establishes a feasible desig Structure for the control logic. This book is about the theory and ap plication of optinal conto particular attention is given to providing. a Figotous yet staightorward foundation for mathematical optimization and uz how practice control system can bo designed. wih this LA FRAMEWORK FOR OPTIMAL CONTROL Optimal control theory describes the application of fowces to a system for the purpose of maximizing some measure of performance or minimizing & out tunetion, I the sfonmation which the contol system mist use ‘ncertain ori the dame system is freed by random csturbance, may ‘por he posible to optimize this eterion with eertainy; the best one can hope todo ito maninize or minimize the expected value ofthe eierion, tiven assumptions about the statics ofthe uncertain factors. Ths leads 60 the concept of “sfochasis™ optimal contol, or optimal contol which Fecopnzes the random behavior of the rstem and attempts to optimize espns or stably om the average rater than with assured precision ‘A stochastic contol sytem really performs two functions; i nt only comvols the dynamic system, estates the system's dyamic state in oder to provide the best “Teedbsck" information for closeé-loop conta. All ‘ont seis ae intended to optimize some exter, whether oF not the triteria ave sated explicit, and there some depree of uncertainty in any2 aration contol system implementation, Consequenty, stochastic optimal control theory has broad application to practical contol syslem syntheses demonstrated in the remainder ofthe book, A ‘By way of introduction, coasider afew examples of dynamic systems that ‘might be controlled in some optimal fashion (ig. 1-1). The figures tre conceptual, and the implied control may not be secomplished easly, - {1h | ‘Tencersutes, ae LL | EE FIGURE 141 Dyan poe con eb ctl (Aca at Shea poet so (esa cmc a : ed meno fr Optimal Coto a Nevertheless, it should be appareat that seemingly diverse systems are fubject to control and that fase solations to contro problems can be ‘Wentifed wring escntaly similar methodologies. “The notion of epimizing the contol policy canbe pursued in each ease, although the “figures of merit” (or performance esteria, cost functions, penalty functions, et) tha reflect successtul contol may be markedly Tliterent, (We wil gencraly rete oa of these as “cost functions” whetber ‘or ot monetary coe i ivolved. In this broad sense, cst implies penalty or ‘Value and it st be minimized oF maximized, as appropiate.) One control ‘hjecive for aircraft would be fo minimize fue us without any penalties for se of conto, and the fuel used defines 2 cst funtion. Another might be to obitin reltvefosensitvty to turbulence with limited motion of the Control surfaces; hence, the eos unetion would contin a weighted sum of ‘motion and con magaitudes, The two contol objectives lad to diferent Contol policies, yet the control tvetores can be developed within a ommoa framework: the optimisation ofa cst funtion. Similar texsoning an be applied to the other examples, im which the “cost” of contol ‘weighed against the “cont” of deviations im dynamic variables to formate tn “optimal” contol policy. Lets consider some cost functions that we might wish o maximize or minimize. These fll ino two categories: discrete objectives chat ae opti- nized “oace sn or all” and continuing (egultory) objectives that shuld toe iainained at opal valves daring the passage of time (including infinite time interval). athe fist eategery, we Bnd the following examples: “+ Minimize the time reguted to taser from one dynamic sate to sather. 1+ Minimize the eneray oF fuel used in ansering from one state to athe. {+ Minimize the eror in arsving atthe final tate {Maximize the return om investment atthe nd of fixed period of + Minimize the ssk in pursuing an ivestment policy, as measured at the end of the investment period Inthe second category, we could have the following 4+ Regulate system to remtin near a fixed nominal condition in ite of sintrances 4 Follow # nominal path with sinimum error, even though system parameters have uncertain vals. Respond to arbitrary changes inthe desired nominal coneiton with ‘nelLbchaved transient in sate and contol variables,4 Iarotecton 1+ Minimize inventories required to produce and deliver a provct in & changing marketplace, There isan important thtd category which wses optimization as an Jnermediat step, ih thatthe cos faneton el i only inecly slated 19 the design objectives. For example, in the design of practical servo. ‘mechanism, the numerical rade between state and congo perturbations ‘may be lest signifiant than the astral frequeney and damping, stability ‘margin, and so on. Weighting factors inthe ont function cane used ‘design parameters to achieve the desired real. The rationale for parsing this approec stat the feasible design produced by optimization generally tse avatlabie contol resources in an efcient manner, requiing, hte intuition or “prior ar wo start the design proces. Furthermore, cet uarantes regarding tbe stability f the controlled sytem can be made, These characterises ate essential, particulary inthe design of contol for systems with many dynamic states and contol variables or those which are ot inherently sable “The motions of dynamic systems evolve as time pastes; hence the erteria to be optimized are expressed naturally in terme of final vale. and timeintegrals ofthe motion and contol variables (i the ime domain) Nevertheless, slaty, response to sinusoidal inputs of varying Irequencies, and sensitivity of response to sytem parameter variation are ot steal practical concern, s0 equivalent Pequeney domain enter ae uta: The Dower of time domain techniques for contol synthesis related to their Eeneraliy; optimal controls for ‘nonlinear, time-varying, systems. at specied (if ot calulated) teadiy, and the simpliiatons forded by arity and time-invaviance aye accommodated in the same thorctic framework. When both linearity and time-invavianee can be sumed, frequency domain conceps aid interpretation and, in some cise cnt symtesis. This book presents stochastic optimal control from & time domsin Viewpoint, providing Trequeney domain’ perspectives whet helpful and snproprite ‘Thete is a natural relationship berween stochastic optimal contol and ‘many mathemacical concept offen collected under the rubric of "artical intligence” (AD. AL algorthns are applicd 0 interpuetiton of data System monitoring and identification, prediction, planning, and design, They may use “prodoetion systems” to model dynamic interactions pred «ated on large data bates, aswell as optimization technigue for decision making under uncertain. Stochastic optimal control theory provides tools {or solving many of these AL problems, ‘What fellows cannot include all’ methodologies for all optimization problems; the book focwes on just those aspects that hive demonstrated practical applications or sow great promise to do 3, Interpretations of the ‘coretcal tnderpinsings of optimization technique, swell as detailed examples of ther se, ee presented Moding Byam Stems s 1.2 MODELING DYNAMIC SYSTEMS Stochastic optimal contol can be appliod to dynamic sytem that consists ‘physica, chen, esonomic, or other dynamic process and it oer Saon, tn order t design the conto logic, a mathematial model of the Uyamic sytem mat be defined. Ie shouldbe understood thatthe mel i, tent e tvogate for the actual system, whose precision i subject to the ‘Ssumions and approximations made by the contol designe "Elements ofthe mathemati! model ean be ateanged in eistne fies cot variables that ken their rexpective roles in the system. Each family Consists of st of scalar quantities andi represented by a single symbol ‘Once the ordering of quantisentalished and xed), tbe symbol ean be interpreted as «column ect Ge Section 2.1) that is synonymous wit the tie set of quantities Symbol (or Veetoe) Dimension > Parameters ' ° Contrib inputs ™ ” ncontelid inputs : (Disturbances) x Dynamic states ” Y Process outputs r . Measurement reoss , 2 Observations , ‘he tlationships between these variables are iustated by Fig. 12-1 “These seven categories of variables Serve distinct purposes in the dynamic Stem, 'The vector of parameter p scales the process response to inputs find its own motions, either in the dynamie proces, the observation, or ren we MGURE TS Benoni tem,“ Introtecton both, The forees on the process (tbe impus) that can be controlled are ‘contained inw, while thse that are beyond contol are contained in w. The Sai vector a sepresents the dynamic condition ofthe process, including fundamental response to ipets an intial conditions, The proces structure insu that x "Yeeds back” into the system through natural dynamic paths; this feedback can moify process response in several way (by sifting the steady-state characterises, causing oscllaions, stailaing or de= stalizng the proces) or examples of specific definitions of parameter, control, disturbance, and state vectors, consider 3 chemical rentor (Fig, 11-1) desied 10 ‘reduce ethylene (W-1), The parameters of the proces, p, ince the ‘Physical characteristics of te eactor as well s product market values and feparation costs. The conte, inchide hydrocarbon. fed rate, Steamhydrocarbon ratio in the ead sream, reactor skin temperature, and feactor outlet presure, Datrtanees, w, might ince variations i the ‘chemical composition of the feedstock and local (uacontelabe) t= perature and pressure variations slong the walls or within the reactor. The Sate components, inelude the carbon deposiion rates at hore pols long the feacor tbe ands measure ofthe ftal nt earnings accumulated from the start ofthe production interval The output vector, y, can contain none o all ofthe preceding familiss, selected components of each vector, or eanslormatons of these vectors, ‘depending on the process and is itsramentaion. In genera the output cannot be measured exactly, so the obscrvation, some combination of the output, y, and measurement error, m. Any dynamic elects associated Wwith'conteolsetution or measurement sensing can be incladed in the plete contoliabity may be lost ’A system is obseroable att ifthe ouput history yy == 4 Is aquate to reconsrt (4) there an independent path between every ompaneat of x and a Teast one clement of, then the process is ky 10 te completely obuervable. Agun, the path may be director indirect, bat ‘ust be ponienteal to al other torn paths i the dynamic system the Single of the clbow of the manipulator arm shown in Fig. Ll-le is ‘ensured the poston of tol atthe end of the arm ean be determined if the angle sensor fas, observabiity may become incomplete These definitions say noting about the potential quslty of conto or ‘observation; they merely indicate whether the natural structures for control fn observation exit Mathematial conditions that mst be satisied for Complete controllability and observabiity can be found in Chapter 2. I thos! eases, the dysamie proces only patallycontolable and otsery~ Ui, either a a consequence of the dynamic process itself or Becauselimited resources prevent implementation of complete contol and obser- ‘ton. Such a process can bo separated into four parts: those subprocesses {ha ate both completely contolable and abeervable, hose thal are one but ‘ot the other, and thse that are neither From 4 pratcal pont of view, complete contolability and observabilty are not required i the uncontoled or unobserved stats are well-behaved. For example, adequate rgit-body contr of a vehile ned not depen o ‘controlling or abserving well-dampod,high-requency elastic modes. I tincontrlld subprocest in an otherwise contolable proces i tale (e, ‘bounded inputs o intial conditions produce bounded Fespons), the process in uid to be sablizele, snd the subprocess's effects cannot cabse the process to diverge. fan nobrerved tubprocess i sable, the otherwise Shservsble process detectable, ax hounds on the observed slates can be ‘extmated, Part conollabity and observabilty may be acceptable, but ‘stable subprocesses that are neither controllable nor observable must be voided beotuse closed-loop (presumably stabilizing) cone ean be applied ‘nly to those elements chat posess both characterstis, More wil be si aout the methods for determining stayin Chapter 2. (Dynamic proceses normaly are continuous funetions of time, so itis spproprste to model the evolution of their motions by iflerental equa tons, These processes are called continour-fine systems, The solution ‘arabes for these equations are contained in the state vector, x0). The Tuwiamental characteristic of continuous-time dynanie systems is tht the rato of change ofa) with respect to time, dad, a function of the "ystems sat, input, and partmeter vectors. Although some processes ae teat described by para dflerenial equations (containing derivatives ith respect to more thin one independent variable), we wil Festict our View © ndnaryillerental equations (containing derivatives with respect to 8 Single independent variable, eg tie), noting thet numecal ape proximations tothe former ca be achieved withthe ater, Figure I. I-Lare provides examples of continous processes Te an increasing numberof applications using digital computers, control settings are ealeated and measurements are made at discrete (olen Period) instants of time. Dilerence equations that are equivalent 10 the ‘orginal diferent equations can be found, and continuoustime stochastic ‘ptimal control solutions are parallted by discrete-time reals, Process that ar rigorously modced by iference equations are called dierte-tme items, Economestie models (.f Fig. -1-1d), soeiologieal modes, and biological model that se dierete measurements made at periodic interval frequently originate as derete-time modsie Consequently,» thorough ‘exposition of dsrete-tine methods can serve x variety of applications. or either coninoowetime or discrete-time modes, the dynamic equa- tions can be clasifed at in Tig. 12-3. It the dynamic coufciens, or parameters, p, ae changiag rapidly in time, in comparison with the time Seale" of mations, the dynamic model must be lime-oaryng. IC the _ Opti Cane Objective ° coecints are relatively constant, a time-invariant model wil suf. If Imotions evidence the superposition characters (.ey doubling the input oF inital conditions doubles the. corteponding response), then linear ods can be wef no, the model must be nonrea ‘Consider the dynimic response of a weathervane to changes in wind ‘iecton and velocity. The wind-ndced torque on the weathervane 5 propostiona othe square of he wind velocity and o the sine ofthe angle @ {etween the wind direction an the inlestor’s angle. For smal, sin) = @ ‘and sin(28)~20, a0 the torque is approximated bya Tine function ofthe fngle. Ifthe wind speed i constant but it diection changes slightly, peturbations in the weathervane's orientation can be described by a ieat, fimesinvarant ferential equation. I the wind sped it ehanging. 38 wel, the linea ferential equation mst contain time-varying cefcient I the Wind undergoes large, rap change in wind direction, becomes large, $in(@)# @, and the weathervane reponwe ean be porteayed accurately only ty nonlinear differential equation 13 OPTIMAL CONTROL OBIECTIVES, 1 wil be helpful to preface the mathematical developments of the book wwith few pictures that strate some ground rues and goals for optimiea- on, While these figures may appear intuitively obvious, they seve as Feminders tht straightforward objectives are central to optimization. I the ‘notation i unfair, it maybe heipfl to skim Section 21 as you go. nthe simplest case, salar cost function J, of scalar variable wis iven, We would lke to find that value of u which caus J, 10 be ‘ina, tha the poi u* showa in Fig. 3-La. As an alternative, we ‘ay wish to id the value u* which masimiacs J, a5 in Fig, 13-1b. The10 tnoacon FIGURE 1344 Fino jet 1 pint (Se actin wth inna) Sigures have boen purposely drawn so Jy =~ Jy; hence the u* hat imines J ako maximizer Js It always it possible to change the search for ‘maximum to the search fora minimum (or vce versa) by changing the sg ff the cost Tunetion. The value of u" which minimizes (maximizes) the function (J) is defined by the same characteristic in both cases. What Is ‘A? What character of the functions distinguishes the maxim fom the "The cost function can establish 9 trade-off between two oF more vari bles. Suppose J isthe sum of two functions, x2) ad J(u), a8 shown in Fig, 12a, whare the sate, sis function of the contol u [te x= x(u)) Then w° enables some “est” combination of J, and J (nd therefore, ‘of rand 1). The optimum point depends on the relative imporianee of J, land JI we make the cost of contol only half as important as before (Fig. 113.20, the optimum point shifts nthe ditection of more contol w achieve Opti Cnr Ojon 7 — FIGURE 13:2 Deaton of ct eon ht combine te eft oo vac (3) at fnce ara eles oe tte span (8) ea ee Sol soon opr lower tate-related cost. ‘The effect on u* would bo the same as that of increasing te snt-oated cost, although the numerial valve of would be diferent, 11 J i scalar function of not one but two independent contol ty ad tit vale i deseribed bya surlace eather than a curve. A minimum valve ‘of found atthe bottom ofa bow shaped contour sin Fig. 13-34; the ‘ninimam’sloeation defined by concurenty optimal valus of the cone {Wos, wT and u8. If one cont has the proper valve but the other does nt, 3 does not take is ninimum valve Contours of equal cos ea be found by making horizontal slices through J's snfae, These can be plot inthe contel "space" (Fg. 13-30), and the optimum control point is defined by the mide of the “thumbprint. [Note tha if we decide to find the optimum conto! point by some te proces, binning wih sonoptimam values of wand ws, e could reseh,FIGURE 13:3 Cos sons wih more ho inl vr (2) Cont een wth Irn ir: nia oat ee ws oa the optimum by slays travsting in dcection that i perpendicular 10 the loca! comstanteost contour. (One additonal piece” of information is required, What ii) ‘Showid there be three or more contol variables, the cost surface becomes hypersurface, he, it no longer can be ploted in dee dimen- fons), However, cqal-cost contours (or the thee-coneol case can be ‘isuaized a surtaces that close aroun the point (wf, uS) at which J is pti Cnt Once 13 maximum oF minimum (Fig. 13-36). Efcient paths to the optimum poi ‘would be perpendicular to those sutfaces. Wi four oF mere contol, the ‘aunl-cout contours themselves become hypersrlaces, Try well be thatthe path Yo the optimum is constrained to ie along or to one tide of a given curve. The former case establishes an equality ‘Conca, (te) =0, an equation relating an ue that mst he ated tthe same time that J mained As shown in Fig, 13-80, the smallest towable value of J the point at which the constraining curve is tangent to the comstant- contour of lowest cost. Tis not an absolute minor, but itis the best that can be done ues the eicumstances. An inequality con sain’ fey fli} =0 oF flay.) =O} separates the con space into ‘cceptable and unacceptable regions. Ifthe acceptable region contains the solute minimum (Fg. 13-40), that of course, the solution. Tit does rot (Fp. 13-40), shen the equaity-consrant minimum isthe answer. "The concep ilastrated nthe last several Rigutes sty apply to the ‘optimization of static (unchanging) systems; however, we can extend these ¥eas o the optimal control of dynamic systems (at Teast figuratively) by ‘making afew changes. Perhaps the most tanficant of these relates 0 the ‘suality contain. Using the vector notation described in Section 2.1, the previous example of sate optimization considered an equality constraint fon the conto hat took the form fo)=0 aay Hind there heen more than one constant, vectr of eqs coud have bbe writen a6 fa=0 432) Optimization of « dynamic system governed by an ordinary diferent equation forthe sate vector, x0, 6 = 4, a. 0.9041 assy is constrained osatisy this equation during the solution interval <= ‘The constraint could be rewrite ts fIa(9, 0, (0, mi, K=O 039) Ineffect, the equaly constant of Fig 1 3-4a ean be multidimensional and contiualy changing in time. Furthermore, sti fusction of pM, W(0- (0, fand X(0) a8 well a the contra, w)- Consequently, +The overall cost function fora dynamic system can only be eaeuated6 Iatretaton Te wil be seen that the dependence ofthe instantaneous optinal control on past history & represented ently through is dependence on the current ‘Our ability to constret plot such a8 Fig. 13-4a from rea data may be tinited by ertor in measurement, uncertainty a the stactre of mathema> tial models, and imprecision in the deintion of parameters. Consequertly, the observed costs and constraints might look ike Fig. 13-Sa or be based From their actual poston. In sich circumstances, it may be desirable 10 forego deterministeoptnuzation and we a stastical approach, which kes {nao avsount ou best knowledge of expected vals an Hkely deviation in scribing the stem (Fig. 13-58). Av inferred feom the Agure, having ‘binned estimates of the expected values of the state and the resulting ‘onttol it pole to use thse sockasc variables in the optiization. ‘Ohservations ofthe sytem can be expressed as 4 vector Fusion, 20) of the matem output an ero, 0 = Hy. A) ass) where the output vector yi is a fnetion ofthe state, control disturbance, fd parameter vector of the system: Hx, (00.01) 0136) wt “The objective isto obtain an estimate of x() fom 20), one that miniizes the degrading effects of lt) and of uncertainties in pi) TAs opts tstimate of can then be incorporated in the formulation ofan optima ‘control strategy nthe mos! genera stochastic optima problem, estimation and contra are coupled Ge, one depends on the othe) however, fora wide clas of problem, the estimation and control phases canbe treated separately. The proces of obtaining optimal estimates parallels the computation of optimal Controls; the processes are said toe mathematical “das” in that smile ‘equations and sompulational sequeness see wed in both cases 14 OVERVIEW OF THE BOOK “The first five chapters of the ook deat with the general problem of ontaling and obmerving sontiner, time-varying systems, and the sith Chapter concentrates attention on near, time-variant systems with con Stantcoctbeient estimation ond control, Chapter 2_ presents a general Inathcmatcal backround, wih cach section supporting the numerically ‘suivant chapter (Section 23 for Chapter 3, ete) For the most pati ‘ot necessary 40 complete all of Chapter 2 before proceeding to the other Prolene ” chapters: The theory of deine nonlinear opin contol is tiven in Chapter 5. Neighorngopinal tajeconte areola prongs for fecack soma an the poste need fr etnaion rote. Opialitering_andprediten for-daccte-tine “pte ae [resend in Copter wth cotinao ine etn Senelope te Tring cae -tor's vanishing smal ine itera’ Soshats aed Cone is frmltd the combined pocewe ef ernnioe ind eta (Chapir New the spec cae of sontling an vertag Ines tnesnvarian physical spon wih conan osicen snl a {stmstos arese, The din of nar mars conten eg Spnal cote tory saved in Chapter 6 & fnlcommmet here in Chance graphical totes throwghowt the Sook ar drawn pnipaly fom (CIGD, CG, He) ey and et) Gee Reterece PROBLEMS 1. Dein the vectors of dynamic proces mode [x py. m2) ae cost fanetion [1] for optimintion i your oe ek ores Co the arses ofthis mel change comic oo they change ey at Aiscrte stants tine? I he proces Coins, bow fst wold ‘iscrete mesurements have tcc fo capture te impetnt deta at time vars Inthe tem ner? None? Time varying? Consider the group of five penis shown inthe dgram. Each Pendlum escates withthe same stra equonc. Ts mone ‘ented by two sate variable components gl a ang a, tence the tl state vector has dimoron 10 Fath pend ou affected by a contol fore, but it deed to povie weno! vector ‘minimum dimeasion. What i the satest dncrson contol vector that provides complete contol? Would these be iret ‘he pendulums had dierent satrl equenies? %- sae ran i ya 3 The pe REFERENCES, SSN ath Cts er No ei, amc en 8 eS hn a cat es re mH ota SS om dt rm tl ce ohne 2 2 THE MATHEMATICS OF CONTROL AND ESTIMATION Optimization is accomplished by using equations that model & dynamic system and represent its conta strategy. The langunge of matiemetie ‘tended to capture broad principles in concive forms ad wile thi ay lead to a certain elegance and efficiency in expression, the notation ‘conventions can appear, at fst bit baling. Optinization i a codiieaton of common sense, and mathematics provides the to for stating an wing that cade, “This chapter presents a review ofthe mathematics of dynamic stems, paticularly a relates to optimal estination and contol ofthese ser Ieannot test every tope i depth, andthe advanced staden may proms through this review quickly "The chapter provides background for ander standing the remainder of the book, and it suggests, sources for further reading inthe mathematical Mtratre, Its parallel supplement ihe than an antecedent tothe other chapters, “The mathematical concepts of special concern inciwe multivariable ‘calculus, linea algetrs, numerical methods, probability and statistics, an teansform analysis. These topes normally are covered in secon nd ‘third-year engineering mathematics courses; hence a background i toee ‘vanced mathematics isnot esental. Nevertheless thee meets may have been sadied fom dieing viewpoints or with similar notation, xt particular sitention is devoted to establishing unfed framework for pplication to contol problems 21 SCALARS, VECTORS, AND MATRICES Seals A real, scalar constant is single quant that does not vary ding & ‘specie problem’s solution interval, The constant may be represented by a0 The Mathai of Catal symbol (ea Eagish or Grek leter)tha is synonymous with a specie ‘Meader (as tne Icom complex numbers, which arc discussed im sebien3, eased in appropriate units. For example, the dimensionless, asc selon constants cand, represent 2.71828. and 3.14159... eeecg: without ambiguity. © gener sybol ca be asigned a ed Tee that pertains to specie problem bat may change from one ease to The next, the mass of an object maybe signed by m i 10 case, mero in the fs cae and 10g ithe second. In either ease, ‘mnsonal values must be consistent with usage in the remainder ofthe lastem, aerate specifications, sch as m= 72.205 Ib-mass or 22.05 b- ease ay be equivalent but aot appropiate if Syme Inemational (SI) tite are ted in related compatatons it he single quantity. docs_vary in space or time or as the rest of changes ioe quantities, tf scalar variable. The quantity (or quan Gch om wich the varable depends can be stated expletly in braekets. THe mews of thrstng rocket vehile decreases with ime # and can be dietoted by mtd. The magnitude of gravitational acceleration g depends on {he stance 7 fom the earth's ceoter, ad it can be expressed as gl”) ‘Vocab can be functions of ther variables; the radios increases as the eat teaver cath, so. 7-1) andthe gravitational acceleration is (0. Ttis boob, scalar constants and varables pormally ate symbolized by TDnercase ric: Exceptions ate mide to agree with widely wed symbols eae cout function J). Explicit expression of variable’ dependence on ‘ice variables me oaly when necessary fo lan: Tr some ese, it 5 mast approprate to consider the variable © be @ function of 8 afcrete sampling Indes, wbich wall i represented by & ‘Misc Ifthe varying mass. measured at TL times with samphing [nora of ts, the sequence of measurements could be cepresented 38 ep to 10. On the other han, te continuous description of mass inthe Sane inerval em), = #10, Vectors ‘A set of seals may have some common tit, and if these scalars aways eee Sheu in ue rletioship to eachother, the ordered et an be teated as Sisal muledimenional quantity. Such quantity called a vector and it fray contin as many of few componens (or elements) a are deemed Tkasary to characterize the st, sealar Hike the volume of water in 8 ‘ines cua be considered a vestor with one component, The foation of & Sinaia of Bees ean be considered «vector with thousands of components, tives foreach bee, Both vector defstions ae valid, yt each is appropriate Linder differen eveumstances, Consequerty, a vector can be represented Oya angle symbol (sual boldface, lowercase Teter ee), but this TZoamsetaton mst be aecomparied By one more piece of information: the dimension ofthe vector. ee fete, Ysa Mai Pa Wn we ako an aed st we mein hat the cnpone are writen one ster he her i ned over ‘The eatin of cation of sgl bos tie component vcr, tw Caste cordate [9 2} tic seen tr hy an vr Spm ee frente owt the duplacements in aed od he he Bee's cao ‘can be expressed as either Pe (2) any reta.mnd aay ‘The subscript salar vals are equivalent othe ina avivlent the in ordered set, tothe dimension n oft vest quite ovo the: Once sng of ‘tives determin Yr each components avec a dnetson nea be Comidered to erslsh» spc of cmon much space lat ‘tua form rater tan 3, wd cordnstes may ot have ene ‘ni; eet egal eptaton an wl lett comerton for express vector st op the commas ‘and arrange the vector's components in a column: a es) ‘This called » column sect, nd ss thervise note, And, ules otherwise noted itis the mpc Sti ft Yes Bk We thst ce we he components ina tow (without te comma) oming ww econ The son of colon eer a ow vc lan eters and the eh inten wit the ope 7 ob y no nl ars ‘The deft of vesorcntants and vate sorespnd 1 the of clase cnt Sms campo de he in which every element is one athe nll bet, in which ever clement is vero. Vecorsanables change as other quantiss change. The poston ofthe rocket vehicle leaving Eat, taking the Earths center rin, i fx] [no] x] |r xo ers) La) [na