Professional Documents
Culture Documents
Hamilton-Jacobi-Bellman Equations
Benjamin Moll
2
Hamilton-Jacobi-Bellman Equation: Some History
v (k0 ) = max e t u(c(t))dt
{c(t)}t0 0
subject to
k (t) = F (k(t)) k(t) c(t)
for t 0, k(0) = k0 given.
r (x, ) = u()
f (x, ) = F (x) x
5
Generic HJB Equation
In the case with more than one state variable N > 1, v (x) RN is
the gradient of the value function.
6
Example: Neoclassical Growth Model
cookbook implies:
u (c) = v (k)
7
Derivation from Discrete-time Bellman
discount factor
() = e
Note that lim0 () = 1 and lim () = 0
kt+ = (F (kt ) kt ct ) + kt
8
Derivation from Discrete-time Bellman
Take 0
v (kt ) = max u(ct ) + v (kt )kt
ct
9
Connection Between HJB Equation and Hamiltonian
Hamiltonian
H(x, , ) = r (x, ) + f (x, )
HJB equation
v (x) = max r (x, ) + v (x)f (x, )
A
Connection: (t) = v (x(t)), i.e. co-state = shadow value
11
Existence and Uniqueness of Solutions to (HJB)
Recall Hamilton-Jacobi-Bellman equation:
{ }
v (x) = max r (x, ) + v (x) f (x, ) (HJB)
A
Two key results, analogous to discrete time:
Theorem 1 (HJB) has a unique nice solution
Theorem 2 nice solution equals value function, i.e. solution to
sequence problem
Here: nice solution = viscosity solution
See supplement Viscosity Solutions for Dummies
http://www.princeton.edu/~moll/viscosity_slides.pdf
Theorems 1 and 2 hold for both ODE and PDE cases, i.e. also with
multiple state variables...
... also hold if value function has kinks (e.g. from non-convexities)
Remark re Thm 1: in typical application, only very weak boundary
conditions needed for uniqueness (s, boundedness assumption) 12
Numerical Solution of HJB Equations
13
Finite Difference Methods
See http://www.princeton.edu/~moll/HACTproject.htm
Functional forms
c 1
u(c) = , F (k) = k
1
Use nite difference method
14
Barles-Souganidis
Shorthand notation
vi = v (ki )
Need to approximate v (ki ).
Three different possibilities:
vi vi1
v (ki )
= vi,B backward difference
k
vi+1 vi
v (ki )
= vi,F forward difference
k
vi+1 vi1
v (ki )
= vi,C central difference
2k
16
Finite Difference Approximations to v (ki )
Forward
Backward
!# !#%$
! !#$
Central
#$ # #%$ "
17
Finite Difference Approximation
FD approximation to HJB is
vi = u(ci ) + vi [F (ki ) ki ci ] ()
vi+1 vi + vi vi1
vi = u(ci ) + si,F + si,B
k k
Notation: for any x , x + = max{x, 0} and x = min{x, 0}
v = u + Av
20
Visualization of A (output of spy(A) in Matlab)
10
20
30
40
50
60
70
0 10 20 30 40 50 60 70
nz = 136
21
The matrix A
vin+1 vin
+ vin = u(cin ) + (v n ) (ki )(F (ki ) ki cin ) ()
Algorithm: Guess vi0 , i = 1, ..., I and for n = 0, 1, 2, ... follow
1. Compute (v n ) (ki ) using FD approx. on previous slide.
2. Compute c n from cin = (u )1 [(v n ) (ki )]
3. Find vn+1 from ().
4. If vn+1 is close enough to vn : stop. Otherwise, go to step 1.
See http://www.princeton.edu/~moll/HACTproject/HJB_NGM.m
Important parameter: = step size, cannot be too large (CFL
condition).
Pretty inefcient: I need 5,990 iterations (though quite fast)
23
Efciency: Implicit Method
25
Non-Convexities
26
Non-Convexities
Consider growth model
v (k) = max u(c) + v (k)(F (k) k c).
c
But drop assumption that F is strictly concave. Instead: buttery
F (k) = max{FL (k), FH (k)},
FL (k) = AL k ,
FH (k) = AH ((k )+ ) , > 0, AH > AL
0.9
0.8
0.7
0.6
0.5
f (k)
0.4
0.3
0.2
0.1
0
0 1 2 3 4 5 6
k
27
#\]A=C] ]A=C]
K
] #A=C]
Standard Methods
TR] R6D4P] RF] Z6MF] V=R<] R=C6] &<6M68FM6]
&<TP]
R<6M6]6X=PRP]1]RM1?63RFMY]=D]M6;=FD] (]V<=3<]=P]8FTD4]26RV66D]R<6]3TMU6]!]]1D4]
1X=P]V<=3<]C66RP]R<6]D636PP1MY]FHR=C1A]3FD4=R=FDP]=;TM6]].6]31AA]=R]FIR=C1A]
Discrete time:]6H6D4=D;]FD]R<6]M6A1R=FD]26RV66D]R<6]HMF4T3R=FD]8TD3R=FD]1E4]R<6]
RM1?63RFMY] rst-order conditions
4=P3FTDR]813RFN ]V6]C1Y]<1U6 ]1P]=D]R<6]31P6]3FDP=46M64]61MA=6M ]R<1R]RM1?63RFMY]
u (F (k) k k ) = v (k )
TDRV=PRP]=D]R<6]U=3=D=RY]F8]R<6]:MPR]6LT=A=2M=TC]HF=DR]TDR=A]R<6]FHS=C1A]RM1?63RFMY]
M613<6P]Z6MF]31H=R1A ]FM]V6]C1Y]<1U6]R<1R]26;=DD=D;]V=R<]1DY]]V6]61R]1AA]
no longer sufcient,
6X=PR=D;]31H=R1A] typically multiple solutions
D]=;TM6]]PT3<]RM1?73RFM=6P]1M6]P<FVD]V=R<]R<6]4FRR65]A=D6P]/6]
C1Y]P<FV]R<1R]V<6D]FHR=C1A]RM1?63RFMY] ]<1P] R<6]8FMC]F8]1]PH=M1A ]R<6] 26RR6M]
some applications: sidestep with lotteries (Prescott-Townsend)
RM1?63RFMY]V=R<]R<6]P1C6]=D=R=1A]U1AT6]F8]31H=R1A]=P]R<6]RM1?63RGMY]V=R<]R<6]A6PP6M]
=D=R=1A]U1AT6]
28
Instead: Using Finite-Difference Scheme
Nothing changes, use same exact algorithm as for growth model with
concave production function
http://www.princeton.edu/~moll/HACTproject/HJB_NGM_skiba.m
0.1 -30
0.08
-40
0.06
0.04
-50
0.02
v(k)
s(k)
0 -60
-0.02
-70
-0.04
-0.06
-80
-0.08
-0.1 -90
1 2 3 4 5 1 2 3 4 5
k k
29
Visualization of A (output of spy(A) in Matlab)
10
20
30
40
50
60
70
80
0 10 20 30 40 50 60 70 80
nz = 154
30
Appendix
31
Why this works? Barles-Souganidis
Growth model
as k 0 and ki k .
33
Why this works? Barles-Souganidis
Theorem (Barles-Souganidis)
If the scheme satises the monotonicity, consistency and stability
conditions 1 to 3, then as k 0 its solution vi , i = 1, ..., I converges
locally uniformly to the unique viscosity solution of (G)
34
Intuition for Monotonicity
Write (S) as
vi = S(k, ki , vi ; vi1 , vi+1 )
For example, in growth model
vi+1 vi
S(k, ki , vi ; vi1 , vi+1 ) = u(ci ) + (F (ki ) ki ci )+
k
vi vi1
+ (F (ki ) ki ci )
k
Monotonicity: S in vi1 , vi+1 ( S in vi1 , vi+1 )
35
Checking the Monotonicity Condition in Growth Model
vi+1 vi
S (k, ki , vi ; vi1 , vi+1 ) = vi u(ci ) (F (ki ) ki ci )+
k
vi vi1
(F (ki ) ki ci )
k
Can check: satises monotonicity: S is indeed non-increasing in
both vi1 and vi+1
36
Meaning of Convergence