EC744 Lecture Note 9 Convergence of Markov Processes

Prof. Jianjun Miao

Policy and Transition Functions. The policy function g induces a Markov process st = (xt, zt) on the product space with transition function P. Theorem. Let (X, X ) and (Z, Z) be measurable spaces; let (S, S) be the product space; let Q be a transition function on (Z, Z) ; and let g : S X be a measurable policy function. Then P ((x, z) , A B) = 1g(x,z)AQ (z, B) all x X, z Z, A X , B Z, denes a transition function on (S, S) . Theorem. If g is continuous and Q satises the Feller property, then P also has the Feller property. Proof. Use the fact
Z
XZ

f x0, z 0 P (x, z) , dx0 dz 0 =

0 Q z, dz 0 f g (x, z) , z

 We are interested in nding conditions under which the process {st} converges to a unique stationary point, and this stationary point does not depend on the initial state. knowing how parameter values aect the behavior of the system. whether a certain model is consistent with a set of data, which are time series for some observable functions of the state. (LLN)

The questions are How to dene a stationary point? How to dene convergence?

Dene an operator T T (A) =

P (s, A) (ds)

Induce a sequence of probability measures t = T t1. An invariant (stationary) distribution of (st) is a xed point of the operator T . Dene Q1 (z, A) = Q (z, A) , Qn+1 (z, A) = Then n = T n1 = T n0.
Z n z 0, A Q z, dz 0 . Q

Markov Chains

Discrete state space S = {s1, s2, ...} Stochastic matrix or transition matrix Pij with raw sum equal to unity A set E S is called an ergodic set if p (si, E) = 1, for si E, and if no proper subset of E has this property

Evolution of distributions t+1 = tP. A distribution is called invariant or stationary if t+1 = t

A state is called transient if there is a positive probability of leaving and never returning.

Example 1. Let P =
"

3/4 1/4 1/4 3/4

S is an ergodic set. lim P n =

"

1/2 1/2 1/2 1/2

Thus, the invariant distribution is = (1/2, 1/2)

Example 2. 1 /2 /2 Let P = 0 1/2 1/2 . 0 1/2 1/2 The set E = {s2, s3} is the only ergodic set, and state s1 is transient.
lim P n = 0 1/2 1/2 .

0 1/2 1/2 0 1/2 1/2

The invariant distribution is = (0, 1/2, 1/2) .

Example 3.

Let P =

0 0 3/4 1/4 0 0 1/4 3/4 . Then: 3/4 1/4 0 0 1/4 3/4 0 0

2n P

1/2 1/2 0 0 1/2 1/2 0 0 0 0 1/2 1/2 0 0 1/2 1/2

2n+1 ,P

0 0 1/2 1/2 0 0 1/2 1/2 1/2 1/2 0 0 1/2 1/2 0 0

S is the ergodic set but it has cyclically moving subsets. But

X 1 N 1 n P = lim N n=0

1/4 1/4 1/4 1/4

1/4 1/4 1/4 1/4

1/4 1/4 1/4 1/4

1/4 1/4 1/4 1/4

The invariant distribution is = (1/4, 1/4, 1/4, 1/4) .

Example 4.

Let P =

3/4 1/4 0 0 1/4 3/4 0 0 0 0 3/4 1/4 0 0 1/4 3/4

. .

n= Then lim P

1/2 1/2 0 0 1/2 1/2 0 0 0 0 1/2 1/2 0 0 1/2 1/2

There are two invariant distributions, = (1/2, 1/2, 0, 0) and = (0, 0, 1/2, 1/2) . 1 2 All convex combinations are also invariant distributions.

Example 5. 1 Let P = 0 1 0 . 0 0 1 s1 is a transient state. There are two ergodic sets E1 = {s2} and E2 = {s3} .
lim P n = 0

0 1 0 . 0 0 1

Each row of this matrix is an invariant distribution.

 If the limit limt t = is independent of the initial condition, we say the markov chain is asymtoptic stationary with a unique stationary distribution
n Theorem. Let P be a stochastic matrix for which Pij > 0 for all i, j and some n 1. Then P has a unique stationary distribution and the process is asymptotically stationary.

Denition Let (S, ) be a metric space; and let S be its Borel sets; let {n} and be measures on (S, S) . Then {n} converges weakly to if
n

lim

f dn =

f d, all f C (S) .

Denition {n} converges strongly to if

n

lim

and if in addition the rate of convergence is uniform for all f B(S, S) such that ||f || = supsS |f (s)| 1.

f dn =

f d, all f B (S, S) ,

Theorem (Existence) If S is a compact metric space and P satises the Feller property, then the Markov process (st) has an invariant distribution. Condition D (Doeblins Condition) There is a nite measure on (S, S), an integer N 1, and a number > 0, such that if (A) , then P N (s, A) 1 , all s S. Theorem (Strong Convergence) Suppose P satises condition D. Suppose in addition that if A is any set of positive measure, then for each s S, there exists n 1 such that P n (s, A) > 0. Then T has a unique invariant PN (ergodic) measure and limN n=1 T n0/N = for all 0 on S. Condition M There exists > 0 and an integer N 1 such that for any A S, wither P N (s, A) , all s S, or P N (s, Ac) , all s S. Theorem (Strong Convergence) If P satises condition M for N 1 and > 0, then there exists a unique probability measure on S such that limn T n0 = strongly for any 0 on S.

P is increasing if f (s0)P (s, ds0) is increasing in s for all bounded and increasing function f . Mixing Condition There exists c S = [a, b] , > 0, and N 1 such that P N (a, [c, b]) and P N (b, [a, c]) Theorem. If P is increasing, has the Feller property, and satises the mixing condition, then P has a unique invariant measure and T n0 converges weakly to for any 0 on S. Application: Policy function g 1. g is increasing, so P is increasing 2. g is continuous, so P satises Feller property 3. P satises a mixing condition.

Theorem (Parametric Continuity) Let be the parameter (metric) space. Assume that a. S is compact; b. if {(sn, n)} is a sequence in S converging to (s0, 0) , then the sequence {Pn (sn, )} converges weakly to P0 (s0, ) ;
c. for each , T has a unique invariant measure .

Then n weakly when n .

Example 1: Stochastic Growth Model with IID shocks DP v (x) = max U (x y) +

v (zf (y)) (dz)

Assumption: For some > 0, ((a, b]) (b a) , all (a, b] Z = [1, z ] . Assumption: f (0) > 0 and f 0 (0) > 1. Choose state space X = [0, x] where x satisfying x = z f () . x The policy function g denes a Markov process on X, corresponding to xt+1 = zt+1f (g (xt)) . This process has a transition function P on (X, X ) . P has the Feller property and since X is compact, P has at least one invariant measure.

Show that P is monotone. Dene z = E [z] and dene x as the unique value satisfying f 0 (g (x)) z = 1. For any x and z dene the sequence 0 (x, z) = x and n+1 (x, z) = f (g (n (x, z))) z. We have P n (x, (n (x, z ) , n (x, z ))) n n, P n (x, (n (x, 1) , n (x, 1 + ))) n n, for all 0 < < z 1, n = 1, 2, .... n=0 Consider the sequence {n (0, z )} . Since 0 (0, z ) = 0 < f (0) z = f (g (0)) z = 1 (0, z ) , it follows by induction that this sequence is increasing. Since it is bounded by x, it converges to a value . We claim > x.

Thus, we can choose N large enough such that N (0, z ) > x, and > 0 such that N (0, z ) > x. Then x < N (0, z ) < N (0, z ) x. Hence P N (0, (x, x]) P N (0, (N (0, z ) , N (0, z )]) N N , as was to be shown. Homework: Show that P N (, (0, x]) P N (, (N (, 1) , N (, 1 + )]) N N , x x x x

Example 2: Equilibrium in a Pure Currency Economy Consider an economy with a large number of households, each composed of one work and one shopper. Household utility E

t=0

where U is bounded and continuously dierentiable. In addition, U (, z) is strictly increasing and strictly concave, U1 (c, ) is strcitly increasing in z. Preference shocks zt is iid with distribution on Z = [z, z] . satises: for some > 0, ((a, b]) (b a) , all (a, b] Z.

tU (ct, zt)

Consider steady state in which the price level p is constant. Let m = M/p. v (m, z) = max U (c, z) +
c,m0 0

subject to

0, z 0 dz 0 v m

m0 + c m y 0 c m 0. There is a unique bouned continuous function v satisfying DP. For each z Z, v (, z) is strictly increasing, strictly concave, and continuously dierentiable in m.

 The optimal policy m0 = g (m, z) is continuous. For each z Z, there exists some (z) > 0 such that g (, z) = y on [0, (z)] , and g (, z) is strictly increasing on [ (z) , ). In addition, (z) y. For each z Z, optimal consumption policy c (, z) is strictly increasing. There exists m > 0 such that g (m, z) = m. For each m 0, g (m, ) is weakly decreasing in z.

The policy function g and the measure induce a transition function P on R+. The ergodic set is X = [y, m] .

The Markov process on (X, X) dened by P satises the Feller property and the mixing condition. Thus, there is a unique stationary distribution Let M be the exogenously given money supply. Then market clearing condition is
Z

m (dm) = M/p.

This equation determines the equilibrium price level.