Stat 150 Stochastic Processes

Spring 2009

Lecture 5: Markov Chains and First Step Analysis

Lecturer: Jim Pitman

Further Analysis of Markov Chains

Q: Suppose = row vector, f = column vector, and P is a probability transition matrix, then what is the meaning of P n f ? Here P n f is the common value of (P n )f = (P n f ) for usual operations involving vectors and matrices. The above equality is elementary for nite state space. For countable state space it should be assumed e.g. that both and f are non-negative. Assume that is a probability distribution, i.e. i >= 0 and i i = 1. Suppose then that X0 , X1 , . . . , Xn is a Markov chain and the distribution of X0 is : P(X0 = i) = i . Then (P n )j = P(Xn = j ). So P n f =
j

(P n )j f (j ) P(Xn = j )f (j )
j

= Ef (Xn ) So the answer is: P n f = Ef (Xn ) for a Markov chain with initial distribution and transition probability matrix P . Notice that the same conclusion is obtained by rst recognizing that P n f (i) = E[f (Xn )|X0 = i]: P n f = (P n f ) =
i

i (P n f )i i E[f (Xn )|X0 = i]

i

= E[E[f (Xn )|X0 ]] = Ef (Xn )

Lecture 5: Markov Chains and First Step Analysis

Q: What sort of functions of a Markov chain give rise to Martingales? Say we have a MC X0 , X1 , . . . with transition probability matrix P . Seek a function h such that h(X0 ), h(X1 ), . . . is a Martingale. Key computation: E[h(Xn+1 )|X0 , X1 , . . . , Xn ] =E[h(Xn+1 )|Xn ] =(P h)(Xn ) where P h(i) =
j

P (i, j )h(j )

= E[h(X1 )|X0 = i] = E[h(Xn+1 )|Xn = i] Notice that if P h = h, then E[h(Xn+1 )|h(X0 ), . . . , h(Xn )] = E[E[h(Xn+1 )|X0 , . . . , Xn ]|h(X0 ), . . . , h(Xn )] = E[h(Xn )|h(X0 ), . . . , h(Xn )] = h(Xn ) Denition: A function h such that h = P h is called a harmonic function associated with the Markov matrix P , or a P -harmonic function for short. Conclusion: If h is a P -harmonic function and (Xn ) is a Markov chain with transition matrix P , then (h(Xn )) is a martingale. You can also check the converse: if (h(Xn )) is a martingale, no matter what the initial distribution of X0 , then h is P -harmonic. Note: A constant function h(i) c is harmonic for every transition matrix P , because every row of P sums to 1. Example: A simple random walk with 1, (1/2, 1/2) on {0, . . . , b}, and 0 and b absorbing: P (0, 0) = 1, P (b, b) = 1, P (i, i 1) = 1/2, 0 < i < b. Now describe the harmonic function solutions h = (h0 , h1 , . . . , hb ) of h = P h. For 0 < i < b, h(i) = j P (i, j )h(j ) = 0.5h(i 1) + 0.5h(i + 1), h(0) = h(0), h(b) = h(b). Observe that h = P h i h(i) is a straight line. So if h(0) and h(b) are specied, get h(i) = h(0) + i h(b) h(0) . So there is just b

Lecture 5: Markov Chains and First Step Analysis

a two-dimensional subspace of harmonic functions. And a harmonic function is determined by its boundary values. Exercise: Extend the discussion to a p , q walk with absorbtion at 0 and b for p = q . (Hint: Consider (q/p)i as in discussion of the Gamblers ruin problem for p = q )

First Step Analysis

First step analysis is the general approach to computing probabilities or expectations of a functional of a Markov chain X0 , X1 , . . . by conditioning the rst step X1 . The key fact underlying rst step analysis is that if X0 , X1 , . . . is a MC with some initial state X0 = i and transition matrix P , then conditionally on X1 = j , the sequence X1 , X2 , X3 , . . . (with indices shifted by 1) is a Markov chain with X1 = j and transition matrix P . You can prove this by checking from rst principles e.g. P(X2 = j2 , X3 = j3 , X4 = j4 |X1 = j ) = P (j, j2 )P (j2 , j3 )P (j3 , j4 ) Back to gamblers ruin problem: Consider random walk X0 , X1 , . . . with 1, (1/2, 1/2) on {0, . . . , b}. We want to calculate P(walk is absorbed at b | starts at X0 = a). New approach: let hb (i) := P(reach b eventually | X0 = i). Now hb (0) = 0, hb (b) = 1, and for 0 < i < b, hb (i) =P (i, i + 1)P(reach b eventually | X1 = i + 1) + P (i, i 1)P(reach b eventually | X1 = i 1) =0.5 hb (i + 1) + 0.5 hb (i 1) Observe that h = hb solves h = P h. So from the results of harmonic functions, hb (i) = i/b. Example: Random walk on a grid inside square boundary. For the symmetric nearest neighbor random walk, with all four steps to nearest neighbors equally likely, and absorbtion on the boundary, a function is harmonic if its value at each state (i, j ) in the interior of the grid is the average of its values at the four neighbors (i 1, j 1). This is a discrete analog of the notion of a harmonic function in classical analysis, which is a function in a two dimensional domain whose value at (x, y ) is the average of its values around every circle centered at (x, y ) which is inside the domain.