E1 222 Stochastic Models and Applications

Problem Sheet 5
1. Suppose we have two boxes labelled 1 and 2. We also have d balls
labelled 1, 2, , d. Initially some of these balls are in box1 and others
are in box 2. At each instant an integer between 1 and d is selected
at random and the ball labelled by that integer is removed from its
box and placed in the other. Let X0 be the initial number of balls in
box 1 and let Xn be the number of balls in box 1 after the exchange at
instant n. Show that Xn is a Markov chain and find its transition probability matrix. Show that the chain is recurrent and find its stationary
distribution. For the case, d = 3, find Px (T 0 = n), for n = 1, 2, 3.
2. Suppose that whether or not it rains today depends on the weather
conditions for the previous three days. Explain how we can set up a
Markov chain model for this. Suppose that if it rained on each of the
previous three days then it will rain today with probability 0.6; if it did
not rain on any of the previous three days then it will rain today with
probability 0.2; in all other cases the weather today would be same as
that yesterday with probability 0.5. Now find the transition probability
for the chain.
3. Consider a Markov chain on nonnegative integers having transition
probabilities, P (x, x + 1) = p and P (x, 0) = 1 p where 0 < p < 1.
Show that the chain is irreducible and recurrent. Find P0 (T0 = n).
Show that the chain has unique stationary distribution and find that
distribution.
4. A transition probability matrix is said to be doubly stochastic if each
of the columns sum to one. (Recall that in any transition probability
matrix, the rows sum to one). Suppose that a Markov chain with
M states is irreducible and that its transition probability matrix is
doubly stochastic. Show that its stationary distribution is given by
(y) = M1 , y.
5. On a road, three out of every four trucks are followed by a car while
only one out of every five cars is followed by a truck. Find the ratio of
trucks to cars on the road.

6. A professor keeps giving a sequence of exams to the class. The exams

are of three types. Let qi denote the probability that the class does well
on exam of type i. It is known that q1 = 0.4, q2 = 0.6, q3 = 0.8. If the
class does well in the current exam, the next exam is equally likely to
be any of the three types. If the class does badly on the current exam,
then the next exam would always be of type 3. What proportion of
exams are of type i, i = 1, 2, 3?
7. Each of two switches is either ON or OFF during a day. On day n each
switch would independently be ON with proability (1 + mn1 )/2 where
mn1 is the number of switches that are ON on day n 1. What is the
fraction of days on which both switches are (i). ON, (ii). OFF.
8. A man has four umbrellas. Everyday in the morning when he sets out
for his office from home he would take an umbrella if it is raining and
if there is an umbrella at home. If it is not raining he goes without an
umbrella. Similarly, in the evenning when he sets out for home from
office he takes an umbrella if it is raining and if he has an umbrella at
his office; if it is not raining he leaves without an umbrella. Suppose
that the probability of rain in the morning or in the evenning is the
same and is given by p. Find the probability that the man is without
an umbrella when it is raining. Can you generalize this to the case
of n umbrellas? How does the structure of the Markov chain change
when the probability of rain in the morning is different from rain in the
evenning.
9. Let Yn be the sum of numbers obtained on n independent rolls of a fair
die, n = 1, 2, . Find
lim P [Yn is divisible by 3]

10. Let {Xn , n 0} be an irreducible, positive recurrent aperiodic Markov

chain whose stationary probabilities are given by . Define another
process {Yn , n 1} by Yn = (Xn1 , Xn ). (That is, Yn keeps track of
the last two states of the original chain). Is Yn a Markov Chain? If so,
find its transition probabilities and limn Prob[Yn = (i, j)].
11. Let X(t) be a WSS stochastic process with autocorrelation R( ). Show
that Prob[|X(t + )X(t)| a] < 2[R(0)R( )]/a2 .
2

12. Consider a stochastic process X(t) = eAt where A is a continuous

random variable with density fA . Express the mean (t) and the autocorrelation R(t1 , t2 ) in terms of fA .
13. Cars pass a certain location on a street according to a Poisson process
with rate . A woman wants to cross the street at that place and she
waits till she can see that no cars would pass that place in the next T
time units. Find the probability that her waiting time is zero.
14. Let {N (t), t 0} be a Poisson process with rate . Let Sn denote
time of the nth event. Find: (i). E[S4 ]; (ii). E[S4 |N (1) = 2]; (iii).
E[N (4) N (2)|N (1) = 3].
15. Customers arrive at a bank at a Poisson rate . Suppose two customers
arrived during the first hour. What is the probability that (i). both
arrived during the first twenty minutes, (ii). at least one arrived during
the first twenty minutes.