Markov 1

Markov Process
1. An example
1.1 Urban and rural populations are 48 million and 116 million respectively. Historical data
reveals that 5 per cent of urban population migrates to rural area and 20 per cent rural
population migrates to urban area. Determine urban-rural population after one year.
(Assuming zero per cent growth)
1.2 Simple arithmetic:
Urban: 48*0.95 + 116*0.3 = 68 million. Rural: 116*0.80 + 48*.05 = 92
1.3 Above example with percentage
Present urban-rural population is 30/70 and next year it would be:
Urban: 30*0.95 + 70*0.3 = 42.50%. Rural: 70*0.80 + 30*.05 = 57.50%
2. Generalization:

2.1

2.2

2.3 Sij i = period and j = sector

Aij i = from and j = to
2.4 S1 = S0T Where S = present share and T is transition matrix
Sn = S0TTTTTT…T {number of Ts are n and T remains constant}
Sn = SoTn
3. Markov Process
Above example is of markov process. This technique has numerous applications in
business; market share analysis, brand switching, consumer purchasing brand A in period
one will purchase brand B in next period, bad debt prediction, determining whether a
machine will have breakdown in the future, in HRM people changing companies etc.
4. Assumptions
4.1 There is a limited number or finite number of possible state. [State: Brands (A, B, C),
conditions (Breakdown, no breakdown), Companies, etc}.
4.2 The probability of changing state remains the same over the time. This means transition
matrix will remain same.
4.3 We can predict any future state from the previous state and the matrix of transition
probabilities.
4.4 The size and makeup of the system (for example, the total number of manufacturers and
customers) do not change during the analysis.
4.5 The states are mutually exclusive and collectively exhaustive. Statistically

October 25, 2010

 Markov 2

5. Example from Anderson Page 796

We have two stores M (Murphy) and A (Ashley). A market research study was conducted
and data collected from 100 from 100 customers. Analysis of data reveals that 90% of
M’s customer will shop at M and remaining 10 % will switch to A in the following week.
Study further reveals that 20% of A will switch to M and 80% will remain with A.

6. Transition matrix P =

7. We have two states and sum of each row = 1 as it is mutually exclusive and completely
exhaustive.
8. Probabilities of states is a row vector
n is period and k number of companies.
9. Π(1) = Π(0)P
Π(2) = Π(1)P
Π(3) = Π(2)P
.
.
.
Π(n + 1) = Π(n)P

10. Market share for next period

Π(1) = Π(0) P = [0.90 0.10]
Π(2) = Π(1) P = [0.83 0.17]
11. Results of the process for 15 weeks are given in the following tables:
State 1 State 2 State 1 State 2
Period 0 1.000 0.000 Period 0 0.000 1.000
Period 1 0.900 0.100 Period 1 0.200 0.800
Period 2 0.830 0.170 Period 2 0.340 0.660
Period 3 0.781 0.219 Period 3 0.438 0.562
Period 4 0.747 0.253 Period 4 0.507 0.493
Period 5 0.723 0.277 Period 5 0.555 0.445
Period 6 0.706 0.294 Period 6 0.588 0.412
Period 7 0.694 0.306 Period 7 0.612 0.388
Period 8 0.686 0.314 Period 8 0.628 0.372
Period 9 0.680 0.320 Period 9 0.640 0.360
Period 10 0.676 0.324 Period 10 0.648 0.352
Above results were obtained through QM. Show on QM
12. If we had 1000 customer, who shopped last at M; our analysis indicates that during 5th
week 723 customers will shop at M and 277 at A.
13. If we had 1000 customer, who shopped last at A; our analysis indicates that during 5th
week 555 customers will shop at M and 445A.

October 25, 2010

 Markov 3

14. Steady-State Probabilities

14.1 Analyzes of tables indicate that as n gets larger, the difference between the state
probabilities for the nth and (n + 1)th period becomes smaller and smaller. We conclude
that when n gets larger, Π(n + 1) and Π(n) becomes equal (almost).
14.2 Its means probabilities will become independent of beginning state when is large.
14.3 The probabilities that we approach after a large number of transitions are referred to as
the steady-state probabilities. It is also called equilibrium condition.
15. Interpretation of Steady-State Probabilities.
15.1 Π(n + 1) = Π(n)P and since difference between Π(n + 1) and Π(n) is very small we may
write Π = ΠP (remove the period as it does not matter)

15.2 [Π1 Π2] = [Π1 Π2]

Π1 = 0.9 Π1 + 0.2 Π2 ----------------- 1

Π2 = 0.1 Π1 + 0.8 Π2 -----------------2
Π1 + Π2 = 1 ----------------- 3
Put (3) in (1) and solve; SS{2/3 and 1/3}
15.3 If we have 1000 customers, markov process model tells us that in the long run, M will get
667 customers and A will get 333 customers.
15.4 Steady-state probabilities can be interpreted as the market share for two stores.
16. Use of Markov
16.1 Store A launch an advertisement campaign instead of getting 10% of M’s share get 15%.
The revised steady-state probabilities would be
Π1 = 0.85 Π1 + 0.2 Π2 ----------------- 1
Π2 = 0.15 Π1 + 0.8 Π2 -----------------2
Π1 + Π2 = 1 ----------------- 3
Put (3) in (1) and solve; SS{0.57, 0.43}. The share of A have been increase
16.2 Suppose market size is 6000 customer, ad campaign will increase customers from 2000
(6000x1/3) to 2580, increase by 580.
16.3 We can do the cost benefit analysis.
17. Account Receivable Analysis
17.1 We have four states (categories); 1. Paid, 2. Bad debt, 3. 0 – 30 days (aging – oldest
unpaid bill is not more than 30 days) and 4. 31 – 90. If any portion of an account balance
exceeds 90 days, that portion is written off as bad debt.
17.2 Suppose that one customer’s account balance on September 30 is as follows:
Aug. 15 Sep. 18 Sep. 28 Total
$ 25 $ 10 $ 50 85

October 25, 2010

 Markov 4

17.3 An aging AR on September 30 would be assign the total balance of $85 to 31 – 90

category because oldest unpaid bill of August 15. On October 7 customer pays $25 (of
August 15). Remaining $60 will be placed in 0 – 30 category. This means transition from
one state to another state.
18. Viewing AR operations as Markov Process
18.1 We will consider each week as a trail of a Markov process with a dollar existing in one of
the following states of the system:
State – 1 Paid category Four states can be placed in
State – 2 Bad debt category any order. However, we are
State – 3 0 – 30 days category using this order to facilitate
State – 4 31 – 90 days category some matrix manipulations.
18.2 We can track the week-by-week status of one dollar by using Markov analysis.
18.3 Based on historical transitions of AR, the following matrix of transition probabilities has
been developed:

18.4 Note that the probability of a dollar in the 0 – 30 days category moving to the paid
category in the next period is 40%. Note also that a dollar in a 0 – 30 days category
cannot make the transition to a bad debt state in one week.
19. Absorbing States
We saw (Sec 17.3) that a dollar can change state from 2 to 3 and vice versa. However,
once a dollar makes a transaction to state – 1, the paid category, the probability of making
a transition to other state is zero. Similarly, once a dollar is in state – 2 (bad debt),
probability of transition to any other state is zero. Thus, once a dollar reaches state 1 or 2,
the system will remain in that state indefinitely. We conclude that all AR eventually be
absorbed into state – 1 or state – 2, hence the name absorbing state.
20. Whenever a Markov process has absorbing states, we do not compute steady-state
probabilities because each unit ultimately ends up in one of the absorbing states.
21. Now we want to know what percentage of 0 – 30 state will go to paid state and what goes
to bad debt and similarly for 31 – 90 state.
22. The Fundamental Matrix
22.1 Computation of absorbing state probabilities requires the determination of a fundamental
matrix. We begin the partitioning the matrix of transition probabilities into the following
four parts:

October 25, 2010

 Markov 5

{Lower parts (2x2) are R and Q respectively}

22.2 A matrix N called fundamental matrix, and given by the formula N = (I – Q) – 1

22.3 N= Teach inverse of 2x2 and show the calculations.

22.4

22.5 Interpretation: 89% of 0 – 30 days category will eventually be paid and 11% will become
bad debt. 74% of 31 – 90 days category will eventually being paid and 26% will remain
unpaid.
23. Establishing the Allowance for Doubtful Accounts – An Example
Suppose that 31 December balance sheet shows that AR in 0 – 30 days category is $1000
and 31 – 90 days category is $2000. In matrix form B = [1000 2000]. Multiply B with
NR i.e. BNR = [2370 630]. Thus out of $3,000, 2370 will be collected and 630
uncollected. Therefore, an allowance for bad debt should be $630.

October 25, 2010