EM Algorithm

1. In a certain type of linkage experiment in genetics one obtains multinomial

data with four cell probabilities (1/2 + /4, (1 )/4, (1 /4), /4). (The
hypothesis of interest in this case is = 1/4.)
(a) Show how the EM algorithm may be used to nd the maximum kike-
lihood (MLE) of , by partitioning the rst cell count.
(b) Apply (a) for a sample of 400 observations providing data: (245, 32, 48, 75).
2. Every human being can be classied into one of four blood groups A, B, o
and AB. There are three underlying allels, A, B, and O, say with frequencies
p, q, and r in the population (p + q + r = 1). Under the model of Hardy-
Weinberg Equilibrium (HWE) the frequencies of the four blood group types
in the population are (p
2
+ 2pr, q
2
+ 2qr, r
2
, 2pq). We observe among N
individuals n
A
, n
B
, n
O
and n
AB
of four types.
(a) Write down the EM algorithm for estimation of (p, q, r).
(b) Estimate (p, q, r) from n
A
= 182, n
B
= 60, n
O
= 176 and n
AB
= 17.
3. Consider a sample of size n from a mixture distribution with unknown
weigts
X
i
g(x) + (1 )h(x), i = 1, . . . , n,
where g(.) and h(.) are known.
(a) Introduce Z
1
, . . . , Z
n
, where Z
i
indicates the distribution from which
X
i
has been drawn, so
X
i
|Z
i
= 1 g(x), X
i
|Z
i
= 2 h(x).
Show that the complete data likelihood can be written as
L
c
(|x, z) =
n

i=1

z
i
g(x
i
) + (1 z
i
)h(x
i
)

z
i
(1 )
1z
i
.
(b) Show that E[Z
i
|, x
i
] = g(x
i
)/

g(x
i
) +(1)h(x
i
)

, and deduce that

the EM sequence is given by

(j+1)
=
1
n
n

i=1

(j)
g(x
i
)

(j)
g(x
i
) + (1

(j)
)h(x
i
)
1
(c) Examine the convergence properties of this EM algorithm on simulated
dataset with n = 25, = 0.3, h(x) = (x), and g(x) = ((x 2)/2)/2,
where denotes the N(0, 1) density.
4. Consider the sample x = (0.12, 0.17, 0.32, 0.56, 0.98, 1.03, 1.10, 1.18, 1.23, 1.67, 1.68, 2.33),
generate form an exponential mixture
pExp() + (1 p)Exp()
All parameters p, , are unknown.
(a) Show that the likelihood h(p, , ) can be expressed as E[H(x, Z)],
where z = (z
1
, . . . , z
12
) corresponds to the vector of allocations of the
observations x
i
, to the rst and second components of the mixture;
that is, for i = 1, . . . , 12,
P(z
i
= 1) = 1 P(z
i
= 2) =
pexp(x
i
)
pexp(x
i
) + (1 p)exp(x
i
)
(b) Construct an EM algorithm for this model, and derive the maximum
likelihood estimators of the parameters for the sample provied above.
5. Suppose X
i
, i = 1, . . . , n, are i.i.d from the mixture distribution
P() + (1 )P(), 0 1, > 0, > 0.
where P() denotes the Poisson distribution mean .
(a) Let Z
i
= e
1
= (1, 0)

if X
i
is from the rst component P() and let
Z
i
= e
2
= (0, 1)

if X
i
is from the second component P(). Let S
j
=
{i : Z
i
= e
j
}, j = 1, 2. Show that the natural sucient statistics for
(, , ) for the family of distribution of the complete data {(X
i
, Z
i
) :
i = 1, . . . , n} are (

S
1
X
i
,

S
2
X
i
,

S
1
, 1).
(b) Hence construct the EM algorithm equations for obtaining the MLE of
(, , ) from a sample x
1
, . . . , x
n
.
(c) Investigate briey the performance of your algorithm when
(i) n = 7, x = (0, 1, 2, 6, 7, 9, 10).
(ii) n = 7, x = (3, 3, 5, 5, 6, 6, 7).
Metropolis-Hastings, Gibbs Samplers
2
1. Calculate the mean of a gamma G(4.3, 6.2) random variable using
(a) Metropolis-Hastings with a gamma G(4, 7) candidate.
(b) Metropolis-Hastings with a gamma G(5, 6) candidate.
2. Students t density with degrees of freedom, T

, is given by
f(x|) =

+1
2

(1 + x
2
/)
(+1)/2
Calculate the mean of a t distribution with = 4 degrees of freedom using a
Metropolis-Hastings algorithm with candidate density
(a) N(0, 1).
(b) t with = 2 degrees of freedom.
3. Consider the following joint density for two random variables X and Y
f
X,Y
(x, y) C
x
n
y
x+a1
(1 y)
nx+b1
for y (0, 1) and x = 0, 1, . . . , n. The parameters a and b are positive real
numbers and n is an integer.
(a) Compute the conditional f
X|Y
and f
Y |X
.
(b) Compute the conditional expect value E(Y |X), E(X|Y ).
(c) Compute the marginal f
X
and f
Y
.
(d) Obtain 1000 realizations of the joint random variables using a Gibbs
Sampler algorithm (to to this, set a = 4, b = 5 and n equal to a value
of your choice.)
(e) Superimpose the histograms of the values you obtain for X and Y with
the plot of their marginal probability function.
4. Consider the posterior distribution on (,
2
) associate with the joint
model
X
i
N(,
2
), i = 1, . . . , n,
N(
0
,
2
),
2
IG(a, b),
where IG(a, b) is the inverted gamma distribution (that is the inverse of
a gamma variable), with density b
a
(1/x)
a+1
e
b/x
/(a) and with
0
,
2
, a, b
specied. Denote x = (x
1
, . . . , x
n
).
3
(a) Prove that the full conditional of and
2
are respectively
|x,
2
N

2
+ n
2

0
+
n
2

2
+ n
2
x,

2

2
+ n
2

and

2
|x, IG

n
2
+ a,
1
2

i
(x
i
)
2
+ b

(b) Obtain 1000 realizations of the joint random variables using a Gibbs
Sampler algorithm (to to this, set
0
,
2
, a, b equal to the values of your
choice.)
5. A hierarchical specication for the normal model is the one-way random
eects model. There are dierent ways to parameterize this model, but a
possibility is as follows
X
ij
N(
i
,
2
), i = 1, . . . , k, j = 1, . . . , n
i
,

i
N(,
2
), i = 1, . . . , k,
N(
0
,
2

2
IG(a
1
, b
1
),
2
IG(a
2
, b
2
),
2

IG(a
3
, b
3
).
(a) Derive the set of full conditional distributions as follow

i
N

2
+ n
i

2
+
n
i

2
+ n
i

X
i
,

2

2
+ n
i

, i = 1, . . . , k,
N

2
+ k
2

0
+
k
2

2
+ k
2

2
+ k
2

2
IG

n/2 + a
1
,
1
2

ij
(X
ij

i
)
2
+ b1

2
IG

k/2 + a
2
,
1
2

i
(
i
)
2
+ b
2

IG

1/2 + a
3
,
1
2
(
0
)
2
+ b
3

where n =

i
n
i
and

=

i
n
i

i
/n.
(b) Implement this Gibb sampler in R with a
1
, b
1
, a
2
, b
2
, a
3
, b
3
are equal to
your choices.
4