GEORGIA INSTITUTE OF TECHNOLOGY

School of Electrical and Computer Engineering

ECE 6605
Information Theory

Assigned: Monday, Sep. 27, 2010

Due: Atlanta and Savannah students: Wed, Oct. 6, 2010
Due: Video students: Wed, Oct. 13, 2010

Problem Set #3

Problem 1-2: Solve the questions 4.12 and 4.13 from Chapter 4 of the textbook (the
second edition).

Problem 3: Let assume the time is discretized to time slots or epochs. At each time
epoch we throw a fair (unbiased) coin. Let X be the waiting time (i.e., the number
of time epochs) for the first heads to appear in successive flips of a fair coin. Thus,
the distribution of the waiting time is given by P r{X = i} = (1/2)i . Let Sn be the
waiting time for the nth head to appear. Thus, we have:

S0 = 0

Sn+1 = Sn + Xn+1 for n = 0, 1, 2, . . . ,

where X1 , X2 , X3 , . . . are iid according to the above distribution.

(a) (10 points) Determine H(S1 , S2 , . . . , Sn ).

(b) (10 points) Does the process Sn have an entropy rate? If so, what is it? If not,
why not?

Problem 4: Suppose that X = {1, 2, 3, 4} and that the probabilities of the four possible
outcomes are p = { 12 , 18 , 14 , 18 }

(a) (2 points) Determine H(X).

(b) (2 points) Let q = { 18 , 14 , 12 , 18 } be probabilities associated with a random variable
Y also defined on the set {1, 2, 3, 4}. Compute H(Y ).
(c) (2 points) Find the relative entropy between p and q, (i.e., D(pkq). Also find
D(qkp).
(d) (6 points) Find a Huffman code for X.
(e) (2 points) Find the expected codeword length for the Huffman code.

1
 (f) (6 points) Now suppose that q had been the true distribution, but the Huffman
code was designed using p as in part d. Find the expected codeword length.
What is the cost for not using the true distribution q to design the code?

Problem 5: Solve the questions 4.28 from Chapter 4 of the textbook (the second edition).