5

Limit Theorems

Excerpts from Introduction to Probability: Second Edition

c
by Dimitri P. Bertsekas and John N. Tsitsiklis
Massachusetts Institute of Technology

Contents
5.1.
5.2.
5.3.
5.4.
5.5.
5.6.

Markov and Chebyshev Inequalities

The Weak Law of Large Numbers .
Convergence in Probability . . . .
The Central Limit Theorem . . .
The Strong Law of Large Numbers
Summary and Discussion . . . .
Problems . . . . . . . . . . . .

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

p.
p.
p.
p.
p.
p.
p.

265
269
271
273
280
282
284

37

38

From Introduction to Probability, by Bertsekas and Tsitsiklis

Chap. 5

5.1 MARKOV AND CHEBYSHEV INEQUALITIES

Markov Inequality
If a random variable X can only take nonnegative values, then
P(X a)

E[X]
,
a

for all a > 0.

Chebyshev Inequality
If X is a random variable with mean and variance 2 , then

2
P |X | c 2 ,
c

for all c > 0.

5.2 THE WEAK LAW OF LARGE NUMBERS

The Weak Law of Large Numbers

Let X1 , X2 , . . . be independent identically distributed random variables with
mean . For every > 0, we have



 X1 + + Xn





P |Mn | = P 
 0,
as n .
n

Sec. 5.3

Convergence in Probability

39

5.3 CONVERGENCE IN PROBABILITY

Convergence of a Deterministic Sequence

Let a1 , a2 , . . . be a sequence of real numbers, and let a be another real
number. We say that the sequence an converges to a, or limn an = a, if
for every > 0 there exists some n0 such that
|an a| ,

for all n n0 .

Convergence in Probability
Let Y1 , Y2 , . . . be a sequence of random variables (not necessarily independent), and let a be a real number. We say that the sequence Yn converges
to a in probability, if for every > 0, we have


lim P |Yn a| = 0.

40

From Introduction to Probability, by Bertsekas and Tsitsiklis

Chap. 5

5.4 THE CENTRAL LIMIT THEOREM

The Central Limit Theorem

Let X1 , X2 , . . . be a sequence of independent identically distributed random
variables with common mean and variance 2 , and define
Zn =

X1 + + Xn n

.
n

Then, the CDF of Zn converges to the standard normal CDF

1
(z) =
2

ex

/2

dx,

in the sense that

lim P(Zn z) = (z),

for every z.

Normal Approximation Based on the Central Limit Theorem

Let Sn = X1 + + Xn , where the Xi are independent identically distributed random variables with mean and variance 2 . If n is large, the
probability P(Sn c) can be approximated by treating Sn as if it were
normal, according to the following procedure.
1. Calculate the mean n and the variance n 2 of Sn .

2. Calculate the normalized value z = (c n)/ n.

3. Use the approximation

P(Sn c) (z),
where (z) is available from standard normal CDF tables.

Sec. 5.6

Summary and Discussion

41

De Moivre-Laplace Approximation to the Binomial

If Sn is a binomial random variable with parameters n and p, n is large, and
k, l are nonnegative integers, then
l + 1 np
p 2
np(1 p)

P(k Sn l)

k 1 np
p 2
np(1 p)

5.5 THE STRONG LAW OF LARGE NUMBERS

The Strong Law of Large Numbers

Let X1 , X2 , . . . be a sequence of independent identically distributed random
variables with mean . Then, the sequence of sample means Mn = (X1 +
+ Xn )/n converges to , with probability 1, in the sense that


X1 + + Xn
P lim
=
n
n

= 1.

Convergence with Probability 1

Let Y1 , Y2 , . . . be a sequence of random variables (not necessarily independent). Let c be a real number. We say that Yn converges to c with probability 1 (or almost surely) if
P


lim Yn = c = 1.

5.6 SUMMARY AND DISCUSSION