K.K.

Gan L5: Maximum Likelihood Method 1

Lecture 5
Maximum Likelihood Method
!

x
=
1
N
x
i
i=1
N
"
! Suppose we are trying to measure the true value of some quantity (x
T
).
" We make repeated measurements of this quantity {x
1
, x
2
, x
n
}.
" The standard way to estimate x
T
from our measurements is to calculate the mean value:
# set x
T
=
x
.
# DOES THIS PROCEDURE MAKE SENSE???
# MLM: a general method for estimating parameters of interest from data.
! Statement of the Maximum Likelihood Method
" Assume we have made N measurements of x {x
1
, x
2
, x
n
}.
" Assume we know the probability distribution function that describes x: f(x, !).
" Assume we want to determine the parameter !.
# MLM: pick ! to maximize the probability of getting the measurements (the x
i
's) that we did!
! How do we use the MLM?
" The probability of measuring x
1
is f(x
1
, !)dx
" The probability of measuring x
2
is f(x
2
, !)dx
" The probability of measuring x
n
is f(x
n
, !)dx
" If the measurements are independent, the probability of getting the measurements we did is:
" We can drop the dx
n
term as it is only a proportionality constant
#
!
L = f (x
1
,")dx # f (x
2
,")dx # # # f (x
n
,")dx = f (x
1
,") # f (x
2
,") # # # f (x
n
,")dx
n
!
L = f (x
i
i=1
N
" ,#) Likelihood Function
K.K. Gan L5: Maximum Likelihood Method 2
" We want to pick the ! that maximizes L:
" Both L and lnL have maximum at the same location.
# maximize lnL rather than L itself because lnL converts the product into a summation.
# new maximization condition:
% ! could be an array of parameters (e.g. slope and intercept) or just a single variable.
% equations to determine ! range from simple linear equations to coupled non-linear equations.
! Example:
" Let f(x, !) be given by a Gaussian distribution.
" Let ! = be the mean of the Gaussian.
" We want the best estimate of ! from our set of n measurements {x
1
, x
2
, x
n
}.
" Lets assume that " is the same for each measurement.
" The likelihood function for this problem is:
!
"L
"#
#=#*
= 0
!
ln L = ln f (x
i
,")
i=1
N
#
!
" ln L
"#
#=#*
=
"
"#
i=1
N
$ ln f (x
i
,#)
#=#*
= 0
!
f (x
i
,") =
1
# 2$
e
%
( x
i
%")
2
2#
2

!
L = f (x
i
,")
i=1
n
# =
1
$ 2%
e
&
( x
i
&")
2
2$
2
i=1
n
# =
1
$ 2%
'
(
)
*
+
,
n
e
&
( x
1
&")
2
2$
2
e
&
( x
2
&")
2
2$
2
Le
&
( x
n
&")
2
2$
2
=
1
$ 2%
'
(
)
*
+
,
n
e
&
( x
i
&")
2
2$
2
i=1
n
-
K.K. Gan L5: Maximum Likelihood Method 3
" Find ! that maximizes the log likelihood function:
" If " are different for each data point
# ! is just the weighted average:
!
" ln L
"#
=
"
"#
nln
1
$ 2%
&
'
(
)
*
+
,
(x
i
,#)
2
2$
2
i=1
n
-
.
/
0
1
2
3
= 0
"
"#
(x
i
,#)
2
i=1
n
- = 0
2(x
i
i=1
n
- ,#)(,1) = 0
x
i
i=1
n
- = n#
# =
1
n
x
i
i=1
n
-
!
" =
x
i
#
i
2
i=1
n
$
1
#
i
2
i=1
n
$
Average
Weighted average
K.K. Gan L5: Maximum Likelihood Method 4
! Example
" Let f(x, !) be given by a Poisson distribution.
" Let ! = be the mean of the Poisson.
" We want the best estimate of ! from our set of n measurements {x
1
, x
2
, x
n
}.
" The likelihood function for this problem is:
" Find ! that maximizes the log likelihood function:
Some general properties of the Maximum Likelihood Method
! For large data samples (large n) the likelihood function, L, approaches a Gaussian distribution.
! Maximum likelihood estimates are usually consistent.
# For large n the estimates converge to the true value of the parameters we wish to determine.
! Maximum likelihood estimates are usually unbiased.
# For all sample sizes the parameter of interest is calculated correctly.
! Maximum likelihood estimate is efficient: the estimate has the smallest variance.
! Maximum likelihood estimate is sufficient: it uses all the information in the observations (the x
i
s).
! The solution from MLM is unique.
" Bad news: we must know the correct probability distribution for the problem at hand!
!
L = f (x
i
,")
i=1
n
# =
e
$"
"
x
i
x
i
!
i=1
n
# =
e
$"
"
x
1
x
1
!
e
$"
"
x
2
x
2
!
...
e
$"
"
x
n
x
n
!
=
e
$n"
"
x
i
i=1
n
%
x
1
! x
2
!..x
n
!
!
d ln L
d"
=
d
d"
#n"+ln" $ x
i
i=1
n
% #ln(x
1
! x
2
!..x
n
!)
&
'
(
)
*
+
= #n+
1
"
x
i
i=1
n
% = 0
" =
1
n
x
i
i=1
n
% Average
K.K. Gan L5: Maximum Likelihood Method 5
Maximum Likelihood Fit of Data to a Function
! Suppose we have a set of n measurements:
" Assume each measurement error (") is a standard deviation from a Gaussian pdf.
" Assume that for each measured value y, theres an x which is known exactly.
" Suppose we know the functional relationship between the ys and the xs:
% !, #...are parameters.
# MLM gives us a method to determine !, #... from our data.
! Example: Fitting data points to a straight line:
" Find ! and # by maximizing the likelihood function L likelihood function:
x
1
, y
1
!
1
x
2
, y
2
!
2
...
" x
n
, y
n
!
n
!
y = q(x,",#,...)
!
q(x,",#,...) = "+#x
L = f (x
i
,",#)
i=1
n
$ =
1
%
i
2&
e
'
( y
i
'q( x
i
,",#))
2
2%
i
2
i=1
n
$ =
1
%
i
2&
e
'
( y
i
'"'#x
i
)
2
2%
i
2
i=1
n
$
!
" ln L
"#
=
"
"#
ln
1
$
i
2%
&
'
(
)
*
+
,
(y
i
,# ,-x
i
)
2
2$
i
2
.
/
0
1
2
3
i=1
n
4 = ,
2(y
i
,# ,-x
i
)(,1)
2$
i
2
.
/
0
1
2
3
i=1
n
4 = 0
" ln L
"-
=
"
"-
ln
1
$
i
2%
&
'
(
)
*
+
,
(y
i
,# ,-x
i
)
2
2$
i
2
.
/
0
1
2
3
i=1
n
4 = ,
2(y
i
,# ,-x
i
)(,x
i
)
2$
i
2
.
/
0
1
2
3
i=1
n
4 = 0
two linear equations
with two unknowns
K.K. Gan L5: Maximum Likelihood Method 6
" Assume all "s are the same for simplicity:
" We now have two equations that are linear in the two unknowns, ! and #.
% We will see this problem again when we talk about least squares (chi-square) tting.
! EXAMPLE:
" A trolley moves along a track at constant speed. Suppose the following measurements of the
time vs. distance were made. From the data nd the best value for the velocity (v) of the trolley.
" Our model of the motion of the trolley tells us that:
!
y
i
i=1
n
" # $
i=1
n
" # %x
i
i=1
n
" = 0
y
i
x
i
i=1
n
" # $x
i
i=1
n
" # %x
i
2
i=1
n
" = 0
!
y
i
i=1
n
" = n#+$ x
i
i=1
n
"
y
i
x
i
i=1
n
" = # x
i
i=1
n
" +$ x
i
2
i=1
n
"
!
" =
y
i
i=1
n
# x
i
2
$ y
i
x
i
x
i
i=1
n
#
i=1
n
#
i=1
n
#
n x
i
2
$( x
i
i=1
n
# )
2
i=1
n
#
and % =
n x
i
y
i
i=1
n
# $ y
i
x
i
i=1
n
#
i=1
n
#
n x
i
2
$( x
i
i=1
n
# )
2
i=1
n
#
61 49 40 33 19 11 Distance d (mm)
6.0 5.0 4.0 3.0 2.0 1.0 Time t (seconds)
!
d = d
0
+vt
!
y
i
i=1
n
"
y
i
i=1
n
" x
i
#
$
%
%
%
%
&
'
(
(
(
(
=
n x
i
i=1
n
"
x
i
i=1
n
" x
i
2
i=1
n
"
#
$
%
%
%
%
&
'
(
(
(
(
)
*
#
$
%
%
%
&
'
(
(
(
Matrix form
Taylor Eqs. 8.10-12
K.K. Gan L5: Maximum Likelihood Method 7
" We want to find v, the slope (#) of the straight line describing the motion of the trolley.
" We need to evaluate the sums listed in the above formula:
!
x
i
i=1
n
" = t
i
i=1
6
" = 21 s
y
i
= d
i
i=1
6
"
i=1
n
" = 213 mm
x
i
y
i
i=1
n
" = t
i
d
i
i=1
6
" = 919 s # mm
x
i
2
i=1
n
" = t
i
2
i=1
6
" = 91 s
2
d
0
= 0.8 mm
!
v =
n x
i
y
i
i=1
n
" # y
i
x
i
i=1
n
"
i=1
n
"
n x
i
2
#( x
i
i=1
n
" )
2
i=1
n
"
=
6$919 #21$213
6$91#21
2
= 9.9mm/ s best estimate of the speed
best estimate of the starting point
K.K. Gan L5: Maximum Likelihood Method 8
0
10
20
30
40
50
60
70
0 1 2 3 4 5 6 7
time (sec)
_____ d = 0.8 + 9.9t
d
i
s
t
a
n
c
e

(
m
m
)
" The line best represents our data.
" Not all the data points are "on" the line.
" The line minimizes the sum of squares of the deviations between the line and our data (d
i
):
MLM fit to the data for d = d
0
+ vt
!
" = data
i
#prediction
i
[ ]
2
i=1
n
$ = d
i
#(d
0
+vt
i
)
[ ]
2
i=1
n
$
Least square fit