4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

GaussMarkovtheorem
FromWikipedia,thefreeencyclopedia

Instatistics,theGaussMarkovtheorem,namedafterCarlFriedrichGaussandAndreyMarkov,states
thatinalinearregressionmodelinwhichtheerrorshaveexpectationzeroandareuncorrelatedandhave
equalvariances,thebestlinearunbiasedestimator(BLUE)ofthecoefficientsisgivenbytheordinary
leastsquares(OLS)estimator.Here"best"meansgivingthelowestvarianceoftheestimate,ascomparedto
otherunbiased,linearestimators.Theerrorsdonotneedtobenormal,nordotheyneedtobeindependent
andidenticallydistributed(onlyuncorrelatedwithmeanzeroandhomoscedasticwithfinitevariance).The
requirementthattheestimatorbeunbiasedcannotbedropped,sincebiasedestimatorsexistwithlower
variance.See,forexample,theJamesSteinestimator(whichalsodropslinearity)orridgeregression.

Contents
1
2
3
4
5

6
7
8
9

Statement
Proof
Remarksontheproof
Generalizedleastsquaresestimator
GaussMarkovtheoremasstatedineconometrics
5.1 Linearity
5.2 Strictexogeneity
5.3 Fullrank
5.4 Sphericalerrors
Seealso
6.1 Otherunbiasedstatistics
Notes
References
Externallinks

Statement
Supposewehaveinmatrixnotation,

expandingto,

where arenonrandombutunobservableparameters,
arenonrandomandobservable(calledthe
"explanatoryvariables"), arerandom,andso arerandom.Therandomvariables arecalledthe
"disturbance","noise"orsimply"error"(willbecontrastedwith"residual"laterinthearticleseeerrorsand
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

1/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

residualsinstatistics).Notethattoincludeaconstantinthemodelabove,onecanchoosetointroducethe
constantasavariable
withanewlyintroducedlastcolumnofXbeingunityi.e.,
for
all .
TheGaussMarkovassumptionsare

(i.e.,alldisturbanceshavethesamevariancethatis"homoscedasticity"),and

for

thatis,theerrortermsareuncorrelated.Alinearestimatorof

isalinearcombination

inwhichthecoefficients arenotallowedtodependontheunderlyingcoefficients ,sincethosearenot

observable,butareallowedtodependonthevalues
,sincethesedataareobservable.(Thedependence
ofthecoefficientsoneach
istypicallynonlineartheestimatorislinearineach andhenceineach
random ,whichiswhythisis"linear"regression.)Theestimatorissaidtobeunbiasedifandonlyif

regardlessofthevaluesof

.Now,let

besomelinearcombinationofthecoefficients.Then

themeansquarederrorofthecorrespondingestimationis

i.e.,itistheexpectationofthesquareoftheweightedsum(acrossparameters)ofthedifferencesbetween
theestimatorsandthecorrespondingparameterstobeestimated.(Sinceweareconsideringthecasein
whichalltheparameterestimatesareunbiased,thismeansquarederroristhesameasthevarianceofthe
linearcombination.)Thebestlinearunbiasedestimator(BLUE)ofthevector ofparameters isone
withthesmallestmeansquarederrorforeveryvector oflinearcombinationparameters.Thisis
equivalenttotheconditionthat

isapositivesemidefinitematrixforeveryotherlinearunbiasedestimator .
Theordinaryleastsquaresestimator(OLS)isthefunction

https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

2/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

of and (where denotesthetransposeof

(mispredictionamounts):

)thatminimizesthesumofsquaresofresiduals

ThetheoremnowstatesthattheOLSestimatorisaBLUE.Themainideaoftheproofisthattheleast
squaresestimatorisuncorrelatedwitheverylinearunbiasedestimatorofzero,i.e.,witheverylinear
combination
whosecoefficientsdonotdependupontheunobservable butwhose
expectedvalueisalwayszero.

Proof
Let

beanotherlinearestimatorof andletCbegivenby
,whereDisa
nonzeromatrix.Aswe'rerestrictingtounbiasedestimators,minimummeansquarederrorimplies
minimumvariance.Thegoalisthereforetoshowthatsuchanestimatorhasavariancenosmallerthanthat
of ,theOLSestimator.
Theexpectationof is:

Therefore, isunbiasedifandonlyif

Thevarianceof is

SinceDD'isapositivesemidefinitematrix,

https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

exceeds

byapositivesemidefinitematrix.

3/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

Remarksontheproof
Asithasbeenstatedbefore,theconditionof
linearunbiasedestimatorof
is
anotherlinearunbiasedestimatorof

Therefore,

isequivalenttothepropertythatthebest

(bestinthesensethatithasminimumvariance).Toseethis,let
.

Moreover,supposethattheequalityholds(

).Ithappensifandonlyif

Rememberingthat,fromtheproofabove,wehave

Thisprovesthattheequalityholdsifandonlyif
asaBLUE.

,then:

whichgivestheunicityoftheOLSestimator

Generalizedleastsquaresestimator
Thegeneralizedleastsquares(GLS),developedbyAitken,[1]extendstheGaussMarkovtheoremtothe
casewheretheerrorvectorhasanonscalarcovariancematrix.[2]TheAitkenestimatorisalsoaBLUE.

GaussMarkovtheoremasstatedineconometrics
InmosttreatmentsofOLS,theregressorsinthedesignmatrix areassumedtobefixedinrepeated
samples.Thisassumptionisconsideredinappropriateforapredominantlynonexperimentalsciencelike
econometrics.[3]Instead,theassumptionsoftheGaussMarkovtheoremarestatedconditionalon .

Linearity
Thedependentvariableisassumedtobealinearfunctionofthevariablesspecifiedinthemodel.The
specificationmustbelinearinitsparameters.Thisdoesnotmeanthattheremustbealinearrelationship
betweentheindependentanddependentvariables.Theindependentvariablescantakenonlinearformsas
longastheparametersarelinear.Theequation
qualifiesaslinearwhile
canbetransformedtobelinearbyreplacing byanotherparameter,say .Anequation
withaparameterdependentonanindependentvariabledoesnotqualifyaslinear,forexample
,where
isafunctionof .
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

4/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

Datatransformationsareoftenusedtoconvertanequationintoalinearform.Forexample,theCobb
Douglasfunctionoftenusedineconomicsisnonlinear:

Butitcanbeexpressedinlinearformbytakingthenaturallogarithmofbothsides:[4]

Thisassumptionalsocoversspecificationissues:assumingthattheproperfunctionalformhasbeen
selectedandtherearenoomittedvariables.

Strictexogeneity
Forall observations,theexpectationconditionalontheregressorsoftheerrortermiszero:[5]

where

isthedatavectorofregressorsfortheithobservation,and

consequently

isthedatamatrixordesignmatrix.

Geometrically,thisassumptionsimpliesthat
product(i.e.,theircrossmoment)iszero.

and areorthogonaltoeachother,sothattheirinner

Thisassumptionisviolatediftheexplanatoryvariablesarestochastic,forinstancewhentheyaremeasured
witherror,orareendogenous.[6]Endogeneitycanbetheresultofsimultaneity,wherecausalityflowsback
andforthbetweenboththedependentandindependentvariable.Instrumentalvariabletechniquesare
commonlyusedtoaddressthisproblem.

Fullrank
Thesampledatamatrix

Otherwise

mustbenonsingular,i.e.itmusthavefullrank.

isnotinvertibleandtheOLSestimatorcannotbecomputed.

Aviolationofthisassumptionisperfectmulticollinearity,i.e.someexplanatoryvariablesarelinearly
dependent.Onescenarioinwhichthiswilloccuriscalled"dummyvariabletrap,"whenabasedummy
variableisnotomittedresultinginperfectcorrelationbetweenthedummyvariablesandtheconstant
term.[7]
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

5/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

Multicollinearity(aslongasitisnot"perfect")canbepresentresultinginalessefficient,butstillunbiased
estimate.Theestimateswillbelesspreciseandhighlysensitivetoparticularsetsofdata.[8]
Multicollinearitycanbedetectedfromconditionnumberorthevarianceinflationfactor,amongothertests.

Sphericalerrors
Theouterproductoftheerrorvectormustbespherical.

Thisimpliestheerrortermhasuniformvariance(homoscedasticity)andnoserialdependence.[9]Ifthis
assumptionisviolated,OLSisstillunbiased,butinefficient.Theterm"sphericalerrors"willdescribethe
multivariatenormaldistribution:if
inthemultivariatenormaldensity,thenthe
equationf(x)=cistheformulaforaballcenteredatwithradiusinndimensionalspace.[10]
Heteroskedacityoccurswhentheamountoferroriscorrelatedwithanindependentvariable.Forexample,
inaregressiononfoodexpenditureandincome,theerroriscorrelatedwithincome.Lowincomepeople
generallyspendasimilaramountonfood,whilehighincomepeoplemayspendaverylargeamountoras
littleaslowincomepeoplespend.Heteroskedacitycanalsobecausedbychangesinmeasurement
practices.Forexample,asstatisticalofficesimprovetheirdata,measurementerrordecreases,sotheerror
termdeclinesovertime.
Thisassumptionisviolatedwhenthereisautocorrelation.Autocorrelationcanbevisualizedonadataplot
whenagivenobservationismorelikelytolieaboveafittedlineifadjacentobservationsalsolieabovethe
fittedregressionline.Autocorrelationiscommonintimeseriesdatawhereadataseriesmayexperience
"inertia."[11]Ifadependentvariabletakesawhiletofullyabsorbashock.Spatialautocorrelationcanalso
occurgeographicareasarelikelytohavesimilarerrors.Autocorrelationmaybetheresultof
misspecificationsuchaschoosingthewrongfunctionalform.Inthesecases,correctingthespecificationis
onepossiblewaytodealwithautocorrelation.
Inthepresenceofnonsphericalerrors,thegeneralizedleastsquaresestimatorcanbeshowntobe
BLUE.[12]

Seealso
Independentandidenticallydistributedrandomvariables
Linearregression
Measurementuncertainty

Otherunbiasedstatistics
Bestlinearunbiasedprediction(BLUP)
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

6/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

Minimumvarianceunbiasedestimator(MVUE)

Notes
1.Aitken,A.C.(1935)."OnLeastSquaresandLinearCombinationsofObservations".ProceedingsoftheRoyal
SocietyofEdinburgh55:4248.
2.Huang,DavidS.(1970).RegressionandEconometricMethods.NewYork:JohnWiley&Sons.pp.127147.
ISBN0471417548.
3.Hayashi,Fumio(2000).Econometrics.PrincetonUniversityPress.p.13.ISBN0691010188.
4.Kennedy2003,p.110.
5.Hayashi,Fumio(2000).Econometrics.PrincetonUniversityPress.p.7.ISBN0691010188.
6.Johnston,John(1972).EconometricMethods(Seconded.).NewYork:McGrawHill.pp.267291.ISBN007
0326797.
7.Wooldridge,Jeffrey(2012).IntroductoryEconometrics(Fifthinternationaled.).SouthWestern.p.220.
ISBN9781111534394.
8.Johnston,John(1972).EconometricMethods(Seconded.).NewYork:McGrawHill.pp.159168.ISBN007
0326797.
9.Hayashi,Fumio(2000).Econometrics.PrincetonUniversityPress.p.10.ISBN0691010188.
10.Greene2012,p.23note.
11.Greene2010,p.22.
12.Kennedy2003,p.135.

References
Plackett,R.L.(1950)."SomeTheoremsinLeastSquares".Biometrika37(12):149157.
doi:10.1093/biomet/37.12.149.JSTOR2332158.MR36980.
Greene,WilliamH.(2012,7thed.)EconometricAnalysis,PrenticeHall.
UseofBLUEinphysics
L.Lyons,D.Gibaut,P.Clifford(1998)."Howtocombinecorrelatedestimatesofasinglephysicalquantity".
Nucl.Instr.andMeth.A270:110.
L.Lyons,A.J.Martin,D.H.Saxon(1990)."Onthedeterminationoftheblifetimebycombiningtheresultsof
differentexperiments".Phys.Rev.D41:982985.doi:10.1103/physrevd.41.982.

Externallinks
EarliestKnownUsesofSomeoftheWordsofMathematics:G(http://jeff560.tripod.com/g.html)
(briefhistoryandexplanationofthename)
ProofoftheGaussMarkovtheoremformultiplelinearregression(http://www.xycoon.com/ols1.htm)
(makesuseofmatrixalgebra)
AProofoftheGaussMarkovtheoremusinggeometry
(http://emlab.berkeley.edu/GMTheorem/index.html)
Retrievedfrom"https://en.wikipedia.org/w/index.php?title=GaussMarkov_theorem&oldid=712569967"
Categories: Statisticaltheorems

https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

7/8

4/7/2016

GaussMarkovtheoremWikipedia,thefreeencyclopedia

Thispagewaslastmodifiedon29March2016,at21:31.
TextisavailableundertheCreativeCommonsAttributionShareAlikeLicenseadditionaltermsmay
apply.Byusingthissite,youagreetotheTermsofUseandPrivacyPolicy.Wikipediaisa
registeredtrademarkoftheWikimediaFoundation,Inc.,anonprofitorganization.

https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

8/8