Professional Documents
Culture Documents
GaussMarkovtheoremWikipedia,thefreeencyclopedia
GaussMarkovtheorem
FromWikipedia,thefreeencyclopedia
Instatistics,theGaussMarkovtheorem,namedafterCarlFriedrichGaussandAndreyMarkov,states
thatinalinearregressionmodelinwhichtheerrorshaveexpectationzeroandareuncorrelatedandhave
equalvariances,thebestlinearunbiasedestimator(BLUE)ofthecoefficientsisgivenbytheordinary
leastsquares(OLS)estimator.Here"best"meansgivingthelowestvarianceoftheestimate,ascomparedto
otherunbiased,linearestimators.Theerrorsdonotneedtobenormal,nordotheyneedtobeindependent
andidenticallydistributed(onlyuncorrelatedwithmeanzeroandhomoscedasticwithfinitevariance).The
requirementthattheestimatorbeunbiasedcannotbedropped,sincebiasedestimatorsexistwithlower
variance.See,forexample,theJamesSteinestimator(whichalsodropslinearity)orridgeregression.
Contents
1
2
3
4
5
6
7
8
9
Statement
Proof
Remarksontheproof
Generalizedleastsquaresestimator
GaussMarkovtheoremasstatedineconometrics
5.1 Linearity
5.2 Strictexogeneity
5.3 Fullrank
5.4 Sphericalerrors
Seealso
6.1 Otherunbiasedstatistics
Notes
References
Externallinks
Statement
Supposewehaveinmatrixnotation,
expandingto,
where arenonrandombutunobservableparameters,
arenonrandomandobservable(calledthe
"explanatoryvariables"), arerandom,andso arerandom.Therandomvariables arecalledthe
"disturbance","noise"orsimply"error"(willbecontrastedwith"residual"laterinthearticleseeerrorsand
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
1/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
residualsinstatistics).Notethattoincludeaconstantinthemodelabove,onecanchoosetointroducethe
constantasavariable
withanewlyintroducedlastcolumnofXbeingunityi.e.,
for
all .
TheGaussMarkovassumptionsare
(i.e.,alldisturbanceshavethesamevariancethatis"homoscedasticity"),and
for
thatis,theerrortermsareuncorrelated.Alinearestimatorof
isalinearcombination
regardlessofthevaluesof
.Now,let
besomelinearcombinationofthecoefficients.Then
themeansquarederrorofthecorrespondingestimationis
i.e.,itistheexpectationofthesquareoftheweightedsum(acrossparameters)ofthedifferencesbetween
theestimatorsandthecorrespondingparameterstobeestimated.(Sinceweareconsideringthecasein
whichalltheparameterestimatesareunbiased,thismeansquarederroristhesameasthevarianceofthe
linearcombination.)Thebestlinearunbiasedestimator(BLUE)ofthevector ofparameters isone
withthesmallestmeansquarederrorforeveryvector oflinearcombinationparameters.Thisis
equivalenttotheconditionthat
isapositivesemidefinitematrixforeveryotherlinearunbiasedestimator .
Theordinaryleastsquaresestimator(OLS)isthefunction
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
2/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
)thatminimizesthesumofsquaresofresiduals
ThetheoremnowstatesthattheOLSestimatorisaBLUE.Themainideaoftheproofisthattheleast
squaresestimatorisuncorrelatedwitheverylinearunbiasedestimatorofzero,i.e.,witheverylinear
combination
whosecoefficientsdonotdependupontheunobservable butwhose
expectedvalueisalwayszero.
Proof
Let
beanotherlinearestimatorof andletCbegivenby
,whereDisa
nonzeromatrix.Aswe'rerestrictingtounbiasedestimators,minimummeansquarederrorimplies
minimumvariance.Thegoalisthereforetoshowthatsuchanestimatorhasavariancenosmallerthanthat
of ,theOLSestimator.
Theexpectationof is:
Therefore, isunbiasedifandonlyif
Thevarianceof is
SinceDD'isapositivesemidefinitematrix,
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
exceeds
byapositivesemidefinitematrix.
3/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
Remarksontheproof
Asithasbeenstatedbefore,theconditionof
linearunbiasedestimatorof
is
anotherlinearunbiasedestimatorof
Therefore,
isequivalenttothepropertythatthebest
(bestinthesensethatithasminimumvariance).Toseethis,let
.
Moreover,supposethattheequalityholds(
).Ithappensifandonlyif
Rememberingthat,fromtheproofabove,wehave
Thisprovesthattheequalityholdsifandonlyif
asaBLUE.
,then:
whichgivestheunicityoftheOLSestimator
Generalizedleastsquaresestimator
Thegeneralizedleastsquares(GLS),developedbyAitken,[1]extendstheGaussMarkovtheoremtothe
casewheretheerrorvectorhasanonscalarcovariancematrix.[2]TheAitkenestimatorisalsoaBLUE.
GaussMarkovtheoremasstatedineconometrics
InmosttreatmentsofOLS,theregressorsinthedesignmatrix areassumedtobefixedinrepeated
samples.Thisassumptionisconsideredinappropriateforapredominantlynonexperimentalsciencelike
econometrics.[3]Instead,theassumptionsoftheGaussMarkovtheoremarestatedconditionalon .
Linearity
Thedependentvariableisassumedtobealinearfunctionofthevariablesspecifiedinthemodel.The
specificationmustbelinearinitsparameters.Thisdoesnotmeanthattheremustbealinearrelationship
betweentheindependentanddependentvariables.Theindependentvariablescantakenonlinearformsas
longastheparametersarelinear.Theequation
qualifiesaslinearwhile
canbetransformedtobelinearbyreplacing byanotherparameter,say .Anequation
withaparameterdependentonanindependentvariabledoesnotqualifyaslinear,forexample
,where
isafunctionof .
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
4/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
Datatransformationsareoftenusedtoconvertanequationintoalinearform.Forexample,theCobb
Douglasfunctionoftenusedineconomicsisnonlinear:
Butitcanbeexpressedinlinearformbytakingthenaturallogarithmofbothsides:[4]
Thisassumptionalsocoversspecificationissues:assumingthattheproperfunctionalformhasbeen
selectedandtherearenoomittedvariables.
Strictexogeneity
Forall observations,theexpectationconditionalontheregressorsoftheerrortermiszero:[5]
where
isthedatavectorofregressorsfortheithobservation,and
consequently
isthedatamatrixordesignmatrix.
Geometrically,thisassumptionsimpliesthat
product(i.e.,theircrossmoment)iszero.
and areorthogonaltoeachother,sothattheirinner
Thisassumptionisviolatediftheexplanatoryvariablesarestochastic,forinstancewhentheyaremeasured
witherror,orareendogenous.[6]Endogeneitycanbetheresultofsimultaneity,wherecausalityflowsback
andforthbetweenboththedependentandindependentvariable.Instrumentalvariabletechniquesare
commonlyusedtoaddressthisproblem.
Fullrank
Thesampledatamatrix
Otherwise
mustbenonsingular,i.e.itmusthavefullrank.
isnotinvertibleandtheOLSestimatorcannotbecomputed.
Aviolationofthisassumptionisperfectmulticollinearity,i.e.someexplanatoryvariablesarelinearly
dependent.Onescenarioinwhichthiswilloccuriscalled"dummyvariabletrap,"whenabasedummy
variableisnotomittedresultinginperfectcorrelationbetweenthedummyvariablesandtheconstant
term.[7]
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
5/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
Multicollinearity(aslongasitisnot"perfect")canbepresentresultinginalessefficient,butstillunbiased
estimate.Theestimateswillbelesspreciseandhighlysensitivetoparticularsetsofdata.[8]
Multicollinearitycanbedetectedfromconditionnumberorthevarianceinflationfactor,amongothertests.
Sphericalerrors
Theouterproductoftheerrorvectormustbespherical.
Thisimpliestheerrortermhasuniformvariance(homoscedasticity)andnoserialdependence.[9]Ifthis
assumptionisviolated,OLSisstillunbiased,butinefficient.Theterm"sphericalerrors"willdescribethe
multivariatenormaldistribution:if
inthemultivariatenormaldensity,thenthe
equationf(x)=cistheformulaforaballcenteredatwithradiusinndimensionalspace.[10]
Heteroskedacityoccurswhentheamountoferroriscorrelatedwithanindependentvariable.Forexample,
inaregressiononfoodexpenditureandincome,theerroriscorrelatedwithincome.Lowincomepeople
generallyspendasimilaramountonfood,whilehighincomepeoplemayspendaverylargeamountoras
littleaslowincomepeoplespend.Heteroskedacitycanalsobecausedbychangesinmeasurement
practices.Forexample,asstatisticalofficesimprovetheirdata,measurementerrordecreases,sotheerror
termdeclinesovertime.
Thisassumptionisviolatedwhenthereisautocorrelation.Autocorrelationcanbevisualizedonadataplot
whenagivenobservationismorelikelytolieaboveafittedlineifadjacentobservationsalsolieabovethe
fittedregressionline.Autocorrelationiscommonintimeseriesdatawhereadataseriesmayexperience
"inertia."[11]Ifadependentvariabletakesawhiletofullyabsorbashock.Spatialautocorrelationcanalso
occurgeographicareasarelikelytohavesimilarerrors.Autocorrelationmaybetheresultof
misspecificationsuchaschoosingthewrongfunctionalform.Inthesecases,correctingthespecificationis
onepossiblewaytodealwithautocorrelation.
Inthepresenceofnonsphericalerrors,thegeneralizedleastsquaresestimatorcanbeshowntobe
BLUE.[12]
Seealso
Independentandidenticallydistributedrandomvariables
Linearregression
Measurementuncertainty
Otherunbiasedstatistics
Bestlinearunbiasedprediction(BLUP)
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
6/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
Minimumvarianceunbiasedestimator(MVUE)
Notes
1.Aitken,A.C.(1935)."OnLeastSquaresandLinearCombinationsofObservations".ProceedingsoftheRoyal
SocietyofEdinburgh55:4248.
2.Huang,DavidS.(1970).RegressionandEconometricMethods.NewYork:JohnWiley&Sons.pp.127147.
ISBN0471417548.
3.Hayashi,Fumio(2000).Econometrics.PrincetonUniversityPress.p.13.ISBN0691010188.
4.Kennedy2003,p.110.
5.Hayashi,Fumio(2000).Econometrics.PrincetonUniversityPress.p.7.ISBN0691010188.
6.Johnston,John(1972).EconometricMethods(Seconded.).NewYork:McGrawHill.pp.267291.ISBN007
0326797.
7.Wooldridge,Jeffrey(2012).IntroductoryEconometrics(Fifthinternationaled.).SouthWestern.p.220.
ISBN9781111534394.
8.Johnston,John(1972).EconometricMethods(Seconded.).NewYork:McGrawHill.pp.159168.ISBN007
0326797.
9.Hayashi,Fumio(2000).Econometrics.PrincetonUniversityPress.p.10.ISBN0691010188.
10.Greene2012,p.23note.
11.Greene2010,p.22.
12.Kennedy2003,p.135.
References
Plackett,R.L.(1950)."SomeTheoremsinLeastSquares".Biometrika37(12):149157.
doi:10.1093/biomet/37.12.149.JSTOR2332158.MR36980.
Greene,WilliamH.(2012,7thed.)EconometricAnalysis,PrenticeHall.
UseofBLUEinphysics
L.Lyons,D.Gibaut,P.Clifford(1998)."Howtocombinecorrelatedestimatesofasinglephysicalquantity".
Nucl.Instr.andMeth.A270:110.
L.Lyons,A.J.Martin,D.H.Saxon(1990)."Onthedeterminationoftheblifetimebycombiningtheresultsof
differentexperiments".Phys.Rev.D41:982985.doi:10.1103/physrevd.41.982.
Externallinks
EarliestKnownUsesofSomeoftheWordsofMathematics:G(http://jeff560.tripod.com/g.html)
(briefhistoryandexplanationofthename)
ProofoftheGaussMarkovtheoremformultiplelinearregression(http://www.xycoon.com/ols1.htm)
(makesuseofmatrixalgebra)
AProofoftheGaussMarkovtheoremusinggeometry
(http://emlab.berkeley.edu/GMTheorem/index.html)
Retrievedfrom"https://en.wikipedia.org/w/index.php?title=GaussMarkov_theorem&oldid=712569967"
Categories: Statisticaltheorems
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
7/8
4/7/2016
GaussMarkovtheoremWikipedia,thefreeencyclopedia
Thispagewaslastmodifiedon29March2016,at21:31.
TextisavailableundertheCreativeCommonsAttributionShareAlikeLicenseadditionaltermsmay
apply.Byusingthissite,youagreetotheTermsofUseandPrivacyPolicy.Wikipediaisa
registeredtrademarkoftheWikimediaFoundation,Inc.,anonprofitorganization.
https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
8/8