4. Introduchon Agronomy : yivlds , Groot of seeds ete. Inspi eee inaphedl yi 14 eebliae (cl age! oA 4 paket Hee ee. Coorong + jabest rak, exchaae rake, de padent price of howses et. varies. of inhest. (X,Y) gb pectic Y from X, . binarg €& £0, 43 YE IR, X omght be ee Erte Veber IR? cate F Ayn, Ho Fun, Of 4d. Regreain Seb—up ys fxs pee cep fivn € Fandom vaciasle , thy scslar, E(E) =O. expeclakim value of E13 ave.” Say that = IR here. Asume that E(e*) <2, The oliste'bubion of E might be Bx (mak of He hme.) oc depads on X, Very itn at will aeume Var Ce} = © (net depraved ing on X) + thomoscedas bsity) ENB: if Var fe) depeals on X= [hebratcedash'eify] | Another ery important aapech is Lo What awe ill Cousn hee “is te care tihee 2 ives in a giver Binike ~clinersionl Vector space J of real-valued fenchins of x.42. Examples of Matar models Example 4: ( One- cay Anais of Variance + ANOVA). let X be a categorical varable, avith aehues in Sita, L3 The set of alk real value funchins on Urs an Ledimasinal Vector space. Ik cerresponds to IR“, whee ong idnbifier PEF ith (a) Or Example 2: (Simple Iinege re grécoron. ) LeL X be a numerical covariasle, that is X EIR. Assuming Yat Hee isa Cinear rebkenship berweon % ond Y, ce nok. y $Gd=athx a ee DX Biample 3 (Peymonial Myre) y fe) = A Kea 3 xe fi 2X xe mel sj £(x) =i hay et Ge |; here 00,447.,44 EIR| Example 4+ ( Simpk Analysis of Covaniance ) Assume Bat X consists of a cabeyocica! varrable CEF4,.., LJ mrt Pf a numerical vaciggie WEIR» A posible model OF avoull consists of fanchon £ of tke form f£(e0) = ale) +be Example 5° (Maltiple bacac Regression. ) Assume that X = (XGWjenng 8 & Leche of of nermerical varvables, A Simple racetel Por He relahonshp behweon X and ¥ Polos Yeas S wy) xl) +e TNB: In prachce, Mulhiple Linear Regeession Models abr oblea include more Complex Predictes su as X()*, xGIX(@) , ete. ad 2. Parameter Eskmabion Here we consider both problems of echmakng £ and ot Co (6 gs Be sherolard cleyekon of Ha error —s Y-f(x)) Ne start from on lata pairs (Xa Va) ,.+, (Xa, MW), hee Kos 4X2) +E, , with stochashica ly inde prndenk chbhed errors Eqpney En. 2.4. Vecher and Madvix Re presentahons: ahs Ya ¥ = | ; / a Yo Ha yamAs Por Fue oleaok. basis Purctons by fai, Pp. Hence 2 can be rite, ao, Pe {oa = Behl) eer S@=({: |. = (3) In addihon, the basis furchons and the observed/choren values of He covariables (X:) olliver He Design madwix" D, fe (Xa) fa (ka) Pp (Xa) ‘ep D= ( fale) kewl 2. fee) | ER” 7 a Ba (Xn) Pen) 2. Ep (td Then our (Crear) regresiian model can be vritten in makix -Vecher mm y= DE+E Tye. with ©: (Xgl ana he convaskin g (2) = (gin foc g: SOR. We can aloo erie Del fC R),.. f(0)] and pS = £(*) 4 2.2. Eskmahon of ©: Ue assume fare (and in the following) Hat nx p nd Bat eo matein D0 Rar fill columa rank, ive. rank (D) =p. Te sum up, rank(d)=p say (3) It means Bet the colums of D ore Crnearly indepmdent. In offer voce , WIPER? \ fo} dy xO, Bab M4is O< Waihi VOD , which meee Hat [DTD is positive definite] A vector Bete? is called leaspequares esiimatoe (Ls) eskicabee FS chy: —~ Ty pelts yea Wye ome. ] In other cxoedls, one choses SEM? in such acy Hat He sum of fapaaet Bie -£6 £14) )* be minimal. The Punchin = Ee, By is Hen Cask squares eshmator of £. Lemma 4: HH! Colume rant Unoler_ (3), a! Ls. ina ay tices es DT D)# oy eof: First Proofs Let us oleBne Q: Tek ’—> Q(H)I=M¥-o7I. Now , let us concider VE IR? and expand G(R VI = N= + bat - pi Sanae U7 = D7 + worn -2<¥- br PY. OR) 9(%) - 2 (Y- oJ ov) + CYDTDV. F ef Steg From Bis, oe icleabity He “a and second orrler ober a Vahives of Q + VQ(W) =-2D (Y-DW). & geadnd (rect) VP (7) = yD @ Hessian (mater) (pe tite phen (rank (0) =p) , DTD is positive clehinie | oll | 02 it clocon't don el). cept ce [s = -Qe)= NF-L. So Q is sticth convex. Since roblem ys sively Convex and diffrentable (over ROS} fay 4 unique solution, arhich is He erthcal pant, 1. seluhan to [Vet) =e] 7(8) =-20°(¥- ~08) = F So we gob ms HS ry? (unoemal equates") “aes TEs Cire) aaF, a. pee Cae Paak(O) 2p Second peook’ peat (34 | deglac = \Nantpasia, cfocan'! nail. act ) = Wea te pe u =n 2¥d%- + ona nr my = AL =F 2eoropro ya oY + Wp Oe te =IVIP + (2- ory ATP) B'D(7- (0D) 207, = PT oT IPD (Ort tH 2 (RBI (d"D) (LS) esomehg mt epi 0 ro So He unigue misimaer of Q & give by Ku8 i Geometric inleprttahon: Led us define Ui=f oe, Vers =f linear Pear of We columns of D.3 =span( fal ),..., fp (F)). Remenber that SF minimizes Fei? — N¥~ OIF, In Per avowls DE pelaiges Re sah att be. He Puncton ve M > IP V-7Il?. @! citer unvole + ¥ coercivity hk ¥ db erika port .MO Fbreny PSN 4c spans Day, De}. So DE i Re odlhepenal projeckor of Vonto Ht, Mole: crotler charackezahim of He osthegonal prejechin of 4 Vector J oaro a Paite- dimensional spacey Soy gevekd vectors Duy... Dp is that V-R(Y) LD Mek... p}. Meaning, KFTOR, D> SO Vi € f4,.-,pF. meaning (YO) =e Pa (TO) OK Remark Th implementahions, it's offen preferable te evar dlircehly inverhng DTD ancl relying on ynaive’ mate's [vector procluct . Inskad , Pachersakos of DTD can be Used ,etc. Remark / Nokabon: me Reminder: V= PO +E. Fane = pd =(o(oT)-* Dy he ne (carne ee Wal eT "= Hf Sed values va=Hyr sith {He D(aTD) “2D IL , hab maki [FOne _clinersioal models Cexamples of & =(0TD)*0°F J Let us fist consider the model |ysere. Anume Haat we observe Ys O+E; (vith Ei's ae usual) dscsn. 4 | Heat (kee p=4 ond Hx) = 1) a cf us slightly generalize by eM oer V2 Ofalx) +2) ae ap one fal) “ee b= ( i] a (Wa) ND (xe = WEP ry s 2 (XY =<$4¥) Fd sola. Zhivy: _ FZ Lyx} Hal?) Ie+ Analysis _of Variance CAN@VA}-exanele Let us reformulate the model: ¥ = Ze, Ayxayy +E. [WLOG, ue take as pang ice f4....277 bea. 7 C8] Let us dence Ye = 2a A Ag nag Eo) and areume Aca latalaeadel | +All Eke! "Ys have been ovcled an Sach A way Pet ve Rad Xe fist , tin Xv = 2 eke. te p= [ shee alist fXe Hj, TE Sn}. she pee SHEEN [inca O° <02,D2> = = 2 (xi-K)=nK-9X =o. Ve hence obbain 4 now doniyn omatber'x ot Ly tyne S=[Ds, 0, ] {3 RE :] deshell alist Chae 2 M-X oethegonal + 7 The corresponchiy model is Y= 4+D(X-X)+E, Hat is, ve have replaced falx)= 2, 4ibo=x by te mtn bans tunchons falx)e-$, (ele 1, 2 = x-&. As Ys aibXt€ , ae can sdoh’ [C=T) oma [a= 8-TR = & a3 | ee now at (wang Hester approach) : th (2) =(3%5) 87 Now n \ BvD end C O O Z-X]" wo» [Eh yet ZYi(X:-K) JAA =a,4 NB: a <0 ee A de> De 4 24 “Gram= Simi > Dy = Di a So Da 2 = ZX, MDall” =£4- =n Bis Da = KA. = = 4 $oRemark: k Hed yregrearion Punchion" $ Can be toritt, ao fe ¥+ 8 (x-¥ bo tet E(Z)= IH arnet toon previns In patheular , Ge Sons icesshions tl) hizo between Cote pia i) oN 9 cet Se > CER) amet Y CSG x-¥) a YS sey) (COS Bete ed amin of ee Back fo ANCOVA model: ro “t xs (GW 7 Ceft.. i ; ver Medel: Yo a(c)tbwte. Goal: eahimake Re paramere (;'s, b) bersech on He Veckor 08 observahims Y= The cbargn -matex fag Ltd cobtemns ? Dstt \e ps pS iC Apessit dog AEE dod T= (We) Ao stated eariem (ANOVAImede!], He L hiret columns are paicvise otegenal. Foc He lack colgmn ; otogonalizahor ail helps Foc Ve(Vii, aad J €F4,..,L}, ctebine + i A Vj) = a7 EM, hee of HE Gn. eG Wa- Wels) \ Aaa ret Then, Re vector Wis W + WG)) = fer) iS ot}beg gyal Go Da, De. ogo f3So ue jeformulate He ANCOVA model as v=ele)+ blw-wlel)+e 5 Bab. fe Noting D=[Ds,.., Du eP ase bbAmtal: Elen \ nla) O yt ah fr Fe -( ana po 7 YO Oe FB | \ / ZN Yi / ¥ (Ca) \ Ren) | F(C.) Het E (wma | A isd | C=b j Zui ie)" / \ a(ce) = Stes =f la) = yc) -0 Wlci)2.2.3 The cecMiciant o eksoaia ont Comparing He Vector Billed alues P= (Kyng ) ee Sbserved res Poses YAEL ome chars te so caled residuals Yi 7 | A commonly tered rearure, Hat quanhfies He quality of He approximation of Vo by Ys We cocfhciuk of hint In B®, one Compares the pearddual Le ob squares Zh. yt exit the tolal sain moot squares = | x;-K)* Ries oben veered e as He dct, of variance” of Y feat can be (brace ely) explaraed fy He covariance X, In mosh Cases OS RS 4. Fer cur Res 4,by clefiaikon. R220 systemabially In cases thee the conchamb i included in He bass siuead (aen). Incleeof in Such cose, yy ean bat fal vector Lonm) a ERAS 2 ¥(E)2 od = tes f \ Coe) F- Senmelanee_ mat Reminder: Assuming Lull columae—rank for He oleoign matsix O, Ave egtablished previously Hat: = e = (ory) *D Y The previous Aosumphions on & ( E(E)=F Vae(E) =» o*9.) Baok He following Consequinces on He L§=Cokmatoc : Lemma 5: Aaaceming Pat HE)=0 onc Va (EZ) =074 ,a. above, ape Rave E(@) =e (unbiasecdness . ) Var) =o*( 0D)“ fteok First let uo pecall that: Vr De +E ( by anrumph'or. ) Solel= (D7) * pF = (oT OY *d" ( pe +z) = (OYD™ ES +O ot DZ. =f& (DY *D"2] (x) 6+ee EE Cause ips not random, > (é)- @ Go 2) =e@+00'*T 42) -e& Ee Regarding He variability of ple LS-echmater , By (4) , ce have : Var (8) =, Var (oro vie) Recher foe Via random vector anth covariance makeix 2 Grol A & (delermiaicht) makin | sui adequate. dimesions , Var( AV) =AZ AT) => Vac) =(07D)"*DT) (0%) ( (oro *D™)™ Pn ah a = o> (DD) DTD (97D) Tee ee ae a= eal | =o* (0D) Examole ( Simple Linear Regeescion ) VeathX4e = F+eb(X-K) +E, | We established Hat He Ls-eshmator for ay Art > (E)> KEY | thee XK =Then, foc any fixed point x EIR, Flx) ls predicted by R(x) =A sbx = &+ 3 (xx). By Aanbiasedaess of He LS-echmator (and Linearity of E): E( $l) = atbx =f). Furthermore , t Var( 4(x)) = Vael &) + ( x =x)? Var(b) (40° since Conds aa ( ee ee ) Wk Vel £60) mana for X2K (and Varl H2l}2& ) (en) As for Cov (4,8) ) we have Lt Cov(a,b) = Cov( &-BX, £) = Cov(&,8)- Var( 2) =o -x ot RIM Beample Simple Analysis PR Covariance . Model? Y= alC) +bW+e = ¥(C) +blw-We)+e. fe eck ea FE ar yu Pe ) mae ere Ath Covavarae Mabe: ot Mo. ° oO a 7 nll) > A © The LS eobimabor for (25) is é-| bo— From Bere, se can real: cota’ (a “ati And Cov (&(¢, 8) =-0" ae | Se Var (2l)-a(i)) = Var (V(«)- U)+0( W0)-M))) aga 4 (MOOG =o fats + 56) ia} + The i* Component of B, Yi- Ye, called He i* residual. Ceomeleccally speaking Y" is “Whe Wiheyon! projechor of Vonks d ahead: senha M, hile ? <5 He ocths gona) prejechion of FV cate fhe orthogonal complemek Mt of M, The LS-eshmatoc e on F solely Pheeugh ¥ as 2k othe geal 4M, eter i @guivalent a. weco, So. that & = (0 4pT7 = (oT Dd) tT P. On fe olbes hand, 2 ce ae on @ since. (4-H)O=O. Se eis r aed hgLet ws sseme Bab Stun, are cerved «nh variance ot< ee. 1h ae red know EC , WENO = A z Ft wuld be q nalurgl eotimator for o%. Kroring tHe grejechin of © orto M* ashead, ss egibinele fo fink ef the eohmator : ma HEN _ vet ou din(ar4) = np Proposition 6: E(S*) 20%. Aasuming dyctter that Ble") S Kort dor some K7O, then ue_Aave : Var( 2) € (k-3) +2 Ft me P r (K-38, if K-3 200 i k-3)4 5% | Soke = max$ o, K-34 Pec: Sreeneeitti We know thapye =(4-H)E = HE with His a-H. So Welit= MAT HE = ETHE > Z Ay -Cov(Ei,€j) api tal S le aed Nou Cov (€i,€;) bo adel ctt cit So WEI = 2 Hi cot = 0% Eta} +-(H) Nos “te(A=H) = tr(4l)-HH) = a-p ae trl H) =P, (see Noh since H teprambs aa otthegons| projechin mihi onto M, a linear supspace of IR", of chimesion p. So +e Badd: E( W2*) = (n-po* lex a