REGRESSION ANALYSIS - Lecture

Multiple linear regression model with K explanatory variables:
Y
t
= |
0
+ |
1
X
1t
+ |
2
X
2t
+ + |
K
X
Kt
+ c
t
(t = 1,2, ,n)

Deterministic component: (complex), K explanatory variables
c
t
: stochastic / random component

REGRESSION ANALYSIS
- MULTIPLE REGRESSION MODEL -
Basics of Regression Analysis
OLS: choose s that minimises sum of squared residuals

coefficients: AY associated with one-unit | in X, ceteris paribus
|
1
: impact on Y of one-unit increase in X
1
, ceteris paribus

|
K
: impact on Y of one-unit increase in X
K
, ceteris paribus
|
0
: value of Y when all the Xs and the error term are zero

REGRESSION ANALYSIS
- MULTIPLE REGRESSION MODEL -
=

=

=
| | | | = =
n
1 t
n
1 t
n
1 t
2
)
Kt
X
K
t 2
X
2
t 1
X
1
t
Y (
2
)
t
Y
t
Y (
2
t
e
EVALUATING THE GOODNESS OF FIT

- STATISTICAL CRITERIA -

Standard error of the estimate (SEE)
SEE = based on s
2
, estimate of true variance (o
2
)
measure of quality of fit for overall equation

n = #observations in series
K = #parameters estimated, including constant,
for , K = 4

Rather: R
2
and F-test used to judge goodness of fit of regression

K n
e
s SEE
2
t
= =

t 3 3 t 2 2 t 1 1 0 t
X
X

Y
| + | + | + | =
DEMONSTRATION : SALES FUNCTION
REGRESSION OUTPUT (Dependent variable: Sales Volume)


Std errors and t-statistics
indication of significance of parameters in function.
each different data set different estimates of the |s
how do estimated coefficients vary depending on the sample?
measure of variability = standard error of estimated coefficients =
estimate of square root of variance of distribution of

the larger the standard error the more the estimates will vary
standard error of the slope coefficient of is given by:

(t=1,2,,n).

s

2
t
) X X (
SEE
)
( SE
E
= |


Std errors and t-statistics
t-statistic (much easier to evaluate):

(k=1,2,,K)

Rule of thumb:
If , t-value , of coefficient |
i
> 1.96: variable i significant
If , t-value , of coefficient |
i
< 1.96: drop variable i

OR use t-table to derive critical value to compare t-statistic to
Reject H
0
(where H
0
: |
i
= 0) if |t
i
| > t
crit

)
( SE
t
k
k
k
|
|
=
|


and

R
2
= proportion of total variance explained by the regression, measure of the
explanatory power of the regression (0 < R
2
< 1 )

R
2
= 0.9: 90% of the variance in dependent variable explained by variation in
independent variable(s)

Each time a variable is added |R
2
: rather use , i.e. R
2
adjusted for
degrees of freedom:

=
2
t
2
t
2
) Y Y (
e
1 R

=
) 1 n /( ) Y Y (
) 1 K n /( e
1 R
2
t
2
t
2
2
R
2
R
2
R

close to one: much variation in dependent variable explained

Rule of thumb:
> 0.8: acceptable for time-series data, lower value acceptable for
cross-section data
2
R
2
R


The F-statistic
test overall significance of equation (joint significance)

Rule of thumb:
If F > 4: coefficients are jointly significant

OR use F table to derive critical value to compare F-statistic to
Reject H
0
(where H
0
: |
1
= |
2
= = |
K
= 0) if F > F
crit

=
) 1 K n /( e
K / ) Y Y
(
F
2
i
2
i


Durbin-Watson statistic (DW) (1)
tests for the existence of first order serial correlation
first order serial correlation will occur when e
t
's are correlated with each
other

Rule of thumb:
0<DW<4, DW ~ 2 (1.5<DW<2.5): no serial correlation
Values closer to 0: positive serial correlation
Values close to 4: negative serial correlation

OR use DW table to derive critical values to compare DW statistic to.

=
=

=
n
1 t
2
t
n
2 t
2
1 t t
e
) e e (
DW

Durbin-Watson statistic (DW) (2)
Critical values indicating the upper (d
U
) and lower (d
L
) values for
combinations of n (number of observations) and K' (number of
independent variables excluding the constant) may be read off in the
DW table.
6 possible results for the Durbin-Watson statistic:

VALUE OF DW RESULT
(4-dL) < DW < 4 Negat ive serial correlat ion
(4-dU) < DW < (4-dL) Result undet ermined
2 < DW < (4-dU) No serial correlat ion
dU < DW < 2 No serial correlat ion
dL < DW < dU Result undet ermined
0 < DW < dL Posit ive serial correlat ion

STATIONARITY VERSUS
NON-STATIONARITY
Stationary series: mean-reverting and constant variance

DGP (data-generating process): AR(1) y
t
= y
t-1
+ v
t
v ~IID(0,o
2
)
y will be:
Weakly stationary where , , < 1
Non-stationary where , , = 1
Explosive where , , > 1

Often we do not consider the latter a possible d.g.p.

Stationarity
STATIONARITY VERSUS
NON-STATIONARITY

-3

-2

-1

0

1

2

10

20

30

40

50

60

70

80

90

y = 0.6*y(-1) + v
Stationarity
STATIONARITY VERSUS
NON-STATIONARITY

0

2

4

6

8

10

20

30

40

50

60

70

80

90

y = y(-1) + v
Stationarity
STATIONARITY VERSUS
NON-STATIONARITY

-2000

0

2000

4000

6000

8000

10

20

30

40

50

60

70

80

90

y = 1.1y(-1) + v
Stationarity
STATIONARITY VERSUS
NON-STATIONARITY
Proof:
y
t
= y
t-1
+ v
t

y
0

y
1
= y
0
+ v
1

y
2
= y
1
+ v
2

= (y
0
+ v
1
)+ v
2

=
2
y
0
+ v
1
+ v
2

y
3
= y
2
+ v
3

= (
2
y
0
+v
1
+ v
2
) + v
3

=
3
y
0
+
2
v
1
+ v
2
+ v
3

Stationarity
STATIONARITY VERSUS
NON-STATIONARITY

therefore

and

=

+ =
2
0
3 0
3
3
i
i
i
y y
=

+ =
1
0
0
t
i
i t
i t
t
y y
Stationarity
STATIONARITY VERSUS
NON-STATIONARITY
Conclusion: What are the properties of stationary data?

E(y
t
) = E(y
t+h
) = < , i.e. constant finite mean over time
E(y
2
t
) = E(y
2
t+h
) = , i.e. constant finite variance over time
E(y
t
y
t-j
) = E(y
t+h
y
t+h-j
) =
ij
< , i.e. constant finite covariance over
time

2
o
2
o
Stationarity
Properties of stationary vs. non-stationary data
Stationary Non-stationary

Variance Finite Unbounded, grows to
to
2
Memory Temporary Permanent
Expected time
between crossings
of y
Finite
(mean reverting)
Infinite (mean is
changing over time)
STATIONARITY VERSUS
NON-STATIONARITY
Stationarity
STATIONARITY VERSUS
NON-STATIONARITY
How are data series transformed?
Stationary variables are obtained by differencing (not necessarily
only once) the non-stationary series. Such a DGP is referred to as
difference stationary.

y
t
= y
t-1
+ v
t
, , = 1
Ay
t
= y
t
y
t-1
= v
t

y
t
~ I(1)

Stationarity
General:
A variable is called I(p) if it must be differenced p times in order to render
it stationary.

We can also say the series contains p unit roots.

For y
t
=y
t-1
+u
t
with u
t
=white noise
If ||<1 then y
t
is I(0)
If ||=1 then y
t
is I(1)

STATIONARITY VERSUS
NON-STATIONARITY
Stationarity
TRANSFORMING A SERIES I(1)

-2

0

2

4

6

8

10

20

30

40

50

60

70

80

90

y

A
y

Stationarity
0
20
40
60
80
100
120
140
70 75 80 85 90 95 00
CPI
TRANSFORMING A SERIES I(2)

Stationarity
0.00
0.04
0.08
0.12
0.16
0.20
70 75 80 85 90 95 00
D_LCPI
-0.04
-0.02
0.00
0.02
0.04
0.06
70 75 80 85 90 95 00
DD_LCPI
y
y
y
CONSEQUENCES OF NON-STATIONARITY
Non-stationary data series could result in spurious regression
Example:
Two non-stationary data series (y and x)
By construction they have nothing in common
y
t
= o + y
t-1
+ v
t

x
t
= + x
t-1
+ u
t

Stationarity
) , 0 ( ~
2
v t
IID v o
) , 0 ( ~
2
u t
IID u o
Variable Coefficient Std. Error t-Statistic Prob.
X 5.494891 0.091884 59.80278 0
R-squared 0.895314 Mean dependent var 51.97351
Adjusted R-squared 0.895314 S.D. dependent var 29.73505
S.E. of regression 9.620824 Akaike info criterion 7.376207
Sum squared resid 8700.663 Schwarz criterion 7.40309
Log likelihood -349.3698 Durbin-Watson stat 0.059368
Included observations: 95
Dependent Variable: Y
Method: Least Squares
Sample: 1901 1995
Stationarity
-30
-20
-10
0
10
20
-20
0
20
40
60
80
100
120
10 20 30 40 50 60 70 80 90
Residual Actual Fitted
Stationarity
Stationarity
Test Test Statistic p-value Conclusion
Jarque-Bera 2.4383 [0.2955] Errors are normally
distributed
ARCH LM 78.8189 [0.0000] Heteroscedasticity
White 30.8526 [0.0000] Heteroscedasticity
Breush-Godfrey 89.5272 [0.0000] Serial Correlation
Lung-Box -0.35 [0.0000] Serial Correlation
Ramsey Reset 9.1089 [0.0105] Misspecification

This non-stationarity of the residuals provides us with important
information concerning the relationship between y
t
and x
t
.
Effectively, where residuals are:
Non-stationary: no causal relationship
Stationary: some evidence that suggests causal relationship
Stationarity

Given the problems associated with non-stationary data series, it is of the
utmost importance to identify the univariate properties of the data
being employed in regression analysis.

Stationarity
Purpose of testing for unit roots:

To establish how many times a variable must be differenced to
render it stationary
Want to avoid spurious regression results (falsely implying
meaningful causal relationship) by testing the residual of the long-
run equation for a unit root

Unit root testing
UNIT ROOT TESTING
Tests available in EViews:
Dickey-Fuller (DF)
Augmented Dickey-Fuller (ADF)
Phillips Perron (PP)

Null hypothesis for all the above tests:
H
0:
= 1 (series contains unit root)

Unit root testing
UNIT ROOT TESTING
Testing for presence of unit roots not straightforward:

Underlying (unknown) d.g.p. may include a time trend
D.g.p. may contain MA terms in addition to being a simple AR
process
Small sample bias: Standard tests for unit roots biased towards
accepting null of non-stationarity (low power of test)
(conclude I(1) when I(0))
Undetected structural break may cause under-rejecting of the null
(too often conclude I(1))
Quarterly data may be tested for seasonal unit roots in addition
Unit root testing
UNIT ROOT TESTING
Example of non-stationary series y
t
=y
t-1
+ u
t
, u
t
~IN(0,1)

Source: Harris 1995:16
STATIONARITY
Long-run models and short-run effects
Example of stationary series y
t
=0.9y
t-1
+ u
t
, u
t
~IN(0,1)

STATIONARITY
Example: Consider a random walk process,
y
t
=y
t-1
+u
t
with u
t
=white noise:
It is difference stationary, since the difference is stationary:
y
t
-y
t-1
= (1-L)y
t
= u
t
It is also integrated, denoted I(d) where d is the order of integration.
Order of integration = number of unit roots contained in series / number
of differencing operations necessary to render the series stationary
y
t
=I(1) above

STATIONARITY
Presence of a stochastic trend (non-stationary) vs. deterministic trend
(stationary) complicates unit root testing

DIFFERENCE-STATIONARITY vs. TREND-
STATIONARARITY
COINTEGRATION THE CONCEPT
An econometric concept which mimics the existence of a long-run
equilibrium among economic time series
A drunk and her dog
COINTEGRATION THE CONCEPT
Data series, although non-stationary, can be combined (linear
combination) into a single series which is itself stationary
+
Say y
t
and x
t
:
must be differenced d times to become stationary
it contains d unit roots I(d)

In general, linear combinations of y
t
and x
t
will also be I(d)

If a vector | exists, such that the disturbance term from regression u
t
=y
t
-x
t

is of a lower order of integration, i.e.
u
t
~I(d-b), b>0,
then x
t
and y
t
are cointegrated of order (d,b)
COINTEGRATION
Example
y
t
~I(1); x
t
~I(1):
If u
t
~I(0), then two series are cointegrated of order CI(1,1)
COINTEGRATION
Example: A cointegrated stable (equilibrium) relationship possibly exist
between m
t
and p
t
(|=1.1085; t=41.92)

COINTEGRATION
COINTEGRATION
Given the following linear relationship
y
t
= |x
t
+ e
t

y
t
and x
t
will be cointegrated only if the residuals,
e
t
= y
t
- |x
t
are stationary or I(0)
Consequently, in the relationship between two variables, both must be
integrated of the same order (and generally, I(1)) for the residuals to be
I(0)
If the number of variables involved in the relationship increases, the
problem becomes more complicated
COINTEGRATION
Consider, for example, a three variable case
y
t
= |
1
x
1t
+ |
2
x
2t
+ e
t

Now it is possible for the variables to be integrated of different orders
and for the error term, to be stationary (e
t
~ I(0))
Suppose that y
t
~ I(0), x
1t
~ I(1) and x
2t
~ I(1)
May suspect that e
t
~ I(1)
However, there could exist a cointegrating vector [|
1,
|
2
] such that
(|
1
x
1t
+ |
2
x
2t
) ~ I(0)
If this is the case, e
t
will be stationary, since y
t
~ I(0) and also
(|
1
x
1t
+ |
2
x
2t
) ~ I(0).

COINTEGRATION
This is a major complication of the entire concept of cointegration in a
long-run relationship and in the stationarity of the error-term
There is a practical hint that can simplify testing procedures
If variables in a long-run relationship are of a different orders of
integration and the order of integration of the dependent variable is
lower than the highest order of integration of the explanatory
variables, there must be at least two explanatory variables
integrated of this highest order if the necessary condition for
stationarity of the error term is to be met
COINTEGRATION
For example
w
t
is I(1) x
t
is I(1) y
t
is I(2) z
t
is I(2)
e
t
may be I(1)
linear combination
c
t
= |
1
w
t
+ |
2
x
t
+ |
3
(o
1
y
t
+ o
2
z
t
)

linear combination
e
t
= o
1
y
t
+ o
1
z
t
c
t
may be I(0)
Regression Model: Y
t
= o + |X
t
+ c
t

Case 1:
Both Y
t
and X
t
are stationary: classical OLS is valid
Case 2:
Y
t
and X
t
are integrated of different orders: regression = meaningless
Case 3:
Y
t
and X
t
integrated of same order, residuals non-stationary:
SPURIOUS REGRESSION PROBLEM
Case 4:
Y
t
and X
t
integrated of same order, residuals stationary:
COINTEGRATION
COINTEGRATION
Cointegration implies:
Two or more series are linked to form an equilibrium relationship
spanning the long run (even if series contain stochastic trends/are
non-stationary).
Series move closely together over time
Difference between them is stable (stationary)

Cointegration mimics the existence of LR equilibrium towards an
economic system converging over time
u
t
may be called disequilibrium error (distance that system is away
from equilibrium at time t)

COINTEGRATION:
ECONOMIC INTERPRETATION
DEMONSTRATION: COINTEGRATION
Estimate demand function for skilled labour in SA:
Long-run equation: LNS = f(LGDPFAC, LWSCPR)

Cointegration test:

-.08
-.04
.00
.04
.08
.12
1970 1975 1980 1985 1990 1995 2000
RES_COINT
Cointegration test:

10%: C(10) = -3.6435
5%: C(5) = -4.0083
1%: C(1) = -4.7586

-2
2
-1
1
T T C(p) | | | + + =

3 CATEGORIES IN EVIEWS:
Coefficient tests
Residual tests
Histogram-Normality Test
Correlogram Q-statistics
Serial Correlation Test
ARCH LM Test
White Heteroscedasticity (without and with cross terms)
Stability tests
Ramsey RESET
Recursive estimates

DIAGNOSTIC TESTING
RESIDUAL TESTS
- NORMALITY -
JARQUE-BERA test statistic:
Tests whether variable is normally distributed
Measures the difference of the skewness and kurtosis of a
series from those of a normal distribution

with S = skewness, K = kurtosis, k = # estimated coeff

H
0
: residuals are normally distributed

Reject H
0
if JB > or if ps0.05
) 2 ( ~ ) (
2
4
) 3 ( 2
6
2
_
+ =
K
k N
S JB
991 . 5 ) 2 (
2
= _
RESIDUAL TESTS
- NORMALITY -
0
1
2
3
4
5
6
7
8
9
-0.04 -0.02 0.00 0.02
Series: Residuals
Sample 1972 2002
Observations 31
Mean 0.001652
Median -0.000818
Maximum 0.027283
Minimum -0.043476
Std. Dev. 0.015517
Skewness -0.386711
Kurtosis 3.639949
Jarque-Bera 1.301634
Probability 0.521620
RESIDUAL TESTS
- SERIAL CORRELATION -
LUNG-BOX Q-statistics:
Tests for assumption that residuals contain no auto
correlation up to order k (residual term is white noise)

with T
j
the j-th serial correlation; T = # obs

H
0
: No autocorrelation up to order k

Reject H
0
if Q > or if ps0.05

Practical problem: Choosing the order of lag
) ( ~ ) 2 (
2
1
2
k T T Q
k
j
J T
T
LB
j
_
=

+ =
) (
2
k _
RESIDUAL TESTS

RESIDUAL TESTS
BREUSCH-GODFREY LM-statistic:
Belongs to class of asymptotic (large sample), Lagrange Multiplier
(LM) tests
Can detect higher order serial correlation
Applicable whether lagged dependent variables included or not
(unlike DW)
Test stat based on auxiliary regression (of y
t
=|x
t
+u
t
)

e
t
= x
t
+oe
t-1
+ oe
t-2
++ oe
t-p
+v
t

H
0
: No autocorrelation up to order p

Reject H
0
if or if ps0.05
) ( ~ nR
2 2
p _
) ( nR
2 2
p _ >
RESIDUAL TESTS

RESIDUAL TESTS
- HETEROSKEDASTICITY -
ENGLES ARCH LM-test:
Belongs to class of asymptotic (large sample), Lagrange Multiplier
(LM) tests
This specification of heteroskedasticity is motivated by the
observation that for many finite time series, the magnitude of the
residuals appears to be related to the magnitude of recent residuals
Test stat based on auxiliary regression

e
2
t
= |
0
+|
1
e
2
t-1
+|
2
e
2
t-2
++|
q
e
2
t-q
+v
t
nR
2
~_
2
(q)

H
0
: No ARCH up to order q in residuals

Reject H
0
if nR
2
> _
2
(q) or if ps0.05

RESIDUAL TESTS

RESIDUAL TESTS
WHITES HETEROSKEDASTICITY LM-test:
Belongs to class of asymptotic (large sample), Lagrange
Multiplier (LM) tests
Test stat based on auxiliary regression (of say
y
t
=b
1
+b
2
x
t
+b
3
z
t
+u
t
):
e
2
t
= o
0
+ o
1
x
t
+o
2
z
t
+ o
3
x
2
t
+ o
4
z
2
t
+ o
5
x
t
z
t
+v
t

nR
2
~_
2
(# slope coeffs in test regression)

H
0
: No Heteroskedasticity up to order q in residuals

Reject H
0
if nR
2
> _
2
or if ps0.05

RESIDUAL TESTS

STABILITY TESTS
- RAMSEY RESET -
RAMSEYS RESET (Regression specification error test):
General test for the following types of misspecification:
Inclusion of irrelevant variables
Exclusion of relevant variables
Test based on augmented regression y=X|+Z+u

H
0
: equation is correctly specified

LR and F-test both test H
0

Reject H
0
if ps0.05
STABILITY TESTS
- RAMSEY RESET -

REGRESSION ANALYSIS - Lecture

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

REGRESSION ANALYSIS - Lecture

Uploaded by

Copyright:

Available Formats

Multiple linear regression model with K explanatory variables:

EVALUATING THE GOODNESS OF FIT

Basics of Regression Analysis

You might also like