Chapter 11

Simple Linear Regression

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-1

Chapter Goals
After completing this chapter, you should be able to:
Explain the simple linear regression model Obtain and interpret the simple linear regression equation for a set of data Evaluate regression residuals for aptness of the fitted model Understand the assumptions behind regression analysis Explain measures of variation and determine whether the independent variable is significant
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-2

Chapter Goals
(continued)

After completing this chapter, you should be able to:

Calculate and interpret confidence intervals for the regression coefficients Use the Durbin-Watson statistic to check for autocorrelation Form confidence and prediction intervals around an estimated Y value for a given X Recognize some potential problems if regression analysis is used incorrectly
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-3

Correlation vs. Regression

A scatter plot (or scatter diagram) can be used to show the relationship between two variables Correlation analysis is used to measure strength of the association (linear relationship) between two variables
Correlation is only concerned with strength of the relationship

No causal effect is implied with correlation

Correlation was first presented in Chapter 3

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-4

11.1 Introduction to Regression Analysis

Regression analysis is used to:
Predict the value of a dependent variable based on the value of at least one independent variable Explain the impact of changes in an independent variable on the dependent variable

Dependent variable: the variable we wish to explain Independent variable: the variable used to explain the dependent variable

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-5

11.2 Simple Linear Regression Model

Only one independent variable, X Relationship between X and Y is described by a linear function Changes in Y are assumed to be caused by changes in X

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-6

Types of Relationships
Linear relationships Y Y Curvilinear relationships

X Y Y

X
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

X
Chap 11-7

Types of Relationships
(continued) Strong relationships Y Y Weak relationships

X Y Y

X
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

X
Chap 11-8

Types of Relationships
(continued) No relationship Y

X Y

X
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-9

Simple Linear Regression Model

The population regression model:
Population Y intercept Dependent Variable Population Slope Coefficient Independent Variable

Random Error term

Yi 0 1Xi i
Linear component

Random Error component

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-10

Simple Linear Regression Model

(continued)

Y
Observed Value of Y for Xi

Yi 0 1Xi i
i
Slope = 1 Random Error for this Xi value

Predicted Value of Y for Xi

Intercept = 0

Xi
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

X
Chap 11-11

Simple Linear Regression Equation

The simple linear regression equation provides an estimate of the population regression line
Estimated (or predicted) Y value for observation i Estimate of the regression intercept

Estimate of the regression slope Value of X for observation i

a bX Y i i
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

The individual random error terms ei have a mean of zero

Chap 11-12

11.3 Least Squares Method

a and b are obtained by finding the values of 0 and 1 that minimize the sum of the squared (SSE resudial) differences : between Y and Y

)2 min (Y (a bX ))2 SSE min (Yi Y i i i

To minimize, differentiate with respect to a and b, and set each result to 0. This generates two simultaneous equations (called normal equations) & two unknowns. Solving for a and b, we get

n xi yi ( xi )( yi )
i 1 i 1 i 1

n xi2 ( xi ) 2
i 1 i 1

a y bx
Chap 11-13

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Simple Linear Regression Example

A real estate agent wishes to examine the relationship between the selling price of a home and its size (measured in square feet) A random sample of 10 houses is selected Dependent variable (Y) = house price in $1000s Independent variable (X) = square feet

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-14

Sample Data for House Price Model

House Price in $1000s (Y) 245 312 279 Square Feet (X) 1400 1600 1700

308
199 219 405

1875
1100 1550 2350

324
319 255

2450
1425 1700

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-15

Excel Output
Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 0.76211 0.58082 0.52842 41.33032 10

The regression equation is:

house price 98.24833 0.10977(squarefeet)

ANOVA
df Regression Residual Total 1 8 9 SS 18934.9348 13665.5652 32600.5000 MS 18934.9348 1708.1957 F 11.0848 Significance F 0.01039

Coefficients Intercept Square Feet 98.24833 0.10977

Standard Error 58.03348 0.03297

t Stat 1.69296 3.32938

P-value 0.12892 0.01039

Lower 95% -35.57720 0.03374

Upper 95% 232.07386 0.18580

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-16

Graphical Presentation
House price model: scatter plot and regression line
450

House Price ($1000s)

400 350 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 Square Feet

Slope = 0.10977

Intercept = 98.248

house price 98.24833 0.10977(squarefeet)

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-17

Interpretation of the Intercept, a

house price 98.24833 0.10977(squarefeet)
a is the estimated average value of Y when the value of X is zero (if X = 0 is in the range of observed X values)
Here, no houses had 0 square feet, so a = 98.24833 just indicates that, for houses within the range of sizes observed, $98,248.33 is the portion of the house price not explained by square feet

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-18

Interpretation of the Slope Coefficient, b

house price 98.24833 0.10977(squarefeet)

b measures the estimated change in the average value of Y as a result of a oneunit change in X
Here, b = .10977 tells us that the average value of a house increases by .10977($1000) = $109.77, on average, for each additional one square foot of size

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-19

Predictions using Regression Analysis

Predict the price for a house with 2000 square feet:

house price 98.25 0.1098 (sq.ft.) 98.25 0.1098(200 0) 317.85

The predicted price for a house with 2000 square feet is 317.85($1,000s) = $317,850
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-20

Interpolation vs. Extrapolation

When using a regression model for prediction, only predict within the relevant range of data
Relevant range for interpolation
450 400 350 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 Square Feet
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-21

House Price ($1000s)

Do not try to extrapolate beyond the range of observed Xs

11.4 Measures of Variation

Total variation is made up of two parts:

SST SSR
Regression Sum of Squares

SSE
Error Sum of Squares

Total Sum of Squares

SST ( Yi Y)2
where:

2 Y)2 SSE ( Y Y SSR ( Y i i i)

= Average value of the dependent variable

i = Predicted value of Y for the given Xi value Y

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Yi = Observed values of the dependent variable

Chap 11-22

Measures of Variation
(continued)

SST = total sum of squares

Measures the variation of the Yi values around their mean Y SSR = regression sum of squares Explained variation attributable to the relationship between X and Y SSE = error sum of squares

Variation attributable to factors other than the relationship between X and Y

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-23

Measures of Variation
(continued)

Y Yi
SST = (Yi - Y)2

SSE = (Yi - Yi )2

_
Y

_ SSR = (Yi - Y)2

_ Y

Xi
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

X
Chap 11-24

11.5 Coefficient of Determination, R2

The coefficient of determination is the portion of the total variation in the dependent variable that is explained by variation in the independent variable The coefficient of determination is also called R-squared and is denoted as R2
SSE SSR regression sum of squares R 1 SST SST total sum of squares
2

note:

0 R 1
2
Chap 11-25

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Excel Output
Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 0.76211 0.58082 0.52842 41.33032 10

SSR 18934.9348 r 0.58082 SST 32600.5000

2

58.08% of the variation in house prices is explained by variation in square feet

SS MS 18934.9348 1708.1957 F 11.0848 Significance F 0.01039

ANOVA
df Regression Residual Total 1 8 9 18934.9348 13665.5652 32600.5000

Coefficients Intercept Square Feet 98.24833 0.10977

Standard Error 58.03348 0.03297

t Stat 1.69296 3.32938

P-value 0.12892 0.01039

Lower 95% -35.57720 0.03374

Upper 95% 232.07386 0.18580

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-26

11.12 Correlation Coefficient

A measure of lineer association between two variables X and Y, denoted r. It is between -1 and +1

S XX r b S XY / KK ( S XX SYY ) SYY

(X
i 1 n i 1

X )(Yi Y )
2 2 ( Y Y ) i i 1 n

(Xi X )

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-27

Standard Error of Estimate

The standard deviation of the variation of observations around the regression line is estimated by
2 ( Y Y ) i i i1 n

S YX
Where

SSE n2

n2

SSE = error sum of squares n = sample size

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-28

Excel Output
Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 0.76211 0.58082 0.52842 41.33032 10

S YX 41.33032

ANOVA
df Regression Residual Total 1 8 9 SS 18934.9348 13665.5652 32600.5000 MS 18934.9348 1708.1957 F 11.0848 Significance F 0.01039

Coefficients Intercept Square Feet 98.24833 0.10977

Standard Error 58.03348 0.03297

t Stat 1.69296 3.32938

P-value 0.12892 0.01039

Lower 95% -35.57720 0.03374

Upper 95% 232.07386 0.18580

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-29

Comparing Standard Errors

SYX is a measure of the variation of observed Y values from the regression line Y Y

small s YX

large s YX

The magnitude of SYX should always be judged relative to the size of the Y values in the sample data i.e., SYX = $41.33K is moderately small relative to house prices in the $200 - $300K range
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-30

Residual Analysis
ei Yi Y i
The residual for observation i, ei, is the difference between its observed and predicted value Check the assumptions of regression by examining the residuals
Examine for linearity assumption Examine for constant variance for all levels of X (homoscedasticity) Evaluate normal distribution assumption Evaluate independence assumption

Graphical Analysis of Residuals

Can plot residuals vs. X
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-31

Residual Analysis for Linearity

Y Y

x
residuals residuals

Not Linear
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Linear
Chap 11-32

Residual Analysis for Homoscedasticity

Y
Y

x
residuals

x
residuals

x Non-constant variance

Constant variance
Chap 11-33

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Residual Analysis for Independence

Not Independent
residuals

X
residuals

Independent

residuals

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-34

Inferences About the Slope

The standard error of the regression slope coefficient (b) is estimated by

SYX Sb SSX
where:

SYX

(X X )
i

Sb

= Estimate of the standard error of the least squares slope

S YX

SSE = Standard error of the estimate n2

Chap 11-35

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Excel Output
Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 0.76211 0.58082 0.52842 41.33032 10

Sb 0.03297
SS MS 18934.9348 1708.1957 F 11.0848 Significance F 0.01039 18934.9348 13665.5652 32600.5000

ANOVA
df Regression Residual Total 1 8 9

Coefficients Intercept Square Feet 98.24833 0.10977

Standard Error 58.03348 0.03297

t Stat 1.69296 3.32938

P-value 0.12892 0.01039

Lower 95% -35.57720 0.03374

Upper 95% 232.07386 0.18580

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-36

Inference about the Slope: t-Test

t-test for a population slope
Is there a linear relationship between X and Y?

Null and alternative hypotheses

H0: 1 = 0 H1: 1 0 (no linear relationship) (linear relationship does exist)

Test statistic

b 1 t Sb
d.f. n 2
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

where: b = regression slope coefficient 1 = hypothesized slope Sb = standard error of the slope
Chap 11-37

Inference about the Slope: t-Test

(continued)
House Price in $1000s (y) 245 312 279 308 199 219 405

Square Feet (x)

1400 1600 1700 1875 1100 1550 2350

Estimated Regression Equation:

house price 98.25 0.1098 (sq.ft.)

The slope of this model is 0.1098 Does square footage of the house affect its sales price?

324
319 255

2450
1425 1700

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-38

Inferences about the Slope: t Test Example

H0: 1 = 0 H1: 1 0 From Excel output:
Coefficients Intercept 98.24833

b
Standard Error 58.03348

Sb
t Stat 1.69296 P-value 0.12892

Square Feet

0.10977

0.03297

3.32938

0.01039

b 1 0.10977 0 t t 3.32938 Sb 0.03297

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-39

Inferences about the Slope: t Test Example

(continued)

Test Statistic: t = 3.329

H0: 1 = 0 H1: 1 0 From Excel output:
Coefficients Intercept 98.24833

b
Standard Error 58.03348

Sb
t Stat 1.69296

t
P-value 0.12892

Square Feet

0.10977

0.03297

3.32938

0.01039

d.f. = 10-2 = 8 a/2=.025 a/2=.025

Reject H0

-t/2 -2.3060

Do not reject H0

0 t/2 2.3060 3.329

Reject H

Decision: Reject H0 Conclusion: There is sufficient evidence that square footage affects house price
Chap 11-40

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Inferences about the Slope: t Test Example

(continued)

P-value = 0.01039
H0: 1 = 0 H1: 1 0 From Excel output:
Coefficients Intercept 98.24833 Standard Error 58.03348

P-value
t Stat 1.69296 P-value 0.12892

Square Feet

0.10977

0.03297

3.32938

0.01039

This is a two-tail test, so the p-value is P(t > 3.329)+P(t < -3.329) = 0.01039 (for 8 d.f.)

Decision: P-value < so Reject H0 Conclusion: There is sufficient evidence that square footage affects house price
Chap 11-41

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

F-Test for Significance

F Test statistic: F MSR MSE
where
MSR MSE SSR k SSE n k 1

where F follows an F distribution with k numerator and (n k - 1) denominator degrees of freedom

(k = the number of independent variables in the regression model)
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-42

Excel Output
Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 0.76211 0.58082 0.52842 41.33032 10

MSR 18934.9348 F 11.0848 MSE 1708.1957

With 1 and 8 degrees of freedom
SS MS 18934.9348 1708.1957 F 11.0848

P-value for the F-Test

Significance F 0.01039

ANOVA
df Regression Residual Total 1 8 9 18934.9348 13665.5652 32600.5000

Coefficients Intercept Square Feet 98.24833 0.10977

Standard Error 58.03348 0.03297

t Stat 1.69296 3.32938

P-value 0.12892 0.01039

Lower 95% -35.57720 0.03374

Upper 95% 232.07386 0.18580

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-43

F-Test for Significance

(continued)

H0: 1 = 0 H1: 1 0 a = .05 df1= 1 df2 = 8

Critical Value:
Fa = 5.32

Test Statistic:
MSR F 11.08 MSE

Decision: Reject H0 at a = 0.05 Conclusion:

F There is sufficient evidence that house size affects selling price
Chap 11-44

a = .05

Do not reject H0

Reject H0

F.05 = 5.32

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Confidence Interval Estimate for the Slope

Confidence Interval Estimate of the Slope:

b1 t n2Sb1
Excel Printout for House Prices:
Coefficients Intercept Square Feet 98.24833 0.10977 Standard Error 58.03348 0.03297 t Stat 1.69296 3.32938 P-value 0.12892 0.01039

d.f. = n - 2

Lower 95% -35.57720 0.03374

Upper 95% 232.07386 0.18580

At 95% level of confidence, the confidence interval for the slope is (0.0337, 0.1858)
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-45

Confidence Interval Estimate for the Slope

(continued)
Coefficients Intercept Square Feet 98.24833 0.10977 Standard Error 58.03348 0.03297 t Stat 1.69296 3.32938 P-value 0.12892 0.01039 Lower 95% -35.57720 0.03374 Upper 95% 232.07386 0.18580

Since the units of the house price variable is $1000s, we are 95% confident that the average impact on sales price is between $33.70 and $185.80 per square foot of house size
This 95% confidence interval does not include 0. Conclusion: There is a significant relationship between house price and square feet at the .05 level of significance
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-46

Pitfalls of Regression Analysis

Lacking an awareness of the assumptions underlying least-squares regression Not knowing how to evaluate the assumptions Not knowing the alternatives to least-squares regression if a particular assumption is violated Using a regression model without knowledge of the subject matter Extrapolating outside the relevant range Start with a scatter plot of X on Y to observe possible relationship

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-47

Strategies for Avoiding the Pitfalls of Regression

Perform residual analysis to check the assumptions
Plot the residuals vs. X to check for violations of assumptions such as homoscedasticity Use a histogram, stem-and-leaf display, box-and-whisker plot, or normal probability plot of the residuals to uncover possible non-normality

If there is violation of any assumption, use alternative methods or models If there is no evidence of assumption violation, then test for the significance of the regression coefficients and construct confidence intervals and prediction intervals Avoid making predictions or forecasts outside the relevant range
Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc. Chap 11-48

Chapter Summary
Introduced types of regression models Reviewed assumptions of regression and correlation Discussed determining the simple linear regression equation Described measures of variation Discussed residual analysis Addressed measuring autocorrelation

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-49

Chapter Summary
(continued)

Described inference about the slope Discussed correlation -- measuring the strength of the association Addressed estimation of mean values and prediction of individual values Discussed possible pitfalls in regression and recommended strategies to avoid them

Statistics for Managers Using Microsoft Excel, 4e 2004 Prentice-Hall, Inc.

Chap 11-50