Regression:

A machine learning perspective

Emily Fox & Carlos Guestrin
Machine Learning Specialization
University of Washington
1 ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
Part of a specialization

©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 This course is a part of the
Machine Learning Specialization

1. Foundations

4. Clustering 5. Recommender
2. Regression 3. Classification
& Retrieval Systems

6. Capstone

3   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

What is the course about?

©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 What is regression?
From features to predictions

ML
Data Regression Intelligence
Method

Input x:
features derived Learn xày
from data
relationship Predict y:
continuous “output” or
“response” to input
5   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Salary after ML specialization

hard work

•  How much will your salary be? (y = $$)

•  Depends on x = performance in courses, quality of
capstone project, # of forum responses, …

6   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Stock prediction
•  Predict the price of a stock (y)
•  Depends on x =
- Recent history of stock price
- News events
- Related commodities

7   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Tweet popularity
•  How many people will retweet your tweet? (y)
•  Depends on x = # followers,
# of followers of followers,
features of text tweeted,
popularity of hashtag,
# of past retweets,…

8   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Reading your mind

Output y
very sad very happy

Inputs x are
brain region
intensities

9   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Case Study:
Predicting house prices

ML
Data Regression Intelligence
Method

$ = ??
price ($)

$ $
$ (y)
+ house house size
attributes (x)

10   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Impact of regression

11   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

Course outline

©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Module 1: Simple Regression
What makes it simple?
1 input and just fit a line to data
y
price ($)

house size x

13   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Module 1: Simple Regression
price ($)

Define goodness-of-fit
metric for each possible line

house size
Gradient descent algorithm
better fit

Get estimated
parameters
slope

- interpret
- use to form
intercept predictions
14   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Module 2: Multiple Regression
y
Fit more complex
price ($)

relationships than
just a line

house size x Incorporate

y more inputs
price ($)

-  Square feet
x[2] -  # bathrooms
-  # bedrooms
-  Lot size
-  Year built
house size x[1]
-  …
15   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Module 3: Assessing Performance
y y
price ($)

price ($)
R F I T
house size x VE size
Ohouse x

Measures of error:
-  Training
-  Test
-  True (generalization)

16   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Module 3: Assessing Performance
y y
price ($)

price ($)
R F I T
house size x VE size
Ohouse x

Bias-variance
tradeoff

17   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Module 4: Ridge Regression
y y
price ($)

price ($)
R F I T
house size x VE size
Ohouse x

Ridge total cost =

measure of fit + measure of
model complexity
bias-variance tradeoff
18   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Module 4: Ridge Regression
How to choose balance?
(i.e., model complexity)
measure of fit + measure of
model complexity

Cross validation

Valid
set

error2(λ) ŵλ(2)  
19   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Module 5: Feature Selection
& Lasso Regression
Lot  size   Dishwasher  
Single  Family   Garbage  disposal  
Year  built   Microwave  
$?
Last  sold  price   Range  /  Oven  
Last  sale  price/sqM   Refrigerator  
Finished  sqM   Washer  
Unfinished  sqM   Dryer  
Finished  basement  sqM   Laundry  loca0on  
#  floors   Hea0ng  type  
Flooring  types   Je]ed  Tub  
Parking  type   Deck  
Useful for efficiency Parking  amount   Fenced  Yard  
Cooling   Lawn  
of predictions and Hea0ng   Garden  
interpretability Exterior  materials  
Roof  type  
Sprinkler  System  
 

…
Structure  style    
 
20   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Module 5: Feature Selection
& Lasso Regression

Lasso total cost =

measure of fit + (different) measure of
model complexity

knocks out certain features…

“sparsity”

Coordinate descent algorithm

21   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Module 6: Nearest Neighbor
y & Kernel Regression
price ($)

Here, this is the

closest datapoint

house size $ = ??? x

22   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Module 6: Nearest Neighbor
& Kernel Regression
Epanechnikov Kernel (lambda = 0.2)

1.5

1 f(x0)

0.5

−0.5
$ = ???

−1

0 0.1 0.2 0.3 0.4 x0 0.6 0.7 0.8 0.9 1

23   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Summary of what’s covered
•  Linear regression
Models •  Regularization: Ridge (L2), Lasso (L1)
•  Nearest neighbor and kernel regression

•  Gradient descent
Algorithms •  Coordinate descent

•  Loss functions, bias-variance tradeoff,

Concepts cross-validation, sparsity, overfitting,
model selection, feature selection

24   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Assumed background

25 ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Math background
•  Basic calculus
- Concept of derivatives
•  Basic linear algebra
- Vectors
- Matrices
- Matrix multiply

26   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Programming experience
•  Basic Python used
- Can pick up along the way if
knowledge of other language

27   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

 Reliance on GraphLab Create
•  SFrames will be used, though not required
-  open source project of Dato
(creators of GraphLab Create)
-  can use pandas and numpy instead
•  Assignments will:
1.  Use GraphLab Create to
explore high-level concepts
2.  Ask you to implement
all algorithms without GraphLab Create
•  Net result:
-  learn how to code methods in Python
28   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  
 Computing needs
•  Basic 64-bit desktop or laptop
•  Access to internet
•  Ability to:
- Install and run Python (and GraphLab Create)
- Store a few GB of data

29   ©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on  

Let’s get started!

©2015  Emily  Fox  &  Carlos  Guestrin   Machine  Learning  Specializa0on