Multisensory

Perception

Alex
Teichman

The Object Multisensory

Recognition
Problem Perception
Boosting

Image
Descriptors
Library Alex Teichman
Applications

Moving
Forward
August 26, 2009
 The Object Recognition Problem

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward

How much does this look like a hand?

 The Object Recognition Problem

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward

Where is the coke can?

 The Object Recognition Problem

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward

What is the object in the box?

 Some Difficulties Among Many

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting
Labels are painful to acquire.
Image
Descriptors
Library

Applications

Moving
Forward
 Some Difficulties Among Many

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting
Labels are painful to acquire.
Image Generalization to data not in your training set can be
Descriptors
Library arbitrarily hard.
Applications

Moving
Forward
 Some Difficulties Among Many

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting
Labels are painful to acquire.
Image Generalization to data not in your training set can be
Descriptors
Library arbitrarily hard.
Applications
Descriptors are generally suited to a particular task.
Moving
Forward
 Boosting - Intuition

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Boosting - Intuition

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Boosting - Math

Multisensory
Perception

Alex
Teichman Training examples (ym , ~xm ) for all m = 1..M, ym ∈ {−1, +1}.
The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Boosting - Math

Multisensory
Perception

Alex
Teichman Training examples (ym , ~xm ) for all m = 1..M, ym ∈ {−1, +1}.
The Object X
Recognition
Problem
min exp(−ym H(~xm )) (1)
H
m
Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Boosting - Math

Multisensory
Perception

Alex
Teichman Training examples (ym , ~xm ) for all m = 1..M, ym ∈ {−1, +1}.
The Object X
Recognition
Problem
min exp(−ym H(~xm )) (1)
H
m
Boosting X
Image H(~x ) = ht (~x ) (2)
Descriptors
Library t
Applications

Moving
Forward
 Boosting - Math

Multisensory
Perception

Alex
Teichman Training examples (ym , ~xm ) for all m = 1..M, ym ∈ {−1, +1}.
The Object X
Recognition
Problem
min exp(−ym H(~xm )) (1)
H
m
Boosting X
Image H(~x ) = ht (~x ) (2)
Descriptors
Library t
(
Applications at if dist(~x , x~t ) < θt
Moving ht (~x ) = (3)
Forward 0 otherwise

Stagewise
P optimization:
minht+1 m exp (−ym (H(~xm ) + ht+1 (~xm )))
 Image Descriptors Library

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image Goal: Plug-and-play image descriptors.

Descriptors
Library

Applications

Moving
Forward
 Image Descriptors Library

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward

See descriptors 2d package

 Hand Detection

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Hand Detection

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Histogram of Oriented Gradient descriptors (3 different
Problem
parameter settings)
Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Hand Detection

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Histogram of Oriented Gradient descriptors (3 different
Problem
parameter settings)
Boosting

Image
Superpixel Color Histogram descriptors (4 different
Descriptors superpixel sizes)
Library

Applications

Moving
Forward
 Hand Detection

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Histogram of Oriented Gradient descriptors (3 different
Problem
parameter settings)
Boosting

Image
Superpixel Color Histogram descriptors (4 different
Descriptors superpixel sizes)
Library

Applications 1 hour to train.

Moving
Forward

Collaboration with Romain and Alex S.

 Video Label Propagation

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward

Label a video given just a few frame labels.

 Mechanical Turk Feedback Loop

Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward

Implementing the feedback loop of the hands training methodology

on mechanical turk with a bottles dataset.
 Online Learning

Multisensory
Perception
X
Alex min exp(−ym H(~xm ))
Teichman a
m
The Object
Recognition Can solve this for all weak classifier responses
Problem
simultaneously.
Boosting

Image
Constant space for any number of training examples.
Descriptors
Library

Applications

Moving Maintain the gradient and Hessian as a stream of labels comes

Forward
into the classifier:
1 X
g= −ym Im
M m
1 X
H= Im IT
m
M m
 Moving Forward

Multisensory
Perception

Alex
Teichman
Labels are painful to acquire.
The Object
Recognition
Problem

Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Moving Forward

Multisensory
Perception

Alex
Teichman
Labels are painful to acquire.
The Object
Recognition
Put classifier predictions on Mechanical Turk, get
Problem corrections, feed back into boosting.
Boosting

Image
Descriptors
Library

Applications

Moving
Forward
 Moving Forward

Multisensory
Perception

Alex
Teichman
Labels are painful to acquire.
The Object
Recognition
Put classifier predictions on Mechanical Turk, get
Problem corrections, feed back into boosting.
Boosting
Generalization to data not in your training set can be
Image
Descriptors arbitrarily hard.
Library

Applications Put classifier predictions on Mechanical Turk, get

Moving corrections, feed back into boosting.
Forward
 Moving Forward

Multisensory
Perception

Alex
Teichman
Labels are painful to acquire.
The Object
Recognition
Put classifier predictions on Mechanical Turk, get
Problem corrections, feed back into boosting.
Boosting
Generalization to data not in your training set can be
Image
Descriptors arbitrarily hard.
Library

Applications Put classifier predictions on Mechanical Turk, get

Moving corrections, feed back into boosting.
Forward
Descriptors are generally suited to a particular task.
 Moving Forward

Multisensory
Perception

Alex
Teichman
Labels are painful to acquire.
The Object
Recognition
Put classifier predictions on Mechanical Turk, get
Problem corrections, feed back into boosting.
Boosting
Generalization to data not in your training set can be
Image
Descriptors arbitrarily hard.
Library

Applications Put classifier predictions on Mechanical Turk, get

Moving corrections, feed back into boosting.
Forward
Descriptors are generally suited to a particular task.
Use a large library of descriptors and allow boosting to
choose which are most useful through the optimization.
 Multisensory
Perception

Alex
Teichman

The Object
Recognition
Problem

Boosting

Image
Descriptors
Library Thanks!
Applications

Moving
Forward