Motion Tracking: Isaac Case

MOTION TRACKING
Isaac Case
Saturday, December 12, 2009

AGENDA
Object Detection
Object Tracking
Moving Camera

OBJECT DETECTION
How do we segment
the image into objects?
Background Subtraction
Frame Difference

BACKGROUND SUBTRACTION
Background must be determined
Select a frame
Average over frames
Statistical method
Assumes images are grey
Can be done with color images, but needs a more

complex method to do difference (other than ‘-’)

BACKGROUND DIFFERENCE
SELECT A FRAME
Pick one Frame
+ Easy
+ Fast
background = im2double(imread(‘background.jpg’))
frame = im2double(imread(‘current_frame.jpg’))
mask = abs(frame ‐ background)
- What happens when the frame changes?

SELECT A FRAME

AVERAGE FRAMES
Average the image values over multiple frames
+ Easy
%%Determine Background
imname = 'image_%03d.jpg' %
back = double(zeros(size(im1))); %blank image
for idx = 20:30
new_im = im2double(imread(sprintf(imname,idx)));
back = back + new_im;
end
back = back./11
%%Difference
mask = abs(frame ‐ background)
- Slower
- How many frames is enough? 10? 20? 30?
How often do we need to do this?

AVERAGE FRAMES
10 Frames 30 Frames 60 Frames

STATISTICAL
Determine standard deviation over range of frames
If the difference between the current frame and the average is greater than the standard
deviation, then it is a foreground pixel
+ More accurate
+ Less effected by noise

%%Determine Background
all_back = repmat(zeros(size(im1)),[1 1 11]); %blank image
for idx = 20:30
new_im = im2double(imread(sprintf(imname,idx)));
all_back(:,:,idx‐19) = new_im
end
avg_back = mean(all_back,3);
std_back = std(all_back,0,3);
%%Difference
diff = abs(frame ‐ background)
diff(diff < std_back) = 0
- Additional Processing
Needs to be constantly updated

STATISTICAL
Average Frame
Standard Deviation Difference of Frame & Average > std dev

+ Based on simple operations (-)
+ Can be very effective in controlled situations
- Hard to determine what is the background
- Real cameras introduce noise
- Changes in lighting & introduction of objects

require updates to background

FRAME DIFFERENCE
Ignore determining what the background is
Look at strict frame to frame differences
+ No need for background updating

delta = 2;
im_prev = im2double(imread(sprintf(imname,idx‐delta)));
im = im2double(imread(sprintf(imname,idx)));
im_next = im2double(imread(sprintf(imname,idx+delta)));
prev_diff = abs(im ‐ im_prev);
next_diff = abs(im ‐ im_next);
combined_diff = cat(3,prev_diff,next_diff);
diff = min(combined_diff,[],3)
- Have to look forward and backward in time

FRAME DIFFERENCE

BLOB DETECTION
Difference images are ‘double’ values
Objects need to be detected as ‘logical’ values
Difference images need to be thresholded
What is the right threshold?
variable?
static?

BLOB DETECTION
> 0.01
> 0.05
> 0.1
NOISE MANAGEMENT
Need to remove noise from a thresholded image
Morphological processing
open
close
dilate
erode
Smoothing & Adaptive filters

MORPHOLOGICAL PROCESSING
diff_image > 0.05

diff_image > 0.05
bwareaopen(image,30)

diff_image > 0.05
bwareaopen(image,30)
imclose(opened_image,strel('disk',5))

SMOOTHING & ADAPTIVE
THRESHOLDING
Lots of noise from the camera
Smooth / reduce the input signal
Humans still detect motion in smoothed or

reduced image
reduce input image (i.e. half)
smooth with gaussian low pass filter

SMOOTH THE INPUT
SIGNAL
Smooth & reduce input

No smoothing of the
image with 5x5
input image
gaussian low pass filter

MORE SMOOTHING
Difference image Thresholded at optimal value

processed with [10 7] (0.0196)
average filter
Note: No morphological
fspecial(‘average’,[10 7]) processing has been done yet

OPTIMAL
THRESHOLDING
Look at the histogram of the

difference image

OPTIMAL
THRESHOLDING

difference image
Analyze the cumulative sum

OPTIMAL
THRESHOLDING

difference image
Analyze the cumulative sum
Use second derivative to find

last inflection point

OPTIMAL
THRESHOLDING
Threshold can be calculated for each frame
no need to guess
less morphological processing (which is also a

guess)
Unfortunately only works on video with noise

(expects noise)

BLOB DETECTION
bwlabel
regionprops
stats =
regionprops(mask,'basic')
Each stat represents a

blob

OBJECT TRACKING
Segment foreground into independent blobs
Find correlation between blobs of previous frames to

current frame
Blobs can merge (and split)
If a blob travels multiple frames, track it (show trail)

BLOB TRACKING &
MERGING
Match Blobs
Look for forward/backward matches
Only Consider blobs that overlap mask
based on linear prediction
Matching based on:
Difference = w0*|distance| + w1*∆Area + w2*∆Color
Match previous frame blobs to current frame based on this

difference (minimize difference)

BLOB MATCHING
T=1 T=2

BLOB TRACKING &
MERGING CONTD.
If multiple blobs match to the same new blob in the

current frame, merge
When drawing bounding boxes, draw boxes around

sub blobs instead of the merged blobs
Before matching for next frame, decompose all blobs

to base components
not merged blob, but original matched sub blobs

MOVING CAMERA
Detect and realign video frame based on camera

motion
Find “static” points
Align images based on points

POINT MATCHING

POINTS
Harris Corners
Find the corners
Match the corners
SIFT
Scale Invariant Feature Transform

HARRIS CORNERS
Harris Corner Detection
Finds corners in an image
Two Options:
Find x shift, y shift for all pixels & determine highest

correlation of x & y shift to determine image shift
faster, less accurate
Use cross correlation to match region around corner to find

match
slower, more accurate

EXAMPLES

EXAMPLES

QUESTIONS?

Motion Tracking: Isaac Case

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Motion Tracking: Isaac Case

Uploaded by

Copyright:

Available Formats

MOTION TRACKING

Saturday, December 12, 2009

Saturday, December 12, 2009

Saturday, December 12, 2009

Background must be determined

Average over frames

Assumes images are grey

Can be done with color images, but needs a more

Saturday, December 12, 2009

- What happens when the frame changes?

Saturday, December 12, 2009

Saturday, December 12, 2009

- How many frames is enough? 10? 20? 30?

How often do we need to do this?

Saturday, December 12, 2009

Saturday, December 12, 2009

+ Less effected by noise

Needs to be constantly updated

Saturday, December 12, 2009

Standard Deviation Difference of Frame & Average > std dev

Saturday, December 12, 2009

+ Based on simple operations (-)

+ Can be very effective in controlled situations

- Hard to determine what is the background

- Real cameras introduce noise

- Changes in lighting & introduction of objects

Saturday, December 12, 2009

Look at strict frame to frame differences

+ No need for background updating

- Have to look forward and backward in time

Saturday, December 12, 2009

Saturday, December 12, 2009

Difference images are ‘double’ values

Objects need to be detected as ‘logical’ values

Difference images need to be thresholded

What is the right threshold?

Saturday, December 12, 2009

Need to remove noise from a thresholded image

Smoothing & Adaptive filters

Saturday, December 12, 2009

Saturday, December 12, 2009

Saturday, December 12, 2009

Saturday, December 12, 2009

Lots of noise from the camera

Smooth / reduce the input signal

Humans still detect motion in smoothed or

reduce input image (i.e. half)

smooth with gaussian low pass filter

Saturday, December 12, 2009

Smooth & reduce input

Saturday, December 12, 2009

Difference image Thresholded at optimal value

Saturday, December 12, 2009

Look at the histogram of the

Saturday, December 12, 2009

Look at the histogram of the

Analyze the cumulative sum

Saturday, December 12, 2009

Look at the histogram of the

Analyze the cumulative sum

Use second derivative to find

Saturday, December 12, 2009

Difference = w0|distance| + w1∆Area + w2*∆Color