Wiimote Infrared Detection

Greg Peaker Max Chiswick

Minority Report Video

Project Goal and Setup

Use infrared for gesture recognition
Works regardless of light and skin tone Infrared detector: Wiimote Infrared emitter: LED attached to hand

Performs gesture recognition Use gestures to control computer applications

Wiimote Features
Built in 1024x768 infrared camera
Processes images using system on a chip and sends post-processed information to computer (or Wii)

Connects to PC with Bluetooth Wii Wiimote Usage:

Wiimote moves around Locates two IR LED emitter arrays in sensor bar

Our Wiimote Usage:

Wiimote stays still IR emitter on finger moves around

Approach and Design

Use off-the-shelf and inexpensive components
Wii remote Infrared emitter: LED, resistor, battery power Free, open source software

Wiimote detects and tracks IR spots Wiimote connects to computer through Bluetooth
Sends data in ASCII at 100 Hz Not used: buttons pressed, acceleration, intensity Used: (x,y) coordinates Interpret user motions/Gesture Recognition Control Windows Media Player and PowerPoint

Library parses ASCII to obtain easily readable parameters

Code in Visual C#

How It Works
Software tracks (x,y) coordinates of IR led Determine start time and coordinates Create array of all coordinates in gesture array from start to end
Sampled >100 times per second

Determine end time and coordinates Directional movements of finger control windows applications

How It Works (2): Interpreting Start and Stop of Start gesture by Motion

Making infrared signal visible Holding signal in place for about 2 seconds Program examines last 3 samples every second If samples are within threshold, begin gesture If samples are not (e.g. random movements), do not begin gesture Once gesture begins, program creates array that keeps track of all (x,y) coordinates End gesture by holding signal in place for about 2 seconds Program continues to examine last 3 samples every second When it finds they are outside of threshold, continues gesture When it finds they are within threshold, ends gesture

How It Works (3): Determining Direction Used

Compare final (x,y) to initial (x,y) Obtain variables x_diff and y_diff If x_diff > threshold, define as East or West If y_diff > threshold, define as North or South If neither > threshold, no motion If one > threshold, basic North, South, East, or West direction If both > threshold, see if one direction is significantly greater than other Interpret this as only the larger one Example: 100 units east and 12 units north probably intended to be only east If both > threshold and neither significantly greater than other Create line between start and end points of gesture array Count every (x,y) element between start and end and count them as above or below line Use this info to determine, for example, whether gesture is South-East or East-South

User Movements
12 total possibilities
Basic 4 directions Direction combinations (order matters)
North-East and East-North are different

Applications
Use directional movements for:
PowerPoint
Next Page (North or East), Previous Page (South or West)

Media Players
Gesture Gesture

South North West East

Down Up Left Right

South West North - West North - East South - East

Previous Volume Down Volume Up Next

Gesture

West South West - North East - North East - South

Mute Mute Pause Stop

Next Steps
Hardware
Better define and increase range Smaller IR transmitter/receiver assembly Use reflection Two finger or more finger tracking allowing increased movements and novel applications

Software
Support applications such as web browsers, Google Earth, etc. Use as mouse cursor with clicking ability Better gesture recognition using machine learning

IR Computer Vision Today

Microsoft Surface
The computer's "vision" is created by a near-infrared, 850-nanometer-wavelength LED light source aimed at the surface. When an object touches the tabletop, the light is reflected to multiple infrared cameras with a net resolution of 1280 x 960, allowing it to sense, and react to items touching the tabletop.

iBar
iBar is an interactive, 10 meter long bar-counter using multi-touch technology. iBar has an intelligent tracking system to detect all objects touching the surface. Integrated video-projectors can project any content dynamically with the movements on the counter. The software is flexible and can be adjusted and enhanced. Objects can be illuminated at their position or virtual objects can be touched with your fingers. Every glass, cellphone, car key, cup, business card or even fingers are recognized when they touch the bar-counter.

Demo

Gesture

Gesture

Gesture

South North West East

Down Up Left Right

South West North - West North - East South - East

Previous Volume Down Volume Up Next

West South West - North East - North East - South

Mute Mute Pause Stop