Matteo Valoriani

Nice to Meet You

Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP for Kinect

Intel Software Innovator: RealSense

email: mvaloriani@gmail.com
twitter: @MatteoValoriani
linkedin: https://it.linkedin.com/in/matteovaloriani

2
Usage Scenarios
RETAIL THERAPY HEALTHCARE EDUCATION TRAINING
You have to be a
magician…

… or at least a
good illusionist
 “Any sufficiently
advanced
technology is
indistinguishable
from magic”

(Arthur C. Clarke)
Sensor Anatomy
Kinect 2 - Specs
Hardware:
Depth resolution:
512×424
RGB CAMERA
RGB resolution:
1920×1080 (16:9)
3D DEPTH SENSOR
FrameRate:
60 FPS
Latency:
60 ms

MULTI-ARRAY MIC
Kinect Adapter
Kinect 2 vs Kinect 1
 Version 1 Version 2
Depth range 0.4m → 4.0m 0.4m → 4.5m
Color stream 640×480 1920×1080
Depth stream 320×240 512×424
Infrared stream None 512×424
Type of Light Light coding ToF
Audio stream 4-mic array 16 kHz 4-mic array 48 kHz
USB 2.0 3.0
# Bodies Traked 2 (+4) 6
# Joints 20 25
Hand Traking External tools Yes
Face Traking Yes Yes+Expressions
FOV 57° H 43° V 70° H 60° V
Tilt Motorized Manual
System / Software Requirements
OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)

CPU Intel Core i7 3.1GHz (or higher)

RAM 4GB (or more)

GPU DirectX 11 supported

USB USB 3.0

Compiler Visual Studio 2012, 2013 (Supported Express)

Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)

Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)

DEMO
Kinect Evolution
Accessing Kinect Data Sources
High-level architecture
Basic Flow of Programming
Kinect for Windows SDK v1

Sensor Stream Frame Data

Kinect for Windows SDK v2

Sensor Source Reader Frame Data

 Source independent to each Data

(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)

 Doesn’t depend on each other Source

(e.g. Doesn't need to Depth Source when retrieve Body Data)
Creating a new store app using Kinect

• In “New Project” create a new Windows Store app

• Enable Microphone and Webcam capabilities
• Add a reference to Microsoft.Kinect
• Use the Microsoft.Kinect namespace in your code
The KinectSensor class
this KinectSensor
this
// Make the world a better place with Kinect
this

Represents a single physical sensor

Always valid: when device is disconnected no more frame are
generated.
Use IsAviable Property to verify if the device is connected
Sources
Readers
Give access to frames
– Events InfraredFrameReader reader =
sensor.InfraredFrameSource.OpenReader();
– Polling reader.FrameArrived +=
InfraredReaderFrameArrived;
...
Multiple readers may be
created on a single source

Readers can be paused

Frame references
void InfraredFrameReader
InfraredFrameArrivedEventArgs
using (InfraredFrame frame =
args.FrameReference.AcquireFrame())
{
if (frame != null)
{
// Get what you need from the frame
}
}
}
Sent in frame event args
AcquireFrame gives access to the actual frame
Frames
• Gives access to the frame data
– Make a local copy or access the underlying buffer directly
• Contains metadata for the frame
– e.g. Color: format, width, height, etc.
• Important: Minimize how long you hold onto the frame
– Not Disposing frames will cause you to not receive more frames
MultiSourceFrameReader
• Allows the app to get a matched set of frames from multiple
sources on a single event
• Delivers frames at the lowest FPS of the selected sources
MultiSourceFrameReader MultiReader =
Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
FrameSourceTypes.BodyIndex |
FrameSourceTypes.Body);

var frame = args.FrameReference.AcquireFrame();

if (frame != null) {
using (colorFrame = frame.ColorFrameReference.AcquireFrame())
using (bodyFrame = frame.BodyFrameReference.AcquireFrame())
using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){
//
}
}
System

Kinect Kinect
Driver Driver

Kinect
Service

Kinect SDK Kinect SDK Kinect SDK Kinect SDK

Application Application Application Application

Kinect for Windows v1 Kinect for Windows v2

Connection

PC PC

Kinect for Windows v1 Kinect for Windows v2

Example Multiple Connection

PC PC PC

Hub

Server
DEMO
Display frame in a Windows Store App
Data Source Details
Physical image sensors, ColorFrameSource,
InfraredFrameSource, DepthFrameSource,
BodyIndexFrameSource, BodyFrameSource, AudioFrameSource
Kinect Data Sources
ColorFrameSource

• 30 or 15 fps, based on lighting

conditions
InfraredFrameSource
DepthFrameSource

– 16-bit distance in millimeters from the

sensor’s focal plane
BodyIndexFrameSource
255
0 1

– 0 to 5: Index of the corresponding body,

as tracked by the body source
– > 5: No tracked body at that pixel
BodyFrameSource
• Range is 0.5-4.5 meters
• Frame data is a collection of Body objects each
with 25 joints
– Each joint has position in 3D space and an
orientation
• Up to six simultaneous bodies
• 30fps
• Hand State on 2 bodies
• Lean
Skeleton vs Body
Coordinate System

 ColorSpace (Coordinate System of the Color

Image)
– … Color

 DepthSpace (Coordinate System of the

Depth Data)
– … Depth, Infrared, BodyIndex

 CameraSpace (Coordinate System with the

origin located the Depth Sensor)
– … Body (Joint)
Coordinate mapping

• Three coordinate systems

Name Applies to Dimensions Units Range Origin

ColorSpacePoint Color
DepthSpacePoint Depth,
Infrared,
Body index
CameraSpacePoint Body
2 pixels 1920x1080 Top left corner
2 pixels 512x424 Top left corner
3 meters – Infrared/depth
camera

• Coordinate mapper provides conversions between each system

• Convert single or multiple points
DEMO
Handle body frames and coordinate mapping
Kinect Studio

Recording and Playback

Recordable Data Sources
Infrared
13 MB/s

Audio Depth
32 KB/s 13 MB/s

Legend
Record/Play

Record Only
Color
BodyFrame
120 MB/s

BodyIndex
DEMO
Kinect Studio 2.0
Recording | Playback
Gesture Recognition

Heuristic + Machine Learning Approaches

DEMO
Programmatic detection of a gesture/pose
Gesture Builder

• New tool, shipping with v2 SDK

• Organize data using projects and solutions
• Give meaning to data by tagging gestures
• Build gestures using machine learning technology
– Adaptive Boosting (AdaBoost) Trigger
• Determines if player is performing gesture
– Random Forest Regression (RFR) Progress
• Determines the progress of the gesture performed by player

• Analyze / test the results of gesture detection

• Live preview of results
Visual Gesture Builder

Your Application
DEMO
Machine Learning Demo
Gesture Recognition
Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to find
good signals for Heuristic
approach
Machine Learning (ML) with G.B.
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress
in a baseball swing)
• Large investment for production
• Danger of over-fitting, causes you
to be too specific – eliminating
recognition of generic cases
Hand Pointer Gestures
Hand pointer gestures

Press

Engagement Targeting

Panning/Zoom
DEMO
Controls Basics – XAML
UserViewer | Engagement | Targeting | Press
Demo
Building ControlsBasics-XAML from Scratch

• File New Project > Grid App

• Add Reference to Microsoft.Kinect.Xaml.Controls
• Enable Microphone + Camera capabilities for app
• Add KinectRegion as container for rootFrame in
App.xaml.cs
• Run it!
Adding Hand Cursor Support Window

Grid
Window

Kinect
KinectRegion
UserViewer
Frame

Frame

Page
Page
“Engagement” for hand cursor
System Engagement Manual Engagement
• Hands over head
• In certain location of room
• Etc…

KinectRegion.SetOnePersonSystemEngagement() KinectRegion.SetOnePersonManualEngagement(…)
KinectRegion.SetTwoPersonSystemEngagement() KinectRegion.SetTwoPersonManualEngagement(…)
DEMO
PHiZ (Physical Interaction Zone) & KinectPointerPoint
PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X
– Between 0.0 and 1.0

KinectPointerPoint->Position->Y
– Between 0.0 and 1.0

X / Y – Allows offscreen info

KinectPointerPoint->Properties->UnclampedPosition->X
KinectPointerPoint->Properties->UnclampedPosition->Y

Z
KinectPointerPoint->Properties->HandReachExtent (raw)
KinectPointerPoint->Properties->PressExtent (affected by user/UI)

Correlating PointerPoint to NUI Data

KinectPointerPoint->Properties->BodyTrackingId
KinectPointerPoint->Properties->HandType
KinectPointerPoint->Properties->BodyTimeCounter
 Press

Recommended Minimum Size: Adapting to smaller visual sizes:

• 208 x 208 (in 1080p resolution) • Make large-size hit testable
• Press attraction towards center • Set KinectRegion.KinectPressInset
• Larger buttons will just (Thickness) to non-visible part of button
attract away from the edge • Attraction region is smaller
Cursors
Demo
Controls Basics – XAML
Scroll
Zoom
Interacting with Controls (in Preview release)
• UI Frameworks
– XAML (Store)
– WPF
• DirectX (Store)
ControlsBasics- ControlsBasics-
App (host KinectRegion, etc…)
WPF XAML
ControlsBasics-
DX
Controls Layer – Cursor rendering, Hit Test, Microsoft.Kinect.
Route PointerPoint via InputPointerManager Xaml.Controls
Microsoft.Kinect.
Wpf.Controls
InputPointerManager, PressableModel,
Microsoft.Kinect.Toolkit.Input
ManipulatableModel

KinectCoreWindow.PointerMoved, Microsoft.Kinect WindowsPreview.Kinect

KinectPointerPoint, KinectGestureRecognizer
Interacting with Content

• Navigating a camera through a 3d space

• Rotating a 3d object
• “Scrubbing” a video
Kinect Resources

• General Info & Blog -> http://kinectforwindows.com

• Purchase Sensor -> http://aka.ms/k4wv2purchase
• v2 Preview SDK -> http://aka.ms/k4wv2previewsdk
• Developer Forums -> http://aka.ms/k4wv2forum
Q&A
Tutto il nateriale di questa sessione sul sito
http://www.dotnetcampus.it

@MatteoValoriani
©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the
U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft
must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after
the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.