DFuse and MediaBroker: System support for sensor-based distributed computing

Kishore Ramachandran (http://www.cc.gatech.edu/~rama) College of Computing Georgia Tech Colleagues: Rajnish Kumar, Bikash Agarwalla, Junsuk Shin, David Hilley, Dave Lillethun, Jin Nakazawa, Bin Liu, Xiang Song, Nova Ahmed, Seth Horrigan, Matt Wolenetz, Arnab Paul, Sameer Adhikari, Ilya Bagrak, Martin Modahl, Phil Hutto

Computing/Communication Continuum
Sensor Network

HPC resources

High connectivity Cameras, sensor nodes

Low connectivity / Wireless High Performance Computing (HPC) resources

Ambient Computing Infrastructure

What does this enable?

Application context

distributed sensors with varying capabilities control loop involving sensors, actuators rapid response time at computational perception speeds

Sample applications

Video based surveillance Transportation Emergency Response

Collaborative search and rescue Evacuation management

Aware Environments

Application Characteristics

Physically distributed heterogeneous devices Interfacing and integrating with the physical environment Diverse stream types (low to high BW) Diverse computation, communication and power capabilities (from embedded sensors to clusters) Stream fusion/transformation, with loadable code Resource scarcities Dynamic join/leave of application components

Key Requirements
Middleware

Programming infrastructure Distributed data fusion Stream data management

Stampede

DFuse MediaBroker
SensorStack

Network Level

Protocol stack Energy efficient routing Grid

Ambient HPC resources

Streamline

Stampede (TPDS 2003)

Distributed programming system covering the continuum Temporal stream data transport Multilingual (C, C++, Java) program components sharing data abstractions Multiple platforms (x86-Linux, ARM-Linux, x86-Windows, x86-Solaris, Alpha-Tru64)
thread

put(ts, item)

get(ts, item)
consume(ts)
Channel

many to many connections time sequenced data correlation of streams

automatic GC

Key Requirements
Middleware

Programming infrastructure Distributed data fusion Stream data management

Stampede

DFuse MediaBroker
SensorStack

Network Level

Protocol stack Energy efficient routing Grid

Ambient HPC resources

Streamline

DFuse (ACM SenSys 2003)

Future Sensor Networks

Collage

Sink
Filter

Todays handhelds, gateways Ubiquitous high-bw sensors

Cameras

Challenges

Overlaying the application onto the physical network Programming abstraction for data fusion

Fusion Channel (a Virtual Sensor)

Producers (sensors or other fusion channels) Consumers

f()

(actuators or other fusion channels)

...

...

Fusion ( Arg[ ])
Filter Collage Sink { .. }

Cost Function

Cameras

Fusion Module Placement Module

Fusion Module

Placement Module

Resource Monitor, Routing Layer Interface

Resource Monitor, Routing Layer Interface Operating System

Operating System

Hardware

Hardware
DFuse functions: 1. Placement of fusion and relay points 2. Plumbing as required 3. Dynamic migration of fusion points

Status of DFuse

Fusion and placement modules implemented on top of Stampede

A prototype iPAQ farm (simulating future sensor network) runs DFuse Stampede and DFuse available for downloads

MSSN, a simulator for sensor networks middleware design guidance (BaseNets04, IJNM05, Wolenetzs thesis)

Available for downloads

Key Requirements
Middleware

Programming infrastructure Distributed data fusion Stream data management

Stampede

DFuse MediaBroker
SensorStack

Network Level

Protocol stack Energy efficient routing Grid

Ambient HPC resources

Streamline

MediaBroker

An architecture for stream management

A clearing house for sensors and actuators in a given space Stream registry, discovery, plumbing, sharing, Dynamic connection of sources (producers) and sinks (consumers) Dynamic injection, and safe execution of transformation code
Feature extraction, fusion

Dynamic sharing of transformations and streams

Elements

Type server: stores data types, relationships, and transformation code Transformation engine: allow safe execution of injected code on cluster nodes Scheduler: manages workload, and allows prioritizing transformation requests Data brokers: manages connections between producers and consumers
Producer Consumer Producer Consumer

Transformation Engine Data Broker Type Server Transformation Engine Scheduler Transformation Engine Data Broker

Data Items Transformation Requests Transformation Code

What it enables

Dynamic instantiations and sharing of transformations

MediaBroker Status

MediaBroker V.1

A subset of the functionalities Application example: Family intercom IEEE PerCom 2004, PMC 2005

MediaBroker++

Currently under development EventWeb built on top

Key Requirements
Middleware

Programming infrastructure Distributed data fusion Stream data management

Stampede

DFuse MediaBroker
SensorStack

Network Level

Protocol stack Energy efficient routing Grid

Ambient HPC resources

Streamline

SensorStack

Adaptability Vs. Stackability of protocol layers

Adaptability deals with cross-layer data

A must for wireless sensor networks

Stackability deals with cross-layer functionalities

A must for modular design

Principle behind SensorStack

Decouple data from functionalities

SensorStack without Cross-layering Support

Fusion requirement

Fusion layer

Neighborhood Information, Topology

Application
Data periodicity, Size, delay tolerance

Fusion data transmission requirement

Flood routing

Data transmission requirement

AODV routing

MAC

Link quality information, Neighborhood status change, topology

Time Sync Service

Time synchronization accuracy, accuracy requirement

Information Exchange Service (IES)

Design Goals:
1. 2. 3. 4. 5.

Efficient use of limited memory Simplifying information sharing interface Extensibility Asynchronous delivery of the information Complex event notification

IES Design

Data management module

Stackability by using standard data interface Publish/subscribe based shared memory system Fully-associative cache for performance

Event management module

Adaptability by notifying when to adapt Complex event notification Reactive memory access

Publisher List Shared Memory

Subscriber List

Date Publisher

put DRE DAE Cache

get

Data Subscriber

Set rules EMM RSE

Event Subscriber

DRE: Data request event RSE: Rule satisfied event DAE: Data available event EMM: Event management module

SensorStack with Cross-layering support

Application Data Fusion Layer
Data Service Layer \ Application Logic

In-stack fusion Information Exchange Service Helper Service Layer

Next-hop selection, Logical naming, Packet scatter/gather

AttributeValue publish/ subscribe

Medium Access Layer Radio

Medium Access, Error Control, Radio Control

Localization, Synchronization Service

Connection

(A) Stack Lay-out

(B) Functionalities

Implemented in TinyOS and iPAQ Linux Initial results (increasing application lifetime) very promising

Application Lifetime Improvement with Cross-layer Information

DFuse performance without Cross-layer information

DFuse performance with Cross-layer information for role-migration

Key Requirements
Middleware

Programming infrastructure Distributed data fusion Stream data management

Stampede

DFuse MediaBroker
SensorStack

Network Level

Protocol stack Energy efficient routing Grid

Ambient HPC resources

Streamline

Pressure Camera

Humidity

Camera P/T/Z

C Recorder V

Gateway

Cluster

Proximity

Grid Infrastructure
Gateway

MediaBroker
Cluster

DFuse Stampede SensorStack

Gateway

Gateway

Cluster

Streamline (MMNC 2006)

Scheduling problem Input:

Computation and communication requirements of various stages of a coarse-grain dataflow graph Application-specified constraints Current resource (processing and bandwidth) availability Resource specific constraints
Placement of the stages of the pipeline on available HPC resources latency and throughput of the application

Output:

Performance criteria:

Streamline Scheduler
S0
S2 S1 S3
Stage Prioritization

R0 R3 R1

{S2 S0 S1 S3}
Application Policies

Resource Filtering

{S2 {R0 R2 R3}}

Resource Selection

Resource Policies

R2

{S2 R0}

Expects to maximize throughput by assigning best resource to most needy stage Additional policies concerning resources, applications, and local schedulers can be incorporated in the cost of a particular assignment

Streamline System Architecture

Results

Streaming Appication Data Sources

Outperforms condor by an order of magnitude for both compute and communication bound kernels, particularly under non-uniform load conditions Performs close to Simulated Annealing but at considerably low scheduling time (by a factor of 1000)

Resource Information Service

Authentication Service

Grid Boundary

Application Policies QoS Monitoring Service

Scheduling Service

Application Information Service

Streaming Application

GRAM

GRAM

GRAM

HPC Resource

HPC Resource

HPC Resource

Streaming Grid

Streamline scheduler integrated into Globus toolkit Example of a mock traffic monitoring app as a service composition using Web Services Blue boxes are the Streaming Grid services

Demo

Demo Output (live if stars are aligned!)

http://www.cc.gatech.edu/~bikash/sgrid/trafficapp/ http://www.cc.gatech.edu/~bikash/sgrid/trafficapp/ trafficapp.html What is the takeaway?

Several technologies working together Service composition, Streamline scheduling, Web Services, Globus toolkit to process the video stream and show the output in the browser Streaming grid instantiates, connect, manages the streaming app

Computing/Communication Continuum
Sensor Network

HPC resources

High connectivity Cameras, sensor nodes

Low connectivity / Wireless High Performance Computing (HPC) resources

Ambient Computing Infrastructure

Conclusions

MediaBroker

DFuse

stream transformation and typed transport engine data fusion architecture distributed programming environment Information Exchange Service for crosslayer support Scheduling support for streaming apps on grid

MediaBroker DFuse Stampede SensorStack

Stampede

SensorStack

Streamline

Streamline

Ongoing Work

Programming tools Adaptive resource management

Marrying grid computing and ubiquitous computing

Wireless networking considerations

sensorstack energy efficient protocols mobility considerations

Web Links

http://www.cc.gatech.edu/~rama http://www.cc.gatech.edu/~rama/stampede http://www.cc.gatech.edu/~rama/up http://www.cercs.gatech.edu/

Applause!!!

Sensor Networks

Current Sensor Network Nodes

Limited capabilities (Mica-2) habitat monitoring, vineyard monitoring

Recent trends

iMotes

Telos motes

8x radio, 4x CPU, power++ >3x radio, similar CPU & power

Future Sensor Network Nodes

Todays handhelds, gateways Ubiquitous high-bw sensors

Source: CACM #47-6 The platforms enabling wireless sensor networks, Hill, Horton, Kling, Krishnamurthy, 2004.

HPC resources

Unix / Linux / XP cluster

Overlaying the fusion graph

uncompress collage

filter

Fusion Applications : need hierarchical fusion support

Family Intercom

Clients connected via MediaBroker Type attributes include audio rate and buffer specs

A client tracking system used for Icombo selection Colocated clients can perform mixing for N-way conferencing

Fusion Module
Structure Management Plumbing Issues

Producers

Consumers

...

...

Fusion Module
Computation Management
Dynamic embedding of user-specified fusion function

Correlation and aggregation of input streams Fusion channel migration

f1()

f()

f2()
...

...

Fusion Module
Computation Management
Dynamic embedding of user-specified fusion function

Correlation and aggregation of input streams Fusion channel migration

Memory Management

f2() f1()

f()

...

...

Fusion Module
Error and Status management Failure / Latency hiding

f2() f1()

f()

...

...

Placement Module
User inputs the task graph

S1
S2 S3
Sources

Collage Filter
Sink (Display)

And, a cost function.

Simple Solution ?

Why not push the fusion points towards its sources ?

Data

sources may be lying all around

Fusion

points may cause data expansion Network is dynamic

DFuse: Placement Modules Algorithm

Three phases:
1.

Nave role assignment

Deploy task graph into the network Start app at a designated root node and delegate task graph subtree roles to richest neighbors recursively

2.

Optimization
Given anticipated application behavior (attributes in the task graph), perform rapid local decisions to adjust which node performs which role. Decisions guided by application-provided cost-function

3.

Maintenance
Monitor actual application behavior and perform less frequent optimizations given application-provided costfunction

Example Cost Function

MT1 (Minimize Transmission Cost1) n1 n2

Source

2 Kbps

2 Kbps 2 Kbps

f()

1 Kbps

Sink

Source

2 Kbps

CMT1 ( n2, f ) = 9 kbps CMT1 ( n1, f ) = 6 kbps

DFuse: Cost Functions

Used in intuitive illustration slides m input data sources (fan-in) n output data consumers (fan-out) MPV (Minimize Power Variance) Attempts to keep the power of network nodes at similar levels. MTP (Minimize Ratio of Transmission Cost to Power) Attempts to keep the time for how long nodes can run a fusion function similar. MT2 (Minimize Transmission Cost 2) Like MT1, but allows role transfer when node power is below a threshold.

MT1 (Minimize Transmission Cost 1)

c(k,f) = cost of node k to perform role f t(x) = transmission rate of data source x hopCount(i,k) = network hops between nodes i and k power(k) = remaining power at node k

Prototype DFuse Implementation

Goal

Hypothesis

Investigate utility of Fusion Point Migration Migration will increase application lifetime, for constant QoS Fusion Module: ARM Stampede port Simulated placement module Interface for coupling, transmission monitoring

Implementation

Simple camera sources, fusion functions and sink

Collage
Filter

Experimental Setup

12 iPAQ 3870s in a familiar 0.6.1 Linux based DStampede 802.11b farm. Only directly adjacent iPAQs in the figure below are considered mutually reachable in 1 hop. Placement module run as a simulation of the distributed algorithm coupled to the farm via an extended fusion module interface Power usage is modeled to be linear to the number of application-level bytes transmitted across a farm hop (simple model)

Prototype DFuse Results: Transmission Cost over Time

Prototype DFuse Results: Variance, Role Transfers, Lifetime

MPV vs MT2 4x less variance Migrations++ 70% lifetime MTP good variance good lifetime

Pervasive Computing with MediaBroker

Family Intercom and Sign Post

Being developed at Georgia Techs Aware Home Replacing a legacy hardware mixing system with a much more scalable system Integration with existing RFID and Vision based tracking system called Sign Post to enable rapid call dispatching and mobility-aware communication