Distributed Systems Group

Prof.dr.ir. Dick H.J. Epema

5 September 2016

5 September 2016
1

The Distributed Systems Group: People

Dick Epema

Alexandru Iosup

Johan Pouwelse

Cloud Computing Online Social Networks

Cloud Computing
Cybercurrencies
Big Data
e-Science
Applied Security
Online Social Networks Online Gaming

5 September 2016

The Distributed Systems Group: Teaching

MSc courses:

IN4150: Distributed Algorithms (Q2)

IN4391: Distributed Computing Systems (Q3) (core Data Sc.&Techn.)

IN4049: High-Performance Computing (Q1)

IN4392: Seminar Cloud Computing (Q1)

IN4253: Applied Security Analysis (Q3)

5 September 2016

(core Software Techn.)

Example Distributed Algorithms (IN4150):

detecting global states
Problem:

record the state of a distributed application

(in an asynchronous system)

Applications: debugging, detecting stable properties

Issues:
1.recording the states of the processes at the same time
2.recording the messages that are in transit

Real-world example:
o recording the amounts of money in bank accounts
at 0:00 AM on Jan 1st by the tax authority

5 September 2016

Topics Distributed Computing (IN4391)

Communication:
Scheduling:
Fault tolerance:
Big Data:
Use case 1:
Use case 2:

5 September 2016

naming and messaging

distributed workloads and
resource management
reliability, availability
consistency, replication,
distributed file systems
scientific/engineering data
processing
massive multiplayer online gaming

The Distributed Systems Group: type of Research

Our research is:
fundamental: devise new application-independent concepts
in distributed systems
experimental: show the value of new concepts in
prototypes or real deployments

5 September 2016

Experimentation: DAS-5
UvA/MultimediaN (62)

VU (136 CPUs)

UvA (36)
System purely for CS research
Operational since June 2015
Specs:
3,200 cores (8-core CPUs)

SURFnet6

10 Gb/s lambdas

TU Delft (48)
5 September 2016

Astron (9)

2.4 GHz CPUs

accelerators (GPUs)
800 TB storage
10 Gb/s Infiniband
Gb Ethernet

Leiden (48)
7

The Distributed Systems Group:

Research Topics
1. Resource Management and Scheduling
2. Big Data Processing (e.g., Graph Processing)
3. Cooperative Systems

5 September 2016

The KOALA multicluster scheduler

deployed
on the DAS
since 2005

parallel MPI applications

workflows
MapReduce applications

Local
Resource
Managers

5 September 2016

LRM

information
service

LRM

LRM

LRM

Processor co-allocation (1)

Idea: split up a parallel application in components and
schedule these in different clusters
Reasons:
o to benefit from available resources (e.g., processors, data)
o application characteristics (e.g., simulation in one location,
visualization in another)

job

Main problems:
o need to coordinate allocations by
autonomous resource managers
o schedule efficiently

5 September 2016

clusters

10

Co-allocation (2): wide-area communication

Co-allocated applications are less efficient due to the
relatively slow wide-area communications

execution
time
(seconds)

32 CPUs

1
2
3
4
number of clusters combined
5 September 2016

11

Co-allocation (3): slowdown

Slowdown of a job:
execution time on multicluster
execution time on single cluster

(>1 usually)

Processor co-allocation is a trade-off between

+ faster access to more capacity
- longer execution times

Many scheduling policies possible for co-allocation

Main finding: co-allocation is beneficial when slowdown 1.20
5 September 2016

12

Big Data Processing

graph processing

5 September 2016

13

Cooperative Systems (1): Tribler

Is based on the BitTorrent P2P file-sharing system
Uses an epidemic protocol for peer and content discovery
Was first released on 17 March 2006 (1,500,000+ downloads)
Enables video-on-demand and live streaming
Is our research vehicle for P2P, Online Social Network
Current focus: reputation, trust, cybercurrencies
Download at www.tribler.org

5 September 2016

14

Cooperative Systems (2): trust/reputation

Problems:
o why help others downloading in P2P systems?
o why contribute to Wikipedia?
o why be good and pay taxes?

Solution: create a trust system without central control

o record decentralized interaction history
o disseminate this history in an attack-resilient way

In Tribler:
o maintain records of uploads and downloads
o compute reputations from resulting graph

5 September 2016

15

Cooperative Systems (3): the blockchain

the blockchain

block 43

5 September 2016

block 44
De blockchain

block 45

16

Organized by the DS Group

icpe2016.spec.org
5 September 2016

17

Doing your MSc in the DS Group

requires you to do at least 2 of the DS MSc courses
is hard work (experiments!!), were not easily satisfied
will push you to a high grade (or, higher than you expected)
lets you collaborate with other MSc and PhD students
learns you to become an independent engineer/researcher
can be done within or outside the university (institutes (NIKHEF)/
companies (Solvinity))
may lead to a conference paper, but much depends on you

5 September 2016

18

Some MSc Research Topics for You

@large

Data Center
Workloads

Cloud Res.Man.
Management

Gaming
@large

Big Data: Graphs

Cybercurrencies
Tribler

Big Data:
Elastic MapRed

5 September 2016

19

More information
MSc coordinator: Johan Pouwelse
Some previous MSc theses:
www.ds.ewi.tudelft.nl/epema/teaching

DS publications
http://publications.st.ewi.tudelft.nl

Home pages:

www.ds.ewi.tudelft.nl

www.ds.ewi.tudelft.nl/epema

www.ds.ewi.tudelft.nl/~iosup

www.ds.ewi.tudelft.nl/pouwelse

Web sites:
KOALA:

www.st.ewi.tudelft.nl/koala

DAS5:

www.cs.vu.nl/das5

Tribler:

www.tribler.org

5 September 2016

20

DS Tag Cloud

5 September 2016

21