Distributed Shared Memory

Outline

Introduction

Shared Memory vs. Distributed

Memory

Distributed Shared Memory

Architecture

DSM vs. Message Passing

Design and Implementation

Consistency Models

DSM Algorithms

Conclusion

Introduction

From the system interconnection perspective, parallel systems fall into

two main categories:
CP
CP
U
U
CP
CP
U
U

Shared Memory

Memor
y

CP
CP
U
U

CP
CP
U
U
Multiprocessors

Tightly-Coupled Multiprocessor
Multiprocessor
Shared Memory architecture
architecture

CP
CP
U
U

Memor
y

CP
CP
U
U
Network

Memor
y

CP
CP
U
U

Memor
y

CP
CP
U
U

Multicomputers
Loosely-Coupled
Distributed Memory

Shared Memory vs. Distributed Memory

Shared Memory

Distributed Memory

Global Address Space

No concept of global address space

Cache Coherent

No concept of cache coherency

Lack of scalability

Scalable performance

Expensive to build

Cost effectiveness: can use commodity, off-theshelf processors and networking

Easy to program, reusability

programmer is responsible for data

communication

Data sharing fast and uniform

Data sharing by message passing, non-uniform

memory access times

Cache coherent means if one processor updates a location in shared

memory, all the other processors know about the update.

Distributed-Shared Memory
Architecture
Definition

A Distributed Shared Memory DSM is an abstraction

that allows the physically separated memories to be
addressed as one logically shared address space.

General Characteristics
Hybrid architecture

Virtual Space shared between all processes

Shared Memory model implemented over

physically distributed memory
Shared-Memory programming techniques can be used
When reading and updating, processes see DSM as an ordinary
memory within their address space

Mapping Manager: Maps shared-memory

address to physical memory (remote or local)

P1

P2

M1

M2

Mn

MM

MM

MM

Pn

Interconnection
Network

Shared Virtual Space

Distributed-Shared Memory
Architecture (contd.)

General Characteristics
Covert Communication operations

Heterogeneous Nodes

The shared memory component can be a cache-coherent SMP

machine and/or a graphics processing unit GPU

Processes on different computers observe the updates made

by one another

Communications are still needed to exchange data, but they are hidden
from the programmer. Inter-process communication transparency

Cache-coherent

SMP or UMA (bus-based) : A model with identical processors that

have equal access times to a shared memory
UMA : Uniform Memory Access
SMP : Symmetric Multiprocessor

P1

P2

M1

M2

Mn

MM

MM

MM

Pn

Interconnection
Network

Shared Virtual Space

Distributed-Shared Memory
Architecture
(contd.)
Advantages

Implicit data sharing

Less expensive to build and scalable

Inherited from the distributed-memory architecture

Very large total physical memory for all nodes

Shields programmer from Send/Receive primitives

Large programs can run more efficiently

Software Portability and Reusability

Programs written for shared memory multiprocessors can be run on DSM systems with
minimum changes

Disadvantages

Little programmer control over actual messages being generated

DSM implementations use asynchronous message-passing

cannot be more efficient than Message Passing implementations

Distributed-Shared Memory
Architecture
(contd.)
Best Suitable

When individual shared data items can be accessed directly

e.g. Parallel Applications

Less appropriate
When data is accessed by request

e.g. Client-Server Systems

A server may be used to assist in providing DSM functionality for data

shared between clients

DSM vs. Message Passing

Property

DSM

Message Passing

Marshalling

No. Variables shared directly

Yes. Programmers job

Address space

Single. Interference may occur

Private. Processes are protected

Data representation

Uniform

Heterogonous

Synchronization

Normal construct for

shared-memory programming

Message passing primitives

Process execution

Non-overlapping lifetimes

At the same time

communications cost

Invisible

Obvious

No evidence against or in favor to any of the two

communication mechanisms

Design and Implementation

Main Issues
Granularity refers to the size of sharing unit that can be uniform chunks of memory or data structures

Structure refers to the arrangement of shared data

Most systems view DSM as a linear array of words

Replacement Strategies

byte, page or complex data structure

Small Pages : increased parallelism
increase in directory size
Large Pages : reduce paging overhead, but increase sharing overhead

Similar to caching mechanisms in MP

In cache systems, LRU is often used
In DSM, shared pages need to be given higher priority than exclusively owned pages
first

Synchronization Primitives

Coherence protocols must ensure the consistency of shared data

DSM must allow simultaneous access to shared data on different machines

single writer, multiple readers, etc.

they could be replaced

Consistency Models
Definition

A memory consistency model for a shared address space specifies constraints on the order in which
memory operations must appear to be performed (i.e. to become visible to the processors) with respect to
one another.

Strict Consistency Model

Any read to a certain memory location returns the value stored by most recent write
operation to that address, irrespective of the locations of the processors performing the
read and the write operation.

Sequential Consistency Model

if the result of any execution is the same as if the operations of all processors were
executed in the same sequential order, and the operations of each individual processor
appear in this sequence in the order specified by its program. (Leslie Lamport)
Definition restated: Sequential consistency requires that a shared memory multiprocessor
appears to be amultiprogramming uniprocessor system to any program running on it.

All instructions are executed in order

Every write operation becomes instantaneously visible throughout the system

Consistency Models (contd.)

Sequential Consistency
Model

Example 1
P1

P2

Data = 2000
{}

while (Head == 0)

Head = 1

= Data

Sequential consistency requires program

order
The write to Data has to complete before
the write to Head can begin
The read of Head has to complete before
the read of Data can begin

Example 2
Initially A = B = 0
P1
A=1

P2

P3

if (A == 1)
B=1
if (B
== 1)
register = A

Sequential consistency can be had if a

process makes sure that everyone has seen
an update before that value is read -

Consistency Models (contd.)

Causal Consistency Model
Writes that are potentially causally related must be seen by all processors in the same order .
Writes that are not potentially causally related may be seen in a different order on different
machines
Processor Consistency Model

Writes done by a single processor are seen by all other processors in the order in
which they were written on that processor.
Writes from different processors may be seen in a different order by different
processors.

Release Consistency Model

Weak consistency with two types of synchronization operations : acquire and release
Each type of operations is guaranteed to be processor consistent

DSM
Algorithms

Server

The Central Server Algorithm

Central Server maintains all shared data

Read Request The server just returns the data

Write Request update the data and send acknowledgement to the client

Two messages for each data access

Implementation

If an applications request to access shared data fails repeatedly, a failure condition is sent to the application

Issues: performance and reliability

Client

A timeout is used to resend a request if acknowledgment fails

Associated sequence numbers can be used to detect duplicate write requests

bottleneck at the server

Possible solutions

Partition shared data between several servers

Use a mapping function to distribute/locate data

DSM Algorithms (contd.)

The Migration Algorithm
Data is always migrated to the site where it is accessed

Allow only one node to access a shared data at a time

Migration Request

Single Reader / Single Writer SRSW protocol

Data is typically migrated between servers in a fixed-size unit called a block

Facilitate the management of data instead of migrating individual data units

Advantages
Takes advantage of the locality of reference

No communication costs are incurred when a process accesses data held locally

Data Block

DSM can be integrated with the virtual memory of the OS at each node
-

The size of the block is chosen to be equal to a virtual memory page or a multiple thereof
A locally-held shared memory page can be mapped into the applications virtual address space

Access to data items on data blocks not held locally triggers a page fault

Normal machine instructions for accessing memory can be used

the fault handler can communicate with the remote hosts to obtain the requested data.

When a data block is migrated away, it will be removed from any local address space it was mapped to

To locate a remote data object:

Use a location server
Broadcast query
Issues

Pages can thrash between hosts: to minimize it, set minimum time for data objects to reside at a node

DSM Algorithms (contd.)

The problem with the previous techniques is the sequential access
to the data block
The Read-Replication Algorithm
Extends the migration algorithm

Replicates data blocks at multiple nodes for read access

Replication can reduce the average cost of read operations

Multiple nodes can have read access or one node write access

Block Request

multiple readers-one writer MRSW protocol

After a write, all copies are invalidated or updated

DSM has to keep track of locations of all copies of data objects

IVY The owner node of data object knows all nodes that have copies

PLUS Distributed linked-list tracks all nodes that have copies

Data Block

Multicast invalidate

Advantages

The read-replication can lead to substantial performance improvements if the ratio of reads to writes is
large

Disadvantages
Write operations might be more expensive since replies may have to be invalidated or updated to maintain
consistency

DSM Algorithms
(contd.)
The Full-Replication Algorithm

sequencer

Extension of read-replication algorithm

Multiple nodes have both read and write access to shared data blocks

multiple-readers multiple-writers MRMW protocol

write

update

Issues

Consistency of data for multiple writers

Solution

use of gap-free sequencer

Client
s

All writes sent to a sequencer

Sequencer assigns sequence number and sends write request to all sites that have copies

Each node performs writes according to sequence numbers

A gap in sequence numbers indicates a missing write request: Node asks for retransmission of missing write
requests

DSM Algorithms
Performance Measure

It needs to take into account the cost of

accessing local and remote data blocks

Parameters
p: cost of sending or receiving
a short packet
P: cost of sending or receiving a data
block, assume P/p equal to 20

S:

r: Read/Write ratio
f: probability of an access fault on a

number of sites participating in

distributed shared memory

non-replicated data block

f `:

probability of an access fault on

replicated data blocks

Conclusion

Being a hybrid of the distributed and shared memory architectures, DSM

systems offer a trade-off between the easy-programming of shared
memory machines and the efficiency and scalability of the distributed
memory systems.

While the programmer is relieved from the communication details, he still

has to take care of many design and implementation issues. The
algorithms mentioned above offer various solutions with cost and
performance varying for each.
No single algorithm is good for all applications.
Algorithms need to be adaptive to application characteristics

Thank you for Paying Attention