Fault Tolerance in Distributed Systems

Yevgeniy Gershteyn
Department of Computer Science
Rochester Institute of Technology
Rochester, NY, USA
yxg7752@cs.rit.edu

February 5, 2003

Abstract constraint of the dependable system is reliability.

This paper reviews fault-tolerance aspects in
distributed systems. Due to possibility of partial
failure in the system, the system should routinely
recover itself from it without disturbing the
overall performance of the system. There exist a
lots of researches how to make distributed
system fault-tolerant. In this paper I analyze
some ideas on different aspects of fault
tolerance, such as new research on corporative
leasing and scalable consistency maintaince in
context distributed networks; improving fault-
tolerance by replicating agents; and problem
specific branch-and bound algorithm for
asynchronious distributed systems.
It is quite different from availability in term of
“that a system can run continuously without
failure” [5] for particular amount of time. Safety
is very important in case of using control
systems, where any failure can cause disaster for
surrounding environment. At last,
maintainability shows level of how fast system
can be repaired and brought back to work.
There exist several techniques to design
fault tolerant systems, such as leasing,
replication. In particular, leasing is kind of
convention between client and server for a
limited time.

1 Introduction 2 Cooperative leases: scalable

“An important goal in distributed systems design
is to construct the system in such a way that it
can automatically recover from partial failures
without seriously affecting the overall
performance” [5(p361)]. The partial failure is the
key problem of the distributed system, which is
differentiate it from a singe processor system.
Distributed systems are designed to work from
different locations or in other words from a
different processors, and each processor may go
off unexpectedly or be disconnected from the
network. In this case designer of the distributed
systems should build system that controls
failures or even better tries to prevent them. The
fault tolerant systems are close related to the
description of the dependable systems, which are
enclosed by following requirements [4]:
availability, reliability, safety, and
maintainability. In particular, an availability
requires from any component be ready to use at
any time. Failure of this constraint might cause
failure of the entire distributed system. Another
consistency maintenance in
content distribution networks.
This article informs about results of the research
which was made with affiliation of the IBM,
Intel, EMC, and Sprint at the University of
Massachusets. The new terms of “cooperative
consistensy along with a mechanism, called
cooperative leases”[1] were introduced. “By
supporting ∆-consistency semantics and by using
a single lease for multiple proxies, cooperative
leases allows the notion of leases to be applied in
a flexible, scalable manner to CDNs”[1]
(Content Distribution Network). The major point
is to divide information stream into different
level of consistency, which depends on level of
actuality of information. In this case, depends on
requirements of the user, more important
information can be obtained faster, rather then in
the same cache as all objects. For example for a
financial company, financial information is more
important than sport scores. This partitioning
may bring efficiency and flexibility of any type
of information.
The authors proposed ∆-consistency
semantics undertaking that all data is up-to-date
as server has it. By adding extra parameter to
original leases approach, authors assume that if
user specifies object update rate, than server will
inform client in certain interval of time. This
brings differentiate server responds to the clients
and make them receive updated information at
requested time. Another modification is “allow a
server to grant a single lease collectively to a
group of proxies, instead of issuing a separate
lease to each individual proxy. For each cached
object, the proxy group designates an
invalidation proxy, referred to as the leader”[1].
The leader interracts with the server about the
leasing, and make it from all proxies in the
group. It brings some downsides of this solution.
First, there is bottleneck, which appears then one
proxy is in use, and second is that leader is
responsible for broadcasting this information to
the group. However, it has some advatages in
this solution, which are: “(i) it reduces the the
amount of state maintained at a server (by using
a single lease to represent a proxy group instead
of an individual proxy); and (ii) it reduces the
number of notifications that need to be sent by
the server (by offloading some of notification
burden to leader proxies)”[1]. Moreover, authors
did not mentioned that leader become a single
point failure, and it needs some kind of
replication techniques for leader of the group, or
every process should be able to be a leader. This
involves additional overhead for each client,
since they have to be able to multicast.
I might sound sceptic, but to prove this
theory, authors used self implemented simulated
environment. There is no guarantee that
simulator performs as real networks would.
However, let’s look on expiremental results. The
“results show a factor of 2.5 reductions in server
message overhead and a 20% reduction in server
state space overhead when compared to original
leases albeit at an increased inter-proxy
communication overhead”[1]. So, in proposed
configuration has a lease server which is
responsible for lease processing. The clients have
lease handler to act as a leader or as a client. As a
result authors claim that theirs new word in fault
tolerance “cooperative leases meets the goals of
flexibility and scalability by (i) employing -
consistency semantics, (ii) using a single lease to
represent multiple proxies and (iii) using
application-level multicast to propagate server
notifications”[1].
3 Improving fault-tolerance by
replicating agents.
Replication of services leads to increase level of
complication of the system. This article
examines “the use of transparent agent
replication, and technique in which the replicates
of agents appear and act as one entity thus
avoiding an increase in system complexity and
minimizing additional system loads.” Another
point is “inter-agent communication, read/write
consistency, resource locking, resource synthesis
and state synchronization. An implementation of
the transparent agent replication for the FIPA-OS
framework is presented and the results of testing
it within a real-world multi-agent system are
shown” [2]
Multi-agent systems (MASs) appropriate
solution for development difficult, distributed
systems, but they are still vulnerable to any kinds
of faults that any distributed system may have.
“The modular nature of a MAS gives it a certain
level of inherent fault tolerance, however the
non-deterministic nature of the agents, the
dynamic environment, the inter connectedness of
the agents and the lack of any central control
point make it impossible to foresee possible fault
states and make fault handling behavior
unpredictable”[2]. Since that, entire system may
crash just because only one agent had a small
error. The authors made their assumptions based
on the open environment system, where agents
are not reliable and might not be fault-tolerant.
“In an open system, agents may be malicious,
poorly designed or poorly implemented, hosts
may get overloaded or fail, network connections
may be slow or down”[2].
“There are four essential characteristics
of a multi-agent system: a MAS is composed of
autonomous software agents, a MAS has no
single point of control, a MAS interacts with a
dynamic environment, and the agents within a
MAS are social (agents communicate and
interact with each other and may form
relationships)” [2]. Moreover, there exist various
fault types, which can bring wide system crash.
These faults can come from program defects,
sudden changes, processor and communication
fault, as well as emerging actions, which are not
predicted. In addition, there are many techniques
to have replication agents, and they either agent-
centric or system-centric. The major differences
between them is that system-centric is more
complex structure, but it can catch system-wide
faults. The mixture of these two types of agents
is classified as a Dependable Mobile Agent
System. However, both of the techniques uses
agent replication aspects, and "the redundent
replicates and any proxies are external to original
agent, using a system centric approach, but
individual agents must have the capability to
utilize replication" [2].
Agent replication is the act of creating
one or more duplicates of one or more agents in
a multi-agent system" [2]. Each of these agents
may perform the same task as the original agent.
Duplicate agents form replicate group and each
member is named as replicates.
As soon as a replicate group is formed and it is
viewable by all members of the MAS, "there are
several ways that agents can interact with it:
• An agent can send requests to each replicate in
turn until it receives an appropriate reply.
• An agent can send requests to all replicates and
select one, or synthesize the replies that it
receives.
• An agent can pick one of the replicates based
on a particular criteria (speed, reliability, etc.)
and interact only with that agent" [2].
There exist heterogeneous and
homogeous replicatios. The major difference
between two of them that first one has separate
implemention then primary agent, and second
one not. However, they have exact copies of
original agent. Homogeous replication may have
same faults as original agent, because it has the
same implementation, and defects of
implementation which were left behind in there
might be catch in all replicas. On other side
heterogeneous replication in case of the fault of
the original agent may continue operating.
Another useful technique to have replicas with
diferent versions of the applications, if
something wrong on new version, which were
supported on previous one, that previous version
replica can take over original agent. As soon as,
group is constucted and operating under MAS,
there are some ussies that may arrise, such as
"inter-agent communication and result synthesis,
read/write consistency, and state
synchronization. In this case, there possibility of
that system may fail at any point. The authors
introduce transparent replication which uses
proxy to communicate between replicas and
clients. So, "proxies provide two important
functions: they make the replicate group appear
to be a single entity and they control execution
and state management of a replicate group" [2].
In details, proxy is providing "communication
between replicates and other agents in the MAS"
[2]. In this case, proxy become single point of
failure, which requires backups of itself and if
one proxy goes down another and another can
replace main proxy's functionalities. This brings
another overhead for this solution, but it pay of
by improving reliability of the system.
Another proxy to exchange data
between replicas, and this proxy makes sure that
all members receive equal data. There is some
downsides of this part such as it become single
point of failure solution. Also, it may cause time
delay between data receieved and data backed up
to the replicates, and they become outdated.
"The other role that proxies can fill is
management of the replicate group. Through
various replicate group management policies,
some of the replication issues can be dealt with.
Three replication management policies are
identified: hot-standby, cold-standby, and active.
" [2]. Depending on policy system desides which
proxy will start after current working proxy goes
down. As well as proxies may "improve
performance managment of the replicate group"
[2].
“This paper introduced the topic of
agent replication and examined the issues
associated with using agent replication in a
multi-agent system. Transparent proxies were
introduced as a method of dealing with the main
issues of agent communication, read/write
consistency and state synchronization”[2]. At
University of Saskatchewan, Saskatoon, Canada
these ideas were implemented and I-HELP MAS
is a working system "for the replication server.
The FIPA-OS [6] agent toolkit was chosen a
platform for the implementation" [2].
4 A Problem-Specific Fault-
Tolerance Mechanism for
Asynchronous, Distributed
Systems.
This article presents new theory on fault-tolerant
mechanism for distributed systems, which is
based on research of different kind of networks,
such as LAN, WAN. Since these networks are
providing various services over the network, it
makes communications between the users
untrusted. So, authors designed the algorithm,
which provides reliable services and scalability
of the resources. In fact, scalability is achieved
by “a fully decentralized algorithm, in which the
dynamically available resources are managed
through a membership protocol” [3]. On other
side, fault tolerance is assured in meaning of
“that the loss of up to all but one resource will
not affect the quality of the solution” [3].
There was “branch-and-bound search
algorithm” used to solve this problem. This
algorithm “is intelligent search method often
used for optimization problems” [3], such as NP-
hard problems. The original problem is
decomposed to smaller problems, and then start
to solve them. In this algorithm, order of the
operations is important. Firstly, in decompose
stage problem splits “into a set of new
subproblems. “ [3] After decomposition problem
is branched. So, next step is branching, during
which “a bound value l(v) on the optimal
solution of subproblem v. This bound value will
be used by Select and Eliminate operations” [3].
During select operation, subproblem is verified
that new bound value is better that previously
known one, if it is than it stores “into the pool of
the active problems”. [3] Otherwise this solution
is eliminated. “The best-known solution is
updated when better feasible solution is found”
[3]. The best performance can be obtained, if
most of operations can be done simultaneously,
and there are some approaches, such as
synchronous or asynchronous, work sharing, and
information sharing mechanisms. Synchronous
design underlying that all processes will wait
each other to complete, and not to wait in the
asynchronous algorithm. Work sharing technique
allows monitoring all processes and assigning
any available slots, so these processes may work
in parallel. Information sharing mechanism
based on storing all information about all known
sharing solutions and its decision is the best-
known solution for this kind of problem.
The internet connected computes were
chosen as target architecture to prove branch-
and-bound search algorithm. This architecture
gives all resources for distributed systems, such
as [3]: scalability, dynamic availability,
unreliability, communication characteristics,
heterogeneity, and lack of centralized control.
Prove is made with some assumptions, but it still
does not fully brake practical point of the
algorithm. However, in my opinion, any
assumptions make any prove or any algorithm
not real and some kind of artificial.
Let us look on algorithm with more
details. “Fully decentralized, asynchronous, fault
tolerant parallel B&B algorithm” [3] is
considered to meet system requirements.
“Each process maintains its local pool of
problems to be solved. When the local pool is
empty, the process sends work requests to other
processes. A process that receives a work request
and has enough problems in its pool removes
some of those problems and sends them to the
requester.” [3] This describes dynamic design
which was chosen by algorithm designers. As
usual dynamic methods provide better
performance and consistency, and in here all
processes update data with better solutions for
future reference and faster decision making
result. However, to make this solution more
reliable some extensions were invented, such as
“a group membership protocol to allow dynamic
variation in the number and components of
resources and a fault tolerance mechanism” [3].
The group membership protocol is here
for “collecting and updating information about
which resources participate in the computation at
any given time” [3]. This works really simple.
When new process or computer comes to the
assigned group, it sends request to join the group
to known server, which controls this group and
knows if anyone from this group is present. If
not, the group is initialized, and on other hand if
group is active, the server just simply joins this
member to this group. On other side when
process lives the group by any causes, such as
self termination or even failure membership
server is aware of these kind processes. As a
result, the system always knows who is where
and what status of the systems’ components are.
Moreover, the server is able to log all activities
of all members, and make sure that process is not
in timeout, due to some inactivity period of it.
The fault-tolerant mechanism does not capture
“failures of computers and restore their data, but
rather focuses on detecting missing results” [3].
This can be obtained by getting placed problem
on the particular place as a node in the tree, and
by calculating code for each location in the tree.
As a result, each node can be identified by
unique code. “Furthermore, given a set of nodes
of the tree, we can easily find its complement,
that is, the list of nodes of the tree that are not in
the given set” [3].
Algorithm was proved using simulation
of the real systems. This simulation, as authors
says, “provides great flexibility in testing a wide
range of B&B strategies in a variety of Internet-
like environments” [3]. Even though,
experiments were made using “small problems”,
result showed that overall performance rate
increased. During these experiments,
implemented simulator provided different
information, such as “execution time,
communication costs, and storage space” [3].

As a result of experiments authors establish “a
failure-recovery mechanism suited for a tree-like
problem space. This mechanism and a low cost
group membership protocol are the ingredients
that transform a rather conventional parallel
branch-and-bound algorithm into a scalable,
reliable, more powerful algorithm, able to exploit
the computational power of hundreds of Internet-
connected resources. Scalability is achieved
through a fully distributed design. The algorithm
is fault tolerant under our assumptions and can
execute and terminate correctly even if only a
single resource remains available” [3].
Moreover, they found solution for “difficult
problems of fault tolerance and termination
detection in distributed environments by
exploiting problem-specific features, specifically
the tree structure of the problem space” [3].
5 Conclusion
In this paper, I made research on some fault
folerance aspects of the distributed systems.
Since I like practical solutions more than
theoretical researches, I tried to analize them
from a practical point of view, and in some cases
these theretical ideas really good, but the
assumptions which were made really smplify the
problem. Anyway, all these reseaches were made
and some results showed that there is big
opportunity to improve the distributed systems
from the fault tolerant perspective. The major
point of all is to make system functioning even if
any part of it goes off. This would make a
system,especially distributed, available, reliable,
safe, and easy maintainable. Since, taking off
one element of the system will not stop its
functions, and during this down time that
element can be improved to make system more
secure or make some updates to it. From these
three papers, my preference was the second one
about improvement for replicating agents. It
makes system more reliable, even if you pay
higher price to achieve it. For example the first
paper, tries to improve distributed system by
inventing new type of leasing, and do not
mention about fault-tolerant issues for the piece
of hardware added to the system. In this case
leader proxy becomes another piece of a single-
point failure. The third paper, is more theoretical
touch in distributed systems, but it still has
powerful canvas for future development. I think,
that my reseach in this topic of the distributed
systems, opened eyes for me, and showed that
there are still holes, and there are ways to
improve existed systems, or even design new
ones based on them.
References
[1] A. Ninan, P. Kulkarni, P. Shenoy, K.
Ramamritham, R. Tewari Cooperative leases:
scalable consistency maintenance in content
distribution networks. Proceedings of the
eleventh international conference on World Wide
Web, Honolulu, Hawaii, USA, pages: 1 – 12,
2002
[2] A. Fedoruk, R. Deters Improving fault-
tolerance by replicating agents. In Proceedings of
the first international joint conference on
Autonomous agents and multi agent systems:
part2, Bologna, Italy, pages 737 – 744, 2002.
[3] A. Iamnitchi, I. Foster A Problem-Specific
Fault-Tolerance Mechanism for Asynchronous,
Distributed Systems. Proceedings of the 2000
International Conference on Parallel
Processing, Toronto, Canada, pages 4 – 13,
August 2000
[4] H. Kopetz, P. Verissimo. Real Time and
Dependability Concepts. In S. Mullender.
Distributed Systems, pp 441-446. Wolkingham:
Addison-Wesley, 2nd ed., 1993.
[5] A. Tanenbaum, M. van Steen. Distributed
Systems: Principles and Paradigms. Prentice
Hall, 2002.
[6] FIPA-OS. http://fipa-os.sourceforge.net,
retrieved February 05, 2003.