Qualication of Formal Safety Assessment:

an exploratory study
Tony Rosqvist
a,
*, Risto Tuominen
b
a
VTT Industrial Systems, PO Box 1301, 02240VTT, Espoo, Finland
b
VTT Industrial Systems, Tampere, Finland
Abstract
Formal Safety Assessment (FSA) is an approach adopted by the International Maritime
Organisation (IMO) to support a systemic and structured assessment of proposals for new
international regulations to improve shipping safety. Several case studies based on the
approach have been conducted during recent years. This paper addresses the issue of con-
dence in a FSA, as encountered during three case studies conducted by the authors over the
years 19982002. A peer review processFSA qualicationis introduced to support the
consolidation of condence in FSA results. Some qualication criteria are suggested.
# 2003 Elsevier Ltd. All rights reserved.
Keywords: Formal safety assessment; FSA; FSA qualication; Risk-informed decision-making; Pre-
cautionary decision-making; Expert judgement
1. Introduction
The business of shipping is governed by a multitude of statutory regulations that
support ship safety at both national and international levels. The safety-related rules
have predominantly been prescriptive, quite often derived as a reaction to a major
incident at sea in order to prevent similar accidents from occurring again. This tra-
ditional approach to rule making has lead to a large number of separate rules.
Typically, the cost-eectiveness of a new rule and its compatibility with other rules
has not been assessed in a formal and consistent way in the traditional rule-making
process. Consequently, rules have been introduced over the years, which are not
optimal, and some may even be considered questionable regarding their overall
inuence on ship safety.
Safety Science 42 (2004) 99120
www.elsevier.com/locate/ssci
0925-7535/03/$ - see front matter # 2003 Elsevier Ltd. All rights reserved.
doi:10.1016/S0925-7535(03)00005-5
* Corresponding author. Tel.: +358-9-4566773; fax: +358-9-4566752.
E-mail address: tony.rosqvist@vtt. (T. Rosqvist).
After some years of discussion and debate about the need for a more systemic,
proactive, and risk-informed regulatory regime in shipping, the International Martime
Organisation (IMO) Maritime Safety Committee (MSC) in 1995 decided to adopt the
concept of Formal Safety Assessment (FSA). This was done in the hope of improving
the IMO rule-making process, and thus further enhancing the safety of shipping. As
stated by Wang (2001), it is considered that Marine safety may be signicantly
improved by introducing a formal goal-setting safety assessment approach so that the
challenge of new technologies and their application to ship design and operation may
be dealt with properly. For a more specic discussion on the expected benets of the
FSA as a regulatory tool, and as a potential framework for safety assurance in shipping
companies, the reader is referred to MSA (1993), Wang (2001) and Peachey (2002).
Following the development and introduction of the FSA method, interim guide-
lines for FSA application were issued by IMO (1997a) to describe and explain the
new method and to support its application and further development in practice.
Since that time, several FSA trial applications and case studies have been carried
out in various IMO member states around the world. Some of the studies have been
issued in direct support for the formulation of new IMO safety regulations (e.g.
DNV, 1997a; IMO, 1997b, 2000a,b), while in some other studies the objective has
been to provide the justication for rule amendments or provisions allowing
deviation/exemptions from a particular rule (e.g. DNV, 1997b). Other studies have
been performed at the national level, for example, in order to support risk decision-
making of the national maritime authorities in specic local applications (eg. Rosq-
vist et al., 2001), or to support the implementation of specic safety measures locally
(eg. Nyman et al., 2002). For a further discussion on ways of FSA applications, see
Wang (2001).
The IMO Formal Safety Assessment method (IMO, 1997a) comprises of ve
consecutive steps as shown in Fig. 1. Complementary discussions of the FSA process
can be found, for example, in Peachey (1997) and in Wang (2001).
In this paper, each step of the FSA-method is addressed from the point of view of
FSA qualication. We give the following denition: FSA qualication means the
procedure of consolidating a decision-makers condence in the results and the recom-
mendations of a FSA. The paper is motivated by the fact that the IMO guidelines
(IMO, 1997a) do not explicitly address the issue of Quality Assurance related to
FSA studies. Especially, the roles of the stakeholders and experts in providing qua-
litative and quantitative information are crucial with respect to the quality of the
FSA. Their importance, as well as the importance of a more formal FSA qualica-
tion procedure, is reected in the observation by Wang (2001): Lack of reliable
safety data and lack of condence in safety assessment have been two major pro-
blems in safety analyses of various engineering activities. Suokas and Rouhiainen
(1993) have introduced a concept systematic quality management in the context
of safety and risk analyses in chemical and process industries. In the nuclear energy
eld quality issues of Probabilistic Safety Assessment (PSA), in relation to harmo-
nising of PSAs, have been addressed in Holmberg and Pulkkinen (2001).
Table 1 lists a set of qualication issues which suggest qualication criteria for
each step of the FSA method. The importance of the issues have been recognised
100 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
during three FSA applications conducted by the authors (Rosqvist et al., 1998, 2001;
Nyman et al., 2002). Some ideas for qualication criteria are found in Holmberg and
Pulkkinen (2001) and Guida et al. (1996). An additional step, Step 0: FSA Scope
and Objectives, has been added in the FSA process. In this step, the norms for
evaluation are established, i.e. the decision rules and decision criteria for acceptance/
rejection and ranking (Holmberg et al., 2002). Because this is a crucial step directing
the FSA work, the authors wish to see it as an integral part of the basic FSA method
(see Fig. 1). In Section 3, after a short review of concepts related to risk assessment,
each issue brought up in Table 1 is separately discussed from the point of view of
FSA qualication.
2. Concepts related to risk assessment
2.1. Denitions
System A set of specied functions and elements interacting with
each other and with the environment to full a purpose.
Fig. 1. The analysis steps of the FSA method according to IMO (1997a).
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 101
Hazard Source of potential harm or a situation with a potential harm.
Hazardous event Event, which can cause harm.
Risk Combination of the frequency or probability of occurrence and
the (negative) consequence of a specied hazardous event.
Decision-maker Actor evaluating the need of FSA for a decision-problem and
evaluating the acceptance of the FSA results and recommendations
based on the condence in the FSA process.
Risk-analyst Normative expert who guides through the steps of the FSA and
develops the risk model.
Expert Domain expert who provides data and information related to
some specic area of concern in any of the FSA steps.
Stakeholder A representative of an interest group providing opinions and
information related to preferences.
2.2. Notation
R x
k
P X > x
k
Risk is the probability of negative consequences exceeding a
specied value x
k
during system lifetime, or within a particular
operating time-frame (e.g. one operating year). The probability
distribution of negative consequences.

i
f g Set of discrete system-states
i
; i 1; . . . ; n f g
x
j
_ _
Set of discrete negative consequences x
j
; j 1; . . . ; m f g
Pr
i
Probability of system-state
i
Pr x
j
_ _
Probability of negative consequence x
j
2.3. Bias, conservatism and precaution in risk assessment
A risk model may be viewed as a model of uncertainties related to the prediction of
negative consequences of operating a system (Nielsen and Aven, 2003). Uncertain-
ties stem from the lack of statistical data and inadequate knowledge about func-
tional and stochastic relationships of system-states, some of which may be
hazardous. If a risk model is also viewed as a mathematical representation of the
system under study, it is a source of uncertainties as well (Zio and Apostolakis,
1996). Furthermore, the system may experience changes over time, changing the
uncertainties that we perceive with respect to hazards.
A system may experience a wide range of system-states during its lifetime. From the
point of view of risk assessment, the hazardous system-states (or events) should be
identied and represented in a risk model. By applying qualitative arguments or deter-
ministic analyses, such as structural analysis, stress analysis, etc. we may derive risk
bounds, either optimistic or conservative, relative to the true risk of the system. It has
to be noted that the level of discretisation (number of system-states considered) is a
modelling choice, contested by alternative discretisation options. Thus, for instance, the
system-state non-severe grounding of a tanker of medium size would actually include
102 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
a range of grounding events (scenarios) from barely noticeable to a near loss of
containment type of grounding (of a tanker of medium size).
Each discrete, and mutually exclusive, system-state
i

i
2 would be associated
with a probability Pr
i
. Furthermore, each hazardous system-state would be asso-
ciated with uncertain negative consequences (damages) X, that may also be dis-
cretised into consequences x
j
, each associated with a probability given the condition

i
; Pr x
j
j
i
_ _
. For instance, to estimate the risk of cargo displacement during a sea
journey, we need to discretise the weather and cargo into specic weather and cargo
conditions (system-states) represented by certain wind speeds, wind directions, wave
types, rain/no rain, cargo type, etc. and to assign a probability for their combined
occurrences. After that we need to assign probabilities to the negative consequences,
given the particular weather and cargo condition.
The risk, R x
k
, can be formulated as
R x
k
P X > x
k

_
2
_
x>x
k
dP xj dP

n
i1
_
x>x
k
dP xj Pr
i
Dr
0
x
k
;
1
; . . . ;
n
f g

n
i1

m
fj:x
j
>x
k
g
Pr x
j
j
i
_ _
Pr
i
Dr
00
x
k
;
1
; . . . ;
n
f g; x
1
; . . . ; x
m
f g 1
Table 1
FSA qualication issues addressed in the paper corresponding to the steps of a FSA study
FSA step Qualication issue
0. FSA Scope and Objectives Decision rules and decision criteria
Is the decision-making rationale explicitly
formulated for transparency?
1. Hazard Identication Stakeholder/expert survey
Are the views of completeness surveyed and analysed?
2. Risk Assessment Expert judgement protocol
Is the elicitation and use of expert judgement credible?
3. Risk Control Options Stakeholder/expert survey
Are the views of completeness surveyed and analysed?
4. CostBenet Assessment Expert judgement protocol
Is the elicitation and use of expert judgement credible?
Stakeholder analysis
Is the distribution of benets and costs addressed
from the point of view of fairness?
5. Recommendations for the Decision-Maker FSA qualication process
Is the FSA quality assured?
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 103
where r
0
(.) and r
00
(.) reect biases dependent of the discretisation of the system
{
ri
} and the consequence parameter x
k
. The Eq. (1) is formally similar to the additive
adjustment-factor approach by Zio and Apostolakis (1996), where the starting point
for risk modelling is the specication of a single best model, based on expert
deliberation of the system, its processes and structural properties. The bias terms in
Eq. (1) are argued to represent model uncertainties in Zio and Apostolakis (1996).
Nielsen and Aven (2003) point out, however, that the term can be viewed as a model
parameter and that it may be unfeasible to quantify it. In the predictive Bayesian
approach (Nielsen and Aven, 2003) all uncertainties are epistemic, related to the
quantities (parameters) of the risk model. It is, however, unclear, how model
uncertainty is addressed during the deliberation of the risk model, and how the
arguments supporting the choice of a certain risk model is communicated to the
decision-maker, in the approach.
The earlier characterisation of biases as optimistic or conservative reects an
underlying assumption of knowing the direction of bias related to a risk mod-
elling assumption, and thus knowing whether the true risk is under- or over-
estimated. In this respect, we will depart from the predictive Bayesian approach
to risk assessment as described in Nielsen and Aven (2003). We will, however,
agree with Nielsen and Aven (2003) that risk management is a matter of
uncertainty reduction and management, based on an analysis of dierent
types of uncertainties and their impact on used decision rules and criteria.
Furthermore, we will show that the bias terms in Eq. (1) do not need to be
quantied, as suggested in Zio and Apostolakis (1996), and adopt the view by
Nielsen and Aven (2003) that they may not even be feasible to quantify in practical
risk analysis.
From the point of view of risk-informed decision-making, it is important to dis-
tinguish two basic types of uncertainties encountered in risk assessment: aleatory
and epistemic uncertainty (USNRC, 1998). Means to resolve these uncertainties are
dierent, as aleatory uncertainty is related to the intrinsic variability of the system
(stochastic property of a system/population variability), and is resolved by system
re-design, whereas epistemic uncertainty pertains to our knowledge and is resolved
by further research. Thus, the uncertainty types in constitutive models (quantity-
oriented/physical or event-oriented/logical) depicting the relationships x
j
(y
i
) should
be kept separated for proper risk interpretation and decision-making. Refer to Bier
(2001), Nielsen and Aven (2003) for complementary views.
From the point of view of precautionary decision-making (Stirling et al., 1999)
conservative modelling assumptions with respect to risk are preferred to optimistic to
ensure that the system does not satisfy an acceptance criterion (a threshold risk
level) falsely. The opposite outcome, i.e. that the system is classied as unacceptable,
when in fact, it is acceptable, is possible in such a case. Thus, the direction of bias is
sucient information of the bias terms in Eq. (1).
At this point we can dene the objective of FSA qualication in more technical
terms: FSA qualication means the procedure of consolidating a decision-makers
condence in the precautionary application of the decision rules and criteria. The
denition of the sets
i
f g, x
j
_ _
, and the estimation of the probabilities Pr
i
and
104 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
Pr x
j
j
i
_ _
, are essential risk modelling choices and tasks, and thus the prime subjects
for qualication.
3. FSA qualication
3.1. Qualifying FSA step 0: FSA scope and objectives
3.1.1. Objective of FSA step 0
The scope and objectives of any FSA study must support the decision-makers
rationale (e.g. precaution) related to risk control and safety improvement. This
rationale is reected by the choice of decision rules and decision criteria.
3.1.2. Types of decision rules and decision criteria
Basically, the selected decision rules and criteria determine the risk modelling
approach. For instance, whether qualitative methods are adequate or not is deter-
mined by the decision criteria utilised in the decision rules. Decision rules combine
measurements and decision criteria to yield an ordering of dierent decision options,
whereas decision criteria are dened as point of views or objectives that specify the
scope of a risk assessment. The decision criteria generally encompass risk criteria, as
well as, for example, economic criteria. Table 2 lists decision rules and criteria for
classifying and ordering of system and risk control options. A discussion on the
scope of a risk assessment is found in Holmerg et al. (2002).
The ALARP-principle or decision rule is perhaps the most commonly adopted
decision rule in risk-informed decision-making. In certain cases it seems, however,
that the decision-makers feel uncomfortable with absolute risk levels, and rather
prefer to make risk comparisons using an existing system as a risk reference (e.g.
Rosqvist et al., 2001). In this case, the decision rule Risk Reduction serves the needs
of the decision-makers better than the ALARP-principle. The combination of the
decision rules Risk Reduction and Cost/Benet reects the rationale of incremen-
tally improving safety over time within some economic constraints. It is important
to note that a biased risk estimate R
^
x
k
may lead to applying a decision rule in
an way that is not precautionary. It is therefore important that uncertainty and
sensitivity analyses are carried out to assess the impact of biases on the decision
outcome.
1
1
For instance, the Risk Reduction decision rule implies that the criterion r
o
=r
n
> 1 is satised, where
r
o
, r
n
denote the estimated risk of the old and the new system, respectively. Denote the true values by r
o
0
and r
n
0
, and the additive model uncertainties by r
o
, r
n
. The earlier condition can now be rewritten as
r
0
0
=r
0
n
> 1 Dr
n
Dr
o
=r
0
n
. If Dr
n
< Dr
o
, it is possible to accept a new system that is more risky than the
old one. Another example is the ICAF criterion: c
n
= r
ref
r
n
_ _
< , where c
n
, r
ref
, are the investment
cost, a xed (for simplicity) risk reference level, and the ICAF acceptance level, respectively. The criterion
can now be rewritten as c
n
= r
ref
r
0
n
_ _
< c
n
Dr
n
= r
ref
r
n
_ _
r
ref
r
0
n
_ _
. Any model uncertainty r
n
>0
may prohibit risk control measures to be taken as the realised acceptance level is in this case less com-
pelling than the formal acceptance criterion.
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 105
The FSA itself may be subject to quality considerations. The perceived quality of
a FSA study certainly aects the decision-maker(s) condence in the results and
recommendations provided by it. Typical questions that are raised are: Is the scope
of the FSA complete?, Is the way of analysis and the logic of inference credible?,
and Is it possible that the biased risk estimates lead to unjustied decisions? Based
on the answers to such questions, the decision-maker seeks evidence whether a FSA
study is sound, or not.
The interdependencies between the various decision rules and criteria utilised in
risk-informed decision-making are outlined in Fig. 2. Risk estimates of system
options are rst checked for compliance with respect to some risk criteria (e.g.
ALARP-criteria), and then for their optimality with respect to some economic
Table 2
Basic types of decision rules and decision criteria
Decision Rule Description of the rule
ALARP-risk (societal risk)
(Melchers, 2001)
Acceptability and intolerability are judged based on
the risk estimate R
^
x
k
; rule for classication.
Cost eectiveness is judged in the ALARPregion
between the risk curves of acceptability and intolerability;
rule for ranking.
Individual risk (HSE, 1989) Acceptability is evaluated against a risk level that can
not be exceeded; rule for classication.
Cost/Benet Acceptability may be based on the ICAF
o
-criterion (i.e.
Optimal Implied Cost of Averting a Fatality) (Kristiansen
and Soma, 2001); rule for classication.
The risk control measure yielding the lowest cost/benet
ratio is the most preferred; rule for ranking
Risk Reduction Acceptability based on risk reduction compared to the risk
of some reference system (the reference system can be a real
or a hypothetical one); rule for classication.
The risk control measure yielding the most preferred risk
reduction based on the risk estimate is selected; rule for
ranking.
Zero Tolerance Acceptability requires zero risk (usually for given system-
states); rule for classication.
Temporary Risk Increase Stop/continue operationdecision based on the
magnitude of the change of risk relative to some predened
reference value; rule for classication.
Relative Risk Signicance Prioritisation of safety improvement in the existing
system; rule for ranking [e.g. Risk Reduction Worth,
Fussel Vesely Importance, Risk Achievement Worth
(Mankamo et al., 1991)]
106 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
decision criterion (e.g. cost/benet). Finally, the choice of action {Accept FSA
results or Rene FSA} is determined by the subjective quality criteria of the deci-
sion-maker(s).
The qualication of the FSA scope and objective is very specic to the regulatory
body and its decision-making culture. The selected decision rules and criteria should
genuinely reect the objectives of the regulatory body to avoid fundamental di-
culties in interpreting the FSA results meaningfully. It is, however, not the task of
the FSA to ensure compliance between the objectives of the regulatory body and the
decision rules and criteria adopted by it. The necessary qualication requirement of
this FSA step would be the explicit formulation of the decision rules and criteria,
based on the understanding of the their role (classication/ranking) and the impact
of biased risk estimates on the precautionary use of the decision rules. In Section 3.6
we will discuss the role of a peer review qualication process in qualifying a FSA.
Refer to Conrad and Ferson (1999), Okstad and Hokstad (2001) and Holmberg et
al. (2002) for a more extensive discussion on decision rules and criteria.
3.2. Qualifying FSA step 1: Hazard Identication
3.2.1. Objective of FSA step 1
The objective of FSA step 1 is to acquire system information for specifying (1) the
set of negative consequences that are relevant for the stakeholders, i.e. {x
j
}, and (2)
the set of the system-states that are deemed hazardous, i.e. {
i
}.
3.2.2. The use of facilitated stakeholder/expert meetings
The FSA Hazard Identication step typically employs stakeholder and expert
interviews and panels to generate a shared understanding of the system, and infor-
Fig. 2. The use of decision rules (examples) for risk-informed decision-making. The condence in the
FSA results and its recommendations is determined by the quality of the FSA.
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 107
mation needed in the risk assessment. The use of Group Decision Support Systems
(GDSS) technology has been found very useful (Rosqvist et al., 1998; Nyman et al.,
2002) to solve some typical problems encountered in expert group meetings (Phillips
and Phillips, 1993). Current GDSS technology oers the opportunity of combining
synchronous and asynchronous meetings, supporting exible agendas for informa-
tion acquisition and generation. Hazardous system-states (or risk scenarios), can be
developed quickly and elaborated simultaneously, revealing the dimensions (char-
acteristics) of the sets {x
j
} and {
j
}.
Any incompleteness in the earlier sets introduces bias in the risk estimate. Thus,
uncertainty introduced in this FSA step may be viewed as completeness uncertainty
(USNRC, 1998). An indication for qualifying the Hazard Identication is provided
by surveying the stakeholders and experts opinions in the completeness of the
hazardous system-states and negative consequences addressed in the FSA. An
indirect way to accomplish this would be to get feedback on the representational
coverage from the selected stakeholders. GDSS can be used for this purpose as well.
An interesting challenge would be the denition of a measure depicting the level of
completeness based on the stakeholders responses. This is, however, beyond the
scope of the paper and is left to the FSA eld to study further. It should be noted
that the stakeholder selection process is presently not included in the FSA method.
3.3. Qualifying FSA step 2: Risk Assessment
3.3.1. Objective of FSA step 2
The objective of FSA step 2 is to specify the functional and stochastic relation-
ships of the system-states and the negative consequences {x
j
} and, in particular, the
probabilities Pr x
j
j
i
_ _
and Pr(
i
) in Eq. (1).
3.3.2. Expert judgement protocol
FSA studies typically rely strongly on expert judgement. Several studies have been
published on the elicitation and use of expert judgement (Keeney and von Winter-
feldt, 1991; Chhibber et al., 1992, Cooke, 1991; Cooke and Goossens, 2000). It is not
the purpose of the paper to review these. The authors put forward the following
theses for specifying an expert judgement protocol for FSAs:
1. The basic frameworkfor using expert judgement inFSAstep2: RiskAssessment
shouldfollowthe phases of the revisedNUREG1150 expert judgement protocol
(Keeney and von Winterfeldt, 1991), i.e. Phase 1: Identication and selection of
the issues (i.e. issues brought fromthe FSAstep 1: Hazard Identication; Phase 2:
Identication and selection of the experts; Phase 3: Discussion and renement of
issues; Phase 4: Training for elicitation; Phase 5: Elicitation; Phase 6: Analysis,
aggregation, and resolution of disagreements; Phase 7: Documentation.
2. Quantities subject to expert elicitation should be decomposed using a common
risk model (e.g. Fault Tree model). This amounts to agreeing on a single
model-of-the-world (Chhibber et al., 1992). This is related to Phase 3 of the
proposed protocol.
108 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
3. The specication of the common risk model parameters should be based on a
consensus seeking process as outlined by the parameter specication proce-
dure in Fig. 3 (see section 3.3.1.2). Classical expert models (see section 3.3.2.2)
suce for this purpose. This is related to Phases 5 and 6 in the protocol. (See
note on rational consensus later)
4. It is the responsibility of the risk analyst to assure that the expert judgement
protocol is informed to the experts before they are consulted, and that the
protocol is followed during the Risk Assessment step of the FSA.
The earlier outline of an expert judgement protocol reects the authors experi-
ences from conducted FSAs that
Specication of a common risk model avoids the problems related to the
aggregation of experts judgements based on dierent modelling (decom-
position) approaches.
It is dicult to nd track records of expert performance with respect to bias.
It is time-consuming to assess possible dependencies between the experts.
The experts prefer to provide percentile values rather than point estimates.
Sophisticated expert models, including parameters for bias and dependence
(Chhibber et al., 1992), are motivated for sensitivity studies as the specica-
tion of the parameters are usually not feasible due to lack of track records.
Rational consensus (Cooke, 1991; Cooke and Goossens, 2000) is an empiri-
cally controlled method to providing credible estimates of risk model para-
meters based on experts judgements. The practical feasibility of calibrating
experts for the elicitation sessions is, however, problematic in the case of FSA
risk models with many parameters requiring dierent expertise and experts.
The following two sections outline expert judgement-based methods for develop-
ing a model structure and for estimating parameters in a way that does not jeo-
pardise the precautionary use of decision rules. The third section addresses the role
of experts in sensitivity analyses.
3.3.2.1. Model bias analysis. The objective of model bias analysis is to ensure that
the decision rules are applied in a precautionary way. Model bias analysis involves
Fig. 3. Parameter specication procedure based on expert judgement (from Nyman et al., 2002).
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 109
making judgements on the direction of bias in the various assumptions underpinning
the risk model of the system (e.g. Fault and Event Tree, Inuence Diagram, etc.).
The assumptions relate to functional relationships of system inputs and outputs,
temporal interdependencies between events, etc. The experts should be able to say
whether the direction of bias is conservative, realistic or optimistic with respect to
risk. As stated earlier in Section 2, deterministic analyses may be utilised to support
model bias analysis. If model uncertainties are deemed large, the problem of over-
conservativeness may arise. This is a problem of credibility of the risk assessment,
and may have to be resolved in a renement of the risk assessment, as decided in
step 5 of the FSA study, the decision being part of the FSA qualication process.
For additional points on model uncertainty, we refer to Zio and Apostolakis (1996).
Model bias analysis is especially important when the considered system represents
a generic system concept rather than a real system. In terms of risk, the generic sys-
tem should represent a conservative rather than an optimistic hypothesised system.
The uncertainty in a system model is referred to as model uncertainty (USNRC,
1998). An example of model bias analysis extracted from a case study by Rosqvist et
al. (2001) is partly reproduced in Table 3.
Table 3
An example of a model bias analysis related to modelling assumptions (part of a table from Rosqvist et
al., 2001)
Modelling assumption Direction of bias Comment
. . . . . . . . .
A_V9: The wave amplitudes are statistically
independent.
realistic Realisations of amplitudes are
independent across the types of
ship motion (i.e. roll, vertical,
horizontal) and over the time,
in irregular seas.
A_C1: The properties of the bulk cargo
formations are time-invariant (with respect
to impulses), i.e. gradual deformations of the
cargo, due to series of impulses, are not
considered.
conservative Wave impacts might change the
system (vessel + cargo) behaviour
over time as the cargo experiences
deformations. Small deformations
probably increase, not decrease,
system stability.
A_C2: Additional water due to rain or
sea-spray, and dierent initial cargo
moisture content levels are estimated
separately and the corresponding specic
amount of water is assumed to be in
equilibrium at the bottom of the cargo at
the beginning of the sea transport.
conservative All water on the bulk cargo
material takes signicant time to
migrate to the bottom of the cargo.
Only in case of long delay between
cargo loading and the start of the
sea transport, or when the sea voyage
is signicantly prolonged, the
migrating water might have a risk
impact.
. . . . . . . . .
110 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
3.3.2.2. Parameter specification procedure. The parameter specication procedure is
a consensus-seeking expert deliberation of parameter values in a risk model. Fig. 3
shows a procedure that was applied in a FSA study by Nyman et al. (2002). In this
procedure, each expert gives his/her percentiles related to the uncertain model
parameter value. If the discrepancies between experts are small, it is expected that,
either a common pair of percentile values, or a mathematical aggregation of the
experts probability distributions, with equal weight of the experts, can be agreed
upon (see Appendix). If agreement cannot be reached, either a renement of the risk
model is needed, incorporating possible new modelling aspects suggested by the
experts divergent quantitative judgements, or more information related to the
parameter needs to be obtained. The denition of the procedure draws on the dis-
cussion of expert consensus types in Budnitz et al. (1998). For a comprehensive
treatment of expert judgement aggregation, refer to Cooke (1991), Mosleh and
Apostolakis (1984) and Chhibber et al. (1992).
If, a common set of percentile values have been agreed upon, the form of the
common probability distribution may be specied. If, the experts do not agree on
common percentiles, expert-specic probability distributions may be determined. As
a rule, a probability distribution function reecting conservatism (i.e. stochastic
dominance with respect to the negative consequence) should be selected in the case
where several candidates are considered. In the classical approach (Cooke, 1991),
the uncertainty related to a risk model parameter is generally given in the form of
the 0.05-, 0.5-, and 0.95-fractiles. Based on this information, samples from the
common or the expert-specic probability distribution can be generated. Alter-
natively, sampling from minimally informative distributions (Cooke, 1995), given
the percentiles, can be performed. Uncertainties introduced in the parameters due to
limited knowledge, divergent views and biases in the expert judgements are referred
to as parameter uncertainties (USNRC, 1998). Refer to Matalas and Bier (1999),
ien and Hokstad (1999) for ideas on the practical assessment of probability dis-
tributions.
It has to be noted that the procedure shown in Fig. 3 induces dependency between
the experts judgements during the deliberation; if the experts agree perfectly on a
point estimate or a probability distribution, all available information has been
shared and utilised, and the experts are completely dependent. However, this is not
to say that an expert should conform if he/she truly thinks that his/her judgement is
a correct one. In that case, mathematical aggregation is certainly preferred to per-
sistent consensus seeking. The particular objective of the procedure is to check that
all observed discrepancies between experts have reasonable grounds and are not just
misconceptions biasing the risk estimate. Fig. 3 is only an outline of the decision
points and the percentile information. It depicts, however, a systematic way to dis-
cuss and reconcile diering views, supporting a common specication of risk model
parameters and the probabilities Pr
i
and Pr x
j
j
i
_ _
.
The earlier described procedure was applied and given credit in the FSA study by
Nyman et al. (2001). The most problematic Fault Tree parameter specications
occurred for very rare system-states. The most extreme percentile values among the
experts usually spanned a range of three orders of magnitude in the study. It was
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 111
expected that for such parameters the experts percentiles would be very dierent
due to judgemental biases (Tversky and Kahneman, 1974), and mathematical
aggregation (see Appendix) was oered as the technique for obtaining the common
specication of the parameters probability distribution.
3.3.2.3. Sensitivity analysis. By calculating the rank correlation between the values
simulated for a particular parameter and the computed values of risk, an indication
of the signicance of the parameter with respect to the risk measure is obtained. The
result of such sensitivity analysis should match the intuitive feel of the experts
regarding the sensitivity of the system for specied system-states and risk model
parameters.
Fig. 4 illustrates the correlation sensitivity
2
of some risk model parameters with
respect to risk, given a certain system-state and parameter values, as calculated in a
case study by Rosqvist et al. (2001). The risk model is deemed credible if the intui-
tions of experts support the computed sensitivity prole for the set of conditions or
system-states {
i
} represented by the risk model parameters.
Fig. 4. Example of sensitivity analysis presented in the form of a Tornado-plot (from Rosqvist et al.,
2001).
2
The correlation sensitivity analysis using rank correlation is based on the Spearman rank correlation
coecient calculations. With this analysis, the rank correlation coecient is calculated between the
selected output variable and the samples for each of the input distributions. The higher the correlation
between the input and the output, the more signicant the input is in determining the outputs value.
112 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
The use of an expert judgement protocol supporting model bias analysis, con-
sensus-seeking parameter specication and sensitivity analysis are argued to be
qualication requirements in the Risk Assessment step of the FSA, as they render
the risk model and the precautionary use of decision rules credible. It has to be
noted that the earlier tasks are iterated in no particular order during the risk mod-
elling process. The risk analysts and the experts needs of particular information
and model renement determine how the risk model is developed.
3.4. FSA step 3: Denition of risk control measures
3.4.1. Objective of FSA step 3
Where necessary, based on the risk assessment results, new risk control options
should be dened, and the risk model rened accordingly to meet the new risk
assessment needs raised by the generated options.
3.4.2. The use of facilitated stakeholder/expert meetings
Any new risk control option introduced in a FSA usually requires modications in
the basic risk model. It is important to check the completeness of the initially dened
sets of the negative consequences {x
j
} and the system-states {
i
}. Consequently, a re-
assessment of the results of FSA step 1: Hazard Identication may be motivated.
This is also depicted in the original FSA process description, as shown in Fig. 1.
Especially, if new sources of uncertainty are introduced by the control options and
the modied risk model, it may lead to more biased risk estimates. Therefore, the
decision rule should be applied with care to avoid false conclusions regarding the
true safety impact of the decision.
From the point of view of qualication of the FSA, surveying the opinions of the
stakeholders regarding the completeness of the generation of potential risk control
options, including consideration of the changes in the sets {x
j
} and {
i
}, is required.
As stated in Section 3.2.2, GDSS can be advantageously used for this purpose.
3.5. FSA step 4: CostBenet Assessment
3.5.1. Objective of FSA step 4
The objective of FSA step 4 is to compute the cost benet ratios,
Q
g
c
g
= r
ref
r
g
_ _
, for all risk control options g=1,. . .,G and identify the option
associated with the minimum value of the ratio, i.e. the least cost related to a unit of
risk reduction. r
ref
is a reference risk level (related to an existing or hypothesised
system, or an absolute risk scale), r
g
and c
g
are the estimated risk and the lifetime
cost measures of option q, respectively. Also the distribution of benets and cost
among the stakeholders is addressed.
3.5.2. Model bias analysis, sensitivity analysis and expert judgement protocol
Evaluation of the cost component is usually not the problem in costbenet
assessment (CBA). Basically the cost component consists of one-time and running
costs cumulating over the predicted lifetime of the system. Thus, the net present
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 113
value of cost has to be calculated. Alternative discounting procedures are described
in French et al. (2001).
The benet part is much more intricate as benets may not only be valued in terms of
risk reduction but also in terms of safety margin or robustness to risk. If all the ratios Q
g
are positive, it is meaningful to talk about risk reduction potential and the cost/benet
ratio is indicative for selecting the most ecient risk control option. However, if, for
some of the options, the risk estimate is zero, i.e. the hazardous system-states modelled
are not reachable, these options are not properly discriminated by the risk based ratio
measure. A measure depicting the safety margin would have to be dened for that
purpose. In principle, some safety margin could possibly be traded-o by, for
example, relaxing the regulatory limits for allowed conditions of operations. Thus,
new business potential could be identied by the magnitude of safety margins.
What is more important, however, is to qualify the precautionary use of a cost/
benet decision rule (see Table 2). Biases in the risk estimate may falsely leave a
prescription to reduce risk unobserved, as discussed in Section 3.1.2. Similarly, dif-
ferent biases in the risk measurement may order the risk control options falsely.
Thus, what has been stated in Section 3.3 regarding qualication of the Risk
Assessment step, applies for the CBA step as well.
3.5.3. The use of Stakeholder Analysis
Stakeholder Analysis provides information about the distribution of benets and
costs of the risk control options, as perceived by the stakeholders. This information
should be used to possibly redene the options. In particular, risk control options
may be augmented with monetary compensation for unfairness that may be per-
ceived by some stakeholders. A spreadsheet model to describe the costbenet dis-
tribution among stakeholders is presented in Gilllan (2002). However, pescriptive
models addressing, for example, compensation between stakeholders have not been
developed to the knowledge of the authors. No decision criteria related to fairness
have been introduced in the FSA method either. It is therefore dicult to state any
qualication criteria for the FSA pertaining to fairness. This will remain an open
question for the FSA eld to answer in the future. A source for references to the
Stakeholder Analysis literature is Donaldson and Preston (1995).
3.6. FSA step 5: Recommendations for the decision-maker
3.6.1. Objective of FSA step 5
As the nal step of the FSA process, the risk and the costbenet impact of the
assessed risk control options are presented to the decision-maker together with an
account of the main risk modelling assumptions, the remaining uncertainties, and a
judgement of the impact of biased risk estimates on the use of decision rules and
criteria.
3.6.2. FSA qualication process and criteria
The procedure of presenting the FSA study and the results should be implemented
as a peer review process where some of the reviewers are independent of the FSA
114 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
study. The peer review process evaluates whether a FSA meets specied qualication
criteria that are generic to FSA studies. The fundamental objective of the FSA
qualication process is to consolidate condence of the decision-makers with respect
to the FSA results and recommendations. The qualication criteria are associated
with methodological quality characteristics which are the basis of risk communi-
cation to the decision-maker (Bier, 2001).
A FSA qualication process is depicted in the process model in Fig. 5. If the
qualication criteria of the FSA are not properly addressed, then it may be possible
to postpone the decision regarding the acceptability of the FSA results and require
some of the remaining uncertainties to be resolved. This corresponds to selecting the
decision option Rene FSA in Fig. 5.
In Table 4, candidates for FSA qualication criteria, together with the methodo-
logical quality characteristics that they imply are shown for each FSA step. The
verication may be operationalised using checklists, with questions related to the
qualication criteria which are answered in a yes/no fashion.
The qualication of the FSA can be supported by a Management System that
includes GDSS features such as simultaneous idea generation, commenting, voting
and surveying. This would be in line with the concept of an enhanced FSA put
forward by Kuo (2002). In such a Management System the qualication of a FSA
can be organised in an iterative way, rening information and resolving uncertainty
where needed until a FSA is quality assured. This would, in the opinion of the
authors, represent a prototype of a mechanism for the use of FSA at IMO, as
proposed by Peachey (2002).
The authors, would like to point out that the earlier discussion of the qualication
criteria is preliminary and primarily reects the experience gained from three FSA
applications studies conducted by the authors (Rosqvist et al., 1998, 2001; Nyman et
al., 2002). More research about the quality of FSAs and the role of qualication
need to be carried out in the FSA community to assure and complement the ndings
and ideas presented here. There is every reason to try to answer the challenge of
Fig. 5. Proposed FSA qualication process. The arrows denote the ow of information.
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 115
consolidating condence in FSAs as discussed in Section 1. Otherwise there is a risk
that the FSA method may not develop into its full potential as a tool for risk-
informed decision-making.
4. Conclusions
Formal Safety Assessment qualication is a peer review process with the objective
of ensuring that a Formal Safety Assessment study has been conducted in a way that
consolidates condence in the results and the recommendations it presents. In the
context of precautionary decision-making, the decision-maker should be condent
in applying decision rules in a precautionary way. It is argued that the FSA method
needs to incorporate qualication criteria related to decision-making transparency,
risk model completeness and credibility of risk analysis. The FSA qualication cri-
teria can be viewed as checkpoints guiding a peer review process in establishing
whether a FSA is qualied, or not. The suggested qualication process and quali-
cation criteria are tentative and subject to elaboration as more applications of the
Table 4
Methodological quality characteristics and qualication criteria for verifying quality of FSAs by a peer
group
FSA step Qualication criteria Quality characteristics
0. FSA Scope and
Objectives
Stakeholders are informed about adopted decision
rules and criteria, veried by the peer group
Transparency
1. Hazard Identication Stakeholders and domain experts feedback on
the completeness of hazard identication process
is recorded and veried by the peer group
Completeness
2. Risk Assessment Model uncertainty of risk model is addressed by
domain experts and veried by the peer group
Credibility
Completeness uncertainty of risk model is
addressed by domain experts and veried by
the peer group
Completeness
3. Risk Control Options Stakeholders and domain experts feedback on
the completeness of risk control option are
recorded and veried by the peer group
Completeness
4. CostBenet Assessment Model uncertainty of cost model is addressed by
domain experts and veried by the peer group
Credibility
Stakeholders feedback on the distribution of
cost of reducing risk and the remaining risk is
recorded and veried by the peer group
Fairness
5. Recommendations for
the Decision-Maker
Peer Review Process Qualied FSA
116 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
FSA will increase the demand of quality assurance and provide new insight on how
to craft a quality assured, i.e. a qualied FSA.
Acknowledgements
The authors are indebted to the anonymous reviewer of the journal, Dr. Jan-Erik
Holmberg and Dr. Urho Pulkkinen for detailed and insightful comments that
helped to improve the paper signicantly.
Appendix. Mathematical aggregation of experts judgements
In the literature, one can nd numerous approaches and methods to the aggregation
of expert judgement. Here we will limit ourselves to the weighted mean methods
(Cooke, 1991), and to two theoretically and practically appealing alternatives: the
weighted arithmetic mean and the weighted geometric mean methods. It should be
pointed out that these aggregation rules can be interpreted in information theoretical
terms; in Pulkkinen (1993), the aggregation formulas are shown to minimise an additive
overall surprisefunction based on the KullbachLeibler divergence measure.
The basic formulas for weighted arithmetic averaging and weighted geometric
averaging of n expert-specic distributions related to a continuous variable x are:
Weighted arithmetic averaging:
f
~
x

n
i1
w
i
f
i
x ;

n
i1
w
i
1 0 4w
i
41 A1
Weighted geometric averaging:
f
~
x c

n
i1
f
i
x
w
i
;

n
i1
w
i
1 0 4w
i
41
c
_
X

n
i1
f
i
x
w
i
dx
_ _
1
A2
From an FSA practitioners perspective, it is rarely the case that the weights of the
experts can empirically be derived from the history or specic tests of experts per-
formances in terms of calibration and informativeness (Cooke, 1991). The validity
of such derivations presumes exchangeability of the calibration and informativeness
scores over the previous and the current estimation tasks. Furthermore, the track
records needed are usually not available. To overcome such problems related to
deriving of the weights, it is the opinion of the authors that practical FSA applica-
tions could simply use equal weighting of thoroughly selected experts. Thus, in gen-
eral, we have w
i
1=n, i1; . . . ; n.
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 117
A further specialisation can be done when experts are used to quantify prob-
abilities and frequencies such as the basic events of a Fault Tree. The natural choices
for generic distribution functions are the lognormal- and the logit-distributions. In
the case, where the aggregation is performed by geometric averaging, we get analy-
tical results for the parameters of the aggregated distribution function, which will
belong to the same function family as the expert-specic distribution functions
(Pulkkinen, 1993). The formulas provided in Table A1 are applicable for equally
weighted expert judgements.
Possible dependencies between the experts judgements are not addressed in the
earlier equations (Eqs. A1A2). Explicit modelling of dependence may be necessary
in a risk assessment involving a lot of uncertain parameters, as dependent uncer-
tainties may signicantly aect the risk results. From a FSA practitioners point of
view the problem of selecting an ecient approach to modelling dependence is not
easy. The dierent approaches (Mosleh and Apostolakis, 1984; Cooke, 1991; Pulk-
kinen, 1993) have to be assessed with respect to applicability, ease of implemen-
tation and transparency. For points of view on dependence, reference is made to
Anderson and Hattis (1999).
References
Anderson, E.L., Hattis, D., 1999. Uncertainty and variability. In: Haimes, Y.Y., Lambert, J.H. (Eds.),
When and How Can You Specify a Probability Distribution When You Dont Know Much? Part II.
Risk Analysis 19(1), 4749.
Bier, V.M., 2001. On the state of art: risk commmunication to decision-makers. Reliability Engineering
and System Safety, 71(2), 151157.
Budnitz, R., Apostolakis, G., Boore, D., Clu, L., Coppersmith, K., Cornell, A., Morris., P. 1998. Use of
technical panels: applications to probabilistic seismic hazard analysis. Risk Analysis, 18(4), 463469.
Chhibber, S., Apostolakis, G., Okrent, D. 1992. A taxonomy of issues related to the use of expert judg-
ments in probabilistic safety studies. Reliability Engineering and System Safety, 38(), 27-45.
Conrad, S., Ferson, S., 1999. Decision Making. In: Haimes, Y.Y., Lambert, J.H. (Eds.), When and How
Can You Specify a Probability Distribution When You Dont Know Much? Part II. Risk Analysis
19(1), 6368.
Cooke, R.M., 1991. Experts in Uncertainty. Oxford University Press, Oxford.
Cooke, R.M., 1995. Unicorn: Methods and Code for Uncertainty Analysis. Delft University of Technol-
ogy, Dept. of Mathematics, Delft, Netherlands.
Table A1
Analytic forms of the parameters of the aggregated distribution obtained by geometric averaging of n
probability distributions
Transformation of uncertain
parameter x
Parameters of the
expert-specic
distributions
Parameters of aggregated distribution obtained
by geometric averaging
Lognormal:ln x $ N ;
2
_ _

i
;
i
~

n
i1
1

2
i
_ _
1

n
i1

2
i
; ~
2
n
2

n
i1
1

2
i
_ _
2
Logit:ln
x
1 x
_ _
$ N ;
2
_ _

i
;
i
Same as above
118 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120
Cooke, R.M., Goossens, L.J.H., 2000. Procedures Guide for Structured Expert Judgment. European
Commission, Nuclear Science and Technology, EURATOM, EUR 18820, Luxembourg.
Det Norske Veritas (DNV), 1997a. Costbenet analysis of existing bulk carriers: a case study on appli-
cation of Formal Safety Assessment techniques. DNV Paper series No. 97-P008. DNV, Oslo.
Det Norske Veritas (DNV), 1997b. Formal Safety Assessment of Helicopter Landing Area on Passenger
Ships as a Safety Measure (DNV Report No. 97-2053). DNV, Oslo.
Donaldson, T., Preston, L., 1995. The stakeholder theory of corporation: concepts, evidence and impli-
cations. Academy of Management Review 20 (1), 6591.
French, S., Bedford, T., Atherton, E., 2001. Supporting ALARP decision-making by Cost Benet
Analysis and Multi-Attribute Utility Theory. Management Science. (Research paper no. 2001/15).
Available from http://www.managementscience.org/research/ab0115.asp.
Gilllan, A. 2002. Stakeholder Analysisthe key to dening who pays for legislation. In: Proc. of the
Intl. Conf. on Formal Safety AssessmentRoyal Institution of Naval Arcitects, London, 1819
September.
Guida, G., Baroni, P.,Cojazzi,G.Pinola, L. Sardella, R. 1996. Preliminary requirements for a knowledge
engineering approach to expert judgement elicitation in probabilistic safety assessment. In: Cacciabue,
P.C., Papazoglou, I.A. (Eds.), Probabilistic Safety Assessment and ManagementEsrel 96 & PSAM
III, 2428 June, Crete, Greece.
Health & Safety Executive (HSE), 1989. Quantied Risk Assessment: Its Input to Decision Making.
HMSO Books, London.
Holmberg, J.-E., Pulkkinen, U., 2001. Experience from the comparison of two PSA-studies. Nordic
Nuclear Safety Research, NKS-36, Roskilde, Denmark.
Holmberg, J.-E., Pulkkinen, U., Rosqvist, T. Simola, K. 2002. Analysis of decision criteria in PSA
applications. In: Bonano, E.J., Camp, A.L., Majors, M.J., Thompson, R.A. (Eds.), Probabilistic Safety
assessment and ManagementPSAM6, 2328 June, Puerto Rico, USA.
IMO, 1997a. Formal Safety Assessment: interim guidelines for FSA application to the IMO rule-making
process (MEPC 40/16). IMO, London.
IMO, 1997b. Formal Safety Assessment: trial application to high speed catamaran vessels (DE 41/INF,7).
IMO, London.
IMO, 2000a. An interim report on FSA study on bulk carrier safety (MSC 72/INF,7&8). International
Maritime Organization, London.
IMO, 2000b. Safety of Bulk Carriersbasis of the international collaborative Formal Safety Assessment
study (MSC 72/INF,18). IMO, London.
Keeney, R., von Winterfeldt, D., 1991. Eliciting probabilities from experts in complex technical problems,
IEEE Trans. Engineering Management 38 (3), 191201.
Kuo, C., 2002. Proposing an Enhanced FSA with a Management System. In: Proc. of the Intl. Conf. on
Formal Safety AssessmentRoyal Institution of Naval Arcitects, London, 1819 September.
Kristiansen, S., Soma, T., 2001. Formal Safety Assessment of Commercial Shipsstatus and unresolved
problems. In: Proceedings of the European Conference on Safety and ReliabilityESREL 2001, Poli-
tecnico di Torino, Italy, 1620 September, vol. 3, pp. 16511658.
Mankamo, T., Po rn, K., Holmberg, J., 1991. Uses of Risk Importance Measures, VTT Technical Report:
Research Notes 1245. VTT, Espoo, Finland.
Matalas, N., Bier, V., 1999. Extremes, extrapolation, and surprise. In: Haimes, Y.Y., Lambert, J.H.
(Eds.), When and How Can You Specify a Probability Distribution When You Dont Know Much?
Part II. Risk Analysis 19(1), 4954.
Melchers, R.E., 2001. On the ALARP approach to risk management. Reliability Engineering & System
Safety 71 (2), 201208.
Mosleh, A., Apostolakis, G., 1984. Models for the use of expert opinions. In: Waller, R.A., Covello, V.T.
(Eds.), Low Probability High Consequence Risk Analysis. Plenum Press, New York.
MSA, 1993. Formal Safety Assessment (MSC 66/14). IMO, London.
Nielsen, T., Aven, T., 2003. Models and model uncertainty in the context of risk analysis. Reliability
Engineering and System Safety 79 (3), 309317.
Nyman, T., Sonninen, S., Ha nninen, S., Rytko nen, J., Tuominen, R., Rosqvist, T., Riska, K., Juva, M.,
T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120 119
Palonen, A., 2002. The Implementation of the VTMIS-system for the Gulf of Finland (Research Report
VAL34-013153). VTT Industrial Systems, Espoo, Finland.
ien, K., Hokstad, P.R., 1999. Handbook for Performing Expert Judgement (Report STF38 A98419).
SINTEF Industrial Management, Trondheim, Norway.
Okstad, E., Hokstad, P., 2001. Risk assesment and use of acceptance criteria for the regulation of dan-
gerous substances. In: Proceedings of the European Conference on Safety and ReliabilityESREL
2001, Politecnico di Torino, Italy, 1620 September, vol. 1, pp. 117124.
Peachey, J., 1997. Formal Safety Assessment for Shipping: Developing its Potential at the IMO. MSA,
Southampton.
Peachey, J., 2002. FSA: ve small steps, but whither the leap. In: Proc. of the Intl. Conf. on Formal Safety
AssessmentRoyal Institution of Naval Arcitects, London, 1819 September.
Phillips, L., Phillips, M., 1993. Facilitated work groups: theory and practice. J. Opl. Res. Soc. 44 (6), 533
549.
Pulkkinen, U., 1993. Methods for combination of expert judgements. Reliability Engineering and System
Safety 40, 111118.
Rosqvist, T., Tuominen, R., Jalonen, J., Salmikuukka, J., Reunanen, M., 1998. Formal Safety Assess-
ment: Trial Application of FSA to the Transportation of Dangerous Goods on Passenger/ro-ro Cargo
Vessels (Report VAL B353). VTT Automation, Tampere, Finland.
Rosqvist, T., Tuominen, R., Reunanen, M., Laaksonen, R., Rintala, S., Sukselainen, J., 2001. FSA Study
on Transportation of Iron Ore Concentrate by Pusher-barges on the Lulea Raahe route (Report RIS
C130). VTT Automation, Tampere, Finland.
Stirling, A., Renn, O., Rip, A., Salo, A., 1999. On Science and Precaution in the Management of Tech-
nological Risk. Final Report for the WC Forward Studies, Science and Technology Policy Research,
University of Sussex.
Suokas, J., Rouhiainen, V. (Eds.), 1993. Quality Management of Safety and Risk Analysis. Elsevier,
Amsterdam, The Netherlands.
Tversky, A., Kahneman, D., 1974. Judgment under uncertainty: heuristics and biases. Science 185, 124
1131.
United States Nuclear Regulatory Commission (USNRC), 1998. An Approach for Plant-specic, Risk-
Informed Decisions on Plant-specic Changes to the Licensing Basis (Regulatory Guide 1.174).
Washington, DC.
Wang, J., 2001. The current status and future aspects in formal ship safety assessment. Safety Science 38,
1930.
Zio, E., Apostolakis, G., 1996. Two methods for the structured assessment of model uncertainty by
experts in performance assessments of radioactive waste repositories. Reliability Engineering and
System Safety 54 (2-3), 225241.
120 T. Rosqvist, R. Tuominen / Safety Science 42 (2004) 99120