Systematic Black-Box Testing of Computer-Based Systems Through Formal Abstraction Techniques

Systematic Black-Box Testing of
Computer-Based Systems through

Formal Abstraction Techniques
Bernhard Klaus Aichernig
Dissertation
vorgelegt zur Erreichung des akademischen Grades
Doktor der Technischen Wissenschaften
an der
Technischen Universitat Graz
Graz, im Janner 2001
Abstract
In the past of computer science, sometimes, a gap between the testing and the formal
methods community could be observed. Testers did not believe in the applicability of
formal verication techniques to real world problems, and formal methods advocates
could not accept testing-techniques as an adequate verication method for producing
correct software. However, today the gap is closing and this thesis contributes to
building the bridge by oering new results on testing based on formal development
techniques.
Others have worked on testing based on formal specications before, but only a
few considered the related development techniques such as program synthesis. Most
of the previous work focused on generating test-cases by partitioning the formal
specication into equivalence classes of input-output behavior. This related work
on testing is reviewed.
In this thesis three formal development techniques have been examined, regarding
their support in the testing of computer-based systems: Light-weight approaches
to formal methods, the renement calculus, and data-reication techniques. As a
result, three novel testing techniques have been developed.
First, a light-weight approach to testing, based on an executable formal speci-
cation, has been applied in two industrial experiments in the domain of voice com-
munication for air-trac control. Abstract VDM prototypes have been interpreted
in order to assess and extend existing system-level test-cases. The encouraging re-
sults demonstrate that formality is essential for testing high-quality systems. Exact
gures of the detected errors in the test documents and of the eorts are presented.
Second, a main contribution of this thesis is a novel approach to calculate test-
cases. It is based on the observation that test-cases are abstractions of system
specications. The renement calculus of Back and von Wright is used to formulate
abstraction rules for deriving correct test-case scenarios from formal specications.
The advantage of this abstraction approach is that simple input-output test-cases as
well as testing scenarios can be handled. This is demonstrated by giving abstraction
rules for partition analysis, mutation testing, and scenarios.
Third, it is demonstrated that abstract formal specications can be used as auto-
matic testing oracles for concrete implementations. Here, classical data-renement
techniques are combined with features of modern tools in order to evaluate test
results with respect to a formal specication.
iii
iv
Zusammenfassung
In der Vergangenheit der Informatik konnte manchmal eine Kluft zwischen den
Anhangern des Testens und der formalen Methoden festgestellt werden. Tester
glaubten nicht an die Anwendbarkeit von formalen Verikationsmethoden in der
realen Welt. Bef urworter von formalen Methoden wiederum, konnten die Testtech-
niken nicht als adaquate Verikationsmethode akzeptieren. Heute aber, schliet sich
die Kluft, und diese Dissertation tragt dazu bei, diese zu uberbr ucken.
Andere haben bereits vorher im Bereich Testen basierend auf formalen Spezi-
kationen gearbeitet, aber nur wenige haben die verwandten Entwicklungsmethoden,
wie die Programmsynthese, mit in Betracht gezogen. Es wird ein

Uberblick dieser
vorangegangenen Arbeiten gegeben.
In Rahmen dieser Dissertation wurden drei formale Entwicklungsmethoden be-
z uglich ihrer Unterst utzung des Testens von computerbasierten Systemen unter-
sucht: Leichte Ansatze zu formalen Methoden, der Renement-Kalk ul und Daten-
verfeinerungstechniken. Das Resultat sind drei neue Testtechniken.
Erstens: Ein leichter Ansatz des Testens, basierend auf ausf uhrbaren formalen
Spezikationen, wurde in zwei industriellen Experimenten im Bereich der Sprachver-
mittlung zur Flugkontrolle angewandt. Abstrakte VDM-Prototypen wurden inter-
pretiert, um existierende Testfalle zu bewerten und zu erweitern. Die ermutigenden
Ergebnisse zeigen, da Formalitat essentiell f ur das Testen von hoch-qualitativen
Systemen ist. Exakte Zahlen der in den Testdokumenten gefundenen Fehler und
des Aufwands werden angegeben.
Zweitens: Ein Hauptbeitrag dieser Arbeit ist ein neuer Ansatz, um Testfalle zu
berechnen. Er basiert auf der Beobachtung, da Testfalle Abstraktionen von System-
spezikationen sind. Der Renement-Kalk ul von Back und vonWright wird verwen-
det, um formale Abstraktionsregeln zu formulieren. Mit diesen Regeln konnen kor-
rekte Testfallszenarien aus formalen Spezikationen abgeleitet werden. Der Vorteil
dieses Abstraktionsansatzes ist es, da sowohl einfache Eingabe-Ausgabe Testfalle
als auch Testszenarien behandelt werden konnen. Es werden Regeln f ur Partitions-
testen, Mutationstesten und Szenarien angegeben.
Drittens: Es wird gezeigt, da abstrakte formale Spezikationen als automatische
Testorakel f ur konkrete Implementierungen verwendet werden konnen. Hier werden
klassische Datenverfeinerungstechniken mit den Moglichkeiten moderner Werkzeuge
kombiniert, um Testergebnisse bez uglich einer formalen Spezikation auszuwerten.
v
vi
Acknowledgments
I would like to express my gratitude to all those people who have supported me
throughout my research.
First of all, I wish to thank Prof. Peter Lucas, my supervisor, for his help and
patience during the course of this work. It was him, who introduced me to formal
development methods and has given me the opportunity to work as his assistant.
Then, I would also like to thank my colleague Andreas Kerschbaumer, who shared
in many fruitful discussions his opinions and points of view with me.
Thanks belong also to my students in software technology. It has been their
constructive feedback on the subject, that convinced me that the formal techniques
presented in this thesis might be accepted by well-educated engineers.
Especially, I want to thank my students Johann Horl and Andreas Gerstinger
who took part in the two industrial experiments at Frequentis. It has been their
ability that made the experiments a success. Both experiments have been funded
by Frequentis.
Finally, I am indebted to my wife, Kathrin, for her support and for encouraging
me all the way. Thanks,
Bernhard K. Aichernig
vii
viii
Contents
1 Introduction 1
1.1 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 The Challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.2 Black-Box and White-Box Testing . . . . . . . . . . . . . . . . 3
1.1.3 Test Automation . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 Formal Development Methods . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1 Formal Specication Techniques . . . . . . . . . . . . . . . . . 6
1.2.2 Design Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.3 Formal Methods and Testing . . . . . . . . . . . . . . . . . . . 9
1.3 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.6 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.7 Structure of this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Review of Related Work on Testing 15
2.1 Partition Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.1 Domain Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.1.2 DNF-Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.3 Separation of the Input . . . . . . . . . . . . . . . . . . . . . . 19
2.1.4 Structural Partitioning . . . . . . . . . . . . . . . . . . . . . . 20
2.2 Mutation Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Structuring Test-Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Classication Trees . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.2 Abstraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Test Sequencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.1 Finite State Machine (FSM) Calculation . . . . . . . . . . . . 23
2.4.2 Renement of a FSM . . . . . . . . . . . . . . . . . . . . . . . 23
2.5 Test Oracles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.6 Algebraic Specications . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.6.1 Testing a Data Type Implementation . . . . . . . . . . . . . . 25
2.6.2 Selection Strategies . . . . . . . . . . . . . . . . . . . . . . . . 26
2.7 Behavioral Specications . . . . . . . . . . . . . . . . . . . . . . . . . 28
ix
x CONTENTS
2.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.8.1 A New Approach to Test Strategies . . . . . . . . . . . . . . . 28
2.8.2 Algebraic vs. Model-Based Specications . . . . . . . . . . . . 29
2.8.3 Taking the Dynamic Behavior into Account . . . . . . . . . . 30
2.8.4 How Much Does It Cost? . . . . . . . . . . . . . . . . . . . . . 30
2.8.5 New Tools New Oracle Approaches . . . . . . . . . . . . . 30
3 A Light-Weight Approach to Testing 33
3.1 Light-Weight Formal Methods . . . . . . . . . . . . . . . . . . . . . . 33
3.2 VDM and Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2.1 The Vienna Development Method . . . . . . . . . . . . . . . . 35
3.2.2 IFAD VDMTools . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3 Test Design with Abstract Prototypes . . . . . . . . . . . . . . . . . . 38
3.3.1 Abstract Prototypes . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.2 The Light-Weight Test Approach . . . . . . . . . . . . . . . . 39
3.4 Formal Specications as a Catalyst in Validation . . . . . . . . . . . . 41
3.4.1 Issues in the Requirements and Test Documentation . . . . . . 41
3.4.2 A Catalyst in Validation . . . . . . . . . . . . . . . . . . . . . 42
3.5 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 42
4 Industrial Experiences 43
4.1 The Application Domain . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.1 Air Trac Control . . . . . . . . . . . . . . . . . . . . . . . . 43
4.1.2 Voice Communication in Air Trac Control . . . . . . . . . . 44
4.1.3 A Network of Voice Communication Systems . . . . . . . . . . 45
4.2 Experiment I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.2.1 Requirements Validation . . . . . . . . . . . . . . . . . . . . . 46
4.2.2 Test Case Validation . . . . . . . . . . . . . . . . . . . . . . . 50
4.2.3 Remarks on Experiment I . . . . . . . . . . . . . . . . . . . . 53
4.3 Experiment II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.3.1 Requirements Validation . . . . . . . . . . . . . . . . . . . . . 55
4.3.2 Examples of Requirement Issues . . . . . . . . . . . . . . . . . 56
4.3.3 Test-Case Validation . . . . . . . . . . . . . . . . . . . . . . . 59
4.3.4 Remarks on Experiment II . . . . . . . . . . . . . . . . . . . . 60
5 Test-Cases are Abstractions! 63
5.1 The New Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.1.1 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.1.2 Test-Design as a Formal Synthesis Problem . . . . . . . . . . . 63
5.2 The Renement Calculus . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.2.1 Contracts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.2.2 Example Contracts . . . . . . . . . . . . . . . . . . . . . . . . 65
CONTENTS xi
5.2.3 Semantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.2.4 Renement and Abstraction . . . . . . . . . . . . . . . . . . . 68
5.3 Test-Cases as Abstractions . . . . . . . . . . . . . . . . . . . . . . . . 69
5.3.1 Input-Output Tests . . . . . . . . . . . . . . . . . . . . . . . . 69
5.3.2 Non-Deterministic Test-Cases . . . . . . . . . . . . . . . . . . 71
5.3.3 Partition Tests . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Testing Interactive Systems . . . . . . . . . . . . . . . . . . . . . . . 72
5.4.1 User Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4.2 Iterative Choice . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4.3 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6 Testing Strategies 77
6.1 Partition Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.2 Structural Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6.3 Mutation Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.4 Calculating Scenarios for Testing . . . . . . . . . . . . . . . . . . . . 83
6.4.1 Critical Remarks on FSM approaches . . . . . . . . . . . . . . 83
6.4.2 Compositions . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.4.3 Scenario Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 84
6.4.4 Scenario Based Testing Strategies . . . . . . . . . . . . . . . . 84
6.5 Example: Process Scheduler . . . . . . . . . . . . . . . . . . . . . . . 85
6.5.1 Interactive Process Scheduler . . . . . . . . . . . . . . . . . . 85
6.5.2 Interaction Partitioning . . . . . . . . . . . . . . . . . . . . . 87
6.5.3 Compositions . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.5.4 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7 Automated Test Result Verication 91
7.1 From Formal Specications to Oracles . . . . . . . . . . . . . . . . . 91
7.1.1 VDM-SL Specications as Oracles . . . . . . . . . . . . . . . . 91
7.1.2 The Homomorphism of Abstraction . . . . . . . . . . . . . . . 93
7.2 Automated Test Evaluation . . . . . . . . . . . . . . . . . . . . . . . 94
7.2.1 Code-Generation of Oracles . . . . . . . . . . . . . . . . . . . 94
7.2.2 Interpretation of Oracles with the Dynamic Link Facility . . . 96
7.2.3 CORBA Oracles . . . . . . . . . . . . . . . . . . . . . . . . . 96
8 Case Study 99
8.1 Roles in Voice Communication . . . . . . . . . . . . . . . . . . . . . . 99
8.1.1 An Abstraction of the Systems State . . . . . . . . . . . . . . 100
8.1.2 Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
8.2 Tester-System Interactions . . . . . . . . . . . . . . . . . . . . . . . . 102
xii CONTENTS
8.3 DNF-Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
8.4 Scenario Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
8.4.1 Interaction Compositions . . . . . . . . . . . . . . . . . . . . . 106
8.4.2 Selecting Test Scenarios . . . . . . . . . . . . . . . . . . . . . 111
9 Concluding Remarks 119
9.1 Research Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
9.1.1 Light-Weight Approach . . . . . . . . . . . . . . . . . . . . . . 119
9.1.2 Abstraction-Based Test Synthesis . . . . . . . . . . . . . . . . 120
9.1.3 Formal Specications as Oracles . . . . . . . . . . . . . . . . . 121
9.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
9.3 Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Bibliography 122
Chapter 1
Introduction
This dissertation presents novel approaches to software testing. The testing tech-
niques are based on formal development methods for software. These formal meth-
ods usually start with a formal description of the software requirements and rene
this specication into an executable implementation. The advantage is the possibil-
ity of proving the correctness of the implementation with respect to its specication.
In this work, formal requirements specications as well as formal development meth-
ods are applied in the design, validation and execution of software tests. It turned
out that abstraction techniques are the key to a more systematic testing process.
This chapter gives a general introduction into the subject, motivations, aims and
scientic contributions of our work.
1.1 Software Testing
1.1.1 The Challenge
Software testing is a challenging task. It should evaluate the software to demonstrate
that it meets the requirements. This is dicult, both in theory and in practice.
However, software is increasingly being used in our daily life as well as in safety-
critical systems where failure could be life threatening. This enforces the need for
an improved systematic discipline of testing which must be scientically justiable.
Testing and Correctness
From theory it is known that testing cannot guarantee the absence of failures in
complex computer-based systems. The reason is the discrete nature of software.
For example, if two test-points in the input-space of a piece of software have been
tested successfully, nothing is known about the behavior of the software for other
inputs situated between these test-points. This is fundamentally dierent to other
engineering disciplines, mainly dealing with continuous physical domains.
1
2 CHAPTER 1. INTRODUCTION
In order to certify a software for being correct with respect to its requirements, it
should be stimulated with all possible inputs. However, the input-space of complex
software is in general too large. Consequently, a representative subset of inputs, the
test-cases, has to be selected. What is considered to be representative is based on a
hypothesis. This means that testing does only show that a system behaves correctly
with respect to its requirements for the actual test-cases, the rest is an assumption.
Therefore, the purpose of software testing is rather the detection of failures than the
demonstration of their absence. Edsgar Dijksta summarized this in his well-known
statement: Program testing can be used to show the presence of bugs, but never to
show their absence [DDH72]. In practice, further diculties with respect to testing
arise.
Testing and Requirements
The testing process is strongly dependent on the quality of the requirements doc-
umentation available to a tester. User requirements that have not been explicitly
documented cannot be systematically tested. It is known from practice that ap-
proximately 60 percent of all defects in the requirements phase are due to missing
requirements [Per95]. Therefore, the systematic capturing of user requirements is a
prerequisite to successful software testing. However, experience shows that require-
ment documents written in a natural language tend to be ambiguous and unsound.
Consequently, formal specication languages have been designed for capturing re-
quirements in an unambiguous and sound software specication. In Chapter 3 our
own experience from two industrial experiments is presented where such formal
specication techniques have been successfully applied for increasing the quality of
functional requirements in a safety-critical domain.
Testing in the Development Process
Testing has to be considered with respect to the development process, too. In the
classical waterfall-model of the software development process, testing is presented as
the last phase [Roy70]. This causes a tendency to cut the testing eorts, if projects
exceed their schedules. The situation for testing is even more dramatic, if it is
realized that most of todays big software projects do not meet their schedules, as
reported in [Gib94] and more recently in [GSPC99]. Consequently, the tests have
to be designed as soon as possible in the development process, after the software
requirements are available.
Execution of the test can take place as soon as software units are ready for com-
pilation or interpretation. If components are ready for integration, these tests are
performed until the nal system-level tests are carried out. The V-model [fSSaCB91]
stresses the dierent levels of testing and shows the relation to the development
phases explicitly.
1.1. SOFTWARE TESTING 3
In Boehms spiral process model testing takes a central role [Boe86]. This ap-
proach proposes rapid prototyping. A prototype of software is an incomplete imple-
mentation of software that mimics the behavior we think the user need [Sta89]. In
rapid prototyping, software is implemented by the incremental development of a pro-
totype that is tested and analyzed before new functionality is added. This process
model is motivated by the fact that testing should be performed as soon as possible.
The earlier in the process a fault is discovered and corrected, the cheaper the correc-
tion early detected faults are cheap faults. Thus, with respect to testing Boehm
says: Do it early! [Boe81]. In Chapter 3, executable formal specications are pro-
posed as a prototype for an early validation of requirements as well as system-level
test-cases.
Becks recent Extreme Programming approach is even more radical [Bec99]. He
rejects the classical process models. However, testing plays a central role in Extreme
Programming. One of its rules is that for each class, sucient test-routines have
to be implemented as a part of the class prior to the methods design. In addition,
more frequent regression tests are proposed after each working day of a program-
ming team, when the software has been extended or small changes took place. The
advantage is twofold: First, the scheduling problem is solved, since testing is an
integral part of the design and implementation process. Second, the testing routines
act as a formal specication of the class methods. This will become more obvious in
Chapter 5, when it is demonstrated that test-cases are in fact formal specications.
1.1.2 Black-Box and White-Box Testing
This thesis is about test techniques. A test technique or test strategy is a systematic
method used to select and/or generate tests to be included in a test-suite. Two
fundamentally dierent categories of test techniques exist, black-box and white-box
techniques. Black-box test techniques are based on requirements. They are therefore
also called behavioral or functional techniques. Testing done under a black-box
strategy is called black-box testing. Black-box test strategies do ignore the internal
structure of the object under test. Thus, a black-box strategy does not consider how
the test-object have been implemented, but what its requirements are. The subject
of this thesis is black-box testing. For a more detailed introduction to black-box
techniques we refer to [Bei95].
In white-box testing, the tests are derived from the structure of the tested object.
Consequently, this testing method is also called structural testing. Well-known
example test strategies of this kind are to execute every statement at least once
(statement testing), to execute every branch at least once (branch testing), and to
test the use of all data object (data-ow testing). For a detailed presentation and
comparison of these techniques we refer to [Bei90] and [Mye79].
In this thesis it is shown that white-box test strategies can be applied to require-
ments, if these requirements are captured in a formal specication. This means that
structural techniques can be applied for black-box testing, if the structure of the
specication is analyzed. One approach is to specify an abstract and executable
prototype solely based on the requirements. Then, for example, branch testing
is possible. This method and its encouraging results are presented in Chapter 3.
A more advanced method derives test-cases directly from a more abstract non-
executable specication by applying synthesis rules to the specication. Here, the
structural techniques are incorporated into the synthesis rules. This novel approach
is shown in Chapter 6.
1.1.3 Test Automation
Testing is expensive and its eorts are often underestimated. The costs of testing
are dependent on the number and subtlety of detected faults during testing. Brooks
points out that software engineers are usually too optimistic with respect to failure
rates, Therefore, testing is usually the most miss-scheduled part of programming
[Bro75]. He proposes that the half of the schedule should be devoted to testing: 1/3
planning, 1/6 coding, 1/4 component test and early system test, 1/4 system test
with all components in hand. Another much-quoted heuristic rule distributes the
eorts as follows: 40 % design, 20 % coding, 40% testing [Per95]. The gures given
by Boehm [Boe75] distinguish between dierent types of software systems. He found
that for testing of command and control systems 34 %, space-born systems 46 %,
operating systems 50 %, scientic systems 30 %, and business systems 28 % of the
total development costs are spent. Boehms gures from 1975 are still in use as their
presentation in the standard literature for software engineering [Som92] indicates.
Although, more elaborate regression models for estimating the eorts of testing exist
(see e.g. [Lyu95, MIO90]), these heuristic data demonstrate how expensive testing
can be.
One way to decrease these huge eorts is the automation of testing. Other
arguments for test automation summarized in [Bei95] are that
manual tests are error-prone: a typical test-case contains 250 characters.
Statistics from professional keypunchers indicate that manually entered tests
have at least one keystroke-error. Furthermore visual test-verication is likely
to fail;
manual testing leads to false condence: manual test-execution is dicult and
boring. Often eort and accomplishment are equated, giving the impression
that hard work means well tested;
dependability requirements demand automated testing: all statistically valid
methods of estimating software dependability are based on fully automatic
test methods for executing enough test-cases;
a test has to be run several times: in practice, tests are executed more than
once. The reasons are that a test-design may have been wrong, a failure
1.2. FORMAL DEVELOPMENT METHODS 5
has to be checked if it is repeatable, a fault has to be demonstrated to the
programmer, the xing of the fault has to be shown.
maintenance needs automated regression testing: In todays software engi-
neering, 80 % of the whole software engineering eorts are in maintenance of
existing code. Only an automated test process of the changed software makes
maintenance feasible.
More detailed justications and examples of test-automation can be found in [FH88,
Hen89, Mil86, Nom87, Per86, VGvM80]. In principal, all three phases of testing can
be automated or at least supported by tools:
1. test design: test-case generators derive test-cases by analyzing a formal repre-
sentation of the software. Most of the tools are based on the program source-
code (white-box testing).
2. test execution: test-scripts or test-routines that can be run automatically make
tests repeatable and reduce costs.
3. test evaluation: automated oracles are programs that evaluate the tests by
comparing the actual test results with the expected outcome.
Traditionally, the automation of the test design has been focused on white-box
techniques. The reason is that the source-code of a programming language is a
standardized formal representation which can be analyzed mechanically. For au-
tomating black-box testing based on the requirements, a formal representation of the
requirements is needed
1
. Formal development methods provide such specication
techniques that enable the automation of black-box testing. A general summary
of the possibilities of automating black-box testing, not solely focused on formal
methods, is [Pos96].
This thesis contributes to the automation of black-box testing with respect to all
three test phases above. In order to understand the contributions, motivations and
aims of this work a short introduction into formal development methods is presented.
The summary of the related work in Chapter 2 will introduce into the existing test
techniques based on formal methods in more detail.
1.2 Formal Development Methods
All classical engineering disciplines rely on mathematics for creating their products.
At a NATO conference in 1968, it was realized that the construction of software is
so complex that its development deserves to be an engineering discipline of its own
the term software engineering was born [NR69]. In the same decade, the rst
formal development methods have been invented.
1
Random testing techniques are exceptions.
Formal development methods are mathematical methods for producing software
(and today digital hardware). Due to the discrete nature of software, the mathemat-
ics needed is fundamentally dierent to that used in other engineering disciplines
mainly dealing with continuous domains. In software engineering, formal logic and
algebraic structures are the most important mathematical tools. The term formal
means the use of methods of reasoning that are sound by virtue of their form and
independent of their content.
The UK Military of Defense standard on the procurement of safety-critical soft-
ware [oD89] denes a formal method as:
A software specication and production method, based on a mathe-
matical system, that comprises: a collection of mathematical notations
addressing the specication, design and development phases of software
production; a well-founded logical system in which formal verication
and proofs of other properties can be formulated; and methodological
framework within which software may be veried from the specication
in a formally veriable manner.
From the denition above it can be concluded that formal specications of require-
ments and formal verication of software are the corner-stone of a formal method.
However, recently light-weight approaches to formal methods have gained popularity
for transferring these techniques into industry [ELC
+
98, JW96, Jon96]. In this con-
text light-weight indicates that the methods focus is rather on specication than on
formal proofs. In Chapter 3 these approaches are examined with respect to testing.
In the following, an overview on the dierent formal techniques should give an
introduction to the advantages mathematics provides for software development
and for testing. A more detailed introduction to formal development methods can
be found, for example, in the formal methods guidebooks of NASA [NAS95, KK97].
1.2.1 Formal Specication Techniques
A formal specication is the documentation of the functional requirements of a
computer-based system in a mathematical language, the specication language.
Since, the semantics of a specication language is formally dened, formal spec-
ications can capture software requirements in an unambiguous and yet abstract
style.
Dependent on the mathematical apparatus used, several dierent specication
techniques have been established. In this work, the following three fundamentally
dierent approaches to specication are distinguished: model-based, algebraic and
behavioral specications. The reason for this classication are the completely dif-
ferent consequences of these techniques for testing (see the survey of related work
in Chapter 2).
Model-based Specication
Model-based specication techniques use mathematical objects for modeling a sys-
tems state explicitly. Numbers, sets, sequences, nite mappings, Cartesian prod-
ucts and composite types are typical types of objects that are applied to model a
system on an adequate level of abstraction. In contrast to conventional program-
ming languages, logical constraints on the types, here called data-invariants, may
be added to specify additional properties of a model. Functionality is specied by
state-changes expressed by functions or relations on the state model. Consequently,
these specications are also called state-based.
Prominent formal methods of this kind are VDM [Jon90, FL98], Z [Spi88, Spi92,
WD96], and B [Abr96]. Furthermore, the formalism of Back and von Wrights
renement calculus is model-oriented [BvW98]. The testing techniques presented
in this work are solely based on model-based specication techniques. However, for
the discussion of the related work the next chapter, a basic knowledge of the other
two specication methods is necessary.
Algebraic Specication
In an algebraic specication technique an object class or type is specied in terms
of the relationships between the operations dened on that type. Unlike in model-
based specications, nothing is said about the internal representation of the dened
objects. Hence, in general, algebraic specications are more abstract than model-
based specications.
Algebraic specication was brought to prominence by Guttag [Gut77] in the
specication of abstract data types. Various notations for algebraic specication
have been developed, including OBJ [FGJM95] , Larch [GHW85, GH93] and CASL
[Mos99]. In RAISE [Gro92, Gro95], algebraic specication and the model-based
techniques from VDM have been combined. A notation independent presentation
of the subject can be found in [HL93].
Behavioral Specication
Abstract specications of the behavior of a system are called behavioral specica-
tions. This type of formalisms describe rather how a system behaves over time than
what the possible computations can be. Control and concurrent systems are typical
application domains for this specication style.
If the description of the behavior should mainly focus on the set of possible
reachable states, nite state machines are used to capture the dynamic state-changes.
If the specication is focused rather on the possible events that can occur in a
system than on its reachable states, then labeled transition systems are in use. The
process algebras of CSP [Hoa85] and the -Calculus [Mil99] belong to this class of
specication techniques.
Further well-known behavioral specication techniques are Petri-nets [Pet77,
Pet81], and Estelle, LOTOS, SDL, [Tur93], well-suited for dening communication
protocols.
However, some development methods, like UML, use diagrammatic behavioral
specications that lack a formal semantics. These methods are not considered to be
formal by the author.
Behavioral specication techniques can be combined with non-behavioral meth-
ods. VDM++ (see Chapter 3) is an example of combining model-based with be-
havioral specications, LOTOS an example of the combination with the algebraic
specication style.
1.2.2 Design Methods
A toolbox of a formal development method should include both, a formal speci-
cation language as well as a formal design technique. Usually, a formal design is
started from an abstract formal specication and then successively rened into an
executable implementation. This process is called renement. In model-based tech-
niques, both, the data model (data renement, also called data reication) as well
as the operations (operational renement) are rened.
Two dierent formal approaches to the renement problem exist, known as invent
and verify and program synthesis. In this thesis it is demonstrated how both methods
can be adapted for the application in testing.
Invent and Verify
In this classical design paradigm, a rened version of the specication is invented and
afterwards veried that the renement is correct with respect to its specication.
Each method provides the proof obligations necessary for proving the renement
relation. For example, the M of VDM (Vienna Development Method) applies
invent and verify [Jon90].
It is demonstrated in Chapter 7 that this approach can be adapted for testing
such that an abstract formal VDM specication can be used as a test oracle.
Program Synthesis
Program Synthesis is the systematic transformation of a specication into an im-
plementation. Formal transformation rules guarantee that the implementation is
correct by construction if these synthesis rules are respected.
Dijkstra has been the rst who proposed this approach for the development of
state-based programs [Dij76]. Additional heuristics for calculating programs have
been presented by Gries [Gri81]. Further progress in program synthesis has been
made by Back and von Wright [BvW98], and Morgan [Mor90]. They extended the
approach to renement calculi with their own language for dening contracts and
large sets of rules for calculating programs.
Program synthesis have been applied to functional programs, too: Martin-Loef
presented an intuitionistic type theory for deriving functional programs from type
declarations [ML85]. Manna and Waldinger [MW92] developed a tabular proof
technique, such that functional programs are developed as side eects during the
proof.
A formal renement calculus enables the development of tools that support
the interactive synthesis of programs. In [vW94] an automation of Back and von
Wrights renement calculus have been described. Coen adapted Martin-Loefs work
and developed a constructive type theory for the theorem prover Isabelle [Coe92].
In this thesis it is the rst time that program synthesis is applied to test-case
synthesis (Chapter 5 & 6). Here, Back and von Wrights renement calculus is used.
1.2.3 Formal Methods and Testing
In the past of computer science, sometimes a gap between the testing and the formal
methods community could be observed. Testers did not believe in the applicability of
formal verication techniques to real world problems, and formal methods advocates
could not accept testing-techniques as an adequate verication method for producing
correct software. However, recently the gap is closing.
Today, light-weight approaches to formal methods invite engineers to take the
advantages of formal specication techniques without focusing alone on correctness
proofs. Having precise and unambiguous formal specications available, is the pre-
requisite in order to automate black-box testing. The importance of black-box
testing has increased, due to the growing awareness that a combination of black-
and the more traditional white-box testing uncovers more defects than applying one
technique solely. Especially, the object-oriented paradigm and the increasing use of
Components Of The Shelf (COTS), shifted the focus of interest towards black-box
approaches [Bei95].
A further reason for an easier integration of testing methods and formal methods
has been the success of model-checking. Model-checking is in fact exhaustive testing
of a formal model. For model-checking to be applicable to software, abstract models
of the software with nite data domains have to be build. These abstractions have to
be proven to be correct. As a consequence, integrated tools with model-checkers and
theorem provers are under development, as presented recently in [Rus00, BGL
+
00].
In this work the application of formal development methods to system-level test-
ing is considered. This part of the testing process will always be necessary. For
proving a computer-based system to be correct, it would be necessary to model the
whole environment the system interacts with which is impossible.
The Philosophical Point of View. The relation between testing and formal
specication is reected in the philosophical points of view of famous Austrians:
Ludwig Wittgenstein, the Wiener Kreis, and Sir Karl Popper. Wittgenstein and the
Wiener Kreis proposed logic for describing and solving scientic problems [Wit21,
Kra50]. Popper realized that only through falsication truth can be obtained. He
consequently proposed his demarcation criterion according to which any empirically
refutable consistent proposition should be regarded as scientic. For theories to be
observationally falsiable, they must be stated precisely and clearly [Pop35, Pop63].
Their results can be applied to software development as well, with the following
consequences:
A formal specication of requirements helps to solve systematically the prob-
lem of software development by oering the apparatus of logic (or more general
mathematics).
The validation of the formal specication as well as the implemented solution
only is feasible through falsication which is testing.
For being falsiable (or testable), the requirements description has to be un-
ambiguous and sound ensured by formal specication and verication tech-
niques.
1.3 Objective
The overall objective of this thesis is to contribute to the combination of formal
development techniques and testing. Despite the fact that a lot of previous work
related to formal methods and testing exists, this thesis should demonstrate that
more can be done. The aim of this thesis is to examine the formal development
techniques with respect to their usefulness in the testing process. Three formal
techniques are considered from a testers perspective:
Light-weight formal methods recently gained increased popularity, due to their
easier integration into an industrial environment. This thesis should demon-
strate the advantages, costs and feasibility of applying this kind of formal
techniques to testing.
Invent and verify development techniques are traditionally applied in formal
program verication with proofs. Where proofs are not feasible, but formal
specication have been successfully implemented, it is our aim to show that
these more abstract specications can be used as test oracles.
Program synthesis techniques ensure the development of correct programs.
Our aim is to develop system-level test-cases with the same level of rigor.
This thesis should prove that program synthesis techniques can be applied to
produce correct test-cases.
1.4. MOTIVATION 11
It has not been our objective to produce a ready to use tool-set for testing software,
but to examine and develop the techniques necessary for developing new testing
tools tools more advanced than available today.
1.4 Motivation
During the last few years, the interest in formal software development has been grow-
ing rapidly. However, despite their long existence since the 60s, formal methods are
not yet widely accepted and applied in the software industry. Our overall motivation
for this work is to facilitate this technological transfer and thus to contribute to the
eorts in turning software development into a mature engineering discipline.
From our experience in industry, we are convinced that test-engineers are a
promising target group for introducing formal methods. Especially, in the domain
of safety-critical systems, testers are really concerned about the eectiveness and
maturity of their methods. In this thesis both, practical results that can be applied
directly, and new theoretical results that need further adaption and tool support to
be transfered into industrial practice are presented.
The central role of testing in software engineering demands scientically justi-
able testing methods. Today, test-plans are rather complex procedures, including
test-cases with several sequences of interaction with a system. In a systematic soft-
ware development process, these test-cases have to be documented correctly and
unambiguously. If a tester cannot rely on his test-plans, they are rather useless.
This was the motivation for our work on the application of program synthesis tech-
niques to test-case synthesis.
A further motivation of our work is the fact that test-cases can be used for
validating a formal specication by users not familiar with formal notations. The
premise for such a test-case based validation is that the test-cases have been correctly
derived from the specication.
1.5 Contributions
In this thesis three formal development techniques are examined, if and how they
support the testing of computer-based systems: Light-weight approaches to formal
methods, the renement calculus, and data-reication techniques. As a result, novel
testing techniques are presented.
The scientic contributions of this work can be summarized as follows:
A light-weight approach to specication based testing has been developed.
The approach is based on executable VDM specications and the commercial
IFAD VDMTools. Two industrial experiments demonstrate its successful ap-
plicability. Figures of the eorts and eciency of the method are presented.
The experiments have been designed, supervised and controlled by the author.
The actual specication work has been done by two master thesis students of
the author.
A main contribution of this work is the realization and formal proof that test-
cases are in fact abstract formal specications. Our abstraction theorems hold
for simple input-output test-cases as well as for test scenarios consisting of
several steps of interaction.
The observed property that test-cases are abstractions of specications lead
to a new test synthesis approach based on program synthesis. In this thesis it
is the rst time that the renement calculus is applied to the problem of test
synthesis. The result is an abstraction calculus for testing in order to calculate
test-cases according to a testing strategy.
New formal synthesis rules for the test strategies of partition testing, mutation
testing and test-sequencing are presented and formally proved.
Finally, it is shown how formal specications can be used as test-oracles. A
framework using VDM and new features of the IFAD VDMTools is discussed.
1.6 Publications
The research for this thesis lead to 10 publications of the intermediate scientic
results.
Our light-weight approach to testing has been described in [HA99, HA00a,
HA00b, AGA00]. It has been shown that this validation approach can be
extended to hybrid systems as well in [AK00]. The article in the IEEE Software
journal [HA00b] was rst submitted to the Fourth International Conference
on Requirements Engineering (ICRE2000). The program committee selected
this article as best paper and forwarded it to the IEEE Software journal.
The work on a calculus for test-synthesis lead to the most recent publication
[Aic01].
The ideas and techniques to use formal specications as test-oracles can be
found in [Aic98, Aic99a, Aic99b, Aic99c].
1.7 Structure of this Thesis
Chapter 1 introduces to the general subject of this thesis: software testing and
formal development methods. Furthermore, the objectives, the motivations, a
summary of contributions, and the structure of this thesis are presented.
1.7. STRUCTURE OF THIS THESIS 13
Chapter 2 summarizes the related work on testing based on formal development
methods. In this chapter, the existing test-case generation techniques based
on formal specications are explained. The summary distinguishes between
testing techniques based on model-based, algebraic, and behavioral formal
specications. Based on this technical review of the related work, the new
contributions of this thesis are discussed.
Chapter 3 presents a light-weight approach to testing based on executable formal
specications. The technique uses an executable VDM specication of the
system requirements and IFAD VDMTools for validating the design of the
system-level test-cases.
Chapter 4 Here, two industrial experiments in the domain of air-trac control are
presented in which the light-weight approach have been applied. Its applicabil-
ity in an industrial environment is demonstrated by the encouraging results of
these experiments. Detailed empirical data support our claims. Furthermore,
example serve to illustrate the proposed method.
Chapter 5 develops the foundation for a new test-synthesis calculus. It is proved
that test-cases are in fact abstract formal specications. This insight holds for
simple input-output test-cases as well as for test scenarios consisting of several
steps of user interaction. The renement calculus is used to formulate and
prove this abstraction property.
Chapter 6 presents formal rules of the new abstraction calculus for test synthesis.
This is the rst time that program synthesis techniques are applied to the
test-case synthesis problem. Dierent test strategies are formulated in this
new formal framework. Rules for partition testing, mutation testing and the
calculation of test scenarios of interactive systems are dened.
Chapter 7 shows how abstract model-based specications can be used as test or-
acles for an automatic evaluation of the test results. The technique is based
on VDM but can be applied to other model-based formal methods. It is ex-
plained how the invent and verify proof technique can be applied to testing.
Furthermore, it is shown that the new features of IFAD VDMTools facilitate
the implementation of this oracle framework for testing.
Chapter 8 demonstrates the new test-calculus by means of a case-study. A voice
communication system for air-trac serves as an example for showing the
application of the dierent testing techniques.
Chapter 9 gives a brief summary of this work. Furthermore, the nal conclusions
are drawn by the author. The thesis closes with an outlook and motivation
of the possible future work that may be initiated by the contributions of this
thesis.
Chapter 2
Review of Related Work on
Testing
In this chapter a review of related work on testing based on model-based formal
methods is presented. Since our approaches to testing is based on the Vienna De-
velopment Method (VDM) and the renement calculus, the previous work on testing
with model-based specications is most closely related to our own work. However,
at the end of this chapter, testing methods based on algebraic and behavioral spec-
ications are presented. The discussion of the testing techniques of these methods
shows the dierent challenges with respect to testing, depending on the choice of
the specication style.
2.1 Partition Testing
The most common strategy for generating tests from model-based specications is
partition testing. Partition testing applies strategies based on dividing the set of all
possible inputs into equivalence classes under an appropriately dened equivalence
relation [HT88, OB88, RC81]. In the late 80s the formal methods community
realized that partition testing techniques may be applied to formal specications.
Especially, for state-based specications the adaption of well-known techniques from
source-code based testing is straight forward.
2.1.1 Domain Testing
Scullard reports in [Scu88] on the industrial use of VDM for selecting test cases in
order to validate a micro-processor. More precisely, domain testing is applied to
VDM specications.
Domain testing analyzes the input-domain of a systems functions, i.e. its signa-
tures. The strategy is based on a simple processing model in which the system under
test accepts an input vector, which then goes through a classication process, e.g.
15
16 CHAPTER 2. REVIEW OF RELATED WORK ON TESTING
a case statement. Each of these cases is processed separately. The main concern in
domain testing is the correctness of the classication process as contrasted with the
correctness of the processing done for each case. Usually, in domain testing, domains
are dened by a set of boundary inequalities. A simple example of a domain testing
technique is the choice of extreme values of input parameters [Coh78, JW94].
Since all input values of a model-based specication have a concrete represen-
tation, domain analysis can be performed on it. Furthermore, preconditions, data-
invariants, and post-conditions can be used to synthesize subdomains. Scullards
test-case extraction process for VDM can be summarized as follows:
1. Domain reduction: Reduction of the signature of the function to basic types
only. Compound types are replaced by the Cartesian product of their compo-
nents. Furthermore preconditions and data-invariants are taken into account
to reduce the input-domain further.
2. External analysis: Selecting a set of typical values for each basic type of the
reduced signature. For ordered types extreme values and one or more central
values are chosen, for unordered types all values are selected.
3. Internal analysis: Identication of possible cases by skolemizing the postcon-
dition of the function.
Scullard gives an example of his technique and discusses that an automated test
generation system would be useful. However, he did not formulize the underlying
partitioning method. This has been done in the subsequent work on VDM and
testing by Dick & Faivre.
2.1.2 DNF-Partitioning
Dick & Faivre. One of the most cited works on test-case generation from model-
based specications is [DF93]. Dick and Faivre realized that transforming a formal
VDM specication into a disjunctive normal form (DNF) constitutes an adequate
partitioning method. A Prolog tool has been developed by them that rewrites data-
invariants, preconditions and postconditions according to the following rules for a
DNF construction:
A B (A B) (A B) (A B)
A B A B
For model-based specications this partitioning process is extended to type asser-
tions of the form x : T denoting the values of variable x to be of type T. A speci-
cation containing x : T can be further partitioned into special values v : T and other
values:
x : T (x = v x ,= v) x : T
2.1. PARTITION TESTING 17
A simple example serves to demonstrate the method of DNF partitioning.
Example 2.1 (DNF partitioning) Consider a specication of a program com-
puting the maximum of two integer numbers. Assuming that the input values are
stored in two state variables x : Z and y : Z and the result in variable max, then
the problem can be specied with the following relational update Max. The square-
brackets denote a relational update [R] that updates the state variables according to
an input-output relation R. This relational update corresponds to a post-condition
in VDM.
Max = [(max = x max = y) max x max y]
Rewriting the disjunction into a DNF gives:
[((max = x max = y)
(max ,= x max = y)
(max = x max ,= y)) max x max y]
Distributing the and simplication results in three disjoint cases:
[(max = x max = y)
(max > x max = y)
(max = x max > y)]
The subexpressions above characterize three subrelations, describing the test parti-
tions for testing Max with respect to its specication.
Stocks. Most of the later work on testing from model-oriented specications is
based on this method. In Stocks PhD thesis and his subsequent work, this formal
partitioning technique is applied to Z [Sto93, SC93b, SC93a, CS94]. Stocks denes
a framework in which classes of test-cases can be dened with Z-schemas. He shows
how dierent partitioning strategies can be applied to Z specications. In addition,
mutation testing is applied to formal specications (see Section 2.2 below).
Horcher & Peleska. In [HP94, HP95] a Z-based tool for partitioning is presented
by Horcher & Peleska. Its industrial application demonstrated that the test-case
selection process was not signicantly faster with this approach. The reason was
that the same degree of understanding for the specication must be reached as for
manual test case selection. However, the evaluation of the tests took only 5% of the
usual time since test oracles had been compiled from the Z specications [HM96].
Test oracles are discussed below in Section 2.5.
Helke et al. [HNS97] demonstrate that a theorem prover like Isabelle [Pau94]
can be used to generate such test-classes based on DNF rewriting. It is shown how
constructive proofs may be used to calculate the partitions of a Z representation in
the HOL instance of this generic theorem prover. HOL stands for higher order logic
and is based on Alonzo Churchs work on a simple theory of types [Chu40].
Van Aertryck. In [Aer98] the tool-supported method CASTING for automat-
ically generating test-cases is presented. CASTING is generic in the sense that
it is applicable to dierent model-based specication languages as well as to pro-
gramming languages. This is realized by an internal representation using attribute
grammars in which the test-case selection strategies can be encoded. An instan-
tiation of CASTING for the B-method is presented, where DNF partitions can be
generated automatically from Bs abstract machine specications. The tool uses a
constraint solver for nding actual test-cases.
Behnia & Waeselynck. A problem concerning DNF partitioning and classical
logics is discussed in [BW99]: DNF rewriting of B specications may lead to unde-
ned partition specications.
Consider a specication expression, where f is a partial function.:
(x dom(f ) f (x) = y)
Applying the DNF approach would result in three disjoint partitions:
(x / dom(f ) f (x) ,= y) (x / dom(f ) f (x) = y) (x dom(f ) f (x) ,= y)
It can be easily seen that the rst two cases are undened, since f (x) is undened.
The authors propose an extended deduction system, where denedness is treated
explicitly by an operator denoting well-denedness. Such problems have been
foreseen by Jones and are solved by extending VDMs logic to a three-valued logic
of partial functions, where the third logic value is undenedness [Jon90, BFL
+
94].
Consequently, for VDM this problem does not exist the partitions above are
well-dened in VDM-SL. This shows that Jones is right, when he says in [Jon98]:
The lessons which I believe it is important to draw from my series of
attempts to nd a satisfactory way to reason about partial functions
include the fact that a real diculty should be faced rather than ignored
( I can accept any approach to this problem much more readily than
pretending it does not exist).
Behnia & Waeselynck solve this problem for the B method by designing an alterna-
tive set of partitioning heuristics that do not generate undened partition predicates.
Their approach is described in Section 2.1.4.
2.1.3 Separation of the Input
Hieron
Hieron proposes another heuristic of partitioning model-based specications that is
coarser than pure DNF-partitioning [Hie97]. His partitioning technique rewrites Z
operation predicates as (X
1
Y
1
) (X
n
Y
n
) where
predicates X
i
refer only to the inputs;
predicates Y
i
refer to the outputs.
This form of the predicates separates the description of the input X
i
from the op-
erational part Y
i
specied by a relation. Since X
i
and Y
i
may contain disjunction,
the rewritten operation predicate is not necessarily in DNF. The motivation is not
to separate constraints that apply to the same category of variables.
For example, let us reconsider an operation for calculating the maximum of two
numbers with the formal input parameters x, y and the output parameter max.
Then the predicate that species the computation of the maximum of x and y
((x > y x = y) (max = x)) ((x < y) (max = y))
is of the desired form (X
1
Y
1
) (X
2
Y
2
). As argued by Hieron, the predicate
X
1
needs not to be further split: it characterizes an input subdomain with uniform
behavior.
However, unlike in our example above, the resulting input domains X
1
and X
2
may overlap. Hence, for obtaining disjoint partitions of the input domain, the
input predicates have to be rewritten with the rule for DNF-partitioning. Then,
the resulting partitions for two predicates ,like in our example, would be: X
1
X
2
,
X
1
X
2
and X
1
X
2
.
A further advantage of this transformation is that non-determinism can be han-
dled as well. A non-deterministic specication of the form X
i
(Y
A
i
Y
B
i
) leads to
three possible states, if rewritten as a DNF:
(X
i
Y
A
i
Y
B
i
) (X
i
Y
A
i
Y
B
i
) (X
i
Y
A
i
Y
B
i
)
However, these three states do not directly correspond to three valid tests if consid-
ered separately, i.e., tests that will not reject a correct program. Here, a union of
two possible test-results is split into three deterministic test-results. If the system
is testes with these three test-cases, two of the test-cases might fail. In Hierons
framework this problem does not occur, since such non-deterministic specications
are not further split.
This partitioning is courser than pure DNF-rewriting as described in the previous
section, since the individual X
i
and Y
i
have not been expanded. Even more control
of the number of test-cases is provided, if the syntactical structure of a specication
language is taken into account. Such an approach is described in the next section.
Donat
Donat [Don98] takes a similar approach as Hierons did. Donat distinguishes between
the stimulus (input) expressions S and the response (output) expressions R in a
systems specication. The dierence to Hierons work is that specications are
rewritten into a disjunction of, what he calls, test-classes:
(S
1
R
1
) (S
n
R
n
)
In contrast to the previous work on partition testing, quantied expressions can
be handled in his partitioning algorithm. This is done by translating a specication
into a test-class normal form, where the quantiers and are shifted inside
conjunctions and outside disjunctions, if possible.
From a test-class S R, a set of test-frames A R is derived. In contrast to
a test-class stimulus S, a test-frame stimulus A does not include disjunctions and
A S holds. A variety of dierent test-frame sets can be derived from a test-class.
Several strategies and their resulting test coverage are described. Donat uses Binary
Decision Diagrams [Bry86] to perform the test frame construction and selection.
2.1.4 Structural Partitioning
The notation of the B method is a general purpose language ranging from abstract
specication constructs to implementation oriented statements. Consequently, spec-
ication and implementation oriented testing strategies can be combined. Recently,
this unication has been presented in [BW99]. The unication is made by extending
the concept of control-ow graphs to non-deterministic specication operators. In
this work, the deterministic programming statements are not rewritten into basic
predicates and then are DNF partitioned, but path-oriented strategies known from
white-box testing techniques are applied. The advantage of considering the syntax
of a modeling language like B, is that a better control of the partitioning is provided:
In general, pure DNF partitioning results into too many test-cases. Therefore, cov-
erage strategies from program testing are extended to handle the whole specication
language of B. The result is a hierarchy of testing strategies:
All-paths criterion: This criterion demands that all feasible paths of the control
ow graph are covered. For a specication that is rewritten as a set of non-
deterministic choices, here denoted by S
1
. . . S
n
, each S
i
is a path on the
control ow graph. For example an if-statement can be rewritten as follows:
if P then S
1
else S
2
(P S
1
) (P S
2
)
Hence, test-cases from two input-domains satisfying P and P have to be
tested for covering the two paths.
2.2. MUTATION TESTING 21
All-paths combinations criterion: This criterion demands that all combinations
of feasible paths on the control ow graph should be covered by test data. If
several input-domains are obtained according to the all-paths criterion, then a
DNF-rewriting of these input-domains provides a coverage of all possible path
combinations.
All-expanded-path-combinations criterion: This criterion splits infeasible path
conditions into further sub-cases. A relational update [P S] may be infea-
sible either if P is false or if P is true and S is infeasible.
These criteria can be further rened by splitting guard condition predicates into
disjoint cases.
2.2 Mutation Testing
An alternative and complementary approach to partition testing is mutation testing.
Mutation testing is a fault-based testing technique introduced by Hamlet [Ham77]
and DeMillo et al [DLS78]. It is a means of assessing test suites. When a program
passes all tests in a suite, mutant programs are generated by introducing faults into
the code of the program under test. Then, the suite is assessed in terms of how
many mutants it distinguishes from the original program.
Budd & Gopal [BG81] applied this concept to the domain of specications given
in predicate calculus. Potential faults are represented as mutants of the specication.
Test data are then generated that can distinguish between the mutated and original
specication. Several promissing program mutations are given as examples, but the
generation technique is left implicit.
Stocks extends this technique in [Sto93] to model-based specication languages
by dening a collection of mutation operators for Zs specication language. An
example for specication mutation is the exchange of the join operator of sets
with intersection . From these mutants, test-cases are generated for demonstrating
that the implementation does not implement one of the specication mutations.
Kuhn developed a similar approach [Kuh99]. Faults were represented as mutated
formal specications and the conditions for distinguishing them from the original
specication were calculated. This information was then used to calculate a fault
coverage hierarchy. However, the conditions for detecting the faults were only shown
for mutations involving Boolean operators.
2.3 Structuring Test-Cases
In general, an automated DNF-partitioning produces many unstructured test-cases.
Therefore, some kind of user interaction is necessary for selecting and structuring
the relevant test-cases in a test suite. Classication trees and abstraction hierarchies
are suitable tools for handling this problem, as explained in the following sections.
2.3.1 Classication Trees
In order to enhance DNF partitioning, the classication tree method is applied for
selecting only interesting partitions in [SCS97]. A classication tree is a graphical
representation of the stepwise partitioning of an input domain into relevant parti-
tions [GG93, GWG95]. The root of the tree represents the complete input domain
of the system under test. Its children nodes are predicates representing the parti-
tions. The tree grows as the partitions are rened. The leaves represent the nal
partitioning of the input domain and serves as the head of a selection table from
which the actual test-cases are chosen. The presented tool provides a graphical user
interface that represents the partitions of a Z specication in such a classication
tree. The advantage of this approach is that the partitions are well structured and
that only relevant ones can be rened or chosen to be nal interactively. This solves
the test-case explosion problem due to DNF partitioning.
2.3.2 Abstraction
Stepney realized the abstraction relation between Z specications of test-cases and
object-oriented Z specications of operations [Ste95]. Test-case specications are
abstract in the sense that only relevant input and before-states are tested and that
only interesting consequences of an operation should be observed. Formally, this
means that for designing a test-case specication of an operation, the precondition
is strengthened and the postcondition is weakened. This is the inverse process of
renement abstraction. In other words, her method of test suit design works by
introducing non-determinism into the original specication.
Her group developed a tool for interactively calculating such partition abstrac-
tions and to structure them for reuse. With this tool, the test-cases of a test suite
are structured in a hierarchy of abstractions visualized in a tree. The root of the tree
is the original specication. Following the tree towards its leaves means abstraction.
The children are either a set of partitions, or more non-deterministic versions of the
parent test-case specications.
Our work can be seen as an extension of this view on testing. Unlike Stepney, we
use the renement calculus for dening the abstraction rules for designing test-cases.
Furthermore, we cover the sequencing of test-cases that is only mentioned by her.
Chapter 5 presents our approach to this topic.
2.4. TEST SEQUENCING 23
2.4 Test Sequencing
Simple input-output test-cases are not sucient for non-trivial systems. An interac-
tive system has to consider the possible user actions. Usually an internal state under
which an operation should be tested cannot be set directly, but must be set up by
invoking a series of preceding operations. The process of nding such sequences of
operations is called test sequencing.
2.4.1 Finite State Machine (FSM) Calculation
For model-based specications, this search for test sequences is non-trivial. The
reason is that before- and after-states are not explicitly available, but implicitly
dened in postcondition relations.
In [DF93] Dick and Faivre describe a test sequencing method, based on the cal-
culation of a nite state machine (FSM) from a VDM specication. Their approach
works as follows:
1. Partitioning of the original specications using the DNF technique described
above. A set of sub-operations is obtained that form the transitions of the
FSM.
2. Calculation of the before- and after-states of each suboperation by existential
quantifying every variable external to the state in question and then simplify-
ing.
3. DNF-partitioning of the disjunction of the set of states obtained in 2. The
result is a description of disjoint states the states of the FSM.
4. Construction of the FSM by resolving the constraints of sub-operations against
states. The FSM has a transition between states s
1
and s
2
if a postcondition
relation R of a sub-operation is satised, hence R.s
1
.s
2
holds. The transition
is labeled with the name of the corresponding sub-operation.
Having the FSM explicitly available, paths through the FSM can be extracted
as test sequences. The strategies for selecting the sequences can be chosen from the
well-known path strategies of white-box testing. Examples for such strategies are
to visit each node (here state) once, or to visit each transition (here sub-operation)
once, or to cover all possible paths without repetition. More about path strategies
can be found in [Bei90] and [Bei95].
2.4.2 Renement of a FSM
Recently, Derrick and Boiten discussed the issues of test sequencing and formal
renement in Z [DB99]. They provide means to calculate a concrete FSM from an
abstract one using a data renement relation (retrieve relation). If this renement
relation is functional and the rened specication is the weakest renement (no
additional operational renement) then disjoint abstract states and transitions will
produce disjoint concrete ones the two FSMs are isomorphic. If the renement
relation is not functional no further transitions can be added, but states may be
identied under renement. However, if further operational renement is performed,
then the process of renement adds new transitions and states by weakening a
precondition. In addition, existing transitions may be deleted by strengthening a
postcondition.
2.5 Test Oracles
In the previous sections, the review focused on test-case generation techniques, es-
pecially on techniques for partitioning the input domain of a system under test. The
second main task in testing, to be considered, is the evaluation of the test results
using an oracle. An oracle predicts the outcome of a given test. Previous work has
shown the derivation of such test oracles from formal specications.
The conceptually simplest form of an oracle is a comparison of the actual output
for some input against a pre-calculated expected output for the same input. An
executable prototype can be used to calculate the expected outputs. Some of the
formal method tools support this kind of prototyping by providing
specication interpreters, and/or
code-generators
for executable specications, e.g. IFAD VDMTools and Atelier B to name only two.
For example, [TDS98] reports the use of a C-prototype that has been generated
from a B specication, for preparing test-cases. In this project, the test-results of
the prototype had been stored into a test le and then were rened into a suitable
form for testing the Ada implementation. Similar works exist for VDM [Dro00,
vdBVW99].
The alternative and more general approach is the use of possibly non-deterministic
formal specications as oracles. The partitioning approaches described above rewrite
a specication as a set of test-class specications with disjoint input domains. These
formal specications do more than describe conditions on the input. The relation-
ship between the input states and output states is precisely specied. This means
that the specication predicates can serve as an oracle if rened into an executable
form.
Consider the specication predicate that denes a computation of the square-
root of a positive number x. If the result is stored in a variable s, and e is a given
precision, then we have the following relation:
-e x s
2
e
2.6. ALGEBRAIC SPECIFICATIONS 25
If this relation is evaluated it can serve as an oracle, and can decide if a test
succeeds. This is in contrast to using a prototype for calculating the actual square-
root which is far more complicated then using the specication above.
In [HM96] such a test environment is described that generates a predicate eval-
uator as an oracle from a Z specication.
In Chapter 7 an alternative approach is shown, where implicit VDM specica-
tions are directly used as test-oracles.
2.6 Algebraic Specications
In the following, an overview of the algebraic testing techniques proposed by Marie-
Claude Gaudel et al. [BGM91, BGM92, Gau95, GJ98] is presented. Although,
algebraic specications are not the subject of this thesis, the review of this work
should explain the dierences to model-based testing techniques. It has to be men-
tioned that Dick & Faivres early testing techniques on VDM have been initially
inspired by this work [BGM91].
Figure 2.1 shows an algebraic specication of sorted lists in the notation of
LOTOS [ISO89]. Such a specication has two parts: a signature = (S, F), where
S is a nite set of sorts and F is a nite name of operation names over the sorts in S,
and a nite set of axioms. The sorts in our examples are List, Elem and Boolean;
the new operation sorted is dened by three axioms.
2.6.1 Testing a Data Type Implementation
Testing code against an algebraic specication consists of showing that the nal
system satises the axioms in the specication. To create tests for a given axiom,
the variables of the axiom are instantiated with values. To run such a test, the
resulting expressions are evaluated. If the results satisfy the axiom then the test is
passed, otherwise it is failed.
An advantage of algebraic specications in testing is the ease with which test
data are constructed. The constructive nature of the specications shows how to
build various instances of the dened data types, which also implicitly include test
sequencing information. There is also great potential for automatically generating
test cases from specications by steadily building up data from the base cases using
constructor operations.
In the SortedList example, a possible test, inspired by the third equation, is:
sorted(cons(A, cons(B, cons(C, nil )))) = (A B) sorted(cons(B, cons(C, nil )))
The corresponding test procedure for testing an implementation P consists of two
computations and one comparison, namely
computing the representation in P of cons(A, cons(B, cons(C, nil ))), calling
the sorted function in P on it, and storing the result;
specification Sorted_List_Example : exit
library Boolean, Lists endlib
type SortedList is
List, Elem, Boolean
opns
sorted : List -> Boolean
eqns
forall xs: List, a, b : Elem
ofsort Boolean
sorted(nil) = true;
sorted(cons(a, nil)) = true;
sorted(cons(a, cons(b,xs)) = (a <= b) and sorted(cons(b,xs));
endtype
Figure 2.1: An algebraic specication of sorted lists as an extension of general lists.
similarly, calling sorted on cons(B, cons(C, nil )), checking if A B and build-
ing the conjunction of the results.
comparing the two results (oracle).
This shows that in algebraic testing, the test-cases are equations and the job of an
oracle is the comparison of the computed results of the left- and right-hand side of
an equation.
2.6.2 Selection Strategies
Gaudel et al. distinguishes between two kinds of selection hypothesis for reducing
the size of test set to practical levels.
A uniformity hypothesis is an assumption that the input-space can be divided
into sub-domains such that if a test set containing a single element from each sub-
domain is passed, then the exhaustive test set is also passed. This corresponds
to the hypothesis used in partition testing. Examples of uniformity hypotheses
are the selection of representative values, or the partitioning of the input-space by
considering the conditions in conditional axioms.
A regularity hypothesis uses a size function from ground -terms to the natural
numbers and assumes that if a set of tests made up of ground terms of size less than
or equal to a given number, then the exhaustive test set is also passed.
Example 2.2 (Algebraic Testing) A rst selection strategy on the SortedList
example (Fig. 2.1) would be to make a uniformity hypothesis on all the variables.
2.6. ALGEBRAIC SPECIFICATIONS 27
Case 3.1: (a <= b) = true and sorted(cons(b,xs)) = true
sorted(cons(a, cons(b,xs))) = true
Case 3.2: (a <= b) = true and sorted(cons(b,xs)) = false
sorted(cons(a, cons(b,xs))) = false
Case 3.3: (a <= b) = false and sorted(cons(b,xs)) = true
Case 3.4: (a <= b) = false and sorted(cons(b,xs)) = false
Figure 2.2: Examples of algebraic test-cases for testing an implementation of sorted
lists as specied in Figure 2.1. The tests are synthesized by unfolding the Boolean
data type denition in the third axiom.
This means that that there are three tests, which are arbitrary instantiations of the
axioms.
If for some reason more tests are desired, one possibility is to unfold the denition
of and in the third axiom. Since the and operations is dened as follows
true and true = true
true and false = false
false and true = false
false and false = false
, the four new conditional axioms in Figure 2.2 are obtained.
More test-cases can be derived, if the <= operation is further unfolded. To
obtain the actual test cases, the variables have to be instantiated. Here, a regularity
hypothesis could be applied that limits the length of the lists to be instantiated to
two.
The method has been automated in the tool LOFT (LOgic for Functions and
Testing) using Prolog [Mar95]. Furthermore, it has been successfully applied on a
substantial case-study [DM91]. Hughes & Stotts applied the technique for testing
object-oriented programs [HS96]. More recently, these algebraic testing techniques
have been combined with behavioral techniques of LOTOS [GJ98]. Testing with
behavioral specications is discussed in the next section.
Currently, Andrej Pietschker is developing a new test-case generator based on
algebraic specications [PF98]. In this work the problem of mapping the abstract
test-cases of the specication to concrete implementation level is considered. Given
the abstract tests, a reverse parsing technique is applied that uses a part of the
specication expressed by means of a grammar to produce tests in the concrete
representation.
Antoy & Hamlet have presented a framework, where implementations are auto-
matically checked against their algebraic specications [AH00]. In contrast to the
algebraic test approach described above, they evaluate the specication by rewriting
it following the axioms. Here, the test-results of the implementation are mapped
and compared to the resulting terms of the specication. Only, simple examples of
testing a set implementation are shown.
2.7 Behavioral Specications
A specication is called behavioral, if the dynamic behavior of a system is described
rather than its data and data transformations. Finite state machines and labeled
transition systems are examples of this kind of specication style. Behavioral spec-
ications are applied in domains where systems have a rather complex state but
simple state transitions, like in communication protocols and control applications.
Behavioral specications are well suited for automatically generating sequences
of test-cases, since feasible applications of state operations are made explicit. There-
fore, Dick & Faivre have shown how to produce a behavioral nite state machine
from a computational VDM specication as has been explained in Section 2.4.
A class of test generation approaches starts directly from such behavioral speci-
cations, like nite state machines [Cho78, FvBKG91, LY94] or nite labeled tran-
sition systems [dNH84, Hen88, Bri89, PF90, Tre92, Pel96, PS96] .
Another class of approaches combines computational and behavioral specica-
tions. As we do, MacColl and Carrington are planing to develop a testing framework
for interactive systems. They use a combination of Z and behavioral specications
(the process algebras CSP and CCS) for specifying such systems [MC99]. In [GJ98]
such behavioral specications are combined with algebraic testing techniques. Fur-
thermore, in [SS98] testing from combined Z and statechart specications is dis-
cussed.
2.8 Discussion
As shown above, a lot of work has been done on generating test-cases from formal
specications. However, in this thesis it is demonstrated that there is still room for
innovation.
2.8.1 A New Approach to Test Strategies
This thesis seeks to extend the body of work for model-based specications summa-
rized above. A new framework under which all testing strategies, like partitioning
and mutation heuristics, can be formally specied, analyzed and used to automate
the generation of test case specications. In contrast to previous works with the
2.8. DISCUSSION 29
same goal, e.g. Burtons recent work on Z [Bur00, BCGM00], our mathematical
treatment of test-cases can handle test-sequencing as well. Unlike the work above,
our approach is based on program synthesis techniques. Since test-case generation
is a synthesis process, we nd it more natural to use a development method like
the renement calculus in order to calculate test-cases. The basis to this uniform
framework is the view that all kinds of test-cases are mathematical abstractions
of model-based specications. This new approach to testing will be presented in
Chapter 5.
Having realized the abstraction relation of test-cases, the renement calculus
can be applied for formalizing synthesis rules for testing. This is the rst time that
Back and von Wrights renement calculus [BvW98] is applied for the development
of test-cases. In Chapter 6 test strategies for partitioning, mutation testing as
well as test sequencing are formalized as proof rules in the renement calculus. It
will be shown that the formalization itself revealed interesting properties of the
heuristics that have to be considered, when automation is envisaged. The uniform
framework based on the abstraction relation provides an environment under which
the relationship between various testing heuristics can be investigated in order to
develop more ecient heuristics.
2.8.2 Algebraic vs. Model-Based Specications
Algebraic specications are not the topic of this thesis. As explained in the introduc-
tion, one of our motivations for this work is the technology transfer of formal spec-
ication techniques into industry. The author shares the opinion of Jones [Jon90]
that model-based specications are more adequate for the practicing engineer and
that algebraic denitions should be left to the formal method experts. Although in
general, algebraic specications are more abstract than model-based specications,
their validation is more dicult for non-mathematicians: It is not easy to determine,
if the axioms in a complex algebraic specication are sucient and sound.
With respect to testing Hayes [Hay86] argues that algebraic techniques are best
suited to testing primitive data types and that, for more complex abstract data
types, a model-based specication is simpler. As noted by Gaudel, predicate logic
specications are more general than algebraic specications. However, the price of
this generality is the restriction that, in general, only specications of classes of
test-cases can be generated automatically. Algebraic techniques can generate test
data as well.
It is our opinion, that a higher degree of automation of test-case generation
should not be gained through a specication technique that is more error-prone
for non-experts in formal methods. The experiments presented in Chapter 3 will
demonstrate how error-prone informal specications are. The situation should not
be made worse by introducing another source of errors.
2.8.3 Taking the Dynamic Behavior into Account
The following chapter will show that todays safety-critical systems may be highly
interactive as well as involve complex data manipulations. Typical test-cases of such
complex systems are scenarios of usage. Thus, a testing strategy must treat both,
the input-output relation as well as the possible sequences of interaction.
As has been described above, recent work on model-based specications has
focused on the combination of behavioral and computational methods for covering
the dierent aspects of functionality. In contrast, this work uses Back and von
Wrights contract language for describing the behavior of interactive systems. It
has been shown that Back and von Wrights renement calculus covers such type of
systems in [BMvW99].
The innovation in the work presented here is that the synthesis of test-sequences
(scenarios) from a specication is considered as an abstraction problem. Our se-
quencing technique does not calculate the complete nite state machine prior to
test-case selection and thus diers from the approaches above. In an environment,
where change requests are common, calculating the whole nite state machine is
not very ecient. Our method focuses on the impact on the possible scenarios by
systematically analyzing interaction compositions.
2.8.4 How Much Does It Cost?
All the methods presented above rely on formal specications and thus will not be
applied in industry if formal specications fail to be accepted. A common question
of the practicing software engineer is: What are the eorts and the payos?, or
simply How much does it cost?.
Too few of the previous works answer this question. One of the reasons is that
often only simple examples serve to demonstrate new research in the area of testing.
In this work, we try to answer the question partly by presenting the results of two
industrial experiments described in the next chapter. In these experiments a light-
weight approach to testing has been used that does not represent fundamentally
new techniques. The test method is based on executable specications. However,
the exact gures of the eorts and detected issues are presented. Furthermore, the
empirical results have motivated our more theoretical research of Chapter 5 and 6.
2.8.5 New Tools New Oracle Approaches
It has been shown that formal specications have been used as test oracles before.
However, in this area of automating the test process, the availability of new tools is
essential. Previous work on oracles has mainly focused on code generation or directly
executable specications. In Chapter 7 we will present a novel oracle technique,
where recently available VDM tools are combined with well-known proof techniques
2.8. DISCUSSION 31
of VDM. This enables the direct use of VDMs pre-post condition specications as
test oracles.
Chapter 3
A Light-Weight Approach to
Testing
In this chapter a test design method is presented that has been applied in two
industrial projects. It is a method for designing system-level test cases for complex
computer-based systems. This light-weight approach is based on executable formal
specications that capture the user requirements of a system under test.
The details of the two projects and their encouraging results are described in
the next chapter. Here, the method and its foundation are introduced. After a
denition and discussion on light-weight formal methods, the Vienna Development
Method is described in more detail. Next, it is shown how VDM and VDMTools can
be used in test design. Finally, the relation between our approach and the important
validation aspect in testing are discussed.
3.1 Light-Weight Formal Methods
During the last few years, the interest in formal software development has been
growing rapidly. One indicator for this growing popularity is the fact that every
main conference on software engineering features sessions about formal methods.
Several reasons can be identied; among others the following three:
the combination of formal and more traditional approaches,
the availability of commercial tools to support the developer,
the beginning awareness that the application of formal methods does not nec-
essarily imply the use of formal correctness proofs.
The last item lead to a new class of formal methods that is often called light-weight
formal methods or formal methods light.
33
34 CHAPTER 3. A LIGHT-WEIGHT APPROACH TO TESTING
These light-weight approaches to formal methods provide the advantages of a
precise and unambiguous specication language to raise the quality of a systems
specication, without focusing on proofs [DKLM98, ELC
+
98, JW96, Jon96].
In this chapter such a light-weight method is presented and its relevance for
system-level testing is discussed and demonstrated.
It is important to distinguish between the terms light-weight and informal meth-
ods.
light-weight implies a formal specication language with a formal semantics. Al-
though, formal proofs are out of a light-weight formal methods scope, formal
verication is possible if needed or wanted.
informal means that no formal semantics of the requirements description is avail-
able. Therefore, formal proofs of properties of a described system are im-
possible. Natural language and the Unied Modeling Language (UML) are
examples of informal specication methods.
The advantages of a light-weight formal method will be demonstrated in the next
chapter, when the results of our industrial projects are presented. These advantages
can be summarized as follows:
+ easy to learn: our students learn our method during a one semester course con-
sisting of two hours of lecture and one hour of exercise per week.
+ easy to apply: formal proofs are dicult or even infeasible for software developers
that are not trained in discrete mathematics. However, experience shows that
software engineers are able to specify their systems in a formal language after
two weeks of training.
+ unambiguous descriptions: the formal language provides a tool for specifying
without the ambiguity introduced by informal description techniques.
+ complete descriptions: in a formal specication it is obvious and precisely de-
ned what kind of properties are described and what is postponed for later
renement.
The disadvantages of skipping a formal verication process are that questions con-
cerning
soundness: a formal specication may contain subtile errors that may solely be
detected by formal proof techniques;
correctness: the correctness of an implementation with respect to its specication
cannot be guaranteed without formal correctness proofs;
deeper insight: in general, formal proofs of system properties provide a deeper
insight into the consequences of system models; cannot be answered without
formal proofs.
3.2. VDM AND TOOLS 35
Some formal methods facilitate the specication in an explicit style such that
a tool can interpret or generate code from this kind of formal specications. The
advantage of these executable specications is
+ that testing can be used for raising the condence in the specication.
The disadvantage of executable formal specications is that
they are in general less abstract than more implicitly dened specications, like
non-deterministic pre- and postconditions.
Our light-weight approach to testing is based on executable specications of a
systems requirements. The method uses these abstract system prototypes not only
for checking the specication, but for:
the validation of the requirements,
the validation of existing system-level test cases,
the design of system-level test-cases, and
the validation of the formal specication.
The formal method that provides such a light-weight application is the Vienna
Development Method supported by IFADs VDMTools. The following section will
give a short introduction to VDM and the used set of tools.
3.2 VDM and Tools
3.2.1 The Vienna Development Method
In the following, the Vienna Development Method (VDM) and its specication lan-
guages are described. The contents are partly based on [PT89, PvKT91, LHB
+
96,
Lar95].
VDM is a formal method for the description and development of computer based
systems. It is a model-based method, i.e. its formal descriptions (VDM specica-
tions) consist of a model of the system being constructed.
A system design is generated through a series of specications, where each spec-
ication is more concrete and closer to the implementation than the previous one
[Bj89]. Each of these development steps introduces proof obligations which, when
appropriately discharged, ensure the (relative) correctness of the implemented sys-
tem.
VDM is one of the most widely used formal methods and has been applied to
the construction of a large variety of software systems [Bee86, Be80, Bod88, BFL96,
Cot84, Han94, JS93, Lam91, MS93, OH95, Sad88, TaA88].
Its history starts in the 60-ies, when, in the IBM laboratory in Vienna, a group of
people, managed by Heinz Zemanek, worked on formal language denition and com-
piler design. The goal was to create an operational semantics for the programming
language PL/I [LW69, Luc87]. By outsiders the meta-language used was dubbed
the Vienna Denition Language or VDL. Due to the complicated formal reasoning
about the operational semantics, a denotational semantics approach, was taken in
1972 with the meta-language Meta-IV. In the past, Meta-IV has been used in many
dierent variants. First of all, these variants reect several dierent styles in the use
of VDM:
the Danish school,
the English school,
the Irish school.
The dierences between the schools is reected in their dierent application areas.
The Danish school focused on modeling large software systems, and thus used VDM
to abstract from complex systems for better understanding, whereas the English
school has concentrated more on the correctness of smaller systems and used a
more implicit style in order to ease the verication of the systems. The Irish school
of VDM can be described as a constructive style of VDM with a heavy use of
algebraic structures, like monoids or groups, for modeling. Here, all specications
are formulated in an explicit (executable) style.
In the mid-eighties the British Standards Institute (BSI) started to harmonize
the dierent variants producing a standard for Meta-IV [Sen87, PT92, Par94]. In
the nineties this was taken up by the International Organization for Standardization
(ISO) [LHB
+
96] and Meta-IV became VDM-SL.
VDM consists of two major components:
The VDM Specication Language, VDM-SL. The language describes a notation
in which VDM specications can be expressed.
A method describing the development from a specication to an implementa-
tion in a certain programming language. The method part of VDM is not as
precisely dened as the VDM-SL notation.
The VDM bibliography [Lar00] contains more than 500 references including tu-
torial text books [Jon90, FL98], a reference manual of VDM-SL [Daw91] and a book
about proofs in VDM [BFL
+
94].
In the 90-ies, an object-oriented version of VDM has been created VDM++.
Beside its object-oriented features, VDM++ oers specication techniques for cap-
turing the dynamic behavior of parallel systems as well as for the timing constraints
of real-time systems. In contrast to VDM-SL, the specication language of VDM++
is not standardized and therefore its syntax and semantics are still subject to changes
[Lan95, DG95, DGP95, KL96, IFA00, Ins00].
3.2. VDM AND TOOLS 37
In the light-weight use of VDM, the author follows the tradition of the Danish
school of VDM. In this thesis the main focus concerning VDM is rather on modeling
complex computer-based systems than on proving their correct implementations.
However, the proof obligations for a correct renement will be used for testing
purposes in Chapter 7.
It might be due to the long tradition of light-weight applications of VDM in
Denmark that the tools that provide the best support of this approach are Danish
the IFAD VDMTools.
3.2.2 IFAD VDMTools
Tool support of formal methods is crucial for transferring these techniques into
industry. Our light-weight approach is based on the features currently oered by
the commercial IFAD VDMTools. Two versions of these tools exist, one for the
traditional VDM-SL, one for the object-oriented VDM++. Both have been used in
our industrial experiments.
The following features are provided by the last versions of IFAD VDMTools:
Syntax checking of VDM specications. To provide a literate programming style
[Knuth84] for specications, the VDM source code may include L
A
T
E
X typeset
text under the UNIX family of operating systems. Under Windows, VDM
specication can be included into MS Word documents.
Static and dynamic type checking features the testing of pre- and postcondi-
tions, and data-invariants.
Pretty printing. The VDM source code in ASCII format is converted into the
mathematical notation for VDM-SL by generating a L
A
T
E
X document under
UNIX.
Execution. Specication interpretation is provided for all executable language con-
structs.
Test coverage analysis. Test coverage information can be automatically recorded
during the evaluation of a test-suite. The specier can at any point check which
parts of the specication are most frequently evaluated and which parts have
not been covered at all.
Source-level debugging. The debug mode is used to interpret the specication.
This mode provides an overview of break points, function trace, and denitions
available in the specication.
Specication management maintains a project by keeping track of the status of
modules and remembering this status from session to session. Furthermore,
all the functionalities are accessible through a graphical user interface.
Code generation An explicit form of a specication that may be compiled into
C++ or Java source code.
Dynamic Link Facility With this feature external C++ libraries may be linked
to the VDM-SL tool. This enables the combined interpretation of VDM-SL
specications and C++ source code [FL96].
CORBA API For VDM++ an API has been developed through which the tool can
be accessed as a CORBA object. This enables the call of the specication from
inside a programming language that is supported by the CORBA standard
[OMG96].
The specication interpreter together with the test coverage analysis tool is
mainly used for our light-weight approach to system-level testing. The possibili-
ties of combining programs and executable specications establishes the test oracle
framework that is introduced in Chapter 2.5.
3.3 Test Design with Abstract Prototypes
3.3.1 Abstract Prototypes
The subject of this thesis are abstraction techniques in testing. In the following,
our light-weight approach that is based on an abstract but executable model of the
systems requirements is introduced. Such an abstract prototype that is dened in
a formal specication language, like VDM-SL, diers from conventional prototypes
as used in rapid prototyping processes in many ways:
abstract model of state: the prototype is abstract in the sense that implemen-
tation details are not included in the model of the state. Here, an adequate
abstraction level mainly depends on the abstraction level of the requirements.
data invariants: the state model can be further constrained by adding logical
properties to a data model. These data invariants must hold before and after
the execution of an operation and can be dynamically checked during inter-
pretation of the prototype.
preconditions: further abstraction of the formal prototype may be gained by re-
ducing the denition of a systems functionality to common cases explicitly
stated in the requirements. Special cases, like exceptions may be subject for a
later, more detailed design. Preconditions are used for dening the property
that must hold for executing an operation succesfully. Here, successful execu-
tion means an execution that terminates. Preconditions may be dynamically
checked during execution of the prototype.
3.3. TEST DESIGN WITH ABSTRACT PROTOTYPES 39
advanced language features: A specication language usually oers more ad-
vanced language constructs than traditional programming languages. This is
due to their mathematical nature. Examples of such features are
logical quantors that are executable over nite domains;
comprehension expressions, like set comprehensions;
pattern matching, like in functional programming languages.
These language features enable an executable formal specication to serve as an
abstract prototype for test-case design and validation. In the following, the approach
is explained in more detail.
3.3.2 The Light-Weight Test Approach
Our method proposes the use of a model-based specication language with an exe-
cutable language subset supported by a specication interpreter. In our case VDM
and IFAD VDMTools have been chosen for building a formal and executable model
representing the requirements of the system under test. Although the model is exe-
cutable, it is still abstract in the sense that implementation details are not included.
Only abstract mathematical objects like sets, sequences, nite mappings are used
to capture the relevant requirements.
Once an executable formal model has been established, the interpreter tool can
be used to run test-cases on this formal prototype. With this test-case animation
already designed test-cases can be validated, or new test cases can be designed using
structural techniques. IFAD VDMTools support this process with their test coverage
analysis tool that highlights uncovered parts in the VDM specication.
Figure 3.1 presents an overview of the processes of our approach that have been
developed for the two experiments in industry. The phases are described as follows:
1. The starting point is the informal requirements documentation containing the
user and software requirements of the system that should be tested.
2. Usually, in an industrial environment a traditional test design process is al-
ready established. This traditional test design method takes the informal
requirement documents and produces one test-case for each requirement.
3. In parallel to the conventional test design, a formal methods expert creates an
abstract VDM prototype out of the informal requirements descriptions. This
formalization process usually highlights a lot of open issues in the requirements.
These are documented in a catalogue of questions concerning the requirements.
4. The requirement engineer reacts to the questions in the catalogue. Some
questions are due to a lack of domain knowledge of the specier, others need
a clarication and then a modication of the requirements. In some cases the
Requirements
Formalization
Requirements
Document
Formal
Specification
Conventional
Test-Cases
Formalized
Test-Cases
Synthesized
Test-Cases
Test-Case
Animation
Test-Case
Formalization
Test-Case
Synthesis
Test-Case
Correction
Requirements
Modification
Test-Case
Generation
Error Report
Catalog of
Questions
Figure 3.1: Processes with inputs and outputs.
questions have to be forwarded to the customer (user) of the system under
development validation has been initiated.
5. When test-cases are available, these are formalized, as well. The formalized
test-cases are sequences of calls to the operations of the abstract prototype.
Furthermore, observer routines are used for checking the test results as spec-
ied in the conventional test documents. This phase uncovers errors in the
test-cases, like missing steps, or wrong input/output specications.
6. The abstract prototype is executed (animated) with the formalized test-cases.
The coverage information that is collected during the test runs highlights un-
covered parts in the requirement specication.
7. Additional test-cases are designed from the test-specication such that 100 %
of the specication are covered.
One will argue that the conventional test-design approach can be skipped and
test-cases can be directly synthesized from the formal specication. However, expe-
rience shows that this twofold test-design approach constitutes a valuable validation
process. This is of high importance, since this kind of test-cases are often used as
acceptance test-cases of the product. Furthermore, it is sometimes the case that a
conventional test engineer has an implicit knowledge of the domain and the test-
cases needed. This means that he designs test-cases for requirements that are not
3.4. FORMAL SPECIFICATIONS AS A CATALYST IN VALIDATION 41
explicitly stated in the requirements Then, it is a promising method to keep the
established test design method and to validate it systematically as described above.
In the following, the validation role of a the formal specication and its impor-
tance for testing is emphasized.
3.4 Formal Specications as a Catalyst in Valida-
tion
3.4.1 Issues in the Requirements and Test Documentation
System-level testing is based on the user and software requirements of the system
under test. Consequently, the quality of the test-cases depends on the quality of the
requirement documentation. Common sources of errors in the test case design are
missing requirements,
ambiguous requirements, or even
unsound requirement descriptions.
This is especially valid for large projects, where system-level tests are designed and
carried out by independent test engineers. This was the case in the two projects
described in the next chapter.
A further source of errors is the complexity of the system-level test-cases itself.
Usually, several interactions with the system are necessary for establishing an inter-
nal state prior to the testing of a given requirement. Designing such a test from an
informal prose document by hand leads to several mistakes in the test design.
Common errors that have been found in the test documentation during our two
experiments are
missing test cases,
missing interactions (test steps) in a test case,
wrong interactions (test steps),
wrong input-data of a test,
wrongly predicted test results.
It has been demonstrated in the two industrial experiments that a light-weight for-
mal method can be used very eciently for detecting issues in both, the requirements
and the test-cases.
3.4.2 A Catalyst in Validation
Our experiments demonstrated the fact that formal specications increased the num-
ber of detected issues in the documents. Therefore, the author describes specication
techniques as a catalyst in the validation process of requirements and system-level
test-cases.
The American Heritage Dictionary denes a catalyst as a substance, usually
present in small amounts relative to the reactants, that modies and especially
increases the rate of a chemical reaction without being consumed in the process.
The presentation of the empirical data gained in the next chapter will demon-
strate the catalyst role of our approach in validating the requirements as well as in
validating test cases.
3.5 Summary and Discussion
In this chapter, a light-weight formal approach to testing has been presented. Light-
weight formal methods and their properties have been discussed. Then, VDM and
its most prominent tool-set has been explained. Both, the specication language
of VDM together with the powerful tool support enables the ecient application
of an abstract prototype in testing as has been explained. Finally, the important
validation role has been discussed.
The chapter does not present any technical novelties concerning tool development
of new formal methods. It is an application of a light-weight formal method for
testing.
However, the strong emphasize of the approach on the validation of both, the
requirements as well as the test-design is new. Especially, to our present knowledge,
it is the rst time that this light-weight method has been applied in two industrial
experiments and that detailed empirical data have been collected data about the
eorts and eciency of the method. These data are needed for convincing decision
makers in industry. It is a prerequisite for transferring the techniques. A fact that
is too often neglected by the research in the domain of formal methods.
Chapter 4
Industrial Experiences
In the previous chapter, a light-weight formal method for testing has been presented.
This chapter reports about our experiences with the application of the method,
gained in two industrial projects. The results of the two experiments support our
claims previously made. The presented data will give insight into the costs and
benets of formal approaches to black-box testing. Short examples from the projects
help to illustrate the approach further. Finally, it is discussed how these experiences
motivated our more advanced test framework described in Chapter 5 & 6.
4.1 The Application Domain
The work presented in this chapter has been performed in two joint projects of the
Austrian company FREQUENTIS
1
and the Institute for Software Technology at the
Graz University of Technology.
The goal of the two experiments has been to apply the light-weight formal test
method in projects of a representative size. The size of the projects made it necessary
that the actual specication work has been carried out by two students in their
masters thesis projects for the degree of a diploma engineer in Telematics
2
. The
students have been educated in formal methods and the test method by the author.
Furthermore, their work has been supervised and steered by him in a very interactive
manner.
4.1.1 Air Trac Control
The core sphere of business of the company FREQUENTIS is air trac control. It
provides voice and data communication systems for safely routing aircrafts between
the worlds airports. The existing voice communication system for air trac control
1
FREQUENTIS Nachrichtentechnik GmbH, Spittelbreitengasse 34, 1120 Vienna, Aus-
tria/Europe, URL: http://www.frequentis.co.at/
2
A combination of computer-science and telecommunications.
43
44 CHAPTER 4. INDUSTRIAL EXPERIENCES
line bound voice communication
voice communication via radio
VCS internal voice communication
Airport with
VCS 3020S system
Another airport
External party
VCS

PSTN
ISDN
Figure 4.1: A typical environment of the VCS 3020S system
VCS 3020 is one of the central products of the company FREQUENTIS and has
been chosen as the target system in the rst project. Voice communication among
air trac controllers and between the controllers and the pilots is crucial to the
safety of commercial aviation. The voice communication system is responsible for
transmitting the commands of the air-trac controllers, concerning starting and
landing clearances, positioning of the air-planes, emergency situations etc.
The work in this domain demonstrates the crucial role that formal specication
techniques play in software engineering. They may serve as a tool for validating
both, the functional requirements as well as existing system-level test-cases. It will
be shown by the data that the validation process of the requirements is essential for
the design of black-box test-cases.
4.1.2 Voice Communication in Air Trac Control
Among other important systems on an airport, like radars, instrument landing sys-
tems, etc., the voice communication system can be seen as the nerve system of
the airport. It serves as the sole communication system between the pilots, the air-
trac control personnel working on the airport (denoted by in Fig. 4.1), the ground
personnel on the runways, other parties external to the airport and even other air-
ports. This typical environment of the FREQUENTIS voice communication system
VCS 3020S is shown in Fig. 4.1.
The system oers all well known services of PABX
3
or PSTN
4
systems, i.e. call
set up and termination, conferencing, call forwarding, call diversion, etc. In contrast
to these systems, it also supports voice communication via radio. The system makes
a clear distinction between line bound duplex voice communication (voice can be
3
PABX . . . Private Automatic Branch Exchange
4
PSTN . . . Public Switched Telephone Network
4.1. THE APPLICATION DOMAIN 45
OP
OP
OP
RIF
LIF
LIF
LIF
VCS 3020S System
PSTN/PABX/
phone set
ISDN
MFC
radio set
SWITCH
Figure 4.2: VCS 3020S system overview
transmitted and received at the same time) and simplex communication via radio
(voice must be either transmitted or received).
Figure 4.2 shows the essential units of the system as well as the interfaces to the
external components. The main voice communication path is established between
the operators working at the operator positions (OP) and line bound parties as well
as radio parties (shown at the left side of the gure). The radio sets, necessary for
communication via radio, are external to the system and are connected to it via
radio interfaces (RIFs). Line bound voice communication is established via the line
interfaces (LIFs), which are highly congurable.
The VCS 3020S system is a switched system intended for real-time transmission
of continuous voice data. This makes it easy to distinguish the VCS 3020S from
both, non-switched broadcast systems and data networks, which are not intended
for real-time transmission of continuous media. The architecture of the system is
distributed and the components are connected by means of bi-directional interfaces.
Furthermore, the safety critical core components of the system are duplicated and
operate in hot stand-by mode.
4.1.3 A Network of Voice Communication Systems
In contrast to the rst project, which covered the already existing voice commu-
nication system VCS 3020S, the second project considered a new communication
system still under development.
The primary goal of the new system is the complete interconnection of air trac
control centers via a fully digital network. The digital network lines will be used for
voice (telephone and radio) and data transmissions. There are several interlinked
air trac control centers, each of them consisting of a network node and a voice
communication system (VCS). See Figure 4.3 for an example of such a network
connecting the main airports in Austria.
One of the advantages of the new system is that lines between the network nodes
VCS
VCS
VCS
VCS
VCS VCS
N1
N3
N5
N6
N4 N2
Figure 4.3: A network for air trac control
are allocated dynamically. Therefore, additional lines can be allocated to compen-
sate a broken line or to provide additional bandwidth in times of high network trac.
This dynamic allocation is also valuable from a cost eectiveness point of view.
Another important concept are roles. Roles are simply functionalities: for in-
stance the functionality to control a certain sector of air trac. These roles - or
functionalities - are not bound to specic OPs, but can be distributed within the net-
work, to optimize the work-load on the controllers. Therefore, it should be possible
to control air space, independent of the physical location of the controller.
4.2 Experiment I
The specication work in this project has been carried out by Johannes Horl, a
student of the author. He had a training of a one semester course in VDM, but
no domain knowledge of voice communication in air trac control. In this project,
VDM
++
has been used the rst time in our group for specifying the requirements
of a non-trivial system.
4.2.1 Requirements Validation
In the following, the techniques and results of modeling the system in VDM
++
are
presented. First, the informal UML model of the object-oriented architecture and
its tool-based mapping to VDM
++
classes are outlined. Then the specication of
frequency coupling serves to demonstrate the specication language of VDM
++
.
Finally, the statistics of the errors found in the informal requirement documents is
presented.
4.2. EXPERIMENT I 47
Architecture of the System
In order to structure the specication, an object-oriented architecture reecting
the physical components has been chosen. The states and methods of the four
classes OP, SWITCH, LIF and RIF dene the main functionality of the system.
Additional auxiliary classes have been used to dene system-wide constraints and
conguration details. Figure 4.4 shows the relations between these classes in UML
notation. The UML class diagrams have been automatically generated from the
formal class specication using the IFAD VDM
++
Toolbox link to the Rational
Rose 98 CASE-tool [IFA98].
LIF
id : LIF_ID
s witch : @SWITCH
RIF
id : RIF_ID
s witch : @SWITCH
OP
id : OP_ID
s witch : @SWITCH
SWITCH
ops : s et of @OP
lifs : s et of @LIF
rifs : s et of @RIF
lifs
0..*
switch
0..*
rifs
switch
switch
0..*
ops 0..*
0..*
0..*
Figure 4.4: Main class relations (in UML notation).
The following specication describes the architecture of the VCS 3020S system in
VDM
++
notation. Here, the specications of the classes RIF and LIF are skipped,
since they are similar to the one of the OP class.
class OP
instance variables
id : OP-ID;
switch : @SWITCH
end OP
class SWITCH
instance variables
ops : @OP-set;
inv ops

card ops 24;
lifs : @LIF-set;
inv lifs

card lifs 96;
rifs : @RIF-set;
inv rifs

card rifs 48
end SWITCH
The bidirectional associations between the components are specied as sets of
references to each other. Additional invariants constrain the number (cardinality)
of possible operator positions and interfaces.
An Example of the Systems Functionality
Within the scope of this project, specication has been limited to the radio commu-
nication system part, since this is the most safety critical part and thus of particular
interest for analyzing the use of formal methods. In order to keep the demonstration
example small, discussion is primarily focussed on the function Coupling/Radio
Re-transmission in this thesis.
Note, that a physical frequency used for voice transmission is associated with
a single Radio Interface (RIF) of the system. The RIF includes hardware and
software necessary for the voice communication and the associated signaling, like
the PTT signal and the SQUELCH-Break signal. It is not possible to change the
physical frequency on which voice is sent or received from within the VCS 3020S
system. The system represents these frequencies as frequency objects, which have a
number of possible operations. The Coupling/Radio Re-transmission function is
one of them.
With this function the operator can couple two or more frequencies. This means
that voice received on one frequency is automatically (re-)transmitted on the other
coupled frequencies. By means of this feature, two pilots, operating on dierent
frequencies, are enabled to talk to each other using the VCS 3020S system. Coupling
can be switched on/o for each frequency at the operator position and all frequencies
coupled form a so-called coupling group. Consequently, several dierent coupling
groups may exist, one at each operator position.
The instance variable frq-couplings of the class SWITCH stores the coupling
groups as a nite mapping from operator positions to sets of frequencies. The
SWITCH has to guarantee, that a frequency can only be member of a single coupling
group and that not more than 15 frequencies are in one coupling group. This
property is expressed by means of a data-invariant denoted by the keyword inv.
instance variables
frq-couplings : OP-ID
m
(FRQ-ID-set);
inv frq-couplings

(s rng frq-couplings card s 15)
(card rng frq-couplings > 1

rng frq-couplings = );
init objectstate

frq-couplings : = ;
The following two methods are called from the OP if frequency coupling is
switched on or o for a frequency. The purpose of the method O-FrqCouplingStart
is to decide, whether frequency coupling is allowed for the given frequency or not.
If the invariant of frq-couplings would be violated, false is returned, otherwise the
frequency is added to the coupling group associated with the OP.
methods
O-FrqCouplingStart (op : OP-ID, frq : FRQ-ID) value B

if frq
rng frq-couplings
(op dom frq-couplings card frq-couplings (op) > 14)
then return false
else ( if op dom frq-couplings
then frq-couplings : = frq-couplings
op frq frq-couplings (op)
else frq-couplings : = frq-couplings op frq;
return true
)
pre op frqs (frq).tx
The method O-FrqCouplingEnd is the counterpart of the previous one. It is
used to switch o the coupling of a particular frequency. The frequency is either
removed from the coupling group, or the whole coupling group entry is removed, if
the frequency was the last one in the coupling group.
methods
O-FrqCouplingEnd (op : OP-ID, frq : FRQ-ID)

if card frq-couplings (op) = 1
then frq-couplings : = op frq-couplings
else frq-couplings : = frq-couplings op frq-couplings (op) frq
pre op dom frq-couplings frq frq-couplings (op)
The specication of the OP is not shown here. It mainly deals with details of the
MMI (Man Machine Interface) in order to call methods of the SWITCH. However,
the specication of the OP has been important, to be able to formulate system
test-cases at the appropriate level of abstraction.
Analysis of Identied Issues
During the formal specication of the VCS 3020S system a total number of 64
ambiguous, contradictory, unclear or erroneous issues have been found in the various
documents of the system. Figure 4.5 shows the statistics of them. The issues
categorized as Reading have been detected by active reading the documents, having
a clear vision of the modeling method to be carried out.
After about a month of practical experience with the system parts, the informal
documents have been read again and thereby more issues have been found, which
have been categorized as Reading with system knowledge. Next the conguration, i.e.
the static initial state of the system, and the functionality of the system have been
specied. The issues found thereby have been named Specication of conguration
and Specication of functionality, respectively. The reader should note that there
is an enormous dierence in the type of issues found by reading the documents
20
8
5
17
12
0 0
2
0
5
10
15
20
25
Reading Reading with system-
knowledge
Specification of
configuration
Specification of
functionality
I
s
s
u
e
s
Ambiguous, contradictiory or unclear issues
Erroneous issues
Figure 4.5: Statistics of the identied issues
in comparison to those found by writing the formal specication. A typical error
found by reading the documents would be a missing character in a table of valid
characters for a congurable name. Contrary to this, an issue found by specifying
the functionality of the system typically deals with undened behavior of the system
under certain conditions or/and after a specic sequence of user actions. It would
have been very dicult or even impossible to nd such issues only by reading or
reviewing the documents, even if this is carried out by a VCS 3020S system expert.
The identied issues have been resolved and the results have been integrated into
the informal documents, which is certainly helpful to improve the overall quality of
the system.
4.2.2 Test Case Validation
The formalization of the test cases is shown by means of an example. Figure 4.6
is a copy of the informal test case Frequency Coupling out of the Internal Test
Specication of FREQUENTIS.
It can be seen in Fig. 4.6 that a test case is divided into several steps. Each step
requires an action from the test-personnel. The expected reaction of the VCS 3020S
system, which is considered to be correct is specied in the reaction column for each
test-step. The test-personnel has to check, whether the reaction of the system is
identical to the specied behavior or not.
The formal specication of test cases follows the informal specication closely.
The actions required for every step of the informal specication are explicitly speci-
ed as method invocations. These methods have a prex E- to indicate the exter-
nal call mechanism. The informally specied reactions after the actions are formally
dened as calls to observation methods with the prex Test-. The VDM
++
spec-
ication in Figure 4.7 is the formalization of the informal test case presented in
Step
No.
Actions Reactions OK Remarks
1 Couple frequency 1 with
frequency 2 by pressing the
<CPL> key on both frequency
objects.
The frequencies are coupled
(the coupling symbol
appears for both frequencies).
All audio received on
frequency 1 is transmitted on
frequency 2 and vice versa.
2 Press PTT Voice is routed to frequency 1
and 2
3 Activate TX for frequency 3 Frequency 1 and 2 are still
coupled, so all audio received on
frequency 1 is transmitted on
frequency 2 and vice versa.
Attention: no voice is routed to
frequency 3
4 Press PTT Voice is routed to frequency 1, 2
and 3.
5 Stop the frequency coupling by
pressing the <CPL> key on
both frequency objects again.
The frequency coupling symbol
( ) disappears.
Figure 4.6: Informal test case for the Coupling/Radio Re-transmission feature.
Figure 4.6.
The action part of the rst step of the informal test case is represented by the
rst two calls of the method E-FrqChangeCoupling of the class op1 (which represents
an Operator Position). This call turns the frequencies with the ids 1 and 2 into
transmit mode and couples them. The following two calls to the method Test-
IsFrqCoupled check if it succeeded. This corresponds to the reaction part of the
informal test case. These test methods return a boolean value and the conjunction
of these values is the total result of the test case.
The test steps 24 have been formalized using the same pattern. The steps 2
and 4 require, that the PTT key has to be pushed (formalized with the E-PttPush
method). Note, that the PTT key also needs to be released after the reaction has
been tested (formalized with the E-PttRelease method). This is not mentioned in
the informal test case. Furthermore, the last three method invocations are not part
of the informal test case specication. These method invocations bring the system
back into its initial state. This is very important, since the next test case to be run
assumes that the system is in this initial state. When the real system is tested, it is
assumed that the the test personnel sets the system into an appropriate state before
a test is performed.
The example shows how the specication based test automation leads to a more
precise test case description. The results of a test case are not relevant for the test
coverage analysis, but further condence into the formal specication is gained by
checking them.
methods
TestCoupling () value B

( dcl result : B : = true;
op1!E-FrqChangeCoupling (mk-FRQ-ID (1)) ;
result : = result op1!Test-IsFrqCoupled (mk-FRQ-ID (1))
op1!Test-IsFrqCoupled (mk-FRQ-ID (2));
op1!E-PttPush () ;
result : = result rif 1!Test-IsPttActiveBy (mk-OP-ID (1))
rif 2!Test-IsPttActiveBy (mk-OP-ID (1));
op1!E-PttRelease () ;
op1!E-FrqChangeTx (mk-FRQ-ID (3)) ;
op1!E-PttPush () ;
result : = result rif 1!Test-IsPttActiveBy (mk-OP-ID (1))
rif 2!Test-IsPttActiveBy (mk-OP-ID (1))
rif 3!Test-IsPttActiveBy (mk-OP-ID (1));
op1!E-PttRelease () ;
result : = result op1!Test-IsFrqCoupled (mk-FRQ-ID (1))
op1!Test-IsFrqCoupled (mk-FRQ-ID (2));
op1!E-FrqChangeRx (mk-FRQ-ID (1)) ;
return result
)
Figure 4.7: Formalized test cases.
Coverage Analysis Results
By means of the IFAD VDM
++
Toolbox all test cases for the specied functionality
have been animated. Thereby, runtime information, i.e. which expressions/statements
have been evaluated, has been collected.
The overall result of this analysis is that approximately 80% of the formal spec-
ication is covered by the test cases, leaving some 20% of the specications func-
tionality untested! The result is of major importance since it points out that an
unacceptable large part of the radio functionality of the VCS 3020S system remains
untested in the system integration test phase. A more detailed summary is shown
in Table 4.1.
The IFAD VDM
++
Toolbox can also generate a colored specication, i.e. the
covered and not covered parts of the specication are pretty-printed with dierent
Class Name Number of Methods Coverage
OP 31 91%
RIF 11 70%
SWITCH 35 79%
average 83%
Table 4.1: Summary of the test coverage results
colors. The colored specication can be used to nd missing test cases. At present,
it is not possible to generate the missing test cases automatically, however, it is
possible to nd the missing test cases systematically. Based on this information
new test cases have been specied and a re-run of the test coverage analysis showed,
that 99% of the formal specication were covered by the combination of the old and
the new test cases, which is an acceptable coverage ratio.
4.2.3 Remarks on Experiment I
Issues
The whole model is about 2400 lines
5
of VDM
++
specication. About 1000 lines
have been used for the test cases and the Workspace, which was necessary to make
the specication executable.
During the development of the formal model, a total number of 64 defects have
been found in the informal system documents. These ambiguous, contradictory,
unclear or erroneous issues have been summarized in a catalogue of questions. All
of them have been resolved, supported by system experts.
A main result of this work was the realization that only 80% of the systems
radio functionality has been covered by the former existing test cases. Consequently,
additional test cases have been derived out of the formal specication. Furthermore,
the specication high-lighted how much more economic test cases could be designed,
in order to cover more system functionality in a single run.
Eorts
During the whole work of this project, the amount of time spent on dierent tasks
has been recorded. The summary in Fig. 4.8 shows the eorts, which have been
necessary for the dierent tasks. This information can be used for estimating the
costs of future projects.
Note, that the time-eorts have been recorded on a per hour basis. This means
that the values have not been estimated after an activity has been performed. The
eort has rather been recorded while the work has been performed.
5
The VDM
++
LOCs have been determined by removing all comments and counting only lines
with more than 2 characters
1,2
2,3
3,5
1,2
1,1
3,0
Read FRQ Documents
Read FM Literature
Formal Specification
Test Coverage Analysis
Modification
Meetings, Presentations
Working times are given in
40 hour units (approx. 1 week)
Figure 4.8: Time analysis for the whole work
As can be seen from the gure, the time eorts have been categorized. The time
spent on reading the documents of the VCS 3020S system (Read FRQ Documents)
was rather short because of an additional practical work with the existing VCS 3020S
system. About twice the time has been spent on reading literature about formal
methods (Read FM Literature). For the specication of the architecture and the
functionality of the system 3
1
2
weeks have been used. The amount of time spent
on the test coverage analysis also includes the time spent to nd the additional
test cases. The time used for the coverage analysis and for modications due to
requirement changes seems to be extremely short. About three weeks have been
spent to prepare and make several presentations and for numerous meetings.
Experience with VDM
++
The specication was done using the VDM
++
language denition of the IFAD
VDM
++
Toolbox version 6.0. Besides some minor errors which have been discovered
in the Toolbox, the main drawbacks, that have been discovered, were found on the
VDM
++
language itself, three of them are worth being pointed out:
Type denitions inside a class are not public. Thus, types dened in one class
cannot be used in another. The consequence is that an articial inheritance
hierarchy has to be created in order to make types available to more than one
class.
Method calls are not allowed in expressions.
Member variables are private and can therefore only be accessed via methods.
This property combined with the previous one can be considered as the worst
drawback of VDM
++
, since it is not possible to make implicit specications
without either calling methods from within expressions in order to retrieve
values from other objects member variables, nor is it possible to get access to
these member variables directly.
4.3. EXPERIMENT II 55
The consequence of these limitations is, that it is in most cases impossible to give
implicit specications, and so we felt that the specication became much too explicit.
It is worth mentioning, that IFAD has also identied these drawbacks independently
and a newer version of the Toolbox with a changed language denition is already
available.
The success and acceptance of our method depends mainly on tool support.
Consequently, the development of tools to support the generation of test cases can
be identied as a central issue for future work. One experience is that professional
testers are seriously interested in the presented techniques, since they try to improve
their testing techniques continuously.
Concerning VDM
++
, the new language version has to be evaluated. Future
case studies will show, if the reported weaknesses have been solved. For the next
experiment, the author has taken the decision that conventional VDM-SL is sucient
for our purposes.
However, it should be pointed out that the encouraging results convinced FRE-
QUENTIS to take the second step: A new project in co-operation with our institute
has been carried out. This time, VDM-SL is used in parallel to an ongoing project,
in which a safety-critical system has to be developed from scratch. The results are
reported in the next section.
4.3 Experiment II
The specication work in this project has been carried out by Andreas Gerstinger, a
student of the author. Due to our experiences with VDM
++
in this project VDM-SL
has been used as the specication language. This decision proved to be right since
a higher level of abstraction of the VDM prototype could be observed.
4.3.1 Requirements Validation
The requirements are contained in a regularly updated document of almost 3000 re-
quirements formulated in natural language. The task of the formal methods engineer
was to create an executable model of a part of the systemspecically a network
nodewhich plays a crucial role. There was also a rst version of the architectural
design document available which was helpful for gaining a general overview of the
network node, however, the formal specication is solely based on the requirements.
The formal methods engineer had no domain knowledge in air trac control.
Therefore, he was introduced to the domain and the system to be developed in a
1
1
2
day training session held by a system engineer. It should be mentioned that
the formal methods engineer was not directly involved in the main projectthe
formalization was done in parallel.
The rst step was an intense study of the requirements and architectural design
document. The formal methods engineer was not given a list of requirements to be
formalized, he was rather instructed to formalize certain functionalities. Although
the requirements document did contain some structure, one of the most challenging
tasks was the selection of the relevant requirements.
The next step was the actual formalization of the selected requirements. The for-
mal model was constructed in two stages: rst a model of the systems state, using
types constrained by data-invariants, then the functional model containing the op-
erations. This formalization process yielded the most valuable output. Formalizing
requirements requires understanding them rigorously and in an unambiguous way.
Since natural language requirements are doomed to be ambiguous, several questions
have been raised during the formalization process. These questions have been incor-
porated into a catalog of questions which was regularly exchanged between the
formal methods engineer and the requirements manager. During the whole project,
108 questions have been raised, with 33 of them resulting in changes in the require-
ments document. These questions uncovered inconsistencies, ambiguities, errors,
omissions, redundancies and inaccuracies in the requirements documentwhich are
unavoidable in textual requirements documents of this size. This led to an improve-
ment in the quality of the requirements.
Studying the documents and selecting the relevant requirements took 5.5 weeks,
the actual formalization process took another 6.5 weeks for 140 requirements. The
size of the formal specication was 60 pages including comments.
This phase could have been completed in considerably shorter time if it would
have been performed by a domain expert. We believe however, that in this phase,
the lack of domain knowledge is not a disadvantagein contrastit might be advan-
tageous because of the type of unbiased questions which arise. This phase resembles
a validation of the requirements.
4.3.2 Examples of Requirement Issues
The following examples demonstrate the kind of issues being raised during the for-
malization process. Relatively simple examples are chosen to keep the explanations
at a minimum.
Layout of a Touch Panel
The example is about the key layout on the touch panels of the OPs. If an OP
handles the functionality of a specic role, we say the OP has the role assigned.
Roles are grouped into function groups. A function group contains roles which have
similar functionality. A VCS may have a special OP with additional functionality
which is called Test-OP. The original requirements for displaying the correct layout
of a touch panel are summarized in Figure 4.9.
If these requirements in plain text are read quickly, they seem to be quite rea-
sonable. However, if they are read more thoroughly they give rise to questions. If
they are read with the process of formalization in mind, they give rise to even more
ID Text
#0025 A function group contains all roles which have about the
same function and therefore have the same key layout.
There are up to 9 function groups 1..9. Each role belongs
to exactly one function group. The function group 0
contains the key layout for OPs which own roles from
more than one function group.
#0042 If an OP has roles from more than one function group
assigned, a default layout shall be displayed.
#0062 If all assigned roles belong to the same function group,
the key layout corresponding to the function group shall
be displayed.
#1047 The key layout of the Test-OP must always be the layout
of the function group 0.
Figure 4.9: Requirements for the touch panel layout on OPs
questions. And once the actual formalization takes place, even more issues may be
raised. This was the process by which the whole project was conducted: some issues
were detected just by reading and having the formalization in mind, but some issues
require the actual formalization to be raised. Since the goal of the formalization
was an executable specicationa high level prototypethe focus was also on im-
plementation. Issues were raised which might otherwise have been detected much
later in the project, namely in the implementation phase.
The most obvious inconsistency is the following. Requirement #0025 is inconsis-
tent in itself: First, it is stated that there are 9 function groups which are numbered
from 1 to 9. The same requirement, however, mentions function group 0! This is
an easy-to-detect inconsistency, which probably does not require formalization to be
uncovered.
However, if we read on, it will be seen that requirement #0042 talks about a
certain default layout. If we dont have the formalization in mind, we might just
think about this as a certain layout. However, if we view the default layoutin
connection with the other requirements, the question is raised if the layout of the
function group 0and the default layoutare equivalent. This is in fact true, so
there is an inconsistent terminology in the requirements.
An unambiguous specication can be given using VDM type denitions:
Functiongroup = N
inv f

f 1, . . . , 9;
Layout = Functiongroup [ DEFAULT;
A data-invariant and a union type model the fact that there are 9 function groups
numbered 1 to 9, but ten layouts: one for each function group plus an additional
default layout.
During the formal specication of the actual layout calculation another issue was
raised. So far we know:
if an OP has roles from only one function group, the layout of the function
group shall be displayed.
if an OP has roles from more than one function group, the default layout shall
be displayed.
the Test-OP shall always display the default layout.
Below, the formal denition is shown:
CalcLayout : APLID VCSID
o
Layout
CalcLayout (opid, vcsid)

( if is-TestOP (vcss, opid)
then return DEFAULT
else let assigned-roles = get-roles-of -OP (rv, opid),
functiongroups = get-fg (roles, vcsid, rid) [ rid assigned-roles in
cases functiongroups:
return DEFAULT,
f return f ,
others return DEFAULT
end
)
The if-clause is used to display the correct layout on the Test-OP. Then, the set
of assigned function groups is constructed from the set of assigned roles with the
help of a set comprehension expression. In the following cases statement, a layout
is returned depending on the set of function groups. A set can have one, more, or
no elements. The case for one or more function groups (i.e. the second and third
branch of the cases statement) is covered in the requirements. However, what if an
OP has no roles (and therefore no function groups) assigned? This case (i.e. the
rst branch of the cases statement) is not covered in the requirements!
This is not necessarily an error in the requirements, since this case may have
been deliberately omitted, and left to the discretion of the engineers responsible
for the following phases. In this case, it could have been left open in the formal
specication as well. However, it was not meant to be left open. There should be a
requirement saying:
If an OP has no roles assigned, a default layout shall be displayed.
ID Text
#1735 If several roles come into question, then an InterCOM
connection will be prioritized. If no InterCOM role is
available, the lowest role will be used.
Figure 4.10: Part of the rules to determine the source of a call
Order of Roles
This is an example of an ambiguity in the requirements. A controller can initiate calls
to other controllers by using the devices on his OP. On the receiving controllers OP,
the call is signaled, together with the source information of who initiated the call, i.e.
a caller can be identied before the call is answered. The diculty with this concept
is the following: since OPs can be assigned functionalities (called roles) dynamically,
the source information depends on the assigned roles of the initiating OP. Therefore,
there have to be some rules of how the source information is determined. The
requirement in gure 4.10 is part of these rules.
It is not important to understand the requirement. However, when formalizing
this requirement, there willat some pointbe the need to determine the lowest
role from a set of roles.
In order to formalize this requirement correctly, there must be a total order
dened on the roles. However, this is not the casethe corresponding requirement
is missing. It is interesting to note, that it was not even clear among the system
specialists which order is meant, since roles can be ordered by dierent attributes.
Even test cases were generated, based on a wrong assumption of an engineer. The
requirement has been claried by exactly stating the desired order of the roles.
4.3.3 Test-Case Validation
For parts of the functionality covered by the formal specication, conventionally
designed test-cases already existed. These test-cases were formulated in a very
clearwe call it semi-formalway. Each test case is broken down to atomic actions
and the results of each action are specied in great detail.
The task of the formal methods engineer was to formalize these test-cases and
animate them on the formal specication using the interpreter tool. There were 65
test-cases consisting of approximately 200 single steps of which 60% were formal-
ized. Due to the detailed specication of the test cases, their formalization was a
straightforward task.
By comparing the expected results specied in the original test-cases with the
results returned by the formal specicationwhich acted as a prototypea test-
case validation has taken place. 16 errors within the original test-cases have been
detected and could therefore be corrected. Furthermore, the animation of test cases
also led to a validation of the formal model. Seven minor errors were found in the
formal specication and consequently corrected. Another output of this stage was
the calculation of the test coverage of the formalized test cases with the help of the
test coverage metrics built-in in IFADs VDMTools.
It was realized that this coverage metrics, which is basically expression coverage,
is rather weak and that complete coverage was quickly attained. This is also due to
the fact that the conventional test-cases were designed in a very thorough manner:
for each requirement one or more test-cases have been specied. The time eort for
the formalization and animation of the test-cases was 2 weeks.
4.3.4 Remarks on Experiment II
Summing up, the formalization of 140 requirements uncovered 108 open issues lead-
ing to 33 changes in the requirements document, and 16 errors in the acceptance
test-cases have been detected. This was done in 14 man weeks. Considering the
fact that requirements engineering is one of the most dicult parts in a project this
eort seems to be acceptable. Especially, in a safety-critical domain like air trac
control, where voice communication must not fail. The customer Austro-Control
appreciated the increased rate of early questions for clarication. The requirement
engineer Robert Aster said:
In the mentioned project, the application of VDM took place at a rela-
tively late stage. At this time, each member of the project team already
possessed a high level of insider-knowledge. Considering this fact, it was
even more astonishing, that the VDM-engineer quickly became familiar
with the complex subject. Subsequently, the application of VDM quickly
led to qualied questions. Consequently, several inconsistencies in the
requirements were uncovered and corrected prior to the implementation
phase. Apart from this, it was realized that several requirements were
well known to the members of the project team without being actually
written down.
This emphasizes the important catalyst role the formalization plays in validation.
However, the experience indicates that this method could be much more eective if
it would be more tightly integrated into the conventional development process.
In this chapter the results of two experiments of applying light-weight formal meth-
ods to testing, have been presented. The experiments have been carried out in an
industrial environment in the highly safety-critical domain of voice-communication
for air-trac control. Both experiments followed the method as described in Chap-
ter 3. Examples of parts of the abstract prototype specication served to illustrate
the formalisms and their role in validation.
4.4. SUMMARY AND DISCUSSION 61
Experiment I detected 64 problems in the requirement documentation of the
existing voice communication system. Furthermore, an unacceptable low coverage
rate of 80 % has been analyzed by testing the abstract prototype with the existing
system-level test-cases. As a consequence, additional test-cases have been derived
from the formal specication. The work has been done in less than 13 man-weeks.
Experiment II detected 33 problems in the requirements documentation of the
new voice communication network under development. This time the existing test-
cases covered 100 % of the abstract prototype, but the 65 test-cases with 200 test-
steps contained 16 faults. Since almost all detected faults occurred in dierent test-
cases, this means that approximately 25 % of the test-cases have been erroneous.
The eort took 14 man-weeks.
These results indicate the ecient applicability of our light-weight approach in
practice. The high number of faults in the requirements reect the fact that require-
ments engineering is a dicult task. It has been shown that formal specications
may help to raise the quality of both, the requirements and the system-level test
cases. That supervised students could do this work underpins our opinion that the
method could be adopted by practicing engineers. Furthermore, the author thinks
that the eorts are justiable, especially for such safety-critical applications.
However, the experiments also showed that more can be achieved. We learned
that executable specications have the disadvantage that non-experienced users tend
to think too operationally. Consequently, the prototypes lack abstraction and in-
corporate too many design details. However, the engineers liked the availability of
test-cases to validate our formal specications.
The solution is a test-case generation method that derives the complex test-
cases from a general non-executable formal specication. These test-cases have to
be correct by denition. A premise for using test-cases for validating a specication
is that they are correct with respect to the specication.
Therefore, a new test-case synthesis method has been developed by the author
and is presented in the following. The general correctness relation for all kinds of
test-cases is abstraction, as is explained in the following chapter.
Chapter 5
Test-Cases are Abstractions!
In this chapter, the foundation for the new test-case synthesis approach is presented.
It is shown that test-cases are abstractions of model-based formal specications. The
framework of the renement calculus is used for formally dening and proving this
abstraction relation for simple input-output test cases, non-deterministic test cases,
partition test cases and test scenarios for interactive systems. A short introduction
to the renement calculus is included.
5.1 The New Idea
5.1.1 Motivations
In the concluding discussion of the previous chapter, the motivation for our new
test-synthesis approach has been given: Test-cases must be correct with respect to
the formal specication. Here, by the correctness of a test-case it is meant that
the functionality described by a test-case (for a given input) must be consistent
with the functionality described by the formal specication. The reason is that the
test-cases should serve as a tool for validating an abstract possibly non-executable
formal requirement specication. Test-cases can be easily understood by a customer
or engineer who is not familiar with a formal specication language.
A further motivation for this work is the need of a general formal framework
for certifying test-case generation tools. Simon Burton has a similar motivation in
his work on Z that has been recently published [Bur00]. However, our framework is
more general as will be seen in the following.
5.1.2 Test-Design as a Formal Synthesis Problem
As can be seen in the previous chapter, test-cases are complex algorithms that have
to be executed either by a tester manually, or by test drivers.
The goal of our framework, is the derivation of such test-cases T from a formal
specication S. A test-case T should be correct with respect to S, and a program
63
64 CHAPTER 5. TEST-CASES ARE ABSTRACTIONS!
P that implements S should be tested with T. Thus, the derivation of T from S
constitutes a formal synthesis problem. In order to formalize a certication criteria
for this synthesis process, the relation between T, S and P must be claried.
It is well-known that the relation between a specication S and its correct im-
plementation P is called renement. We write
S _ P
for expressing that P is a correct renement (implementation) of S. The problem
of deriving an unknown P? from a given S is generally known as program synthesis:
S _ P?
In this chapter, it is shown that correct test-cases T are abstractions of the
specications S, or:
T _ S _ P
Consequently, test-case synthesis is a reverse renement problem. The reverse
renement from a given S into an unknown test-case T? is here called abstraction,
and denoted as:
S _ T?
Hence, formal synthesis techniques can be applied for deriving test-cases from
a formal specication. This is the most important idea in this thesis. Its main
contribution is to use the theory of Back and von Wrights renement calculus
[BvW98] for formulating abstraction rules for deriving correct test-cases.
The renement calculus is a theory for reasoning about the correctness and the
renement of programs. In the following it is shown how its program synthesis
techniques can be applied to test-case synthesis.
5.2 The Renement Calculus
5.2.1 Contracts
The prerequisite for testing is some form of contract between the user and the
provider of a system that species what it is supposed to do. In case of system-
level testing usually user and software requirement documents dene the contract.
Formal methods propose mathematics to dene such a contract unambiguously and
soundly. In the following two chapters the formal contract language of Back and von
Wright [BvW98] is used. It is a generalization of the conventional pre- and post-
condition style of formal specications known in VDM, B and Z. The foundation of
this renement calculus is based on lattice-theory and higher-order logic (HOL).
5.2. THE REFINEMENT CALCULUS 65
A system is modeled by a global state space . A single state x in this state
space is denoted by x : . Functionality is either expressed by functional state
transformers f or relational updates R. A state transformer is a function f :
mapping a state space to the same or another state space .
A relational update R : Bool species a state change by relating the
state before with the state after execution. In HOL, relations are modeled by func-
tions mapping the states to Boolean valued predicates. For convenience, a relational
assignment (x: = x
[ b) is available and generalizes assignment statements. It sets

a state variable x to a new state x
such that b, relating x and x
, holds.
The language further distinguishes between the responsibilities of communicating
agents in a contract. Here, the contract models the viewpoint of one agent called
the angel who interacts with the rest of the system called the demon. In our work
following [BvW98, BMvW99], the user is considered the angel and the system under
test the demon. Relational contract statements denoted by R express relational
updates under control of the angel (user). Relational updates of the demon are
denoted by [R] and express updates that are non-deterministic from the angels
point of view. Usually, we take the viewpoint of the angel.
The contract statement f ) denotes a functional update of the state determined
by a state transformer f . There is no choice involved here, neither for the angel nor
the demon agent, since there is only one possible next state for a given state.
Two contracts can be combined by sequential composition C
1
; C
2
or choice op-
erators. The angelic choice C
1
. C
2
and the demonic choice C
1
C
2
dene non-
deterministic choice of the angel or demon between two contracts C
1
and C
2
. Fur-
thermore, predicate assertions p and assumptions [p] dene conditions the an-
gel, respectively the demon, must satisfy. In this language of contract statements
p; f ) denotes partial functions and p; [R] pre-postcondition specications. Fur-
thermore, recursive contracts, using the least x-point operator and the greatest
x-point operator , are possible for expressing several patterns of iteration.
The core contract language used in this work can be summarized by the following
BNF grammar, where p is a predicate and R a relation.
C: = p [ [p] [ R [ [R] [ C; C [ C . C [ C C [ X C
To simplify matters, we will extend this core language by our own contract
statements. However, all new statements will be dened by means of the above
core language. Thus, our language extensions are conservative. This means that no
inconsistencies into the theory of the renement calculus are introduced by our new
denitions.
5.2.2 Example Contracts
A few simple examples should illustrate the contract language. The following con-
tract is a pre- postcondition specication of a square root algorithm:
x 0 e > 0; [x: = x
[ -e x x
2
e]
The precondition is an assertion about an input variable x and a precision e. A
relational assignment expresses the demonic update of the variable x to its new value
x
. Thus, the contract is breached unless x 0 e > 0 holds in the state initially.
If this condition is true, then x is assigned some value x
for which -e x-x
2
e
holds.
Consider the following version of the square root contract that uses both kinds
of non-determinism:
x, e : = x
, e
[ x
0 e
> 0; [x: = x
[ -e x x
2
e]
In this contract the interaction of two agents is specied explicitly. This contract
requires that our agent, called the angel, rst chooses new values for x and e. Then
the other agent, the demon, is given the task of computing the square-root in the
variable x.
The following example should demonstrate that programming constructs can be
dened by means of the basic contract statements. A conditional statement can be
dened by an angelic choice as follows:
if P then S
1
else S
2
P; S
1
. P; S
2
Thus, the angel agent can choose between two alternatives. The agent will,
however, always choose only one of these, the one for which the assertion is true,
because choosing the alternative where the guard is false would breach the contract.
Hence, the agent does not have a real choice if he wants to satisfy the contract.
Alternatively, we could also dene the conditional in terms of choices made by
the other agent (demon) as follows:
if P then S
1
else S
2
[P]; S
1
[P]; S
2
These two denitions are equivalent. The choice of the demon agent is not
controllable by our agent, so to achieve some desired condition, our agent has to
be prepared for both alternatives. If the demon agent is to carry out the contract
without violating our agents assumptions, it has to choose the rst alternative when
P is true and the second alternative when P is false.
Iteration can be specied by recursive contracts (X C). Here X is a variable
that ranges over contract statements, while (X C) is the contract statement C,
where each occurrence of X in C is interpreted as a recursive invocation of the
contract C. For example, the standard while loop is dened as follows:
while g do S od (X if g then S; X else skip )
We write skip = id) for the action that applies the identity function to the
present state.
5.2. THE REFINEMENT CALCULUS 67
5.2.3 Semantics
The semantics of the contract statements is dened by weakest precondition pred-
icate transformers. A predicate transformer C : ( Bool) ( Bool) is a
function mapping postcondition predicates to precondition predicates. The set of
all predicate transformers from to is denoted by ( Bool)
( Bool).
The dierent roles of the angel and the demon are reected in the following
weakest-precondition semantics. Here q denotes a postcondition predicate and
a particular state, p is an arbitrary predicate, and R a relation. Following the
convention, we identify contract statements with predicate transformers that they
determine. The notation f .x is used for function application instead of the more
common form f (x).
p.q p q (assertion)
[p].q p q (assumption)
R.q. ( R.. q.) (angelic update)
[R].q. ( R.. q.) (demonic update)
(C
1
; C
2
).q C
1
.(C
2
.q) (sequential composition)
(C
1
. C
2
).q C
1
.q C
2
.q (angelic choice)
(C
1
C
2
).q C
1
.q C
2
.q (demonic choice)
In this semantics, the breaching of a contract by our angel agent, means that the
weakest-precondition is false. If a demon agent breaches a contract, the weakest-
precondition is trivially true. The semantics of the specication constructs above
can be interpreted as follows:
The weakest precondition semantics of an assertion contract reects the fact
that, if the nal state of the contract should satisfy the post-condition q,
then in addition the assertion predicate p must hold. It can be seen that the
global state is not changed by an assertion statement. Consequently, the angel
breaches this contract if p q evaluates to false.
The semantics of an assumption shows that the demon is responsible for satis-
fying an assumption predicate p. If the assumption does not hold, the demon
breaches the contract and the angel is released from the contract. In this case,
the weakest-precondition trivially evaluates to true.
The angelic update denition says that a nal state must exist in the rela-
tion R, such that the postcondition q holds. The existential quantier in the
weakest-precondition shows that the angel has control of this update. The an-
gel can satisfy the contract, as long as one update exists that satises the post-
condition. In the set notation this update is dened as R.q. R.q ,= .
This is in contrast to the denition of the demonic update. Here, all possible
nal states have to satisfy the postcondition. The reason is that the demonic
update is out of our control. It is not known, to which of the possible states,
described by the relation R, the state variables will be set. In the set notation
this update is dened as [R].q. R. q.
The weakest-precondition of two sequentially combined contracts is dened by
the composition of the two weakest-preconditions.
The angelic choice denition shows that the weakest-precondition is the union
of the weakest-precondition of the two contracts. Thus, a further choice of the
angel, further weakens the weakest-preconditions.
The demonic choice is dened as the intersection of the weakest-preconditions
of the two contracts. Thus, demonic choice means a strengthening of the
weakest-preconditions.
For further details of the predicate transformer semantics, we refer to [BvW98].
5.2.4 Renement and Abstraction
The notion of contracts includes specication statements as well as programming
statements. More complicated specication statements as well as programming
statements can be dened by the basic contract statements presented above. The
renement calculus provides a synthesis method for rening specication statements
into programming statements that can be executed by the target system. The re-
nement rules of the calculus ensure by construction that a program is correct with
respect to its specication.
Formally, renement of a contract C by C
, written C _ C
, is dened by the
pointwise extension of the subset ordering on predicates: For being the after state
space of the contracts, we have
C _ C
q ( Bool) C.q C
.q
This ordering relation denes a lattice of predicate transformers (contracts) with
the lattice operators meet and join .. The top element is magic.q true, a
statement that is not implementable since it can magically establish every postcon-
dition. The bottom element of the lattice is abort.q false dening the notion
of abortion. The choice operators and negation of contracts are dened by point-
wise extension of the corresponding operations on predicates. A large collection of
renement rules can be found in [BvW98, Mor90].
Abstraction is dual to renement. If C _ C
, we can interchangeable say C is

an abstraction of C
. In order to emphasize rather the search for abstractions than

5.3. TEST-CASES AS ABSTRACTIONS 69
for renements, we write C _ C
to express C
is an abstraction of C. Trivially,
abstraction can be dened as
C _ C
_ C
Hence, abstraction is dened as the reverse of renement.
5.3 Test-Cases as Abstractions
In the following we will demonstrate that test-cases common in software engineering
are in fact contracts highly abstract contracts. To keep our discussion simple,
we do not consider parameterized procedures, but only global state manipulations.
In [BvW98] it is shown how procedures can be dened in the contract language.
Consequently, our approach scales up to procedure calls.
5.3.1 Input-Output Tests
The simplest form of test-cases are pairs of input i and output o data. We can
dene such an input-output test-case TC as a contract between the user and the
unit under test:
TC i o x = i ; [y: = y
[ y
= o]
Intuitively, the contract states that if the user provides input i , the state will be
updated such that it equals o. Here, x is the input variable and y the output
variable.
In fact, such a TC is a formal pre-postcondition specication solely dened for a
single input i . This demonstrates that a collection of n input-output test-cases TCs
are indeed pointwise dened formal specications:
TCs TC i
1
o
1
. . . . . TC i
n
o
n
Moreover, such test-cases are abstractions of general specications, if the specica-
tion is deterministic for the input-value of the test-case, as the following theorem
shows.
Theorem 5.1 Let p : Bool be a predicate, Q : Bool a relation on
states, and TC i o a test-case with input i in variable x and output o in variable
y. Then
p; [Q] _ TC i o (x = i ) p ([ x = i [; Q) [ y: = o [
, where
[ p [ denotes the coercion of a predicate p : Bool to a relation (here x = i ).
The relation [ p [ : Bool is dened as follows:
[ p [ .. = andp.
[ f [ denotes the coercion of a state transformer f : to a relation (here
y: = o). The relation [ f [ : Bool is dened as follows:
[ f [ .. f . =
the composition operator ; is overloaded for relations. The relation composition
P; Q is dened as follows:
(P; Q).. ( P.. Q..)
Proof.
p; [Q] _ TC i o
by denitions
r p. Q. r (x = i ). [y: = y
[ y
= o].r
by denition of demonic relational assignment
r p. Q. r (x = i ). ( y
(y
= o) r[y: = y
])
by simplication of update
r p. Q. r (x = i ). r[y: = o]
by denition of substitution r: = (y: = y
[ y
= o).
p. Q. (y: = y
[ y
= o). (x = i ).
distributivity, subset denition
( (x = i ). p.)
(
(x = i ). Q..
(y: = y
[ y
= o)..
)
denitions
(x = i ) p [ x = i [; Q [ y: = o [
Theorem 1 shows that only for deterministic specications, simple input-output

test-cases are sucient, in general. The theorem becomes simpler if the whole input
and output is observable.
Corollary 5.1 Let p : Bool be a predicate, Q : Bool a relation on
states, and TC i o a test-case, where the whole change of state is observable. Thus,
input i : and output o : . Then
p; [Q] _ TC i o p.i Q.i .o
Proof. The corollary follows from Theorem 1 and the assumption that i : and
o : .
The fact that test-cases are indeed formal specications and as Theorem 1 shows
abstractions of more general contracts explains why test-cases are so popular. They
are abstract, and thus easy to understand. Furthermore, they are formal and thus
unambiguous.
Furthermore, the selection of certain test-cases out of a collection of test-cases
can be considered as abstraction:
5.3. TEST-CASES AS ABSTRACTIONS 71
Corollary 5.2
TC i
1
o
1
. . . . . TC i
n
o
n
_ TC i
k
o
k
for all k, 1 k n.
Proof. The theorem is valid by denition of the join operator a .b _ a or a .b _ b,
respectively.
5.3.2 Non-Deterministic Test-Cases

In general, a contract can permit more than one result. In this case, testing the
requirements with simple input-output values is insucient. An output predicate
: Bool can be used for describing the set of possible outputs. We dene such
a test-case as follows:
TCp i x = i ; [y: = y
[ ]
For being a correct test-case with respect to a contract this type of test-case
should be an abstraction of the contract.
Theorem 5.2 Let p : Bool be a predicate, Q : Bool a relation on
states, and TC2 i o a test-case with input i in variable x and output in variable y
such that the output predicate holds . Then we have:
p; [Q] _ TCp i (x = i ) p ([ x = i [; Q) [ [
The theorem shows that a test-case for non-deterministic results can be calcu-
lated by strengthening the precondition to a single input value and weakening the
postcondition to the outputs of interest. The fact that the output predicate might
be weaker than Q represents the case that not all properties of an output might be
observed. This can be useful if not all variables or only selected properties of the
output should be checked.
5.3.3 Partition Tests
Partition analysis of a system is a powerful testing technique for reducing the pos-
sible test-cases: Here, a contract is analyzed and the input domains are split into
partitions. A partition is an equivalence class of test-inputs for which the tester
assumes that the system will behave the same. These assumptions can be based
on a case analysis of a contract, or on the experience that certain input values are
fault-prone.
In case of formal specications, the transformation into a disjunctive normal
form (DNF) is a popular partition technique as already mentioned in the discussion
of the related work in Section 1. This technique is based on rewriting according the
rule A B (A B) (A B) (A B).
A partitioning of a contract statement p; [R] is a collection of n disjoint par-
titions p
i
; [R
i
], such that
p; [R] = p
1
; [R
1
] . . . . . p
n
; [R
n
]
and
i , j 1, . . . , n i ,= j p
i
p
j
=
These partitions describe classes of test-cases, here called partition test-cases.
Often in the literature, if the context is clear, a partition test-case is simply called
a test-case.
Partition test-cases are abstractions of specications, too:
Theorem 5.3 Let p
i
; [R
i
] be a partition of a specication p; [R]. Then
p; [R] _ p
i
; [R
i
]
Proof. The result follows directly from the denition of partitioning above, and the
denition of ..
Above, only the commonly used pre-postcondition contracts have been consid-
ered. They are a normal form for all contracts not involving angelic actions. This
means that arbitrary contracts excluding . and R can be formulated in a pre-
postcondition style. (see Theorem 26.4 in [BvW98]). However, our result that
test-cases are abstractions holds for general contract statements involving user inter-
action. In order to justify this, user-interaction has to be discussed with respect to
testing. The next section will introduce the necessary concepts.
5.4 Testing Interactive Systems
The synthesis of black-box tests for an interactive system has to consider the pos-
sible user actions. Furthermore, simple input-output test-cases are insucient for
practical systems. Moreover, sequences of interactions, called scenarios, are nec-
essary for setting the system under test into the interesting states. Consequently,
scenarios of the systems use have to be developed for testing.
Scenarios are gaining more and more popularity in software engineering. The
reasons are the same as for other test-cases: Scenarios are abstractions of an inter-
active system. For a comprehensive introduction into the dierent roles of scenarios
in software engineering see [JKS98]. In this work, the focus is on validation and
verication.
5.4. TESTING INTERACTIVE SYSTEMS 73
5.4.1 User Interaction
Testing interactive systems, typically involves the selection of a series of parameters.
Some of these parameters can be entered directly, some have to be set up, by initiat-
ing a sequence of preceding actions. Adequate test-cases should distinguish between
these two possibilities of parameter setup. Therefore, simple pre-postcondition con-
tracts are not sucient to specify test-cases. Moreover, the testers interaction with
the system has to be modeled.
We dene an atomic interaction IA of a tester, as a composition of the testers
system update T and the following systems response Q.
IA T; [Q]
The fact that we dene an atomic interaction by means of angelic and demonic
updates does not exclude other contract statements for modeling interaction. Theo-
rem 13.10 in [BvW98] states that T;[Q] is a normal form, thus arbitrary contract
statements can be dened by means of interactions.
In this context a simple input-output test-case TCI i o includes the actual setting
of the input variable to i .
TCI i o x: = x
[ x
= i ; [y: = y
[ y
= o]
Again the abstraction relation holds for this kind of test-cases.
Theorem 5.4 Let T : Bool and Q : Bool relations on states,
and TCI i o a test-case with input i in variable x and output o in variable y. Then
T; [Q] _ TCI i o [ x: = i [ T Q [ y: = o [
Proof.
The theorem holds by homomorphism and monotonicity properties. For ab-
stracting an interaction, demonic updates may be weakened and angelic updates
strengthened.
The proof is similar to that of Theorem 5.1.

5.4.2 Iterative Choice
The application of an iterative choice statement for specifying and rening interac-
tive systems have been extensively discussed in [BMvW99]. This statement, intro-
duced in [BvW98], is dened as a recursive selection of possible interactions S.
do
n
i
g
i
: : S
i
od (X g
1
; S
1
; X . . . . . g
n
; S
n
; X . skip)
The skip statement, models the users choice of stopping the dialog with the system.
denotes the least x-point operator. In general, a recursive contract X S is
interpreted as the contract statement S, but with each occurrence of statement
variable X in S treated as a recursive invocation of the whole contract.
The iterative choice statement follows a common iteration pattern, called angelic
iteration. This iteration construct over S is dened as the following xpoint:
S
(X S; X . skip)
Therefore, we have
do
n
i
g
i
: : S
i
od = (g
1
; S
1
. . . . . g
n
; S
n
)
Iterative choice should not be mixed with guarded command iterations used by
Dijkstra [Dij76]. Guarded command iterations are strong iterations dened by S
(X S; X skip) with, in contrast to angelic iteration, the termination out of a

users control.
In [BMvW99] renement rules for iterative choice are given. However, for testing
we need abstraction rules for the synthesis of test-cases scenarios are our goal.
5.4.3 Scenarios
An arbitrary scenario SC of an interactive system with n possible interactions S
i
and of length l is a sequence of l sequential user interactions S
i
. We write a sequence
comprehension expression
S
i
(k) [ (1 i n) (1 k l ))
to denote such arbitrary sequences, where k is the position in the sequence. It
should be mentioned that this sequence comprehension expression is not a valid
predicate transformer, but rather serves as a scheme for sequences of predicate
transformers. We use sequence comprehensions as a convenient notation, but they
cannot be dened in higher-order logic.
Scenarios are abstractions of interactive systems, modeled by iterative choice, as
the following theorem shows.
Theorem 5.5
do
n
i
g
i
: : S
i
od _ (g
i
; S
i
)(k) [ (1 i n) (1 k l ))
Proof. The theorem is valid by denition of the angelic iteration statement and thus
by denition of iterative choice:
do
n
i
g
i
: : S
i
od
skip . g
1
; S
1
. g
2
; S
2
. g
1
; S
1
; g
1
; S
1
. g
1
S
1
; g
2
S
2
. . . .
Hence, by denition of . any choice of sequences of g
i
S
i
is an abstraction.

However, for test-case generation, we are only interested in valid scenarios. A
scenario is considered a test-scenario if it terminates for every possible initial state.
Thus its weakest precondition should be true for an unspecied nal state:
S
i
(k) [ (1 i n) (1 k l ).true = true
Consequently, the abstraction should not equal the abort statement. Since abort
is the bottom element of the predicate transformer lattice, it is the trivial ab-
straction of every statement. Therefore, we dene a notion of testing abstraction
_
T
S _
T
T S _ T abort
and get the abstraction rule for testing scenarios:
Theorem 5.6 Let g(k) denote the guard at the kth position in a scenario and as-
sume that the system specication is consistent. Hence we assume that for all inter-
actions S
i
.true g
i
. Furthermore, g(l + 1) ,= false should be an arbitrary predicate
called the goal.
do
n
i
g
i
: : S
i
od _
T
(g
i
; S
i
)(k) [ (1 i n) (1 k l ) g
i
(k) S
i
(k).g(k + 1) g(1) = true)
Proof. Abstraction follows from Theorem 5.5. Termination is valid by induction:
The weakest precondition of the rst interaction is true, due to the assumption
that for all interactions S
i
.true g
i
and g(1) chosen to be true. Consequently S
i
(1)
terminates. An interaction S
i
(k +1) terminates due to the fact that its pre-condition
g(k + 1) can be reached by denition.
This abstraction rule denes the calculation of valid test scenarios. The goal
predicate is a condition that denes the states that should be reached by a sequence
of interactions. Trivially, it can be chosen to be true. For developing a scenario for
setting a system to a certain state, this goal predicate represents the corresponding
state description.
The theorem above shows that the question if a scenario terminates, can be
reduced to the question if two following interactions are composeable. From this
observation a new testing strategy will be derived in the next section.
In this chapter, the theory of the renement calculus have been used for showing
that test-cases are abstractions of formal specications (contracts). The abstraction
relation have been proved for dierent kinds of test-cases. Simple input-output test-
cases, non-deterministic test-cases, test partitions, and test-sequences, here called
test scenarios, have been covered. The presented theorems clarify the role that
test-cases play in a development process, formally we have:
test-cases _ specication _ implementation
The fact that many dierent types of test-cases can be handled, demonstrates
the generality of our approach of choosing abstraction as the correctness criterion for
test-cases. In contrast to previous work on testing, one criterion covers the simplest
form of input-output test-cases as well as the most complex sequences of test-cases.
Susan Stepney did use an abstraction relation, as well [Ste95]. However, in her
work abstraction is mainly used for nding and structuring partition test-cases for
object-oriented Z specications. She covered neither input-output test-cases, nor
test-sequences.
The important conclusion that can be drawn from this chapter is: Test-cases are
abstract formal contracts that partially dene a systems functionality.
The fact that test-cases are more abstract than formal specications that cover
the whole input-output domain, like pre- postconditions, does not imply that test-
cases are better suited for specication purposes.
Test-cases are indeed too abstract for replacing specications using a formal spec-
ication language. However, the reasons why test-cases are so popular in software-
engineering can be given in the presented context:
Test-cases are formal contracts. Therefore, they are more precise than natural
language specications.
Test-cases are abstract. The fact that test-cases include less detail than pro-
grams or pre- postcondition specications is the reason why they are easily
and quickly understood by engineers as well as by customers.
The theorems that have been presented in this chapter, are the prerequisite for
developing formal synthesis rules for testing. In the following chapter, such formal
abstraction rules are presented. The rules dene how test-cases can be calculated
from a formal specication. The dierent kinds of rules represent the dierent test
strategies how useful test cases are to be selected.
Chapter 6
Testing Strategies
A strategy for selecting test-cases is based on a hypothesis about faults in a pro-
gram. This chapter shows that such strategies can be formulated as formal synthesis
rules for deriving test-cases. This is a consequence of our observation that correct
test-cases are abstractions of contracts, as has been explained in Chapter 5. The ab-
straction rules of the prominent strategies of partition testing and mutation testing
are presented. In addition, it is shown that structural testing strategies are covered
by this approach.
Furthermore, a new technique for calculating sequences of test-cases, here called
scenarios, is presented. In contrast to previous work on test sequencing, our ap-
proach does not need to compute a nite state machine. This is in contrast to
previous work on testing from state-based specications. A well-known example
from the testing literature serves to demonstrate this unusual application of the
renement calculus in order to synthesize tests rather than implementations.
6.1 Partition Testing
Partition testing techniques are based on a uniformity hypothesis. This strategy
assumes that a system shows the same behavior for a certain partition of input
values. Therefore, once the equivalence partitions are selected, it is sucient to test
one case for every partition.
For partitioning model-based specications, [DF93] proposed the rewriting of
specications into their disjunctive normal form (DNF). This popular technique is
based on rewriting disjunctions with:
a b (a b) (a b) (a b)
The proof of the following abstraction rule for this well-known strategy high-
lighted a problem with this technique. In general, DNF-rewriting results in disjoint
partitions. Applied to a relational specication it gives disjoint input-output rela-
tions, but the input partitions (the domain of the relation) may overlap.
77
78 CHAPTER 6. TESTING STRATEGIES
In such pathological cases, selecting a test-case for each partition is not cor-
rect. If test-cases are derived for overlapping domains and the specication is non-
deterministic then two test-cases with the same input may dene two dierent de-
terministic outputs.
Therefore, the disjointness of the resulting input partitions dom.Q
i
is assumed
in the following synthesis rule.
Theorem 6.1 Let a and b be arbitrary Boolean expressions and Q
1
, Q
2
, Q
3
be
relations. The following rule for deriving partition test-cases from pre-postcondition
contracts is then generally valid:
[x: = x
[ a b] _ [Q
1
],
[x: = x
[ a b] _ [Q
2
],
[x: = x
[ a b] _ [Q
3
],
i , j 1, 2, 3 i ,= j dom.Q
i
dom.Q
j
=
[x: = x
[ a b] _ dom.Q
1
; [Q
1
] .
dom.Q
2
; [Q
2
] .
dom.Q
3
; [Q
3
]
Proof.
[x: = x
[ a b]
= by the DNF-partitioning rule
[x: = x
[ (a b) (a b) (a b)]
= by denition of demonic choice
[x: = x
[ (a b)] [x: = x
[ a b] [x: = x
[ a b]
_ by _-assumptions and monotonicity of
[Q
1
] [Q
2
] [Q
3
]
= by explicitly stating the domains as assumption
[dom.Q
1
]; [Q
1
] [dom.Q
2
]; [Q
2
] [dom.Q
3
]; [Q
3
]
= by the fact that the choice is deterministic, since:
(1) the domains are disjoint by assumption,
(2) the whole input domain is covered,
since abstraction cannot reduce the domain of a relational update.
dom.Q
1
; [Q
1
] . dom.Q
2
; [Q
2
] . dom.Q
3
; [Q
3
]
The derivation rule of Theorem 6.1 yields three disjoint partitions, although a
derived partition might be empty, thus dom.Q
i
= false. It follows from Theorem 5.3
that each of these partitions is an abstraction of the original contract. It is obvious
that the rule has to be applied recursively to the resulting partitions if further sub-
partitions are needed.
The proof of Theorem 6.1 shows that the derivation rule yields partition test-
cases that cover the full input domain. Furthermore, it follows from the assumptions
that for one partition more non-determinism can be introduced due to abstraction.
This reects the possibility of weakening certain tests.
Weakening of Tests: The rule shows that test-cases can be further abstracted
by weakening a demonic update. In the rule above, for example the premise [x: =
x
[ a b] _ [Q
1
] indicates such a possible weakening of a test-case. This form of
weakening a test-case is a further strategy for test-case design. The reason for this
further abstraction of a partition might be that
a tester does not wish to observe the (demonic) changes of the whole state
space, or
not the whole state and thus its updates are observable.
Therefore, parts of the demonic updates of the system are skipped in the test-case
specications. Formally this is an abstraction process carried out by weakening a
post-condition.
Example 6.1 The rule is illustrated by deriving the test-cases from a specication
of the computation of the minimum of two numbers.
[z : = z
[ (z
= x x y) (z
= y x y)]
= by the partitioning rule
x < y; [z : = z
[ (z
= x x y) (z
,= y x < y)] .
x > y; [z : = z
[ (z
,= x x > y) (z
= y x y)] .
x = y; [z : = z
[ (z
= x x y) (z
= y x y)]
= by simplication
x < y; [z : = z
[ z
= x] .
x > y; [z : = z
[ z
= y] .
x = y; [z : = z
[ z
= x z
= y]
In the third test partition x = y a further abstraction by weakening the rela-
tional update might be applied. For testing the correct computation for input x = y,
a tester might-choose to compare the new value z
only with one input variable. If

x is compared to z , then the following three test cases are obtained:
_ by weakening the post-condition in the third test-case
x < y; [z : = z
[ z
= x] .
x > y; [z : = z
[ z
= y] .
x = y; [z : = z
[ z
= x]

Note that it has been the search for a valid abstraction that highlighted the
necessary condition of disjointness. The last step in the proof shows that only if the
demonic choice is in fact deterministic it can be transformed into an angelic choice
. of partitions. For a general demonic choice of partitions, the abstraction relation
would not hold. The consequence could be the derivation of incorrect test-cases.
The second example shows such a pathological case, where the rule must not be
applied.
Example 6.2 The example demonstrates the consequence, if the rule is applied to
non-disjoint partitions.
[z : = z
[ z
= 0 z
= 1]
_ by a wrong application of the partitioning rule
true; [z : = z
[ z
= 0 z
,= 1] .
true; [z : = z
[ z
,= 0 z
= 1] .
false; [z : = z
[ z
= 0 z
= 1]
= by simplication
true; [z : = z
[ z
= 0] . true; [z : = z
[ z
= 1]
_ by selecting input-output test-cases
(TC 1 0) . (TC 1 1)
The result of this wrong application of the rule is the derivation of two deter-
ministic test-cases for a non-deterministic result. If applied in testing an implemen-
tation of this specication, these test-cases indicate an error in the implementation
that does not exist. The correct solution, for non-disjoint domains is to merge the
partitions. Then, only non-deterministic test-cases would have been possible, like
TCp 1 (z
= 0 z
= 1).
6.2 Structural Partitioning

In the review of the related work, structural approaches for partitioning B-specications
have been discussed (see Section 2.1.4). Here, the syntax of the specication lan-
guage is considered for partitioning. This structural technique provides a better
control of the partitioning process, then pure DNF-partitioning.
Our abstraction approach can be extended for dening test-case synthesis rules
for dierent kinds of specication statements. For example, the rule for partitioning
6.3. MUTATION TESTING 81
a conditional specication can be dened as follows:
Theorem 6.2 Given a conditional contract statement as dened in Section 5.2.2.
C
1
and C
2
be arbitrary contracts and P a predicate. Then, a conditional contract
can be partitioned by
C
1
_ C
1
,
C
2
_ C
2
if P then C
1
else C
2
_ P; C
1
. P; C
2
Proof.
The rule is valid by the denition of the if-statement and the monotonicity of the
_ relation.
Synthesis rules, like the one for if-statements, can be given for arbitrary speci-
cation statements. Each such syntax oriented synthesis rule reects a test-selection
strategy, known from white-box testing. For example, the rule above represents
branch testing, a strategy, where every branch in the control-ow should be tested
once.
The following example demonstrates that a structural strategy might help to
reduce the set of test-cases.
Example 6.3 The rule is illustrated by deriving the test-cases from an alternative
specication of the computation of the minimum of two numbers.
if x y then [z : = z
[ z
= x] else [z : = z
[ z
= y]
= by the partitioning rule for if-statements
x y; [z : = z
[ z
= x] . x > y; [z : = z
[ z
= y]
In contrast to the three test-cases of Example 6.1, here only two test-cases are
derived for the same problem.
6.3 Mutation Testing

Mutation testing is a fault-based testing technique introduced by Hamlet [Ham77]
and DeMillo et al [DLS78]. It is a means of assessing test suites. When a pro-
gram passes all tests in a suite, mutant programs are generated and the suite is
assessed in terms of how many mutants it distinguishes from the original program.
The hypothesis is that programmers only make small errors. Stocks extends this
technique in [Sto93] to model-based specication languages by dening a collection
of mutation operators for Zs specication language. An example for specication
mutation is the exchange of the join operator of sets with intersection . From
these mutants, test-cases are generated for demonstrating that the implementation
does not implement one of the specication mutations.
We can formulate abstraction rules for deriving mutation test-cases for a given
contract and its mutants.
Theorem 6.3 Given a contract C and a mutation C
m
generated by applying a
mutation operator m such that m.C = C
m
holds. Then a test-case TC i o that
covers the mutation must be derived by the rule
C _ TC i o,
C
m
,_ TC i o
C _ TC i o
The rule shows that a test-case for nding errors in a mutant, has to be, rst, a
correct test-case of the original contract, second, it must not be a an abstraction of
the mutated contract. Test-cases that are abstractions of the mutations do not cover
the mutated parts of a contract. Consequently, the coverage criteria of a collection
of test-cases T for a contract C in mutation testing with a collection of mutation
operators M can be given by:
m M tc T (C _ tc m.C ,_ tc)
This represents a new approach to the quality criteria for test-cases based on
mutations. Again, the minimum example serves to illustrate the synthesis rule.
Example 6.4 This example shows how mutation testing techniques might be ap-
plied to specication based testing. Consider this specication C of a minimum
computation:
[ z
= x] else [z : = z
[ z
= y]
In mutation testing, the assumption is made that programmers produce small
errors. A common fault made by programmers is that operators are mixed. In this
example a possible fault would be to implement the < operator instead of the .
Hence, a mutation operator
m(... ...) = ... ...
is dened that changes all occurrences of in a contract to . Applying this
mutant operator m.C = C
m
computes the following mutant:
[ z
= x] else [z : = z
[ z
= y]
6.4. CALCULATING SCENARIOS FOR TESTING 83
The synthesis rule says that a successful test-case for this mutation must not be
an abstraction. Since
[ z
= x] else [z : = z
[ z
= y] ,_ x < y; [z : = z
[ z
= x]
but
[ z
= x] else [z : = z
[ z
= y] _ x < y; [z : = z
[ z
= x]
a test-case x < y; [z : = z
[ z
= x] is a valid test-case for detecting a specication

mutation of this kind in our implementation. On the other hand, the test-case
x = y; [z : = z
[ z
= x] would not detect the mutation since it is an abstraction

of both, the original specication and the mutated one.
6.4 Calculating Scenarios for Testing

In this section an alternative strategy for the sequencing of test-cases into test-
scenarios is proposed.
6.4.1 Critical Remarks on FSM approaches
In the literature on related work on test-sequencing for model-oriented specications,
authors have been concentrating solely on the approach proposed by Dick and Faivre
in [DF93]. This strategy rst calculates partitions of the available operations and
states. Then a nite state machine (FSM) is constructed by calculating possible
transitions between the states. The result is a graph with nodes that are state
partitions and transitions that are operation partitions. To derive test-sequences
(scenarios) the tester follows the paths in the graph. See the related work summa-
rized in Chapter 2 for examples of this approach.
One disadvantage of this technique is that the whole FSM has to be calculated in
advance, even if full coverage is out of the testers scope due to resource limitations.
This situation is even worse: Due to the focus on state partitions, the number of
states increases exponentially with the number of partitioned state variables. Hence,
rather large FSMs have to be calculated in advance. The second disadvantage is that
a state based testing strategy is enforced, although the contract does not emphasize
states but, like for interactive systems, possible interactions are the central paradigm
of description.
In the following, a scenario oriented testing strategy is proposed. We call it a lazy
technique, since the test-cases are calculated by need. It does not calculate a FSM,
since it is not based on states. It is based on atomic scenarios, called compositions.
6.4.2 Compositions
We dene a composition of an interactive system as a terminating sequential com-
position of two interactions:
Composition = C
1
; C
2
The following corollaries follows directly from Theorem 5.6 and denes a rule for
calculating such compositions.
Corollary 6.1 For a consistent specication of interactions we have that
(p g
a
) S
a
.g
b

do
n
i
g
i
: : S
i
od _
T
p g
a
; S
a
; g
b
; S
b
where 1 a, b n holds and p is an arbitrary predicate such that p ,= false.
In practice, we will not calculate the compositions from the original specica-
tion, but will previously perform a partition analysis on the interactions, leading to
more (partition) interactions. However, the approach keeps the same. These com-
positions should be calculated for all interaction partitions of interest. Next, these
compositions are combined into scenarios.
6.4.3 Scenario Synthesis
The following rule denes the general calculation of scenarios by combining two
compositions of interest.
Corollary 6.2 Let the interactions with indices 1 i , j , k n be interactions of
an interactive system with n interaction partitions, then
do
n
i
g
i
: : S
i
od _
T
p
1
g
i
; S
i
; g
j
; S
j
p
2
g
j
; S
j
; g
k
; S
k
p p
1
g
1
S
j
.(p
2
g
2
)
do
n
i
g
i
: : S
i
od _
T
g
i
; S
i
; g
j
; S
j
; g
k
; S
k
In order to generate valid scenarios, a tester can e.g. start by an initial interaction
with a guard equal to true and then he further searches for compositions leading to
his test-goal. Which scenarios and how many scenarios are tested, depends on the
testing strategy.
6.4.4 Scenario Based Testing Strategies
The new test approach can be divided into three phases:
1. calculation of interesting partitions for each interaction.
6.5. EXAMPLE: PROCESS SCHEDULER 85
2. calculation of compositions.
3. combination of compositions to validate or to generate test-scenarios.
Dierent test-coverage strategies can be derived, determined by the strategy for
combining the compositions. Interesting scenario analysis strategies are:
Derive scenarios that include for each partition
one composition consisting of the partition: for each partition one scenario.
all possible compositions consisting of the partition: for each partition, one
scenario for each interaction reaching the partition.
all possible combinations of compositions between two interactions of interest:
all scenarios leading from one interaction of interest to another.
all possible combinations of compositions: all possible scenarios.
The strategies are similar to the testing strategies used in data-ow testing
[Bei90]. The dierence is that here atomic scenarios, called compositions, are con-
sidered, and in data-ow testing data-objects. An example will serve to demonstrate
the approach at work.
6.5 Example: Process Scheduler
In this section, the application of our test synthesis method is demonstrated by
an example, well known in the formal methods testing literature. It is the process
scheduler introduced in [DF93] and translated to Back and von Wrights contract
notation. We have chosen this example, although it is not new, because it allows a
comparison to the traditional FSM approach most easily. Test-cases for industrial
examples from our projects have been calculated, too. These more complex examples
will be published in future publications.
6.5.1 Interactive Process Scheduler
The system consists of processes either ready to be scheduled or waiting to become
ready and, optionally, a single active process. These processes are identied by a
unique Pid Nat. A process cannot be both ready and waiting, and the active
process is neither ready nor waiting. In addition, there must be an active process
whenever there are processes ready to be scheduled. The scheduling algorithm is
not further specied.
We can model the interactions with this process scheduler as shown in Figure 6.1.
In this specication a : Pid [ nil is a global variable representing an optional active
process, the global variable r : set of Pid and w : set of Pid represent the sets of ready
Init [a, r, w: = a
, r
, w
[ a
= nil r
= ]
New p: = p
[ p
,= a p
, (r w); [w: = w
[ w
= w p
set
]
Ready p: = p
[ p
w; [a, r, w: = a
, r
, w
[w
= w-p
set

a = nil (r
= r a
= p)
a ,= nil (r
= r p
set
a
= a)]
Swap a ,= nil; [a, r, w: = a
, r
, w
[r = (a
= nil r
= )
r ,= (a
r r
= r-a
set
)
w
= w a
set
]
Figure 6.1: Contracts of the process schedulers initialization and interactions.
and waiting processes. Furthermore, p : Pid is a global input variable solely used for
setting a parameter.
New introduces another process, Ready puts a process into the ready state, and
Swap changes the active process. In order to prevent a confusion with assertions,
p
set
and a
set
are used for denoting the sets p, a containing the single element
p and a. Swap is an example of an operation with a precondition a ,= nil that
cannot be directly satised by choosing a parameter. This is in contrast to Ready,
where the precondition w ,= follows from the angelic update p: = p
[ p
w
and thus must not be explicitly stated.
The interactive process scheduler can be dened by iterative choice of these
interactions. The initialization statement should be executed once prior to user
interaction. In Figure 6.2 this model is shown. Note that the precondition of Swap
has become a guard. Furthermore, necessary conditions such that a parameter
selection is possible are documented in the guards. Here w ,= is such a precondition
of Ready.
Scheduler Init; do true : :New
w ,= : :Ready
a ,= nil : :Swap
od
Figure 6.2: Contract of the interactive process scheduler.
6.5. EXAMPLE: PROCESS SCHEDULER 87
Ready
1
a = nil; p: = p
[ p
w; [a, r, w: = a
, r
, w
[w
= w-p
set

r
= r a
= p ]
Ready
2
a ,= nil; p: = p
[ p
w; [a, r, w: = a
, r
, w
[w
= w-p
set

r
= r p
set
a
= a]
Swap
1
a ,= nil r = ; [a, r, w: = a
, r
, w
[(a
= nil r
= )
w
= w a
set
]
Swap
2
a ,= nil r ,= ; [a, r, w: = a
, r
, w
[(a
ready r
= r-a
set
)
w
= w a
set
]
Figure 6.3: Partitioned process scheduler.
6.5.2 Interaction Partitioning
For test-case synthesis, we rst partition the basic interactions. Here, our parti-
tion strategy will be based solely on the case distinctions in the contract. As a
consequence, the interaction New is not partitioned. Figure 6.3 presents the new
partitions after some simplication.
Further partitioning based on interesting states would be possible. Here, for
example, New may be further partitioned into cases where w = nil and w ,= nil .
Any partition is possible and can be formulated as a rewriting rule, such that the
resulting partitions are correct abstractions as stated in Theorem 2.
As a consequence of this partitioning, a new interactive system contract can be
given. In this new description shown in Figure 6.4, the partition preconditions are
incorporated into the guards. This is necessary such that our scenario synthesis
approach works.
Scheduler
Init; do true : :New

w ,= a = nil : :Ready
1
w ,= a ,= nil : :Ready
2
a ,= nil r = : :Swap
1
a ,= nil r ,= : :Swap
2
od
Figure 6.4: Partitioned contract of the interactive process scheduler.
a = nil; New; Ready
1
a ,= nil; New; Ready
2
Swap
1
; Ready
1
card w > 1; Ready
1
; Ready
2
card w > 1; Ready
2
; Ready
2
Swap
2
; Ready
2
r = ; Ready
1
; Swap
1
card r > 1; Swap
2
; Swap
2
r = a ,= nil; New; Swap
1
Ready
2
; Swap
2
card r = 1Swap
2
; Swap
1
a ,= nil; New; Swap
2
S; New
Figure 6.5: Compositions of the process scheduler.
6.5.3 Compositions
The next step, is the calculation of atomic scenarios the compositions. The
calculation is done by applying the rule of Corollary 3 to the partitioned interactive
system contract. In many cases, the precondition p of a composition is stronger than
the guard g
a
of the rst interaction S
a
. In the formula of Corollary 3 this means
that p ,= true. The reason for the additional constraint p is that the interaction
S
a
does not guarantee that g
b
is satised. This ts perfectly into our approach,
since precondition strengthening is in fact abstraction. However, such strengthening
indicates paths that are more dicult to establish. In the trivial case p = g
b
, which
means that the precondition of the composition is the conjunction of the two guards
g
a
and g
b
.
The compositions of the scheduler are listed in Figure 6.5. In this presentation
the guard assertions g are skipped. Only if a guard g has been strengthened by
a precondition p the additional assertion is shown as the precondition p of the
composition of the two interactions.
This collection of possible compositions has several advantages: (1) Several sce-
narios can be calculated step-wise, without calculating the weakest precondition for
the whole sequence of interactions again and again. (2) It carries the information
which interactions are easily established and which are dicult to set up. For setting
up a goal as quickly as possible, choosing simple compositions will lead to shorter
scenarios. On the other hand, strong preconditions indicate that these combinations
are complicated to carry out. A tester should include such complex combinations.
The compositions are grouped by the second statement, which is more practical
for searching scenarios backwards. Backwards scenario development is more useful
if scenarios are used to reach a certain test goal, as will be seen next.
6.5.4 Scenarios
Applying the rule for composing two compositions, correct scenarios can be synthe-
sized in a systematic way. In Figure 6.6 one scenario for testing each partition is
presented. For each scenario the additional precondition to be established is docu-
mented. Scenario
3
serves to discuss the synthesis process in more detail.
The actual scenario synthesis starts with the last statement. Here, this is Ready
2
,
the interaction to be tested. From Figure 6.5 it can be seen that four compositions
are available. Ready
1
is chosen because a scenario for Ready
1
is already available.
However, the new precondition forces to choose New twice.
New is chosen, because it is the most promising: Here, New can follow each
statement S, since it has the precondition true. It is a general strategy to choose
interactions with weak guards.
Scenario
4
and Scenario
5
shows that scenarios can be reused for setting a system
in a state of interest.
Based on the table of possible compositions, all scenarios according to one of
the selection strategies that have been presented in the previous section can be
calculated. Here, we applied the rst strategy: One scenario for each partition.
It should be noted that for this simple strategy, not all compositions have to be
calculated in advance. However, compositions carry the information which combi-
nations are most easily achieved, those with the weakest additional precondition.
Trivially, it equals true.
What we have presented, is to our knowledge, the rst application of the renement
calculus for generating test-cases. In this chapter several formal abstraction rules for
calculating correct test-cases have been presented. These rules represent dierent
strategies for test-case selection, known as DNF-partition testing, structural testing
and mutation testing.
The presented synthesis rules for scenario calculation dene an alternative method
for nding sequences of interactions. In contrast to nite state machine (FSM) based
approaches, the focus is on nding possible compositions of interactions. Which
compositions are possible is determined by the abstraction rules. A well-know ex-
ample has been translated into an interactive contract specication and served for
illustrating purposes of the method.
The advantage of our scenario calculation method is that it can be applied by
need. Therefore, we call it a lazy technique. Dependent on the systems level of
reliability new test-scenarios can be added. The focus on compositions, high-lights
which sequences appear to be trivial and which seem to be more-interesting, from a
testers perspective.
It is the authors opinion that an abstraction calculus for testing, provides deeper
Scenario
1
Init; (Testing New)
New
Scenario
2
Init; (Testing Ready
1
)
a = nil; New;
Ready1
Scenario
3
Init; (Testing Ready
2
)
a = nil; New;
a = nil card w > 0; New;
card w > 1; Ready
1
;
Ready
2
Scenario
4
Scenario
2
; (Testing Swap
1
)
Swap
1
Scenario
5
Scenario
3
; (Testing Swap
2
)
Swap
2
Figure 6.6: Testing scenarios for the process scheduler.
insight into the methods of test-case derivation. Especially, the understanding of
new black-box testing strategies is supported by reasoning about the synthesis pro-
cess. The examined testing strategies showed that even existing techniques reveal
interesting properties. New test-case generation techniques may be the result.
Chapter 7
Automated Test Result
Verication
In this chapter the possibilities to automate the test-result verication through for-
mal requirement specications are explored. More precisely, the formal method
VDM (Vienna Development Method) serves to demonstrate that abstract require-
ment models can be used as test oracles for concrete software. It is shown that
techniques of VDMs invent and verify development method can be adapted to set
up a test environment. The automation of the resulting testing frame-work is based
on modern CASE-tools that support a light-weight approach to formal methods
as explained in Chapter 3. Here, the specication language used is VDM-SL, but
the results are easily transferred into similar model-based methods such as B, Z or
RAISE.
7.1 From Formal Specications to Oracles
The Vienna Development Method provides the two concepts needed to support
the presented testing process. First, the formal specication itself, which precisely
denes what a function or operation should compute. Second, the concept of data-
reication, that provides a formal denition of abstraction. In the following, both
concepts are explained and their role in our testing approach is claried.
7.1.1 VDM-SL Specications as Oracles
As already mentioned, in VDM a system is specied in terms of abstract math-
ematical models. VDM-SL, the general purpose specication language of VDM,
provides mathematical objects, like sets, sequences, maps etc., to model a systems
state. The functionality is formulated by imperative operations, which manipulate
these abstract state representations or applicative functions. Two styles can be
distinguished to dene functionality: implicit and explicit denitions. An implicit
91
92 CHAPTER 7. AUTOMATED TEST RESULT VERIFICATION
operation denes what should happen by pre- and post-conditions.
The following example is inspired by the industrial projects that have been pre-
sented in Chapter 4 in which a voice communication system for air trac control
has been formally specied. Here, an abstract view of the systems radio switch is
modeled. The switch assigns frequencies to operator positions on which communi-
cation may be established. The VDM-SL model refers to frequencies and operator
positions by the identiers FrqId and OpId. The relation is modeled by a nite map-
ping from operator positions to a set of frequencies. A map can be interpreted as a
two-column table whose left- and right-side entries can be accessed by the domain
(dom) and range (rng) operators.
The set of frequencies represents frequency coupling, which means that commu-
nication is forwarded to all frequencies in the set. A requirement of the switch is
that a frequency must not belong to two dierent coupling sets. This is expressed
by means of a data-invariant, stating that for all two frequency sets, with more than
one element, they may not have frequencies in common.
Switch-a = OpId
m
FrqId-set
inv s

fs1, fs2 rng s card fs1 > 1 card fs2 > 1 fs1 fs2 =
A function couple-frequency, dened by pre- and post-conditions, adds frequencies
to an operator position. The pre-condition says that if the operator position has
already a frequency associated, then no frequency set with a cardinality greater than
one must exist that already contains the frequency to be added. This would lead to
violation of the systems invariant.
The post-condition establishes the following relation between the input (f, op
and s) and the resulting output r: If the operator position has already a frequency
associated, then the resulting map equals the old one with the input frequency added
to the existing set of frequencies of the operator position. This is expressed by over-
riding () a map entry. In case of a new operator position, the result equals the old
switch with the new map entry.
couple-frequency (f : FrqId, op : OpId, s : Switch-a) r : Switch-a
pre op dom s fs rng s f fs card fs > 1
post if op dom s
then r = s op s (op) f
else r = s op f
From a testers perspective, the post-condition serves as a test oracle. In general,
an oracle is any program, process or data that species the expected outcome of a
set of tests as applied to a tested object [Bei90]. Here, the oracle is a predicate,
a Boolean function, which evaluates to true if the correct output is returned, with
7.1. FROM FORMAL SPECIFICATIONS TO ORACLES 93
C
A
C
A
r r
op
abstract
op
concrete
Figure 7.1: The morphism of abstraction.
the premise that the pre-condition holds. The signature of the post-condition oracle
above is:
post-couple-frequency : OpId FrqId Switch-a Switch-a B
The two arguments of type Switch-a dene the relation between the old and new
value of the switch. Note that pre-conditions dene the input values for good tests,
where valid outputs can be expected.
However, the systems model and consequently the oracle is abstract and cannot
be directly used as a test-oracle for a test on implementation level. What is needed,
is a link between the abstract level and the concrete level. In VDM, and other formal
methods, this link is provided by a precise denition of abstraction.
7.1.2 The Homomorphism of Abstraction
The M in VDM is based on a renement of an abstract specication to a more
concrete one, which usually is carried out in several steps. Rening the data model
is called data-reication, which is correct if it establishes a homomorphism between
the abstract and rened, more concrete, level.
In Figure 7.1 our notion of abstraction is represented by a commuting diagram,
where operations (op) are viewed as functions from input to output states. The
abstract operation op
abstract
manipulates an abstract state representation of type A.
The concrete operation op
concrete
incorporates detailed design decisions and maps
a rened input state to an output state. Examples for such data-renement would
be to implement a set through a balanced binary-tree, or the other way round, to
view a data-base system as a set of data. The relationship between the abstract and
the concrete is dened by a a function r mapping a concrete state to its abstract
counterpart.
The diagram shows that the retrieve function r is a homomorphism for which
the following formula holds:
in : C op
abstract
(r(in)) = r(op
concrete
(in))
Using an implicit function denition on the abstract level, the formula for a
correct implementation (renement) is:
in : C pre-op
abstract
(r(in)) post-op
abstract
(r(in), r(op
concrete
(in)))
This simply means that the abstract post-condition must hold for the concrete
input and output, mapped to the abstract level by r, if the pre-condition is satised.
Modern tools allow the interpretation of explicit VDM-SL denitions. If the
retrieve function and the post-condition predicate are dened explicitly, the formula
above can be evaluated and thus provides a testing frame-work inside VDM-SL.
However, to test functionality implemented in a programming language like C++,
the language gap between programming languages and the specication language
has to be overcome. In the following it is shown how modern tools provide these
bridges and help to automate the approach.
7.2 Automated Test Evaluation
The combination of the notion of abstraction and the possibility to use specica-
tions as test oracles leads to a testing framework with abstract oracles. Modern
tools like the IFAD VDMTools (see Chapter 3) allow the interpretation or even
code-generation of such pre- and post-conditions which leads to an automated test
evaluation through post-condition oracles.
In Figure 7.2 the data-ow of the new testing framework is presented. An imple-
mentation is executed with a concrete test input (state). An implemented retrieve
function maps the concrete input and output to its abstract representations. A
pre-condition check validates the input and feeds it into the oracle which checks the
relation to the produced output. If the post-condition evaluates to true, the test
passed.
The advantage of the approach is the automation of black-box testing by the
usage of structural test-data. Another possibility would be to code the reverse of r
in order to test with abstract test-data derived from the specication.
In the following, the possibilities for automation by using the IFAD tools are
explained in more detail.
7.2.1 Code-Generation of Oracles
The IFAD VDMTools provide a C++ code-generator, which translates explicit
VDM-SL specications to C++ source code, using a VDM library. With this tool the
7.2. AUTOMATED TEST EVALUATION 95
postcondition
oracle
implementation
precondition
check
r r
concrete input concrete output
abstract
input
valid abstract
input
abstract
output
input valid? test ok?
Figure 7.2: DFD of the testing framework.
post-condition functions, like in our Switch example, can be automatically trans-
lated to C++ . Even the pre-condition with its quantors over nite sets can be
code-generated. Below the generated C++ code for post-couple-frequency is shown:
Bool post_couple_frequency(int f, int op, Map s, Map r) {
Bool varRes;
Bool cond;
cond = (Bool) s.Dom().InSet(op);
if (cond.GetValue()) {
Map var2;
Map modmap;
Set tmpVar2;
Set var2;
var2 = Set().Insert(f);
tmpVar2 = (Set) s[op];
tmpVar2.ImpUnion(var2);
modmap = Map().Insert(op, tmpVar2);
var2 = s; var2.ImpOverride(modmap);
varRes = (Bool) (r == var2);
}
else {
Map var2;
Map modmap;
modmap = Map().Insert(op, f);
var2 = s; var2.ImpOverride(modmap);
varRes = (Bool) (r == var2);
}
return varRes;
}
What remains to implement manually is the retrieve function. Special C++
generator functions in the VDM library facilitate the conversions of C++ types into
VDM objects.
7.2.2 Interpretation of Oracles with the Dynamic Link Fa-
cility
Another approach is to use the specication interpreter in order to evaluate the oracle
inside VDM-SL. The dynamic link facility may be used to call the C++ functions to
be tested from inside the IFAD Toolbox [FL96]. The dynamic link facility enables
a VDM-SL specication to execute parts written in C++, during interpretation. It
is called dynamic link facility because the C++ code is dynamically linked with the
Toolbox.
In order to access the implementation, a dlmodule has to be dened which con-
tains the functions declarations that are to be tested. For our example that would be
dlmodule CPP
exports
functions couple-frequency-ext : Z Z Z
m
Z-set Z
m
Z-set
end CPP
if we model frequencies and operator positions as integers (Z). In order to test
couple-frequency in this case two functions are to be implemented. One mapping
the input to the implementation representation and calling couple-frequency, and the
retrieve function which converts the output back to the abstract VDM-SL data-type.
With this approach abstract VDM-SL test-cases are t into the implementation.
Hence, the following must hold for a correct implementation op
concrete
:
in : A pre-op
abstract
(in) post-op
abstract
(in, r(op
concrete
(rep(in))))
where the function rep:A C maps the abstract input to a concrete representation.
7.2.3 CORBA Oracles
A third elegant way to bridge the language gap is CORBA, the Common Object
Request Broker Architecture [OMG96]. The VDM
++
Toolbox provides a CORBA
based API (Application Programming Interface) through which a program can ac-
cess the functionality of the VDMTools [Sto98].
The CORBA interface of the VDMTools is dened in IDL, the interface denition
language of CORBA. The IDL provides a language independent interface that maps
the dierent programming languages to the specication language of VDM. The set
of programming languages that are supported is dependent on the object request
broker (ORB) that is in use.
The IDL interface of the VDMTools, denes the VDMTools as a class of objects.
This tool-class includes a set of methods through which the full functionality of a
running VDM toolbox may be invoked. Thus, all of the functionality that usually is
invoked through the graphical user interface may be called from other applications.
This includes the loading of a specication into the tool, starting the type-
checker, and, most important for our approach, the invocation of the specication
interpreter. IFADs intention for this API is the combined execution of VDM spec-
ications and program source-code. This facilitates, for example, the integration of
graphical user interfaces with executable VDM specications for validation purposes.
Since the API is based on the distributed architecture of CORBA, the program ob-
jects and the VDM interpreter may run on dierent machines that communicate
over a network.
However, with this technique a post-condition oracle can be accessed and evalu-
ated through the API, as well. The VDMTools may even run on a dierent (test-
)server providing the VDM oracles. Since CORBA IDL converts dierent program-
ming languages, the test approach can be applied to as many programming languages
as IDL mappings have been implemented, e.g. C++ and Java.
In this chapter we have presented an approach for automated test evaluation through
VDM-SL oracles. We have shown the general strategy and presented several ways
how the resulting test approach can be automated. The three presented solutions
for automation are based on the commercial IFAD VDMTools.
To our present knowledge, only one automated black-box testing approach has
been based on an explicit mapping from an implementation level to a formally speci-
ed and abstract post-condition oracle [BF86]. In this early work, the post-condition
oracle had to be translated manually into Prolog. Of course, other formal model
oriented approaches to testing have been published, but they dier in two aspects:
First, the focus lies on test-case generation, rather than on oracle generation and au-
tomated test-result verication [Las90, DF93, Sto93, HNS97, PS97, Don97, Aer98,
Ped98]. Hence, the abstraction problem is not considered at all. Second, explicit
specications serve as active oracles which calculate and compare the specied out-
put to the program under test [Har92, ALR98, TDS98]. In contrast to our passive
oracles, these solutions cannot handle non-deterministic results.
Our idea has been presented in [Aic98] the rst time. This work could also
be extended with test-case generation techniques in order to automate the whole
test process. However, the automated test-result verication technique especially
supports random testing.
In future we envisage an instrumentation of objects with post-condition oracles.
Objects can be instrumented through inheritance without changing the actual code.
Then, testable objects inherit the functionality from its superclass and provide ad-
ditional retrieve and oracle functions as testing methods.
Our test-approach has been successfully applied in the masters thesis project of
our student Pascal Fenkam [Fen00]. Currently, an object-oriented Java framework
is under development that eases the application of the presented test approach.
We feel that the application of both, abstract formal specications and formal
development tools to testing, as presented here, is a powerful combination.
Chapter 8
Case Study
This chapter demonstrates our abstraction test approach by a case study that is in-
spired by our industrial experiments. The simplied requirements of a functionality
taken from a voice-communication system for air-trac control serve as an illus-
trating example. First, the formalization of the requirements is presented. Then,
a contract that species the interactions of tester with the system is presented.
From this contract test scenarios are calculated by searching compositions of the
partitioned operations.
8.1 Roles in Voice Communication
The functions that have to be formalized in this case study provide the management
of roles in a voice-communication system for air-trac control.
A role is a group of functionalities that is available at an operator position: for
instance the functionality to control a certain sector of air trac. These roles
or functionalities are not bound to specic operator positions (OPs), but can
be dynamically distributed within the network, to optimize the work-load on the
controllers. Therefore, it should be possible to control air space, independently of
the physical location of the controller.
The functionality that is modeled in our systems view is
role request: an operator position requests a role, if possible the role is trans-
ferred to the operator position.
role release: an operator position releases one of its roles.
call: a call is established from one operator position to a another by dialing
the phone number of a role.
hang up: a call is released.
enable OP: an operator position is enabled.
99
100 CHAPTER 8. CASE STUDY
disable OP: an operator position is disabled.
The details of the functionality above, are presented in the following formal speci-
cation. First, the global state is modeled, then the operations are dened.
8.1.1 An Abstraction of the Systems State
The global state space of the system is dened by the three state variables for
ops:Ops, the operator positions connected to the voice-communication system,
roles : Roles that are statically and system-widely dened,
calls : Calls, currently established between operator positions.
An array of operator positions Ops represents the existing working places of the
air-trac controllers. Each operator position Op is modeled by its set of function
groups fgs that are installed at this operator position, the roles that are currently
associated to this OP, and a Boolean value denoting if the OP is enabled. Here,
function groups and roles are identied by natural numbers.
Ops array 1..opmax of Op
Op record fgs : set of Nat
roles : set of Nat
enabled : Bool
The globally known roles in a system are dened by an array of roles. Each
role represents a function group fg and a number num through which an operator
position can be called.
Roles array 1..rmax of Role
Role record fg : Nat
num : Nat
Currently established calls between operator positions are represented by a set
of calls. A call consists of its source src, the initiator of the call, and the target
operator position trg.
Calls set of Call
Call record src : Nat
trg : Nat
External parties calling via radio or phone interfaces are not included in this
model. The reason is that it is assumed that external calls have the same behav-
ior as calls between operator positions with respect to roles. Consequently, only
operator to operator calls are used for testing the roles functionality. This unifor-
mity hypothesis on calls demonstrates that an abstract view on a system already
establishes a testing strategy.
8.1. ROLES IN VOICE COMMUNICATION 101
8.1.2 Operations
In the following the functionalities under consideration are specied formally. Each
of the operations depends on the operator position at which the operation is initi-
ated. Auxiliary variables are used for storing the parameter values that are explicitly
chosen by a user. Assertions further restrict a users choice to cases, where a function
may be applied successfully.
A role request ReqRole is initiated by a users selection of an operator position o
and a role r. A succeeding assertion further restricts the possible choices to operator
positions that have less than 16 roles associated and that are enabled. Furthermore,
for a requested role, the corresponding function group must be installed on the
chosen operator position. If this holds, then the system demonically updates the
state variable ops such that the requested role is associated to the selected operator
position. This implies that the role may be removed from another operator position.
The syntax of accessing a eld in an array and in a record is the same. For
example, ops[o][roles] denotes the roles eld in the oth entry of variable ops. The
operator # denotes the cardinality of a set.
ReqRole o, r: = o
, r
[ 1 o
opmax 1 r
rmax;
#ops[o][roles] < 16;
ops[o][enabled];
roles[r][fg] ops[o][fgs];
[ops: = ops
[ op (r ops
[op][roles] op = o)
(r , ops
[op][roles] op ,= o)]
Releasing a role, RelRole, is specied by rst selecting an appropriate operator
position o and a role r, and then removing the role from the operator position. The
users (angelic) selection is restricted to operator positions that have the role to be
released.
RelRole o, r: = o
, r
[ 1 o
opmax 1 r
rmax;
ops[o][enabled];
r ops[o][roles];
[ops: = ops
[ op r , ops
[op][roles] ]
A Call to another operator position via the phone number of a role, involves the
selection of the calling operator position o1 and the role number n. A selection is
valid if there exists both, a role with this number, and an enabled and not calling
target operator position with this role associated. Furthermore, the calling operator
position must not take part in a call already. For valid input, the system successfully
selects the target operator position o2 and establishes a call to o2.
Call o1, n: = o1
, n
[ 1 o1
opmax;
ops[o1][enabled];
,c calls c[src] = o1 c[trg] = o1;
r, op roles[r][num] = n
r ops[op][roles]
ops[op][enabled]
c calls c[trg] ,= op c[src] ,= op;
[o2: = o2
[ r roles[r][num] = n r ops[o2
][roles]];
[calls: = calls
[ calls
= calls Call (o1, o2)]

A HangUp disconnects a call at an operator position o. A precondition is that a
call is established previously. The remove of a call is specied implicitly, by stating
that a call is removed in which o is either the source or the target.
HangUp o: = o
[ 1 o
opmax;
c calls c[src] = o c[trg] = o;
[calls: = calls
[ c calls
= calls\c (c[src] = o c[trg] = o)]

EnableOp enables a selected operator position, if it has been disabled in advance.
EnableOp o: = o
[ 1 o
opmax;
ops[o][enabled];
ops[o][enabled]: = T
DisableOp disables an enabled operator position. As a consequence, all roles of
the operator position are released.
DisableOp o: = o
[ 1 o
opmax;
ops[o][enabled];
ops[o][enabled]: = F;
ops[o][roles]: = ;
With these denitions our abstract view on the functionality of the voice commu-
nication system is precisely dened. However, in the next section it will be seen that
for testing purposes the possible user interactions of the system should be explicitly
specied.
8.2 Tester-System Interactions
In the previous section, the operations of the system have been formally specied.
Most of the formal methods, like VDM and Z, stop the specication process at this
point. However, in Back and von Wrights contract language the behavior of the
user, her interactions, can be dened more precisely.
8.2. TESTER-SYSTEM INTERACTIONS 103
One possibility would be to include the operations in an iterative choice state-
ment as has been done in the process scheduler example of Chapter 6. However, from
a testers perspective this would not be the most elegant choice for this example.
The dierence to the process scheduler is that the communication system has a
distributed interface the operator positions. Consequently, a single tester has to
move locally when operations from a dierent operator position are to be initiated.
Therefore, it makes sense to
separate the selection of operator positions from the other user interactions;
design test scenarios such that operations initiated at one operator position
are grouped together.
The result of this considerations is the specication of the voice communication
system below. In this contract, the selection of the operator position is separated
from the other selections and is included in an enclosing iterative choice statement.
Furthermore, the operations have been divided into their angelic contract part
and their demonic contract part. The angelic part of an Operation is denoted by
Operation
a
and represents the users inputs (angelic updates) and his constraints
(assertions). The demonic part is denoted by Operation
d
and represents the reactions
of the system (demonic updates). This separation makes the contract more readable,
but has its advantages for deriving the test scenarios, too, as will be seen later.
VCS ops, roles, calls: = ops
, roles
, calls
[ ops
[roles] = calls
= ;
do true : :o: = o
[ 1 o opmax;
do ops[o][enabled] #ops[o][roles] < 16 : : ReqRole
a
; ReqRole
d
ops[o][enabled] ops[o][roles] ,= : : RelRole
a
; RelRole
d
ops[o][enabled]
(c calls c[src] = o c[trg] = o) : : Call
a
; Call
d
ops[o][enabled]
(c calls c[src] = o c[trg] = o) : : HangUp
d
ops[o][enabled] : : Enable
d
ops[o][enabled] : : Disable
d
od
od
The contract of the VCS can be interpreted from a testers point of view as
follows:
1. In advance of the actual testing, the testing environment has to be set up. This
involves the installation of function groups at operator positions, the entering
of role data (function group and number). Furthermore, certain operator
positions may be disabled in advance. These data can be t into the system
over a special working place for parameterizing the VCS. This initialization
is modeled by an angelic update with the constraints that no calls can be
established in advance, and no roles can be previously associated.
2. The tester selects an operator position and initiates one of the following oper-
ations:
(a) If an enabled operator position has less than 16 roles associated, then a
new role may be requested:
ReqRole
a
r: = r
[ 1 r
rmax roles[r
][fg] ops[o][fgs]
ReqRole
d
[ops: = ops
[ op (r ops
[op][roles] op = o)
(r , ops
[op][roles] op ,= o)]
(b) If an enabled operator position has at least one role associated, then a
role can be released:
RelRole
a
r: = r
[ 1 r
rmax r
ops[o][roles]
RelRole
d
[ops: = ops
[ op r , ops
[op][roles]]
(c) If an enabled operator position is not already involved in a call, then a
call can be initiated:
Call
a
n: = n
[ r, op roles[r][num] = n
r ops[op][roles]
ops[op][enabled] c calls c[trg] ,= op c[src] ,= op
Call
d
[o2: = o2
[ r roles[r][num] = n r ops[o2
][roles]];
[calls: = calls
[ calls
= calls Call (o, o2)]

(d) If an enabled operator position is involved in a call, then the operator
can stop calling by hanging up. Here, no further selection of the tester is
needed. Therefore, a demonic update is sucient:
HangUp
d
[calls: = calls
[ c calls
= calls\c (c[src] = o c[trg] =

o)]
(e) A disabled operator position may be enabled:
EnableOp
d
ops[o][enabled]: = T
(f) An enabled operator position may be disabled:
DisableOp
d
ops[o][enabled]: = F; ops[o][roles]: = ;
(g) The tester may skip the operation selection and may choose a new oper-
ator position for entering his commands.
3. The tester decides to stop testing.
8.3. DNF-PARTITIONING 105
8.3 DNF-Partitioning
As in the process scheduler example, the next step is the partitioning of the inter-
actions. This can be done by applying the rule for generating a disjunctive normal
form (DNF).
However, in this contract none of the demonic updates includes cases such that
the DNF rule can be applied. The only that is included in ReqRole
d
is inside a
-quantier expression which cannot be split apart.
The only partitioning that can take place is the partitioning of the guard expres-
sion for hanging up. There, the existential quantied expressions can be split into
two separate expressions:
(c calls c[src] = o c[trg] = o)
= by splitting the quantier
((c calls c[src] = o) (c calls c[trg] = o))
= by DNF-rewriting

= by the invariant: only one call per operator position
(c calls c[src] ,= o c[trg] = o)
(c calls c[src] = o c[trg] ,= o))
((c calls c[src] = o c[trg] = o))

= by the invariant: self-calls are not possible
(c calls c[trg] = o)
(c calls c[src] = o))

This gives two partitions for hanging up. In the rst, the called operator hangs
up. In the second, the calling operator hangs up.
Note that the guard for the call operation cannot be partitioned since
c calls c[src] = o c[trg] = o c calls c[src] ,= o c[trg] ,= o
Consequently, the partitioned contract VCS
is only modied with respect to the

HangUp operation.
VCS
ops, roles, calls: = ops
, roles
, calls
[ ops
[roles] = calls
= ;
do true : :o: = o
[ 1 o opmax;
do ops[o][enabled] #ops[o][roles] < 16 : : ReqRole
a
; ReqRole
d
ops[o][enabled] ops[o][roles] ,= : : RelRole
a
; RelRole
d
ops[o][enabled]
(c calls c[src] = o c[trg] = o) : : Call
a
; Call
d
ops[o][enabled] (c calls c[src] = o) : : HangUp
d
ops[o][enabled] (c calls c[trg] = o) : : HangUp
d
ops[o][enabled] : : Enable
d
ops[o][enabled] : : Disable
d
od
od
The contract VCS is an example of a class of systems where domain partitioning
is trivial and thus not sucient. The complex guards indicate that, here, the chal-
lenging task is the sequencing of the tests the test scenario synthesis. As explained
in the previous chapter, our strategy starts with the search for valid compositions.
8.4 Scenario Synthesis
8.4.1 Interaction Compositions
Our test synthesis method for test scenarios is based on compositions (see Chap-
ter 6). These compositions are pairwise sequential compositions of interactions.
Their construction plays three dierent roles:
1. compositions are atomic elements for nding possible scenarios;
2. the compositions highlight which interactions are more interesting than others
with respect to testing;
3. by searching valid compositions, the preconditions and guards of the interac-
tions of a contract are validated.
In the following, a set of compositions for each of the interactions in the VCS
contract is presented. Since the contract consists of 7 basic operations on a certain

8.4. SCENARIO SYNTHESIS 107
operator position, theoretically 7 7 = 49 compositions might exist. However, for
testing only a reduced number of compositions is necessary. The following type of
compositions can be skipped:
non-terminating composition: the weakest precondition of the sequential com-
position is false. Thus for an arbitrary predicate q, (S
1
; S
2
).q = false. This
represents a non-feasible test sequence.
independent composition: the weakest precondition of the sequential composi-
tion S
1
; S
2
equals the conjunction of the weakest preconditions of S
1
and S
2
.
Thus, for an arbitrary predicate q, (S
1
; S
2
).q = S
1
.q S
2
.q. This represents a
test sequence, where S
1
does not contribute in setting up the systems state,
such that S
2
can be invoked. These compositions are not interesting from a
testers point of view.
The following compositions are grouped by the second operation.
ReqRole
Five operations may precede ReqRole and yield independent compositions. A pre-
ceding HangUp is not interesting for testing since it constitutes an independent
composition.
(C1) ops[o][enabled] ops[o][roles] < 15 rmax 2
r1, r2 r1 ,= r2 roles[r1][fg] ops[o][fgs] roles[r2][fg] ops[o][fgs]
r1 , ops[o][roles];
ReqRole
a
; ReqRole
d
;
ReqRole
a
; r , ops[o][roles]; ReqRole
d
(C2) ops[o][enabled] ops[o][roles] < 16
r roles[r][fg] ops[o][fgs];
ReqRole
a
; ReqRole
d
;
ReqRole
a
; r ops[o][roles]; ReqRole
d
(C3) ops[o][enabled] r r ops[o][roles];
RelRole
a
; RelRole
d
;
ReqRole
a
; ReqRole
d
(C4) ops[o][enabled] (r, op op ,= o r ops[op][roles] ops[op][enabled]
c calls c[trg] ,= op c[src] ,= op)
roles[r][fg] ops[o][fgs] #ops[o][roles] < 16;
Call
a
; Call
d
;
ReqRole
a
; roles[r][num] = n; ReqRole
d
(C5) ops[o][enabled] r roles[r][fg] ops[o][fgs]
#ops[o][roles] < 16;
Enable
d
;
ReqRole
a
; ReqRole
d
ad C1: This constitutes the request of two dierent roles. The additional assertion
r , ops[o][roles] after the second ReqRole
a
reduces the possibilities of the
testers role selection to new roles. This is the reason why the operations
have been split into an angelic and demonic part. This additional assertion is
consistent with our approach of abstraction, since reducing the choices of the
angel is abstraction.
The precondition states that the operator position (OP) must be enabled; two
further roles can be added to the OP; at least two roles exist in the system;
two roles exists that have the permission to be added to the OP and one role
is not added yet.
ad C2: This means requesting the same role twice. The additional assertion r
ops[o][roles] after the second ReqRole
a
reduces the possibilities of the testers
role selection to a role that has been already added to the OP.
The precondition states that the operator position must be enabled; one fur-
ther role can be added to the OP; one role exists that can be added to the
OP.
ad C3: For releasing a role and then requesting one, at least one role must be
associated to the enabled OP previously.
ad C4: This represents the interesting case, where during a call the role of the
partner is requested. This raises the question if this should be possible an
example for the validation role of the composition synthesis process.
The precondition states that the OP must be enabled; a role associated to a
second enabled OP exists which is not involved in a call and this role can be
requested.
ad C5: This composition results in a case where the rst role is added to the OP
after its enabling. For enabling an OP it must be disabled in advance, fur-
thermore the preconditions for requesting a role must hold.
RelRole
Only three operations can be composed with RelRole to independent and termi-
nating compositions. Enabling and disabling operations cannot directly precede
RelRole. A preceding HangUp and Call result in an independent composition. Since
Call is interesting from a validation point of view its composition is included, the
question is: Is it possible to release a role during a call?
(C6) ops[o][enabled] r roles[r][fg] ops[o][fgs]
ops[o][roles] = ;
ReqRole
a
; ReqRole
d
;
RelRole
a
; RelRole
d
(C7) ops[o][enabled] #ops[o][roles] 2;
RelRole
a
; RelRole
d
;
RelRole
a
; RelRole
d
(C8) ops[o][enabled] (r, op op ,= o r ops[op][roles] ops[op][enabled]
c calls c[trg] ,= op c[src] ,= op)
ops[o][roles] ,= ;
Call
a
; Call
d
;
RelRole
a
; roles[r][num] = n; RelRole
d
ad C6: This composition represents the case, where a certain role is requested and
afterwards released. Here, the precondition is further restricted (abstracted)
to guarantee that the same role is requested and afterwards released: No roles
should be associated previously. The more general case of requesting one role
and releasing another would be less interesting.
Therefore, the precondition states that the OP must be enabled; a role with
the appropriate function group exists; and the set of associated roles is empty.
ad C7: This is a test for releasing two roles successively. A precondition is that
two roles are associated to the enabled OP.
ad C8: This composition represents a test for checking if releasing a role is possible
during a call. The complex precondition shows that this composition is an
independent one.
Call
The two interesting compositions for testing the call functionality, is to hang up and
then to initiate a call. For hanging up we have identied two partitions. The other
possible preceding operations are not very interesting for testing due to their high
independence.
(C9) ops[o][enabled] (c calls c[src] = o);
HangUp
d
;
Call
a
; Call
d
(C10) ops[o][enabled] (c calls c[trg] = o) (op ops[op][roles] ,= );
HangUp
d
;
Call
a
; Call
d
ad C9: This is the case, where the OP has called another one, hangs up, and
again initiates a call. Due to the composition the precondition becomes rather
simple.
The precondition says that a call must exist that has been initiated by the
OP, and OP must be enabled.
ad C10: This is the case, where the OP has been called by another one, the tester
at the called OP hangs up, and again initiates a call.
Here, the precondition in addition demands that another OP exists that has
a role associated. It might be the case that the other OP, which previously
called, has meanwhile released its roles.
HangUp
The only interesting and possible preceding operation for hanging up at an operator
position is the initiation of a call. The other partition of HangUp can only be reached
by rst changing the actual operator position.
(C11) ops[o][enabled] (r, op r ops[op][roles] ops[op][enabled]
c calls c[trg] ,= op c[src] ,= op);
Call
a
; Call
d
;
HangUp
d
ad C11: This represents a test-case for testing the hanging up during an initiated
call. The precondition is the precondition of Call .
EnableOp
For enabling an operator position, the only possible preceding operation at the same
operator position is its disabling.
(C12) ops[o][enabled];
Disable
d
;
Enable
d
ad C12: Disabling and then enabling demands that the operator position is rst
enabled.
DisableOp
Disabling an operator position is trivial in the sense that it is possible after each
preceding operation. This operation builds independent compositions with every
other operation. The question the experts have to answer is: Is this a good de-
sign decision? Again, the validation process of the specication (design) has been
improved by raising this kind of questions.
8.4.2 Selecting Test Scenarios
Based on the compositions above, dierent strategies for selecting the scenarios are
possible. Examples are:
1. test all (partitioned) operations once: a maximum of one scenario per available
operation is needed;
2. test all compositions once: a maximum of one scenario per composition;
3. test each composition with all possible scenarios between a given operation of
interest and the composition;
4. test each composition with all possible preceding scenarios: maximum number
of test-cases;
5. test each operation with all possible preceding scenarios including independent
compositions: maximum number of test-sequences.
The strategies are ordered by the increasing number of needed scenarios for covering
these criteria. The additional tests of independent compositions in the last strategy
might detect unintended feature interactions [MZ94].
In the following, the test scenarios for the rst (Strategy 1) and second strategy
(Strategy 2) are presented.
Strategy 1
The following six test scenario contracts have been developed following Strategy 1.
This strategy only demands one test for each partitioned operation.Therefore, the
simplest compositions can be chosen to set up a state such that the operation to be
tested can be invoked.
Where compositions have been used, a scenario is annotated by the identier of
the composition. The process follows a backtracking approach where the synthesis
process starts from the operation under test and ends at the parameterization of the
system.
In the test-cases, certain input selections have been further abstracted to con-
crete test-data. This has been done by adding an additional assertion after an
angelic update. In the contract language, the sequential composition of an angelic
update and an assertion R; p reduces the range (nal states) of the relation R
to (ran.R p).
Scenarios for Testing Each Operation
(T1) ops, roles, calls: = ops
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; ReqRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
(C6) ReqRole
a
; ReqRole
d
;
RelRole
a
; RelRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[2][enabled]
roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; ReqRole
d
;
o: = o
[ o
= 2;
Enable
d
;
Call
a
; Call
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[2][enabled] roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; ReqRole
d
;
o: = o
[ o
= 2;
Enable
d
;
(C11) Call
a
; Call
d
;
HangUp
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[2][enabled]
roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; ReqRole
d
;
o: = o
[ o
= 2;
Enable
d
;
(C11) Call
a
; Call
d
;
o: = o
[ o
= 1;
HangUp
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled];
o: = o
[ o
= 1;
Enable
d
;
Disable
d
It should be noted that the test-case (T1) is included in (T2), and (T3) is included
in (T4). Therefore, the two test-cases (T1) and (T3) can be skipped for testing all
operations once. They are redundant.
It should be obvious that all these tests have to be carried out on all operator
position in the voice communication system.
Strategy 2
Next, the scenarios for covering all of the 12 compositions are presented. Here, we
will skip those scenarios that are detected to be redundant during the development
of the test-cases, i.e. scenarios that would test compositions that are already covered
by other test scenarios.
Scenarios for Testing Each Composition
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] roles
[1][fg] ops
[1][fgs]
roles
[2][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
(C1) ReqRole
a
; r = 1; ReqRole
d
;
ReqRole
a
; r = 2; ReqRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
(C2) ReqRole
a
; r = 1; ReqRole
d
;
ReqRole
a
; r = 1; ReqRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
(C6) ReqRole
a
; r = 1; ReqRole
d
;
(C3) RelRole
a
; r = 1; RelRole
d
;
ReqRole
a
; r = 1; ReqRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[2][enabled]
roles
[1][fg] ops
[1][fgs] roles
[1][fg] ops
[2][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; r = 1; ReqRole
d
;
o: = o
[ o
= 2;
Enable
d
;
(C4) Call
a
; roles[1][num] = n; Call
d
;
ReqRole
a
; r = 1; ReqRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] roles
[1][fg] ops
[1][fgs]
roles
[2][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
(C1) ReqRole
a
; r = 1; ReqRole
d
;
ReqRole
a
; r = 2; ReqRole
d
;
(C7) RelRole
a
; r = 1; RelRole
d
;
RelRole
a
; r = 2; RelRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[1][enabled]
roles
[1][fg] ops
[1][fgs] roles
[2][fg] ops
[2][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; r = 1; ReqRole
d
;
o: = o
[ o
= 2;
(C5) Enable
d
;
ReqRole
a
; r = 2; ReqRole
d
;
(C8) Call
a
d
;
RelRole
a
; roles[r][num] = n; RelRole
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[2][enabled]
roles
[1][fg] ops
[1][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; r = 1; ReqRole
d
;
o: = o
[ o
= 2;
Enable
d
;
(C11) Call
a
d
;
(C9) HangUp
d
;
Call
a
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled] ops
[2][enabled]
roles
[1][fg] ops
[1][fgs] roles
[2][fg] ops
[2][fgs];
o: = o
[ o
= 1;
(C5) Enable
d
;
ReqRole
a
; r = 1; ReqRole
d
;
o: = o
[ o
= 2;
(C5) Enable
d
;
ReqRole
a
; r = 2; ReqRole
d
;
(C11) Call
a
; roles[1][num] = nCall
d
;
o: = o
[ o
= 1;
(C10) HangUp
d
;
Call
a
d
, roles
, calls
[ calls
= ops
[roles] =
ops
[1][enabled];
o: = o
[ o
= 1;
(C12) Enable
d
;
Disable
d
These nine test scenarios include all twelve compositions that have been identied
to be of interest for testing. Test-case (T5) includes composition (C1) and thus test-
case (T1) might be skipped. However, a tester may want to check the results of the
compositions at the end of a test-case. Then, test-case (T5) should be included and
the other skipped test-cases for composition (C5), (C6), and (C11) may be added.
Depending on the need and on the test hypothesis made, more tests can be
designed. The following possibilities for extending the set of test-cases can be con-
sidered:
More scenarios may be developed following the other scenario strategies.
The scenarios should be tested on all operator positions in the voice commu-
nication system.
Dierent roles than the rst two might be chosen.
Additionally, extreme values might be chosen. Examples are maximum and
minimum numbers of roles and function groups at an operator position.
Additional mutation tests might be invented from the specication. Then, it
can be checked if the test scenarios would detect the specication mutations,
by showing that the derived scenarios are not abstractions of the specication
mutants.
In this chapter, an important functionality of a real voice communication system
has served as a case study for illustrating our new test approach based on formal
contracts. An abstract model of the operations for role management has been con-
structed. Next, a contract that denes the possible tester-system interactions has
been presented, and then test-scenarios have been derived (abstracted) from this
formal contract. Therefore, the contract had been rst partitioned and then inter-
action compositions have been developed. From these compositions, a set of test
scenarios has been constructed.
The example demonstrates that a rather complex functionality, as the manage-
ment of roles in a voice communication system, can be specied in a rather coarse
contract. The key to this simplicity of the contract is abstraction abstraction to
a level needed for system testing, not for testing the specication.
The roles management functionality has been part of our second experiment
on light-weight approaches as described in Chapter 4. The executable VDM-SL
specication turned out to be much more complex (30 pages). One reason is the
large set of auxiliary functions that has to be dened for manipulating the state
space. In our contract, these state updates are specied implicitly by relations
expressed by quantiers that are not executable but more compact.
The second reason, why the implicit model of this chapter is less complex, is more
trivial: Requirements concerning exceptional behavior and architectural details have
been skipped. Mathematically, this means that data-abstraction has been applied
for modeling the central behavior rst. And the test scenarios of the example show
that this central behavior is complex enough. It is our strong opinion that, rst,
the most important parts of a system have to be understood, validated and tested.
Then, the view on a system (its model) may be rened further and more details
can be considered. The renement calculus provides the theory for this step-wise
top-down approach.
Renement is an important aspect in our new test approach. In our method
the use of the renement calculus supports the data-renement of a specication
together with its test-cases, since both are contracts. This is in contrast to test-
approaches that use nite state machines (FSM), where renement is rather dicult.
Derrick and Boiten showed that states and transitions in the FSM might disappear
or are to be added under data-renement [DB99]. In our approach the interaction
compositions and thus the scenarios can be data-rened quite straight-forwardly.
See [BMvW99] for an example of data-renement of an interactive system in the
renement calculus.
Finally, the validation aspect of constructing the compositions should be pointed
out. Several missing preconditions have been detected, due to this scenario synthesis
method. Further questions have been raised by considering, if a possible composi-
tion of interactions does indeed reect the intended behavior. This questions are
easily communicated to a customer, since they are caused by concrete examples of
behavior.
Chapter 9
Concluding Remarks
In this thesis three formal development techniques have been examined regarding
their support in the testing of computer-based systems: Light-weight approaches to
formal methods, the renement calculus, and data-reication techniques. As a re-
sult, novel testing techniques have been developed. In this chapter some concluding
remarks regarding this work are provided. This is done by rst reviewing the results
of this research. This is followed by a presentation of areas of future work.
9.1 Research Results
The research of this thesis resulted in three new approaches to testing:
1. a light-weight approach to specication based testing that has been suc-
cessfully applied in industry,
2. an abstraction-based test synthesis method that uses the mathematical
framework of the renement calculus,
3. a method for using abstract formal specications as automated test-oracles.
In the following the contributions of these test methods are discussed in more detail.
9.1.1 Light-Weight Approach
The presented light-weight approach uses abstract VDM prototypes for assessing
and designing test-cases. It is light-weight in the sense that no proofs are carried
out in this method. Two industrial experiments in the domain of air-trac control
demonstrated the relevance of our method.
This part of the work is an application of an existing light-weight formal method
to testing. However, the strong emphasis of the approach on the validation of
both, the requirements as well as the test-design is new. Especially, to our present
knowledge, it is the rst time that this light-weight method has been applied in two
119
120 CHAPTER 9. CONCLUDING REMARKS
industrial experiments and that detailed empirical data have been collected data
about the eorts and eciency of the method. The results of these experiments
demonstrated the main advantages of our method:
the formal specication of the requirements is a catalyst in validation in the
sense that many questions are raised during the modeling process: Exper-
iment I detected 64 problems in the given requirement documentation and
Experiment II found 33 problems;
previously designed test-cases can be improved: Experiment I uncovered an
unacceptable low coverage of appr. 80 % in the existing test-cases. Experiment
II found that appr. 25 % of the existing test-cases were erroneous;
engineers like the availability of test-cases to validate our formal specications.
Having an abstract prototype, an engineer or customer is not forced to under-
stand the formal specication. The test-cases demonstrate the consequences
of the model by examples;
the eorts are acceptable low.
However, the experiments also showed that more can be achieved. We learned that
executable specications have the disadvantage that non-experienced users tend to,
and sometimes are forced to, think too operationally. Consequently, the specica-
tions lack abstraction and incorporate too many design details.
9.1.2 Abstraction-Based Test Synthesis
This part of the work resulted in the most important contributions of this thesis.
It has been observed that test-cases are abstractions of formal specications. This
lead to a new test synthesis approach based on program synthesis. In this thesis
it is the rst time that the renement calculus is applied to the problem of test
synthesis. The result is an abstraction calculus for testing in order to calculate test-
cases according to a testing strategy. New formal synthesis rules for several test
strategies have been dened. This research has identied and proved the following
facts:
test-cases can be viewed as formal contracts,
test-cases are abstractions of formal specications,
program synthesis techniques can be applied to the test-case synthesis problem,
the renement calculus of Back and von Wright provides a powerful mathe-
matical framework for dening an abstraction calculus for testing,
simple input-output test-cases as well as complex test scenarios can be treated
by an abstraction approach to testing,
9.2. FUTURE WORK 121
rather dierent testing strategies can be formulated as formal abstraction rules:
partition testing, structural testing, mutation testing, and scenario testing
(test sequencing).
It is the authors opinion that an abstraction calculus for testing, provides deeper
insight into the methods of test-case derivation. Especially, the understanding of
new black-box testing strategies is supported by reasoning about the synthesis pro-
cess. The examined testing strategies showed that even existing techniques reveal
interesting properties. New test-case generation techniques may be the result.
For example, the presented synthesis rules for scenario calculation dene an
alternative method for nding sequences of interactions. In contrast to nite state
machine (FSM) based approaches, the focus is on nding possible compositions
of interactions. Which compositions are possible is determined by the abstraction
rules.
9.1.3 Formal Specications as Oracles
In this part of the research, the possibilities to automate the test-result verication
through formal requirement specications have been explored. More precisely, the
formal method VDM (Vienna Development Method) serves to demonstrate that
abstract requirement models can be used as test oracles for concrete software. It
is shown that techniques of VDMs invent and verify development method can be
adapted to set up a test environment.
Our ideas dier to most of the previous work on test-result verication in two
aspects:
a possibly non-executable specication may serve as a rather general oracle
for checking the test results. Non-deterministic test results can be handled as
well.
the abstraction problem is solved by using VDMs data-reication techniques
to relate the test-data for the implementation to the abstract specication
oracle.
9.2 Future Work
Naturally, further extensions to our new abstraction calculus for test-synthesis must
be included in the list of future work. More synthesis rules can be found that
represent other test strategies that are not considered in this thesis. This includes,
strategies for testing several forms of iterations, procedure calls and data-types.
One could think of a calculus for design-patterns that provides a testing strat-
egy for each pattern. Integration tests could be designed for certain software ar-
chitectures, like for pipe-and-lters, object-oriented architectures and black-board
systems.
122 CHAPTER 9. CONCLUDING REMARKS
Furthermore, the test calculus could be extended to handle concurrency. The
framework of action systems [BS94], an extension of the renement calculus for
describing reactive systems, could be used for dening new rules.
The synthesis rules could be rened such that they dene a structural operational
semantics of the test-selection strategy. Like for programming languages it should
be possible to derive test-case generators out of this operation semantics denition.
Furthermore, program transformation tools could be applied to derive the test-cases
semi-automatically.
The development of future tools that support our test-synthesis process is one
of our central interests. We envisage the use of a theorem prover to calculate the
scenarios. The author has already started rst experiments with constructive proofs
in Isabelle [Pau94]. Alternatively, model-checkers may be used for checking the
abstraction relations. However, for the application in industry new interfaces are
needed that hide the complex interior of a theorem prover. The advantage of our
method is that a test-case generator based on our approach can be certied by
proving that the correct abstractions are calculated.
We hope that the presented work stimulates further research on test-synthesis
based on other program-synthesis approaches. Especially, the application of program
synthesis and transformation tools for testing could be a promising topic of future
research.
9.3 Epilogue
The introduction started with the sentence Software testing is a challenging task.
If testing is taken seriously, it could be as challenging as the development of the pro-
gram itself. Therefore, the test design deserves the same level of rigor than software
design. In this thesis, we have provided new methods that facilitate this aim. The
view that test-cases are abstractions of specications should help to further unify
eorts in the testing and formal methods community. We conclude by expressing
the hope that in future more people will be aware of the central property:
test-cases _ specication _ implementation
Bibliography
[Abr96] Jean-Raymond Abrial. The B-Book: Assigning Programs to Mean-
ings. Cambridge University Press, Trumpington Street, Cambridge
CB2 1RP, Great Britain, 1996.
[Aer98] Lionel Van Aertryck. Une methode et un outil pour laide à la
generation de jeux de tests de logiciels. PhD thesis, Universite de
Rennes, January 1998.
[AGA00] Bernhard K. Aichernig, Andreas Gerstinger, and Robert Aster. Formal
specication techniques as a catalyst in validation. In Proceedings of
the 5th Conference on High-Assurance Software Engineering, 15th17th
November 2000, Albuquerque, New Mexico, USA. IEEE, 2000.
[AH00] S. Antoy and D. Hamlet. Automatically checking an implementation
against its formal specication. IEEE Transactions on Software Engi-
neering, 26(1):5669, 2000.
[Aic98] Bernhard K. Aichernig. Automated requirements testing with abstract
oracles. In ISSRE98: The Ninth International Symposium on Soft-
ware Reliability Engineering, Paderborn, Germany, pages 2122, IBM
Thomas J.Watson Research Center, P.O.Box 218, Route 134, York-
town Heights, NY, USA, November 1998. Ram Chillarege. ISBN 3-00-
003410-2.
[Aic99a] Bernhard K. Aichernig. Automated black-box testing with abstract
VDM oracles. In John Fitzgerald and Peter Gorm Larsen, editors,
Workshop Materials: VDM in Practice!, Part of the FM99 World
Congress on Formal Methods, Toulouse. Springer, 1999.
[Aic99b] Bernhard K. Aichernig. Automated black-box testing with abstract
VDM oracles. In M. Felici, K. Kanoun, and A. Pasquini, editors, Com-
puter Safety, Reliability and Security: proceedings of the 18th Interna-
tional Conference, SAFECOMP99, Toulouse, France, September 1999,
volume 1698 of Lecture Notes in Computer Science, pages 250259.
Springer, 1999.
123
124 BIBLIOGRAPHY
[Aic99c] Bernhard K. Aichernig. Executable specications in software reliability
engineering. In 2nd Workshop on Formal Descriptions and Software
Reliability (FDSR), Boca Raton, Florida, 31st of October 1999.
[Aic01] Bernhard K. Aichernig. Test-case calculation through abstraction. In
Proceedings of Formal Methods Europe 2001, FME 2001, March 1216
2001, Berlin, Germany, Lecture Notes in Computer Science. Springer
Verlag, 2001. To be published.
[AK00] Bernhard K. Aichernig and Reinhold Kainhofer. Modeling and validat-
ing hybrid systems using VDM and Mathematica. In C.Michael Hol-
loway, editor, Lfm2000, Fifth NASA Langley Formal Methods Work-
shop, Williamsburg, Virginia, June 2000, number CP-2000-210100 in
Conference Publication, pages 3546. NASA, June 2000.
[ALR98] Sten Agerholm, Pierre-Jean Lecoeur, and Etienne Reichert. Formal
specication and validation at work: A case study using VDM-SL.
In Proceedings of Second Workshop on Formal Methods in Software
Practice, Florida, Marts. ACM, 1998.
[BCGM00] Simon Burton, John Clark, Andy Galloway, and John McDermid. Au-
tomated V & V for high integrity systems, a targeted formal methods
approach. In C.Michael Holloway, editor, Lfm2000, Fifth NASA Lan-
gley Formal Methods Workshop, Williamsburg, Virginia, June 2000,
number CP-2000-210100 in Conference Publication. NASA, June 2000.
[Be80] D. Bjrner and O. Oest (eds.). Towards a Formal Description of Ada,
volume 98 of Lecture Notes in Computer Science. Springer-Verlag,
1980.
[Bec99] Kent Beck. Extreme Programming Explained: Embrace Change.
Addison-Wesley, 1999.
[Bee86] D. Beech, editor. Concepts in User Interfaces: A (VDM) Reference
Model for Command and Response Languages, volume 234 of Lecture
Notes in Computer Science. Springer-Verlag, 1986.
[Bei90] Boris Beizer. Software Testing Techniques. Van Nostrand Reinhold,
New York, 2nd edition, 1990.
[Bei95] Boris Beizer. Black-Box Testing: Techniques for Functional Testing of
Software and Systems. John Wiley & Sons, Inc., 1995.
[BF86] R.E. Bloomeld and P.K.D. Froome. The application of formal methods
to the assessment of high integrity software. IEEE Transactions on
Software Engineering, SE-12(9):988993, September 1986.
BIBLIOGRAPHY 125
[BFL
+
94] Juan C. Bicarregui, John S. Fitzgerald, Peter A. Lindsay, Richard
Moore, and Brian Ritchie. Proof in VDM: A Practitioners Guide.
FACIT. Springer-Verlag, 1994.
[BFL96] T.M. Brookes, J.S. Fitzgerald, and P.G. Larsen. Formal and informal
specications of a secure system component: Final results in a com-
parative study. In Marie-Claude Gaudel and Jim Woodcock, editors,
FME96: Industrial Benet and Advances in Formal Methods, pages
214227. Springer-Verlag, March 1996.
[BG81] Timothy A. Budd and Ajei S. Gopal. Program testing by specication
mutation. Computer Program Testing, pages 129148, 1981.
[BGL
+
00] S. Bensalem, V. Ganesh, Y. Lakhnech, C. Munoz, S. Owre, H. Ruesz,
J. Rushby, V. Rusu, N. Shankar, E. Singerman, and A. Tiwari. An
overview of SAL. In C.Michael Holloway, editor, Lfm2000, Fifth
NASA Langley Formal Methods Workshop, Williamsburg, Virginia,
June 2000, number CP-2000-210100 in Conference Publication, pages
187196. NASA, June 2000.
[BGM91] G. Bernot, M.-C. Gaudel, and B. Marre. Software testing based on for-
mal specications: A theory and a tool. Software Engineering Journal,
6(6):387405, November 1991.
[BGM92] G. Bernot, M.-C. Gaudel, and B. Marre. A formal approach to soft-
ware testing. In Maurice Nivat, Charles Rattray, and Teodor Rus,
editors, Proceedings of the Second International Conference on Alge-
braic Methodology and Software Technology, Workshops in Computing,
pages 243253, London, May 2225 1992. Springer-Verlag.
[Bj89] Dines Bjrner. Specication and transformation, towards a meaning
of M in VDM. Lecture notes to IFIP TC2/WG2.2 South America
Tutorial: Formal Description of Programming Concepts, April 1989,
February 1989. A revised and expanded version of a TAPSOFT article.
[BMvW99] Ralph Back, Anna Mikhajlova, and Joakim von Wright. Reasoning
about interactive systems. In Jeannette M. Wing, Jim Woodcock, and
Jim Davies, editors, FM99 Formal Methods, World Congress on
Formal Methods in the Development of Computing Systems, Toulouse,
France, September 1999, Proceedings, Volume II, volume 1709 of Lec-
ture Notes in Computer Science. Springer, 1999.
[Bod88] Grahan Boddy. The use of VDM within the Alvey Flagship Project. In
R. Bloomeld, L. Marshall, and R. Jones, editors, VDM 88: VDM
The Way Ahead, pages 153166. VDM-Europe, Springer-Verlag,
September 1988.
126 BIBLIOGRAPHY
[Boe75] B.W. Boehm. The high cost of software. In E. Horowitz, editor, Practi-
cal Strategies for Developing Large Software Systems. Addison-Wesley,
Reading MA, 1975.
[Boe81] Barry W. Boehm. Software Engineering Economics. Prentice-Hall,
Englewood Clis, NJ, 1981.
[Boe86] Barry W. Boehm. A spiral model of software development and en-
hancements. ACM Software Engineering Notes, 11(4):2242, 1986.
[Bri89] E. Brinksma. A theory for the derivation of tests. In Peter H.J. van Eijk,
Chris A. Vissers, and Michel Diaz, editors, The Formal Description
Technique LOTOS: Results of the ESPRIT/SEDOS Project, pages 235
247. Elsevier Science Publishers North-Holland, 1989.
[Bro75] Frederick P. Brooks. The Mythical Man-Month, Essays on Software
Engineering. Addison Wesley, 1975.
[Bry86] Randal E. Bryant. Graph-based algorithms for Boolean function ma-
nipulation. IEEE Transactions on Computers, 35(8):677691, August
1986.
[BS94] R.J.R. Back and K. Sere. From modular systems to action systems. In
Proceedings of Formal Methods Europe94, Spain, October 1994, Lec-
ture Notes in Computer Science. Springer-Verlag, 1994.
[Bur00] Simon Burton. Automated testing from Z specications. Technical re-
port, Department of Computer Science, University of York, November
2000.
[BvW98] Ralph-Johan Back and Joakim von Wright. Renement Calculus,
a Systematic Introduction. Graduate Texts in Computer Science.
Springer, 1998.
[BW99] Salimeh Behnia and Helène Waeselynck. Test criteria denition for B
models. In Jeannette M. Wing, Jim Woodcock, and Jim Davies, editors,
FM99 Formal Methods, World Congress on Formal Methods in
the Development of Computing Systems, Toulouse, France, September
1999, Proceedings, Volume I, volume 1709 of Lecture Notes in Computer
Science, pages 509529. Springer, 1999.
[Cho78] T.S. Chow. Testing software design modeled by nite-state machine.
IEEE Transactions on Software Engineering, 4(3), 1978.
[Chu40] Alonzo Church. A formulation of the simple theory of types. Journal
of Symbolic Logic, 5:5668, 1940.
BIBLIOGRAPHY 127
[Coe92] Martin David Coen. Interactive Program Derivation. PhD thesis,
St. Johns College, University of Cambridge, 1992.
[Coh78] E.I. Cohen. A Finite Domain-Testing Strategy for Computer Program
Testing. PhD thesis, Ohio State University, June 1978.
[Cot84] I.D. Cottam. The rigorous development of a system version control
program. IEEE Transactions of Software Engineering, 10(2):143154,
March 1984.
[CS94] D. Carrington and P. Stocks. A tale of two paradigms: Formal meth-
ods and software testing. In Proceedings of the 8th Z User Meeting.
Springer-Verlag, 1994.
[Daw91] John Dawes. The VDM-SL Reference Guide. Pitman, 1991. ISBN
0-273-03151-1.
[DB99] John Derrick and Eerke Boiten. Testing renements of state-based
formal specications. Software Testing, Verication and Reliability,
9:2750, July 1999.
[DDH72] O.-J. Dahl, E.W. Dijkstra, and C.A.R. Hoare. Structured Programming,
chapter I. Notes on Structured Programming, page 6. A.P.I.C. Studies
in Data Processing. Academic Press, London and New York, 1972.
[DF93] Jeremy Dick and Alain Faivre. Automating the generation and sequenc-
ing of test cases from model-based specications. In J.C.P. Woodcock
and P.G. Larsen, editors, FME93: Industrial-Strength Formal Meth-
ods. Springer-Verlag, April 1993.
[DG95] Eugène D urr and Stephen Goldsack. The development of concurrent
and real-time systems using VDM
++
, June 1995.
[DGP95] Eugene D urr, Stephen J. Goldsack, and Nico Plat. Object reication in
VDM
++
. In M. Wirsing, editor, ICSE-17 Workshop on Formal Methods
Application in Software Engineering Practice, pages 194201, 2425
April 1995.
[Dij76] E.W. Dijkstra. A Discipline of Programming. Series in Automatic
Computation. Prentice-Hall International, 1976.
[DKLM98] Martin Dunstan, Tom Kelsey, Steve Linton, and Ursula Martin.
Lightweight formal methods for computer algebra systems. In IS-
SAC 98: International Symposium on Symbolic and Algebraic Com-
putation, University of Rostock, Germany, August 1998.
128 BIBLIOGRAPHY
[DLS78] R. DeMillo, R. Lipton, and F. Sayward. Hints on test data selection:
Help for the practicing programmer. IEEE Computer, 11(4):3441,
April 1978.
[DM91] P. Dauchy and B. Marre. Test data selection from algebraic spec-
ications: Application to an automatic subway module. In A. van
Lamsweerde and A. Fugetta, editors, 3rd European Software Engineer-
ing Conference, ESEC91, volume 550 of Lecture Notes in Computer
Science, 1991.
[dNH84] R. de Nicola and M. Hennessy. Testing equivalences for processes.
Theoretical Computer Science, 34, 1984.
[Don97] Michael R. Donat. Automating formal specication-based testing. In
Michel Bidoit and Max Dauchet, editors, TAPSOFT 97:Theory and
Practice of Software Development, 7th International Joint Conference
CAAP/FASE, volume 1214 of Lecture Notes in Computer Science,
pages 833847. Springer-Verlag, April 1997.
[Don98] Michael R. Donat. A Discipline of Specication-Based Test Derivation.
PhD thesis, Department of Computer Science, The University of British
Columbia, September 1998.
[Dro00] Georg Droschl. Formal Specication and Analysis of Requirements in
Software Development. PhD thesis, Institute for Software Technology,
TU-Graz, Austria, April 2000. Supervisor: Peter Lucas.
[ELC
+
98] S. M. Easterbrook, R. Lutz, R. Covington, J. Kelly, Y. Ampo, and
D. Hamilton. Experiences using lightweight formal methods for re-
quirements modeling. IEEE Transactions on Software Engineering,
24(1), January 1998.
[Fen00] Pascal Christian Fenkam. Dynamic user management system for web
sites. Masters thesis, Institute for Software Technology, TU-Graz, Aus-
tria, September 2000. Supervisor: Harald Gall and Peter Lucas.
[FGJM95] K. Futatsugi, J.A. Goguen, J.P. Jouannaud, and J. Meseguer. Princi-
ples of OBJ2. In Proceedings of the 12th ACM Symposium on Principles
of Programming Languages, New Orleans, pages 5266, 1995.
[FH88] P.A. Freeman and H.S. Hunt. Software quality improvement through
automated testing. In Fifth International Conference on Testing Com-
puter Software, Washington, DC, June 1316, 1988, 1988.
[FL96] Brigitte Frohlich and Peter Gorm Larsen. Combining VDM-SL speci-
cations with C++ code. In Marie-Claude Gaudel and Jim Woodcock,
BIBLIOGRAPHY 129
editors, FME96: Industrial Benet and Advances in Formal Methods,
pages 179194. Springer-Verlag, March 1996.
[FL98] John Fitzgerald and Peter Gorm Larsen. Modelling Sytems, Practical
Tools and Techniques. Cambridge University Press, 1998.
[fSSaCB91] ESA Board for Software Standardisation and Control (BSSC). ESA
Software Engineering Standards. European Space Agency Publications
Division, ESTEC, Noordwijk, The Netherlands, February 1991. Doc.
Id.: ESA PSS-05-0 Issue 2.
[FvBKG91] G. Fujiwara, S. von Bochmann, F. Khendek, and A. Ghedamsi. Test
selection based on nite-state models. IEEE Transactions on Software
Engineering, 17(6), 1991.
[Gau95] Marie-Claude Gaudel. Testing can be formal too. In TAPSOFT95:
Theory and Practice of Software Development, 6th International Joint
Conference CAAP/FASE, volume 915 of Lecture Notes in Computer
Science, pages 8296. Springer-Verlag, May 1995.
[GG93] M. Grochtmann and K. Grimm. Classication trees for partition test-
ing. Software Testing, Verication and Reliability, 3:6382, 1993.
[GH93] John V. Guttag and James J. Horning. Larch: Languages and Tools
for Formal Specication. Texts and Monographs in Computer Science.
Springer Verlag, 1993.
[GHW85] J.V. Guttag, J.J. Horning, and J.M. Wing. The Larch family of speci-
cation languages. IEEE Software, 2(5):2436, 1985.
[Gib94] W. Wayt Gibbs. Softwares chronic crisis. Scientic American, Septem-
ber 1994.
[GJ98] Marie-Claude Gaudel and Perry R. James. Testing algebraic data types
and processes: A unifying theory. Formal Aspects of Computing, 10(5
& 6):436451, 1998.
[Gri81] David Gries. The Science of Programming. Texts and Monographs in
Computer Science. Springer Verlag, 1981.
[Gro92] The RAISE Language Group. The RAISE Specication Language. The
BCS Practitioners Series. Prentice-Hall, 1992.
[Gro95] The RAISE Method Group. The RAISE Development Method. The
BCS Practitioners Series. Prentice-Hall, 1995.
130 BIBLIOGRAPHY
[GSPC99] Neil Gross, Marcia Stepanek, Otis Port, and John Carey. Software hell.
Business Week, pages 3844, December 1999.
[Gut77] J. Guttag. Abstract data types and the development of data structures.
Communications of the ACM, 20(6):369405, 1977.
[GWG95] M. Grochtmann, J. Wegner, and K. Grimm. Test case design using
classication trees and the classication-tree editor. In Proceedings of
the 8th International Quality Week, San Francisco, 1995.
[HA99] Johann Horl and Bernhard K. Aichernig. Formal specication of a voice
communication system used in air trac control, an industrial appli-
cation of light-weight formal methods using VDM++ (abstract). In
J.M. Wing, J. Woodcock, and J. Davies, editors, Proceedings of FM99
Formal Methods, World Congress on Formal Methods in the Devel-
opment of Computing Systems, Toulouse, France, September 1999, vol-
ume 1709 of Lecture Notes in Computer Science, page 1868. Springer,
1999.
[HA00a] Johann Horl and Bernhard K. Aichernig. Requirements validation of
a voice communication system used in air trac control, an industrial
application of light-weight formal methods (abstract). In Proceedings
of the Fourth International Conference on Requirements Engineering
(ICRE2000), June 1923, 2000, Schaumburg, Illinois, page 190. IEEE,
2000. Selected as one of three best papers.
[HA00b] Johann Horl and Bernhard K. Aichernig. Validating voice communi-
cation requirements using lightweight formal methods. IEEE Software,
pages 2127, May/June 2000.
[Ham77] Richard G. Hamlet. Testing programs with the aid of a compiler. IEEE
Transactions on Software Engineering, 3(4):279290, July 1977.
[Han94] Kirsten Mark Hansen. Formalising railway interlocking systems. In
Nordic Seminar on Dependable Computing Systems, pages 8394, Tech-
nical University of Denmark, August 1994. Department of Computer
Science.
[Har92] Andrew Harry. The value of reference implementations and proto-
typing in a formal design and testing methodology. Report 208/92,
National Physical Laboratory, Queens Road, Teddington, Middelsex
TW11 0LW, UK, October 1992.
[Hay86] I.J. Hayes. Specication directed module testing. IEEE Transactions
on Software Engineering, 12(1):124133, 1986.
BIBLIOGRAPHY 131
[Hen88] M. Hennessy. An algebraic theory of processes. MIT Press, 1988.
[Hen89] B.M. Henderson. Big brother automated test controller. In Sixth
International Conference on Testing Computer Software, Washington,
DC, May 2225, 1989, 1989.
[Hie97] R.M. Hierons. Testing from a Z specication. Software Testing, Veri-
cation and Reliability, 7:1933, 1997.
[HL93] Ivo Van Horebeek and Johan Lewi. Algebraic Specications in Software
Engineering, an Introduction. Springer Verlag, 1993.
[HM96] E.M. Horcher and E. Mikk. Test automation using Z specications. In
Tools for System Development and Verication, Workshop, Bremen,
Germany, BISS Monographs. Shaker Verlag, 1996.
[HNS97] Steen Helke, Thomas Neustupny, and Thomas Santen. Automating
test case generation from Z specications with Isabelle. In ZUM97,
1997.
[Hoa85] C.A.R. Hoare. Communicating Sequential Processes. International Se-
ries in Computer Science. Prentice Hall, 1985. ISBN 0-13-153271-5
(0-13-153289-8 PBK).
[HP94] Hans-Martin Horcher and Jan Peleska. The role of formal specications
in software testing. In Tutorial Notes for the FME94 Symposium.
Formal Methods Europe, October 1994.
[HP95] Hans-Martin Horcher and Jan Peleska. Using formal specications to
support software testing. Software Quality Journal, 4(4):309327, De-
cember 1995.
[HS96] Merlin Hughes and David Stotts. Daistish: Systematic algebraic testing
for OO programs in the presence of side-eects. In ISSTA96, 1996.
[HT88] R. Hamlet and R. Taylor. Partition testing does not inspire condence.
In Second Workshop on Software Testing, Verication and Analysis,
Ban, Canada, July 1921, 1988, 1988.
[IFA98] IFAD. INFORMA project EP23163: User manual for the Rose-VDM
++
link, January 1998. Doc.Id.: INFORMA-IFAD-40-V1.3.
[IFA00] The VDM Tool Group, IFAD. VDMTools: The IFAD VDM
++
Lan-
guage. The Institute of Applied Computer Science, Forskerparken 10,
5230 Odense M, Denmark/Europe, 6.6 edition, 2000.
132 BIBLIOGRAPHY
[Ins00] The Institute of Applied Computer Science, Forskerparken 10,
5230 Odense M, Denmark/Europe. VDMTools: VDM
++
Toolbox User
Manual, 6.6 edition, 2000.
[ISO89] ISO. Lotos: A formal description technique based on the temporal or-
dering of observational behaviour. Technical Report 8807, International
Standards Organisation, 1989.
[JKS98] Mathias Jarke and Reino Kurki-Suoni. Special issue on scenario man-
agement. IEEE Transactions on Software Engineering, 24(12), 1998.
[Jon90] Cli B. Jones. Systematic Software Development Using VDM. Series
in Computer Science. Prentice-Hall, second edition, 1990.
[Jon96] Cli B. Jones. Formal methods light: A rigorous approach to formal
methods. IEEE Computer, 29(4):2021, April 1996.
[Jon98] Cli B. Jones. Some mistakes I have made and what I have learned from
them. In Egidio Astesiano, editor, FASE98, volume 1382 of LNCS,
pages 720. Springer, 1998.
[JS93] P.S. Jackson and P.A. Stokes. Formal Specication and Animation of a
Water Level Monitoring System. Technical Report INFO-0428, Atomic
Energy Control Board, Ottawa, Canada, March 1993.
[JW94] B.C. Jeng and E.J. Weyuker. A simplied domain-testing strategy.
ACM Transactions on Software Engineering and Methodology, 3:254
270, July 1994.
[JW96] Daniel Jackson and Jeannette Wing. Formal methods light:
Lightweight formal methods. IEEE Computer, 29(4):2122, April 1996.
[KK97] John C. Kelly and Kathryn Kemp. Formal methods, specication and
verication guidebook for software and computer systems, volume ii: A
practitioners companion, planning and technology insertion. Technical
Report NASA-GB-001-97, NASA, Washington, DC 20546, May 1997.
[KL96] Stuart Kent and Kevin Lano. Axiomatic semantics for VDM
++
,
August 1996. ESPRIT project 6500: AFRODITE, Doc.ID.:
AFRO/IC/SKKL/SEM/V1,
ftp://theory.doc.ic.ac.uk/papers/Lano/VPP/skkl.ps.
[Kra50] Victor Kraft. Der Wiener Kreis. Der Ursprung des Neopositivismus.
Springer-Verlag, Vienna, Austria, 1950.
BIBLIOGRAPHY 133
[Kuh99] D. Richard Kuhn. Fault classes and error detection capability of
specication-based testing. ACM Transactions on Software Engineer-
ing and Methodology, 8(4):411424, October 1999.
[Lam91] Richard Lampard. Specication of the Two-way Handshake in VDM.
Report 188/91, National Physical Laboratory, Queens Road, Tedding-
ton, Middelsex TW11 0LW, UK, September 1991.
[Lan95] Kevin Lano. Formal Object-Oriented Development. FACIT. Springer-
Verlag, London, 1995.
[Lar95] Peter Gorm Larsen. Towards Proof Rules for VDM-SL. PhD thesis,
Technical University of Denmark, Department of Computer Science,
March 1995. ID-TR:1995-160.
[Lar00] Peter Gorm Larsen. The VDM bibliography. Technical report, The
Institute of Applied Computer Science, Forskerparken 10, DK-5230
Odense M, Denmark, 2000.
[Las90] Janusz Laski. Data ow testing in STAD. The Journal of Systems and
Software, 12(1):314, 1990.
[LHB
+
96] P. G. Larsen, B. S. Hansen, H. Bruun, N. Plat, H. Toetenel, D. J. An-
drews, J. Dawes, G. Parkin, et al. Information technology program-
ming languages, their environments and system software interfaces
Vienna Development Method specication language part 1: Base
language, December 1996. International Standard ISO/IEC 13817-1.
[Luc87] Peter Lucas. VDM: Origins, hopes, and achievements. In Bjrner,
Jones, Airchinnigh, and Neuhold, editors, VDM 87 VDM A Formal
Method at Work, pages 118. VDM-Europe, Springer-Verlag LNCS 252,
1987.
[LW69] P. Lucas and K. Walk. On the formal description of PL/I. Annual
Review Automatic Programming Part 3, 6(3), 1969.
[LY94] D. Lee and M. Yannakakis. Testing nite-state machines: state iden-
tication and verication. IEEE Transactions on Computers, 43(3),
1994.
[Lyu95] Michael R. Lyu, editor. Handbook of Software Reliability Engineering.
McGraw-Hill, 1995.
[Mar95] B. Marre. LOFT: A tool for assisting selection of test data sets from
algebraic specications. In TAPSOFT95: Theory and Practice of Soft-
ware Development, 6th International Joint Conference CAAP/FASE,
134 BIBLIOGRAPHY
volume 915 of Lecture Notes in Computer Science, pages 799800.
Springer-Verlag, May 1995.
[MC99] Ian MacColl and David Carrington. A model of specication-based
testing of interactive systems (abstract). In J.M. Wing, J. Woodcock,
and J. Davies, editors, Proceedings of FM99 Formal Methods, World
Congress on Formal Methods in the Development of Computing Sys-
tems, Toulouse, France, September 1999, volume 1709 of Lecture Notes
in Computer Science, page 1862. Springer, 1999.
[Mil86] E.F. Miller. Mechanizing software testing. In TOCG Meeting, Westlake
Village, CA, April 15, 1986, San Francisco, 1986. Software Research
Associates.
[Mil99] Robin Milner. Communicating and Mobile Systems: the Pi-Calculus.
Cambridge University Press, 1999.
[MIO90] John D. Musa, Anthony Iannino, and Kazuhira Okumoto. Software
Reliability: Professional Edition. McGraw-Hill, New York, 1990.
[ML85] P. Martin-Lof. Mathematical Logic and Programming Languages, chap-
ter Constructive mathematics and computer programming, pages 167
184. International Series in Computer Science. Prentice-Hall, 1985.
[Mor90] Carrol C. Morgan. Programming from Specications. Series in Com-
puter Science. Prentice-Hall International, 1990.
[Mos99] Peter D. Mosses. CASL: A guided tour of its design. In Proceedings of
WADT98, volume 1589 of Lecture Notes in Computer Science, pages
216240. Springer-Verlag, 1999.
[MS93] Paul Mukherjee and Victoria Stavridou. The formal specication of
safety requirements for storing explosives. Formal Aspects of Comput-
ing, 5(4):299336, 1993.
[MW92] Zohar Manna and Richard Waldinger. Fundamentals of deductive
program synthesis. IEEE Transactions on Software Engineering,
18(8):674704, August 1992.
[Mye79] G.J. Myers. The Art of Software Testing. John Wiley & Sons, New
York, 1979.
[MZ94] Peter Mataga and Pamela Zave. Formal specication of telephone fea-
tures. In J. P. Bowen and J. A. Hall, editors, Z User Workshop, Cam-
bridge 1994, Workshops in Computing, pages 2950. Springer-Verlag,
1994.
BIBLIOGRAPHY 135
[NAS95] NASA. Formal methods specication and verication guidebook for
software and computer systems, volume I: Planning and technology
insertion. Technical Report NASA-GB-002-95, NASA, Washington,
DC 20546, USA, July 1995.
[Nom87] T. Nomura. Use of software engineering tools in Japan. In Ninth Inter-
national Conference on Software Engineering, Monterey, CA, March
30April 2, 1987., 1987.
[NR69] Peter Naur and Brian Randell. SOFTWARE ENGINEERING. NATO
Science Committee, January 1969.
[OB88] T.J. Ostrand and M.J. Balcer. The category-partition method for spec-
ifying and generating functional tests. Communications of the ACM,
31(6):676686, 1988.
[oD89] The UK Ministry of Defence. Defence standard for military safety-
critical software 00-59. draft, 1989.
[OH95] Takahiko Ogino and Yuji Hirao. Formal methods and their applications
to safety-critical systems of railways. QR of RTRI, 36(4):198203, De-
cember 1995.
[OMG96] OMG. The common object request broker architecture and specica-
tion, revision 2.0. Technical report, OMG, July 1996.
[Par94] Graeme I. Parkin. Vienna development method specication language
(VDM-SL). Computer Standard & Interfaces, 16:527530, 1994. Special
issue with the general title: The Programming Language Standards
Scene, Ten Years On.
[Pau94] L.C. Paulson. Isabelle A Generic Theorem Prover, volume 828 of
Lecture Notes in Computer Science. Springer Verlag, 1994.
[Ped98] Jesper Pedersen. Automatic test case generation and instantiation for
VDM-SL specications. Masters thesis, Department of Mathematics
and Computer Science, Odense University, September 1998.
[Pel96] Jan Peleska. Test automation for safety-critical systems: Industrial
application and future developments. In Marie-Claude Gaudel and
Jim Woodcock, editors, FME96: Industrial Benet and Advances in
Formal Methods, pages 3959. Springer-Verlag, March 1996.
[Per86] I.M. Perelmuter. Directions of automation in software testing. In Third
Conference on Testing Computer Software, Washington, DC, Septem-
ber 29 October 1, 1986, 1986.
136 BIBLIOGRAPHY
[Per95] William Perry. Eective methods for software testing. John Wiley &
Sons, 1995.
[Pet77] J. Peterson. Petri nets. ACM Computer Surveys, 9(3):223252, 1977.
[Pet81] J.L. Peterson. Petri Net Theory and the Modelling of Systems.
McGraw-Hill, New York, 1981.
[PF90] D.H. Pitt and D. Freestone. The derivation of conformance tests from
LOTOS. IEEE Transactions on Software Engineering, 16(12), 1990.
[PF98] Andrej Pietschker and John S. Fitzgerald. Techniques for automated
test generation from system specications. In 28th International Sym-
posium on Fault-Tolerant Computing, Munich, June 23-25, 1998. IEEE
Computer Society, 1998.
[Pop35] Karl R. Popper. Logik der Forschung. Wien, 1935.
[Pop63] Karl R. Popper. Conjectures and Refutations. The Growth of Scientic
Knowledge. London, 1963.
[Pos96] Robert M. Poston. Automating Specication-Based Software Testing.
IEEE Computer Society Press, Los Alamitos, California, 1996.
[PS96] Jan Peleska and Michael Siegel. From testing theory to test driver
implementation. In Marie-Claude Gaudel and Jim Woodcock, editors,
FME96: Industrial Benet and Advances in Formal Methods, pages
538556. Springer-Verlag, March 1996.
[PS97] J. Peleska and M. Siegel. Test automation of safety-critical reactive
systems. South African Computer Jounal, 19:5377, 1997.
[PT89] Nico Plat and Hans Toetenel. Tool support for VDM. Technical Report
89-81, Delft University of Technology, November 1989.
[PT92] Nico Plat and Hans Toetenel. A formal transformation from the
BSI/VDM-SL concrete syntax to the core abstract syntax. Technical
Report 92-07, Delft University, March 1992.
[PvKT91] Nico Plat, Jan van Katwijk, and Hans Toetenel. Applications and
benets of formal methods in software development. Technical Report
91-33, Delft University of Technology, Faculty of Technical Mathemat-
ics and Informatics, April 1991.
[RC81] D.J. Richardson and L.A. Clarke. A partition analysis method to in-
crease program reliability. In Fifth International Conference on Soft-
ware Engineering, San Diego, CA, March 912, 1981, 1981.
BIBLIOGRAPHY 137
[Roy70] W.W. Royce. Managing the development of large software systems. In
IEEE WESCON, pages 19, August 1970.
[Rus00] John Rushby. Disappearing formal methods. In Proceedings of the
5th Conference on High-Assurance Software Engineering, 15th17th
November 2000, Albuquerque, New Mexico, USA, pages 9596. IEEE,
2000. Invited talk.
[Sad88] S.J. Sadler. Nuclear reactor protection software, an application of
VDM. Appendix to: VDM 88. VDM - The Way Ahead. Springer-
Verlag, 1988. From: Rolls-Royce and Associates Limited.
[SC93a] P.A. Stocks and D.A. Carrington. Test templates: A specication-
based testing framework. In Proceedings of the 15th International Con-
ference on Software Engineering, pages 405414, August 1993.
[SC93b] Phil Stocks and David Carrington. Test template framework: A
specication-based testing case study. In Proceedings of the Interna-
tional Symposium on Software Testing and Analysis, pages 1118, 1993.
[SCS97] Harbhajan Singh, Mirko Conrad, and Sadegh Sadeghipour. Test case
design based on Z and the classication-tree method. In Proceedings
of the 1st International Conference on Formal Engineering Methods
(ICFEM 97), 1997.
[Scu88] G.T. Scullard. Test case selection using VDM. In R. Bloomeld,
L. Marshall, and R. Jones, editors, VDM88, VDM The Way Ahead,
2nd VDM-Europe Symposium, Dublin, Ireland, September 1988, Pro-
ceedings, volume 328 of Lecture Notes in Computer Science, pages 178
186. Springer Verlag, 1988.
[Sen87] Dev Sen. Objectives of the British standardization of a language to
support the VDM. In Airchinnigh Bjrner, Jones and Neuhold, editors,
VDM 87 VDM A Formal Method at Work, pages 321323. VDM-
Europe, Springer-Verlag LNCS 252, 1987.
[Som92] Ian Sommerville. Software Engineering. International Computer Sci-
ence Series. Addison-Wesley, 4th edition, 1992.
[Spi88] J.M. Spivey. Understanding Z, a Specication Language and its formal
semantics. Cambridge University Press, 1988.
[Spi92] J. Michael Spivey. The Z Notation: A Reference Manual. Prentice-Hall
International Series in Computer Science. Prentice-Hall, New York,
N.Y., second edition, 1992.
138 BIBLIOGRAPHY
[SS98] S. Sadeghipour and H. Singh. Test strategies on the basis of extended
nite state machines. Technical report, Daimler-Benz AG, Research
and Technology, 1998.
[Sta89] M.E. Staknis. The use of software prototypes in software testing. In
Sixth International Conference on Testing Computer Software, Wash-
ington, DC, May 2225, 1989, 1989.
[Ste95] Susan Stepney. Testing as abstraction. In J. P. Bowen and M. G.
Hinchey, editors, ZUM 95: 9th International Conference of Z Users,
Limerick 1995, volume 967 of Lecture Notes in Computer Science.
Springer, 1995.
[Sto93] Philip Alan Stocks. Applying formal methods to software testing. PhD
thesis, The Department of computer science, The University of Queens-
land, 1993.
[Sto98] Ole Storm. The VDM Toolbox API users guide. Technical report,
IFAD, 1998.
[TaA88] G.S. Teo and M. Mac an Airchinnigh. The use of VDM in the spec-
ication of chinese characters. In VDM 88 VDM The Way Ahead,
pages 476499, September 1988.
[TDS98] H. Treharne, J. Draper, and S. Schneider. Test case preparation using
a prototype. In Didier Bert, editor, B98, volume 1393 of LNCS, pages
293311. Springer, 1998.
[Tre92] J. Tretmans. A Formal Approach to Conformance Testing. PhD thesis,
Universiteit Twente, the Netherlands, December 1992.
[Tur93] Kenneth J. Turner, editor. Using Formal Description Techniques
Estelle, LOTOS, SDL. John Wiley and Sons Ltd., 1993.
[vdBVW99] Manuel van den Berg, Marcel Verhoef, and Mark Wigmans. Formal
specication and development of a mission critical data handling sub-
system, an industrial usage report. In John Fitzgerald and Peter Gorm
Larsen, editors, Workshop Materials: VDM in Practice!, Part of the
FM99 World Congress on Formal Methods, Toulouse. Springer, 1999.
[VGvM80] U. Voges, L. Gmeiner, and A.A. von Mayerhauser. Sadat an
automated test tool. IEEE Transactions on Software Engineering,
16(3):286290, 1980.
[vW94] J. von Wright. Program renement by theorem prover. Technical re-
port, Abo Akademi University, Dept. of Computer Science, 1994.
BIBLIOGRAPHY 139
[WD96] Jim Woodcock and Jim Davies. Using Z Specication, Renement,
and Proof. International Series in Computer Science. Prentice Hall,
1996.
[Wit21] Ludwig Wittgenstein. Tractatus logico-philosophicus. Annalen der
Naturphilosophie, 1921.

Systematic Black-Box Testing of Computer-Based Systems Through Formal Abstraction Techniques

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Systematic Black-Box Testing of Computer-Based Systems Through Formal Abstraction Techniques

Uploaded by

Copyright:

Available Formats

Systematic Black-Box Testing of

Computer-Based Systems through

[ b) is available and generalizes assignment statements. It sets

such that b, relating x and x

for which -e x-x

, we can interchangeable say C is

. In order to emphasize rather the search for abstractions than

Theorem 1 shows that only for deterministic specications, simple input-output

5.3.2 Non-Deterministic Test-Cases

The proof is similar to that of Theorem 5.1.

(X S; X skip) with, in contrast to angelic iteration, the termination out of a

5.5. SUMMARY AND DISCUSSION 75

only with one input variable. If

80 CHAPTER 6. TESTING STRATEGIES

6.2 Structural Partitioning

6.3 Mutation Testing

= x] is a valid test-case for detecting a specication

= x] would not detect the mutation since it is an abstraction

6.4 Calculating Scenarios for Testing

Init; do true : :New

= calls Call (o1, o2)]

= calls\c (c[src] = o c[trg] = o)]

= calls Call (o, o2)]

= calls\c (c[src] = o c[trg] =

((c calls c[src] = o) (c calls c[trg] = o))

((c calls c[src] = o) (c calls c[trg] = o))

(c calls c[src] = o c[trg] ,= o))

((c calls c[src] = o c[trg] = o))

(c calls c[src] = o))

is only modied with respect to the

ops, roles, calls: = ops

contract is presented. Since the contract consists of 7 basic operations on a certain

You might also like