Mutation Testing Techniques: A Comparative Study

Soukaina Hamimoune Bouchaib Falah

School of Science and Engineering School of Science and Engineering
Al Akhawayn University in Ifrane Al Akhawayn University in Ifrane
Ifrane, Morocco Ifrane, Morocco
s.hamimoune@aui.ma b.falah@aui.ma

Abstract— Testing is a very crucial phase in any software

development, in which various testing techniques are used with
the intent of finding software defects. Different approaches have
been suggested to effectively accomplish an application testing,
and testers shall choose the most adequate one in terms of cost
and efficiency. Mutation has been ranked as one the most
effective testing techniques in assessing the quality of input values
and test cases. However, it has been neglected by many testers
because of the costly techniques it encompasses. This paper
conducts a comparative study of four different mutation testing
techniques (class-level operators, method-level operators, all
operators, and random sampling). The purpose of this research
paper is to conduct the most effective and reliable mutation
testing technique. To do this, an empirical study on five different
java applications is presented and focused on investigating each
mutation technique. Based on the results of this experimental
comparison, the all operators sampling technique is the most
optimized and effective mutation testing method.
The goal is to cause the test case to fail so that it would be able
to detect the change in the mutant code. In another word as
deducted in [6], if the mutant produces a result that is different
from the original program, the tester can deduce that the
program contains a syntactic error that needs to be corrected.
In this case, the test suite is said to be-efficient and the mutant
is referred to as a killed mutant. When this happens, the
mutant is considered dead and no longer needs to remain in
the testing process since the errors injected to the mutant have
been detected. Otherwise, if the test suite hasn’t been able to
identify the presence of an error, then it is inadequate and the
mutant is said to be alive. Normally, a mutant remain alive
because it is equivalent to the original program. As illustrated
in Fig.1, the output of an equivalent mutant is the same as the
original one, and it can’t be killed.

Keywords— Software Testing; Mutation Testing; Mutant;

Mutation operators

I. INTRODUCTION
Software testing is an important and practical technique to
efficiently detect errors in complex software systems.
Mutation testing is concerned to be a structural testing method
that aims at assessing the adequacy and effectiveness of test
suites, along with estimating the number of faults present in
the systems under test. The essence of mutation testing arises
from the fault injection techniques in which a fault is
intentionally introduced to test the error handling of the
system that should eventually be revealed by adequate tests. In
fact, mutation was originally thought of as applying Fig. 1: Mutation Testing Approach
modification to the original program for the purpose of
generating numerous mutants, which are faulty versions; each Historically, general-purpose programming languages had
containing one change. Consequently, the mutated program been targeted by mutation testing application to measure the
code is slightly changed in comparison to the original one. adequacy of a test suite and had been widely used in
Afterwards, each mutant proposed as in [5] will be tested experimental studies. It was compared to a list of software
against a suite of test cases in order to determine the testing techniques including white-box widely used
effectiveness of these tests in detecting the injected errors.1 techniques, and mutation still remains as the most effective of
all techniques. Yet, mutation testing techniques require many

978-1-5090-5579-1/16/$31.00 ©2016 IEEE

mutants execution which result on high and massive
computational cost. When generating tests to use in
experiments, a common, although not universal practice, is to
generate multiple sets of tests to satisfy the same criterion or
according to the same procedure, and then to compute their
average performance.
However, because of the set of limitations, mutation
testing is encountering. It’s considered close to the testing
theory yet very far from the industrial use. Many researchers
justify the unpopularity of mutation testing among industrial
due to the high cost associated with the technique [4,
6].Mutation testing requires large computation resources that
necessitate a large storage space and time. The cost of the
technique raises with the increase of the project scope and
complexity. Howeden in [8] has estimated that the number of
mutants generated for an n-line program will be of n2 order.
To verify and determine equivalent mutants to exclude them
from the estimates, human effort is a necessity in the mutation
testing approach of detecting and eliminating system faults. In
fact, in [10] equivalent mutants are considered to be similar to
the original program semantically since they produce the same
output, but they are syntactically different from it. Offutt et al.
[11] led an average estimation of 8.8% of equivalent mutants
present for each program.
Various researches has spent engaging efforts in
order to remediate the ongoing issue of the high costs
associated with mutation testing as a technique. The results of
these researches and empirical studies have led to the
development of various techniques with two major focuses.
The goal of the first set of strategies is to the mutation the cost
reduction mainly by emphasizing on the optimization of some
particular steps during the mutation testing approach namely
selective mutation and higher-order mutation. In [6] it had
been mentioned that the next set of strategies aims to improve
the effectiveness of the technique by reducing the occurrence
of equivalent mutants. Consequently, these mutants cannot be
caught, and therefore, probably other approaches that deal
with these mutants need to be proposed.
This paper presents a new comparative study of
mutation testing techniques, which aims to reduce the cost of
mutation by deriving the most effective mutation testing
technique. The approach operates under the assumption that
the more the number of killed mutants is, the more the highest
mutation score is computed, hence the mutation technique is
said to perform an effective and efficient testing.
Our approach was conducted on five different open
source applications downloaded from different internet
repositories .The results were compared with the Selective
Mutation and Strong Mutation techniques. Strong Mutation
performs mutation by applying all operators and testing all
mutants with the four suggested mutation techniques (class-
level operators, method-level operators, all operators, and
random sampling). [10, 12] tackled selective mutation and
defined it as a do-fewer approach that seeks to reduce the cost
of mutation testing by omitting the operators that generates the
highest number of mutants.
The paper is organized as follows: section 2
investigates the existing literature on do-fewer approaches. It
analyzes the four mutation testing techniques. Section 3
presents the mutation operators designed for Java. Section 4
describes the steps of the suggested algorithms. A description
of the tools and the applications used in the experiment is
presented in section 5. Section 6 reports the major results of
the empirical study. Finally, section 7 develops a comparative
study based on the previous results to present the most
effective mutation testing technique.
II. RELATED WORK
In the software engineering life cycle, software
testing is the fourth pole and the most crucial one. There are
several techniques and approaches that are used to test the
correctness and quality of software. Mutation Testing is a
testing technique that is characterized by its ability to test
internal code of a program through which testers intend to
measure the effectiveness of test suites. Frankl et al. [7] and
Offutt et al. [5] conducted that mutation testing was more
effective at finding faults than data-flow. Walsh [15]
empirically found that mutation testing is more powerful than
statement and branch coverage. They compared mutation
testing technique with other testing techniques and ignored
comparing mutation testing techniques with each other.
Although mutation testing is powerful, it is still
considered as a complicated costly technique and very time
consuming which triggered the rise of many researches to
optimize and reduce its cost. B. Falah and K.Magel [1]
proposed an approach to reduce significantly the number of
test cases needed to reveal the presence of a large percentage
of the errors that would be discovered through existing
approaches. Another approach of tests’ reduction is what
Zhangl et al [9] has suggested, by reordering tests, which a test
killing the mutant (when it can be killed) must run earlier. On
the other hand, Bottaci [17] proposed that, by adapting the
meta-mutant design the generation mutants should be delayed
until run-time. Respectively, one of the problems that may
affect the efficiency of mutation testing that had been tackled
for decades is the occurrence of the equivalent mutants. One
way to avoid it [19] has proposed a comparative study of the
existing techniques that had been developed for this purpose to
eliminate the frequency of equivalent mutants within the phase
of mutants’ generation.
Several scholars in [10,13] have studied the process
of mutation testing and suggested that it is possible to reduce
the cost of mutation testing by following these three
approaches: do-fewer, do-smarter, do-faster.
 Do-fewer approach aims to run fewer mutants
without incurring large losses in information. In do-fewer
approach, strategies are used to select subset of the mutants
from the set of mutants generated in such way that it is
sufficient to assess test cases [10, 12]. Examples of do-fewer
approach are selective mutation and mutant sampling.
Do-faster approach as in [12] aims to generate and
run mutants as fast as possible, thought developing a set of
cost reduction algorithms. Some do-faster approaches include
schema-based mutation analysis and separate compilation.
Mothra [5] and Proteum [4] are well known and
widely used mutation tools. Various tools that support
mutation testing had been suggested; Hafiz [18] has developed
a tool to create a plugin for Eclipse framework as an
enhancement of MuJava. The ultimate goal of the innovative
tool that Hafiz [18] suggested is to have an upgraded version
of MuJava that will tolerate the execution of programs with
various issues like exceptions, infinite loops, etc.
III. MUTATION TESTING
Mutation testing aims at generating a set of test cases that
are adequate enough to successfully find all possible faults in
a program. It is used to test a test case and kill all of its
mutants. A test case that could achieve to kill all non-
equivalent mutants is said to be adequate with a high score. A
mutant score is measured by the total number of killed
mutants over the non-equivalent ones.
Fig. 2: Equivalent Mutants Example
A. Mutation Operators
The mutation testing techniques are very crucial in
evaluating, comparing, and improving the quality of a test
suite. Nonetheless, the value of mutation testing depends on
the set of mutants used in the evaluation. These mutants are
formed from the original program through the use of
predefined mutation operators. In order to perform syntactic
modification on the program, mutation operator is a rule that
substitutes section of the source code. Since mutation is
always based on mutation operators, researchers have
designed and developed a set of mutation operators to support
various programming languages such as Java. The quality of
mutation operators is the key to mutation testing.
The main goal behind mutation operator’s use is to
generate many faulty version of the original program [4].
These versions are called mutants. A mutant is created by
applying mutation operator on the original source code. An
operator is a rule that substitutes section of the source code in
order to perform syntactic modification on the program [2].
Fig.3 illustrates the creation process of mutants.

Fig. 3: Mutants Creation Process

Mutation score is a real value from 0.0 to 1.0 that is
considered the highest score. To attain the best mutation score, The adequacy of mutation testing resides in the
the test suite should successfully kill all the mutants. However, effective mutants’ choice. The test engineer runs the mutants
some of the mutants created are equivalent [10]. Therefore, to against test cases and compare their behavior with the
get the highest possible score, all non-equivalent mutants need behavior of the original program for the intention to
to be detected. Such mutants only contribute in increasing the identifying faulty results. The percentage of mutants killed by
mutation testing computational cost. They do not help the test cases is the mutation score [2].
suite establishing
should successfullywhether
kill all the amutants.
particular
However,test
somecase is effective
of the mutants created areinequivalent
[10]. discovering
Therefore, to getfaults in apossible
the highest program orallnot.
score, Also, themutants
non-equivalent need to be detected. Researchers have suggested different mutation operators
identification
Such ofmutants only contribute
whether a mutant in increasing the mutation
is equivalent to testing computational
the original is dofor
cost. They
mutant not each programming language. Our research emphasizes on
help establishing whether a particular test case is effective in
theoretically hard to decide. Fig.2 illustrates an example of discovering faults in a program or where the focus is only on the operators that deal with
Java,
not. Also, the identification
equivalent mutants. of whether a mutant is equivalent to the original mutant is theoretically
object-oriented feature. The mutation operators supported by
hard to decide. Figure 2 illustrates an example of equivalent mutants.
Java are divided into two main categories: class -level
operators and method-level operators.
Original Program P Mutant Program P

If (a ==2 && b ==2) If (a ==2 && b ==2) 1) Class-Level Operators

c = a * b; c = a + b;
The value of c will always result to 4. No test
would be able to detect the fault with this program.
Figure 2: Equivalent Mutants Example
In the late 90’s the mutation class operators were
introduced to address the faults associated object- oriented
features. They were developed based on object-oriented
features [16]. Object oriented programs can be distinguished
from traditional programs in many characteristics. They are
often structured differently and contain new features such as

3.1 Mutation Operators

The mutation testing techniques are very crucial in evaluating, comparing, and improving the
quality of a test suite. Nonetheless, the value of mutation testing depends on the set of mutants
used in the evaluation. These mutants are formed from the original program through the use of
predefined mutation operators. In order to perform syntactic modification on the program,
mutation operator is a rule that substitutes section of the source code.

Since mutation is always based on mutation operators, researchers have designed and developed a
set of mutation operators to support various programming languages such as Java. The quality of
inheritance, polymorphism, encapsulation, and dynamic Java-Specific JTI This keyword Insertion
binding. Examples of the modification of these features, by the Features
JTD This keyword Deletion
class mutation operators, include deleting the super keyword
JSI Static modifier Insertion
for inheritance and changing a cast type for the polymorphism.
JSD Static modifier Deletion
The class level operators considered for MuJava are JID Member variable Initialization
divided in four categories [10] according to their usage in Deletion
Object Oriented Programming: encapsulation, polymorphism,
JDC Java-supported Default
inheritance, and Java specific features. These four groups are constructor Creation
based on the language features that are affected [3]. The first
three groups target features common to all OO programing EOA Reference assignment and
content Assignment
languages while the fourth category depends solely on Java. replacement
EOC Reference comparison and
Class-level operators make changes to the program
content Comparison
syntax by inserting, deleting or modifying the expressions replacement
under test. Operators have been defined for each category. In
EAM Accessor Method change
all, there are 29 class-level operators in MuJava. Table I
presents the class operators supported by java as well as their EMM Modifier Method change
description.

TABLE I. CLASS-LEVEL OPERATORS IN MUJAVA

Language Operator Description
2) Method-Level Operators
Feature In addition to class level operators, MuJava, also, has
method operators that are known as traditional operators and
Encapsulation AMC Access Modifier Change that are developed based on procedural language features [16].
Inheritance IHI Hiding variable Insertion
They perform modifications to statements by inserting,
replacing, or deleting primitive operators [10]. There are six
IHD Hiding variable Deletion categories of primitive operators:
IOD Overriding method Deletion
1. Arithmetic operators
IOP Overriding method calling
Position 2. Relational operators
3. Conditional operators
IOR Overriding method Rename 4. Shift operators
ISI Super keyword Insertion 5. Logical operators
ISD Super keyword Deletion
6. Assignment operators

IPC Explicit call of a Parent's According to the number and the type of operands,
Constructor
some of the method level operators are divided into two or
Polymorphism PNC New method Call with child three operators. Because MuJava has been created for Java
class type programs, only operators used in Java are considered for the
PMD Member variable Declaration creation of this tool. Table II presents the method operators
with parent class type [16] for Java:
PPD Parameter variable
Declaration with child class
type TABLE II. CLASS-LEVEL OPERATOR IN MUJAVA

PCI Type Cast operator Insertion Language Operator Description

Feature
PCD Type Cast operator Deletion
PCC Cast type Change Arithmetic AOR Arithmetic Operator

PRV Reference assignment with AOI Arithmetic Operator Insertion

other comparable Variable
AOD Arithmetic Operator Deletion
OMR Overloading Method contents
Replace Relational ROR Relational Operator

OMD Overloading method Deletion Conditional COR Conditional Operator

OAC Arguments of Overloading COI Conditional Operator Insertion

method call Change COD Conditional Operator Deletion
 Shift SOR Shift Operator Replacement length of the classes varied between 48 LOC and 347 LOC.
Logical LOR Logical Operator Replacement Table III describes the applications used in our experiment.
LOI Logical Operator Insertion
LOD Logical Operator Deletion TABLE III. DESCRIPTION OF EMPIRICAL STUDY SUBJECT

Assignment ASR Assignment Operator Application Classes

# of Testing Classes Number LOC

classes of
Methods
IV. EMPIRICAL STUDY BlackJack 7 Dealer
1) Automated Mutation Tool Hand

LogicalFacade
One of crucial and resource consuming steps of mutation
testing is the creation and execution of mutants. Many User 465 120
research studies have been conducted to automate this process Player
by developing mutation software tool to successfully perform
Hand
this task. As part of our research, we have used a mutation-
testing tool, MuClipse [1] as an integrated development Card
environment, Eclipse provides many project management and Coffee Maker 4 CoffeeMaker
development interfaces that greatly enhance designing and
implementing a Java project. MuClipse is an Eclipse Plug-in Inventory
316 50
that bridges between the existing MuJava API and the Eclipse InventoryException
Workbench.
Recipe

This tool was based on an implementation of the CruiseControl 4 CarSimulator

mutation tool MuJava, developed by Offutt [16] and his team. Controller
The architecture of MuClipse is highly related to the 544 89
CruiseController
architecture of MuJava. In fact, it uses the same mutation
operators supported by MuJava. The tool implements a ―do- SpeedController
faster approach using Mutant Schemata Generation technique Elevator 8 Elevator
(MSG) [1]. MSG creates a single-meta-mutant for all existing
mutants. This requires two compilations: the compilation of ElevatorControl
the original program and the compilation of the meta-mutant ElevatorGroup
[4]. This results in a reduced time during mutants’ creation. ElevatorInterface
676 148
Along with being an important tool for object Floor
oriented mutation testing, MuClipse provides a list of helpful FloorControl
features that are relevant to the performed experiment. Among
FloorInterface
the elements MuClipse has been distinguished from other
tools; is the possibility of being fully integrated with the ArrivalSensor
development environment and also allowing having a wide Find 1 Find 37 48
range of mutation operators that include both class-level and
method-level operators. Respectively, it has a good interface
that is visually useful for mutants’ statuses examination (alive
or killed) besides the support of mutants’ generation and In order to illustrate the presence or absence of each
execution in two distinct phases. of the four class-level operators’ categories in our applications
under test, Table IV classifies the subject applications into
Overall, MuClipse had been chosen for provides an these operators’ categories.
API that simplifies the process of performing mutation testing.
TABLE IV. DESCRIPTION OF EMPIRICAL STUDY SUBJECT
2) Subject Applications Application # of Inheritance Polymor Java
classes phism Specific
In order to test the proposed approach, five small Java features
open source applications of different lengths and complexity BlackJack 7 No No Yes
have been selected and downloaded from academic websites.
Coffee Maker 4 No No Yes
The applications contained from 2 to 9 classes in total. The
 CruiseControl 4 No Yes Yes checked to run the test cases using class level operators or
Elevator 8 Yes Yes Yes method level operators.
Find 1 No No Yes

Furthermore, to investigate the presence or the TABLE VI. METHOD –LEVEL OPERATORS MUTATION
absence of each method level operator in our subject Applic Class Unit Testing
applications, Table V categorizes, for each subject ation
application, the method-level operators into six operators’
categories depending on the type of primitive operator. Name # of Killed Live Sco Mean
method muta muta re Score
Both Logical and Assignment operators exist in the s nts nts
chosen applications. Coffee Coffee Maker 8 155 162 48
Maker
Inventory 10 78 31 71 65.33
TABLE V. METHOD –LEVEL DESCRIPTION OF EMPIRICAL STUDY
SUBJECT Recipe 14 84 25 77

BlackJ Card 8 35 2 94
ack
Dealer 1 12 0 100

Hand 3 27 0 100 94.2

LogicFacade 17 58 17 77

User 5 30 0 100

Cruise CarSimulator 11 57 163 25

Contro
l Controller 9 63 37 65 55.25

CruiseControl 3 1 0 100
3) Subject Applications
SpeedControl 7 32 71 31

The method that had been conducted in this study is Elevat ArrivalSensor 3 3 4 42
or
based on the results from the metric value of mutation score ElevatorInterf 7 1 115 0
that had been presented above. The test cases have been ace

developed using unit testing, and then run on the applications ElevatorContr 10 3 7 30
chosen after these programs have been modified automatically ol

using Muclipse as a mutation testing tool. ElevatorGroup 9 29 87 25 24.8

Elevator 24 45 147 76
As for the purpose of this research mutation testing
FloorControl 3 0 63 0
tool with its different techniques had been used. For each
class, all methods had been tested with a great focus on FloorInterface 5 14 66 17
complex ones. The approach started with the creation of one Floor 12 10 157 5
or more tests for each method depending on the complexity. Find Find 4 176 45 76 76
Afterwards, test cases were run on the mutated programs four
times, by modifying the mutant operators for each round. The
four techniques that had been experienced are method
operators, class operators, all operators and a random TABLE VII. CLASS –LEVEL OPERATORS MUTATION
sampling subsequently. Applic Class Unit Testing
ation
The mutation score had been calculated for each
class within the application and respectively for each Name # of Killed Live Sco Mean
method muta muta re Score
technique. In order to expedite the process of comparison of s nts nts
the techniques effectiveness between different programs, a
Coffee Coffee Maker 8 1 74 1
mean mutation score had been calculated by application. Maker
Inventory 10 31 19 62 39.6
4) Research Results Recipe 14 13 17 56
BlackJ Card 8 10 1 90
ack
Tables VI, and VII are a representation of mutation Dealer 1 1 0 100 93
score values by class and a mean score by application. In
Hand 3 5 0 100
the process of mutants ‘generation, it had been manually
 LogicFacade 17 3 1 75 ElevatorContr 10
ol 3 9 25
User 5 4 0 100
ElevatorGroup 9 90 30 75
Cruise CarSimulator 11 6 10 37
Contro Elevator 24 41 548 6
l Controller 9 3 4 42
51.75 FloorControl 0 24 78 20
CruiseControl 3 1 0 100
FloorInterface 1 15 68 28
SpeedControl 7 4 10 28
Floor 0 35 176 17
Elevat ArrivalSensor 3 3 4 42
or Find Find 0 49 147 75 73
ElevatorInterf 7
ace 1 2 33

ElevatorContr 10
ol 1 4 20
TABLE IX. RANDOM SAMPLING OPERATORS MUTATION
ElevatorGroup 9 13.86
0 2 0 Applic Class Unit Testing
Elevator 24 ation
9 56 7
FloorControl 0 15 0 0 Name # of Killed Live Sco Mean
FloorInterface method muta muta re Score
1 2 33 1
s nts nts
Floor 0 19 0 0
Coffee Coffee Maker 8 178 167 67
Find Find 0 4 0 0 0 Maker
Inventory 10 138 50 78 72.3
Recipe 14 101 38 72
Table VIII shows that the all operators sampling method BlackJ Card 8 46 2 95
ack
that generates mutants using both method and class Dealer 1 13 0 100
operators. While Table IX exemplify the random
Hand 3 32 0 100 96.6
sampling for mutant generation, this task had been
performed manually by an external third party manually. LogicFacade 17 51 7 88
A total of 30 operators had been selected randomly from User 5 34 0 100
43, which represents 70% of the operators. Afterwards, Cruise CarSimulator 11 64 172 27
scores had been calculated and reflected in the table Contro
l Controller 9 68 41 68
73.75
CruiseControl 3 1 0 100

TABLE VIII. ALL OPERATORS MUTATION SpeedControl 7 56 0 100

Applic Class Unit Testing Elevat ArrivalSensor 3 4 6 40

ation or
ElevatorInterf 7
ace 20 119 10
Name # of Killed Live Sco Mean
method muta muta re Score ElevatorContr 10
ol 3 9 25
s nts nts

Coffee Coffee Maker 8 ElevatorGroup 9 90 30 75 27.63

178 167 67
Maker
Inventory 10 Elevator 24 41 548 6
138 50 78 72.3
Recipe 14 FloorControl 0 24 78 20
101 38 72
BlackJ Card 8 FloorInterface 1 15 68 28
46 2 95
ack
Dealer 1 Floor 0 35 176 17
13 0 100
Hand 3 Find Find 0 49 147 75 73
32 0 100 96.6
LogicFacade 17 51 7 88
User 5 34 0 100 The comparison had been led by considering two
Cruise CarSimulator 11 64 172 27
perspectives: the comparison of the given techniques’
Contro
Controller 9
performance within the same application, and from a bigger
l 68 41 68
73.75 scope between different applications.
CruiseControl 3 1 0 100
SpeedControl 7 56 0 100 It had been identified that within the same application
Elevat ArrivalSensor 3 4 6 40 4/5 applications had the all operators sampling method as the
or
ElevatorInterf 7 27.63 most effective mutation testing technique giving the high
ace 20 119 10 mutation score for each of the classes encompassed within the
application, and also the mean score. As for CruiseControl in
Fig.4 application the all operators’ technique is ranked the
most efficient by a high score of 73.75 % followed by the
method level operators. Similarly, this great performance was
demonstrated within other applications
Fig. 4: CruiseControl Mean Mutation Score per Technique
The all operators sampling technique have shown a
great performance within most of the classes by resulting in a
high mutation score. However, for Find in Fig.5 application
this technique had been slightly behind the method operators’
technique. The 3% difference could be explained by the
absence of the class level operators which has an impact on
the all operators sampling mutation score
Fig. 5: Find Mean Mutation Score per Technique
From a bigger scope and by comparing the given
techniques by application it had been highly recognizable
that the all operators sampling is the technique with the
highest mutation mean score.
Fig. 6: Mean Mutation Score per Application
V. CONCLUSION AND FUTURE WORK
The conducted experiment tried to empirically
examine the effectiveness of mutation testing strategies. The
results of the experiment performed, indicate that the all
operators sampling is a possible mutation technique that is
effective enough to kill a large number of mutants and test the
adequacy of test cases generated for a particular program. This
research has suggested that the choice of the type of mutation
operators can have an impact on the effectiveness of mutant
detection. It has also confirmed that all sampling operators are
more prevalent and efficient for small and middle sized
applications.
This research opens the opportunity to consider a
possible relation between the features of the program –such as
size - and the type of mutation operators used. This would
allow the tester to make the right mutation testing technique
choice corresponding to the application size.
A continuation of the paper could examine the effect
of size and complexity of application on the type of mutation
operator used, and would suggest the effective technique that
could fid to any application. Furthermore, an experiment
would be performed for the same purpose using seeded
programs instead of the automatic generation of mutants.
This is to come up with precise preferred technique
that would guide the choice of operators. As a part of future
work, the Cloud security testing will be performed to satisfy
the current web-based systems and application [21, 22, 23].
References
[1] B. Falah, K. Magel. ―Test Case Selection Based on a Spectrum of
Complexity Metricsǁ. Proceedings of 2012 on mutation operators and
selective mutation strategies: An empirical studyǁ, Vol. 9, Issue 4, pp.
205–232, December 1999.
[2] Elfurjani S, Mresa.B,― Efficiency of mutation operators and selective
mutation strategies: An empirical study, vol. 19 Issue 2, pp.111-131,
June 2009.
[3] Macario Polo , Mario Piattini , Ignacio García-Rodríguez, ―Decreasing
the cost of mutation testing with second-order mutantsǁ, Software
Testing, Verification International Conference on Information
Technology and Software Engineering (ITSE), Lecture Notes in
Electrical Engineering , Volume 212, 2013, pp. 223-235.
[4] M. E. Delamaro and J. C. Maldonado. Proteum – A tool for the
assessment of test adequacy for C programs. Proceedings of the
Conference on Performability in Computing Systems, pages 75–95, July
1996.
[5] R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J.
Offutt. An extended overview of the Mothra software testing
environment. Proceedings of the Second Workshop on Software Testing
Verification and Analysis, pages 142–151, July 1988.
[6] B. Falah, S. Bouriat, O. Achahbar,ǁ RSM: Reducing Mutation Testing
Cost Using Random Selective MutationTechnique,ǁ Malaysian Journal
of Computer Science. Vol. 28(4), 2015.
[7] P. G. Frankl, S. N. Weiss, and C. Hu. All-use vs mutation testing: An
experimental comparison of effectiveness. Journal of Systems Software,
28(3):235–253, September 1997.
[8] Ph.D. Dissertation. State University of New York at Binghamton,
Binghamton, NY, USA.
[9] Howden, W.E. Functional Program Testing And Analysis. McGraw-
Hill, New York, 1967.
[10] L. Zhang, D. Marinov, and S. Khurshid, ―Faster mutation testing
inspired by test prioritization and reduction,ǁ in Proceedings of the 2013
International Symposium on Software Testing and Analysis. ACM, Jul.
2013, pp. 235–245.
[11] Maryam Umar, 2006. An Evaluation of Mutation Operators for
Equivalent Mutants, Master Thesis. King’s College, London, United
Kingdom. Advisor Mark Harman.
[12] R. A. DeMilloand A. J.Offutt, ―Experimental results from an automatic
test casegeneratorǁ,ACM Transactions Software Engineering
Methodology,Vol. 2, Issue 2, pp. 109 –127, April 1993.
[13] Moohebat, M., Raj, R.G. , Kareem, S.B.A., Thorleuchter, D.,
―Identifying ISI-indexed articles by their lexical usage: A text analysis
approachǁ, Journal of the Association for Information Science and
Technology, Vol. 66, No. 3, pp. 501–511. doi: 10.1002/asi.23194.
[14] Eric Wong, ―Mutation Testing for The New Centuryǁ. Advances in
Database Systems, Vol. 24. Springer, New York. 978-1-4757-5939-6,
2001.
[15] J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate
tool for testing experiments? In 27th International Conference on
Software Engineering, pages 402–411, May 2005.
[16] P. J. Walsh. A measure of test case completeness. PhD thesis, Univ.
New York, 1985.
[17] Y. S. Ma, and J. Offut, ―Description of Class-level Mutation Operators
for Java,ǁ November 29, 2005.
[18] L.Bottaci, “Type Sensitive Application of Mutation Operators for
Dynamically Typed Programs ,” Paris Conference, pp. 126–131, Oct.
2010.
[19] M.Hafiz- Mutation Testing Tool for Java, September 2008,
http://www.munawarhafiz.com/research/mutationtesting/MutationTestin
g.pdf
[20] Overcoming the Equivalent Mutant Problem: A Systematic Literature
Review and a Comparative Experiment of Second Order Mutation. IEEE
transactions on software engineering, vol. 40, no. 1, january 2014.
[21] S. Aljawarneh, B. Alshargabi, S. MA Hayajneh, and A. T Imam,
“Integration of E-learning and Cloud Computing Platform Through
Software Engineering,” Recent Patents Comput. Sci., vol. 8, no. 2, pp.
100–105, 2015
[22] S. Aljawarneh, C. Laing, and P. Vickers, “Verification of Web Content
Integrity: A new approach to protecting servers against tampering,”
PGNET 2007 8th Annu. Postgrad. Symp. Converg. Telecommun. Netw.
Broadcast. Liverpool, UK, pp. 28–29, 2007.
[23] S. Aljawarneh, “Cloud security engineering: Avoiding security threats
the right way,” Cloud Comput. Adv. Des. Implementation, Technol., p.
147, 2012.