You are on page 1of 11

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information

and Wireless Communications, 2011 (ISSN: 2220-9085)

EPS2Way: An Efficient Pairwise Test Data Generation Strategy


K.F. Rabbi1, Sabira Khatun1, Che Yahaya Yaakub1 and M. F. J. Klaib2 Faculty of Computer Systems & Software Engineering, Universiti Malaysia Pahang, Pahang, Malaysia. 2 Faculty of Science and Information Technology, Jadara University, Jordan. Corresponding E-mail: sabira@ump.edu.my

ABSTRACT
Testing is a very important task to develop error free software. The funds and release time of a software product is limited, thus, it is nearly hard to execute the exhaustive test i.e., to test all combinations of input data. Pairwise (2 way) test data generation strategy maintains higher reduction of exhaustive numbers at the same time, it is also a low cost solution to test any software product thoroughly. Usually, in software, faults are caused by unusual combination of input data. Hence, optimization in terms of number of generated test-cases and execution time is in demand. This paper proposes an efficient pairwise search approach (EPS2Way) of input values for optimum test data generation. This approach searches the most coverable pairs by pairing parameters and adopts one-test-at-a-time to construct final test suites. EPS2Way is effective in terms of number of generated test cases and execution time compared to other existing strategies.

very labor-intensive and expensive [1]. Approximately 50% of project money goes under software testing. Thus, the focus is to find an automatic and costeffective software testing and debugging strategy to ensure high quality software release [2]. Nowadays research and investigations on software testing focuses on test coverage criterion design, test-case generation problem, test oracle problem, regression testing problem and fault localization problem [1]. Among these problems test-case generation problem is a valuable one and come forth in producing error free software [1]. To solve this problem, Pairwise strategy (i.e. two-way interaction) has been renowned as an effective test case reduction strategy and able to detect from 60 to 80 percent of the faults [3, 4]. For example, the proofing tab under option dialog in Microsoft excel (Figure 1), there are 6 possible configurations needed to be tested. Each configuration takes two values (checked or unchecked) and on top of that the French modes takes 3 possible values and Dictionary language takes 54 possible values. So to test this proofing tab exhaustively, the number of test cases need to be executed is 26 x 54 x 3 i.e. 10,368. Assuming each test case may 1099

KEYWORDS
Combinatorial interaction, Software testing, Pairwise testing, Pairwise search approach, Test case generation.

1 INTRODUCTION In the process of software engineering, software testing and debugging is still

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

consume 4 minutes to execute; results around 28 days to complete the exhaustive test of this proofing tab [3, 4].

EPS2Way prototype tool, the empirical results with its efficiency and comparison, and finally the conclusion. 2 RELATED WORKS Observed facts show that lack of testing for both functional and nonfunctional is one of the major sources of software and systems bug/errors [7, 8]. National Institute of Standard and Technology (NIST) estimated that the cost of software failure to the US economy at $6 X 1010, which was the 0.6 percent of GDP [9, 10]. Around one-third of this cost can be reduced by improving software testing structure. Hence, automatic testing is one of the critical concerns [11]. Through system automation, software development process may become more practical and scalable. However, the automated generation and reduction of test cases are challenging [12]. The underlying problem is known as NP-hard thus researchers have focused on the techniques that search to find near optimal test sets in a reasonable time [13, 14]. Pairwise testing is a significant approach for software testing as it provides efficient error detection at a very low cost. It shows a good balance by linking the magnitude and effectiveness of combinations. It requires, combination of any two parameter values has to be covered by at least one test case [14, 15]. From the point of pairwise there are some pre-defined rules to calculate the test cases directly from the mathematical functions, which are known as algebraic strategy [4]. On the other side, 1100

Figure 1: Microsoft Excel Proofing Option

Similarly, for a hardware product which has 30 on/off switches, to test all possible combination may need 230 = 1,073,741,824 test cases, and consume 10,214 years by considering 5 minutes duration for each single test case [4]. Nowadays, research work in combinatorial strategy aims to generate smallest amount of performable test suits [5]. The solution of this challenge is known as non-deterministic polynomialtime hard (NP-hard) [6]. So far many approaches have been proposed to solve this problem for last few decades [5-8, 10-14] but yet optimum one is in demand. This paper introduces an efficient pairwise search approach for test data generation. This strategy combines two parameters collectively (a pair) and searches for another parameter combination to generate test suits from the parameter pairs in terms of optimum size and time consumption. The paper is organized as follows. The related work detail is presented in Section 2. Followed by the proposed EPS2Way strategy, the developed

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

computational approaches are based on the calculation of coverage of generated pairs, followed by an iterative or random search technique to create test cases. IRPS algorithm [6] uses the computational approach. It is a deterministic strategy which generates all pairs and then stores it to the linked list. Finally it searches the entire list, select best list and empties the list. When all list become empty, the collection of best list is determined as the final test suite. The Automatic Efficient Test Generator (AETG) [16] and its deviation mAETG [6] generate pairwise test data using computational approach. This approach uses the Greedy technique to build test cases based on covering as much as possible uncovered pairs. AETG uses a random search algorithm [17]. Genetic Algorithm (GA) and Ant Colony Algorithm (ACA) are the variants of AETG [18, 19]. Genetic algorithm [18] creates an initial population of individuals (test cases) and then the fitness of those individuals is calculated. Then it starts discarding the unfit individuals by the individual selection methods. The genetic operators such as crossover and mutation are applied on the selected individuals and this continues until a set of best individuals found. Ant Colony Algorithm [18] candidate solution is associated with the start and end points. When an ant chooses one edge among the different edges, it would choose the edge with a large amount of pheromone which gives the better result with the higher probability.

The In-Parameter-Order (IPO) [12] strategy starts with an empty test set and adds one test at a time for pairwise testing. It creates the test cases by combination of the first two parameters, then add third and calculate how many pair is been covered and so on until all the values of each parameter is checked. This approach is deterministic approach. AllPairs [20] algorithm can generate test suites covering all pairwise interactions within a reasonable time. The strategy seems to be deterministic strategies since the same test suite are generated every run time. The Simulated Annealing (SA) [21] algorithm is also a deterministic strategy with the same generated test suite for every run time. Generalization of Two Way test data (G2Way) [22] is one of the excellent tools based on computational and deterministic strategy. It is based on backtracking algorithm and uses customized markup language to describe base data. The G2Way backtracking algorithm tries to combine generated pairs so that it covers highest pairs. Finally after covering all the pairs, the test case treats as a final test suite. One of the recent algorithms is effective random search based pairwise test data generation algorithm named R2Way to optimize the number of test cases [23]. R2Way takes a random number from 0 to maximum possible exhaustive number of test cases and creates a test case based on some binary calculations. 1101

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

Java program was used to test the performance of R2Way. It is able to support both uniform and non-uniform values effectively with near optimum performance in terms of number of generated test cases and time consumption.

Table 1: Example parameters with values

Parameters A a1 Values a2

B b1 b2

C c1 c2

D d1 d2

E e1 e2

F f1 f2

3 PROPOSED EPS2Way STRATEGY The proposed efficient pairwise test data generation strategy works in three (3) steps as follows: I. Firstly, it generates pair parameters and their values. Then, the value of one pair is combined with another pair by calculating the highest possible coverable pairs. In this way it constructs a test case and adds to the final test suit. Finally, it constructs the test cases from uncovered pairs through an adjustment algorithm.

EPS2Way first generates header pair parameters which are AB, CD, EF and then calculates the possible values of all the pairs as shown in Table 2. In this example, each of AB, CD and EF pair contains 2 x 2 = 4 pair of values and stored in the memory respectively. The expression can be expressed as follows: NP = [V1 * V2] (1)

II.

Where, NP = Number of Pairs, V1 = Value of Parameter 1, V2 = Value of Parameter 2.


Table 2: Generated header pair parameters and all possible values

III.

Pair Parameters Generated all possible test cases

AB [a1, b1] [a1, b2] [a2, b1] [a2, b2]

CD [c1, d1] [c1, d2] [c2, d1] [c2, d2]

EF [e1, f1] [e1, f2] [e2, f1] [e2, f2]

To make this easily understandable, a scenario is presented in terms of example as follows: 3.1 Pair Generation EPS2Way strategy/algorithm first takes the parameters and corresponding values to make the 2 way combinations. The combinations stored into the memory. As an example, suppose, there are six (6) parameters A, B, C, D, E, and F, each with 2 values as shown in Table 1.

3.2 Test Case Generation At the beginning, each AB pair tries to combine with one of the CD pairs. If combined pairs give highest or maximum coverage, then it tries to combine with the values of EF pairs. So the test case generation approach is in greedy mode and constructs test cases one at a time. In Table 2, EPS2Way tries to combine [a1, b1] pair with four possible values of 1102

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

CD which are: [c1, d1], [c1, d2], [c2, d1], [c2, d2]. The first combined AB-CD pair with the highest coverage looks for the available pairs again from EF header pair among: [e1, f1], [e1, f2], [e2, f1], [e2, f2] as shown in Table 3. Finally, the combined AB-CD-EF pair with highest coverage is then added to the final test suit. Table 3 shows, one of AB pairs [a1, b1] searches for the best pairs among the available pairs of CD and its output should be only one, which is [a1, b1, c1, d1] as the first uncovered final test-case with full coverage. Again, generated pair [a1, b1, c1, d1] search for the available pairs of EF and the generated output is [a1, b1, c1, d1, e1, f1]. Same procedure is followed by other AB pair parameters to generate other final test cases. The highest coverable pairs are stored in the final test set.
Table 3: Example of pair search and final test case generation

[a2, b1]

[a2, b2]

d1] [c2, d2] [c1, d1] [c1, d2] [c2, d1] [c2, d2] [c1, d1] [c1, d2] [c2, d1] [c2, d2]

[a2, b1, c2, d1]

[a2, b2, c2, d2]

f1] [e2, f2] [e1, f1] [e1, f2] [e2, f1] [e2, f2] [e1, f1] [e1, f2] [e2, f1] [e2, f2]

[a2, b1, c2, d1, e2, f1]

[a2, b2, c2, d2, e2, f2]

3.3 Pair Covering & Test Case Adjustment At this stage, the adjustment algorithm performs an exhaustive search for all possible uncovered pairs and tries to adjust all uncovered pairs. When an adjustment is done it is added to the final test set. Figure 2 shows the test case generation adjustment algorithm and Figure 3 shows the flowchart of the strategy.
Adjustment Algorithm to Generate Test Suits () Begin Let PP = {} represents the set of all possible pairs Let PS = {} represents the pairs where all the PS stores in PP Let PB = {} represents the best pairs set which cover highest pairs Let PF = {} as empty set

Initi al pairs

Avai Best lable uncover pairs ed pairs [c1, d1] [c1, d2] [c2, d1] [c2, d2] [c1, d1] [c1, d2] [c2,

[a1, b1]

[a1, b1, c1, d1]

[a1, b2]

[a1, b2, c1, d2]

Availa Best ble uncove Pairs red pairs [e1, f1] [e1, [a1, b1, f2] c1, d1, [e2, e1, f1] f1] [e2, f2] [e1, f1] [a1, b2, [e1, c1, d2, f2] e1, f2] [e2,

1103

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

represents the Final test suits Let CB as number = 0 represents the best covering number Let CC as number = 0 represents the current covering number For each PS as P1 in PP For each next PS as P2 in PP Add P1 with P2 and put in P CC = Get coverage pair number of P IF CB is less or equal to CC Put CC into CB Put P2 into PB End IF End For Add P1 with PB and store to PF End For End Figure 2: EPS2Way pseudo code for test case generation from uncovered pairs.

4 EPS2Way PROTOTYPE TOOL A prototype tool is developed with simple GUI to investigate EPS2Way performance in terms of the number of generated test cases & execution time. The tool accepts both uniform and nonuniform values and shows the required number of test cases. Figures 4 to 7 shows the screen shot of PS2Way prototype tool.

Figure 4: Generator Window

Figure 3: Flow chart of the EPS2way strategy

Figure 5: Same parameter & value window

1104

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

Figure 6: Different Parameter and Value window

Figure 4 shows the initial windows containing two buttons. The value of the T-way field is 2 (un-editable) as the current algorithm supports pairwise (2way) testing. The buttons Same Params & Values and Differ Params and Values are for uniform and nonuniform input values respectively. Clicking on the Same Params & Values opens input popup windows shown in Figure 5. The Number of Parameter field takes the required number of parameters as input and the Number of values for each parameter takes input values for each parameter. Figure 6 shows the popup window which can take non uniform values. Here, the values can be added on the list box. After clicking on the ok button the number of test cases and the test case suits are produced as shown in Figure 7.

Figure 7: Generated Test cases

5 EMPIRICAL RESULTS To evaluate the efficiency of proposed EPS2Way, we have considered six (6) different system configurations. Among those the first three (3) are non-uniform parameterized values and the rest are uniform as follows: S1: 3 parameters with 3, 2 and 3 values. S2: 3 parameters with 2, 1 and 3 values. S3: 5 parameters with 3, 2, 1, 2 and 2 values. S4: 3 2-valued parameters S5: 3 3-valued parameters S6: 4 3-valued parameters. The consideration of the parameters and assumptions are according to some of 1105

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

the related existing algorithms that support pairwise testing to compare our results with those. Table 4 shows the comparison of generated test suite size by proposed EPS2Way with others. The shadowed Sys AETG IPO [16] [12]

cells indicate the best performance in term of generated test case size. NA indicates that data are not available. It shows that proposed EPS2Way produces the lowest number of test cases for each configuration by showing its superiority.

Table 4: Comparison based on the generated test size

TConfig Jenny TVG ALL G2Way R2Way Proposed [24] [25] [26] Pairs [22] [23] EPS2Way [20] 9 S1 NA 9 9 9 9 9 9 9 6 S2 NA 6 6 6 6 6 6 6 7 S3 NA 7 8 8 8 9 7 7 S4 NA 4 4 5 6 4 4 4 4 S5 NA 10 9 9 10 10 10 10 9 S6 9 10 9 13 12 10 10 9 9 programming language, Linux For a fair comparison of execution time Operating System. (i.e., complexity) among related test IPO [12]: Intel P II 450 MHz, Java strategies, either computing environment programming language, Windows 98 should be same or need the source code operating system. (usually not available most of the cases). GA [18], ACA [19]: Intel P IV 2.26 All Pairs tool [20] is free to download GHz, C programming language, and can be executed using any platform. Windows XP operating system. Also R2Way [23] is our proposed G2Way [22]: Intel P IV 1.6 GHz, 1 strategy and have source code. Hence we GB RAM, C++ programming have compared the execution time of language, Windows XP operating EPS2Way with All Pairs and R2Way system. using the same platform as follows: Intel P IV 3 GHz, 1 GB RAM, Java programming language, and Windows XP OS. For some of the other existing strategies, the platform instances and data are taken from References [22]. We have managed to execute our EPS2Way strategy using the following platform for respective comparison with: AETG [16], AETGm [6], SA [21]: Intel P IV 1.8 GHz, C++

1106

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

To compare the complexity, we have assumed the following five (5) configurations. T1: 3 3-valued parameters, T2: 4 3-valued parameters, T3: 13 3-valued parameters, T4: 10 5-valued parameters, T5: 1 5-valued parameters, 8 3-valued parameters and 2 2-valued parameters.

Table 5 shows that in terms of execution time, EPS2Way gives the better result for first three system configurations T1, T2 and T3. For the other two systems T4 and T5 the execution time is also very close to the lowest values with negligible difference. Hence EPS2Way outperforms other existing strategies.

Table 5: Comparison Based on Execution Time (in seconds)

Sys T1 T2 T3 T4 T5

AETG [16] NA NA NA NA NA

AETGm [6] NA NA NA NA 58

IPO [12] NA NA NA 0.05 NA

SA [21] NA NA NA NA 214

GA [18] NA NA NA NA 22

ACA [19] NA NA NA NA 31

ALL Pairs [20] 0.08 0.23 0.45 1.02 0.35

G2Way [22] 0.047 0.062 0.25 0.687 0.33

R2Way [23] 0.07 0.16 1.44 20.3 5

Proposed EPS2Way 0.027 0.062 0.2 0.06 0.43

In this paper we have proposed pair parameter based search algorithm for pairwise (2-way) test case generation. The correctness of the proposed strategy is apparent. The algorithm is efficient in terms of execution time and able to generate highly reduced test suites to fulfill the current demand by software development companies. The proposed algorithms could be further extended to support higher t-way interaction testing which is under investigation in University Malaysia Pahang. 6 CONCLUSIONS 7 ACKNOWLEDGEMENTS The authors would like to thanks Universiti Malaysia Pahang for financial support under University Research Grants (RDU 100361) to finalize this work. 1107

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

8.

8 REFERENCES
1. Xiang Chen, Qing Gu, Jingxian Qi, Daoxu Chen, Applying Particle Swarm optimization to Pairwise Testing, in proceedings of the 34th Annual IEEE Computer Software And Application Conference, Seoul, Korea, 2010. Yingxia Cui, Longshu Li, Sheng Yao, A New strategy for pairwise test case generation, in proceedings of the third international Symposium on Intelligent Information Technology Application, NanChang, China, 2009. Y. Lei, R. Kacker, D.R. Kuhn, V. Okun, and J.Lawrence, "IPOG: A general strategy for tway software testing, in proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, Tucson, Arizona, 2007. M. I. Younis, K. Z. Zamli, N. A. Mat Isa, Algebraic Strategy to Generate Pairwise Test Set for Prime Number Parameters and Variables, in proceedings of the IEEE international conference on computer and information technology, Kuala Lumpur, Malaysia, 2008. Mohammad F. J. Klaib, Sangeetha Muthuraman, Noraziah Ahmad, and Roslina Sidek, A Tree Based Strategy for Test Data Generation and Cost Calculation for Uniform and Non-Uniform Parametric Values, in proceedings of the 10th IEEE international conference on computer and information technology, West Yorkshire, UK, 2010. M. I. Younis, K. Z. Zamli, N. A. Mat Isa, IRPS - An Efficient Test Data Generation Strategy for Pairwise Testing, in Proceedings of the 12th international conference on Knowledge-Based Intelligent Information and Engineering Systems, Lecture Notes In Artificial Intelligence, Springer-Verlag, 2008. D. Leffingwell and D. Widrig, Managing Software Requirements: A Use Case Approach, Addison Wesley, 2003.

9.

10.

2.

11.

3.

12.

13. 14.

4.

5.

15.

6.

16.

17. 18.

7.

R. L. Glass, Facts and Fallacies of Software Engineering, Addison Wesley, 2002. National Institute of Standards and Technology, The Economic Impacts of Inadequate Infrastructure for Software Testing, Planning Report, 2-3 May, 2002. Mark Harman and Phil McMinn, A Theoretical and Empirical Study of SearchBased Testing: Local, Global, and Hybrid Search, IEEE Transactions on Software Engineering, vol. 36, no. 2, pp-226-247, 2010. P. McMinn, Search-Based Software Test Data Generation: A Survey, Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004. Y. Lei and K. C. Tai, "In-Parameter-Order: A Test Generation Strategy for Pairwise Testing", in proceedings of the 3rd IEEE International conference on High-Assurance Systems Engineering, Washington, DC, USA, 1998. D. Gong, X. Yao, Automatic detection of infeasible paths in software testing IET Software, vol. 4, no. 5, pp-361-370, 2010. Jangbok Kim, Kyunghee Choi, Daniel M. Hoffman, Gihyun Jung, White Box Pairwise Test Case Generation, in proceedings of the IEEE Seventh International Conference on Quality Software, Oregon, USA, 2007. Zainal Hisham Che Soh, Syahrul Afzal Che Abdullah, Kamal Zuhari Zamli, A Parallelization Strategies of Test Suites Generation for t-way Combinatorial Interaction Testing, in proceedings of the IEEE International conference on Information Technology, International Symposium , Kuala Lumpur, Malaysia, 2008. D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, "The AETG System: An Approach to Testing Based on Combinatorial Design," IEEE Transactions on Software Engineering, vol. 23, no. 7, pp. 437-444, 1997. M. Harman and B.F. Jones, Search-based Software Engineering & Information and Software Technology, pp. 833-839, 2001. T. Shiba, T. Tsuchiya, and T. Kikuno, "Using Artificial Life Techniques to Generate Test Cases for Combinatorial

1108

International Journal on New Computer Architectures and Their Applications (IJNCAA) 1(4): 1099-1109 The Society of Digital Information and Wireless Communications, 2011 (ISSN: 2220-9085)

19.

20.

21.

22.

23.

24. 25.
26.

Testing," in proceedings of the 28th Annual Int. Computer Software and Applications Conf. (COMPSAC04), Hong Kong, 2004. Xiang Chen, Qing Gu, Xin Zhang, Daoxu Chen, Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization, in proceedings of the 9th International IEEE Conference on Quality Software, Jeju, Koria, 2009. J. Bach, "Allpairs Test Case Generation Tool", Available from: http://tejasconsulting.com/opentestware/feature/allpairs.html (Last access date: 27th Sep. 2009) James D. McCaffrey, Generation of Pairwise Test Sets using a Simulated Bee Colony Algorithm, in proceedings of the IEEE International Conference on, Information Reuse & Integration, Las Vegas, USA, 2009. M. F. J. Klaib, K. Z. Zamli, N. A. M. Isa, M. I. Younis, and R. Abdullah, "G2Way A Backtracking Strategy for Pairwise Test Data Generation," in proceedings of the 15th IEEE Asia-Pacific Software Engineering Conf, Beijing, China, 2008. S. Khatun, K. F. Rabbi, C. Y. Yaakub and M. F. J Klaib, A Random Search Based Effective Algorithm for Pairwise Test Data Generation in proceedings of IEEE International Conference on Electrical Control and Computer Engineering 2011, Kuantan, Malaysia, 2011. "TConfig," Available at: http://www.site.uottawa.ca/~awilliam/. (Last access date: 27th Sep. 2009). "Jenny," Available at: http://www.burtleburtle.net/bob/math/. (Last access date: 27th Sep. 2009). "TVG," Available at: http://sourceforge.net/projects/tvg. (Last access date: 27th Sep. 2009).

1109