Melih Kemal Tekin Graduate Program in Engineering and Technology Management Boazii University - SPRING 2011 ETM591 Term

Project

13.04.2011

TITLE: Software Reliability Techniques, Testing Approaches, Capability Maturity Model Integration ADVISOR: Ali Rza KAYLAN, Ph.D., Professor CONTENTS: Software Reliability Models o A General Theory of Software-Reliability Model Introduction A General Theory of Software Reliability Derivation Average Size of Remaining Errors o Case1: Error Sizes are Constant and Equal o Case2: Error Sizes are Constant but Unequal o Case3: Error Sizes are Variable Apparent Error Density o Case1: Real and Apparent Error Densities are Equal o Case2: Real and Apparent Error Densities are Unequal Software Workload o Case1: Workload is Constant o Case2: Workload Shows Asymptotic Growth o Case3: Workload has a Rayleigh Pattern Derivation of Classical Software Reliability Models Linear Models Error Detectability is Constant o Jelinski-Moranda Model o Shooman Model o Musa Model Geometric Models Error Detectability is Geometrically Decreasing Rayleigh Model Error Detectability is Linearly Increasing o A Reliability Model for Object Oriented Software Introduction Mutation Operators for Object Oriented Software Complexity of Object Oriented Software Test Effectiveness A New Reliability Model for Object Oriented Software Experimental Results Tekin|1

Conclusion o An Accurate Model of Software Reliability Introduction An Incomplete Measurement of Traditional Software Reliability Restudy the Software Reliability Measurement Rethinking the Software Complexities Rethinking the Effectiveness of Software Testing An Improved Model for Software Reliability Examples and Experimental Results Conclusion o An Approach for Software Reliability Model Selection Introduction Background Classification of Software Reliability Models Early Prediction Models Software Reliability Growth Models (SRGM) Input Domain Based Models Architecture Based Models Hybrid Black Box Models Hybrid White Box Models Criteria Used For Software Reliability Model Selection Life Cycle Phase Output Desired by the User Input Required by Model Trend exhibited by the data Validity of Assumptions According to Data Nature of Project Structure of Project Test Process Development Process Algorithm SoReMS: Algorithm for Model Selection Example Conclusion and Future Work Software Capability Maturity Model o Extension of the SEI Software Capability Maturity Model to Systems Introduction Overview of Software Capability Maturity Models (S/W CMMs) the SEI S/W CMM Other S/W CMMs Extending the SEI S/W CMM to Systems CMM (SCMM) Tekin|2

Direct Extension of S/W CMM Additional Technical Areas Additional Management and Business Areas Conclusion o MIS-PyME Software Measurement Capability Maturity Model Supporting the Definition of Software Measurement Programs and Capability Determination Introduction Related Work MIS-PyME Methodological Framework MIS-PyME Measurement Capability Maturity Model MIS-PyME Measurement Capability Maturity Model : Levels and Attributes o Level 0: Incomplete o Level 1: Performed Process o Level 2: Managed Process o Level 3: Established Process o Level 4: Predictable Process o Level 5: Optimizing Process MIS-PyME Assessment Process: Determining the Measurement Capability of the Organization MCMM Interface: Integration of the Measurement Maturity Model Into MIS-PyME Methodology MIS-PyME Measurement Maturity Model Case Study Conclusion and Further Research Software Test Models o A Study of Rational Test Rules for Software Process Model Introduction Three Software Process Models and Rationality Test Methods for Them Rationality Test Method for Activity Diagram Process Model Rationality Test Method for Petri Net Software Process Model Rationality Test Method for Data Flow Diagram Software Process Model General Rationality Test Rules for Software Process Model and Test for the Rules Correctness General Rationality Test Rules for Software Process Model Correctness Test for the Rules Conclusion o Research and Establishment of Quality Cost Oriented Software Testing Model Introduction Definition of Quality Cost o Control Cost o Failure Cost Tekin|3

Distribution of Quality Cost In Software Test Model Relationship Between Control Cost and Software Quality Relationship Between Failure Cost and Software Quality Balance Relationship of Control Cost and Failure Cost Study On Typical Software Test Model Introduction of Models Feature Comparisons of Typical Software Test Model Problem Analysis Quality Cost 3D Model Principles o Define the Activities In the Software Test Model Based On Quality Cost Type. o Closely Integrate the Quality Cost and Software Test Processes. o Build the Measuring Mechanism To Software Quality In the Process of Software Test Quality Cost 3D Model System Structure Balance Software Quality and Quality Cost Control Cost Activity and Failure Cost Activity In QC 3D Model Computation of Testing Cost o Computation of Control Cost o Computation of Failure Cost o Computation of Total Testing Cost Balancing Control Cost and Failure Cost Case Study of QC 3D Model Project Background Evaluation of Project Implementation Cost Data Analysis Conclusion Research On A Behavior-Based Software Test Process Model Introduction Proposition of A Behavior-Based Software Test Model Application of the Behavior-Based Software Test Model Conclusions the W-Model For Testing Software Product Lines Introduction Development of Software Product Lines Software Product Lines Product Line Software Engineering the W-Model For SPL Test Domain Testing o Domain Component Testing Tekin|4

 REFERENCES:

o Domain Integration Testing o Domain System Testing Application Testing o Regression Testing of Applications o Application Testing Coverage o Application Variant Test Strategies of SPL Testing o Pure Application Strategy o Sample Application Strategy o Commonality and Reuse Strategy Related Works Conclusions and Future Work

Trachtenberg, M.; , "A general theory of software-reliability modeling," Reliability, IEEE Transactions on , vol.39, no.1, pp.92-96, Apr 1990 o Abstract: A general theory of software reliability that proposes that software failure rates are the product of the software average error size, apparent error density, and workload is developed. Models of these factors that are consistent with the assumptions of classical software-reliability models are developed. The linear, geometric and Rayleigh models are special cases of the general theory. Linear reliability models result from assumptions that the average size of remaining errors and workload are constant and that its apparent error density equals its real error density. Geometric reliability models differ from linear models in assuming that the average-error size decreases geometrically as errors are corrected, whereas the Rayleigh model assumes that the average size of remaining errors increases linearly with time. The theory shows that the abstract proportionality constants of classical models are composed of more fundamental and more intuitively meaningful factors, namely, the initial values of average size of remaining errors, real error density, workload, and error content. It is shown how the assumed behavior of the reliability primitives of software (average-error size, error density, and workload) is modeled to accommodate diverse reliability factors Peng Xu; Shiyi Xu;, "A Reliability Model for Object-Oriented Software," Test Symposium (ATS), 2010 19th IEEE Asian , vol., no., pp.65-70, 1-4 Dec. 2010 o Abstract: Software reliability is one of the important attributions of dependable systems. A number of software reliability models have been developed by now, but few of them take object-oriented features into account. Nowadays, more and more software systems are developed in object-oriented technology and object-oriented programming languages contain new language features, most notably inheritance, polymorphism, and dynamic binding which have a strong impact on the reliability of software. In this paper, we propose an accurate model for object-oriented software reliability estimation. The central idea of the strategies proposed is to incorporate the influence factors of Tekin|5

complexity of the software under test and the test effectiveness into the object-oriented software reliability model so as to make the software reliability model more adequate and accurate to the real situation for object-oriented software. Results from substantial experiments have shown the rationality and usefulness of the new model. Shiyi Xu;, "An Accurate Model of Software Reliability," Dependable Computing, 2007. PRDC 2007. 13th Pacific Rim International Symposium on, vol., no., pp.77-84, 17-19 Dec. 2007 o Abstract: Software reliability is one of the important attributions of dependable systems. However, for the time being, the assumptions that software reliability measurement makes do not deal with some important factors that influence the accuracy of the software reliability models such as the complexity of the software and the effectiveness of test suite, resulting in inaccuracy or even incorrectness in evaluating the software reliability. Therefore, in this paper, we propose some practical strategies in order to improve the accuracy of software reliability measurement. The idea of the strategies presented in this paper is to incorporate the important factors related to the precision of software reliability measurement into reliability models, which mainly include complexity of software and the test effectiveness, making the software reliability models more adequate and accurate to the real measurement. Experimental results based on five programs selected have shown the feasibility and usefulness of the new approach. Asad, C.A.; Ullah, M.I.; Rehman, M.J.-U.;, "An approach for software reliability model selection," Computer Software and Applications Conference, 2004. COMPSAC 2004. Proceedings of the 28th Annual International, vol., no., pp. 534- 539 vol.1, 28-30 Sept. 2004 o Abstract: Software reliability models are used for the estimation and prediction of software reliability. Selection of a software reliability model for use in a particular case has been an area of interest for researchers in the field of software reliability. Tools and techniques for software reliability model selection found in the literature cannot be used with a high level of confidence as they use a limited number of model selection criteria. In this paper, we have classified software reliability models according to software development life cycle (SDLC) phases. We have identified and defined a number of criteria (with importance level) for software reliability model selection. We have proposed an algorithm based on these criteria for the selection of software reliability models along with an example. This algorithm can be used in different phases of SDLC and applies to various classes of software reliability models. Shere, K.D.; Versel, M.J.;, "Extension of the SEI software capability maturity model to systems ," Computer Software and Applications Conference, 1994. COMPSAC 94. Proceedings., Eighteenth Annual International , vol., no., pp.195-200, 9-11 Nov 1994 o Abstract: Substantive improvements in software quality, and meaningful estimates of software cost and development schedules require applying systems engineering techniques to software engineering. Specifically, it is necessary to define existing processes, to assess these processes, to determine the desired process and to implement these processes. This approach is often called business process engineering. The Software Engineering Institute (SEI) developed a capability maturity model (CMM) for software. This software CMM (S/W CMM) is used to assess an organization's current Tekin|6

software process. Whereas the SEI S/W CMM is a very good model for assessing many software processes, it does not include many system level issues related to software development. These issues include safety, security, training and maintainability. An initial system oriented CMM is presented, together with a general discussion of some procedures for defining a more detailed system-oriented CMM. The paper is intended to stimulate discussion of the subject and to promote a generic system-oriented CMM that is suitable for tailoring to individual organizations Maria Diaz-Ley, Felix Garcia, Mario Piattini, MIS-PyME software measurement capability maturity model - Supporting the definition of software measurement programs and capability determination, Advances in Engineering Software, Volume 41, Issues 10-11, October-November 2010, Pages 1223-1237, ISSN 0965-9978, DOI: 10.1016/j.advengsoft.2010.06.007. o Abstract: One important reason for the failure of measurement program implementation is that the maturity of companies as regards measurement has not been taken into account during the definition phase. Unfortunately, the major methods and frameworks that support measurement programs - such as Goal Question Metric (GQM), Goal-Driven Software Measurement, GQ (I) M, PSM and ISO/IEC 15939 - do not address this issue explicitly. This can especially affect small and medium settings, where a low measurement maturity level is typical and where there are more measurement implementation limitations. This is the case both as far as software engineering is concerned and with respect to measurement knowledge. Nevertheless, companies do wish to define measurement programs adapted to the measurement maturity of the company and they want to improve their measurement maturity. There are few measurement assessment models and these are neither well-known nor applied in industry. This paper presents a measurement capability maturity model which supports companies both in defining their measurement programs, as adapted to the measurement maturity of the company, and in detecting measurement improvement suggestions. Bao, E.; , "A Study of Rationality Test Rules for Software Process Model," Information Management, Innovation Management and Industrial Engineering, 2008. ICIII '08. International Conference on , vol.1, no., pp.28-32, 19-21 Dec. 2008 o Abstract: Parallel system evaluation environment (PSEE) is a new generational CASE environment offering software engineers automatic support. To reuse rationality test modules for software process models in PSEE systems, we firstly studies three software process models as well as their rationality test methods, then generate the general rationality test rules, which are finally proved to be correct. The general rules act as a basis for the reuse of rationality test modules. Yuyu Yuan; Shen Gu; , "Research and Establishment of Quality Cost Oriented Software Testing Model," Electrical and Computer Engineering, 2006. CCECE '06. Canadian Conference on , vol., no., pp.2410-2415, May 2006 o Abstract: In a competitive market, quality is essential to the software. Software test plays an important role during the software-developing period. Any saving on software test will greatly reduce the total cost of software. The key point of this paper is to build a Tekin|7

software test process with the cost control management and to make tradeoff between the quality and the cost. Typical software testing model are analyzed and compared. A novel QC 3-D test model is introduced. The model combined with software test process, software quality cost and testing type. The computational formula is defined to calculate software quality cost. A software quality balance law is introduced to balance control cost and failure cost. By adjusting software test method and strategy, organization can reach the objective of the best balance between software quality and cost. An enterprise case is studied and the result is analyzed to prove the basic law. The result shows an optimistic point that will prove the accurate of the model and the law. Yongzhong Lu; Simeng Da; Jun Yang; Junli Song; Songlin Nie; , "Research on a Behavior-Based Software Test Process Model," Computer Science and Computational Technology, 2008. ISCSCT '08. International Symposium on , vol.2, no., pp.114-117, 20-22 Dec. 2008 o Abstract: Based on a detailed investigation of a variety of current software test models, a novel behavior-based software test model is first proposed. It includes several activities, namely test design, test plan, test case generation, test execution, test result analysis and test reuse. And they are laid out in the whole lifecycle of software development process. Then the model is intentionally applied to a large software test application project which is called American on-line as early as possible. The practical result shows that the software test model has a good bug-mining operation and can generate test cases efficiently in the regression test. Li Jin-hua; Li Qiong; Li Jing;, "The W-Model for Testing Software Product Lines," Computer Science and Computational Technology, 2008. ISCSCT '08. International Symposium on , vol.1, no., pp.690-693, 20-22 Dec. 2008 o Abstract: Testing is one of the important means of software quality assurance. Each software development paradigm requires an appropriate test model. Software product line engineering emerges as a new software development method. This paper puts forward a new test model for software product lines, the W-model, describes two separate and closely related sub-models of domain test and application test. Key issues and activities specific to testing software product lines such as variability test and regression test are addressed and explained with an example.

Tekin|8