Classifying COTS Products

Letizia Jaccheri and Marco Torchiano

Department of Computer and Information Science, Norwegian University of Science and Technology, Trondheim Norway {letizia,marco}@idi.ntnu.no

Abstract. Classes of COTS products can be derived by classification attributes, which define a Cartesian space. Examples of such attributes are the architectural level, the kind of the COTS product (is it a standard, or a service, or an executable component?), and the software life cycle phase in which the product is used (is it a development tool or an executable component?). COTS products belonging to the same class can be evaluated and compared by means of evaluation attributes, such as price or type of license. This work has been conceived mainly for learning purposes. Building a classification schema and filling it with products is a way for COTS product familiarization. In addition, the process of defining classes and filling them with COTS poses new research questions, like why is this class empty?, or which are the relationships between these two classes?. The result of classification and evaluation process cannot have general validity if it not customized for special organization goals. These customization issues are outside the scope of this work.

Introduction

The requirements for being on the market today can be summarized by three keywords: faster, better and cheaper. A key factor in this strive for faster-bettercheaper is the use of innovative software technologies and the related implementations, i.e. software products available on the market, often dubbed with the buzz-word COTS (Commercial-Off-The-Shelf). Before adopting a COTS product, an organization must learn about it, and evaluate it, particularly with reference to similar products. One way to facilitate the COTS learning process is that of classifying COTS into classes of similar items and to make comparisons in the context of each class. The classification process gives a set of principles and themes, which may result in research issues for the field [3]. Our classification of COTS is intended to partition items into sets whose elements are comparable. Classes cannot be defined one time forever, but they are organization and goal specific. Classes can be defined by intuition, like the class of all DBMS, the class of all web browsers, etc.. Several works deal with classification of commercial products; either proposing taxonomies of application domain [3] or defining a set of attributes, which define a Cartesian space used to characterize COTS products [2][5]. Other works deal with the problem of evaluating and selecting COTS products using a set of attributes [1][4][6][7]. The idea underlying our work is that attributes can be divided into classification and evaluations ones. Some attributes are clearly classification ones. An example is

the phase in which the COTS is used, which can be either during development or as a component of the final system. Other attributes are evaluation ones. Price is the typical attribute one wants to use to make comparisons about different COTS. On the other hand one could be interested in studying the category of COTS, within a given price class, thus making the price attribute a classification one. This work has been conceived in the context of a fifth year course in software technology, which was run for first time at the Norwegian University of Science and Technology during the Autumn 2001. The goal of the course is to make the students reflect over how to learn, evaluate, and start to use software technology. Central in the course is a COTS classification and evaluation process. The process consists of several activities; the most important being Classification Attributes definition, COTS Classification, Evaluation Attributes definition, COTS Evaluation. Apparently, a possible outcome of this work is a software technology database which records information about COTS and their relationships and which can be used during technology selection and evaluation. However, this is not the goal of our work. We are in fact skeptical about the general validity of such a database as technology changes too fast and the information contained in the database would be doomed to become soon obsolete. Moreover, to keep the database up to date requires a huge effort, which is not affordable for most organizations whose mission is different from that of developing and maintaining such kind of information (see for instance www.ovum.com as an example of organization, which has the main mission of providing and updating such information). The process of investigating COTS product features in order to classify them is a useful activity. On one end it forces the classifier to look at products from an unusual point of view, which can uncover otherwise unnoticed features; on the other end it stimulate reflection on the results of the classification process, resulting in interesting research questions, like why is this class empty? or which are the relationships between these two classes?. This paper is mainly about the classification attributes definition phase. For more information about the whole process or the single phases, refer to [8]. The rest of the paper is organized as follows: section 2 elaborates the concept of classification attributes and proposes a set of three classification attributes. Section 3 reports about the process of assigning COTS items, to the classes and to evaluate such items. Section 4 analyze the relationships with related works. Section 5 is a preliminary evaluation of our work which has been performed in the context of a University undergraduate course. Conclusions are given in section 6.

Finding Classes

The first step in our proposal is the definition of the criteria for classifying COTS products. The approach we adopt is to define attributes that define a Cartesian space. 2.1 Classification Attributes

We identified three attributes that are useful in order to classify products: architectural

level, artifact kind, and life-cycle phase. These attributes can be associated with orthogonal axes: architectural level, product kind, life-cycle phase. Thus a given COTS product can be characterized by means of three coordinates in the resulting Cartesian space. These three attributes have to be regarded as three possible classification attributes and not as the classification attributes. Other interesting attributes, which can supplement or even substitute these three or can be found. For instance, the domain for which the COTS component is conceived, or its functionality, etc.. 2.1.1 Architectural Level Architectural issues concern both the top-level architectural pattern (or style) chosen for the system and the role the item plays in the architecture. Examples of top-level architectural patterns are: centralized, client-server, 2-tier, 3-tier, peer-to-peer, pipe and filter, and blackboard. The possible values of the architectural level attribute consist of a pair composed of: an architectural pattern, a role defined in that pattern. E.g. with the architectural pattern 3-tier, each COTS can play the role of either the client, server, or data. Examples of COTS for which the attribute architectural level assumes the value client are Html language, and the WEB browser Opera. Examples of COTS for which this attribute assumes the value server are: java servlet and Oracle application server. Examples of COTS having architectural level data are MySQL and Oracle DBMS. 2.1.2 Product Kind COTS components deal with different kind of artifacts, not only with the software properly said. We say that the attribute kind of a COTS component can assume one of the following values: executable statements either in source or binary form, e.g. Java Compiler javac and Macromedia flash. standard publicly available and approved, e.g. the HTML specification. service provided by other through some network, for instance web-based development services (such as project hosting, version control, bug and issue tracking, project management, backups and archives, and communication and collaboration resources), e.g. SourceForge.net. 2.1.3 Life-Cycle Phase Another distinction can be found when looking at when, during the life cycle of a system, a given COTS product is used. Each COTS product can be used during: development of the system, for instance Macromedia Flash, execution as a part of the running system, e.g. Opera and MySQL.

Table 1. Classification dimensions. Attribute Architectural Level Artifact Kind Life-cycle Phase Possible values c (client) e (executable) d (development) s (server) st (standard) e (execution) d (data) sv (service)

Using the encoding shown in Table 1, classes can be identified by means of a triple. E.g. (c,e,d) identifies the class of products that can be used for the client level (c), are executable (e), and are used during the development phase (d). 2.2 Generating Classes

After deciding a set of attributes and their possible values, classes can be derived. This process leads to a number of classes (Nc), which can be calculated starting from the number of possible values along the three axes: Nc = Va V k V p . (1)

If we consider three attributes described above: Va : number of architectural combinations is equal to 3 (client, server, data); Vk : number of possible kinds is equal to 3 (executable, standard, service); Vp : number of phases is equal to 2 (development, execution). Thus the number of possible classes Nc = 18. There can be items for which it is not possible to assign a given value to an attribute. In this case we can say that its value is not applicable. This would lead to a greater number of classes.

3
3.1

Classification and evaluation

Classification

Once the classes have been defined, COTS products can be assigned to classes. To do that, for each COTS product, values must be assigned to the classification attributes. It happens that it is not possible to assign a single value to an attribute. Consider for example the XML standard. XML can be used in every layer of the 3-tier architecture. Possible uses of XML include: data storage format or query language (XQuery) in the data layer distributed object communication format (SOAP) in the server layer presentation (XHTML) in the client layer XML is not a COTS, but the SOAP standard (which is based on XML) is a classifiable COTS. Another possible solution, when an item leads to ambiguity, is to try splitting it into smaller items and classify them. E.g. for Java Run Time Environment (JRE), the question architecture level may lead to the answers client

and server. If we split the JRE up into components the client compiler is at client level and the beans component is at server level. Adopting this approach, the list of COTS to be evaluated may grow as new items obtained by splitting existing items are added. For some COTS items, it is not possible to assign a unique value to a given classification attribute. For example a CASE development tool can be used to develop models at client, server, and data level. In this case we say that the attribute cannot be specified. When considering also the non-specified value, a greater number of classes arise. For simplicity reasons we choose not to generate a-priori classes for al combinations of attribute values that include the not-specified values, but we rather delay the creation of these classes when items belonging to them are found. 3.2 Evaluation

3.2.1 Attributes Definition In addition to the classification above, each COTS product can be characterized by evaluation attributes. By help of these attributes and their values it will be possible to compare items belonging to the same class. While several of these attributes are common across the classes, some attributes are specific of a single class. Examples of common attributes are: price, market share, license type, compatibility issues, etc. Examples of attributes for DBMS are transaction speed, scalability issue, etc.. For each attribute, we must define its measurement scale, which can be: nominal, ordinal, ratio, and absolute. Examples of evaluation attributes, which can be applied to several classes, are presented in the following Table 2.
Table 2. Evaluation attributes.

Attribute Acquisition cost Ownership cost Market size Market share License type

Scale Ratio Ratio Ratio Ratio Nominal

The cost of a software product can be split into two parts: the cost of acquisition of the product or of its license and the cost derived from its support, maintenance, and setup. The diffusion of a COTS product can be defined in terms of market size and market share. The market size is an absolute number representing the number of people that use items similar for the given one. For instance the market size for Internet Explorer (IE) is the number of user web browsers. Market share is the fraction of the market that uses the specific product. For instance the market share for Internet explorer is the number of user of Internet Explorer itself. Another interesting attribute for evaluating a software product is the type of license under which it is sold, e.g. single user, open source, or GNU.

3.2.2 Assigning Values to Evaluation Attributes The assignment of values to an attribute must respect the scale for that attribute. The operations on attributes must also respect scale constraints, e.g., while is it possible to assert that an item is better than another one with respect to its price, we will not able to assert any ordering based on the maturity attribute unless we change its scale.

Related Works

Glass et al. [3] survey taxonomies of application domains. They emphasize the confusion between problem-oriented criteria and solution-oriented ones. This distinction is somehow related to the distinction we propose between classification and evaluation attributes. Classification can be seen as a problem oriented operation, while evaluation deals with solution related issues. Carney and Long [2] propose the classification of COTS products using a bidimensional cartesian space. In addition they reports some examples that populate this space. The dimensions they define are origin and modifiability. The origin dimension addresses the way the product is produced; ranging from products developed on purpose to commercial components with a large number of customers. The modification dimension describes to which extent the product either can or must be modified by the system developer that uses the component. Morisio and Torchiano [5] propose a classification framework for COTS products. The purpose of their framework is twofold: first it is a tool to precisely define what is the meaning of COTS product, second it represents a way of specifying which subclasses of products are addressed by a given work. The aim of their work is mainly classification. It is similar to the approach presented in this paper. Boloix et al. [1] propose a framework for evaluating technology; it is based on metrics that address three perspectives (or dimensions): the project, the system and the environment. Each dimension encompasses three factors. In order to keep the process simple, only three ratings are used: basic, intermediate, and advanced. One of the first approaches proposed for COTS selection is Off-The-Shelf Option (OTSO) [4]. Starting from a set of requirements specifying the system component, using the analytic hierarchy process approach, a decision taxonomy and a set of measures is defined to select the most suitable COTS component in a given requirements context. The IusWare methodology [6] is based on the multi-criteria decision aid approach and consists of two main phases: design of an evaluation model and application of the model. The design phases can be broken into: identification of relevant actor, identification of evaluation type, definition of a hierarchy of attributes, definition of the measures, choice of an aggregation technique. A recent approach based on metrics for COTS selection is COTS Acquisition Process (CAP) [7]. This process is made up of three parts: Initialization, Execution and Reuse. The first part deals with the acquisition process planning and its cost estimation. Second part provides guidance for performing the COTS assessment and taking the make-or-buy decision. The third part is responsible for storing all the information gathered for reuse in future COTS acquisition processes.

Provided suitable measurement and comparison criteria are defined, the evaluation attributes can be used for COTS selection purposes. In this phase one of the above methodologies can be adopted.

Experience

The initial assessment of our framework has been performed in a fifth year course in the field of Software Technology at Norwegian University of Science and Technology. In this section we concentrate about the use of our classification proposal in this course. Students were ask to perform two individual assignments, each lasting three weeks: 1. Assignment 1: students are asked to propose a list of COTS components; classify them according to the classes generated by the three attributes in section 2.1; propose a list of at least six evaluation attributes. Students can choose COTS components freely. As a starting point, a list of web sites was provided in [8]. 2. Assignment 2: starting from a list of 18 evaluation students are asked to assign values to these attributes to four COTS components (assigned to each student by the teacher). The 18 evaluation attributes are designed by the teacher and other researchers. At the end of the course there was a workshop during which students worked together with 12 panelists from research centers and universities, 10 being local and two being foreigners. There were 34 students that participated to the project. Each of them proposed a variable number of products, ranging from a minimum of 4 to a maximum of 36. A total of 162 different products were suggested. Table 3 shows the main results of the classification process. The classes that got the most numbers of items were: server-side languages (s, st, e) (23 items), client development tools (c, e, d) (19 items), client execution engines (c, e, e) (19 items), client-side languages (c, st, e) (14 items), server-side engines (s, e, e) (9 items), DBMS (d, e, e) (10 items). Several items could not be fully classified. In particular 57 items get not applicable on the architectural level attribute, 3 get not applicable on the artifact kind attribute, and 6 items get not applicable on the life-cycle phase attribute. Fig. 1 shows the frequency of occurrence of the 18 classes during the classification of the COTS products. The 6 classes collected 90% of items while the remaining 10% was scattered across other 7 classes, which accounted each for a percentage of items ranging from 1% to 3%. Finally 5 classes remained empty, they are (s, sv, e); (s, sv, d); (c, sv, d); (d, e, d); (d, sv, d). Focusing the attention on the empty classes, we can observe that four of them have the sv value for the artifact kind dimension. This may imply that there are few service products available on the marketplace, and thus there is a lack of products in a certain area of software technology, this is a suggestion for new commercial opportunities.

The discussions at the workshop confirmed our hypothesis that the process we followed is useful for COTS product familiarization and that this process may be extended by COTS product evaluation by prototyping. The panelists agreed about the fact that an evaluation database should be considered as a learning medium rather than a final contribution. The main drawback that was highlighted during the workshop about our classification framework was the lack of concern about the domain. In a real world instantiation of the framework the application domain should be taken into account. In a cross-domain context it could appear directly as a dimension; in a more simple case it could appear indirectly by introducing a dimension that deals with the functionalities of the COTS products.
25%

20%

15%

10%

5%

0%

(s,st,e) (c,e,d) (c,e,e) (c,st,e) (s,e,e) (d,e,e) (d,st,e) (c,st,d) (c,sv,e) (s,e,d) (s,st,d) (d,st,d) (d,sv,e) (c,sv,d) (s,sv,d) (s,sv,e) (d,e,d) (d,sv,d)

Fig. 1. Distribution of items

Conclusions

We have proposed a framework for COTS classification, which is based on a set of attributes. This paper proposes three attributes: kind, architectural, level, and phase. These three attributes must be regarded as an example of possible classification and not as the attributes for classification. Classification attributes are different from evaluation attributes. Evaluation and comparison of COTS products can be performed on a homogeneous set of products, i.e. those that classification process put into the same class. COTS classification is useful for several purposes. First, our classification attributes lead to classes, some of which denote well known classes of elements, like for example the class of all browser, while some other are empty classes. Empty classes lead to the research question Why is this class empty? A class may be empty because existing COTS have been forgotten or because there do not exist COTS with those characteristics on the market. If such COTS do not exist, an obvious reason may be that that class does not make sense. A more interesting reason may be that COTS like that have not been yet developed, and this may open for new research. One can for example ask: Is the class of tools for development of server level standard not interesting?

Table 3: Items and their classification.

Classification archi kind phase s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st e s st d s e e s e e s e e s e e s e e s e e s e e s e e s e e s e e s e d d sv e d st e d st E d st e d st d d e e d e e d e e d e e d e e d e e d e e d e e d e e d e e c sv e c st e c st e c st e c st e c st e c st e c st e c st e c st e c st e c st e c st e c st e c st e c st d c st d c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e c e e Class Name Size Paint Shop Pro 7.0 MacroMedia Flash Amaya Macromedia Dreamweaver Macromedia Fireworks WAP Emperor WAP Pro Java ME compiler SmartPhone Emulator Macromedia homesite Java ME wl. Emulator MS emb. VC++ Emulator Code Warrior for Palm IBM perfect photo Java ME wl. Compiler MS Frontpage Netscape Composer Oracle think9i Palm Mobile iNet kit AltaVista Mobile Access C++ Java Speech MS ActiveX XML DTD UML server side engines Rational RUP XMI c c c c c c c c c c c c c c c c c c c e e e e e e e e e e e e e e e e e e e sv st st st st st st st st d d d d d d d d d d d d d d d d d d d e e e e e e d d d Standards

COTS Product CORBA Perl Java Language SMIL MS ASP Java Servlet Java Beans Java RMI Java Server Pages MS DCOM CGI CORBA IIOP spec. Java EJB SOAP Ada Language Java Connector Java Message Queue Java NDI Java SSE PHP RPC SSH XQL Java IDL Orion Application Server Oracle Application Server Sybase Adaptive Server IBM HTTP Server MacroMedia ColdFusion Srv. Apache HTTP server Jigsaw MS Biztalk server 2000 MS Exchange 2000 ORBacus IBM Websphere Jasmine asp Java JDBC SQL 91 MS ODBC Clustra C++ API IBM DB2 Oracle DBMS Sybase DBMS MS SQL Server MS Access Borland Interbase DBMS Clustra DBMS MS project central db MySQL Sybase Industry Warehouse Yospace HTML XHTML Java Applet Dynamic HTML WML CSS Java ME MIDP MS Pocket PC Jscript Java Phone JavaScript MacroMedia ColdFusion ML MathML WebTV XSLT Java 3d MS Pocket PC PoketC Winamp Opera MS Internet Explorer Lynx Fetch! MacroMedia Shockwave Neoplanet Java ME runtime Java Plugin Netscape Communicator MS Pocket PC IE Palm Reader Acobat Reader Java ME preverifier MS Media Player MS Pocket PC Media Player Open SSH Palm Multimail Shoutcast

server side languages

client development tools

-,sv,e programming languages

19 1

development standards

(s, st, d)

23 1

(d, st, e) (d, st, d)

3 1

DBMS

executable components

(s, e, d) (d, sv, e)

9 1 1

(c, sv, e)

10 1

14 (c, st, d) 2

client execution engines

XML MS Excel MS Office MS Word MS Office XP MS Outlook Age of Empire Java HotSpot Runtime Java VM MasterBooter MS Powerpoint MS RegClean Oracle 8i Rational Rose Argo UML MS Visual C++ Java SDK Together Control Center Java Compiler Java ME SDK MS Visual Studio MS Visual Studio .NET MS emb. VC++ Compiler MS emb. VC++ Editor Rational Purify + Text Pad GNU Emacs IBM VisualAge Java Forte Java IDE Jbuilder Jedit MacroMedia ColdFusion Studio MS emb. Remote Spy++ MS emb. VC++ RegEdit MS Sourcesafe MS Visio MS Visual FoxPro PowerBuilder Rational Apex Rational family Rational Requisite Pro MS Windows IBM OS/2 MS Windows XP MS .NET Palm Oracle

st e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e -

e e e e e e e e e e e e d d d d d d d d d d d d d d d d d d d d d d d d d d d d d -

12

client side languages

development tools

29

-,e,-

-,-,-

19

Other research questions may be generated by looking at the existing or not existing relationships between classes of COTS. Like, are there any relationships between the COTS for client standard development and the COTS for server standard development?. Another interesting consequence of our work is to assess its validity for learning purposes both in academia and in industry. We believe that one gets a deep knowledge of COTS components by taking them in use. Another interesting question is which is the relationship between a classification work like the one we propose and a learning program based on active use of COTS components for, for example, prototype development?. Finally, to our knowledge, there is no work in the literature that either proposes the distinction of classification and evaluation attributes or emphasizes the benefits of classification for learning new technologies. Further steps will be. First, for each of the available class, study the evaluation data and try to make comparisons, if possible, among the items of each class. This may lead to further evaluation steps to improve the quality of our evaluation data. Second, change the classification attributes, like making domain as a classification attributes. This will lead to different classes. For the new classes, comparisons among items in single classes will be made. Finally, we have to add several COTS components to our pool of components and eventually remove some items, or even classes of items if we discover that they cannot be regarded as COTS components.

References
1. 2. 3. 4. Boloix, G., Robillard, P.N.: A software system evaluation framework. IEEE Computer, 28(10): 1726, December 1995. Carney, D., Long, F.: What Do You Mean by COTS? Finally a Useful Answer. IEEE Software, 17(2): 83-86, March/April 2000. Glass, R., Vessey, I.: Contemporary Application Domain Taxonomies. IEEE Software, 12(4): 6376 July 1995. Kontio, J.: A Case Study in Applying a Systematic Method for COTS Selection. Proc. of IEEE-ACM 18th Int. Conf. on Software Engineering (ICSE), pp 201-209, Berlin, Germany, March 25-29, 1996. Morisio, M., Torchiano, M.: Definition and Classification of COTS: a proposal. Proc. of International Conference on COTS Based Software Systems (ICCBBS), pp 165-175, Orlando (FL), February 4-6, 2002. Morisio, M., Tsoukis, A.: IusWare: A methodology for the evaluation and selection of software products. IEE Proceedings Software Engineering, 144(3):162-174, June 1997. Ochs, M.A.; Pfahl, D.; Chrobok-Diening, G.; Nothhelfer-Kolb, B.: A Method for Efficient Measurement-based COTS Assessment ad Selection Method Description and Evaluation Results. Proc. IEEE 7th International Software Metrics Symposium, pp 285296, London, England, 4-6 April 2001. SIF80AT - A course in new software technologies, home page at http://www.idi.ntnu.no/emner/sif80at/

5.

6. 7.

8.