2009 Second International Conference on Computer and Electrical Engineering
An Investigation of the Weaknesses of the ISO 9126 International Standard
Rafa E. Al-Qutaish
Al Ain University of Science and Technology - Abu Dhabi Campus,
Abu Dhabi, P.O. Box: 112612, UAE
e-mail: rafa@ieee.org
AbstractSince 2005 and up-to-date, the International
Organization for Standardization (ISO) is ongoing to update
the current ISO 9126 international standard on software
product quality measurement. However, this current standard
will be replaced by the quality measurement division (ISO
25020, ISO 25021, ISO 25022, ISO 25023, and ISO 25024) of
the upcoming ISO 25000 series of international standards on
software quality requirements and evaluation (SQuaRE). This
paper presents an investigation of the weaknesses of the
current ISO 9126 by collecting various viewpoints of interested
researchers and practitioners. Furthermore, this investigation
will tackle the potential solutions of such weaknesses to be
taken into account when preparing the new quality
measurement division of the ISO 25000 series of international
standards.
Keywords-ISO 9126; ISO 25000; software quality; software
product
I. INTRODUCTION
In 1991, ISO published its first international consensus
on the terminology for the quality characteristics for software
product evaluation, which is called ISO 9126 on Software
Product Quality Characteristics and Guidelines for their Use
[1].
Afterwards, from 2001 to 2004, ISO published an
expanded four-part version, containing both the ISO quality
models and inventories of proposed metrics for these models.
The current version of the ISO 9126 series now consists of
one International Standard [2] and three Technical Reports
[3-5].
Currently, ISO has recognized a need for further
enhancements to ISO 9126, primarily as a result of advances
in the field of information technologies and changes in
environment [6]. Therefore, ISO is now working on the next
generation of software product quality standards which will
be referred to as Software Product Quality Requirements and
Evaluation (ISO 25000). This series of standards will replace
the current ISO 9126 and ISO 14598 series, and will consist
of five divisions [7], each of them may contain one or more
documents:
ISO 2500n on Quality Management.
ISO 2501n on Quality Model.
ISO 2502n on Quality measurement.
ISO 2503n on Quality Requirements.
ISO 2504n on Quality Evaluation.
In this paper we will investigate the weaknesses of the
current ISO International standard (ISO 9126). However, we
978-0-7695-3925-6/09 $26.00 2009 IEEE
DOI 10.1109/ICCEE.2009.83
will collect a set of weaknesses from our previous work [8,
9, 10, 11, 12, 13] and from other researchers and
practitioners [14, 15, 16, 17, 18, 19, 20]. Furthermore, this
paper will provide a set of proposed suggestions to overcome
these weaknesses. Thus, these proposed suggestions can be
taken into account on the development of the up-coming
quality measurement division (ISO 25020, ISO 25021, ISO
25022, ISO 25023, and ISO 25024) of the ISO 25000
(SQuaRE: Software Product Quality Requirements and
Evaluation) International series of standards.
The rest of this paper is organized as follows: section 2
presents an overview of the ISO 9126 International Standard
and its contents. Section 3 provides a general overview of the
quality measurement division of the ISO 25000 series os
international standards. Section 4 analyses and investigates
the weaknesses of the ISO 9126. Finally, section 5 concludes
the paper and gives some suggestions for the improvement of
the upcoming ISO 25020n on quality measurement which is
a part of the ISO 25000 series of international standards.
II. ISO 9126: AN OVERVIEW
The ISO 9126 series of standards now consists of one
international standard (IS) and three technical reports (TR),
that is:
ISO IS 9126-1 on Software Product Quality Model
[2].
ISO TR 9126-2 on Software Product External
Quality Metrics [3].
ISO TR 9126-3 on Software Product Internal Quality
Metrics [4].
ISO TR 9126-4 on Software Product Quality in-Use
Metrics [5].
The first document of the ISO 9126 series (Quality
model) contains two-parts quality model for software
product quality [5]:
Internal and External Quality Model.
Quality in-Use Model.
The first part of the two-parts quality model determines
six characteristics in which they are subdivided into twenty-
seven sub-characteristics for internal and external quality [2].
These sub-characteristics are a result of internal software
attributes and are noticeable externally when the software is
used as a part of a computer system, see Figure 1. Whereas,
the second part of the two-part model indicates four quality
in-use characteristics [2], see Figure 2. However, the
definitions of each of these characteristics and sub-
characteristics are available in the ISO 9126-1 [2].
275
 Figure 1. ISO 9126 quality model for external and internal quality (characteristics and sub-characteristics).
Figure 2. ISO 9126 quality model for quality in-use (characteristics).
The second [3], third [4] and fourth [5] parts of the ISO
9126 series provide the following information:
Sets of metrics for each external quality sub-
characteristic, internal quality sub-characteristic and
quality in-use characteristic.
Explanations of how to apply and use these sets of
metrics.
Examples of how and where to apply these metrics
during the software product lifecycle.
III. THE QUALITY MEASUREMENT DIVISION OF ISO
25000: AN OVERVIEW
One of the objectives of this new ISO 25000 series (and
what makes it different from the current ISO 9126 series) is
the harmonization of its contents with the software
measurement terminology of ISO 15939 [10] which itself
based on the ISO metrology terminology [21].
Figure 3 shows the structure of the quality measurement
division (ISO 2502n) series that is to replace the current
four-part ISO 9126 series of standards [22]. This proposed
276
quality measurement division (ISO 2502n) will consist of
five documents:
ISO 25020 on Measurement Reference Model and
Guide.
ISO 25021 on Quality Measure Elements.
ISO 25022 on Measurement of Internal Quality.
ISO 25023 on Measurement of External Quality.
ISO 25024 on Measurement of Quality in Use.
The work on ISO 25000 is being carried out by Working
Group 6 (WG6) of the software and system engineering
subcommittee (SC7) of the ISO/IEC joint technical
committee (JTC1) on Information Technology, that is,
ISO/IEC JTC1/SC71.
1
The author is a Canadian representative and committee
member of the ISO/IEC Joint Committee 1 Subcommittee
7 (JTC1/SC7) Work Group 6 on Software Evaluation and
Measurement.
 Figure 3. The structure of the measurement division (ISO 2502n series).
IV. THE WEAKNESSES OF ISO 9126
In this section we will discuss the weaknesses of the ISO
9126. However, these weaknesses will be classified into two
classes, that is, the weaknesses which we have identified
during our previous work, and the weaknesses which are
discovered by other researchers and practitioners.
A. Self-Discovered Weaknesses
During our previous research, we have found the
following weaknesses of ISO 9126:
1) The terminology needs to be harmonized with the
ISO International Vocabulary of Basic and General
Terms in Metrology [8].
2) The metrics in ISO 9126 parts 2, 3, and 4 are
completely merged and need to be classified into
base and derived metrics to be aligned with the ISO
15939 [9].
3) It is to be noted that the ranges of the results of many
of the metrics in ISO 9126 are between 0 and 1.
Therefore, it is easy to convert them to percentage
values. However, from our point of view, these
results would be easier to understand if they were
ranked in terms of qualitative values; for example,
for task completion, if the percentage result is 100%,
then the completion of the task is labeled excellent;
if the result is 80%, then the completion of the task is
labeled very good; and so on [10, 13].
4) When mapping the ISO 9126 to ISO 12207, many of
the ISO 9126 quality metrics referred to ISO 12207
processes. However, as known, each process in ISO
12207 contains a number of different activities.
Thus, it is more usable for the ISO 9126 users to
refer to the activities of the ISO 12207 [11, 12].
B. The Weaknesses Discovered by other Researchers
The following are the weaknesses which have been
identified by other researchers:
1) Jung et al. [14] stated that including the six
compliance sub-characteristics leads to two possible
277
results. The first possibility is that the compliance
sub-characteristic for each characteristic belongs to
its characteristic as defined in ISO 9126. The second
possibility is that the six compliance sub-
characteristics constitute a separate dimension [14].
Furthermore, their results reveal ambiguities in the
way that ISO 9126 is structured in terms of
characteristics and sub-characteristics.
2) Chua and Dyson [15] - in their investigation -
uncovered some inherent weaknesses in the ISO
9126 quality model, particularly with regards to the
Usability characteristic. To make the model simpler
to use for educators, who may not be usability
experts, they propose that this characteristic should
be extended to include more specific factors such as
consistency, simplicity, legibility and color use. It is
also suggested that a Help sub-characteristic be
included as part of Usability, mainly to ensure that
this important factor should not be neglected. In
addition, we propose the inclusion of user
satisfaction as a global characteristic to summarize
the general impact of the system on the user in their
specific educational context and given their specific
requirements. With these improvements, ISO 9126
could be a useful model for evaluating the quality of
e-learning systems [14].
3) Based on the ISO 9126 standard, as well as an
investigation of other models proposed by various
Quality Measure Elements (ISO 25021)
Measurement of External Quality (ISO 25023)
Measurement of Internal Quality (ISO 25022)
Measurement of Quality in Use (ISO 25024)
easurement Reference Model and Guide (ISO 25020)
authors, Abran et al. [16] stated that revised structure
for a usability model need to be proposed, including
the addition of two characteristics to the ISO 9126
standard which had been selected as their baseline:
learnability and security. Then, specific metrics for
 these characteristics were presented using the
structured hierarchy of ISO 9126 [16].
4) Begh [17] stated that he could not apply the ISO
9126 metrics directly because they are imprecise. In
addition, the project team in his study devoted
considerable effort to redefining the metrics and
applying the formalized approach with counting rules
and profiles [17]. The complex attributes based on
standards required a careful examination of the
standards, but they eventually turned out to be easier
to define [17].
5) Koscianski and Costa [18] stated that the
normalization of metrics and scores must accomplish
the following: every metric used must be mapped to
the same reference scale [18]. The sample metrics of
ISO/IEC 9126-2 and ISO/IEC 9126-3 generally
return values in the range [0, 1], where 1 means the
best possible result. Furthermore, qualitative metrics
may use the same concept [18].
6) Behkamal et al. [19] stated that the ISO 9126 quality
model is a general one. In addition, they noted that in
order to apply this standard, it needs to be
customized [19].
7) Erazo [20] has analyzed the ISO 9126 quality model
and presented the results of an actual evaluation,
illustrating each characteristic with their respective
valued aspects.
V. CONCLUSION
Nowadays, ISO is ongoing to update the current ISO
9126 international standard on software product quality
measurement. However, this current standard will be
replaced by the upcoming quality measurement division
(ISO 25020, ISO 25021, ISO 25022, ISO 25023, and ISO
25024) of the ISO 25000 series of international standards on
software quality requirements and evaluation (SQuaRE).
This paper presented an investigation on the weaknesses
of the current ISO 9126 international standard by collecting
different viewpoints from our previous researches and from
interested researchers and practitioners. However, this paper
has tackled the following suggestion to resolve such
weaknesses:
Ensure that the terminology on software product
quality measurement is fully aligned with the classic
measurement terminology in the sciences and in
engineering.
Identify and classify the ISO 9126 metrics into base
and derived metrics to make it easy to determine
which base metrics should be collected to be used
subsequently in computing the other derived metrics.
Convert the result of the ISO 9126 metrics to
percentage values to be more understandable.
To overcome the problem on the mapping between
ISO 9126 and ISO 12207, a cross-reference numbers
from ISO 12207 can be used. For example, the
cross-reference number 5.3.9 is referring to primary
278
processes, development process and software
qualification testing activity, respectively, Fig. 3 for
an example on where the ISO 9126 internal quality
metrics could be measured within the ISO 12207
verification activities.
Ignore the compliance sub-characteristic for each
characteristic in ISO 9126 quality model.
Additional empirical work is necessary for clarity
and validates the ISO 9126 quality model structure
in terms of characteristics and sub-characteristics.
The usability characteristic should be extended to
include more specific sub-characteristics such as
consistency, simplicity, legibility, color use, and
help.
Include the addition of two characteristics to the ISO
9241-11 standard which had been selected as their
baseline: learnability and security. Then, specific
metrics for these characteristics need to be presented
using the structured hierarchy of ISO 9126.
Redefine the ISO 9126 metrics and apply a
formalized approach with counting rules and profiles
to the ISO 9126 metrics.
Map each of the ISO 9126 metrics to the same
reference scale.
Make the ISO 9126 be ready for customization to be
used in any domain.
The ISO 9126 quality model characteristics need to
be ranked by their usability and importance.
REFERENCES
[1] ISO, Software Product Evaluation - Quality Characteristics
and Guidelines for Their Use (ISO/IEC IS 9126). Geneva,
Switzerland: International Organization for Standardization
1991.
[2] ISO, Software Engineering - Product Quality - Part 1: Quality
Model (ISO/IEC IS 9126-1). Geneva, Switzerland:
International Organization for Standardization, 2001.
[3] ISO, Software Engineering - Product Quality - Part 2:
External Metrics (ISO/IEC TR 9126-2). Geneva, Switzerland:
International Organization for Standardization, 2003.
[4] ISO, Software Engineering - Product Quality - Part 3: Internal
Metrics (ISO/IEC TR 9126-3). Geneva, Switzerland:
International Organization for Standardization, 2003.
[5] ISO, Software Engineering - Product Quality - Part 4: Quality
in-Use Metrics (ISO/IEC TR 9126-4). Geneva, Switzerland:
International Organization for Standardization, 2004.
[6] M. Azuma, "SQuaRE: The Next Generation of ISO/IEC 9126
and 14598 International Standards Series on Software Product
Quality," Proc. of the European Software Control and Metrics
Conference (ESCOM01), London, UK, 2001, pp. 337-346.
[7] ISO, Software Engineering - Software Product Quality
Requirements and Evaluation (SQuaRE) - Guide to SQuaRE
(ISO/IEC 25000), Geneva, Switzerland: International
Organization for Standardization, 2005.
[8] A. Abran, R. E. Al-Qutaish, and J.-M. Desharnais,
"Harmonization issues in the updating of the ISO standards on
software product quality," Metrics News: Journal of the
Software Metrics Community, Vol. 10, No. 2, 2005, pp. 35-
44.
[9] A. Abran, R. E. Al-Qutaish, J.-M. Desharnais, and N. Habra,
"An information model for software quality measurement
with ISO standards," Proc. of the International Conference on
Software Development (SWDC-REK'05), Reykjavik, Iceland,
2005, pp. 104-116.
[10] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego,
"Analysis of the ISO 9126 on Software Product Quality
Evaluation from the Metrology and ISO 15939 Perspectives
WSEAS Transactions on Computers, Vol. 5, No. 11, 2006,
pp. 2778-2786.
[11] R. E. Al-Qutaish, "Measuring the Software Product Quality
during the Software Development Life-Cycle: An
International Organization for Standardization Standards
Perspective," Journal of Computer Science, Vol. 5, No. 5,
2009, pp. 392-397.
[12] R. E. Al-Qutaish and H. Jalfawareh, "Mapping Between ISO
9126 on Software Product Quality Metrics and ISO 12207 on
Software Life Cycle Processes," Proc. of the International
Arab Conference on Information Technology (ACIT'06),
Yarmouk University, Irbid, Jordan, 2006.
[13] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego,
"Investigation of the Metrology Concepts in ISO 9126 on
Software Product Quality Evaluation," Proc. of the 10th
International Conference on Computers (ICComp'06),
Vouliagmeni, Athens, Greece, 2006, pp. 864-872.
[14] H.-W. Jung, S.-G. Kim, and C.-S. Chung, "Measuring
Software Product Quality: A Survey of ISO/IEC 9126," IEEE
Software, Vol. 21, No. 5, 2004, pp. 89-92.
279
[15] B. B. Chua and L. E. Dyson, "Applying the ISO 9126 Model
to the Evaluation of an e-Learning System," Proc. of the 21st
Conference of the Australasian Society for Computers in
Learning in Tertiary Education (ASCILITE'04), Perth,
Australia, 2004, pp. 184-190.
[16] A. Abran, A. Khelifi, W. Suryn, and A. Seffah,
"Consolidating the ISO Usability Models," Proc. of 11th
International Software Quality Management Conference,
Glasgow, Scotland, UK, 2003.
[17] J. Boegh, "Certifying Software Component Attributes," IEEE
Software, Vol. 23, No. 3, 2006, pp. 74 81.
[18] A. Koscianski and J. C. Bracarense Costa, "Combining
Analytical Hierarchical Analysis with ISO/IEC 9126 for a
Complete Quality Evaluation Framework," Proc. of the 4th
IEEE International Symposium and Forum on Software
Engineering Standards (ISESS'99), Curitiba, Brazil, 1999, pp.
218-216.
[19] B. Behkamala, M. Kahanib, and M. K. Akbar, "Customizing
ISO 9126 quality model for evaluation of B2B applications,"
Information and Software Technology, Vol. 51, No. 3, 2009,
pp. 599-609.
[20] S. Riascos-Erazo, "Evaluation of the Quality (norm ISO
9126) in Applications Educational Multimediales,"
FORMATEX Magazine, 2006, Vol. 5, pp. 832-836.
[21] ISO, International Vocabulary of Basic and General Terms in
Metrology (VIM). Geneva, Switzerland: International
Organization for Standardization, 1993.
[22] ISO, Software Engineering - Software Product Quality
Requirements and Evaluation (SQuaRE) - Quality Measure
Elements (ISO/IEC TR 25021). Geneva, Switzerland:
International Organization for Standardization, 2007.