Information & Management 40 (2003) 677–690

The impact of software process improvement on quality:

in theory and practice
Noushin Ashrafi*
Department of MSIS College of Management, University of Massachusetts, 100 Morrisey Blvd., Boston, MA 02125, USA
Accepted 22 August 2002

Abstract

To remain competitive, software companies must establish practices that enhance quality and advance process management.
To this end, they have increasingly turned to software process improvement (SPI) methodologies, of which the ISO 9000
standards and the capability maturity model (CMM) are the best known. The underlying principle of both methodologies is to
assess organizational capabilities to produce quality software, but they depend on different underlying processes.
Whether the practices advocated by these methodologies lead to high-quality software has been the topic of ongoing debates.
Both scholars and practitioners are looking for hard evidence to justify the time and effort required by such guidelines to improve
the software-development process and its end product.
In this paper, we investigate the impact of SPI methodologies on software quality, first by theoretical comparison and then
with empirical data. Our findings reveal that each methodology has had a different level of impact on software quality factors.
These findings could help software-development organizations select the methodology or combination that best meets their
quality requirement.
# 2002 Elsevier Science B.V. All rights reserved.

Keywords: Software process improvement; Software quality assurance; The capability maturity model; ISO 9000-3; Quality factors

1. Introduction Software organizations differ in their strategies for

Despite rapid advances in all facets of technology, the
software industry is still struggling with the formidable
task of developing software applications that meet
quality standards, time pressure, and budget constraints
[26,29]. Cost and time elements are quantitative, and
therefore can be measured and evaluated. Quality on the
other hand, is a multi-dimensional concept, difficult to
define and measure. The focus of this paper is on the
quality aspect of software development.
*
Tel.: þ1-617-287-7883; fax: þ1-617-287-7877.
E-mail address: noushin.ashrafi@umb.edu (N. Ashrafi).
improvement of the quality of their products. In the
past, looking for a quick solution, some organizations
have tried to use special packages or approaches to
their software-development processes, hoping to get
quality software products out fast and within budget.
Since these efforts did not deliver as promised, the
slower paced and methodical approaches, such as
software process improvement (SPI), started to gain
momentum [13].
Two established sources, the ISO 9000-3 standards
and the capability maturity model (CMM), have pro-
vided guidance for software-development companies.
While these SPI techniques may be considered main-

0378-7206/$ – see front matter # 2002 Elsevier Science B.V. All rights reserved.
PII: S 0 3 7 8 - 7 2 0 6 ( 0 2 ) 0 0 0 9 6 - 4
678 N. Ashrafi / Information & Management 40 (2003) 677–690
stream approaches in the software industry, their
deployment is expensive and time consuming. There-
fore, organizations are looking for hard evidence to
justify the time and effort required to use these guide-
lines to improve the software-development process
and thus hopefully the end product.
The purpose of this paper is to examine SPI meth-
odologies, specifically ISO 9000-3 and the CMM, to
find out to what extent they address quality factors,
and once implemented, their impact on various quality
factors. This line of analysis should be of value in
determining which SPI methodology should be
adopted to build quality into the software-develop-
ment process.
2. Background
The elusive concept of software quality has been the
topic of debate since computers were first invented
[19]. Manufacturing quality concepts initiated by
Deming [5] and continued by Juran and Gryna [17]
such as conformance for requirements and fitness for
customer use, have been applied to software products.
Pressman [30] and Humphrey [14] showed quality as a
multi-dimensional concept perceived differently in
different domains. Fournier [9] has indicated that,
‘‘quality very often signifies different things to differ-
ent people in the context of a software system.’’
To illustrate various aspects of software quality,
software professionals pioneered a quality model that
attempted to identify factors representing the beha-
vioral characteristics of the software system. Each
factor was subsequently associated with at least two
or more software quality attributes used to enforce the
definition of a specific quality factor [23]. Throughout
the years, experts have come up with different versions
of the model. While varying in detail, all versions lay a
foundation to cover all dimensions of quality for a
software product, ensuring that quality is built into the
software system package being created [7,10,11,27].
The Software Quality Assurance (SQA) group, which
is a professional organization founded to promote
the quality assurance profession through proliferation
and advancement of high professional standards,
embraced this framework as a guideline to judge
the quality of a software system. Table 1 shows a
version of the software quality model printed in the
Table 1
Software quality factors
Quality of design
Correctness Extent to which the software conforms
to its specifications and conforms to its
declared objectives
Maintainability Ease of effort for locating and fixing a
software failure within a specified time period
Verifiability Ease of effort to verify software features and
performance based on its stated objectives
Quality of performance
Efficiency Extent to which the software is able to do
more with less system (hardware, operating
system, communications, etc.) resources
Integrity Extent to which the software is able to
withstand intrusion by unauthorized
users or software within a specified
time period
Reliability Extent to which the software will
perform (according to its stated objec
tives) within a specified time period
Usability Relative ease of learning and the
operation of the software
Testability Ease of testing to program to verify that
it performs a specified function
Quality of adaptation
Expandability Relative effort required to expand soft
ware capabilities and/or performance
by enhancing current functions or by
adding new functionality
Flexibility Ease of effort for changing the soft
ware’s mission, functions or data to
meet changing needs and requirements
Portability Ease of effort to transport software to
another environment and/or platform.
Reusability Ease of effort to use the software (or its
components) in another software
systems and applications
Interoperability Relative effort needed to couple the
software on one platform to another
software and/or another platform
Intra-operability Effort required for communications between
components in the same software system
Source: the Handbook of Software Quality Assurance, Prentice
Hall, 1998.
handbook of SQA. This model identifies 14 quality
factors in three stages of the development life cycle:
design, performance, and adaptation. Although it is
difficult, if not impossible, to develop a piece of
software that adheres to all of these 14 factors, the
model provides a good frame of reference to under-
stand software quality. We used this quality model
throughout the study.
 N. Ashrafi / Information & Management 40 (2003) 677–690
Finding a way to improve software quality for all
software-development organizations is not possible;
what is considered the ultimate target for quality in
one organization may not be important to another; for
example, an organization that produces mission cri-
tical software, considers reliability to be the most
important factor while, portability may be a necessity
for organization that produces a software product for a
variety of platforms. Obviously, increasing some qual-
ity factors may cause others to decline.
What an experienced software developer wants is
an approach that reflects his or her organization and its
goals. Organizations that find themselves in a process
of choosing a methodology to improve their quality
management system face a dilemma: which, if any of
these methodologies meet their quality requirements
[8].
Also, there has been some concern about the lack of
hard evidence that SPI methodologies do indeed
improve the quality of software products [16,24,28,31].
Recently, a number of empirical studies addressed
the effect of the CMM or ISO device on quality. These
studies, however, measure software quality as reduced
number of defects and less rework, hence offering a
narrow definition of quality. CMM specifies 18 key
process areas (KPAs) categorized into five process
maturity levels. Studies have shown that a higher
maturity level leads to increased productivity and
quality, defined as reduced defects and rework
[12,15,21,22]. A similar study investigated the emer-
ging ISO/IEC 15504 international standard on soft-
ware process assessment [6]. Their findings indicate
that improvements in process capability are associated
with reduction in rework.
The Software Engineering Institute’s report
recently summarized the observations from a 1999
survey of high maturity CMM-based efforts. It basi-
cally provided a perspective on the practices of high
maturity level organizations regarding their manage-
ment, engineering, tools, and technology [4].
Several other studies have been conducted to com-
pare and contrast ISO 9000-3 and CMM. Most have
focused on one-on-one comparison of CMM key
process areas and ISO 9000-3 requirements [1–
3,20,25,33]. Although very useful in answering ques-
tions such as ‘‘Is complying with a certain level of
CMM equivalent to complying with ISO standards?,’’
they do not assess and compare the impact of the two
679
approaches. In fact, there has apparently been no effort
to investigate SPI methodologies on various dimen-
sions of software quality, such as design, performance,
and adaptation.
In our attempt to do this, first we need to compare
and contrast SPI and SQA approaches to the devel-
opment process and how they define and embrace
quality issues.
3. Comparing SPI with the SQA factors
Although both SQA guidelines and SPI methodol-
ogies more or less claim to address the same subject B,
quality assurance (QA) B, they attempt to do so from
different starting points.
SQA treats processes as part of a ‘‘package’’ with
factors and attributes that can be simply illustrated in
tables and cross-charts. Fournier notes that, ‘‘Quality
. . . encompasses the complete system package being
created, including its supportive elements, such as
system documentation, staff training, and even post-
installation support processes B, in other words, all
those items that meet the expectations of our custo-
mers . . . in addition to the software product.’’
In contrast, SPI sees the development process as the
deciding factor in software improvement and tends to
take most other (but by no means all) factors from
outside the domain of QA. For example, CMM
describes the principles and practices intended to help
organizations improve the maturity of their software
process development through an evolutionary path
from ad hoc and chaotic to mature and disciplined
[34]. A formal definition given by the Software Engi-
neering Institute (SEI) shows the emphasis on process
rather than product: ‘‘the capability maturity model
summarizes the key practices of a key process area and
can be used to determine whether an organization or
project has effectively implemented the key process
areas.’’
ISO 9000-3 guidelines emphasize consistency
through extensive documentation. It states, ‘‘ . . . this
part of ISO 9000 is intended to describe the suggested
controls and methods for producing software which
meets a purchaser’s requirements. This is done pri-
marily by preventing non-conformity at all stages
from development through to maintenance.’’ [18].
Though expectations may have been heightened by
680 N. Ashrafi / Information & Management 40 (2003) 677–690

the title of ISO 9000-3 ‘‘quality management and
quality assurance standards: guidelines for the appli-
cation of ISO 9001 to the development, supply and
maintenance of software,’’ it has a very narrow focus,
as spelled out in its Section 1: ‘‘this part of ISO 9000
deals primarily with situations where specific software
is developed as part of a contract according to pur-
chaser’s specifications.’’
Both of these definitions lead us to believe that SPI
models tend to reflect a developer’s view, while SQA
reflects more of a user’s view. This contrast creates a
Heisenberg effect: following one or the other of these
approaches, you concentrate either on development
processes, or the external attributes of the product, but
not on both at the same time.
Basically, SPI methodologies provide a generic
recipe for improving the software-development pro-
cess. This would have a better chance of raising
product improvement if we could establish a frame-
work within which improvement can be tailored to
meet business needs and organizational quality
requirements.
Questions such as: ‘‘Do process improvement activ-
ities reflect the understanding of the relationship
between process characteristics and quality improve-
ment goals?’’ have remained unanswered. To address
such concerns and specifically to identify the strengths
and weaknesses of SPI methodologies as they relate to
quality, we mapped software quality factors to ISO
clauses and key processes of CMM.
This mapping provided theoretical evidence that
ISO and CMM cover quality factors as described in the
handbook of SQA. The next step for this study was
to find out whether there is a correlation between
theory and practice when deploying SPI methodolo-
gies. Hence, we conducted an empirical study, in
which we were surveying the users of these meth-
odologies (software developers) and asking for their
perception of the impact of SPI methodologies on
quality factors.
For our theoretical assessment, we used SQA as the
initial basis for our mapping. Quality factors are
divided into three categories: design, performance,
and adaptation. Table 2 depicts the mapping of quality
factors affecting design to ISO standards and CMM
key process areas.
Thus, ISO 9000-3 covers all three factors that
improve the quality of design whereas CMM falls
short of addressing maintainability. It should be noted
that ISO uses a more direct language to address
correctness and verifiability then CMM.
Next, we examined the coverage of quality factors
for performance by ISO 9000-3 clauses and CMM
process areas. Table 3 demonstrates the mapping of
these quality factors.
All factors for the quality of performance are
addressed both by ISO 9000-3 and CMM, but the
language used is not as clear as that for quality of
design. For example, both methodologies cover effi-
ciency; however, they refer to efficiency of process
rather than efficiency of product. This raised the
question: Does improving the efficiency of process
improve the efficiency of product? Both address
usability through training, rather than ease of use of

Table 2
Quality of design

Quality factors ISO standards CMM key process areas: goals

Correctness: extent to which the software
conforms to its specifications and conforms
to its declared objectives
Maintainability: ease of effort for locating
and fixing a software failure within a
specified time period
Verifiability: ease of effort to verify
software features and performance based
on its stated objectives
5.9.2: provisions should be made for
verifying the correctness and completeness
of copies of the software product delivered
5.6.2: subsequent processes: the product
should be designed to the extent practical
to facilitate testing, maintenance and use
5.4.6: the supplier should draw up a plan
for verification of all development phase
outputs at the end of each phase
Product engineering: consistently performs
a well-defined engineering process that
integrates all the software engineering
activities to produce correct, consistent
software products effectively and efficiently
Not available
Project tracking and oversight: establishes
a common understanding between the
customer and software project of the
customer’s requirements that will be
addressed by the software project
 N. Ashrafi / Information & Management 40 (2003) 677–690 681

Table 3
Quality of performance

Quality factors ISO standards CMM key process areas: goals

Efficiency: extent to which the software
is able to do more with less system
(hardware, operating system,
communications, etc.) resources
Integrity: extent to which the software
is able to withstand intrusion by
unauthorized users or software
within a specified time period
Reliability: extent to which the software
will perform (according to its stated
objectives) within a specified time
period
Usability: relative ease of learning
and the operation of the software.
Testability: ease of testing to program
to verify that it performs a
specified function
5.4.1: the organization of the project resources,
including the team structure, responsibilities,
use of sub-contractors and material resources
to be used
5.3.1: . . ., the supplier should have a complete,
unambiguous set of functional requirements; in
addition, these requirements . . . include, but
are not limited to, the following:
performance, safety, reliability, security,
and privacy; these requirements should be
stated precisely enough so as to allow
validation during product acceptance
5.3.1: . . ., the supplier should have a complete,
unambiguous set of functional requirements. In
addition, these requirements . . . include, but
are not limited to, the following: performance,
safety, reliability, security, and privacy; these
requirements should be stated precisely
enough so as to allow validation
during product acceptance
6.9: the supplier should establish and
maintain procedures for identifying the
training needs and provide for
training of all personnel performing activities
affecting quality
5.7: testing may be required at several levels
from the individual software item to the
complete software product, there are several
different approaches to testing and integration
Product engineering: consistently performs
a well-defined engineering process that
integrates all the software engineering
activities to produce correct, consistent
software products effectively and efficiently
Configuration management: establishes and
maintains the integrity of the products of
the software project throughout the project’s
software life cycle
Project tracking and oversight: establishes
a common understanding between the
customer and software project of the
customer’s requirements that will be
addressed by the software project
Training program: the purpose of training
program key process area is to develop
skills and knowledge of individuals so
they can perform their roles effectively
and efficiently
Peer reviews: the purpose of peer review is
to remove defects from the software
work products early and efficiently. . .

the product. Although the ultimate goal is the same,
enhancing the ability to use the product via training
focuses on the users, whereas ease of use focuses on
the product. Table 4 shows the mapping of the quality
factors for adaptation to CMM key process areas and
ISO standards.
Comparatively, ISO 9000-3 guidelines provide bet-
ter coverage. However, the ISO 9000-3 clauses that
cover the quality factors for adaptation and perfor-
mance are not as strong as those for design.
The mapping revealed that the coverage of quality
factors by the two SPI methodologies ranges from
explicit to implicit or not covered at all. For example,
both ISO and CMM cover design factors explicitly,
except for maintainability. Both cover all of the
performance factors, but the level of coverage is
mixed. ISO addresses adaptation implicitly except
for portability. CMM does not address adapta-
tion factors except for flexibility, which is covered
lightly.
4. The empirical study
To find out the consequences of their coverage in
practice, and to attempt to assess whether using these
methodologies does indeed improve the quality of a
piece of software, we embarked upon an empirical
study. We targeted developers who have used CMM
and ISO and asked them to evaluate the impact of SPI
on the quality of the design, performance, and adapta-
tion of their software products. We compared their
responses on the basis of the type of SPI methodology
they used.
682 N. Ashrafi / Information & Management 40 (2003) 677–690

Table 4
Quality of adaptation

Quality factors ISO standards CMM key process areas: goals

Expandability: relative effort required to 5.10.1.c: functional expansion or performance NA

expand software capabilities and/or
performance by enhancing current
functions or by adding new functionality
Flexibility: ease of effort for changing the
software’s mission, functions or data to
meet changing needs and requirements
Portability: ease of effort to transport
software to another environment and/or
platform
Reusability: ease of effort to use the software
(or its components) in another software
systems and applications
Interoperability: relative effort needed to
couple the software on one platform to
another software and/or another platform
Intra-operability: effort required for
communications between components in
the same software system
improvement: functional expansion or
performance improvement of existing
functions may be required by the purchaser
in the maintenance stage
5.10.7: the supplier and purchaser should Project tracking and oversight:
agree and document procedures for establishes a common understanding
incorporating changes in a software between the customer and software
product resulting from the need to project of the customer’s requirements
maintain performance that will be addressed by the
software project
NA NA
5.6.2.c: use of past design experience: NA
utilizing lessons learned from past design
experiences, the supplier should avoid
recurrence of the same or similar problems
6.1.3.1.c: (procedures should be applied to NA
ensure that) all interfaces to other software
items and to hardware (can be identified for
each version of the software)
5.7.3.b: any discovered problems and their NA
possible impacts to and other parts of the
software should be noted and those
responsible notified

We selected four professional organizations in New In the second part of the survey, we asked questions
England and distributed our survey to their members: about the impact of SPI methodologies on quality; for
The SPI Network (SPIN), The Association of Com- design, this is determined by its correctness, main-
puting Machinery (ACM) Boston Chapter, The Amer- tainability, and verifiability. To analyze the impact of
ican Society for Quality (ASQ) of Massachusetts, and the SPI methodologies, we asked the respondents to
The New England Software Quality Assurance Forum rank their perceived impact from very low ¼ 1,
(NESQAF). The method of data gathering was that of low ¼ 2, average ¼ 3, high ¼ 4, to very high ¼ 5.
semi-structured interviews: the surveys were handed Fig. 1 shows histograms indicating the proportion of
out in person, and questions were answered as they developers that perceive the impact of the methodol-
were raised. We approached more than 200 members ogy they use on design factors from very low to very
of these organizations and screened out those who high.
were not part of a SW development team. Our findings
and analysis are based on the 67 responses that we Table 5
collected. Percentage of respondents using a SPI methodology
The first part of the questionnaire was designed to
ISO CMM ISO & Other Total
identify the users and non-users of SPI methodologies. CMM
We separated projects that were developed using an
SPI 16% 10% 12% 22% 60%
SPI methodology from those that were not. Table 5
Non-SPI 40% 40%
depicts this distribution.
 N. Ashrafi / Information & Management 40 (2003) 677–690 683

Fig. 1. Impact of software process improvement methodology on the quality of design.

Overall, this shows that our respondents perceived
the impact of using ISO on design factors much
stronger than that of CMM. They reported, however,
that when companies adopt both ISO and CMM a
higher level of correctness, maintainability, and ver-
ifiability is achieved.
Histograms provide a graphical description of our
respondents’ perception; however, they do not tell the
whole story. For example, while the mode shows the
highest frequency for a level of perception, it does not
measure variability. Numerical descriptive measures,
such as mean and standard deviation provide a mea-
sure of central tendency and of variability, respectively
hence providing additional insight into comparing the
methodologies. Low variability indicates agreement
between respondents on the level of impact and man-
ifests predictability, which is an important factor in
selecting a methodology. Table 6 shows the average
684 N. Ashrafi / Information & Management 40 (2003) 677–690

Table 6 considered the best. Obviously, deploying both ISO

Mean and standard deviations for design quality factors
9000-3 and CMM is costly and time consuming, and
Correctness Maintainability Verifiability only companies who are truly committed to improving
the development process would use both. In practice,
ISO SB ¼ 3.64 SB ¼ 3.27 SB ¼ 3.50
SS ¼ 1.21 SS ¼ 1.19 SS ¼ 1.27 the developers of mission critical software, where
correctness is of crucial importance use both meth-
CMM SB ¼ 3.44 SB ¼ 3.33 SB ¼ 3.00
SS ¼ 1.88 SS ¼ 2.12 SS ¼ 2.12 odologies [32]. In our study, 12% of the respondents
reported using both.
ISO and CMM SB ¼ 3.67 SB ¼ 3.67 SB ¼ 3.33
Many companies that cannot afford to deploy both
SS ¼ 1.03 SS ¼ 1.51 SS ¼ 1.21
techniques should select the methodology that meets
Other SB ¼ 3.50 SB ¼ 3.18 SB ¼ 3.25
their requirements. Our study suggests that using ISO
SS ¼ 0.80 SS ¼ 0.98 SS ¼ 0.87
would have a greater impact on the quality of design
than using CMM or other methodologies.
The quality of a software product depends upon a
and standard deviation for design factors for each good design, and good performance. This is measured
group. by the efficiency, integrity, reliability, testability and
The table suggests that the mean values for four usability of a software product. Fig. 2 demonstrates
categories are all above average (3) and quite close. our respondents’ perspectives on the impact of SPI on
Standard deviations provide additional information performance quality factors.
regarding the consistency of the perception among Our data for performance is not as obvious as that
the users of a methodology. The largest variability was for design. Using both ISO and CMM has the greatest
detected among those who use only CMM; whereas impact on reliability and testability, which are two
those who use other, mostly ‘‘home grown,’’ meth- related factors. It seems that other methodologies
odologies were cohesive in their perception of the combined showed the highest impact on efficiency,
impact. and that usability can be improved using ISO alone. It
Fig. 1 and Table 6 are consistent with our mapping is not clear which methodology has the highest impact
for ISO, whose requirements explicitly address the on integrity. Table 7 depicts mean and standard devia-
issues of correctness, maintainability, and verifiability. tion for each factor and clarifies some of the ambiguity
The empirical study indicates that most of the respon- in Fig. 2. While histograms did not indicate which
dents thought that ISO had a high impact on these methodology has the highest impact on integrity,
factors. The results, however, are somewhat incon- Table 7 shows them to be ISO and CMM, thus
sistent for CMM. Although it covers correctness and indicating that the combination is the best.
verifiability and not maintainability, the responses are As we pointed out earlier, performance quality
moderate and close for them. Using both techniques is factors are not addressed as strongly as in design,

Table 7
Mean and standard deviations for performance quality factors

Efficiency Integrity Reliability Usability Testability

ISO SB ¼ 2.64 SB ¼ 2.64 SB ¼ 3.64 SB ¼ 3.73 SB ¼ 3.70

SS ¼ 1.12 SS ¼ 0.81 SS ¼ 1.03 SS ¼ 0.90 SS ¼ 1.06
CMM SB ¼ 2.75 SB ¼ 2.75 SB ¼ 3.67 SB ¼ 3.33 SB ¼ 3.78
SS ¼ 2.05 SS ¼ 2.19 SS ¼ 2.00 SS ¼ 2.00 SS ¼ 1.99
ISO and CMM SB ¼ 2.83 SB ¼ 3.17 SB ¼ 4.00 SB ¼ 2.50 SB ¼ 3.67
SS ¼ 0.75 SS ¼ 0.98 SS ¼ 1.10 SS ¼ 1.22 SS ¼ 1.51
Other SB ¼ 3.18 SB ¼ 3.00 SB ¼ 3.58 SB ¼ 3.42 SB ¼ 3.67
SS ¼ 1.17 SS ¼ 1.18 SS ¼ 1.08 SS ¼ 1.31 SS ¼ 1.30
 N. Ashrafi / Information & Management 40 (2003) 677–690 685

Fig. 2. Impact of software process improvement methodology on the quality of performance.

686 N. Ashrafi / Information & Management 40 (2003) 677–690

Fig. 3. Impact of software process improvement methodology on the quality of adaptation.

except for reliability and testing. Table 7 shows mean and a mixed for usability. In Table 3 the language used
values mostly below average (3) for efficiency and by ISO for addressing efficiency is not very strong,
integrity, above average for reliability and testability, and Fig. 2 indicates the percentage of respondents that
 N. Ashrafi / Information & Management 40 (2003) 677–690 687

Table 8
Mean and standard deviations for adaptation quality factors

Expandability Flexibility Portability Reusability Interoperability Intra-operability

ISO SB ¼ 3.60 SB ¼ 3.50 SB ¼ 3.10 SB ¼ 3.50 SB ¼ 3.20 SB ¼ 4.10

SS ¼ 0.97 SS ¼ 1.08 SS ¼ 1.60 SS ¼ 1.08 SS ¼ 1.23 SS ¼ 0.99
CMM SB ¼ 4.00 SB ¼ 3.00 SB ¼ 3.13 SB ¼ 3.13 SB ¼ 3.00 SB ¼ 3.33
SS ¼ 1.87 SS ¼ 1.93 SS ¼ 2.03 SS ¼ 2.10 SS ¼ 1.77 SS ¼ 2.18
ISO and CMM SB ¼ 3.80 SB ¼ 3.40 SB ¼ 3.60 SB ¼ 2.80 SB ¼ 3.40 SB ¼ 3.40
SS ¼ 1.10 SS ¼ 1.14 SS ¼ 1.34 SS ¼ 0.84 SS ¼ 0.89 SS ¼ 1.14
Others SB ¼ 2.80 SB ¼ 3.40 SB ¼ 2.80 SB ¼ 3.20 SB ¼ 3.30 SB ¼ 3.55
SS ¼ 1.14 SS ¼ 1.17 SS ¼ 1.55 SS ¼ 1.40 SS ¼ 1.25 SS ¼ 1.29

Fig. 4. Decision tree for selecting SPI methodology.

688 N. Ashrafi / Information & Management 40 (2003) 677–690
perceived its impact as high or very high are 27 and
0%, respectively. Also, apparently CMM covers effi-
ciency only lightly, and its histogram shows that
respondents are not impressed by its impact on effi-
ciency. The histograms in Fig. 3 depict the reported
impact on the factors of the quality of adaptation; it
was quite uneven.
Altogether, the histograms do not provide a clear-
cut distinction between methodologies in their impact
on adaptation factors. Table 8 shows means and
standard deviations for these factors.
Table 8 indicates that using ISO, CMM, or both has
greater impact on most adaptation factors. This contra-
dicts our mapping results that showed that CMM does
not cover expandability, portability, reusability, inter-
operability, and intra-operability, and covers flexibil-
ity lightly. According to the respondents, ISO ranks
high in its impact on all adaptation factors, even
though it does not cover portability at all and its
coverage for the rest is quite light and not direct.
4.1. The results
Our empirical data basically supports the theoreti-
cal framework and indicates that process improvement
enhances software quality factors; however, the levels
of impact on various quality factors differ depending
on which methodology is used. To put our entire
findings in perspective, we created a decision tree
that organizations could use as a guideline for the
selection of an SPI methodology that meets their
quality requirements. Fig. 4 shows this decision tree.
Real-time software products, where performance is
a critical factor, include reservation and trading soft-
ware applications. Organizations involved in devel-
oping this type of software product should use ISO, if
their primary concerns are usability; use ISO and
CMM, if integrity, reliability, and testability are the
most important quality factors for them, and use other
methodologies if efficiency is the primary concern.
The impact on adaptation quality factors is the most
difficult to analyze. Histograms did not provide a
clear-cut selection for those organizations with adap-
tation factors as quality requirements. However, while
the empirical findings, more or less, supported the
results of theoretical investigation for design and
performance factors, the results for adaptation factors
mostly contradicted the findings of our mapping.
The intrinsic characteristics of CMM and ISO
9000 could explain their shortcomings here. Both meth-
odologies primarily deal with the status quo and try to
improve the existing development process rather
than engaging in innovation that ultimately requires
adaptation quality factors. Their guidelines reflect
their philosophy. Hence, one would believe that
for organizations involved with developing software
products that need to adapt to constant change,
home grown’’ SPI methodologies are the best strategy.
However, our empirical findings did not support this
idea.
The majority of our respondents ranked the impact
of using ISO and CMM together first for those quality
factors that address technical aspects of software
development (correctness, maintainability, verifiabil-
ity, reliability, and testability). On the other hand,
improving managerial aspects (efficiency, usability,
flexibility, reusability, and intra-operability) did not
require using both ISO and CMM. This seems counter-
intuitive, because ISO and CMM both address man-
agerial issues extensively.
The CMM users consistently showed a wide dis-
crepancy in their perceptions of CMMs impact on
design, performance, and adaptation factors. One
possible reason is that the respondents belonged to
firms at different levels of CMM maturity.
5. Conclusions and limitations
The key to survive is to develop and market quality
products, on time, and within budget. In this study we
examined SPI methodologies as they relate to quality
guidelines. We cross-referenced factors to ISO 9000-3
clauses and key process areas of the CMM to learn
whether SPI methodologies deal with SQA guidelines
as part of their process improvement strategy. We
conducted an empirical study to find the extent of
the impact of SPI methodologies on quality factors in
practice. Based on our findings, we built a decision
tree, which could be used as an instrument for meth-
odology selection.
Our study produced some inconclusive results,
including, for example, the inconsistency between
the theoretical framework and the empirical results
for adaptation factors, wide variability among CMM
users, and the possible biases inherent in the perception
 N. Ashrafi / Information & Management 40 (2003) 677–690
of technical people for managerial issues. This
research has two limitations:
1. Statistical analysis is descriptive rather than
inferential. We chose descriptive statistics because
it does not require a scientific sampling procedure.
Although our sample size is large enough for
inferential statistics, our primary concern was the
integrity and accuracy of our responses; therefore,
rather than random sampling, we concentrated on
developers, with whom we could meet face-to-
face and know that we could gauge their opinions
and their perceptions of SPI methodologies.
2. The ISO 9000-3 clauses and the CMM key process
areas are intentionally vague to allow organizations
to interpret them according to their own quality
system. Our interpretation of their strengths and
weaknesses as they relate to quality factors is also
subjective and may differ from that of other.
References
[1] S.R. Bailey, A.J. Donnell, R.M. Hough, Process Control (ISO
9000) and Process Improvement (APEX) in the Transmission
Systems Business Unit, AT & T Technical Journal (1994)
17–25.
[2] R.C. Bamford, W.J. Deibler, When the CMM meets ISO
9001, cross-talk, The Journal of Defense Software Engineer-
ing (1998) 3–8.
[3] R.C. Bamford, W.J. Deibler, Comparing and contrasting ISO
9001 and the SEI capability maturity models, IEEE
Computers 26 (10), 1993, pp. 68–70.
[4] CMM-CMU/SEI-93-TR-25, Key Practices of the Capability
Maturity Model, Version 1.1, Carnegie Mellon University,
Pittsburg, 1993, pp. 0–11.
[5] W.E. Deming, Quality, Productivity, and Competitive Posi-
tion, MIT Center for Advanced Engineering Study, Cam-
bridge, MA, 1982.
[6] K.E. Emam, A. Birk, Validating the ISO/IEC 15504 measure
of software requirements analysis process capability, IEEE
Transactions on Software Engineering 26 (6), 2000, pp.
541–566.
[7] M.W. Evans, J. Marciniak, Software Quality Assurance and
Management, Wiley, New York, NY, 1987.
[8] B. Fazlollahi, M.R. Tanniru, Selecting a requirement
determination methodology-contingency approach revisited,
Information and Management 21, 1991, pp. 291–303.
[9] R. Fournier, Practical Guide to Structured System Develop-
ment and Maintenance, Prentice-Hall, Englewood Cliffs, NJ,
1991, Chapter 13, pp. 316–322.
[10] D. Garvin, What Does ‘‘Product Quality’’ Really Means?,
Sloan Management Review, 1998, pp. 25–45.
689
[11] T. Gilb, Principals of Software Engineering Management,
Addison-Wesley, Reading, MA, 1987.
[12] D.H. Harter, M.S. Krishan, S.A. Slaughter, Effects of maturity
on quality, cycle time, and effort in software product develop-
ment, Management Science 46 (4), 2000, pp. 451–466.
[13] J. Herbsleb, D. Zubrow, D. Goldenson, W. Hayes, M. Paulk,
Software quality and the capability maturity model, Commu-
nications of ACM 40 (6), 1997, pp. 30–40.
[14] W.S. Humphrey, Managing the Software Process, Addison
Wesley, Reading, MA, 1989.
[15] J. Ingalbe, D. Shoemaker, V. Jovanovic, A MetaModel for
Capability Maturity Model for Software, in: Proceedings of
the AIS, August 2001.
[16] C. Jones, The economics of software process improvement,
IEEE Computer 29 (1), 1996, pp. 95–97.
[17] J.M. Juran, F.M. Gryna, Quality Planning and Analysis,
McGraw-Hill, New York, 1993.
[18] R. Kehoe, A. Jarvis, ISO 9000-3: A Tool for Software Product
and Process Improvement, New York, 1996.
[19] B. Kitchenham, S.L. Pfleeger, Software quality: the elusive
target, IEEE Software 13 (1), 1996, pp. 12–21.
[20] D.H. Kitson, Relating the spice framework and SEI approach
to software process assessment, Software Quality Journal 5
(3), 1996, pp. 145–156.
[21] M.S. Krishan, C.H. Kriebel, S. Keker, T. Mukhopadhyay, An
empirical analysis of productivity and quality in software
product, Management Science 46 (6), 2000, pp. 754–759.
[22] M.S. Krishan, M.I. Keller, Measuring process consistency:
implications for reducing software defects, IEEE Transactions
on Software Engineering 25 (6), 1999, pp. 800–815.
[23] J.A. McCall, et al. Factors in Software Quality, RADC-TR-
77-369, 13441-5700, Rome Air Development Center, Griffiss
Air Force Bas, NY, November 1977, pp. 1–3.
[24] L. Osterweil et al., Strategic directions in software quality,
ACM Computing Surveys 28 (4), 1996, pp. 738–750.
[25] M. Paulk, How ISO 9001 compares with the CMM, IEEE
Software 12 (1), 1995, pp. 74–83.
[26] M.J. Pearson, C.S. McCahon, R.T. Hightower, Total quality
management: are information systems managers ready?
Information and Management 29, 1995, pp. 251–263.
[27] W.E. Perry, Quality Assurance for Information Systems:
Methods, Tools, and Techniques, QED Technical Publishing
Company, Wellesley, MA, 1991.
[28] S.L. Pfleeger, Does organizational maturity improve quality?
IEEE Software 13 (5), 1996, pp. 109–110.
[29] D.D. Phan, J.F. George, D.R. Vogel, Managing software
quality in a very large development project, Information and
Management 29, 1995, pp. 277–283.
[30] R.S. Pressman, Software Engineering: A Beginner’s Guide,
McGraw Hill, New York, 1988.
[31] R.S. Pressman, Software process perceptions, IEEE Software
13 (16), 1996, pp. 16–18.
[32] H. Saiedian, L. McClanahan, Frameworks for quality
software process: SEI capability maturity model versus ISO
9000, Software Quality Journal 5, 1996, pp. 1–23.
[33] S.A. Sheard, The frameworks quagmire, cross-talk, The
Journal of Defense Software Engineering (1997) 1–8.
690 N. Ashrafi / Information & Management 40 (2003) 677–690
[34] G.J. Van der Pijla, G.J.P. Swinkelsb, J.G. Verrijdtc, ISO 9000
versus CMM: standardization and certification of IS develop-
ment, Information and Management 32 (6), 1977, pp. 267–274.
Noushin Ashrafi is an associate profes-
sor in the Management Science and
Information Systems Department, Col-
lege of Management at the University of
Massachusetts Boston. Dr. Ashrafi re-
ceived her PhD in management and
information systems from the University
of Texas at Arlington. Her extensive publications on software
reliability, software process improvement, technology and society,
and application of mathematical models to assess fault tolerance
software have appeared in journals such as IEEE Transactions on
Software Engineering, IEEE Transactions on Software Reliability,
IEEE Computer Society, Information and Management, The
Journal of Information Technology Management, Information and
Software Technology, Information System Management, Computer
and Society, and Journal of Database Management. Dr. Ashrafi has
made numerous presentations on various topics of information
systems in national and international conferences.