Improving Defect Classification:

Why, How, and When

SEPG North America Conference
May 6-7, 2014

Mark Stein
Keymind, A Division of Luminpoint

Davide Falessi
Fraunhofer Center for
Experimental Software Engineering
Agenda

• WHY is it important to have a good defect

classification schema?
• HOW do you improve your schema?
• HOW do you know if you’ve made an improvement?
• WHEN is the right time to implement this kind of
improvement in your organization?

2
• 25-person division of a 500-person
umbrella organization, Luminpoint,
Inc.
• IT and Creative division of
Luminpoint
• Specializes in Web-based application
development with strong focus on
user-centered design and full
Section 508 compliance
• Achieved CMMI-DEV v1.3 Maturity
Level 5 in March 2012

3
• A not-for-profit applied research &
technology transfer organization
• Affiliated with the University of Maryland
• Our philosophy: understanding the context
and suggesting appropriate improvements
• Our Process Improvement approach
– Understanding the effort (e.g., gap analyses)
– Planning the improvement initiative
– Implementing the improvement initiative plan
– Conducting an assessment
• Other Competencies
– Software/Systems Management, Measurement and
Empirical Studies
– Knowledge Management
– Software Architecture, Testing, and IV&V

4
The Importance of a Good Defect
Classification Schema

• If your only goal in tracking defects is to make sure they are

resolved, then a description and status could be enough.

• If your organizational goals focus on software quality and

defect reduction, then more information is needed.

• How do you define a “good” classification schema? The CMMI

provides process insights versus classification details and
recommends basing the quality of individual processes on
how well it supports your project and organizational goals.

• Your needs may change as your organization matures.

5
Prediction Model (Defects in Production)

The old classification

schema made it
challenging to exclude
certain defects, such as
those that were not a
result of our processes,
or could not be identified
through testing.

Understanding the
effectiveness of different
testing activities was
essential to designing the
next version of the
model.

6
Classification Schema (BEFORE)

Title:
Description: Our original
classification schema
Priority:
supported our
Severity: organization well
through ML 2 and 3
Defect Category:
Version Injected:
Version Fixed: As we focused on
some of the high
Phase Detected: maturity process
Tool/Method Used: areas, limitations of
the schema became
Component: apparent.
Assignee:
Reported By:

7
Classification Schema (BEFORE)

Title:
Good at telling us
Description: what type of defects.
Priority: Not as good at telling
Severity: us where they were
coming from, or
Defect Category: helping us determine
Version Injected: which ones to include
in our HM analyses.
Version Fixed:
Phase Detected:
Tool/Method Used: Good at telling us how
Component: we found the defects.

Assignee: Not as good at telling

us the relative
Reported By:
effectiveness of our
methods.

8
Defect Categories (BEFORE)
Describes Describes
Failure Source
Accessibility Defect X
Code Defect – Content X X
Code Defect – HTML X X
Code Defect – Formatting X
Code Defect – Logic X X
Code Defect – Data Access X X
Code Defect – Data Migration X
Configuration Defect X X
Misunderstood Requirement X
Missed Requirement X
Non-Standard Compliance X X
Usability Defect X
Security Finding X
Supplier Defect X

9
Defect Categories (BEFORE)
Describes Describes
Failure did this
When Source
Which process would we
Accessibility Defect X originate?
defect
focus on to prevent this
Code Defect – Content X X
kind of defect in the
Code Defect – HTML X X future?
Code Defect – Formatting X a defect we
Is this
Code Defect – Logic X consider for
should X
Code Defect – Data Access our prediction
X model? X
Code Defect – Data Migration X improvements to
Are past
Configuration Defect X ourXanalysis & design
Misunderstood Requirement Was this defect processes
X reducing
Missed Requirement unavoidable? software
X defects?
Non-Standard Compliance X X
Usability Defect X Is this a defect?
Security Finding X IS a defect?
What
Supplier Defect X

10
 Defect Failure (AFTER)
Missing, Incorrect, or Incomplete Functionality/Results:The functionality or results
Missing, Incorrect, or Incomplete
are different from the expected ones whether missing, or incomplete (e.g., search Must describe the
resultsFunctionality/Results
presented are incomplete, email not automatically generated, or other manifestation of the
documented requirements and wireframed features are not provided to the user). defect
Unexpected
However, when formattingTermination (error)
is incorrect, use "Incorrect Formatting" failure option.
Unexpected Termination (error): The system unexpectedly stops during processing
(e.g., system crash, 404-page not found, page not displayed, or system is
Incorrect Formatting
unresponsive).
Is a required attribute
when initially logging
Incorrect Formatting: The output is incorrectly displayed.
Usability
Usability: Any feature, function, or facet of the user interface or its organization the defect
that violates established principles of usability (e.g., visibility, feedback, etc.,) or
that isAccessibility
likely to lead to user error, delay, confusion, or the failure to complete a task.
Accessibility: Defects that prevent full accessibility of an application or web site Has to be from the
(e.g., Section 508, using a screen reader, or other assistive technologies). perspective of the end
Performance
Performance: A system feature does not meet requirements in terms of speed or user or customer
capacity (e.g., export of data is too slow).
Security/Vulnerability
Security/Vulnerability: A feature or information is available to an unintended user.
Non-compliance to Standard: A defect related to non-adherence to a standard Is a helpful
eitherNon-compliance
official, e.g., SCORM, or anto Standard
internal one such as our various coding standards, discriminator when
naming conventions, and standard architecture. Note that this type of defect including or excluding
cannot be identified by the user. It does NOT include 508 which has its own
Other
category–see "Accessibility" failure type.
data from analysis

11
 Defect Source (AFTER)
Requirement - Incomplete from Customer: The customer provided an unclear or
Requirement - Incomplete from Customer
incomplete requirement.
Must describe the
Requirement - Incomplete Internally:
Requirement - IncompleteThe requirement was not documented or not
Internally
clearly specified.
cause of the defect, or
the point at which the
Requirement
Requirement - NotThe
- Not Implemented: Implemented
requirement was documented and included
defect was injected
in the wireframes, but was not implemented in the software.
Information
Information Architecture:Architecture
Issues with the wireframes and/or work flow diagrams
including insufficient detail and/or misrepresentation of a requirement.
Content:Content
Incorrect or incomplete content provided. Less about blame,
Data Migration: Problems related to data migration. more about
Data Migration
Partner/Supplier: Defects found in deliverables or services provided by a partner determining which
or supplier. processes to focus on
Partner/Supplier to prevent defects in
Interface Design: Issues with the implementation of HTML, CSS, images, colors,
Interface
etc., related Designof the design, NOT mistakes in the design process.
to the execution the future
For the latter use "Information Architecture" source.
Configuration
Configuration - Application: –Incorrect
Application
software configuration, i.e.., missing
component or module, incorrect version, incorrect Web config file, a script that Is also a helpful
shouldConfiguration
have been run that was - Server/Environment
not. discriminator when
Configuration - Server/Environment: Incorrect server or network configuration, including or excluding
Codesecurity settings, ports, IIS, DNS, etc.
i.e., incorrect data from analysis
Code: Requirements have been defined and implemented but software is not
Other
functioning correctly due to incorrect logic, calculations, or other coding errors.

12
Defect Detection Method (BEFORE)

Defect detection tools/methods Testing Activities

Manual Testing Automated Unit/Integration Testing

Automated Code Review Automated UI Testing
Peer Review Internal Systems Testing
Stress and Performance Testing User Acceptance Testing (UAT)
Unit Testing
UI Testing Tool
Usability Testing
Product Usage

Measured number of defects per Measured level of effort (hours) for each
method, per software version testing activity, per software version

Not a foolproof way to determine the number of

defects found per hour of testing effort

13
Defect Detection Method (AFTER)

Defect detection tools/methods Testing Activities

Automated Unit/Integration Testing Automated Unit/Integration Testing

Automated UI Testing Automated UI Testing
Internal Systems Testing Internal Systems Testing
User Acceptance Testing (UAT) User Acceptance Testing (UAT)
Load/Performance Testing Load/Performance Testing
Usability Testing Usability Testing
Security Testing Security Testing
Compliance Testing Accessibility/Compliance Testing
Product Usage
Peer Review

Now we have both number of defects found per method, plus the amount of effort
expended per method. Helps us make decisions on how to allocate V&V resources.

14
Validation Methodology

 Sampled 10 defects from our repository and asked 19

subjects to classify them.

 Each subject classified 10 defects, 5 with the current

schema, 5 with the proposed one.

 Total of 190 defect classifications (85 for each schema)

 We measured Correctness in terms of Agreement:

percentage of classifications where, for the same defect,
different people selected the same category.

15
Validation Results

Agreement using the

Subjects’ opinion about the new vs. old schema old schema:

39%
Agreement using the
new schema:

64%
Statistical difference:
Note: participants received no training prior to taking survey. P-value < 0.001

16
Rollout

• Created new defect failure/source fields, disabled existing category

field
• Developed plan for reclassification of ~ 5000 defects
– Automatic: ~ 50%
• Scripted changes to existing categories where they mapped
directly to new failure/source fields
– Manual: ~ 50%
• Current, active projects: Recategorized all production and
pre-production defects.
• Older, or decommissioned projects: Focused on production
defects, January 2011 or later
• Delegated recategorization task to project managers –
took 120 hours of effort over the course of 14 weeks

17
When to Implement the Change

• It depends on the goals of your organization

• For Keymind, we planned and budgeted to improve our classification post
high maturity appraisal
– We consciously decided to keep things static pre-appraisal so as not to impact
existing baselines and models.

– Old schema brought challenges, but these were not show-stoppers when it came
to achieving our level 4/5. However, they did limit our ability on improving both our
models and testing effectiveness.

– Focused instead on applying improvement efforts (with limited resources) in

preparation for the appraisal.

• Lesson Learned: We had many interesting arguments about the definition

of defects along the way – maybe these types of conversations can be
though of as indicators of opportunities for improvement(!)

18
Conclusions

• A robust defect classification schema can provide

relevant measurement data to help an organization
meet/monitor organizational level goals

• When to implement depends on context of the

organization and its priorities at the time

• A robust methodological approach can validate if an

improvement was attained (or not). It also provides a
nice set of high maturity evidence (OPM SP3.3).

19
References

• Bug classification schema

– Chillarege, et al., “Orthogonal defect classification-a concept for in-process
measurements,” IEEE Transactions on Software Engineering, vol. 18, no. 11, pp. 943 –
956, 1992.
– IBM, http://researcher.watson.ibm.com/researcher/view_project_subpage.php?id=5020
– HP - Grady, Robert B., “Practical Software Metrics for Project Management and Process
Improvement”. Prentice Hall, Inc., (1992), pp. 122-137, 223-227.

• Quality of bug classification schema

– Freimut, et al., “An industrial case study of implementing and validating defect
classification for process improvement and quality management,” METRICS 2005.
– K. El Emam and I. Wieczorek, “The repeatability of code defect classifications,” in
Software Reliability Engineering, 1998.
– Falessi et al., “On Failure Classification: The Impact of 'Getting it Wrong‘”, ICSE 2014.

20
Q &A

21