Metrics Training Program Conducted

by QAI
 Measurement Framework
Measurement
Did it Work?
Performance
Process
Measurement

Improvement Comparison

Improvement Performance Multiple Level of

Plan Goals Responsibilities

Analysis
Projects… Project Managers

What Should Processes… PEG, QAD

Change? Business… Business Manager
 Improvements and
Measurement
Define GQM
Measuremen Procedures
t Need
Identify Collect
Scope Data
Continuous
Improvement
Model
Your Measurement
Evolve Process
Process Need to
Analyze be a Dynamic One!!
Data
CMM
Without the right information, you are
just another person with an opinion-
Tracy O’Rourke

Why We Measure What We Measure

• To Characterize Project
• To Evaluate Product
• To Predict Quality
• To Improve Process

Metrics- KISS Approach

Easy to Understand, Able to Improve, Simple to
Define, Easy to Measure
 Measurement Scales
• Nominal- Type of Project
• Ordinal- Customer Satisfaction Level
• Interval- Temperature
• Ratio- Mean Time Between Failure
• Absolute- Difference of One, Origin
Present, e.g.-Number of Defects
 Philip B Crosby-Quality
Without Tears
Performanc
e Standards Procedures

Material
Information

INPUTS OUTPUTS

Facilities and Training and

Equipment Knowledge
 What to Measure?
• Basic Measures-
Size
Common Assumptions
Effort
2. Cost is Proportional to
Schedule Effort
Defect
2. Schedule= Cycle Time
Cost
Cycle Time is the Minimum Completion Time for a
Particular Type of Project
 What to Measure?
• Derived Measures
Productivity
Quality
Change
Defect Leakage
Effort Distribution
Rate of Defect Injection
Review Effectiveness
Risk
 Derived Measures
• Productivity- WC per PH, Hours per WC
• Defect Injection Rate- Denominator may
not be in PH alone (Defect/Page for Tech.
Doc. Activity)
• Change- Change in Size is a Surrogate
measure for Change in Requirement
 Derived Measures
• Quality-
Cost of
Quality={(Review+Rework+Testing+Change+
Audit+Training Effort+Defect
Prevention)/Total Effort}*100
Cost of Poor
Quality={(Rework+Testing+Change+Audit
Effort)/Total Effort}*100
Try to Identify the Value Added Items
 Value Added Items
• Product Quality-
if Delivered Defect/Size
Delivered Defect= Defect Injection
Rate*Size*(1-Defect Containment
Efficiency)
Defect Prevention, Training reduces DIR
Review Increases Defect Containment
Efficiency
 What is Risk?
• Risk is an event
• Risk is unpredictable
• It always has negative impact

Risk is a Derived Measure

Consider External Dependencies before assessing Risks
 Review and Test
Effectiveness
• Review Effectiveness= Weighted
Defects found in Review/Review
Effort
• Testing Effectiveness= Weighted
Defects found in Testing/Testing
Effort
 The Management
Dashboard

Quality
Productivity
Developme
nt/Mainten Delivery
Workload ance
Demand
Capability
Tools Readiness
 Quality
Functional Technical Satisfaction Defects
Quality of Design Service Level MTBF
O/P Quality Delivered

Defects
Reliability Interop- Quality may be measured
w.r.t. the implicit and
Security erability specified requirements
Usability Maintain-
ability
 Productivity

WC/D
W ay
C/
P
H

Size Size
 Delivery Rate

%
Co
m
mi
t
m
en
t

Deviation from estimate

 Development:Maintenance
Ratio
• D:M Ratio=Effort in Creation/(Total
Effort-Effort in Creation)
 Tool Penetration
• Tool Penetration
Tool A
Phase

Tool B

% Using the tool

% Not Using the tool, who are

supposed to use it
 Others
• Capability- In terms of Maturity
Level
• Readiness- Response Time, Server
Downtime
• Productivity- Use Run Charts to
track over a period of time, Team
Size, Domains, Technology
 Project Management
• Metrics Required for-
Team Members/Module Leaders
Project Managers
Senior Management
Quality Management
 Metrics- TM/ML
• Individual Planning (PSP)- Tasks and
Durations
• Individual Tracking- Allocated vs
Actual Schedule, No. of Hours for
Completion of Tasks
• Quality Tracking- Reviews/ Testing
Vs Faults
 Metrics- Project
Managers
• Required Data for Project
Management- Size (Actual and
Estimated), Task Allocation and
Completion Schedule, Actual and
Estimated Effort, Impact of Size
Variation on Schedule
• Project Review Meetings
Project Review- Progress
Tracking
• Schedule- Original, Latest,Actual
• Risk- Planned, Forecast, Actual, Index
• Requirement Volatility-Impact on Effort
• Staffing
• Cumulative Milestones
• Variation Chart
 Project Review- Quality
Tracking
• In Phase Defects-Errors
• Planned vs Actual- Product Quality
• Cost of Quality, Cost of Poor Quality
• Phase Containment Efficiency
• Delivered Defect Density
 Metrics –Senior
Management
• Planned Completion Date vs Status
• Planned Efforts vs Status
• Planned Size vs Status
• Injected and Detected Defects-Planned vs Actual
• Testing-Test Progress vs Status of Completion
• Filed Test Bugs-Status
• Delivered Defects-Status
• Technology Changes- Progress (Schedule)
• Process Changes (Schedules)
• Customer Satisfaction Measurement Status
 Management of Quality
• Review Data
• Testing Data
• Delivered Defect Data
 Process Control
• Process Metrics
• PDB
• PCB
Project’s Process Control
• Process Capability Levels
• Actual Measurement
• Special Cause of Variation
• Projects Process Control
- Control Schedules
- Control Efforts
- Control Quality
 Organization’s Process
Improvements
• Decrease of Variation
- Estimation Variance
- Defects Variance
- Rate of Completion
• Increase of Performance
- Improve in Productivity
- Decrease of Cost (CoQ)
- Decrease in Cycle Time
- Decrease of Delivered Defects
 Metrics and CMM KPAs
• Metrics at Level 2- Estimates and Actuals
• Metrics at Level 3- Threshhold for
Management of Size, Effort, Schedule,
Cost, Quality, Critical Resources, Risks
• Metrics Database
• Defect Data analysis
 Metrics and CMM KPAs
• Metrics at Level 4-
• Process Management
- Baselines
- Process Stabilization
- SPC
- Process Goals and Projects’ Process Control

• Quality Management-
- Quality Goals
- Quality Progress
- Defect Goals
 Metrics and CMM KPAs
• Metrics at Level 5-
• Defect Prevention
- Reduce CoQ, CoPQ
• Technology and Process Change
Management-
- Reduce Cycle Time
- Improve Productivity
- Reduce Defect Density
 Seven QC Tools
• Flow Diagrams
• Cause-Effect Diagrams
• Check Sheets
• Histograms
• Pareto Charts
• Scatter Diagrams
• Control Charts
 Seven QC Tools
• Cause and Effect Diagram: Three
Type
• Dispersion Type
• Production Process Classification
Type
• Cause Enumeration Type (Methods,
People, Input, Tools)
 Seven QC Tools
• Histogram- Be aware of the Time Interval
• SPC- Peer Review (Separate for Doc. Inspection
and Code Inspection) and Test Effectiveness can
be under SPC
• Achieve Process Stability before using control
charts
• Update Control Limits at Regular Intervals
• Pareto Charts- Always Mention the Time Period
• Structured Brainstorming
 Causal Analysis Method
• Analyze Defect Data
• Pareto Analysis
• Brainstorming
• Cause-Effect Analysis
• Action Planning
• Implementation of Action Plan
• Check for DP Effectiveness
- Decrease in CoQ, CoPQ
- Absent of Similar Defects in future reviews
 Recommended Readings
• Measures for Excellence- Putnam
• Measuring the Software Process-
Florac, Carlton
• Practical Software Metrics- Brady
• Memory Jogger II
• Humphrey’s book on Software
Process