Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iii

United Way of America Task Force on Impact: Members Involved

in Development of Manual ....................................................................................................................... v

National Human Service Organization Representatives Meeting

to Advise Project Staff and Consultants .......................................................................................... vii

Glossary of Selected Outcome Measurement Terms ................................................................... xv

Introduction to Outcome Measurement ............................................................................................... . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........................................................................................... 1
Why Measure Outcomes? ............................................................................................................................... .4
About This Manual .......................................................................................................................................... 5
Notes About Terminology ....................................................................................................................... .7
In Conclusion ............................................................................................................................................ 8
Exhibits
A: Summary of Program Outcome Model .............................................................................................. 3
B: Measuring Program Outcomes: Eight Steps to Success ................................................................ 6

Step 1: Get Ready . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11

Task 1: Assemble and Orient an Outcome Measurement Work Group ..................................................... 12
Work Group Responsibilities ................................................................................................................. 13
Work Group Membership....................................................................................................................... 14
Initial Discussions ................................................................................................................................... 16
Definition of Terms ......................................................................................................................... 16
Value of Outcome Measurement ................................................................................................... 16
Limitations and Potential Problems of Outcome Measurement .................................................. 21
Expectations of Agency Director, Board, Other Key Players ...................................................... 24
Available Resource Materials ........................................................................................................ 25
Task 2: Decide Which Program(s) to Start With ........................................................................................ 25
Task 3: Develop a Timeline ........................................................................................................................ 27
Task 4: Distribute Your Game Plan to Key Players .................................................................................. .30

Exhibits
1-A: Responsibilities of Outcome Measurement Work Group .......................................................... 15
1-B: The Traditional Service Program Model ..................................................................................... 17
1-C: The Program Outcome Model ...................................................................................................... 18
1-D: Hints as to What Are and Are Not Outcomes.............................................................................. 19
1-E: Examples of Diverse Programs and Possible Outcomes ............................................................ 20
1-F: Examples of Potential Benefits of Outcome Measurement ........................................................ 22
1-G: Guidelines for Incorporating Outcome Data in Funding Decisions .......................................... 26
1-H: Sample Timeline for Planning and Implementing Outcome Measurement
in a Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ......................... 28

Worksheet
1: Timeline for Planning and Implementing Outcome Measurement ............................................... 29
ix
Step 2: Choose the Outcomes You Want to Measure ............................... .31
Levels of Outcomes ........................................................... 32
Task 1: Gather Ideas for What Your Program's Outcomes Are from a
Variety of Sources ..................................................... .33
Task 2: Construct a Logic Model for Your Program .................................. 38
Issues in Defining Outcomes ............................................... .49
Preparing your Logic Model ................................................. 52
Task 3: Select the Outcomes That Are Important to Measure ......................... .53
Task 4: Get Feedback on Your Logic Model and the Outcomes Selected
for Measurement ...................................................... 56
Exhibits
2-A: Inputs Through Outcomes: The Conceptual Chain .......................... 34
2-B: Conducting Focus Groups ............................................. .36
2-C: Examples of Outcomes Based on Observations of a Neighborhood
After-school Program ................................................. .39
2-D: Southside Children's Agency ............................................ 41
2-E: Example Logic Models for Four Programs of Southside Children's Agency
(horizontal format) ................................................... 42
2-F: Example Logic Models for Four Programs of Southside Children's Agency
(vertical format) ...................................................... 45
2-G: Questions Regarding Proposed Program Logic Model and Outcomes .............57
Worksheets
2: Program Logic Model ................................................... 44
3: Assessing Possible Outcomes ................................ , ............ .54

Step 3: Specify Indicators for Your Outcomes ..................................... 59

Task 1: Spedfy One or More Indicators for Each Outcome . , ..........................59
Task 2: Dedde What Factors Could Influence Partidpant Outcomes .... ; ............... 68
Spedal Problems with Hard-to-Measure Outcomes ................................. .74
Exhibits
3-A: Sample Outcome Indicators: Southside Children's Agency ................. : ..64
3-B: Example Outcomes and Outcome Indicators for Various Programs .............. 66
3-C: Example of Value of Outcome Data Broken Out by Influencing Factors .......... 69
3-D: Partidpant and Program Characteristics That May Influence Outcomes ......... .70
3-E: Factors That May Influence Partidpant Outcomes:
Southside Children's Agency ............................................ 72
Worksheets
4: Outcome Measurement Framework ........................................ 60
5: Outcome Measurement Framework: Factors That May Influence
Partidpant Outcomes ................................................... 75

X Measuring Program Outcomes: A Practical Approach

Step 4: Prepare to Collect Data on Your Indicators .................................................................. . . .
........................... 81
Task 1: Identify Data Sources for Your Indicators ..................................................................................... 81
Task 2: Design Data Collection Methods.................................................................................................... 87
Preparing Data Collection Instruments ................................................................................................. 89
Developing Data Collection Procedures ............................................................................................... 95
At What Milestones Should Data Be Collected? .......................................................................... 96
Who Is Considered a Participant? .................................................................................................. 97
Include All Participants or Only a Sample? ................................................................................... 97
Who Will Collect the Data? ............................................................................................................ 98
How Will Confidentiality Be Protected? ....................................................................................... 98
How Will Participants Be Informed About the Data Collection Process? ................................ 100
Task 3: Pretest Your Data Collection Instruments and Procedures......................................................... 102
Exhibits
4-A: Advantages and Disadvantages of Data Sources for Outcome Indicators ................................. 86
4-B: Comparison of Major Data Collection Methods ......................................................................... 88
4-C: Sample Outcome Measurement Framework: Southside Children's Agency ........................ 90
4-D: Considering Cultural Issues in Instrument Development ........................................................... 94
4-E: Sample Pledge of Confidentiality for Data Collectors ................................................................ 99
4-F: Sample Consent Form ................................................................................................................. 101

Step 5: Try Out Your Outcome Measurement System ......................................................... 105

Task 1: Develop a Trial Strategy ............................................................................................................... 106
Task 2: Prepare the Data Collectors .......................................................................................................... 107
Task 3: Track and Collect Outcome Data .................................................................................................. 107
Task 4: Monitor the Outcome Measurement Process ............................................................................... 110
Exhibits
5-A: Sample Master List of Data Elements Needed for Each Participant ........................................ 107
5-B: Sample Data Collection Log ....................................................................................................... 109

Step 6: Analyze and Report Your Findings ............................................................................... 113

Task 1: Enter the Data and Check for Errors ............................................................................................ 114
Task 2: Tabulate the Data ........................................................................................................................... 115
Task 3: Analyze the Data Broken Out by Key Characteristics ................................................................ 116
Task 4: Provide Explanatory Information Related to Your Findings ...................................................... 118
Task 5: Present Your Data in Clear and Understandable Form ............................................................... 118
Exhibits
6-A: Sample Spreadsheet for Programs Without Computers ............................................................ 115
6-B: Outcome of Delivering Healthy Babies, by Age of Mother ..................................................... 117
6-C: Sample Outcome Data Table: Southside Children's Agency Teen Mother
Parenting Education Program ..................................................................................................... 119
6-D: Percent of Participants with Pre- to Post-Test Increases of Six Points
or More ........................................................................................................................................ 120
6-E: Changes in Youths' Conscientiousness About Homework ...................................................... 121

Contents xi
Step 7: Improve Your Outcome Measurement System ..............................125
Task 1: Review Your Trial-run Experience, Make Necessary Adjustments,
and Start Full-scale Implementation ...................................... 126
Task 2: Monitor and Review Your System Periodically ....................... _ ......127

Step 8: Use Your Findings .................................................... 129

Internal Uses of Outcome Findings ............................................. 130
Provide Direction for Staff ................................................. 130
Identify Training and Technical Assistance Needs ............................... 131
Identify Program Improvement Needs and Strategies ............................131
Support Annual and Long-range Planning ..................................... 140
Guide Budgets and Justify Resource Allocations ................................. 140
Suggest Outcome Targets .................................................. 140
Focus Volunteers' Attention on Policy and Programmatic Issues .................... 141
External Uses of Outcome Findings ............................................. 142
Recruit Talented Staff and Volunteers ......................................... 142
Promote Your Program to Potential Participants and Referral Sources ................ 143
Identify Partners for Collaborations .......................................... 143
Enhance Your program's Public Image ........................................ 144
Retain and Increase Funding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..... 145

Exhibits
8-A: Outcome Findings by Program Unit and Level of Difficulty of
Participants' Problems ................................................ 133
8-B: Comparative Findings for Two Youth Services Approaches: Team vs.
Individual Activity Emphasis ........................................... 134
8-C: Percent of Participants with Pre- to Post-Test Increases of Six Points or
More, 1995 and 1996 ................................................. 136
8-D: Southside Neighborhood Children's Agency Child Abuse Public Awareness
and Prevention Program: Actual Outcomes vs. Targets ...................... 138
8-E: Burglaries in Southside Neighborhood Before and After Implementation
of Neighborhood Organizing Program ....... , ...........................139

xii Measuring Program Outcomes: A Practical Approach

Appendix:
Issues in Developing Data Collection Instruments and Procedures ........ 14 7
Record Extraction Forms............................................................................................................................ 147
Instrument Development ..................................................................................................................... 147
Procedural Issues Specific to Record Extraction ............................................................................... 149
Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..1 SO
Instrument Development ..................................................................................................................... 150
Content ........................................................................................................................................... 150
Wording ......................................................................................................................................... 153
Format and Style ........................................................................................................................... 155
Procedural Issues Specific to Questionnaires .................................................................................... 156
Administration ............................................................................................................................... 156
Voluntary Completion ................................................................................................................... 159
Participant Confidentiality ........................................................................................................... 159
Anonymity ..................................................................................................................................... 159
Times for Contacting Potential Interview Respondents ............................................................. 160
Response Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ......................... 160
Costs ............................................................................................................................................... 160
Trained Observer Ratings ........................................................................................................................... 162
Instrument Development ..................................................................................................................... 162
Procedural Issues Specific to Trained Observer Ratings................................................................... 162

Exhibits
1: Example of a Record Extraction Form ......................................................................................... 148
2: Survey Questions on Family Counseling ....................................................................................... 152
3: Examples of Fixed-choice Response Sets ..................................................................................... 154
4: Summary Comparison of Questionnaire Administration Options ............................................... 158
5: Tips for Improving Mail Survey Response Rates.......................................................................... 161
6: Sample Rating Scale for Trained Observer Rating of a Playground
for Preschool Children .................................................................................................................... 163

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .165
Performance Measurement .......................................................................................................................... 165
Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ................. 166
Performance Indicators ................................................................................................................................ 168