EVALUATION

Pat Ray M. Dagapioso Project Development

Project Development Cycle

Project Planning

Project Evaluation

Project Implementation

Project Evaluation
It is a systemic and objective assessment of an ongoing or completed project (ILO Technical Operation Manual) Systematic Investigation of the worth or merit of an object, i.e., project (NSF Handbook for Project Evaluation)

Why is Evaluation Important?

Evaluation is Important
The importance of Evaluation lies in its aims:
A. Determine the level of achievement of the project objectives B. Development effectiveness, Efficiency, Impact and Sustainability C. Provides information to help improve the project D. Building understanding, motivation and capacity amongst involved in the project

Evaluation is Important
The importance of Evaluation lies in its aims:
E. Helping empower project partners F. Provides information to help improve the project, and G. Provides new insights or new information that was not anticipated

Projects and its Impacts

Outputs Inputs

(Investments)

(Products & Services Provided)

Outcomes (Immediate Objective)

Impact (Long-Term Sustainable Changes)

Managing for Impact

Managing for Impacts

Outcomes:
Medium-term effects of project outputs. Observable changes that can be seen from project interventions.

Impacts
Positive and negative; and primary and secondary effects of a project intervention. Often detectable after several years.

Levels of Evaluation

Levels of Evaluation
Program

Project

Project

Projects

Activities

Activities

Activities

Levels of Evaluation
Project Evaluation:
Focuses on an individual project funded under the umbrella of a program. Evaluation collects data to improve the project as it develops and progresses. Information is also gathered for the purpose of whether the project has proceeded as planned, and has it reached its programs goal or the projects objectives

Types of Evaluation

Types of Evaluation (Who Does it?)

Types of Evaluation according to who does it:
A. Self Evaluation B. Internal Evaluation C. Independent Evaluation D. External Evaluation

Evaluators Self-Evaluation Agency/Project Managers Themselves From the Agency but not necessarily part of the project External Evaluators + Independent Members from the Agency External Evaluators with no previous links with the project

Degree of Impartiality Low

Costs to the Agency Low

Internal Evaluation

Medium

Medium

Independent Evaluation External Evaluation

Medium to High

High

High

High

Evaluation Types

Frequency of Evaluation

Frequency of Evaluation
Per ILO Technical Cooperation Manual, the frequency of evaluation proceeds in:
A. Technical cooperation projects of 18 months duration or less require a final evaluation upon completion; B. Projects of 18 to 30 months duration require annual reviews and a final evaluation upon completion; C. Projects of over 30 months duration require annual reviews, an interim evaluation and a final evaluation upon completion; D. All projects with a budget of over US$ 500,000 require an independent evaluation at least once during the project cycle; and E. Independent evaluations are required before starting a new phase.

Frequency of Evaluation
Annual reviews
Self-Evaluations that are managed by the project management itself.

Interim Evaluations
Takes Place halfway through the project implementation.

Final Evaluations Independent Evaluation

Can either be final or interim evaluations.

Additional Evaluation
Takes place when there is a need for readjustment of the project objectives

Additional Independent Evaluations

Recommended for projects that lasts up to 4 years.

Types of Evaluation according to the Phase of the Project

Types of Evaluation according to its Phase:
A. Formative B. Summative

Formative Evaluation
Formative Evaluation assesses initial and ongoing project activities. 2 Components:
A. Implementation Evaluation B. Progress Evaluation

Formative Evaluation
Implementation Evaluation
The purpose of Implementation evaluation is to assess whether the project is being conducted as planned.

Formative Evaluation
Progress Evaluation
The purpose of progress implementation is to assess progress in meeting goals of the program and the project. This evaluation measures the progress at the various stages of intervention.

Summative Evaluation
Summative evaluation assesses the quality and impact of a fully implemented project. Summative Evaluation collects information about outcomes and related processes, strategies and activities.

Summative Evaluation
Basic questions:
A. To what extent does the project meet the stated goals? B. Are greater number of the target population has shown significant changes? C. Which components are effective? Which are not? D. Were the results worth the cost? E. Is the program sustainable?

Evaluation Phase Proper

What to Evaluate?
Evaluation Criteria:
A. Relevance and Strategic fit of the project B. Validity of the Project Design C. Project Progress and Effectiveness D. Efficiency of resource use E. Effectiveness of the Management Arrangement F. Impact Orientation and Sustainability of the Project

Relevance and Strategic Fit

Does the project address a relevant need and decent work deficit? Was a needs analysis carried out at the beginning of the project reflecting the various needs ofdifferent stakeholders? Are these needs still relevant?

Relevance and Strategic Fit

Have new, more relevant needs emerged that the project should address? How does the project align with and support national development plans, thenational poverty reduction strategy (PRS), national decent work plans, national plans of action on relevant issues (e.g., on employment creation, child labour, antitrafficking, etc.), as well as programmes and priorities of the national social partners?

Validity of Design
What was the baseline condition at the beginning of project? How was it established? Was a gender analysis carried out? Are the planned project objectives and outcomes relevant and realistic to the situation on the ground? Do they need to be adapted to specific (local, sectoral, etc.) needs or conditions? Is the intervention logic coherent and realistic? What needs to be adjusted?

Project Progress and Effectiveness

Is the project making sufficient progress towards its planned objectives? Will the project be likely to achieve its planned objectives upon completion? Have the quantity and quality of the outputs produced so far been satisfactory? Do the benefits accrue equally to men and women? Are the project partners using the outputs? Have the outputs been transformed by project partners into outcomes?

Project Progress and Effectiveness

How have stakeholders been involved in project implementation? How effective has the project been in establishing national ownership? Is project management and implementation participatory and is this participation contributing towards achievement of the project objectives? Has the project been appropriately responsive to the needs of the national constituents and changing partner priorities?

Project Progress and Effectiveness

Has the project been appropriately responsive to political, legal, economic, institutional, etc., changes in the project environment? Has the project approach produced demonstrated successes?

Efficiency of Resource Use

Have resources (funds, human resources, time, expertise, etc.) been allocated strategically to achieve outcomes? Have resources been used efficiently? Have activities supporting the strategy been costeffective? In general, do the results achieved justify the costs? Could the same results be attained with fewer resources? Have project funds and activities been delivered in a timely manner?

Effectiveness of Management Arrangements

Are management capacities adequate? Does project governance facilitate good results and efficient delivery? Is there a clear understanding of the roles and responsibilities by all parties involved? Does the project receive adequate political, technical and administrative support from its national partners? Do implementing partners provide for effective project implementation?

Impact Orientation and Sustainability

Can observed changes (in attitudes, capacities, institutions, etc.) be causally linked to the projects interventions? In how far is the project making a significant contribution to broader and longer-term development impact? Or how likely is it that it will eventually make one? Is the project strategy and project management steering towards impact? What are the realistic long-term effects of the project on the poverty level and the decent work condition of the people?

Impact Orientation and Sustainability

Is there a need to scale down the project (i.e., if the project duration is shorter than planned)? Can the project be scaled up during its duration? If so, how do project objectives and strategies have to be adjusted? Has the project successfully built or strengthened an enabling environment (laws, policies, peoples attitudes, etc.)? Are the project results, achievements and benefits likely to be durable? Are results anchored in national institutions and can the partners maintain them financially at end of project?

Methodologies Used to Acquire Data

Techniques Used to Acquire Data

1. Survey 2. Interview 3. Focus Groups 4. Observation 5. Tests 6. Document Studies 7. Key Informant 8. Case Studies

Advantage/Disadvantages: Here.

Evaluation Report

Evaluation Report
Before any evaluation report will be presented, a terms of reference (TOR) of the evaluation will be first presented (and prepared by the evaluation manager) for comments to the following stakeholders (ILO) and before the evaluation proper:
A. Project Manager B. Main National project partners C. ILO Field Office Director (for ILO only) D. Field Technical Specialist E. Donor

Terms of Reference
Contents of the Terms of Reference of the Evaluation:
A. Introduction and rationale for evaluation B. Brief background on project and context C. Purpose, scope and clients of evaluation D. Suggested analytical framework E. Main outputs F. Methodology to be followed G. Management arrangements, work plan and timeframe

What happens with the terms containing in the TOR?

Once the TOR is approved by the Project Management, the evaluation manager can then undergo the proper evaluation phase as stipulated in the TOR.
The evaluation manager is assured of efficient use of resources: 1. Gathering relevant information prior and during the evaluation 2. Ensuring the efficient use of time 3. Arranging for the evaluator to meet the right people 4. Scheduling interviews and meetings with partners, and 5. Ensuring efficient logistical arrangements.

Evaluation Report
The Evaluation Report Contains:
Abstract Brief background on the project and its logic Purpose, scope and clients of evaluation Methodology Review of implementation Presentation of findings regarding project performance Conclusions Recommendations Lessons learned Annexes

Evaluation Report
Example of an Evaluation Report (per World Bank):
Here.

People Who Evaluate

Evaluation Managers
The evaluation manager is responsible for managing all independent and internal evaluations. The evaluation manager should have no links to the project decision-making and hence should not be the technical or administrative backstopper of the project.

Evaluation Managers
Roles:
Determine the target audience for the evaluation and the key evaluation questions theevaluation should answer; Prepare the draft TOR for the evaluation (final approval is given by the evaluation focal person) and send a copy of the approved TOR for information; Identify the evaluation consultant(s), and obtain final approval for their recruitment from the evaluation focal person; Ensure smooth organization of the evaluation process and proper support to the evaluation team;

Evaluation Managers
Roles:
Ensure proper stakeholder involvement in the entire evaluation process; Ensure that gender issues are considered throughout the evaluation process; Manage the process of preparing the evaluation report (including circulating the draft report and collecting comments); Submit the final evaluation report to the evaluation focal person for final review Ensure proper follow-up on the recommendations and dissemination of lessons learned

Evaluators
The evaluator carries out the evaluation and prepares the evaluation report according to theTOR. Roles:
Adhere to internationally-accepted good practices and solid ethical principals; Be skilled in implementing diverse evaluation methodologies; Ensure the evaluation is an inclusive and participatory learning exercise; and Be culturally- and gender-sensitive

The End. Thank You!