Module II

Setting Objectives and Selecting Data Gathering Methods
In this module we will be covering the preparation phase of the reporting process. Here you will learn
how to set objectives for the investigation and select the most efficient and effective methods to gather
your data.
We will look at various levels and aspects of evaluation, so we can be clear when investigating about
what we are looking for and what we can expect to see.
We will discuss approaching reporting as a process, and how we can work from either a preset
reporting format or Terms of Reference to develop a plan to gather the information we need.
Setting Research Objectives

Obviously, reports are a non-negotiable part of our work. Apart form the fact that we have to write
reports, what are the different purposes of reporting?
Reasons can include:
To facilitate smooth implementation of programs
Accountability and transparency
To provide a record of progress / success
To investigate concerns from management or end users
Monitoring activities, use of resources
Gain support for our work and advocate change
Assist management thinking
Within these general purposes, each report will have its specific objectives. We need to be clear about
these before selecting our data gathering methods.
Why exactly are you conducting the investigation? What is the history and what are the current
issues to be explored? How will the results of the investigation be used to aid management
thinking in the future?
Are you looking at strengths and weaknesses of a particular programme / project?
Are you looking at processes?
Or are you investigating the outcomes of the programme / project?
2
Extract from The Reporting Skills and Professional Writing Handbook
This module is ELD Publications 2008 and distributed as a free service to those who cannot buy the Handbook and CD
The entire Handbook and CD are available from www.reportingskills.org/order.htm
The Four Levels of Evaluation

There are, broadly, four different levels of information we can get from end users.
Effective evaluation should cover all of these levels. However, at earlier stages in the project we may
not expect to be able to easily evaluate the higher levels.
Reactions and Feelings

Although positive reactions from end users do not necessarily mean your interventions are having any
impact, we still need to evaluate whether the work we are doing is appreciated and understood. Positive
reactions can indicate that the plan has considered end users short-term needs; and has been
communicated clearly. Looking at this level is very important when considering the appropriateness of
the methods we are using in the field. End users satisfaction with service delivery is essential for
project success. This can be investigated even at the earliest stages of a project.
Learning and Knowledge / Results
These are changes in knowledge or attitudes as a result of intervention. We may want to measure to
what extent we have transferred knowledge to end users. This is a precondition for any behavioural
change.
Change in Behaviour / Skill
At this level we investigate whether end users are applying the knowledge / learning and doing things
differently. As change in behaviour follows knowledge, this may only be seen in the medium term.
Effectiveness / Change
This is the change we are striving for as a result of changed behaviour. This tells us whether the plan /
intervention logic actually worked. For example, if at the learning and knowledge level community
groups learned the need for hygiene; and this led to a change in behaviour such as an end to open
defecation; did this, at the effectiveness level, lead to the goal of improved health?
3
The Four Levels and the Logical Framework

To put these levels further into context, it is useful to compare them with the hierarchy of intervention
logic in the Logical Framework Analysis.
Using an example of a health project that aims to contribute to a reduction in HIV through changing
behaviours of commercial sex workers (CSWs):
Reactions and Feelings relates closely to the activities conducted. If the method of service delivery is
not well-received by the target group, there will be little participation.
CSWs will not be open to the ideas presented. As a reporter you need to regularly monitor how end
users feel about the method of service delivery / implementation.
Learning and Knowledge are measurable and can be tested. These are the outputs or results of the
activities understanding of HIV transmission and its prevention, knowledge of legal protection and
understanding of rights and how to assert these.
At the Skills / Behaviour level we have our specific project objective, in this case a change in
behaviour such as increased use of condoms. High-risk behaviour is reduced and assertiveness
practised.
Effectiveness here will relate to the project goal that of reduction in HIV transmission among the
target group.
4
Specific Issues for Evaluation Reports

If the above were not enough, current EU guidelines require also that your evaluation consider the
following specific areas:
Project Preparation and Design
Relevance of the Project
Efficiency
Effectiveness
Outcome and Impact
Findings and Recommendations
Lessons Learned
Quality / Sustainability / Replicability
While this might seem quite a challenging way to present your analysis, lets remember that analysis is
simply breaking things up into simple elements. (And, later, we will show you not just how to break
things up, but how to put them back together again.)
This format allows us to examine a project from many different aspects and, if well understood, will lead
to an extremely effective evaluation / assessment.
I used this format for our charitys 2006 assessment. Previously, we had mainly evaluated performance
how many training programmes and assignments undertaken, and so on. This approach was
extremely useful in helping us really understand where we were, what we really had achieved and
where we needed to focus to improve performance, as well as ensuring transparency with our partners.
(Remember that transparency is an opportunity to show others what really has been happening and
this includes achievements that performance- based reports can overlook.)
As for many this format can be confusing, lets look at each section in more detail to see how we can
use it to help us in setting clear objectives. Later in this module we will look at working with preset
formats as they are actually very helpful if we use them when planning our evaluation / investigation
rather than just approaching them as ways to organise our writing.
5
What preparatory activities were carried out?
Were they relevant?
Were they reflected in the final project design?
Were any necessary preparation activities omitted?
Was the initial intervention logic / plan appropriate?
Relevance
How has the project adapted itself over time?
Does it meet the identified needs of the target group?
Efficiency
Were things done right?
Which activities were undertaken in order to get project results? Were these conducted
efficiently?
Could the results have been achieved more economically?
Were the activities carried out on time? / Was the timing appropriate?
Did the management structure support or hinder efficient implementation?
Was the implementation methodology appropriate?
Effectiveness
Were the right things done?
Were the assumptions in the plan realised?
If the assumptions were not realised, how did this affect the project?
Was the project flexible enough? / How was it adapted?
Should the assumptions be changed?
How effective were project strategies and activities in achieving the project objective?
Outcome and Impact
What were the actual outcomes (intended / unintended) for the end users?
What are the planned and unplanned impacts (positive / negative) of the project?
Findings and Recommendations
What were the main successes and failures of the project so far?
What are the challenges and how can they be addressed?
Which changes / actions are needed for appropriate and successful implementation?
Lessons Learned
What organisational and operational lessons can be drawn?
How can they be incorporated into ongoing activities?
Quality, Sustainability and Replicability
How replicable are the successful project activities?
Is the exit strategy well planned?
What are the quality assurance measures?
6
Some More Terms Explained in Detail

Now we cant assume that this has explained everything 100% in fact, if this section has had you
scratching your head, thats understandable. (As long as you are not banging your head against the
wall!) We cant anticipate every question you have, which is why your feedback is welcome. Meanwhile,
I will briefly explain further two places where a lot of people get confused.
Please note, by the way, that I am not claiming that everyone will agree with these explanations.
Different organisations and different individuals will have a different understanding. The purpose of this
section is to help you improve your own conceptual clarity on what all these terms mean.
Effectiveness vs. Efficiency
Efficiency is a measure of speed and cost. Efficiency says getting something done quickly and cheaply
is more important than it done perfectly later. Efficiency measures results against inputs. For example,
studying this programme is efficient - it gets a lot of knowledge across quickly and cheaply.
Effectiveness is a measure of quality and goodness. Effectiveness says the opposite, that getting it right
is more important than cost or time. That's why some people would prefer to spend hundreds of dollars
plus expenses to attend our training programmes - they feel that face-to-face group learning is more
effective.
Sometimes we can have both - and sometimes we can't. A trade off could be the quantity of advocacy
materials produced / cost / speed of delivery (Efficiency) vs. quality of materials, how widely they are
read, whether they get to the right people, etc. (Effectiveness)
Impact vs. Outcome
In a sense, these two are very similar. The major difference is that an Outcomes Evaluation refers
specifically to the qualitative changes on people (end users, beneficiaries). Impact may include this, but
is wider in scope, and can also include changes which are environmental or social.
Typically, we can also say that our outcomes are usually anticipated and desirable results. For
example, micro-credit for rural women may have the desired outcomes of increased personal self
esteem, greater decision making power within the family and increased involvement in the community.
There may be other outcomes which were unanticipated both positive and negative but if project
preparation and design were done right we should know what Outcomes Indicators we are looking for.
Impact is often an unanticipated result of achieving our goal. For example, one project which brought
water taps directly to village houses achieved its performance objectives and its goal of reducing
womens labour. This was time that had, previously, been spent walking to and from the nearest river.
However, the social impact was that village women lost a valuable forum for mutual support time
spent at the river where women could interact, sharing on how to deal with domestic problems and
reproductive health issues.
7
Field Reporting / Monitoring

Not all reports will be evaluations. In fact, the most frequent type of report is the monitoring report
usually a visit to the field / project site conducted on a regular basis. These reports actually provide
more than just a way of tracking / monitoring implementation and fine-tuning delivery: they are also the
foundation on which major evaluations will be based.
You can imagine, if you like, a house of cards. The bottom of the pyramid represents the field reports
the monthly or quarterly snapshots and analysis of process and progress. The next level is reports
which represent major milestones end of Year 1, mid-term, etc. At the top is the final evaluation.
So, its essential that the regular project monitoring and its reporting are solid and effective otherwise
there is no foundation for the final evaluation. Effective monitoring reports make evaluations simpler.
When I am reviewing our charitys work each year, the majority of the investigation has already been
done through our newsletters (reporting to partners) and database.
Field reporting, then, needs to be as well-planned as any evaluation. A typical field report would include
most of the following:
Basic Information about the Project
Project Title
Location
Implementing Organisation
Phase
Reporting Period
Events during the Reporting Period (positive and negative)
Achievements (in comparison with targets)
Planned Activities
Process
Administrative / Financial Management
Human Resources
Logistics
Conclusions
Recommendations
Annexes / Other Information
8
Approaching Reporting as a Process

Whether you are preparing a field / monitoring or evaluation report, the process is still going to be the
same.
The first thing is to set your research / investigation objectives. What exactly are you investigating? If
this is not already explicit for example, through Terms of Reference given to external consultants
you will need to be clear before you can take any action.
Are you evaluating / assessing a project? If so, the EU guidelines provide a useful framework for
gathering and presenting ideas. Or are you looking at one specific area? For example, your reporting
objective might be to investigate:
Process / Efficiency
Outcomes and Impact
Performance
Some specific issue or problem which has arisen
Next, you need to consider what you can realistically expect to evaluate. In early stages, you may only
be able to look at processes and evaluate Reactions and Feelings. If you are further down the line and
the project has more history, you will be able to move higher up the Four Levels of Evaluation. Knowing
where you are historically in the project is essential.
Once you know what you are looking for and you are clear about your objectives, the next steps are:
Selecting Data Gathering Methods
Conducting the Research
Analysing the Results
Now that we have covered the purpose of reporting, the different levels and aspects of evaluation and
setting objectives, lets look at how to get the information for the report.
Ways of Gathering Information

There are six broad ways for gathering information. Whatever approach you use, the actual methods
will fall into one of these categories.
Survey questionnaires, checklists
Interview structured or unstructured, getting first-hand responses from end users
Desk Study reviewing existing documentation: reports, publications, web sites
Observation visiting the project site and personally observing what is happening
Focus Group Discussion facilitated meetings with groups of end users around a particular
issue
Case Study an in-depth investigation over time into one particular end users experience and
outcomes of the programme
For each method, consider:
What is the overall purpose?
What are the advantages of using this method?
What are the limitations or risks?
9
Survey
The survey method is appropriate when we
need to get a lot of information quickly.
Relatively easy to design and administer,
questionnaires / surveys have benefits of
being:
Non-threatening: there is no-one to

judge responses, so end users do not
have to please the questioner
Anonymous: end users can freely

respond knowing that their identity will
be kept private
Relatively cost-free
Easy to compare: makes analysis

easier (if the questionnaire is
designed with the analysis in mind!)
Things to consider are:
Respondents might not complete carefully
Questions have to be designed carefully in order not to bias the response from the end user
This method does not develop any kind of relationship with the end user
Surveys never get the full story
Interview
Interviews help us gain some
more depth and insight to really
understand how the project is
affecting end users or how they
feel about it. Here we get both
range of information we can
cover a lot of issues as well as
depth. We can respond
immediately and investigate
deeper any responses, while
developing a relationship with the
respondent.
However, interviews take time.
While the interview may be
structured (i.e. the same
questions are asked to all
respondents), their flexible nature
means that analysis and crosscomparison may be difficult: after all, everyone has a different story. There is also the danger that the
questioner may bias the responses, or that the respondent may give face- saving answers.
10
Desk Study
Reviewing secondary
documentation is a good way to find
out how a project is doing without
actually interrupting the process.
This is from a review of reports and
other project documents.
In the best scenario, we have
access to a huge amount of
information without having to
interrupt the programmes
operations. However, it can be time
consuming; we need to know what
we are looking for or can get lost.
Sometimes the information is
incomplete or cannot be verified. Its
also inflexible: you can only use
what already exists.
Observation
Observation for example, through
visits to project sites lets us get
verifiable, first-hand information
about how things are actually done.
It is very useful for observing
processes as they happen for
example, observing a User Group
Meeting.
Despite being time consuming, a big
advantage is that we can respond
immediately. We can ask questions
to gain further understanding of
what we see, and instead of asking
what if? can actually take action
and see what happens.
However, understanding what we
see isnt always easy to interpret, and categorising the information collected can be tricky. Another
drawback of observation is that our presence can influence others behaviour, and what we see may
not be typical of what happens when we are not present.
11
Focus Group Discussion

Originally developed as a quality tool in
marketing, the Focus Group Discussion
(FGD) is useful for exploring group
perceptions on a topic in depth. These
could be reactions and feelings of end
users or exploring group concerns.
FGDs an also help us to resolve
emerging conflicts and reach
participatory decisions.
The FGD is efficient in that we can get
both range and depth of information in a
short time. It also serves to
communicate key information about our
programmes to end users.
The limitations of this method include:
Scheduling the discussion can

be can be difficult
Analysing the discussion afterwards may be complex
We need to ensure that the FGD is conducted by a skilled facilitator
Participants may not give honest responses, and try to please the facilitator or fit in with the
group
Relationships between the group members may mean that some members do not express their
ideas openly
The culture of the group may not lend itself easily to the divergence of opinion necessary to
reach true consensus
Case Study
The Case Study focuses on depth, and aims to fully
understand a particular end users experience of a
programme. It gives a full picture of the end users
experience of the project inputs, processes and results,
providing a powerful way to demonstrate the benefits of
the programme to outsiders look at how many INGOs
use case studies in their fundraising with the general
public.
The limitations are that:
It is very time consuming to collect
It has to be planned and conducted from the start

of the programme not just added later. (Thats
a Success Story)
The Case Study gives depth of information, not

breadth
Analysis and, particularly, cross-comparison, can

be difficult
12
Choosing Your Methods

Before you choose your methods, you need to be fully clear about the purpose of the investigation /
evaluation and what it will cover. We are going to look at three examples here the EU evaluation
guidelines, the field reporting format, and a specific example based on the Terms of Reference
provided by RHIYA. (You saw the summary in Module One.) We will finish this section with a checklist
to consider which applies to selecting data gathering methods for any type of report of investigation.
Evaluations
The EU guidelines are already broken down into the various sections, with questions as guidance for
each section. It is now simple enough to work down the list like this:
Source of data
Methods
Comments

What preparatory
activities were carried
out?
Were they relevant?
Were they reflected in
the final project?
Was the initial
intervention logic /
plan appropriate?
Relevance of the Project
How has the project
adapted itself over
time?
Does it meet the
identified needs of
the target group?
Efficiency
Were things done
right?
Etc.
Etc.
Sources of Data can refer to documents, but will often be people: programme staff, community
members, etc.
It will also help if you take the guidelines and develop these into a Mind Map. Mind Mapping is a key
tool for reporters which is fully explained in Module Three as a tool for data analysis.
However, it can be integrated into the reporting process from the setting objectives stage. The example
on RHIYA will demonstrate this more fully. A Mind Map of the EU guidelines can be found below.
13
14
Progress Reports
A prescribed report format helps donors / managers to get consistent information and analysis from
reporters. It also helps them to cross-compare, for example to compare the performance in one district
with another.
A lot of writers complain that the report format doesnt really help them, that they dont know how to fit
their ideas into it. Usually, this is because they look at the format at the Drafting stage after they have
conducted the research / investigation.
Preset formats can actually be extremely useful if they are used at objectives setting stage. They can
help us clarify our objectives and select the most appropriate methods to gather the data.
The format here was shared with us by the staff of one international organisation in Lao PDR.
PROGRESS REPORT
Project Title
Location
Implementing Organisation
Phase
Period
1. Events
2. Achievements
3. Planned Activities
4. Process
(a) Admin / Finance
(b) Human resources
(c) Logistics
5. Conclusions
6. Recommendations
Annexes / Other information
15
Through mapping the template (see below), we were able to:
Use the map to see the bigger picture
Visualise what each section of the report should include and how it would be organised
Select data gathering methods for each section
16
Assessments based on Terms of Reference

We saw the summary of the RHIYA assessment in the previous module. This was one case where we
used a single Mind Map throughout the assignment. At the initial stage the map was used to clarify the
Terms of Reference and select data gathering methods. Later, as data was gathered and analysed, this
was fed into the map and provided the draft for the report. The same map was used as a visual for
presentation to donors, Government and other stakeholders.
Following this process led to improved management of the assignment, more consistency, and faster
understanding of the project (after all, we were external consultants with a very short time to become
experts). In short, the work was done very quickly, and the results were good.
The original Terms of Reference stated the following objectives:
To review and assess the overall achievements and implementation status of the RHIYA
programme the final status
To prepare and present review / assessment report that covers September 2003 to February
2006
To develop and present the recommendations for future programme actions
To suggest possible linkages with different donors and national strategies
The key objectives are in the first statement: review and assess the overall achievements and
implementation status.
These Terms of Reference were then developed into a Mind Map which you can see here.
Based on this, we were easily able to identify our data sources and select the most efficient methods to
get the information required.
Methods selected were Desk Study, Interview, Survey, FGD and Observation (the last one to get an
actual feel for the project in action).
General Principles
First of all, remind yourself of your research objectives. What exactly are you trying to find out? When
choosing your methods, bear in mind that you are trying to get the most useful and relevant information
17
in the most economical way. (This is another example of the trade off between Effectiveness and
Efficiency.)
Things to consider are:
What information do we need to make decisions?
How much can we get quickly, cheaply and simply (for example, using a questionnaire)?
How reliable will the information be?
What extra methods should we use if we need more information?
Will donors or management and other decisions-makers approve the methods?
Are the methods appropriate for the target group? (Can they complete a questionnaire? Will we
be allowed access to documents?)
Can we expect the audience to conform to the methods? (Will they be able to fill out
questionnaires correctly, engage in interviews or focus groups? Are they literate? Will they just
give face-saving answers?)
Do we have the skills to conduct these methods?
Can we easily analyse the results?
Typically, a combination of methods works best. For example, a questionnaire can quickly collect a
great deal of information from a lot of people, and then interviews can get more in-depth information
from certain respondents about their answers to the questionnaires.
For a project evaluation, a simple combination of methods would typically be:
Review of proposals and periodic reports and other project related documents
A briefing on the project to the evaluation team by project staff (including implementation
process, results, challenges faced, lessons learnt and best practices)
Field visits to project districts to observe project activities, consulting with project beneficiaries,
running Focus Group Discussions, and conducting interviews with the community, Government
officials and other stakeholders
18
Extended Task
This is the first part of an extended task you will return to
several times as you work through this programme.
Eventually you will draft and edit the report, but first we
need to set our objectives and data gathering methods.
As a senior in-country staff of an international child rights
NGO, you have been asked to develop a brief (2 3
page) report on the situation of children in Nepal.
Ultimately, this document will be included in the
fundraising prospectus which is distributed to the
organisations strong funding base in the USA. The text
may also be published in various development journals
and the national press as part of your organisations
advocacy work.
How would you gather data for this report efficiently and economically?
Method
Yes / No
Comments
Survey
Interview
Desk Study
Observation
Focus Group
Discussion
Case Study
19
Suggested Solution
Dont worry if you selected different methods there is no fixed way to do this. However, applying those
general principles, we suggest:
Survey NO
As this is not an evaluation, we dont really have a target

group to survey, so we will have to pass on this method.
Interview YES
There are certainly some key people we should interview

child rights activists, representatives from concerned
Government line agencies, various other NGOs working with
a child focus.
Desk Study YES
There is a lot of existing documentation, but this will be a

major tool for us. Our own documentation should be the
primary source, as we can be sure of the methodology.
Government reports will give us the demographic information
we need. We can look on- line into the status in other
countries for comparison (WHO, World Bank indicators, etc.)
Observation YES
Although there is no specific target group and this is quite a

general report, it will be useful to see first hand some of what
we are describing: options can be visiting a local Non-formal
Education project or street children drop-in centre.
Focus Group Discussion

NO
Difficult to identify relevant target group and this is not an

evaluation.
Case Study NO
Difficult to arrange (too late!) and this is not a full evaluation.
20
Endnote
As we reach then end of this second module, lets review some of the key points.
The main issues covered were Setting Objectives and Data Gathering Methods. Weve seen that there
are four levels of evaluation Reactions & Feelings, Learning & Knowledge, Skills & Behaviour and
Effectiveness. The most effective evaluations cover all the levels, but at earlier stages we may only be
able to gather evidence for some of the levels.
We also looked at different aspects we look for in evaluation Preparation & Design, Relevance,
Efficiency, Effectiveness, Outcomes and Impact. While these may appear sometimes perplexing, they
do ensure that we conduct our investigation from a enough viewpoints to really understand what is
going on.
This whole preparation phase is essential. It allows us to clarify our objectives and build a framework /
map of what we need to know so that we can select appropriate methods to gather data. We must
clarify our objectives and know specifically what questions we have before we can proceed. It is useful
at this stage to map the objectives, and use this framework throughout the research, analysis, planning
and drafting stages. So, if there is a template for the report, you need to map this out; or develop a map
based on the Terms of Reference for the assignment. The Mind Mapping tool is more fully discussed in
the next module which covers Analysing and Interpreting Information.
21

Module II

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Module II

Uploaded by

Copyright:

Available Formats

Setting Objectives and Selecting Data Gathering Methods

Setting Research Objectives

To facilitate smooth implementation of programs

Accountability and transparency

To provide a record of progress / success

To investigate concerns from management or end users

Monitoring activities, use of resources

Gain support for our work and advocate change

Assist management thinking

Are you looking at strengths and weaknesses of a particular programme / project?

Are you looking at processes?

Or are you investigating the outcomes of the programme / project?

The Four Levels of Evaluation

Reactions and Feelings

The Four Levels and the Logical Framework

Specific Issues for Evaluation Reports

Project Preparation and Design

Relevance of the Project

Outcome and Impact

Findings and Recommendations

Quality / Sustainability / Replicability

Project Preparation and Design

What preparatory activities were carried out?

Were they relevant?

Were they reflected in the final project design?

Were any necessary preparation activities omitted?

Was the initial intervention logic / plan appropriate?

How has the project adapted itself over time?

Does it meet the identified needs of the target group?

Were things done right?

Could the results have been achieved more economically?

Did the management structure support or hinder efficient implementation?

Was the implementation methodology appropriate?

Were the right things done?

Were the assumptions in the plan realised?

Was the project flexible enough? / How was it adapted?

Should the assumptions be changed?

Outcome and Impact

Findings and Recommendations

What are the challenges and how can they be addressed?

What organisational and operational lessons can be drawn?

How can they be incorporated into ongoing activities?

Quality, Sustainability and Replicability

How replicable are the successful project activities?

Is the exit strategy well planned?

What are the quality assurance measures?

Some More Terms Explained in Detail

Field Reporting / Monitoring

Basic Information about the Project

Events during the Reporting Period (positive and negative)

Achievements (in comparison with targets)

Administrative / Financial Management

Annexes / Other Information

Approaching Reporting as a Process

Outcomes and Impact

Some specific issue or problem which has arisen

Selecting Data Gathering Methods

Conducting the Research

Analysing the Results

Ways of Gathering Information

Survey questionnaires, checklists

Interview structured or unstructured, getting first-hand responses from end users

Desk Study reviewing existing documentation: reports, publications, web sites