Paper submitted to the International Conference on Impacts of Agricultural Research and

Development, to be held on February 4-7, 2002, Meliá Confort Hotel, San José, Costa Rica

Impact Pathway Analysis:

An approach to strengthening the impact orientation
of agricultural research

Andreas Springer-Heinze1, Frank Hartwich2,

J. Simon Henderson3, Douglas Horton4, Isaac Minde5

Draft, 15 January 2002 – Do not quote

Contents:

1. Introduction: The research impact problem

2. The idea of the impact chain and its shortcomings
3. From impact chains to pathway analysis
3.1 Constructing impact pathways
a) Determining processes and stages
b) Analysing a pathway independent of research outputs
3.2 The methodological status of pathway analysis
3.3 Pathways as a means to enhance the impact orientation of research
4. Pathways and impact evaluation methods
4.1 Improving the design of impact evaluation studies
4.2 Combining different methods to collect and interpret evidence
5. Introducing the impact pathway: Examples in Eastern and Southern Africa

1
Dr. Andreas Springer-Heinze, Project “Knowledge systems in rural development”. GTZ, POB 5180.
65760 Eschborn, Germany. Tel +49-6196-791441, Fax-7162, E-mail: andreas.springer-heinze@gtz.de
2
Dr. Frank Hartwich, ISNAR Office at IICA - Costa Rica, POB 55-2200, San Jose, Costa Rica
Tel. Bus: +506-216-0248, E-mail: f.hartwich@cgiar.org
3
Mr. J. S. Henderson, Performance & Impact Programme, Natural Resources Institute, Central
Avenue, Chatham Maritime, Kent ME4 4TB, United Kingdom, Tel +44 (0) 1634 883296, E-mail:
j.s.henderson@gre.ac.uk
4
Dr. Douglas Horton, ISNAR, Box 93375, 5093 AJ Den Haag, The Netherlands, Tel. +31 (70)
3496197, E-mail: E-mail: d.horton@cgiar.org
5
Dr. Isaac Minde, Co-ordinator of ECAPAPA (ASARECA), Entebbe, Uganda. Tel. +256-41-321780
E-Mail: ECAPAPA@imul.com
1. Introduction: The research impact problem

Since a couple of years, research donors increasingly ask for a credible assessment of the
impact of agricultural research. Impact assessment is expected to shed light on the
effectiveness of research programmes and to provide the necessary development focus for
new investments. This applies equally to international and national agricultural research
institutes working for development goals. From the perspective of development policy, the
issue is not limited to looking back at past efforts. The objective is to make research more
impact-oriented in the first place. Research institutions have to actively strive for impact as
well as demonstrate their success.
Evaluating impact is an important means towards achieving both ends. However, if impact
assessment (IA)6 is done at all, the current practice does not address these concerns very
convincingly. There are a number of typical problems of in the way IA methodology is
frequently applied to agricultural research:
• IA is to a large extent driven by external forces and carried out by external evaluators.
This and the fact that many evaluators are economists often implies that the IA
approach gives great weight to formal rigour and is driven by methodological
concerns in the first place. This is often is at the expense of the responsiveness of IA
to the needs of potential users of the information generated. The publication interest
of some evaluators can also militate against an easy reception by users.
• Most IA studies are motivated by a need to account for past efforts and not by the
objective to foster learning and improve agricultural research per se.
• Economic IA (EIA) employs a “black-box” evaluation approach focused on research
inputs (investment) at one end and economic gains at the other, ignoring all the
processes in between. Nevertheless, even if evidence suggests that an observed
productivity change is linked to previous research, this information alone is of little use
to guide decisions on future research.
• EIA pursues aggregate impacts and is macro-oriented, hence produces little
information about the research process. Yet, we also need to understand how a
difference is made, i.e. how the research has to be organised in the first place, before
it can have a chance of generating development benefits.
• Given the complexities of agricultural change, evaluators tend to consider a few
variables only. Often, studies reduce the effort to measuring changes in technology
use or in the production figures of a particular commodity, assuming that the
observed changes can easily be attributed to research, while the technical progress
often is simply assumed to translate into social benefits. Claiming positive research
impacts without a serious effort to understand the real causes of change and
checking for alternative hypotheses is less and less credible.
• EIA often does not involve the stakeholders and potential users (donors, research
managers, researchers, farmers and other clients). Hence, there’s little buy into the
issues, methods or results and little understanding of what IA can do and where its
limitations are.
As a consequence, researchers and research stakeholders have only poor knowledge about
the way research programmes actually contribute to agricultural development and often
ignore relevant development opportunities and trends. At the same time, the typical
bottlenecks to achieving progress with technological research (e.g. a lack of complementary
services, input and product market disfunctionalities, etc.) are not well understood and taken
due account of. Following a narrow concept of impact, research institutes often do not see
the necessity to study the complex process of agricultural innovation and their own role in it
more thoroughly

6
In the following, we will use the terms impact assessment (IA)and impact evaluation as synonyms.

2
Another consequence of the current practice in conducting impact assessment is the limited
use made of IA studies. In fact, research institutes have little incentive to utilise the
information resulting from IA studies. Contenting themselves with numbers denoting adoption
rates or rates of return, researchers have little to learn for the subsequent rounds of research
planning and implementation. Instead, research managers continue to formulate research
topics on the basis of previous research, existing capacity, ad-hoc analysis of
problems/intuition, and donor interests. Research thus misses a chance to gradually improve
and become more development-minded.
Summing up, one of the root causes for the weak impact orientation of agricultural research
in many developing countries is the lack of an explicit strategy that articulates how research
is expected to contribute to development. To a considerable extent, this is due to the
limitations of current approaches to IA which are largely driven by external accountability
requirements. The disregard for the views of stakeholders and the methodological
reductionism leads to the practical irrelevance of IA results. As a result, the current practice
of IA is of little use for improving the impact orientation of agricultural research.
The present paper advances the pathway analysis approach as a means to improve the
conduct of research IA, so that it performs the function of orienting research towards impact
better and obtains greater internal value to the research managers.

2. The idea of the impact chain and its shortcomings

Enhancing the prospects for achieving a result by performing purposeful action starts by
establishing the link between the action and its intended effects. Conventional logic
expresses the relation by stating an if-then relationship: ´If action A is performed, result B
follows´. Arranging a series of such if-then relationships, we arrive at a chain that leads from
an initial activity to expected subsequent events as the basic format for describing intentional
actions.
The principle is applied in the terminology of the Logical Framework (the “logframe”), a
planning tool, which consists of four hierarchical levels, reaching from activities to outputs, to
objectives on to goals. Each level is logically linked to the next higher level by an “if-then”
statement. The repeated if-then relationships and underlying assumptions provide the “theory
of action” for a project.
For evaluation or impact assessment purposes, the logframe may be transformed into an
“impact chain”. Just as the logical framework does, the impact chain describes the intended
sequence of events linking the activities with outputs, outcomes and ultimately with “impact”.
This sequence can be visualised as follows:
Activities (Inputs) – Outputs – Outcome (Use of the outputs) – Impact
The impact chain concept and its terminology are fairly basic to evaluation (e.g. Alex, 1998).7
Its purpose is to retrace the intentions behind the initial activities, and state a possible
sequence of events that can be checked against empirical evidence. Some even make the
logical framework, and thus the linear impact chain, the sole basis for research evaluation
(e.g. Balzer / Nagel, 2001). Evaluation means to go through the entire chain from activities to
planned products on to the intended impacts (objectives), including the intermediate step
“outcome” that lies in between. If the internal logic of an impact chain sequence is assumed
to be true, an impact assessment study could can the effort to checking on the initial and the
final steps by measuring the respective indicators.
Applied to research, the basic structure of an impact chain would include the key steps
Research input – research output – use of research output – impact (on society)

7
also compare to the impact chain as applied to research on transport systems
http://www.inrets.fr/ur/dest/Sitpro2.htm.

3
In agricultural research, it is often equated with the conventional understanding of technical
change in agriculture and takes the form of the linear technology transfer concept´, spanning
the sequence
Research input – Output (a technology) – Technology adoption – Impact.
Sometimes this formulation is completed with an intermediary step of technology
dissemination / extension preceding adoption. This concept of an agricultural research
impact chain is quite widespread. At least implicitly, it can be found in many impact studies,
for which it provides the basic structure and formulates the tasks: determining the initial
research investment, describing the technologies released and measuring adoption. Only
economic studies arrive at the upper end, calculating economic benefits.
The linear chain as the normal model for agricultural innovation and the related conception of
impact studies have met with dissent. Most of the criticism is not specific to research but
applies to the idea of a linear impact chain in general:
The first objection regards the internal assumption of causality in the impact chains. In fact,
the assumed chain of events leading from an investment in public R&D projects to
development is quite long. As the analysis progresses from the origin – research – and down
along an assumed succession of effects, the observed changes become less and less
attributable to the initial research investment. Expanding the horizon of analysis leads to a
growing complexity in the study of social and economic change making it quite difficult to
single out the effect of research, a problem known as the “attribution gap” (Kuby et al., 2000,
2000; Kuby 2000). The variety of actors involved increases and hence the possibility of
conflicting views on the ongoing change and the factors driving it. Therefore, the causal
linkages between research and development are doubtful. Human development is not
“caused” by intentional action but follows an unpredictable and unique, i.e. historical path.
Yet, unless the chain contains information about the intermediate steps, it fails to provide
insight into the processes of change.
A second observation is that there is not one but many impact chains and multiple sources of
agricultural change. Development theory advances different hypotheses that explain
agricultural development not only by technology, but also by e.g. policies, structural change,
infrastructure conditions, institutional factors or social behaviour. In addition, we observe
negative trends as well, e.g. the depletion of natural resources or the dwindling biodiversity,
the origins of which need to be explained also. One has to note that, e.g., the step between
“use” and “impact” in fact covers a myriad of processes working both directions. For
example, technology use leads to changes in yields or quality, which may produce changes
in production, in prices and in income/social status, maybe driving out competitors. Social
status can have repercussions on technology use and/or affect the consumption patterns,
which might have some effect on nutritional intake and health etc. Usually, it is impossible to
detect and describe more than some of these changes. It is also highly unlikely that the
effects along the different lines can be measured as one moves away from research and
toward the expected long-term impacts. Therefore, evaluators have to be selective.
Both points apply to agricultural research as well. A problematic issue of the research impact
chain in particular, is that the “chain” follows only one activity and one output through a
hypothetical path to achieve a long-term impact. But in reality, any problem-oriented
agricultural research program would have many activities that produce many outputs, which
are expected to produce the effects in combination. So, even the assumption of one starting
point for the chain is a gross simplification.
Obviously, the impact chain concept does not offer more than a first approximation The idea
that impact is generated through a linear causation from research and technology transfer to
economic gains cannot be maintained. The best one can hope for, is to detect patterns of
(partial) change processes and to reach a consensus on the relevant aspects, leaving aside
most of the complexity.

4
3. From Impact Chains to Pathway Analysis
The chances of research making a significant contribution to the process of agricultural
change, and of credibly demonstrating it, depend on how well the innovation process is
understood in the first place. Being impact oriented and understanding impact processes are
closely related.
Our proposition is to use the chain concept as a tool to formulate research strategies as well
as to structure research impact studies. We suggest to keep the principle of a logical
sequence and the four major steps as the starting point for the analysis. But, given the
logical and practical shortcomings of the impact chain concept explained in chapter 2,
dismiss the supposition of just one standardized and linear chain. Instead, the elaboration of
impact chains is made a task in itself.
The idea is to study one or several sequences looking for the best fit to real change
processes. The result of such an exercise can be considered an impact pathway. The
approach differs from impact chains as it formulates sequences case by case constructing
pathways for each research programme in question. This implies to adapt, reformulate or
differentiate the basic sequence to fit particular commodities or farming systems. Each
pathway is another case study, relative to the specific research strategy and its objectives. It
consciously establishes an action theory for each case, instead of relying on the same type
of chain representing a standard model of innovation (e.g. of the type technology research/
extension/ adoption/ impact). Thus, it involves the task to advance hypotheses about the
processes through which research may contribute to development, identifying critical
success factors, determining their influence on objectives and their relationship to one
another. In the absence of a universally applicable theory of agricultural innovation, from
which to deduce such hypotheses, the approach limits the effort to producing plausible
arguments why a research intervention may or may not have (had) the intended
consequences.
Constructing impact pathways is not another impact assessment method but rather a way to
integrate and organise information so that the role of research in development becomes
more transparent. Its purpose is to create a framework to guide and link the different efforts
to achieve greater impact. This idea is similar to other approaches to qualitative modelling of
change processes, such as “outcome mapping” (Smutylo 2001, IDRC) or the construction of
“impact flow diagrams” (Guijt, 1998). Whereas the impact chain is derived from a planning
logic and always starts with project interventions (research inputs and outputs), pathway
analysis may as well initiate the sequence with critical factors outside a project, such as e.g.
market change. The idea of pathway analysis therefore is more open and allows to look on
reality independent of any particular research programmes.
Pathway models can be used for several purposes. For one, they allow to (re)construct the
action theory of a research programme for evaluative purposes. The pathway discloses the
assumptions about agricultural development that are implicit in the research strategy, puts
them to debate, and serves as a framework for the design of specific impact studies. In a
planning perspective, pathway analyses generate strategic knowledge that is useful for
decisions on new research topics, the planning and implementation of current programmes
and for seeking partnerships. They help to operationalise an organisation’s strategy to deliver
developmental impact. The main users, therefore, are researchers and research managers.
Rules for pathway analyses
A basic principle is that the different groups contributing to the research success, viz. the
researchers and research managers, their clients, and the planners and evaluators of
research, co-operate on equal terms. Each of these groups has specific knowledge about
(parts of) a potential pathway but none can claim superior knowledge about the pathway as a
whole.
Just like any study or planning exercise, pathway analyses are constrained by the decision-
making situation. Therefore, and that is a second consideration, it is necessary to determine

5
the procedures to follow (who to involve and how, when to stop, etc.) and the methodological
approach to take (constructing the pathway forward or backward, number of logical links
considered, empirical methods etc.) according to the initial constraints. Procedures depend
on the objectives of an evaluation or planning exercise and should not occupy more than a
reasonable proportion of the available resources.
The third aspect is the form of actually constructing and phrasing arguments. The reasoning
is presented in the form of statements about the expected (or observed) changes in
agriculture (e.g. technology choice, farm organisation, productivity of natural resources, etc.),
specifying the grounds and conditions under which these changes can be claimed to be
results of the research intervention. The decisive point is that the issues are presented in a
form that is accessible and understandable to stakeholders, so that all partners in the change
process are able to bring in their own knowledge of the ongoing processes. Impact pathways
should be described diagramatically, so that they can actually serve as a framework for
analytical work and the communication between all stakeholders. The argumentation has to
be open to controversy and invite learning. The following considerations concentrate on the
third aspect.

3.1 Constructing impact pathways

The starting point for constructing pathways is the series of steps as taken from the logical
framework. Applying them to agricultural research, we can establish a pathway by a
sequence of equivalent stages, viz. research input, research output(s), outcome (changes at
the level of research clients, i.e. use of a technology, or organisational change), and “impact”,
i.e. social change. In the logical framework, each of these stages is a measurement point at
the same time. In order to understand how the stages are linked, a pathway analysis also
specifies the processes that lead from one stage to the next. Thus, four points and three
processes may be distinguished: (1) the research process leading to the research output, (2)
the agricultural innovation process leading to technical or organisational change, and (3) the
agricultural (rural) development process leading to economic and social change. The first
process is internal to the research institution and under its control.

Fig. 1 Basic elements of an agricultural research impact pathway

Programme-internal path Programme-external path

Measuring Measuring Measuring Measuring

point point point point
Research Research Research Process of Measures of Process of Economic,
intputs process outputs agricultural technical or agricultural environ-
innovation organisational development mental and
change (e.g. social
use of a new change
technology)

Figure 1 shows the principal elements of an impact pathway in generic terms. This graphic
does not differ very significantly from the impact chain concept set out earlier. However,
there are differences:
• The concept not only states the measuring points for progress along the chain
(shaded columns) but also the processes leading up to them. This is very important
given the attribution gaps between the different stages. Even if the change in
indicators can be clearly assessed, researchers and evaluators need to know
whether that change can be linked to the previous research or may be caused by
other processes. This is in difference to the logframe-based approach to impact
6
 monitoring (Balzer/ Nagel: “Logframe based impact monitoring within the CGIAR
system”, April 2000). It is important to note, that the three processes it names, require
quite different forms of analysis.
• While an impact chain states a linear sequence, the pathway allows for other external
factors influencing the processes and being influenced by them. The arrows are
meant to characterise these effects: External variables feed into the processes and
possibly contribute to the measurable effects as well. At the same time the processes
may have other effects than the ones represented in the sequence. The
measurement of the indicators at each stage serves to account for research success
only to the extent that the analyses of the intermediate processes confirm the
existence of the chain connections.
The graphic presented in figure 1 is a generic pathway, which serves as a starting point for
pathway analysis.
Pathway analysis means to fill the abstract terms with content, to systematically devise a
concrete pathway that leads from a given research programme to development impacts. This
implies to define the content of the steps in the pathway and find indicators for the measuring
points, as well as to describe the processes linking the stages. The task is to construct a
pathway step by step, forming hypotheses about a likely sequence of events leading to
development. As this paper concentrates on impact issues, we will concentrate on the
programme-external part of the pathway in the following.

Approaches to constructing pathways

a) Determining processes and stages
Pathway analysis starts with defining a research output, i.e. the product(s) of a research
project or programme. By fixing this starting point the stage is set for identifying related
issues and interrelations as the analysis unfolds. The second step is to trace the expected
downstream effects of the research output(s). The essential point here is to avoid jumping to
quick assumptions, or simply filling in standard categories in an impact chain (e.g. “adoption
of technology”). Instead, pathway analysis has to recognize the expectations and constraints
of research users, the activities of other stakeholders, and to clarify the conditions under
which research outputs are likely to be taken up.
This is achieved by a variety of qualitative methods. The most important question is who, i.e.
which organisations, groups and individuals, are able and interested to use the products of
research. By communicating with these clients, it is possible to identify the desired and actual
changes that can be expected as the immediate consequences of using the research
products. Research clients are also able to specify other stakeholders and the critical factors
that need to be in place so that organisational and/or technical change in agricultural
production and rural livelihoods may take place. Making the logical link from research outputs
to the next measurement point requires a description of the processes in between. Basically,
describing a process involves to separate the pathway into further (sub-)sequences of
events, adding intermediate steps. A “big step” such as from research output to technical
change is subdivided into minor steps, each of which explains the connection in more detail.
This method of constructing a pathway comes close to the “outcome mapping” approach
used by IDRC, which relies on intensive collaboration with programme users. Outcome or
impact mapping is a participatory approach to reflecting about the way how a development
intervention intends to advance its goals (David et al., 2000). The method can be used to
identify expected impact and impact indicators at an early stage. Usually a map of
interrelated effects of a development intervention is drawn on board under the guidance of a
group of affected people. Metaplan techniques can be used for the purpose of identifying
critical factors, direct and indirect consequences, and causal linkages. Impact mapping is
more suitable than formal surveying when investigating intangible benefits and changes in

7
human relationships and behaviour, especially on sensitive issues such as gender effects
and equity issues.
By collaborating with stakeholders and “outcome mapping”, the pathway may gradually be
linked from the output stage to ensuing technical and organisational change. The mapping
exercise generates a more or less plausible argument, why and how the research output
feeds into the ongoing decisions and changes made by the research clients.
Another visual method is the “critical path analysis”. This is essentially a diagram in which the
various effects of development activities are represented by nodes connected by causal links (see
for example Henderson et al. 2000). Design is flexible, in so far as it can be tailored to the
characteristics of the intervention to be evaluated. The nodes of the pathway define events or
factors that are critical to the successful adoption and application of development outputs. These
“critical success factors” can be internal to the development process or external. The links in the
pathway identify the influence and dependency between the critical success factors. Analytical
tools exist to identify critical paths and probabilities.
In any case, the construction of the pathway is limited by the increasing number of factors
and stakeholders to consider in the analysis as one gets beyond the initial users of research
results. As Smutylo observes, there is a threat of “crouching impact”, as subsequent effects
along the pathway get lost in the increasing importance and turbulence of the contextual
variables. The conventional impact chain becomes a mere “chain of influence”, with the
influence of the research project not reaching much beyond its immediate users. As a
consequence, IDRC limits outcome mapping to evaluating “the changes in behaviour,
relationship, activities and/or actions of the people, groups and organizations with whom a
program works directly” (Smutylo, 2001). The idea of “chain of influence” is shown in Fig.2.

Fig. 2 A “chain of influence” (adapted from Smutylo, 2001)

Research Research Use of

Process Output Output

Yet, there might be another way of extending the pathway further downstream. This is by
starting another, second exercise in pathway analysis beginning with the next stage in the
generic pathway concept, i.e. the ongoing technical (or organisational) change. The idea is to
study the effects of technical change independent of the question to what extent it has been
influenced by research in the first place. There is a choice of economic modelling and
qualitative methods to achieve this. A qualitative approach, quite similar to the one used by
IDRC for determining outcomes, is the “impact mapping method” practised by the European
Union for evaluating economic investments of the European regional development fund. It
consists in a series of stakeholder conferences, in which the evaluators collect and jointly
analyse the consequences of the productive investment and define the respective indicators.
This goes beyond the utility of the investment for the beneficiary, though it still does not
reach far enough to capture ultimate “impacts”, the long-term social, economic,
environmental effects.
Both in the case of outcome as well as impact mapping, the exercise delivers a series of
different effects, so that the pathway is no longer linear but splits up into branches. The
research output does not only (and not directly) impinge on technology use. In the positive
case it can also influence stakeholder knowledge, decision-making and overall capacity,
which, in turn, will translate into further effects. Hence, it is likely that additional stages and

8
measurement points will be introduced that differ significantly from the generic sequence,
with which the analysis had started.

b) Analysing a pathway independent of research outputs

So far, the method for constructing the pathway started from the research output, working its
way downstream. Another promising approach to developing pathways takes the change in
technology as its point of departure. The line of sight is turned around: The evaluator looks
for the factors explaining the desired (or already given) change, rather than trying to deduce
it from the initial research output. This can lead to entirely new pathways as the factors
driving change may have little to do with research. Pathways describing development trends
can provide valuable insight both for evaluating as well as for planning research.
IFPRI has conducted a series of pathway analyses of this kind. In a definition of John Pender
et al. (1999), “a development pathway represents a common pattern of change in resource
management, associated with a common set of causal and conditioning factors” (p. 2). In the
Honduras case study of IFPRI, the pathway analysis starts with a survey of the actual
change in primary and secondary occupations in a given region over a period of 20 years.
The results of the survey are used to define typical patterns of change, such as “horticultural
expansion”, “coffee expansion” or “specialization in forestry”.
The next step is to relate the observations to a number of explanatory variables, i.e.
“pressure/shift factors”, as well as “community baseline conditions” and “local markets and
institutions”. The IFPRI study collects the necessary data through surveys and secondary
sources including, e.g. aerial photography. Alternatively, the empirical data might as well
generated by establishing lists of critical factors (markets, infrastructure, availability of
services etc.) together with stakeholders as in the impact mapping approach.
Although many combinations of variables are theoretically possible, the study confirmed that
the number of distinct pathways is relatively small. The pathways constitute stable patterns
of change that systematically differ from each other and that are “largely determined by
factors affecting comparative advantage, including agricultural potential, market access and
population pressure” (p.41). The pathway framework allows to critically assess the role of
different types of technology in the development process and thus helps to derive potential
research strategies.
Another option for developing pathways is the supply chain analysis of agricultural
commodities. Although supply chains trace the flow of production and service activities and
not the sequence of causes and effects in a change process, they may be used to determine
the explanatory variables for technical change. An example is a product innovation, e.g. a
high-value vegetable. Supply chains allow to trace back the technological requirements of
the market (e.g. in production technology, quality certification or packaging) and thus show
how research can impact on the innovation or, ex-post, how it might already have contributed
to making it possible.

3.2 The methodological status of pathway analysis

Pathways are conceptual models of innovation and agricultural change either applied to a
research area, a commodity, a natural resource, or a region. They provide a holistic view of
the change processes that can be linked, directly or indirectly, to a research programme
without attempting to capture more than those aspects that the actors involved in the
analysis deem relevant. The intervening factors, steps in the change processes and
measurement points are shown graphically in form of one or several chains or diagrams.
This representation is kept open for revision and conceived to evolve over time in
accordance with the growing knowledge about the ongoing agricultural innovation processes.
As a conceptual framework made up of discursive statements, a pathway can easily
accommodate and integrate insights derived from a variety of empirical methods (see

9
chapter 4, below). The pathway approach thus keeps the idea to link research interventions
with overall development goals expressed in the impact chain concept but tries to mitigate
some of the conceptual difficulties it entails. By combining measurable indicators with
process analyses, pathway analysis helps to address the attribution problem and take
account of the many factors other than research that drive change. Summarising the
characteristics of the pathway approach, we can retain the following suggestions:
(a) A pathway may be constructed for any particular technology or research output or for
research on a commodity, a farming system or a location. This allows pursuing the
pathway in either a “forward” direction – from research to developmental goal or
“backward” from developmental trends toward their causes.

(b) The pathway is always divided into at least three segments, with each segment and
measurement point being a separate area of study. There is no need to cover the
whole chain with the same degree of accuracy.

(c) Different methods can be combined as analysts can use different methods for each
segment. The segmentation allows to take account of the different stakeholders,
ways of reasoning change processes, and the differences in data availability.

(d) The pathway may be split up into subchains in the analysis. For example, if it turns
out that the innovation process is determined by important variables other than the
respective research output, an alternative chain may be established, in which those
variables are placed at the starting point (e.g. outputs of marketing projects instead of
research outputs). In this way, the research contribution can be assessed more
realistically and other interventions, e.g. policy decisions, are included as well.

3.3 Pathways as a means to enhance the impact orientation of research

Any internal change in research institutes has to be introduced with a view to enhance the
prospects of impact. This is most obvious when it comes to the downstream linkages from
the research outputs to their use by clients. Client and service orientation of research
therefore is regarded as a key requirement by many observers.
Yet, researchers need to look beyond the immediate use of its outputs in order to increase
the relevance of the research work. Orienting research towards impact implies observing
agricultural development and considering the role of research in it. This is where pathway
analysis comes in. The function of pathways is clearly expressed in the approach taken by
CIFOR: “A research impact strategy demands that the ´pathways´ by which research outputs
are most likely to yield impact (positive or negative) be clearly identified and re-evaluated
throughout the planning and implementation of the research effort” (CIFOR, 2000, p.3).
“Research strategies must go beyond the production of research outputs and consider
strategies to optimise impacts across the entire set of possible impact pathways.” (CIFOR,
2000, p.4).
Research managers only have control over research activities but, obviously, not over the
process of innovation and change as a whole. In terms of the pathway concept, only the
“programme-internal path” (see Fig. 1) is directly accessible to research management.
Through pathway analysis, researchers can identify and address the patterns of change
more effectively. Working the pathway backwards, the analysis reveals opportunities for
research, e.g. technological needs arising from export opportunities. At the same time, those

10
problem situations can be identified, in which it would be pointless to conduct research, e.g.
under adverse conditions of markets and infrastructure. On the other hand, the identification
of critical success factors and bottlenecks also helps to justify new research projects
precisely addressing those bottlenecks. Pathway analysis therefore is as much a tool for
discovering research opportunities as for avoiding failures.

4. Pathways and Impact Evaluation

Pathway analysis is a general approach to conceptualising impact processes and thus
generates a framework for different research planning and evaluation tasks. In the planning
field, research institutes may benefit from conducting pathway analyses by improving the
selection of research topics, improving the management of research delivery and/or by
enhancing the operational effectiveness of research programmes. Another important use of
pathways is impact evaluation, which embraces the functions of impact monitoring,
programme evaluation and impact assessment. The following sections concentrate on the
relation of pathways to impact monitoring or impact assessment (IA). There are two uses of
pathways for this purpose, in the design of impact studies and in the selection of methods for
implementing them.

4.1 Improving the design of impact evaluation studies

Using the impact pathway perspective changes the way overall impact assessment studies
are designed. Many guidelines for impact monitoring and impact assessment recommend a
typical procedure to follow (cf. Lobb-Rabe, 2000, following Thurau; Herweg at al., 1999):
According to them, establishing an impact assessment study involves a series of
methodological steps, from the formulation of IA objectives to the use of its results. All
proposals of procedures include, at some stage, the task of formulating one or several
hypotheses of how impact is achieved. Impact hypotheses should state the expected effects
for the project interventions and present a form of “theory” of how a project generates
change. The hypotheses if impact was or will be achieved are tested by measuring
respective indicators which are defined in the subsequent step of those assessment
methodologies. Pathways can be a valuable instrument in performing this task, as the
pathway approach explicitly aims at producing hypotheses and, in addition, organizes them
into a systematic framework. The pathway helps in organizing and uniting explicitly different
existing hypothesis about how impact is achieved. For example, some efforts in research on
small-scale banana production, which we assume here to be a cash crop for many
households in a particular development region, may only lead to impact in synergy with
efforts of strengthening the rural marketing system of bananas in the region. The pathway
approach in such a case would not discriminate achievements of banana production
research but would clearly indicate the critical factor of marketing that should be equally
addressed when starting on the banana production research.
Clearly, the various processes and measurement points in the pathway differ in terms of
analytical complexity and measurability. Assessing technology adoption may be relatively
straightforward, at least in some cases, while the social and economic consequences of the
technical change place difficult demands on evaluators. Pathways offer to accommodate
these differences: The concept of pathway analysis is open in the way that it allows to split
the impact pathway into several segments analysed separately. The different processes (the
research process, the innovation process and the agricultural development process) can be
studied separately and independent of each other while maintaining a comprehensive view
on the assumed causation of impact. Carrying the evaluation perspective to the aggregate
level of development, makes it necessary for the other actors besides agricultural research to
observe the ongoing agricultural development process as well. Therefore, it is useful to share
the responsibility for studying aggregate change and exchange evaluation results.

11
Another advantage consists in the communication function of a pathway perspective. It
provides a common framework that allows combining the views of stakeholders participating
in the analysis with any type of secondary information. As long as the overall structure of the
pathway remains in the format of discursive statements all stakeholders can understand,
some aspects of the pathway may be analysed through formal methods. E.g., econometric
analyses could verify the likely connection between a production increase in one commodity
(e.g. bananas) and economic welfare of farmers. This issue can be studied even
independent of the question whether the production increase has been brought about by
technology or rather by improved market access. The results of such exercises can then be
built into the pathway in the form of a qualitative statement covering one downstream
segment of the banana research pathway. This provides a hint concerning the potential
effects of banana research (provided the other factors support this as well). Within the
pathway perspective, the result might provide a meaningful argument even for those who do
not know the econometric model in detail and treat it as a black box.
Besides integrating and organising the information generated during an impact assessment
study, the well-founded description of an impact pathway is in itself an element in
demonstrating impact, especially if it is built on stakeholder perceptions. However, any
pathway remains to be a set of critical assumptions of how impact is achieved. In any case,
we can hardly expect anything beyond more or less plausible arguments corroborating its
suppositions.

4.2 Combining different methods to collect and interpret evidence

Using pathways for evaluative purposes requires to make the link to sources of empirical
information and of value judgements. It is one of the advantages of the pathway concept that
it can embrace many different types of analyses and methodologies fitting the different
stages in the pathway. Pathways thus facilitate mixed-method approaches while maintaining
the focus on the entire change processes in question. Because of its discursive nature,
pathways also allow to integrate the results of qualitative and quantitative methods referring
to the same stage in the pathway. In fact, they encourage the use of mixed-method
approaches at each stage.
Methods to be used draw from the entire impact assessment and evaluation field and may
derive from social research, economic analysis or simply logic, the indicators being
qualitative or quantitative. The available methodology can roughly be classified according to
the purpose:
• Formal models for research impact assessment: Formal, especially econometric
models conceptually link research inputs (investment) and aggregate social welfare
(e.g. economic surplus) and hence implicitly cover the whole chain. Formal models
render a separate pathway analysis obsolete. At most, pathway analysis in the sense
explicated in Chapter 3, can be used to translate the implicit assumption of the model
into discursive assertions.
• Tools for pathway analysis: These imply the different approaches mentioned in
Chapter 3, such as impact mapping, stakeholder analysis or critical path analysis.
• Individual methods to collect and interpret evidence: This category embraces all
methods that are useful analysing elements of a pathway, measuring change of
variables, checking the individual hypotheses involved in the pathway, or describing
processes according to the segments in it.
In the following, a few examples of methods fitting into the third category:
Activity and output monitoring using logical frameworks: The logical framework is used for
measuring the progress of a project or program towards the objectives. Activities, outputs and
program objective(s) are linked to verifiable indicators, stating means of verification and critical
assumptions. Monitoring has to ensure that the intervention performs well, and is done by
12
measuring indicators against predefined levels, and by checking whether activities are carried out
and outputs achieved.
Adoption studies study the rates of technology use and the effects of technology adoption on the
intended primary users of any technologies developed in the past. This includes to model the type
of adoption process (early adopters and late adopters) and carry out field surveys.
Formal and informal rural surveys: Formal surveys generate quantitative data about the
variables referred to in impact hypotheses. They are based on sampling and questionnaires.
Results of surveys can be subjected to statistical inference and are used to back a hypothetical
link within the pathway. The informal survey also is a systematic, but less structured approach to
acquire information about rural conditions and their change. “Rapid rural appraisals” include a
series of mostly qualitative data collection methods that allow to come up with empirical insight
quickly,
Case studies are an appropriate assessment technique where the conditions of the change
processes are complex or historically unique, so that only a detailed description of any particular
case can clarify the causal relationships involved. as a whole. Case studies illustrate the pathway
and provide backing to its hypotheses.
Peer review is the judgment of quality of development work by specialists of equal rank. The most
common example for peer review is the review process in scientific journals. However, peer
evaluation or expert hearings can also extend to any project activity, proposal, program, or final
product, providing insider information at low cost. Peer reviews are procedures for judgments.
They may follow a specific structure are remain open, so that the peers define the criteria against
which development efforts is evaluated themselves. The bias towards subjective perceptions can
be avoided through the design of the analyses, e.g. by using methods such as the “Delphi
method” or the “Analytic Hierarchy Process”.
Indicator measurement: Measuring and comparing indicator values over time is a conventional
method for describing change. Indicators are derived from the characteristics of the development
process, each of which provides some insight into its extent or quality. Often, a system of „adding
up“ indicator measures is applied to derive a more comprehensive measure of performance.
Elaborate indicator systems aggregate the set of partial indicators and arrive at bottom-line scores
reflecting the overall quality and impact of the development efforts conducted. Complex
aggregation systems can integrate a large set of indicators. using scoring methods to derive the
weights used for aggregating.
Economic assessments evaluate the economic benefit at the user end of development efforts
applying economic production theory. Economic impact assessment is based on the rate of
adoption of new technologies and the changes in yield and production, and derives conclusions
on their economic value in quantitative terms (e.g. income, social welfare, distribution effects) The
principle is cost effectiveness or the benefit-cost ratio.
Any analysis which contributes to understanding the impact pathway is useful, as long as it is
put into the frame of the pathway model. This excludes comprehensive and sophisticated
econometric methods which already implicitly cover the whole chain from research
investments to aggregate economic impact and thus make a parallel pathway analysis
meaningless.

5. Introducing the impact pathway: Examples in Eastern and Central Africa

Research institutions clearly have to strengthen their impact orientation, if they are to reverse
the downward trend in funding and become more effective. It is not enough to study impact.
What we need is the use of evaluation results to actually improve research processes. There
are reasons to suggest that pathway analysis can be a cornerstone to achieve this.

The ideas of “impact orientation” and the impact pathway approach have become the subject
of a new endeavour carried forward by the Association for Strengthening Agricultural

13
Research in Eastern and Central Africa (ASARECA), a regional association of agricultural
networks with support from GTZ, the German Agency for Development Co-operation.
Deliberate efforts to introduce the understanding and practice of impact pathway and impact
orientation into Eastern and Central Africa began in early 1998. Three institutions;
ASARECA, ECART and CTA with technical backstopping from GTZ held four planning
meetings to prepare for a “Regional Workshop on Impact Assessment”. This workshop was
held in Entebbe Uganda from 16-19 November 1999 (GTZ 2000). The main objective of the
workshop was to elaborate recommendations to improve the implementation of impact
assessment in Eastern and Central Africa identifying key strategic elements in the region and
providing operational guidance.. The specific objectives were to strengthen the partnership
between African and European research and development institutions, to synthesize
available knowledge and analyse the current situation of impact assessment in the region,
This workshop was attended by high profile agricultural research administrators (Directors of
Agricultural Research) in the countries of the region who occupy in a way both the demand
and supply sides for impact. “Supply” because they have to demonstrate to donors and their
governments that the research being conducted is generating positive impact to the society -
and “demand’ because, them as research administrators have to demand from their
scientists that what is being done makes a positive difference to the society.
The workshop was remarkable in several ways, as it underscored the importance and
demand for impact, identified key constraints in imparting impact culture in agricultural
research in the region, and provided legitimacy, recommendations and directives to initiate
rigorous and strategic activities to bring about impact culture in agricultural research in the
region.
Based on the recommendations of this workshop, a two-pronged regional strategy for
improving impact orientation and impact assessment in agricultural research emerged in the
ASARECA region. The first strategy entails introducing and strengthening the concept of
impact orientation in the National Agricultural Research Systems. This was initiated through
preparation and appraisal of a three-year project proposal intended to cover largely nine of
the 10 ASARECA countries. The appraisal involved key representatives from nine of the
ASARECA countries. This proposal has been submitted to the German Ministry for
International Cooperation requesting for funding.
The second strategy involved the introduction of the concepts of impact culture, impact
orientation, indicator measurement and impact pathways to all 19 regional networks,
programmes and projects (NPPs). Between July 2000 and September 2001, the
coordinators and the key socio-economists of the NPPs met four times - in Antananarivo,
Nairobi, Entebbe and Dar-es-Salaam with a view to introduce and consolidate the
understanding and application of the fundamental impact concepts.
The deliberations confirmed the relevance of a segmented impact pathway approach to
conceptualising joint evaluation efforts for the regional research programmes. Figure 3
shows the principal set-up of the impact monitoring and evaluation system, relating the steps
in a generic pathway with three stages to the notion of which hierarchical levels should be
responsible for what type of evaluation and the frequency with which the different stages
come under review. The idea is that the research outputs are monitored annually by the
different research projects participating in the program, that assessing client benefits is a
task of the regional research programmes as a whole, while the assessment of development
change is left to strategic reviews taking place every 4 to 10 years.
The framework serves as the basis for constructing the specific impact pathways for the
different programmes, where each program has to fill in the relevant content. The starting
point to achieve this are the existing logframes of the different research programs: It as been
decided to use the logframe indicators for the evaluation purpose. The challenge then is to
analyse the links between the three levels by describing the processes that lead to the
technical, organisational and economic change at the level of the research clients in the

14
respective commodity sector, as well as the overall trends explaining agricultural
development in the region.

Figure 3: The role of activity, project, programme and organization in the impact pathway

Ultimate
development
change

Organizational or
strategic level
Client
benefits

Programme Level Impact chain

Output

Project Level

Annually 3-6 years 4-10 years

Evaluation Impact Assessment

The continuing initiatives are becoming useful to scientists and administrators holding
various positions along the impact pathway in identifying and recognizing their roles in
working towards impact. For example, although the activity, project, programme and
institutional leaders all aim at contributing to ultimate development change, there are
differences in accountability and in what is expected from each one of them along the chain.
These continuing efforts will enable agricultural research to consciously organize their work
so as to manage for results.

References

Balzer, G. / Nagel, U. (2000): „Logframe-based impact monitoring within the CGIAR system”.
Paper on behalf of the Technical Advisory Committee of the CGIAR, Rome.
CIFOR (2000): „Centre Contribution: CIFOR“. Paper prepared for the Technical Advisory
Committee of the CGIAR, Rome.
David, S./ Kirkby, R. and S. Kasozi (2000): “Assessing the Impact of Bush Bean Varieties on
Poverty Reduction in Sub-Saharan Africa: Evidence from Uganda”. (Network on Bean
Research in Africa. Occasional Publication Series, No. 31). Kampala: CIAT

15
GTZ (ed.) (2000): “ECART/ASARECA/ CTA workshop on impact assessment of agricultural
research in Eastern and Central Africa”. Entebbe, Uganda, 16-19 November, 1999.
Eschborn, Germany: GTZ
Guba, E.G. / Lincoln, Y.S. (1989); “Fourth Generation Evaluation”. Newbury Park, California:
Sage Publications
Guijt, I. (1998): "Participatory Monitoring and Impact Assessment of Sustainable Agricultural
Initiatives: An Introduction to Key Elements". (SARL Discussion Paper No. 1, July 1998)
London: IIED
Henderson, S./ Burn, B./ Lowore, J., and N. Marsland (2000): “Adaptation and application of
a balanced scorecard approach to research performance assessment: the case of UK-
supported miombo woodland research in Malawi”. In “ECART/ASARECA/ CTA workshop on
impact assessment of agricultural research in Eastern and Central Africa”. Entebbe,
Uganda, 16-19 November, 1999. Eschborn, Germany: GTZ
Herweg et al. (1999): „Impact Monitoring of Soil Conservation Projects. Guidelines”.
Kuby, T. et al. (2000): “Lead Paper by the Workshop Preparation Group”. In:
“ECART/ASARECA/ CTA workshop on impact assessment of agricultural research in
Eastern and Central Africa”. Entebbe, Uganda, 16-19 November, 1999. Eschborn,
Germany: GTZ
Lobb-Rabe (2000):
Pender, J. / Place, F. and S. Ehui (1999): “Strategies for Sustainable Agricultural
Development in the East African Highlands”. (EPTD Discussion Papers No. 41) Washington:
IFPRI
Smutylo, T. (2001): “Crouching Impact – Hidden Attribution. Overcoming Threats to Learning
in Development Programs”. (IDRC Evaluation Unit), Ottawa: IDRC.
Stufflebeam, D.L. (1984): „The CIPP Model for Program Evaluation”. In: Madaus, G.F., Scriven,
M.; Stufflebeam, D.L. (Eds.): “Evaluation Models: Viewpoints on Educational and Human Services
Evaluation”. Boston: Kluwer Publications

16