Meaning, Need, Features, and Indicators
Introduction
Topic 27 introduced Raab et al.'s "impact evaluation" and Kirkpatrick's "Results" level; Topic 25 introduced "impact" as one of OECD-DAC's six criteria. This topic goes deeper into impact assessment as its own specialised methodology — the most demanding, and most valuable, form of evaluation, precisely because it asks a genuinely hard question: not just "did things change," but "did this programme cause that change?"
๐ฏ Learning Outcomes
- Define impact using the counterfactual formula: impact = Y1 − Y0.
- Explain why impact assessment is needed, using CGIAR's Standing Panel on Impact Assessment as an example.
- Describe the core features and challenges of impact assessment: attribution, counterfactual estimation, time lag, and cost.
- Work through a numerical example distinguishing raw before-after change from genuine programme impact.
- List common indicators used in agricultural impact assessment.
๐ Why This Matters
Because rigorous impact assessment requires large budgets and extended timelines, many under-resourced evaluations are forced to rush studies using small samples and short timeframes — meaning weak impact assessment is often not a methodological failure so much as a resourcing failure, a distinction worth keeping in mind when interpreting any impact claim.
1. Meaning: Impact as a Counterfactual Comparison
Impact is formally defined as the difference between an outcome indicator (Y) with the intervention (Y1) and the same indicator without it (Y0): impact = Y1 − Y0. The Y0 term — the counterfactual — is what makes impact assessment distinct from simpler evaluation: it requires estimating what would have happened to the same people or area if the programme had never occurred, which can never be directly observed and must instead be estimated, typically through a comparison or control group.
2. Need: Why Impact Assessment Exists as Its Own Field
The Consultative Group on International Agricultural Research (CGIAR) established its Standing Panel on Impact Assessment (SPIA) in 1995 specifically to improve the rigour of impact studies across international agricultural research and to feed findings back into research planning. Its existence reflects a genuine need: without credible impact evidence, it is difficult to justify continued investment in agricultural research and extension to funders and governments, connecting directly to Topic 26's accountability purpose and Topic 29's instrumental use of findings.
3. Features and Challenges of Impact Assessment
Attribution
Establishing a genuine causal link between a specific programme and an observed outcome is difficult, since many other factors — weather, markets, other programmes — could also explain the change.
Counterfactual Estimation
Constructing a credible comparison of what would have happened without the intervention typically requires a control or comparison group as similar as possible to the treatment group in every respect except the intervention itself.
Time Lag
Genuine impacts — changed income, improved livelihoods — often take years to materialise, well beyond the timeline of a typical terminal evaluation (Topic 27), requiring impact assessments to be conducted well after a programme ends.
Cost
Rigorous impact assessment requires substantially more budget and time than routine evaluation, which is precisely why many under-resourced settings compromise on sample size or timeframe.
4. A Worked Numerical Example: Why Raw Before-After Change Overstates Impact
A widely used illustration from the Inter-American Development Bank's guide to impact evaluation shows why a simple before-after comparison can be misleading. Suppose both a treatment group and a control group start with the same agricultural gross margin of $100 per hectare. Over the project period:
| Group | Starting Margin | Ending Margin | Raw Change |
|---|---|---|---|
| Treatment (received the programme) | $100 | $150 | +$50 |
| Control (did not receive the programme) | $100 | $120 | +$20 |
The treatment group's raw change is +$50 — but $20 of that reflects a general upward trend affecting everyone (better weather, rising prices), captured by the control group's own change. The genuine programme impact is the difference between the two changes: $50 − $20 = $30, not the full $50. This is the same difference-of-differences logic already introduced in Topic 30's Anganwadi Worker example, now applied to an agricultural outcome.
⚠️ Without a Control Group, You Cannot Separate These Two Effects
A before-after evaluation with no comparison group — the design used in several studies referenced in Topic 30 due to real field constraints — would have reported the full $50 as "impact," overstating the programme's genuine contribution by attributing a general time trend entirely to the intervention.
5. Indicators Used in Agricultural Impact Assessment
| Indicator Category | Example |
|---|---|
| Productivity | Crop yield, agricultural gross margin per hectare |
| Adoption | Proportion of farmers adopting a recommended practice |
| Diversity | Crop/variety diversity indices (e.g., Shannon, Berger-Parker) |
| Income/Livelihood | Household income, poverty status |
| Knowledge/Behaviour | Kirkpatrick's Learning and Behaviour levels (Topic 27), as a leading indicator of eventual impact |
๐พ Extension Angle
A well-known impact study by Evenson (1997) evaluated the Training and Visit extension system in Kenya (Topic 19) — one of the earlier serious attempts to move beyond simple before-after farmer surveys toward genuine counterfactual-based impact assessment of an extension delivery model, illustrating how the concepts in this topic apply directly to the same T&V history already covered earlier in this course.
๐ฎ๐ณ Indian Institutional Context
CGIAR-affiliated research relevant to India, along with ICAR's own research evaluation processes, draws on the same SPIA-style rigour when assessing the returns to Indian agricultural research and extension investment — using control-group and counterfactual-based designs where feasible, rather than relying solely on the simpler before-after farmer surveys more commonly used in routine KVK-level programme evaluation (Topic 30).
๐ Beyond Agriculture
The same counterfactual logic underlies impact assessment in health and education programming worldwide — randomised controlled trials, the gold standard for establishing a credible counterfactual, are used to assess the impact of everything from vaccination campaigns to corporate training investments, following exactly the same attribution and counterfactual challenges described in Section 3.
Frequently Asked Questions
- CGIAR Standing Panel on Impact Assessment (SPIA). (2020). 2020 SPIA Approach to Impact Assessment for CGIAR, Technical Note No. 8. Rome: CGIAR Independent Advisory and Evaluation Service.
- Inter-American Development Bank (IADB). Designing Impact Evaluations for Agricultural Projects. Washington, DC: IADB.
- Waddington, H., et al. Evaluating the impact of agricultural extension programmes in sub-Saharan Africa: Challenges and prospects. [Source of the impact = Y1 − Y0 counterfactual definition.]
- Evenson, R. E. (1997). The economic contributions of agricultural extension to agricultural and rural development. In B. E. Swanson, R. P. Bentz, & A. J. Sofranko (Eds.), Improving Agricultural Extension: A Reference Manual. Rome: FAO. [Impact evaluation of Kenya's T&V extension system — cross-referenced, see Topic 19.]