Not just what to evaluate, but why — and how well the evaluation itself should be done
Introduction
Topic 25 established what evaluation means and introduced the OECD-DAC's six criteria for judging what a programme achieved. This topic goes one level deeper on two fronts: first, the distinct purposes an evaluation can serve, since the same programme might be evaluated very differently depending on why the evaluation is being done; and second, the principles that determine whether the evaluation itself — as a piece of work — was actually done well.
๐ฏ Learning Outcomes
- Distinguish the main purposes evaluation can serve: improvement, accountability, and knowledge generation.
- Explain OECD-DAC's two principles for applying its evaluation criteria.
- Describe the Joint Committee on Standards for Educational Evaluation's four attributes of a well-conducted evaluation: utility, feasibility, propriety, and accuracy.
- Distinguish evaluation criteria (what to evaluate) from evaluation standards (how well the evaluation itself was done).
๐ Why This Matters
OECD-DAC's own guidance is explicit that its six criteria should never be applied mechanistically — the criteria to emphasise, and how, should depend on the evaluation's actual purpose and the needs of its stakeholders. An evaluation can technically address all six criteria and still fail to be useful if it was never designed with a clear purpose in mind.
1. Purposes of Evaluation
Building on Topic 25's Raab et al. (1987) definition — that evaluation exists specifically to guide decision making — evaluations broadly serve three distinct purposes:
Improvement
Feeding findings back into the current or next cycle of programme design (Topic 9), often through ongoing, formative evaluation conducted while a programme is still running.
Accountability
Demonstrating to funders, sponsoring departments (Topic 22), or oversight bodies that resources were used appropriately and results were achieved.
Knowledge Generation
Building a broader evidence base about what works, informing needs analysis (Topic 12) and design decisions for entirely future programmes, not just the one being evaluated.
These purposes are not mutually exclusive, but they do pull in different directions: an evaluation designed purely for accountability (a clean, defensible final report) looks different from one designed purely for improvement (frequent, informal, rapid feedback loops).
2. OECD-DAC's Two Principles for Using Evaluation Criteria
The OECD-DAC's six evaluation criteria, covered in Topic 25, come with two guiding principles for how they should actually be applied:
Principle 1 — Apply Thoughtfully, Not Mechanically
The criteria should be contextualised to the specific intervention rather than treated as a fixed checklist to be worked through uniformly regardless of circumstances.
Principle 2 — Fit the Criteria to the Evaluation's Purpose
Which criteria matter most, and how deeply each should be examined, depends on why the evaluation is being conducted and what its stakeholders actually need to know — directly echoing Section 1's purposes.
3. Principles for Good Evaluation Practice: The JCSEE Standards
The OECD-DAC criteria and Raab et al.'s definition both describe what evaluation should look at. A separate, complementary question is how well the evaluation itself is conducted as a piece of work. The Joint Committee on Standards for Educational Evaluation (JCSEE) — an ANSI-accredited standards body sponsored by 17 North American professional organisations — sets out attributes of evaluation quality widely used across education, health, and development evaluation:
Utility
Does the evaluation serve the actual information needs of its intended users?
Feasibility
Is the evaluation realistic, practical, and cost-effective given genuine resource constraints?
Propriety
Is the evaluation conducted legally, ethically, and with respect for the rights and dignity of those involved?
Accuracy
Does the evaluation produce and convey technically sound, valid information about the programme's actual worth?
๐ Two Different Layers
Think of the OECD-DAC criteria (Topic 25) as answering "what dimensions of the programme should we look at?" and the JCSEE standards as answering "was the looking itself done well?" A perfectly designed evaluation that examines relevance, effectiveness, and impact (good criteria coverage) can still fail if it was conducted unethically, produced findings no one used, or was too expensive and slow to be practical — a failure of standards, not of criteria.
⚠️ A Common Tension
Utility and feasibility often pull against each other in practice: the most useful, thorough evaluation design is rarely the cheapest or fastest one to execute. Extension organisations with limited staff time and budget (Topic 10's organising elements) must regularly negotiate this trade-off rather than pretending it doesn't exist.
๐พ Extension Angle
The propriety standard has particular weight in extension evaluation: an evaluator questioning farmers about a programme's shortcomings needs the same humility and respect for local knowledge already established as a core competency in Topic 21 — an evaluation conducted in a way that embarrasses or talks down to farmer respondents violates propriety even if its data collection is technically accurate.
๐ฎ๐ณ Indian Institutional Context
ATMA and KVK reporting obligations (Topic 22) illustrate the utility-versus-feasibility tension directly: district-level extension staff, already stretched thin across administrative and field duties (Topic 19's single-purpose vs. multi-purpose history), must produce evaluation reports useful enough to justify continued funding while remaining feasible to complete within their existing time and resource constraints.
๐ Beyond Agriculture
Corporate training evaluation faces an identical utility-feasibility trade-off — a rigorous randomised-control evaluation of a training programme's business impact would be more accurate, but is rarely feasible given typical corporate L&D budgets and timelines, pushing most organisations toward lighter, faster (but less definitive) evaluation approaches instead.
Frequently Asked Questions
- OECD-DAC. (2019, revised February 2020). Better Criteria for Better Evaluation: Revised Evaluation Criteria and Principles for Their Use. Paris: OECD Development Assistance Committee Network on Development Evaluation (EvalNet).
- Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The Program Evaluation Standards: A Guide for Evaluators and Evaluation Users (3rd ed.). Thousand Oaks, CA: Corwin Press. [Joint Committee on Standards for Educational Evaluation.]
- Raab, R. T., Swanson, B. E., Wentling, T. L., & Clark, C. D. (Eds.). (1987). A Trainer's Guide to Evaluation. Rome: Food and Agriculture Organization of the United Nations. [Cross-referenced — see Topic 25.]