Read an Evaluation Plan Before the Results Arrive
Read an evaluation plan before results arrive to see what it can answer, what evidence it uses and which limits matter for the decision ahead.
Will an evaluation tell you whether a policy worked, or only whether people received a service? An evaluation plan can help you tell the difference before results arrive. It sets out the questions, evidence and comparisons that will shape what the eventual report can reasonably say.
Start with the decision you care about: checking a public claim, improving a programme or considering whether it should continue. Then read the parts of the plan that bear on that decision. Think of the plan as a recipe: its ingredients and method matter, but they do not tell you how the meal will taste.
Evaluation plan claims
Begin with the evaluation questions. Look for the change the plan proposes to examine, who is meant to experience it and over what period. “Did the scheme work?” is too broad to assess on its own. The plan may instead ask whether eligible households enrolled, received a service or experienced a change in a defined outcome.
These questions are related, but they are not interchangeable. A delivery question concerns what happened during implementation. An outcome question concerns a change for people or places. An impact question asks whether the policy caused a change, rather than merely occurring alongside it. If a public claim concerns reduced hardship but the plan measures only service uptake, the evaluation may still be useful; it simply cannot settle that larger claim.
Next, find the measures. Check whether each question has an observable indicator and whether the plan explains how it will be recorded. A count of appointments can show service contact, not whether someone’s circumstances improved. Administrative records, questionnaires, interviews and observations each capture different parts of the picture. The plan should make the connection between question and measure clear.
Look for definitions. Does “participant” mean everyone offered the service, those who enrolled or those who completed it? Does “employment” mean any paid work or sustained work? Without clear definitions, a result may sound precise while referring to a shifting group or outcome. Note any terms the plan leaves open; they may matter when a later report presents a headline.
Comparison and causation
If your question is whether the policy itself made a difference, turn to the proposed comparison. An outcome can improve after a programme begins without the programme being the reason. Economic conditions, other services and changes in who takes part may also affect what happens. A comparison is the plan’s way of estimating what might have happened without the policy, though no design answers that counterfactual perfectly.
Look for the comparison group or other basis for estimating that alternative. The plan might compare eligible people who took part with similar people who did not, compare places with different exposure to the policy, or examine outcomes before and after introduction. Each approach relies on assumptions. Ask how the groups differ at the outset, whether other changes could explain the result and how the plan will consider those possibilities.
Say a council is introducing a tutoring service and plans to compare pupils who attend with pupils who do not. Imagine the plan records their starting circumstances, including what support they already receive, and then describes how it will account for those differences. The resulting comparison would give readers information about whether the groups were alike at the outset; it would not, by itself, settle what the service caused. If the plan offers no way to examine those differences, treat any causal claim with caution.
A plan without a credible comparison can still describe delivery, participant experience or change over time. The useful judgement is narrower: can it answer the causal question people are asking, or is it designed to answer a different one?
Evidence for improvement
If you want to know what to change while a programme is running, give implementation questions more attention. Check whether the plan looks at reach, access, consistency and participant experience, not just final outcomes. A service can have a sound idea and still miss people who need it because referrals are unclear, opening hours do not fit or staff cannot deliver the intended support.
Find out when information will be collected and who will use it. Feedback gathered only at the end may explain what happened after the opportunity to adjust has passed. The plan might combine routine service records with interviews or observations, helping decision-makers see both patterns and possible reasons. Ask whether it distinguishes between a programme not being delivered as intended and an approach that was delivered but did not produce the expected change.
Check whose experience is represented. People who complete a service may be easier to hear from than those who leave early or never enrol. If the plan includes only the first group, it may overlook barriers that matter for improvement. Look for a clear account of how views from people with different experiences will inform decisions.
Evidence for resource decisions
If the evaluation will inform continuation or expansion, inspect its limits before treating it as a funding verdict. Ask whether the plan covers delivery costs and capacity, the settings in which the programme operates and the people it reaches. A result in one context may not transfer cleanly to another if staffing, eligibility rules or local needs differ.
Check the planned scope and timing. An assessment may capture early service use but miss changes that take longer to emerge. A plan confined to one location may offer detailed local insight without establishing what would happen elsewhere. Neither choice is automatically wrong; the question is whether the evidence will be adequate for the decision under consideration.
Look for a statement of limitations and plans to report unexpected effects. Outcomes can differ across groups, and a policy may create burdens alongside benefits. If the plan names only intended effects, ask how unintended consequences will be recorded and discussed.
What should you check?
Who is included?
Check whether the plan names the eligible population, actual participants and any comparison group separately. If these groups are blurred, ask for clarification before treating the results as representative of any one group.
What counts as success?
Look for measures tied to the stated questions, with definitions and data sources. If a measure captures activity rather than an outcome, keep that distinction in view when judging later claims.
How is the comparison built?
Find the proposed basis for estimating what would have happened without the policy and the assumptions it relies on. If no comparison is planned, treat causal conclusions as outside the plan’s clear reach.
What could the plan miss?
Check for groups, outcomes, time periods or unintended effects that sit outside the design. If the plan names a limitation, use it to interpret findings; if it does not, ask which important questions remain unanswered.
Reading note decisions
Before results appear, record the plan’s main question, primary measure, proposed comparison, one stated limitation and the decision the evaluation is meant to inform. Use these concrete checks to decide what to follow up:
- Check whether the main question matches the public claim; if it does, compare the claim with the planned answer, and if it does not, describe the narrower question the plan addresses.
- Check whether each measure has a definition and a data source; if both are clear, use them to interpret the result, and if either is missing, ask how the outcome will be identified.
- Check whether the plan separates people offered the service from those who enrolled or completed it; if it does, keep those groups distinct, and if it does not, ask whose experience the result describes.
- Check whether a comparison is planned for a causal question; if it is, examine its assumptions, and if it is not, avoid treating change over time as proof of policy impact.
- Check when information will be collected; if it is collected while changes remain possible, look for how decision-makers will use it, and if it arrives only at the end, treat it as evidence for later learning rather than immediate adjustment.
- Check whether the plan includes people who left early or never enrolled; if it does, consider their accounts alongside completers’ views, and if it does not, ask how access barriers will be identified.
- Check whether the plan describes its limits and unexpected effects; if it does, carry those limits into your reading, and if it does not, ask how outcomes outside the original aims will be reported.
When the report arrives, compare its findings with your note. Did it answer the planned question? Were the measures available, and did the comparison hold up as described? Did the report acknowledge missing evidence or changes in delivery? These checks help separate a bounded finding from a claim that reaches beyond the design.
Reading an evaluation plan in advance is most useful when you match your scrutiny to the decision at hand. For a public claim, test whether the measures match the promise; for causation, inspect the comparison; for improvement, look at delivery and whose experience is heard; for a funding choice, consider scope and limits. That is the practical value of an evaluation plan: it helps you judge what the eventual result can support, and what it cannot.
