Switch course ▾
Session 2. Weighing Evidence and Testing a Policy Document
Objective. Weigh evidence with levels, systematic reviews and effect sizes in plain words, and appraise a policy document claim by claim for support, fit and cost.
Not all evidence weighs the same, and a confident document is not the same as a well-supported one. Leaders are asked to agree to programmes, purchases and rules on the strength of a leaflet or a speech. This lesson gives you plain ways to weigh what you are shown: how far a study can be trusted, what a review adds, what a number means in practice and what to ask the author.
Key ideas
An evidence ladder, and its limits
- Rough order from weakest to strongest for 'did it work?': opinion or one story, a single before-and-after, a comparison without a lottery, a lottery trial, many trials combined in a systematic review.
- The higher you go, the better other explanations are ruled out. But the ladder is a guide, not a law: a careful comparison can beat a sloppy trial.
- Nutley, Powell and Davies (Nesta, 2013) argue that no single ranking fits all questions; what counts as good evidence depends on the question and the use. Interviews can explain 'why' and 'how', which trials cannot.
Systematic reviews and effect sizes
- A systematic review fixes its question, search and quality rules before it starts, then combines what the qualifying studies found. A meta-analysis adds the numbers into one average. An ordinary literature review may quietly pick what suits the writer.
- Ask of any review: how many studies, how good, from where, how different from each other? Studies with dull results are published less often (publication bias), which can flatter an approach.
- Effect size puts differences on one scale. The EEF Toolkit translates it into months of progress and rates trust with padlocks (course 11.4). Treat these as guides to compare, not promises for your class.
Value for money and fit
- Impact alone is not the decision. Compare cost in money AND teacher time, and how long results take to show. The World Bank 'Smart Buys' report sorts options into great, good, promising and bad buys for low- and middle-income countries.
- A headline lesson of that report is that buying equipment without teacher training or better school management often fails to improve learning. Check what comes with the purchase.
- Fit: evidence from another country or age group is a reason to try, not to adopt. Ask what differs: class size, language of instruction, textbook, teacher training, budget.
Five checks for any policy document
- 1 WHO wrote it and why (funder, seller, ministry, researcher)? 2 WHAT question does it answer? 3 WHICH evidence is cited, and can you open it? 4 DOES the recommendation follow from that evidence? 5 DOES it fit our context and budget?
- Red flags: one study treated as proof, only success stories, 'all schools' with no numbers, no costs, no risks, figures without a source, and claims in the summary that the body does not support.
- Judge each claim separately: supported, partly supported, not supported, or cannot tell. Then write the one question that would change your mind. A document is rarely all good or all bad.
A ladder for 'did it work?'
What a systematic review does and does not do
Think of a school inspector who decides in advance which classes to visit, how to judge them and what counts as good, then visits all of them and writes down what she found. A systematic review works the same way with studies. Because the rules are fixed before the search, it is harder to pick only the studies that agree with the writer.
But a review can only be as good as the studies inside it. Ask how many there are, how good, from which countries, and whether they differ widely. If most are small, short or from one country, say so when you report 'the review found'.
Numbers need a unit and a price
An effect size or 'three months of progress' is a comparison, not a promise. It describes an average across many classes. In your school the same approach may gain more or less, depending on how well it is carried out and who the pupils are.
Always set the impact beside the cost. Two approaches with the same gain are not equal if one takes ten minutes a day of teacher time and the other needs new equipment and a week of training. The World Bank report asks the same question at national level: which options give the most learning for each unit of money?
Examples from the classroom
The trainer's slides
Scene. A trainer visits Bukhara and shows a slide: 'Used in 40 countries.' Dilnoza, a deputy head, asks, 'Used, or shown to work?' The trainer cannot say. Dilnoza writes down three questions: what was compared, in which countries, and at what cost per school. She sends them politely and waits for written answers before signing anything.
Insight. Use is not evidence of effect; ask for the comparison and the cost.
Two cheap options
Scene. A cluster in Samarkand compares two ways to help weak readers. Option A, paired reading, takes 15 minutes a day and no money. Option B is a software licence at a fee per pupil. A summary rates both as helpful. Jasur notes that A fits the timetable and budget, while B needs a computer room most schools do not have, so he proposes a pilot of A.
Insight. Compare options on impact, cost in money and minutes, and fit, not on impact alone.
The one-country review
Scene. Malika reads a review of 30 studies that favours a homework approach. She sees that 27 studies come from one country, with large well-funded classes of a different kind than hers. She tells her board, 'On average the approach helped in those studies; for our schools we should pilot it first.'
Insight. State the limits of a review in the same breath as its result.
Common pitfalls
- Treating the highest rung of the ladder as automatically right for every question.
- Quoting 'months of progress' without the cost or the strength of the evidence.
- Believing a review because it is large, without asking who and what is inside it.
- Accepting 'research shows' when no one can name the study.
Test a Policy Extract Claim by Claim
- Open the Activity Kit: an INVENTED two-paragraph policy extract with eight numbered claims, and the claim table. Trios (5 min): read the extract once without marking anything.
- Trios (20 min): for each claim mark supported, partly, not supported or cannot tell, give the reason in a few words, and write the one question you would ask the author. Use the five checks.
- Two trios (10 min): compare your tables. For each difference, find the fact in the extract that decides it. Do not average; argue until one reading wins.
- Part B, trios (10 min): fill the traffic-light table for three options in the kit (impact evidence, trust, cost in money and minutes, fit). Choose which option you would pilot first and say what would change that.
- Whole group (10 min): the facilitator reads the key for claims 1, 4 and 7. Which wording tricks did your trio miss, such as 'up to', 'all', 'shows' or 'proves'?
- Alone (5 min): choose the real document you will appraise for your leadership task and note who will review your table with you.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which claim looked the strongest on first reading and weakest after the checks?
- How would you say 'not supported yet' to a ministry officer or a visiting trainer politely?
- When is it right to act on weak evidence, and what safeguards would you add?
Reflect
- Which policy or training offer in the last two years was adopted mostly on trust in the speaker?
- Who in your network could read a study with you and be honest about its weak points?
- What does 'good enough evidence' mean for a small, reversible pilot versus a district-wide purchase?
Try it tomorrow
- Take one claim from a document you hold and place its evidence on the five-rung ladder, in pencil.
- Write the cost of one approach you like in money AND teacher minutes per week.
- Write the one question to the author that would most change your mind, and send it.
Further reading and links
Evidence levels are a guide, reviews are only as good as their studies, and impact must be read with cost and fit. Judge every claim of a document separately and ask the one question that would change your mind.