Switch course ▾
Session 5. Designing a Small Policy Pilot and Keeping It Fair
Objective. Design a small policy pilot on one page with measures, decision rules, a budget in money and teacher minutes, and an equity check, and defend it in a short mock board meeting.
A pilot is the cheapest way to find out whether an idea fits your schools, and the fairest way to stop an idea that does not. It only works if it is designed to answer a question, if the decision rules are written before the results, and if it asks who benefits and who is missed. This lesson shows how to write rules you can live with and how to measure reach by group so that an average does not hide a gap.
Key ideas
Why pilot
- A pilot is a question you ask the system cheaply before you commit. The OECD book Governing Education in a Complex World presents experimentation as a way to govern complex systems, moving away from a fixed linear policy cycle toward approaches that adjust as the system changes.
- The same chapter names a tension: narrow experiments can be evaluated rigorously, but their results may not carry across the wider network of people involved. It suggests experimenting at an appropriate scale. A pilot in one class tells you little about a district; a pilot in four varied schools tells you more.
- Good pilots iterate. J-PAL describes Teaching at the Right Level as beginning as a pilot by Pratham in India that was evaluated and refined over fifteen years before it spread to a growing number of African countries. Plan to adjust, not just to prove.
The one-page pilot
- Write on one page: the question; the change (what teachers will actually do, in minutes); who and where (several different schools, with a comparison class or a waiting group where possible); how long, allowing for the delays you mapped in session 1; and the cost in money and teacher minutes.
- Use four kinds of measure: an outcome measure (did pupils or families gain?), a process measure (did the change really happen, and for how many?), an unintended-effects measure (workload, complaints, narrowing) and an equity measure (who gained and who did not). One measure of each is enough.
- Set the decision rules before you see results: if this, scale; if that, adjust; if the other, stop. A rule written in advance protects you from reading what you hoped to see. Course 14.7 Education Research & Policy gives the habits for reading the results fairly.
Equity is not the same as equal
- Equal means the same for everyone. Equitable means each learner gets what they need to reach the same goal. A free workbook for every pupil is equal; extra reading time for pupils who start behind, in their language of instruction, is equitable. A fair pilot says which one it is doing and why.
- UNESCO's 2020 Global Education Monitoring Report, 'Inclusion and education: all means all', finds that well-intentioned laws and policies often fail in practice, and points to governance and finance, curricula and assessment, teacher education, school infrastructure and relationships with students, families and communities as areas that can drive inclusion.
- An OECD report on equity and quality notes that a lack of fairness and inclusion can lead to school failure, and that on average about one in five young adults leaves before completing upper secondary education. Treat equity as part of the design from page one, not as a paragraph at the end.
An equity lens and the ethics of testing on children
- Ask five questions of every pilot: Who might be left out (rural pupils, girls or boys, pupils with disabilities, a language of instruction, poorer families, pupils who cannot stay after lessons)? What barrier would stop them? What adjustment removes it? Who carries the cost, in teacher time or family money? How will we see who gained and who did not?
- Testing involves children, so keep to a few firm habits. No pupil should be worse off than under current practice; a comparison group waits for the new approach rather than missing out. Tell families what is happening and why, offer an opt-out where practical, and follow school and ministry rules on pupil data and consent.
- Keep data minimal and anonymous in reports, never use a pilot to rank or judge individual teachers, and report weak or negative results as honestly as good ones. A pilot that finds a plan does not work has done its job.
The pilot cycle
Write the decision rules before the results
A decision rule has three parts: a measure, a threshold and an action. 'If at least four of five families in every school type are reached and the family check is higher than in the waiting group, scale; if rural reach is low, fix delivery first; if teacher time is above the plan, simplify.' Add a rule for a mixed result, because most results are mixed, and say who decides and when.
With four or five schools, differences can arise by chance, so treat the result as a signal for the next step, not as proof. The habits of reading results fairly from course 14.7 Education Research & Policy apply. A rule that says 'any improvement means success' will always be met and teaches nothing.
Measure reach before results
An average can look good while the pupils who most needed the approach did not get it. Tally reach by group: urban and rural, language of instruction, girls and boys, and any other group that matters in your school. Keep the tally as simple counts without names. If reach differs by group, look for the barrier before judging the approach.
Adjustments are often cheap: a different time of day, a spoken summary beside the written one, a second language version, a way to take part from home. Ask families how they would like to be told, and watch that no group is labelled by the way the pilot describes it.
Examples from the classroom
The lesson-visit round
Scene. Jasur's cluster near Samarkand wants to try a weekly round in which maths teachers visit each other for 15 minutes and talk for 15. Before starting, the four heads write the rules: stop if teacher time passes 45 minutes a week, adjust if fewer than half the teachers take part, and look at pupils' short maths check against a waiting school before deciding whether to scale.
Insight. Rules that include a stop condition protect both teachers and the idea.
The waiting class
Scene. Aziz in Fergana pilots a progress note in one Grade 4 class and keeps the parallel class as a waiting group. A parent asks why her child's class is not first. Aziz explains that the parallel class will receive the notes next term and that this lets the school see fairly whether the notes help. The parent asks to see the note when it is ready.
Insight. A waiting group is fair when everyone receives the approach in the end.
The bus at one o'clock
Scene. Malika in Tashkent runs a reading club at lunch in two rural and two urban schools. Her simple tally shows that in the rural schools few pupils who depend on the 1:00 pm bus attend. She asks the heads to move the club to 12:20 in those schools for four weeks and keeps the tally to see whether reach changes.
Insight. Check reach by group before judging results.
Common pitfalls
- Rolling out from a pilot that ran only in the strongest schools.
- Changing the success measure after the results are in.
- Treating the same treatment for everyone as fairness when some pupils cannot take part.
- Reporting good news and quietly dropping weak or negative results.
Design and Defend a Pilot
- Whole group (10 min): open the kit. The facilitator walks through the one-page pilot canvas and the equity lens using the model pilot in the answer key (read only the canvas headings aloud).
- Trios (25 min): choose idea card 1, 2 or 3 (or a real idea from your leadership tasks). Fill the pilot canvas: question, change, who and where, duration, four measures, decision rules, cost in money and teacher minutes, and two risks with actions.
- Trios (10 min): apply the equity lens. List three groups who might be left out, one barrier for each and one adjustment. Add or sharpen the equity measure on your canvas.
- Part B, mock board (15 min): join another trio. One trio presents for three minutes, starting with the ask. The other trio is the board and asks the three hard questions from the kit, then decides: approve, approve with changes, or not yet, with one reason. Swap roles.
- Whole group (15 min): which hard question was hardest? The facilitator reads the model pilot in the key. Compare your decision rules with it: which of yours would have told you to stop?
- Alone (10 min): write your leadership task from the kit and choose the person or meeting you will present to.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which decision rule on your canvas could really tell you to stop, and how would you feel if it did?
- Who was most easily forgotten in your first draft, and what adjustment brought them back?
- What would you say to a head who wants to roll the idea out to all schools now, on the strength of enthusiasm?
Reflect
- Which idea in your school has been rolled out without a pilot, and what would a pilot have told you earlier?
- Which pupils in your community are most often the last to benefit from a new programme, and why?
- What would help your colleagues accept that a pilot which stops an idea is a success?
Try it tomorrow
- Write a three-part stop rule (measure, threshold, action) for one idea in your school.
- Tally who takes part in one existing club or catch-up activity by group, without names.
- Ask one parent how they would like to be told about a new programme and what worries them.
Further reading and links
Pilot small and varied, with four measures and decision rules written in advance. Put equity into the design from page one, protect the children involved, and report honest results, including a decision to stop.