Switch course ▾
Session 4. Measuring Results: Scale, Adapt or Stop
Objective. Read pilot data honestly with its limits, apply a decision rule set in advance, decide to adopt, adapt or abandon, and plan scaling as a series of small tests.
The end of a pilot is the moment when hope is strongest and judgement is weakest. A teacher who loves the idea sees success in any number; a sceptical colleague sees failure in the same number. Agreeing the decision rule beforehand, counting pupils, naming other explanations and treating scaling as a new test are the habits that keep you honest. This session practises them on sample data and then on your own.
Key ideas
Read the data honestly
- Compare the end measure with the baseline, taken with the same tool. Count pupils, not only averages: 'how many reached the line' is clearer than a mean mark. Compare the change in the pilot class with the change in the comparison class.
- Ask what else could explain the result: an easier end ticket, a keen teacher, the novelty of something new, different pupils, absences. A pilot with two classes and 30 pupils each gives a signal, not proof of cause.
- Add the human side: a few pupil comments, one or two work samples, the teacher's own diary. Numbers say what changed; comments help explain why.
Decide by a rule set before you start
- Write the decision rule in the plan: 'Continue if the pilot class gains at least 10 points more than the comparison class, preparation takes under 15 minutes a week, and the change is used in at least 10 of 12 lessons.'
- A rule written in advance stops hopeful reading. Without it, a teacher who loves the idea sees success in any number, and a sceptic sees failure in the same number.
- Three outcomes: ADOPT (keep it as it is), ADAPT (change something, usually to reduce cost or fit other classes, and test again), ABANDON (stop). Every outcome is a legitimate result.
Scaling is a series of small tests
- A change that worked in one teacher's class has not yet been shown to work in another. Scale in steps: your other class, one colleague, the grade team, then the school. Each step is a new small pilot with its own check.
- Separate the CORE (what must stay for it to work, such as 'draw the strip before calculating') from the FLEXIBLE parts (paper, examples, pair or solo). Write both down, so a colleague with 38 pupils can adapt without losing the core.
- Count what scaling costs: preparation, materials, a short coaching visit. If the change works only with an exceptional teacher, say so honestly.
Stopping is a result
- If the data show little gain or too high a cost, stopping is good professional practice. It saves colleagues weeks of effort and frees time for a better idea.
- Write what you learned: what the pupils did, what the data showed, why you think it happened, and what you would try next. A one-paragraph honest report is more valuable than a polished success story.
- Do not hide a weak result and do not stretch a good one. Report numbers with their limits, and thank the pupils and colleagues who took part.
Three honest decisions
Adopt
Keep it as it is
- Rule met on every condition
- Cost acceptable
- Limits stated
Adapt
Change, then test again
- Gain good, cost too high
- Works in one class only
- Fix one part
Abandon
Stop and share what you learned
- Little or no gain
- Cost too high
- Write a short honest report
What a small pilot can and cannot show
A pilot in two classes of about thirty pupils can show a signal: the change looks promising, harmless or too costly. It cannot show that the change caused the result, nor that it will work for every teacher. Differences in the groups, an easier end ticket, a keen teacher and the freshness of something new can all add to the gain.
Honest reading has three habits. Compare with the baseline and with a comparison class. Count pupils who reached the line rather than only averages. Say out loud what else could explain the result, and put two limits in your report. A small result stated honestly is more useful to colleagues than a big result stated loosely.
Scaling: adapt, do not copy
Work on scaling education innovations stresses that what happens at scale depends on how the idea is designed, how it is delivered, and the conditions around it. In a school, this means that a method that worked in your class of 24 may meet a class of 38, a different textbook or a teacher with less time.
Write down the core that must stay and the parts that may flex, and plan each step as a new small test: your other class, one colleague, the grade team, the school. Offer support at each step, such as ready templates or one visit. If the change works only with exceptional effort, record that as a result.
Examples from the classroom
Both classes improved in Bukhara
Scene. Malika's Grade 9 pilot class rose from 8 to 17 pupils fixing their own errors, out of 24. The comparison class rose from 9 to 14 out of 25. Her rule said at least 10 percentage points more than the comparison. The pilot class gained about 38 points and the comparison about 20, a difference of about 18. She adopts, but notes the comparison class also improved, so part of the rise probably has other causes.
Insight. Reading against a comparison class prevents crediting the change with everything.
Class of 38 in Namangan
Scene. Jasur tries Malika's peer-feedback cards in his Grade 9 class of 38 in Namangan with no extra support. Cards are lost and noise is high. Malika visits, and they find that her core is 'one card per pair, two minutes, one thing to fix'. Jasur uses the same core with half the cards in class, and the pairs sit closer. They agree to test again for three weeks.
Insight. Naming the core and the flexible parts lets a colleague adapt without losing what matters.
Stopping near the Aral Sea
Scene. Aziz teaches Grade 7 in a school in Nukus. His pilot, a long weekly project, raised ticket results by only 3 pupils out of 26 against a rule of 8, and took four hours of preparation a week. He abandons it, writes a one-paragraph report with what pupils said, and shares it with the grade team. A colleague was about to try the same project and instead chooses a shorter version.
Insight. An honest decision to stop saves colleagues effort and is a result of the pilot.
Common pitfalls
- Reading the end result before deciding what counts as success.
- Presenting a small, two-class result as proof that a method works everywhere.
- Hiding a weak or mixed result because the idea was popular.
- Rolling out to the whole grade next week without a new small test.
Read the Pilot and Decide
- Open the Activity Kit. Pairs (15 min): in Table A, work out the baseline and end percentages for the pilot class and the comparison class, the gain of each, and the difference. Record it in Table B.
- Pairs (10 min): in Table B list at least four other explanations for the result and write two limits you would state in a report.
- Pairs (10 min): apply the decision rule in Table C to the sample. Decide adopt, adapt or abandon, and write the reason in one sentence.
- Second activity, groups of 4 (13 min): in Table D plan the next scaling step for the sample: who, what stays as core, what may flex, what you will measure, what support you will give.
- Alone, then pairs (17 min): in Table E write the decision rule for your own pilot, if you have not yet. Then read your own data so far (a mid-pilot check is fine), and tell your partner what you decide and what the limits are.
- Alone (5 min): write your school-based task from the Kit.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which other explanation worried you most in the sample, and how could a pilot design reduce it?
- Was it hard to decide 'adapt' when the main number looked good? Why?
- What would make you able to say 'we stop' about your own favourite idea?
Reflect
- Which of your own results did you read too hopefully in the past?
- Who could review your numbers with you and challenge them kindly?
- Which colleague's class would be the best next test, and why?
Try it tomorrow
- Write your decision rule in the pilot plan with two or three numbers before you look at the end data.
- Ask a colleague to name one explanation for your result that you have not considered.
- Write the core and the flexible parts of your change on half a page for a colleague who might try it.
Further reading and links
Set the rule first, count pupils, name the limits, and treat adopt, adapt and stop as equally honest results.