Switch course ▾
Session 5. Measuring at Scale, Stopping Rules and the Scaling Brief
Objective. Read results across schools by fidelity and school type instead of one average, write stopping rules with numbers before launch, and present a one-page scaling brief with a clear decision.
Once a practice is in many schools, the numbers arrive in many shapes, and it is tempting to report the best-looking one. Leaders who last are those who set up fair, simple measures before the start, read results by school type, diagnose disappointing results before reacting, and stop on written rules without blaming people. In this session you practise exactly that, and finish with a one-page scaling brief you can take to your board or district.
Key ideas
Do not expect the pilot's result to repeat
- Results often change when a practice leaves its inventor: ordinary teachers, thinner support, harder schools. Banerjee and colleagues describe how the early versions of Teaching at the Right Level did not produce effects in the regular government school system, and how repeated testing and redesign produced two versions that raised learning at scale.
- So measure at every ring, not only at the end. Five things matter: REACH (how many classes), FIDELITY (core moves seen), OUTCOMES against the baseline, EQUITY (results by school type) and COST per pupil. If you measure only outcomes, you cannot tell whether a weak result is the idea or the delivery.
- Use the same simple measure in every school (for example the same end-of-unit task marked with the same key) and record it as counts, such as 9 of 30 pupils, not only percentages. Small schools have few pupils, so one pupil moves their percentage a lot; say so.
Read the variation, not only the average
- An average gain across 12 schools can hide three schools with no gain at all. List every school: type, pupils, correct before and after, core moves seen out of five. Then group them by fidelity and by gain and ask what each group needs.
- Four groups: HIGH fidelity and good gain: LEARN FROM them. High fidelity and small gain: RETHINK the idea or the measure. LOW fidelity and good gain: CHECK what else helped. Low fidelity and small gain: fix SUPPORT. Treat these as clues; without a comparison group you cannot prove cause.
- A cheap fairer comparison: if schools start in two waves, the schools that start next term can act as a comparison for this term, provided they are similar and you state differences. Always report results separately for village and town, small and large schools, and by language of instruction.
Stopping rules written before launch
- Course 14.5 taught you the stop list and red lines for a pilot. At scale you need rules for the whole programme, written before launch while nobody is emotionally attached: triggers with numbers, the action (PAUSE, ADJUST or STOP) and who decides. Written rules make stopping a planned decision, not a defeat.
- Five typical triggers: (1) harm to a pupil or broken consent: STOP at once; (2) core moves seen in fewer than half of classes after two cycles despite coaching: PAUSE expansion and fix support; (3) no gain in most schools with good fidelity after two cycles: STOP scaling and return to pilot; (4) cost per pupil above the agreed limit or no year-3 payer: ADJUST or stop; (5) the gap between school types widening: PAUSE and add support.
- When you do stop, stop well: tell schools and thank them, keep and share what worked, archive the materials and the lessons learned, and release the hours for something else. A programme that ends honestly leaves teachers more willing to try the next idea.
The scaling brief: one page for a decision
- A scaling brief gathers the whole course on one page: the practice and its CORE; the evidence and its limits; the rings and the MODEL; the support and THREE-YEAR COST with the year-3 payer; EQUITY (barriers and adaptations); GOVERNANCE (owner and decision rights); MEASURES with thresholds; STOPPING RULES; and the DECISION you ask for.
- Keep the habits from course 14.5: lead with the decision, give counts with their limits, say 'suggests' not 'proves', and credit colleagues. Add what is new at scale: results by school type, what you do NOT yet know, and what would make you stop.
- Ask for a staged decision, not a blank cheque: approve Ring 2 for one term with the thresholds above, and meet again to decide Ring 3. Staged decisions are easier to approve and easier to reverse, and they keep the evidence in step with the spending.
Diagnose before you decide
A measure schools can compare
Choose one task for the whole network: the same end-of-unit task, the same marking key and the same timing in every school. Take the baseline before the practice begins. Record counts, such as 11 of 30 pupils correct, because small schools make percentages jump. Add a balancing measure from course 14.5, for example lesson time or pupils' confidence, so that a gain is not bought at a hidden cost.
To keep marking fair, run a simple moderation: each school sends 10 anonymous scripts, two teachers from different schools mark them with the key, and any disagreements are discussed. This costs a few hours and protects the whole programme from the doubt that one school marked generously. Do not use results to rank teachers; use them to decide where help should go.
When results disappoint: diagnose before you decide
A weak result can have several causes, and each leads to a different action. Ask in order: Was the practice delivered (core moves seen)? Was the result measured fairly (same task and key)? Did the context differ (a new timetable, many absences)? Is the idea itself weak, or is the support thin? Only after these questions should the decision follow: continue, adjust, pause or stop.
The study of Teaching at the Right Level in government schools shows why this patience pays: early designs did not work in the regular system, and repeated testing and redesign produced two versions that did. Stopping early is right when the rules say so, but giving a redesigned version a fair test in one more ring is often the better use of the evidence you have.
Examples from the classroom
A small moderation
Scene. Eight schools in a Samarkand cluster send 10 anonymous scripts each, 80 in all. Two teachers from different schools mark each script with the key. They disagree on 9 scripts, mostly about how much explanation earns the second mark. They revise the key with an example for each mark and re-mark those 9. Pupils' counts changed in three schools by one or two pupils.
Insight. A small moderation makes results comparable and protects the programme's credibility.
Diagnosing a quiet school
Scene. A mountain school with 15 pupils shows no gain. Before judging the idea, the network lead checks: core moves seen 2 of 5, no coaching visit in three weeks because the road was closed, and the lessons were moved to Friday afternoon. She arranges a phone check-in, a buddy visit after the road opens and a timetable change. She does not mark the school 'failed'.
Insight. Low fidelity with no gain points first to support and conditions, not to the idea.
Stopping well
Scene. After two cycles, a Karshi district programme meets its written stopping trigger: fewer than half of classes show four core moves despite coaching, and the year-3 payer withdrew. The steering group, chaired by Gulnora, stops the expansion. Schools are thanked in writing, the card and task sets are kept in each school, and the three schools that used it well continue with their own budget.
Insight. A planned, respectful stop keeps what worked and leaves teachers willing to try again.
Common pitfalls
- Changing the task or the key between schools and then comparing the results.
- Reporting only the best school or the pooled average, and not results by school type.
- Reacting to a weak result with blame before checking delivery, measurement and conditions.
- Continuing past a written trigger because the programme has become a matter of pride.
Measure, Decide, Brief
- Pairs (20 min): in Table A, calculate the gain in percentage points for each of eight sample schools, then the pooled figures for all schools, town schools and village or mountain schools. Assign each school to a group (LEARN FROM, RETHINK, CHECK or SUPPORT).
- Pairs (15 min): in Table B, write five stopping rules with numbers for the sample programme, each with an action and who decides.
- Trios (15 min): in Table C, find the problems in a weak draft brief and rewrite each sentence so that it is honest, specific and asks for a decision.
- Part B, alone (25 min): in Table D, draft the one-page scaling brief for YOUR practice and network, using your work from sessions 1 to 4.
- Pairs (10 min): present your decision line and your first stopping rule aloud in one minute. Your partner asks one hard question; note it and add a line to the brief.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- What did the pooled average hide in the sample data, and who would have been left behind if you had only read the average?
- Which of your stopping rules would be hardest to carry out when the time comes, and who will make sure it is?
- What is the one thing you do NOT yet know that your brief must say plainly?
Reflect
- Which programme in your area was extended on the strength of one good average, without looking at school types?
- Who in your system has the authority to stop a programme that is not working, and do they know it?
- What would you want a school board to ask you before approving a scale-up?
Try it tomorrow
- Choose one task and key that every school in your network could use, and write down when the baseline will be taken.
- List your five typical stopping triggers with numbers and the person who decides each.
- Draft the first two lines of your scaling brief: the decision you ask for and the result with its limits.
Further reading and links
Measure at every ring, read results by fidelity and school type, write numbers into your stopping rules before launch, and ask for staged decisions on one honest page.