Switch course ▾
Session 5. Bias, Ethics and Closing the Loop
Objective. Recognise common biases and pitfalls in school data, apply a simple data-ethics check, and close the loop with a plan, one change, a second measurement and a decision.
Numbers that guide teaching can also mislead it. A test in the wrong language, a missing group, a target that teachers begin to chase, a list with names on a wall: each can do harm even when everyone means well. This session trains your eye for the most common pitfalls and gives you a short ethics check you can run in two minutes. It then shows how to turn one finding into a small experiment, measure it again in the same way and decide by a rule written beforehand.
Key ideas
Where data go wrong
- The test does not measure what we think: a word problem in a language the pupil is still learning measures reading as much as mathematics. This is measurement bias.
- Some pupils are missing: absent, moved, not entered. The figure then describes only those who are left (selection).
- We read noise as a story, or one number as a label: 'a weak pupil', 'a weak class'. A single mark is a snapshot, not a person.
When a measure becomes a target
- Campbell's law (and the similar Goodhart's law): the more a number is used for high-stakes decisions, the more it is gamed and the more it distorts what it measures.
- Typical signs: coaching only pupils near the pass mark, marks drifting upward, difficult pupils kept out of the test, teaching only what is tested.
- Protection: use several measures (results, progress, attendance, pupils' work), look at all pupils and not only those near a line, and keep stakes low for individuals.
A simple data-ethics check
- Ask six questions: What is the purpose? Do I need names, or will codes do? Who will see it? How long will it be kept? Do pupils and parents know what is collected and why? Could it label a child or shame a colleague?
- Collect the minimum, share the minimum, delete when done. Keep lists with names in school, never in a personal messaging group.
- Never paste pupil names, marks or colleagues' results into an AI chatbot or any online tool you do not control; invented sample numbers are fine for learning how a method works.
Close the loop: plan, do, study, act
- PLAN: one question, one change, a prediction, the measure and how it is taken, the baseline, and the noise margin. DO: make the change in the lessons and note what actually happened.
- STUDY: measure the same way as the baseline, compare with the baseline, with a comparison group if you have one, and with normal variation. ACT: decide in advance a rule, then adopt, adapt or drop.
- Write one page that your team can reopen in a year: question, change, numbers, decision, what you would do differently. Share failures too; they save colleagues a term.
Plan, Do, Study, Act
When the target eats the measure
Donald Campbell, a social scientist, observed that the more an indicator is used to make important decisions, the more it is distorted and the more it distorts the process it was meant to monitor. In schools you can recognise it early: effort moves to the pupils just below a pass mark, marks creep up, weak pupils are quietly kept out of the count, and teaching narrows to the tested items. None of this needs bad people; it needs one number with high stakes.
The protection is design, not suspicion: report several measures together, include every enrolled pupil, moderate marks with a colleague, and use data to ask questions rather than to reward or punish. Also test the opposite risk: ignoring data because they might be gamed. The aim is a balanced set that is hard to game all at once.
A loop is not closed until it is measured twice
Improvement work often stops at the action: a new routine, a new worksheet, a workshop. The Plan-Do-Study-Act cycle adds what is usually missing, which is the Study step with a baseline taken before the change, the same measure afterwards and a decision rule written in advance. The Education Endowment Foundation points out that good ideas can fail through poor implementation; the Study step is how you notice whether it is happening.
Decide the rule before the data arrive, for example: 'at least 12 more pupils succeed, adopt; 5 to 11, adapt and repeat; fewer than 5, drop or redesign'. A rule written first stops hopeful reading of a modest result, and 'drop' becomes a respectable outcome because it frees time. Then keep a one-page log, so that next year's team does not rediscover the same lesson.
Examples from the classroom
Word problems in a second language
Scene. A Samarkand school gives a Grade 4 mathematics test with word problems in Russian. The Uzbek-medium class scores 52% and the Russian-medium class 78%. The head says the first class is weak at maths. Dilnoza proposes giving the same problems in Uzbek to the first class for a re-check; 71% solve them. The test had measured reading Russian as much as mathematics.
Insight. Check what the test really measures before judging the pupils; this is measurement bias.
The 80% target
Scene. A Tashkent school sets a target: 80% of Grade 9 must pass. Teachers coach 14 pupils close to the pass mark and let the weakest six slide. The rate reaches 79%. Aziz, a mathematics teacher, raises the question at a team meeting, and the team adds two measures: progress by band and the results of the weakest six, and agrees to report all three together.
Insight. When a single threshold is the target, add measures that cover every pupil.
Drop it, and be glad
Scene. Jasur's team in Bukhara trials a new 20-minute Friday review. Baseline: 41 of 120 pupils solve a mixed item. Rule: adopt at 12 more, adapt at 5 to 11, drop below 5. After six weeks: 44 of 120, three more, within noise. They drop the review, keep a one-page log, and use the freed 20 minutes to try the word-problem routine instead.
Insight. A rule written in advance makes dropping a change a clear, respectable decision.
Common pitfalls
- Giving a target such weight that the number becomes the job.
- Sending lists with pupil names to a group chat or pinning them on a wall.
- Changing three things at once, so that nobody can tell which one worked.
- Declaring success without a baseline or with a different test the second time.
Spot the Pitfall, Then Close a Loop
- ACTIVITY A (40 min). Open the Activity Kit: six Pitfall cards and the Ethics check. All sample scenes. Groups of three or four.
- Groups (25 min): for each card, name the pitfall (measurement bias, selection, gaming, noise, labelling and privacy, no baseline), the harm it could do, and a fix. Apply the six ethics questions to the cards that involve names.
- Whole group (15 min): each group presents its worst card and its fix. Agree a list of three 'never do' and three 'always do' rules for your own schools.
- ACTIVITY B (30 min). Pairs fill in the one-page PDSA sheet in the Kit. Use the sample (word problems, Grade 5) or an anonymised issue from your own school team. Write a decision rule before any new data exist.
- Pairs (15 min): fill the PLAN box with a question, one change, a prediction, the measure, the baseline and the noise margin, then the decision rule (adopt, adapt, drop) and the date of the second measurement.
- Swap with another pair (10 min): ask the five reviewer questions in the Kit. Then alone (5 min): write your school-based task.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which pitfall have you seen in your own school? What would have been a kinder and more accurate way to use the same data?
- What does a decision rule written in advance protect you from?
- What will you tell pupils and parents about the data you collect and why?
Reflect
- Which number in your school has become a target, and what has it started to change in classrooms?
- Which improvement did you adopt without a second measurement? What would you now check?
- Who in your school is responsible for protecting pupil data, and do colleagues know who that is?
Try it tomorrow
- Pick one measure your school reports and write what it might distort if it became a target.
- Run the six ethics questions on one list or sheet you keep, and fix the weakest answer.
- Write a PDSA sheet for one change with a baseline, a decision rule and a date for the second measure.
Further reading and links
Check the test, who is missing and what the target does to behaviour; protect names; then act, measure the same way again and decide by a rule written in advance.