Switch course ▾
Session 2. Piloting an AI Tool: A Small Evaluation Plan
Objective. Design a four-week pilot of one AI use with two to four colleagues, a baseline, two or three simple measures, a data rule, a stop rule and an offline fallback, and decide whether to adopt, adapt or stop.
Many tools arrive in schools because someone liked a demonstration, and they stay because nobody asked whether they helped. A small pilot gives you a polite way to find out. It protects pupils, because a mistake affects two classes and not the whole school, and it protects colleagues, because the decision rests on what happened in your own classrooms. This lesson goes further than the key ideas: how to choose a narrow question and measures, and how to read small, messy numbers without fooling yourself.
Key ideas
Why pilot first
- A pilot is small, time-limited and reversible. It answers one question before the school spends money, time and trust.
- A good demo or a keen colleague is not evidence. The classroom is where most tools meet slow internet, mixed languages and 35 pupils.
- Keep it to two to four teachers and about four weeks. Larger pilots cost more and are rarely better designed.
The pilot card
- One question, for example: 'Does this tool cut the time to prepare reading texts at three levels without lowering quality?'
- Who and where: which teachers, which classes, which dates. Plus a data rule (nothing about named pupils goes in) and an offline fallback.
- Two or three measures and a stop rule: for example, stop at once if a serious error reaches pupils or if checking takes longer than writing by hand.
Baseline and fair evidence
- Record how things are now before you start: minutes spent, quality of the last set of texts, how many pupils reached the exit ticket. Without a baseline you can only say 'it felt better'.
- Include the cost: extra checking time, errors found, and a pupil voice (a three-question exit ticket). Volunteers are usually enthusiasts, so say who took part.
- Small numbers give signals, not proof. A new tool also feels exciting at first, so look at weeks 3 and 4, not week 1.
Decide, report, protect people
- End with one of three decisions: ADOPT (with the conditions), ADAPT (change and test again) or STOP. 'Stop' is a good result: it saved the school from a bad purchase.
- Write a one-page report with the numbers, the problems and the decision, and share it in a 20-minute meeting. Do not name pupils or judge colleagues.
- Tell parents when pupils will use a tool, and follow school rules on consent. Every step must have an offline fallback so teaching never stops.
A four-week pilot
A narrow question and honest measures
'Does the tool work?' is too big to answer. 'Does it cut the time to prepare three-level reading texts for Grade 5 without lowering language quality?' can be answered in four weeks. Choose measures that a busy teacher can record in two minutes: preparation minutes including checking, the number of errors found, a simple quality checklist applied to one sample, and a three-question pupil exit ticket without names.
Avoid claiming effects on test scores after four weeks; too many things change at once. Agree in advance what would count as a good result, for example 'at least 10 minutes saved per set and no more than two errors found per set'. Writing this before you start stops everyone from adjusting the target after they see the data.
Reading small and messy numbers
With two teachers and a few classes you cannot prove anything; you can see signals. Look at the pattern by week, not only the average. Time often rises in week 1, while people learn the tool, then falls. Ask what else changed: a holiday, an exam week, a new class. Ask who volunteered, because keen teachers may get results that others will not.
Put numbers beside voices. A pupil comment such as 'the text was easier to read' or a teacher note such as 'Uzbek sentences sounded unnatural' explains what the numbers cannot. Then choose one of three decisions: adopt with conditions, adapt and test again, or stop. Say what is still unknown, and who will check it next term.
Examples from the classroom
Week 1 costs more
Scene. Dilnoza and Aziz pilot an AI tool for reading texts in two Samarkand schools. Their baseline is 40 minutes per set. In week 1 they need 55 minutes because they are learning to ask and to check. By week 3 they need 25. If the head had looked only at week 1, the tool would have been dropped. A weekly log showed the true pattern.
Insight. Look at the trend across weeks; the first week often hides the real picture.
A holiday in the middle
Scene. Malika pilots a quiz generator with her Grade 7 class in Fergana. In week 3 Navruz holidays cut the lessons and pupils return tired, and their exit tickets dip. She does not blame the tool. She notes the holiday in the log and compares weeks 1, 2 and 4 instead. Her report says what the data show and what is uncertain.
Insight. Name other events that could explain the numbers before you credit or blame the tool.
The pilot that stopped itself
Scene. In a Bukhara school, a pilot of an AI feedback tool had a stop rule: stop at once if a serious error reaches pupils. In week 2 a draft comment contained a wrong historical date and was almost sent home. The two teachers stopped, wrote a one-page report and the head thanked them. The school avoided a purchase it would have regretted.
Insight. A stop rule turns a failed pilot into a useful decision.
Common pitfalls
- Choosing a question so broad that no measure can answer it.
- Starting without a baseline and then relying on how the tool felt.
- Changing the target after seeing the data.
- Counting only enthusiastic volunteers and forgetting to say who took part.
Repair the Pilot, Read the Data
- Open the Activity Kit: SAMPLE 'Plan A', a weak pilot plan for an AI reading-text tool at School No. 21 (3 min).
- Pairs (12 min): find at least five weaknesses in Plan A. Name the missing baseline, measure, data rule, stop rule and fallback.
- Pairs (13 min): rewrite it on a blank pilot card: one question, who and where, four weeks, two or three measures, data rule, stop rule, fallback.
- Trios (12 min): read the SAMPLE four-week results table. Say what the data show, what they do not show, and choose ADOPT, ADAPT or STOP with one condition.
- Second short activity (10 min): hold a mock 10-minute decision meeting with the script in the Kit. One person chairs, one reports, one is the head who asks 'so is it good?'.
- Alone (5 min): write your school-based task from the Kit.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which weakness in Plan A is most common in real school projects?
- What could the data not tell you, and what would you collect in the next four weeks?
- How did you explain 'adapt' to the head who wanted a yes or no?
Reflect
- Which AI use in your school deserves a pilot before it spreads further?
- What would make you stop a pilot early?
- Whose voice is missing from your usual evaluation of new tools?
Try it tomorrow
- Write one narrow pilot question for an AI use that a colleague already tries.
- Record your baseline for one week: minutes, one sample of work and a three-question exit ticket.
- Agree in writing what result would make you adopt, adapt or stop, before the first day.
Further reading and links
Pilot one question with a baseline, two or three simple measures, a data rule, a stop rule and a fallback, then decide: adopt, adapt or stop.