Switch course ▾
Session 3. Evaluating AI Claims, Vendors and Risks
Objective. Test vendor and media claims with eight practical questions, spot weak evidence and data red flags, keep a short risk register for bias, errors and over-reliance, and say a respectful 'not yet' when evidence is missing.
Almost every AI product arrives with confident words: smarter, safer, proven. A teacher leader does not need to be a technical expert to test them. You only need a habit of asking the same few plain questions every time, and the patience to write the answers down. This lesson goes beyond the key ideas: it shows how to rank the strength of evidence behind a claim, and how to read data terms and total costs before a tool touches your pupils.
Key ideas
A claim is not evidence
- 'Raises results', 'AI-powered', 'safe' and 'trusted by thousands' are sales words. Ask who says it, based on what, for which pupils, and compared with what.
- Popularity is not impact. A tool used in many schools may still not improve learning, and a study run by the seller needs a second look: who paid, who measured, who was compared?
- Evidence from other countries, ages or languages may not fit Samarkand, Grade 6 or an Uzbek-medium class.
Eight questions for any vendor
- Learning and language: what pupil outcome, what evidence for pupils like ours, is Uzbek and Russian quality tested?
- Data and control: what is collected, where it is stored, who sees it, whether it trains models, whether we can delete it; can the teacher see, edit and override?
- Cost, access and exit: total three-year cost, what happens when payment stops, does it work on weak internet or paper, can we leave with our data?
Three risks and a short register
- Bias: examples and images may show only one kind of family, region or name. Test with local names and settings, and look for who is missing.
- Errors: fluent text can be wrong. Keep the checking step from course 8.7 as a rule, not a favour. Over-reliance: if pupils or teachers stop thinking first, skills fade, so keep tasks where thinking comes before the tool.
- A register has five columns: risk, how likely, how bad, our action, who watches. One page is enough, reviewed each term.
Say a respectful 'not yet'
- A good rule: no tool touches pupil data until it passes the eight questions and a small pilot. 'Free' often means paid in data.
- 'Not yet' is not 'never'. Say what would change your answer: independent evidence, Uzbek-language testing, a deletion promise in writing.
- Keep a decision file: the eight answers, the register and the date. Paper in a folder is enough, and it protects you from pressure.
Strength of evidence (weaker to stronger)
How strong is the evidence?
Evidence comes in strengths. The weakest is a testimonial ('teachers love it'). Next are usage numbers ('used in thousands of schools'), which show popularity, not learning. A report written by the seller is better, but ask who measured and who was compared. A small independent study is stronger. The strongest is an independent comparison of similar pupils with and without the tool, over enough time, using a measure of learning rather than of clicks.
Then ask about fit. Was it tested with pupils of your age, in your language, with your textbooks and your connection? An English-language result with university students tells you little about Grade 6 pupils in Samarkand. 'We could not find evidence for your case' is an honest finding. It means: pilot it before you trust it.
Data terms and the real cost
Read the data terms like a contract, because they are one. Ask in writing: what is collected, for what purpose, where it is stored, who can see it, how long it is kept, whether it is used to train models, whether it is shared with other companies and how deletion works. If an account is required, prefer school accounts without real names.
Then count the cost over three years: licences, the time teachers spend learning and checking, devices, internet and replacement if the vendor closes or raises prices. 'Free' is often paid for with data or advertising. Ask what happens to your files when you stop. Keep the answers, the date and the person who gave them in a decision file, so that a new head or colleague can see why a decision was made.
Examples from the classroom
Compared with what?
Scene. A visitor tells Jasur's Grade 6 team in Fergana that a reading app 'raises results by 30%'. Jasur asks three questions: compared with what, over how long, and for pupils like ours? The visitor cannot say. Jasur writes the three questions on a card and gives it to the head. The team agrees to pilot the app for four weeks before any purchase.
Insight. A number without a comparison is a slogan, not evidence.
The sign-in page
Scene. Dilnoza finds a free quiz app that asks pupils to register with their full name and phone number. She prints the terms and finds that data may be shared with 'partners'. She tells the deputy head 'not yet', explains why and offers an alternative: a paper quiz from the textbook. She also writes to the company asking for a deletion promise.
Insight. Reading data terms before use, and keeping a paper alternative ready, protects pupils and relationships.
Three-year cost
Scene. A Bukhara school is offered a language tool at a low first-year price. Aziz adds the second and third years, two training afternoons for 12 teachers and the cost of a spare device for the lab. The total is three times the headline. The head decides to wait and to ask for a trial price for the full period.
Insight. Count the total cost and the teacher time, not only the first price.
Common pitfalls
- Treating the number of schools using a tool as proof that it improves learning.
- Accepting 'secure' or 'compliant' without asking what is stored and who sees it.
- Judging a tool on a polished demonstration, not on a trial in your own classroom.
- Making a decision without writing down the questions asked and the answers received.
Three Vendor Sheets
- Open the Activity Kit: three SAMPLE vendor sheets (a learning app, a reading-text tool and an AI-writing detector). All names and claims are invented (3 min).
- Pairs (15 min): for each sheet fill the eight-question table with 'answered', 'not answered' or 'red flag'. Underline the claim with the weakest evidence.
- Pairs (12 min): choose one sheet to take to a pilot, or none. Write five lines of your reasoning and the three things you would still ask the vendor.
- Trios (10 min): build a five-line risk register for the chosen tool, covering bias, errors and over-reliance. Give each risk an action and an owner.
- Second short activity (10 min): rehearse the 'not yet' script in the Kit. One person is an enthusiastic vendor or colleague, the other answers calmly. Swap.
- Alone (5 min): write your school-based task from the Kit.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which claim sounded most convincing before you asked 'compared with what?'
- Which data answer would make you stop a purchase at once?
- What helped you stay respectful while saying 'not yet'?
Reflect
- Which AI claim have you heard recently that you accepted without asking for evidence?
- Which risk is most likely in your school: bias, errors or over-reliance?
- Who puts pressure on you to adopt tools quickly, and how will you reply?
Try it tomorrow
- Pick one claim you heard this month and write the three questions: compared with what, how long, for whom.
- Ask for the data terms of a tool your school uses and mark every sentence you do not understand.
- Start a decision file with one page per tool: date, questions, answers and decision.
Further reading and links
Ask who, based on what, for whom and compared with what; check data and exit terms; keep a one-page risk register; say 'not yet' with a reason.