Switch course ▾
Session 2. Reading Research Summaries Critically
Objective. Explain an effect size in plain words, tell correlation from cause, and use a six-question checklist to judge whether a research summary or a training claim fits your own school.
Research summaries reach teachers through training slides, articles and well-meaning colleagues, and the numbers in them sound precise. A precise number is not the same as a reliable guide for your Grade 4 class in Fergana. In this lesson you go beyond the plain-words effect size from the key ideas: you learn what a number leaves out, three ordinary reasons why a gain can appear without the method causing it, and how to ask the right question when someone says 'research shows'.
Key ideas
An effect size in plain words
- An effect size answers: how big is the gap between two groups, compared with how much pupils differ from each other anyway?
- The same gap of 2 marks is large when almost all pupils score alike, and small when marks are spread from very low to very high.
- It is not a percentage of pupils who improved. Compare effect sizes only when the tests and pupils are alike.
Correlation is not cause
- Two things that go together are linked, not necessarily causing each other. Pupils who attend after-school classes score higher, but they may also be more motivated or have more support at home.
- Three explanations to try each time: A causes B; B causes A; a third thing causes both.
- To suspect a cause, we need a fair comparison: groups that were alike at the start, and one difference between them.
Averages, context and who is speaking
- A summary gives an average over many studies. Your Grade 3 class in a rural school in Fergana may differ from the pupils, subjects and years in those studies.
- Check what was measured (a test of memory, or understanding?), for how long, and whether the same people who sold the idea also measured it.
- A good summary admits its limits. Be careful when one is sure about everything.
Six questions for any claim, and the learning-styles case
- 1 Who were the pupils? 2 Compared with what? 3 What was measured, and when? 4 How big was the difference? 5 Is my school like theirs? 6 Who says so, and what do they gain?
- Case: 'teach each pupil in their learning style'. People do have preferences. But reviews of the research, for example by Pashler and colleagues, found no good evidence that matching teaching to a style improves learning.
- A better match is method to content: a map for geography, a diagram for the water cycle, practice for a skill. That helps every pupil.
From 'they go together' to 'one causes the other'
What a single number leaves out
An effect size depends on what was measured and when. A test written about exactly the topic that was taught usually shows a bigger gain than a broad end-of-year test. A short study can look different from a long one, and younger pupils may respond differently from older ones. So two numbers that look comparable may come from very different situations. Kraft makes the further point that cost and the possibility of doing it in many schools also matter when judging a result.
This is why a good summary tells you who, what and how long, not only how much. When you meet a number, ask: measured with what, after how long, in which pupils? Treat comparisons across different studies with caution unless those three answers match.
Three ordinary reasons for a gain without a cause
First, the groups may differ at the start: the class that tried a method was already stronger or more motivated. Second, time passes: pupils improve with ordinary teaching during a term, so before-and-after gains need a comparison. Third, novelty and effort: teachers who try something new often prepare harder and pupils respond to the change itself. None of these means the method is useless; they mean the result is not yet clean.
A fair comparison answers the first two by using similar groups, a baseline and the same measure. The third is answered over time: does the gain remain after the novelty fades, and can ordinary teachers get it with ordinary preparation?
Examples from the classroom
Tutoring and marks
Scene. A report on Tashkent schools says pupils who attend private tutoring have higher mathematics marks. A parent asks Malika whether the school should recommend tutoring. Malika thinks of three explanations: tutoring raises marks; pupils who already do well are the ones whose families pay for tutoring; or both come from families who prioritise school. She tells the parent that the link is real but the report cannot show tutoring is the reason, and offers advice on free revision help.
Insight. Check A causes B, B causes A and a third factor before accepting a cause.
A narrow test, a broad test
Scene. Aziz in Bukhara tries a new way of teaching fractions in Grade 6. Pupils do much better on his own end-of-chapter quiz on fractions but about the same as the parallel class on the district test that covers the whole year. He had been ready to tell colleagues the method was powerful. Instead he reports both results and says the method helped pupils with what was taught, and that the effect on broader maths is not yet known.
Insight. A test close to the taught content tends to show a bigger gain than a broad test; report both.
A summary from another country
Scene. Jasur in Samarkand reads that a teaching approach based on small groups of four helped pupils on average. His classes have 38 pupils, a long narrow room and two textbooks for three pupils. He lists what he cannot tell from the summary: the class sizes in the studies, the age of pupils and the subjects. He decides to try the approach with groups of six for one unit and compare with the parallel class.
Insight. Averages come from other settings: ask who was studied and adapt the plan to your room.
Common pitfalls
- Reading an effect size as 'the percentage of pupils who improved'.
- Comparing numbers from studies that used different tests, ages or lengths.
- Accepting 'research shows' without asking who was studied and compared with whom.
- Dropping a good idea because a summary average is small, without looking at whether it fits your pupils.
Read Three Summaries
- Pairs (10 min). Open the Activity Kit: Part A, two SAMPLE tables of marks. In both, the average gap is 2 marks. Look at the whole lists and decide which comparison shows the bigger effect, and say why in one sentence.
- Trios (25 min). Part B: three SAMPLE research or training claims. For each, apply the six questions and write what you can say and what you cannot say. Mark any place where the claim confuses correlation with cause.
- Second activity (10 min). Choose one claim and rewrite it honestly in two sentences, as a careful teacher would say it to a head of studies: what is known, what is not, what I would try first.
- Whole group (15 min): share the rewrites. Which words made a claim more honest (for example 'tends to', 'in these studies', 'we do not know yet')?
Your facilitator will guide you through this activity in the session.
Questions to discuss
- In Part A, why did the same 2-mark gap mean different things?
- Which of the six questions do you most often forget to ask?
- How can you tell a colleague that a popular claim is weak without making them feel foolish?
Reflect
- Which research claim have you repeated to colleagues without checking its source?
- Did you ever believe in learning styles? What did you do differently because of it?
- Which of your pupils' groups might differ from the groups in a typical published study?
Try it tomorrow
- Next time you hear 'research shows', ask the three questions: who, compared with what, measured how.
- Take two lists of marks from your own classes and look at both the gap and the spread before saying anything.
- Rewrite one claim from a training slide using 'tends to' and 'in these studies'.
Further reading and links
Ask how big the gap is compared with the spread, whether it shows cause, and whether it fits your school. Honest summaries say 'tends to'.