Switch course ▾
Session 4. Analysing Simply and Claiming Fairly
Objective. Clean and tabulate data, compute simple measures of change, draw honest graphs, code interview notes, name other explanations and write a claim that fits the evidence.
Analysis is where small studies usually go wrong, not through difficult mathematics but through hurry: a single average, a dramatic graph, a claim that is bigger than the evidence. With ten or twenty pupils you do not need statistics software. You need a clear table, two or three honest numbers, a picture that starts at zero and the habit of asking, 'What else could explain this?'
Key ideas
Look first, calculate second
- Put every pupil in one row: code, before, after, change. Use only pupils who have BOTH tests and say how many were left out and why (absent, moved).
- Check the marks again against the key for a few papers. One wrong total can change a small group's mean.
- Look at the raw numbers for a minute before computing anything. Is there a pupil who stands out, or a score that looks like a mistake?
Simple measures that tell the story
- MEAN = total divided by the number of pupils. MEDIAN = the middle score when you put them in order. Use both: one very high or low score pulls the mean but not the median.
- Compare CHANGE, not only the final scores, because groups may start differently. Count how many pupils gained 2 or more, 1, or nothing.
- Example: group A went from a mean of 4.5 to 6.2 (+1.7); group B from 4.5 to 5.4 (+0.9). Seven of ten pupils in A gained 2 points; none in B did.
Honest tables and graphs
- One graph, one message. Bar charts start at ZERO, bars are labelled with the numbers, and both groups use the same scale. Never cut the axis to make a gap look big.
- For small groups a dot plot is best: one dot per pupil's change, so readers see the spread and not only the average.
- A table should have few columns and a clear title. Put the exact numbers in the table and a picture next to it; readers trust what they can check.
Pupils' words, other explanations and the claim
- Interviews: read all notes, label repeated ideas with short codes (for example R = remembered better, T = too fast) and count how many pupils mention each. Quote one short line, no names.
- List other explanations and what would check each: a different class, a more eager teacher, practice on the same test, a Navruz week lost, absent pupils. Say which you could not rule out.
- Write the claim at the strength the evidence allows: 'In these classes, pupils with the routine gained more' is fair; 'The routine causes better results in all schools' is not. Small groups call for 'repeat before deciding widely'.
Strength of a claim, from weakest to strongest
Why individual pupils matter
An average can hide more than it shows. If group A's mean rises from 4.5 to 6.2, that could be seven pupils who each gained two points, or one pupil who jumped by ten and nine who barely moved. Both give a similar average, but only the first says the routine reached most of the class. Always look at the change of each pupil and count how many gained a lot, a little or nothing.
Compare groups on change, not only on the final score, because groups rarely start equal. If one class starts at 4.5 and another at 3.0, a final difference of 1.5 points says nothing about the routine. In a small group a single absent pupil can shift the mean by half a point, so state how many pupils are in each row and why anyone was left out.
Claims that fit
There are three honest levels of claim. A description: 'In these classes, pupils with the routine gained 1.7 points and the others 0.9.' A suggestion: 'This points to the routine helping, but the groups were small and we used one test.' A recommendation: 'We propose to repeat it with the classes swapped before extending it.' Avoid causal words such as 'proves' or 'causes' unless your design can really carry them.
For every claim list at least one other explanation and what you did or could do about it: equal baselines, a second marker, a swap of classes. Pupils' words add meaning, but counting how many mention a theme only shows what was said, not whether it is true. When the evidence is mixed, say so: a mixed result is still useful because it tells the council what to test next.
Examples from the classroom
One big gain
Scene. Dilnoza in Samarkand sees the mean of her Grade 6 group rise from 5.0 to 6.4 and is pleased. When she lists each pupil's change she finds that two pupils gained five points and the other eight gained under one. She reports the median change too, and writes that the routine may help some pupils a lot and most only a little.
Insight. Look at each pupil's change: an average can hide a few big gains.
Starting from different places
Scene. Jasur in Bukhara compares two Grade 5 classes on a decimals test. Class A scores 7.1 and class B 6.3 at the end. Then he checks the baseline: A started at 6.5 and B at 5.9. The gains are 0.6 and 0.4, a far smaller gap than the final scores suggested, and he says so in the report.
Insight. Compare gains, not final scores, when groups started differently.
Counting what pupils said
Scene. Malika in Tashkent interviews six pupils about a paired-reading routine. She marks each note with R (read more smoothly), T (too long) or S (shy to read aloud). R appears in four notes, T in one and S in two. She writes 'four of six said they read more smoothly', adds one short quote with no name, and notes that she interviewed only volunteers.
Insight. Counting themes shows what was said; naming who was interviewed shows how far to trust it.
Common pitfalls
- Reporting only the average and never looking at individual pupils.
- Drawing a bar chart whose axis starts at the lower of the two means.
- Comparing final scores of groups that started at different levels.
- Using 'proves', 'always' or 'everyone' in a claim from two small groups.
Read the Evidence of a Term Study
- Open the Activity Kit: SAMPLE before-and-after scores of 10 pupils in each of two Grade 6 classes (A with the routine, B as usual), six interview notes and a blank analysis sheet. (5 min)
- Pairs (15 min): fill the change columns, then find for each class the mean before, mean after, mean change, median after, and how many pupils gained 2, 1 and 0.
- Pairs (10 min): draw two graphs on paper: a bar chart of mean change from zero, and a dot plot of the pupils' changes. Compare with the 'cut-axis' graph in the Kit and say what it exaggerates.
- Trios (15 min): code the six interview notes into themes and count them. Then list three other explanations for the result and the one check you would do for each.
- Second short activity, pairs (10 min): write the claim at three strengths (too strong, fair, too weak). Choose the fair one and add a sentence that says what you would do next.
- Alone (10 min): draw the empty analysis sheet for your OWN study (table columns, measures, graphs) and write your school-based task from the Kit.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- How did looking at the change per pupil add something to the two means?
- Which other explanation worried you most, and what is the cheapest way to check it?
- Which words in your claim would you remove if a sceptical colleague read it?
Reflect
- Have you ever reported an average without looking at individual pupils?
- Which claim in your own work could be a little too strong?
- What is the smallest extra evidence that would make your result more convincing?
Try it tomorrow
- Put your pilot data in a table with one row per pupil, then compute the change for each pupil.
- Draw a bar chart of two means on graph paper starting from zero, and put the numbers on top of the bars.
- Write your claim at the three levels (description, suggestion, recommendation) and underline the one you will use.
Further reading and links
Tabulate every pupil, compare change, draw from zero, count the pupils' words and claim only what the evidence allows.