Switch course ▾
Session 5. Fair Tests, Teaching to the Test and Reporting Results
Objective. Review items for bias and unnecessary language load, tell good test preparation from teaching to the test, and report results by outcome in a way that colleagues, pupils and parents can use.
A test can be well planned and well written and still be unfair, because something outside the subject decides who answers correctly: an unfamiliar setting, a long sentence, a stereotype, a marking habit. A fair test can also be used badly when results become a ranking or when preparation squeezes out the rest of the curriculum. This lesson widens your view from the item to the whole assessment: checking fairness quickly, how pressure on scores changes teaching, and how to report so that results help rather than label.
Key ideas
Where bias hides in an item
- Bias means an item is harder or easier for some pupils for reasons unrelated to the skill being tested. Common sources: a context some pupils do not know (a ski resort, a metro map for pupils from villages), a heavy language load, stereotypes about who does what, and unfamiliar formats.
- Language load matters in Uzbek- and Russian-medium schools, and for pupils who study in a language they speak less at home. If reading is not the target, use short sentences, familiar words and no double negatives.
- The test: would two pupils with the same skill but different backgrounds, home languages or gender have the same chance of the right answer? If not, the item is measuring something besides the skill.
A short fairness review routine
- Ask a colleague who is not the author to read the test with a short checklist for 10 minutes: unfamiliar contexts, stereotypes, language difficulty, unclear instructions, formats the class has not met. Mark each item OK, REVISE or QUESTION.
- Ask two or three pupils to say in their own words what an item asks. If they read it differently from your intention, rewrite it. Use a balance of names, settings and roles (Dilnoza and Aziz, village and city, many kinds of work).
- Where the school runs parallel Uzbek and Russian versions, do not translate word for word. Ask a bilingual colleague to check that both versions are equally clear and equally difficult. Follow school and ministry rules for extra time or other adjustments, and record what was given.
Good preparation or teaching to the test?
- Good preparation: teach the whole curriculum well, spend a lesson or two on the format (reading instructions, managing 40 minutes, filling an answer sheet), and use past-style items as one source of practice beside textbook tasks.
- Narrowing: dropping topics that did not appear recently, replacing practicals and projects with timed papers, teaching tricks instead of understanding, or practising the exact items to be used. Researchers who study high-stakes testing, such as Daniel Koretz, warn that the more a score matters, the less it may tell about real learning.
- A check: give the same content in a new form after a delay. If pupils can solve practice-style items but not the same idea in a different form, the preparation has narrowed learning.
Reporting results so they can be used
- Report by outcome, not only by total. 'Aziz reached the ratio and percentage outcomes; reasoning in a new situation is the next step' helps more than '14 out of 20'. Add one concrete next step and return work quickly so pupils can correct it.
- Do not rank pupils in public or compare them by name. For colleagues and the school, give a class summary by outcome (for example 'In my class of 24, 20 reached outcome B and 6 reached outcome D'), with what the test did and did not cover.
- Say only what the data show. One class test on one day does not describe a pupil's ability or your whole school, and it cannot be placed on the scale of an international study. Keep results private to those who need them.
Preparing pupils for a test
Good preparation
- Teach every topic
- One lesson on format
- Past-style items as one source
- Explain and re-solve in new form
Grey zone
- Test-taking tips
- Timed practice
- Always check content comes first
Narrowing
- Drop untested topics
- Replace practicals with papers
- Practise the exact items
Fairness goes beyond one item
Fairness has several layers. The item layer is the bias checklist. The administration layer asks whether every pupil sat the test under similar conditions: same time, same instructions, a quiet room, any agreed adjustment recorded. The marking layer asks whether the same answer earns the same mark whoever wrote it: mark without names, count spelling and handwriting only if they are part of the outcome, and use one rubric for every script.
Then there is opportunity to learn. A test is fair only to pupils who were taught the content. If a unit was cut short or a class missed three lessons, note it and do not read the results as a verdict. Work on culturally responsive assessment suggests tasks should respect pupils' backgrounds and not depend on one cultural setting; vary settings and names to apply this.
Pressure on scores and honest reporting
When scores carry high stakes for pupils, teachers or schools, pressure to raise them grows. Daniel Koretz, who studies this, argues that the more a score matters, the less reliably it may reflect real learning, because teaching shifts toward what is tested. Your defence is simple: teach the whole curriculum, vary item forms, and check from time to time that learning transfers to a new form of the same idea.
Report in a way that supports learning. Black and Wiliam describe feedback that tells a pupil about the specific qualities of the work and how to improve it, without comparing pupils. Do the same in reports: outcomes, evidence, a next step. For colleagues, give class summaries by outcome with the number of pupils, and say what the test did not cover. Never place a class score on an international scale.
Examples from the classroom
The map the village pupils had never seen
Scene. Dilnoza's Grade 8 item in Samarkand asks pupils to read a metro map of Tashkent to plan a journey with changes. A colleague in a village school nearby says her pupils have never used a metro. The skill tested is reading a network and counting changes. Dilnoza keeps the skill but adds a small bus-route diagram, which all pupils can read in the item itself. Other teachers adopt the approach.
Insight. Remove the barrier unrelated to the skill, or give the information the item needs.
Two languages, one test
Scene. A Grade 7 school in Tashkent gives the same maths test in Uzbek and in Russian. Aziz notices that the Russian version of one item uses a rare word for 'discount', and the Uzbek version uses a longer phrase. A bilingual colleague rewrites both with the everyday word and checks that the two versions are equally clear. The mathematics is unchanged. Parents read both versions the same way.
Insight. Parallel language versions need checking for equal clarity, not word-for-word translation.
A summary instead of a ranking
Scene. At a staff meeting in Fergana, the head of studies asks Jasur for a ranked list of his Grade 8 pupils. Jasur brings a class summary by outcome: in his class of 24, 20 reached the ratio outcome and 6 reached the context-problem outcome. The staff agree to add one reasoning task to every Grade 8 unit. No pupil is named and nobody is ranked.
Insight. A summary by outcome gives the school something to act on without labelling pupils.
Common pitfalls
- Checking bias only on the items and forgetting administration conditions and marking habits.
- Reading results from a class that missed lessons as a verdict on the pupils.
- Using the exact practice items on the test and calling it preparation.
- Comparing a class test score with an international scale or ranking pupils in public.
Fairness Review, Preparation Sort and Report
- Trios (10 min): open the Kit. Use the bias checklist on the four sample items. Mark each OK, REVISE or QUESTION and write the reason in a few words.
- Trios (10 min): rewrite the weakest item so it keeps the skill but removes the unrelated barrier. Check your rewrite with the 'same skill, same chance' test.
- Trios (10 min): sort the eight preparation practices in the Kit into GOOD, GREY and NARROWING. Agree one sentence you could say to a head of studies about the grey ones.
- Pairs (15 min): from the Kit data write (a) a four-sentence outcome-based note for a parent and (b) a three-line class summary for colleagues with one decision for teaching. Swap and check that nothing goes beyond the data.
- Second short activity, pairs (15 min): role-play the conversation with a head of studies who asks for a ranked list of pupils. Use the five-step script in the Kit. Swap roles and give one piece of feedback each.
- Alone (10 min): write your school-based task from the Kit, including how you will assemble your final course folder.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which barrier in the sample items would your own pupils meet, and how would you find such barriers in your own tests?
- Which practices in the grey zone are you tempted to use, and how will you check that learning is not narrowing?
- What did the outcome-based report give a parent that '14 out of 20' did not?
Reflect
- Which pupils in my class might find my tests harder for reasons that have nothing to do with the subject?
- How do I describe results to parents now, and what could I change?
- What is the one tool from this course I will use again next month?
Try it tomorrow
- Ask a colleague to run the bias checklist on your next test and mark each item OK, REVISE or QUESTION.
- Replace one pupil-ranking line in a report with one outcome statement and a next step.
- After a preparation period, give one item on the same idea in a new form and see whether pupils can still solve it.
Further reading and links
Review items for bias, prepare pupils by teaching the whole curriculum, and report by outcome without ranking or over-claiming.