Switch course ▾
Session 3. Item Analysis: Which Questions Tell Me What?
Objective. Calculate the share of pupils who got each item right, compare strong and weak pupils, and read wrong answers for misconceptions.
When a test goes badly, the first instinct is to look at the pupils. Item analysis asks you to look at the test as well. By counting how many got each question right, comparing your strongest and weakest pupils, and reading the wrong answers, you separate three different situations: a topic you taught well, a topic that needs another try, and a question that was never fair. Fifteen minutes by hand is enough, and it tells you where your next lesson should begin.
Key ideas
Item by item, not only pupil by pupil
- A total mark tells you about a pupil. A column of correct and wrong answers for one item tells you about the teaching and the question.
- Item analysis means looking down each column: how many got it right, who got it right, and what the others wrote.
- You can do it by hand in 15 minutes for a class of 25 and 6 items.
Difficulty: the share who got it right
- Difficulty (also called facility) = pupils correct / pupils who took the item x 100. 15 of 20 correct is 75 percent.
- A rough guide, not a law: above about 90 percent the item is easy and teaching can move on; below about 30 percent the item is hard and needs a closer look.
- A hard item is not automatically a teaching failure. First ask whether it was taught and whether the question is clear.
Strong versus weak: does the item separate?
- Put the papers in order of total score. Count how many of the top 6 and the bottom 6 got the item right.
- A good item is answered right by most strong pupils and fewer weak pupils. If strong pupils also fail, suspect the question or a gap in teaching for everyone.
- If weak pupils get it right and strong pupils do not, check the answer key: a mistake in the key is more common than we like to admit.
Wrong answers are information
- In a multiple-choice item, count how many chose each wrong option. A wrong option chosen by many is a pattern, not random guessing.
- Ask what thinking would lead to that answer. 1/2 + 1/3 = 2/5 comes from adding tops and adding bottoms, a misconception about what fractions are.
- For written work, sort scripts into piles by the kind of error. Three piles with a count in each is enough to plan a re-teach.
Illustrative item difficulty (% correct), sample quiz
Three checks before you blame the pupils
When one item has a very low score, ask three questions in order. Was it taught? Look at your plan and the textbook: a pupil cannot answer what the unit did not cover. Is it clear? Read the wording as a pupil would; long sentences, unfamiliar words or a confusing picture make a question harder than the idea. Is the key right? A wrong answer key is more common than we expect, especially when questions are copied from another source.
Only when all three checks pass should you conclude that the idea itself was not learned. At that point the wrong answers become your best guide, because they tell you what the pupils thought instead.
Wrong answers designed to teach you
A diagnostic multiple-choice question is written so that each wrong option matches one specific misconception. For 1/2 + 1/3 the options 2/5 (adds tops and bottoms), 2/6 (adds tops, multiplies bottoms) and 5/6 (correct) all point somewhere different. If most of the wrong pupils pick the first, you know what to teach, and you know it from the answer, not from a guess.
You can write such questions yourself in ten minutes: list the three mistakes you see most often on this topic, and turn each into an option. Keep a notebook of the mistakes; the same ones come back every year, and a question bank built from them improves each time you use it.
Examples from the classroom
The popular 2/5
Scene. In Grade 6 in Samarkand, 9 of 20 pupils choose 2/5 for 1/2 + 1/3, and only 6 choose 5/6. Dilnoza does not re-explain the algorithm. She draws a half and a third of the same bar, asks 'is 2/5 even more than a half?' and lets pupils see that the answer is too small. Three days later the same item scores much higher.
Insight. A popular wrong answer is a window onto thinking; a picture can challenge the idea directly.
Everyone failed question 6
Scene. Only 3 of 20 pupils answer a long word problem on 'a fraction of a fraction', and two of the three are not among the top pupils. Jasur runs his three checks. Not taught in the unit; wording of 45 words; key correct. He drops the item from the marks, notes 'teach next unit' and does not change his plan for the class.
Insight. A very hard item with weak discrimination points to the question, not to a failure of all pupils.
A key that was wrong
Scene. Nodira, Grade 7 in Tashkent, sees that on Item 8 the weakest pupils did better than the strongest ones. She checks the key and finds she typed B instead of C. With the key corrected, the item behaves normally. Without the strong-versus-weak check she would have written a note about 'confused clever pupils'.
Insight. If strong pupils fail and weak pupils pass, check the answer key first.
Common pitfalls
- Concluding 'they did not learn it' before checking whether it was taught, clear and keyed correctly.
- Looking only at the percentage and ignoring which wrong answer was chosen.
- Treating the 90 and 30 percent marks as strict laws rather than rough guides.
- Using item analysis to rank pupils instead of to improve the teaching.
Item Analysis by Hand
- In pairs (15 min): use the results table in the Activity Kit (20 pupils, 6 items). Count the correct answers in each column and compute the difficulty of every item.
- In pairs (10 min): papers are already in order of total. Count the correct answers of the top 6 and the bottom 6 pupils for each item.
- Alone (10 min): for Item 4, count how many pupils chose each option A to D and write what thinking leads to the popular wrong answer.
- As a group (15 min): choose the item you worry about most and decide what to check first. Look at Item 6 especially.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which item was hardest and which was easiest? Would you teach them differently on this evidence?
- What did the popular wrong answer in Item 4 tell you that the percentage did not?
- Item 6 was hard even for strong pupils. What are three things to check before concluding that the pupils did not learn?
Reflect
- Which question in my last test did almost everyone fail, and did I ever ask why?
- Could any of my questions be unclear, or taught too little?
- What misconception do I meet again and again in my subject?
Try it tomorrow
- After your next short quiz, count correct answers for each item and write the difficulty beside it.
- For the hardest item, run the three checks (taught, clear, key right) and write the result.
- Write one multiple-choice question for your subject with three wrong options, each tied to a mistake you often see.
Further reading and links
Read each item three ways: how many got it right, whether strong pupils did better than weak, and what the wrong answers show.