Switch course ▾
Session 2. Instruments, Validity and Reliability
Objective. Choose and build simple instruments (short test, rubric, interview, observation), check their validity and reliability in plain terms, and pilot them before the study.
A study can be perfectly planned and still mislead if the instruments are poor. A test that partly measures reading, a rubric that two markers read differently, or an interview that puts words in pupils' mouths will each distort the picture quietly. The good news is that simple checks, a pilot and a second marker, catch most of the problems in an afternoon, and they cost nothing but a little colleague time.
Key ideas
Match the instrument to the question
- A short TEST shows what pupils can do. A RUBRIC judges the quality of longer work such as a paragraph or a report. An INTERVIEW shows how pupils think and why. OBSERVATION shows what happens in lessons.
- Use two kinds on one question, for example a test for the outcome and an interview for the reasons. When both point the same way, you can trust the finding more.
- More is not better. Two or three instruments you can use well in a real term beat six you cannot finish.
A good short test
- Eight to twelve items, 15 minutes at most, each item tied to one thing the pupils were taught, with a written answer key and one clear right answer.
- Before and after, do not give the very same paper if pupils can remember it. Use two versions with different numbers but the same skills at the same difficulty.
- Avoid a test so easy that everyone scores full marks, or so hard that everyone scores nearly nothing: then nobody can show change.
Rubrics, interviews, observation
- RUBRIC: three or four levels, each described by what you can SEE in the work (for example 'gives a reason and an example'), not by words like 'good' or 'weak'.
- INTERVIEW: four or five open questions, ten minutes, five to eight pupils who differ (strong, middle, quiet). Ask 'What was different?' not 'Didn't it help you?' Write short notes at once.
- OBSERVATION: choose one thing to count (for example pupils who answer aloud in a 5-minute window), use a tally sheet, and observe at the same point in the lesson each time. A colleague observing is better than you.
Validity and reliability in plain words
- VALIDITY asks: does this instrument measure what I really mean to measure? A fractions test full of long sentences also measures reading. Check each item against what was taught and what you want to know.
- RELIABILITY asks: would I get a similar result if I repeated it or someone else marked it? Test it simply: a colleague marks 10 papers with your key or rubric and you count how often you agree.
- Pilot before the real study: try the instrument on a few pupils outside the study. Note unclear items, the time needed and every disagreement between markers, then fix them. Only then start the baseline.
Which instrument answers which question
Short test
what pupils can do
- 8-12 items, 15 minutes
- Key with one right answer
- Two parallel forms
Rubric
quality of longer work
- 3-4 levels
- Describe what you can see
- Two markers on 10 papers
Interview / observation
how and why
- 4-5 open questions
- Tally one behaviour
- Pilot first
Items, wording and the hidden skill
Every test item asks for more than one skill. A word problem asks for fractions, but also for reading, for patience with a long text and for knowing the context. If the aim is to find out about fractions, strip the language to what is needed: short sentences, familiar situations, numbers pupils can handle. If the aim is to see whether pupils can apply fractions in a real situation, the reading belongs to what you measure, and you should say so.
Read each item and ask: what would a pupil have to know or do to get this right, and is that exactly what I taught and want to learn about? Remove anything that fails: content not yet taught, trick wording, two correct answers. Then ask a colleague to answer the test as a pupil and tell you where they hesitated.
Making marking steady
Reliability is mainly about marking. Short answer items with a key are steady if the key lists acceptable answers. Longer work is steadier with a rubric whose levels describe what you can see, and with two or three anchor papers that show the levels in practice. Before marking the real set, two markers mark the same ten papers independently and compare.
Count two things: how often you agree exactly, and how often you are within one level. If you are within one level in most cases but exact in few, the level descriptions are too fuzzy; discuss the papers where you differ and rewrite the descriptions with something observable. Do this before the baseline, because changing the rubric after the study starts breaks the comparison.
Examples from the classroom
Cotton in the word problem
Scene. Jasur in Bukhara writes a Grade 6 fractions test whose problem about a cotton harvest has a long introduction. In the pilot, five pupils who can add fractions in the exercise book get the problem wrong, and a colleague says it was the reading. He shortens the text to two sentences, keeps the context, and the same pupils now succeed.
Insight. Validity: the item must measure fractions, not the length of the text.
Two teachers, one rubric
Scene. Dilnoza and Malika in Tashkent mark ten Grade 5 paragraphs with a four-level rubric and agree exactly on four of ten, within one level on nine. They discuss paragraph 6, where they differ by two levels, and find that 'uses a reason' is unclear. They add 'gives a because-sentence' and mark a new set of ten: seven agree exactly.
Insight. Reliability improves when descriptions say what to look for; test it again after you fix it.
A leading question
Scene. Aziz in Fergana plans to ask pupils, 'Didn't the starter help you remember?' In a pilot interview with a former class, all five say yes without detail. His mentor suggests 'What, if anything, was different at the start of lessons?' The answers now vary, and one pupil says the starter felt rushed.
Insight. Open questions invite real answers; leading questions only return the researcher's own view.
Common pitfalls
- Giving the same paper before and after so pupils simply remember the answers.
- Making a test so easy or so hard that nobody can show change.
- Changing the rubric wording after the baseline has been marked.
- Asking leading interview questions and then counting the agreement as evidence.
Build and Pilot an Instrument Set
- Open the Activity Kit: a SAMPLE question, a draft Grade 6 fractions test with hidden flaws, two markers' marks for 10 paragraphs and four interview questions. (5 min)
- Pairs (15 min): fill the matching table: for each of three research questions, choose two instruments and say what each will show. Then do the same for your OWN question.
- Pairs (15 min): critique the draft test item by item. Find at least four problems of validity or clarity (reading load, content not taught, two right answers, too easy, cannot be marked steadily) and rewrite two items.
- Trios (15 min): reliability check. Count how often the two markers agree exactly and within one level on the 10 paragraphs. Find where they differ most and rewrite two level descriptions so that a third person could mark them the same way.
- Second short activity, pairs (10 min): rewrite the four leading interview questions as open questions and decide which five to eight pupils you would invite and why.
- Alone (5 min): list your instruments, who will pilot them and when; write your school-based task from the Kit.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which flaw in the draft test would have fooled you in your own test?
- What did the agreement count tell you, and what did you change in the rubric?
- Which instrument in your study is the weakest and how will you strengthen it before week 1?
Reflect
- Have you ever drawn a conclusion from a test that measured something else than you thought?
- Who could be your second marker, and what would you give them?
- How many instruments can you really handle in a normal term?
Try it tomorrow
- Give one draft item to a colleague as if they were a pupil and note where they hesitate.
- Mark five papers twice, a day apart, and count how often your two marks match.
- Rewrite two interview questions so that they cannot be answered with yes or no.
Further reading and links
Two well-chosen instruments, checked for what they measure and for steady marking, piloted before week 1.