Switch course ▾
Session 1. What Benchmarks and Tests Really Measure
Objective. Describe in plain terms what a standard, a national-style test and the PISA, TIMSS and PIRLS frameworks ask of pupils, and classify items by content and by thinking level (know, apply, reason).
Colleagues talk about PISA, TIMSS and national tests as if they were one mysterious thing. In your classroom the useful question is simpler: what do these documents actually ask pupils to do, and does my own test ask the same kind of thing at the level of my standard? This lesson helps you read frameworks as design guides, spot the thinking behind an item, and avoid two traps: copying a test's look without its demand, and treating a test as a verdict on your class.
Key ideas
Three reference points
- The standard says what pupils should know and be able to do. A national-style test, for example a DTM-style paper (usually centrally set, with many closed items and a strict time limit; check the current official specification), samples part of it in one format. International studies sample it in other ways.
- PISA (for 15-year-olds) describes reading, mathematics and science literacy: using knowledge to understand and solve real-life problems. TIMSS (grades 4 and 8) describes maths and science content areas plus three thinking skills: knowing, applying, reasoning. PIRLS (grade 4 reading) describes two reading purposes and four comprehension processes.
- You do not have to chase every test. Read each framework once, then take only the ideas that fit your standard and your pupils.
Know, apply, reason
- In this course we sort items into three levels. KNOW: recall a fact, rule or routine. APPLY: use knowledge in a familiar situation. REASON: judge, justify, compare or solve in a new situation. This follows the idea of thinking levels in the frameworks, in simple words.
- The level depends on what the pupil must do, not on how difficult the topic feels. Recalling a hard date is still KNOW; explaining a simple graph can be REASON.
- A good test has all three. Too much KNOW rewards memory only; too much REASON can hide whether pupils know the basics.
Competency-based items in plain terms
- A competency-based item gives a realistic situation (a stimulus: a short text, table, price list or graph) and asks pupils to use knowledge and skills to answer a question about it.
- The story must be needed. Test: delete the story. If the question still makes sense unchanged, the story was only decoration and the item is an ordinary exercise.
- Such items often need pupils to choose which information to use, take several steps, or explain a decision. They take more time, so use a few, well marked.
What results can and cannot tell you
- Large international studies describe groups of pupils and systems, drawn from samples. They are not a report on your class or on one child, and they should not be used to rank colleagues.
- This course makes no claim about any country's results. Use the frameworks as design guides for good items, not as scoreboards.
- The most useful question for you is: do my own tests ask pupils to do what my standard says, at the level it says?
Three thinking levels used in this course
Reading a framework as a teacher
A framework is a recipe, not a result. It says which content areas a test covers, which thinking skills it asks for, how many items of each kind appear and how they are written. TIMSS, for example, names content domains for each grade and three cognitive domains. PIRLS names reading purposes and comprehension processes. PISA describes literacy in reading, mathematics and science as using knowledge in real situations.
You do not need all 200 pages. Read the introduction and the page that lists the categories, then pick two or three ideas that match your standard: perhaps a pupil should explain, not only state; perhaps a text should be read for a purpose. Write them on a card by your desk. Your own test is better when it borrows ideas, not when it copies layout.
From verbs to items
The fastest way to check alignment is by verbs. Take one outcome from your standard, for example 'compare two fractions and justify the choice'. 'Compare' and 'justify' are not recall. An item that asks only 'Which fraction is larger?' tests part of the verb; an item that adds 'Explain how you know' tests all of it. This is the same move frameworks make when they separate knowing from applying and reasoning.
Do this for each outcome: underline the verb, ask what a pupil would write or do to show it, and check whether your test lets them. Then count. If the verbs ask for reasoning but 90 percent of your marks go to recall, the test says one thing while the standard says another, and pupils learn from the test what really counts.
Examples from the classroom
Same topic, different demand
Scene. Dilnoza teaches Grade 8 in Samarkand and sets two questions on percentages. Question 1: 'What is 25 percent of 80?' Question 2: 'A shop offers 25 percent off a coat that costs 80 000 so'm, and another shop offers 20 000 so'm off. Which is better? Explain.' Her pupils get Question 1 right 22 times out of 24, and Question 2 only 9 times.
Insight. The topic is the same but the thinking level is not; a test with only Question 1 would hide what pupils cannot yet do.
The story that was only decoration
Scene. Jasur writes a Grade 7 item: 'Aziz walked to the Fergana bazaar. He bought 3 kg of tomatoes at 6 000 so'm per kg. What is 3 x 6 000?' A colleague deletes the story and the question is unchanged. Jasur rewrites it with two prices on a sign and asks which stall gives more tomatoes for 20 000 so'm, so the sign is needed to answer.
Insight. A competency-based item needs the situation to be necessary; otherwise it is an ordinary exercise with decoration.
Using a framework without chasing a score
Scene. At a school in Bukhara the head of studies asks Malika to prepare Grade 9 for an international study. Malika instead reads the public framework with two colleagues, picks the idea that pupils should explain with evidence, and adds one explanation question to each unit test. She tells the head she is using the framework to improve tests, not to predict results.
Insight. Frameworks help most as design guides for your own assessment, not as scoreboards.
Common pitfalls
- Copying the layout of a national-style test and calling it aligned when the items are all recall.
- Believing that PISA, TIMSS and PIRLS measure the same things, so one practice set serves all.
- Judging the level of an item by how hard the topic feels instead of by what the pupil must do.
- Treating the results of a large study as a verdict on your class or on one child.
Sort the Items, Then Check Your Own Test
- Trios (10 min): open the Activity Kit. Read the one-page 'frameworks in plain terms' and the 8 sample items. Each person reads 2-3 items aloud and says what the pupil must do.
- Trios (15 min): sort the 8 items by thinking level (know, apply, reason) and by format (closed or open). Where you disagree, write the pupil's action in one verb before you decide.
- Whole group (10 min): compare with the answer note in the Kit. Discuss the two borderline items and match items 5, 6 and 7 to the PIRLS processes and items 1, 2 and 4 to the TIMSS thinking skills.
- Pairs (10 min): take your own last test (names removed). Count its items by level in the Kit table. Which level is missing? Does it match the verbs of your standard?
- Second short activity, pairs (10 min): rewrite one KNOW item from your test as an APPLY item with a local situation, and one as a REASON item that asks for a decision and a reason. Swap with another pair and apply the 'delete the story' test.
- Alone (10 min): write your school-based task from the Kit and agree with a colleague when you will do it.
Your facilitator will guide you through this activity in the session.
Questions to discuss
- Which item was hardest to classify, and what did the pupil's action tell you?
- Which level was missing or rare in your own test, and why do you think it happened?
- Which rewritten item still looks like decoration, and how could the situation become necessary for the answer?
Reflect
- Which of my usual test items would a pupil answer correctly without understanding?
- Which verb in my standard do I almost never test?
- What myth about national or international tests have I heard in my staff room, and what would the framework say?
Try it tomorrow
- Choose one outcome from your standard, underline its verb and write the item that would show it.
- Count the items in your last test by know, apply and reason and note the missing level.
- Rewrite one item so that the situation is necessary to answer it, and ask a colleague to try the 'delete the story' test.
Further reading and links
Read the standard and the frameworks in plain terms, sort items by what the pupil must do, and build tests with all three levels.