This academic year we launched a research project into maths assessment with No More Marking. Part of this project consisted of two relatively open-ended maths problems designed to expose pupils’ mathematical reasoning. These were marked using comparative judgement.
How comparative judgement works
In comparative judgement, teachers don’t award marks according to a markscheme. Instead they compare two pieces of work and decide which one is better. By combining many judgements by many judges these pieces of work can placed in a rank order and fitted to a distribution in order to give a scaled score.
Comparative judgement is appealing because it looks at whether a piece of work is good rather than whether it matches a markscheme. No markscheme is perfect, so good pieces of work that don’t quite match the criteria can be under-graded. Similarly bad pieces of work that happen to match the criteria can be over-graded.
What comparative judgement gives you is an aggregation of what your pool of judges think about which pieces of work are better than others. Their judgements aren’t based on criteria or rules. They’re based on those judges’ expertise of the area in which they’re making judgements.
How we applied it in this research
We gave around 1700 Year 7 pupils this question:
Ari has a pile of cookies and a pile of boxes.
When he puts seven cookies in each box, he has one empty box left over.
When he puts six cookies in each box, he uses all the boxes but has one cookie left over.
How many cookies did Ari have?
Pupil responses were then assessed using comparative judgement by a pool of 142 serving maths teachers. Each piece of work was judged ten times. This reached a reliability of 0.85, which is equivalent to a margin of error of two marks on a forty mark paper. Using rubrics for a similar assessment would typically give a five mark margin of error.
The result is that we have a rank order, with relatively high agreement amongst teachers, of 1700 pieces of mathematical problem-solving. Teachers were given no instruction on how to make judgements other than a note that they are allowed to reward representations such as pictures, and need not only reward more formal “working out”. Beyond this we simply appealed to their teacherly sense of what good mathematical problem-solving is.
Before reading further, you are strongly advised to do the problem that was set for pupils (set out above). You’re about to see some pupil answers, and what we say about them will make much more sense if you have done the problem.
A model solution.
How important is the correct answer?
If you have done the problem correctly then you will know that the answer is 49. There are eight boxes, and when boxes are filled with seven cookies you use exactly seven boxes but have one left over. When filled with six cookies you can pack 48 cookies but have one cookie left over.
Solving this problem requires you to compare the six and seven times tables, looking for a pair of numbers where: (a) the number in the seven times table is one greater than the number in the six times table, and (b) the former of these is one place earlier in its times table than the latter. In (a) the number being one greater represents the extra cookie, and in (b) the number being earlier in its times table represents needing one less box.
Knowing this, which of the following pieces of work would you judge as being a better piece of problem solving?
Piece 1
Piece 2
[Note: Pieces of work were judged as a pair, so these pupils answers to this question were judged as a pair with their answers to another (slightly easier) question. Both of these pupils had the same answer to the other question.]
Piece 1 has got the correct answer, but not told us about their reasoning process.
Piece 2 has got the incorrect answer. They have told us about their reasoning process – and they’ve told us that it was wrong. They looked for the lowest common multiple (if we assume this is what they meant where they say factor), which is not a step towards solving this problem correctly.
So which is a better piece of maths? The correct answer without working, or the wrong answer with misguided working? Read on to find out what the teachers who judged this thought. But first, another question.
How important is the reasoning process vs how it’s communicated?
Let us introduce Piece 3. Remember that solving this problem relies on finding a pair of numbers, one in each times table, where the number in the seven times table is one greater than the number in the six times table.
Piece 3
In Piece 2 they found the lowest common multiple in both times tables. Then they embark on some incorrect reasoning to explain why this solves the problem. Their explanation is well-presented, but it misses
Piece 3, like Piece 2, has got an incorrect answer. But in Piece 3 the pupil has correctly identified that they need a pair of numbers where one is bigger than the other (35 and 36) and that the answer is the one in the seven times table (hence their answer of 35). Their error is to pick a pair where the number in the six times table is greater instead of the number in the seven times table.
Piece 3 is much closer to the reasoning one needs to solve the problem, but it is not at all explained. Piece 2 is further away, but explained. Which is the better piece of maths?
What teachers think good problem-solving looks like
Our sample of maths teachers overwhelmingly thought that Piece 2 was the best piece of maths out of these three. Piece 2 received a scaled score of 91 – right at the top of the distribution. Pieces 1 and 3 received similar middling scores (51 and 56 respectively).
This tells us something fascinating about what we are and aren’t valuing in school mathematics, and about the habits we form as teachers. Do we reward working because of the value of having a record of the procedure followed, or do we reward it because a correct answer is more valuable when accompanied by an argument that supports it?
Teachers consistently scored answers obtained by detailed but invalid reasoning more highly than correct answers without working. Is this because we think that explicit reasoning is a necessary part of good maths, and a correct answer cannot compensate for its absence? Is it that we’re conditioned by exam preparation to value working out disproportionately highly?
Teachers also consistently scored answers with invalid reasoning but more thorough explanations more highly than valid reasoning with less thorough explanations. Is this because we think that good communication is a necessary part of good mathematics? Or is it that we’ve told pupils that “worded questions need worded answers” so many times we’ve internalised it?
It’s possible to argue that these are value judgements stemming from a more holistic view of what good maths is, or that they’re just reflections of what’s important in exam preparation. What do you think?
Let us know in the comments, or by tagging us on Twitter/X. We’re @AxiomMaths.