Comparative Judgement compared with traditional writing assessments

Data from NAPLAN in Australia

Comparative Judgement (CJ) is an innovative way of assessing writing. Traditional marking involves the use of a rubric; CJ relies on many educators making many holistic judgements about the quality of different pairs of scripts.

Over the last six years, we have used CJ to assess nearly two million pieces of writing from students in England, Wales, the US and Australia.

Over that time, there are two questions in particular that we have encountered again and again.

• One: how do the results of a Comparative Judgement assessment compare to those from a traditional writing assessment? We have done various pieces of research on this in the past. Broadly speaking, the results are similar enough for us to suggest that the different types of assessment reward the same construct. To the extent that the results do disagree, we would say that CJ is often giving the fairer and more valid result, and that rubric-based assessment can end up with distorted results driven by a very rigid tick-box approach.

• Two: is there an issue with Comparative Judgement and handwriting bias? Because individual CJ decisions are quick, people sometimes worry that teachers may be using shortcuts like handwriting instead. Again, our previous investigations have shown that handwriting bias is much more likely to be an issue with traditional marking, where each script is only ever seen once by one teacher. However, there is perhaps a broader issue about whether handwritten or typed submissions are fairer, regardless of assessment method.

Now, we have some fascinating new data from Australia that sheds light on both these questions.

We recently ran an Australian CJ assessment, and we can compare students’ results on this task with their results on the national Australian writing assessment, NAPLAN.

This comparison will help us see the relationship between the two different types of writing assessment.

Not only that, but this year, for the first time, NAPLAN introduced typed writing assessments. (UPDATE: Typed submissions have in fact been gradually introduced since 2018). The CJ assessment remains handwritten. So comparing NAPLAN and CJ scores can also shed some light on the importance and signficance of handwritten assessments.

Here’s a graph showing the results of a cohort of 60 Year 5 students. The graph compares their NAPLAN data from 2023 and their averaged CJ data from 2022 and 2021.

Insights

So what can we learn from this graph?

• As we’ve found previously, there is good evidence that CJ and the NAPLAN rubric assessment are measuring the same construct. You can see that the data trends towards the top right, showing that students with higher NAPLAN scores get higher CJ scores.

• As well as the broad correlation, there are no outliers in the bottom right-hand quadrant. This means there are no students getting a high CJ score and a low NAPLAN score.

• So, for most of the cohort CJ and NAPLAN scores are comparable, even though the NAPLAN assessments were typed and CJ handwritten.

• HOWEVER: there are 6 distinct outliers in this data set in the top left-hand corner: students 7, 13, 14, 16, 21 and 34. These students got low scores on the handwritten CJ task, but higher ones on the typed NAPLAN task. We haven’t seen significant outliers like this on previous comparisons between handwritten CJ & handwritten rubric assessments (eg see here and here). So what is going on?

Analysing the outlying scripts

When we look at these scripts in closer detail, it does appear that for some of them, handwriting may have played a part in the score discrepancy.

Here is a sample of a handwritten and typed response from student 16.

Their handwriting is difficult to read, and it does also make it hard to see if they are punctuating sentences correctly. By contrast, their typed response is much easier to read, and you can tell that they are punctuating sentences correctly. This student struggles with the physical aspect of handwriting, so typing their response may free them up to concentrate on the meaning and mechanics of their writing.

Similar patterns were observed for students 7 & 34, and all 3 students are are funded students accessing tutoring and Tier 3 instruction.

Which is the truer or fairer representation of their standard - typed or handwritten? You can probably argue this either way. On the one hand, being freed from the physical challenges of writing does seem to have allowed these students to express themselves more fluently. On the other hand, you could say that handwriting still does play a role in life and in cognitive development, and students who struggle with handwriting may benefit from more practice and assessment of it.

Overall conclusions

• As with our previous research, we find good evidence to suggest that Comparative Judgement and traditional rubric assessment are broadly rewarding the same constructs.

• We find that some students do better on typed assessments than handwritten ones.

Learn more

Our next Australian information webinar is on the 17th October at 4.30pm AEDT - register here.

Our next UK information webinar is on the 16th November at 4pm - register here.