Can AI solve handwriting bias?
Human assessment with AI transcriptions
One of the most common questions we get asked about essay marking is: are teachers and exam markers biased by bad handwriting?
The short answer is yes, some of them are.
Handwriting bias in traditional human marking
Back in 2018, the English exam regulator Ofqual published a fascinating study looking at the thinking processes of experienced examiners in English Language GCSE. Here are some of the comments the examiners made about the scripts they were marking.
I see football, I see band 4, straightaway.
If you read none of these words, and you see they’ve done two-and-a-half sides, it’s written in paragraphs, the handwriting’s readable. And a candidate who gets 5 out of 8 for question one looks like that often.
— There’s a lot going on there. Doesn’t like to stretch himself too much, this is definitely a boy.
— You’re right it is a boy.
— Scruffy writing, can’t be bothered writing too much
So yes, handwriting bias exists, even amongst experienced examiners!
Handwriting bias in human Comparative Judgement
Comparative Judgement is different from traditional marking, requiring humans to read two pieces of writing and decide which is better. It is a more reliable and quicker form of assessment than traditional marking.
At No More Marking, we set up our Comparative Judgement assessments so that every piece of writing is seen 20 times. This helps mitigate the problem of handwriting bias, and indeed of all errors and biases. By combining together lots of different judgements from different teachers, the errors and biases will cancel each other out. That’s very different from traditional marking, where most scripts are only seen once by one marker.
However, whilst this process might mitigate handwriting bias, we can’t claim that it completely eliminates it. This was brought home to us in our recent trial of AI Comparative Judgement, where we compared the decisions of AI judges and human judges. Whilst they mostly agreed, many of the biggest disagreements were the result of human teachers being biased against bad handwriting. Here is a good example. The teacher was given this decision, and said the script on the left was better. The AI disagreed, and thought the script on the right was better.
The teacher who made the original decision reviewed it and conceded that she had made an error and had been unduly influenced by the poor handwriting of the script on the right.
Probably the only surefire way to completely eliminate handwriting bias would be to eliminate handwriting - and there are all kinds of good educational reasons why you wouldn’t want to do that.
But AI can help us solve this problem, and provides us with two ways in which we can retain handwritten responses and reduce or even eliminate handwriting bias.
AI judging
Our AI Comparative Judgement trial shows that we can use AI as a complement to human judging, and one that is largely immune to handwriting bias, because the AI transcribes the scripts before judging them. Our current model is to run assessments that involve 90% of decisions being made by AI and 10% by humans. This reduces workload dramatically, but still retains a significant human element which allows us to compare the human and AI decisions and easily identify any human or AI errors.
AI transcription
The first step of our AI judging process involves the scripts being transcribed. As we were developing this process, we thought to ourselves: why not let our human judges see the transcriptions too?
That’s our latest innovation. We have built a new feature into our judging interface that lets teachers toggle easily between the original handwritten response and an AI transcription. You can read more about it here and watch a short video below.
This new feature allows us to keep our assessments handwritten - with all the educational benefits that brings - but for teachers to view a transcription when they judge the writing.
We will be trialling this new feature in our upcoming summer AI assessments. Stay tuned for the results!
Love the idea here, but honestly, none of the bad handwriting examples here come close to the sort of bad handwriting I see with dyslexic/dysgraphic students, who struggle not just with handwriting but also with spelling, capitalization, punctuation etc. Brilliant, very superior IQ kids who have fabulous writing content but just can't get past either the associations of neatness with ability or the assumptions that they must be careless or lazy (since they are obviously not unintelligent). Not sure how AI would combat that. Does the AI transcription correct spelling and mechanics to combat that bias as well?
It's nothing new to hear that teachers perceive a link between handwriting ability and/ or neatness and subject knowledge or ability. It's the same perception that drives the relentless focus on what kids look like rather than their learning and outcomes. I think there is a significant percentage of parents and teachers, and those who wield power, who see a correlation between school students adhering to a strict formal school uniform and attainment and achievement. What do teachers new to the profession think about h/w? I would bet that few current ITT courses offer much input on teaching handwriting despite the proven links between correct letter formation and spelling and similar for number. Being able to write neatly and accurately is a useful, transferable skill and still essential for exams. But it has to be viewed in proportion. I can't see that an AI program will alter teachers' day to day bias if they still see ' neat and tidy' as an end in itself. I can remember ' Lovely neat work' being a commonly seen comment on many kids' work and not only back in the day.