Paper and on-screen assessments

Is the medium the message? Is the quality of thinking tied to the means?

Last week, the English exam board AQA announced that they are planning to introduce digital exams over the next few years.

At No More Marking, we have assessed nearly 2 million pieces of writing over the last six years. Our platform supports hand-written paper assessments, and typed on-screen assessments, so we have experience of working with traditional and digital assessment.

What’s better - digital or paper assessment?

Too often, the digital / paper debate is seen as one that is just about efficiency and technological progress. It is easy to caricature the annual exam season, with its crates of papers and postal deliveries, as a very antiquated and backward system. Just as it is much easier to apply for a driving licence or passport online, it seems obvious that we should be able to sit all our exams on-screen too.

But it is not as straightforward as that. When you translate an exam from paper to screen, in a certain sense it is not the same exam any more. There are persistent and significant ‘mode effects’, which mean that students respond differently to the same question depending on the mode it’s presented.

Last year, I spoke to the researcher John Jerrim who has done a lot of research on mode effects in the international PISA tests. He told me that in one of his papers, the extent of the mode effects was one of the biggest and most surprising findings he’s come across.

In this paper, Jerrim and his co-authors analysed the results of over 3,000 students in Germany, Ireland and Sweden who had all taken the 2015 PISA tests in reading, maths and science. The students were randomised into two groups, one who took the test on paper and one on computer. The paper-based group did 20 scaled score points better than the computer-based group - the equivalent of about 6 months of additional schooling.

Why might this effect happen? There is some evidence to suggest that we don’t think as deeply when reading and writing on screen. When we read a text on screen, we scan and scroll far more than when we read it on paper. When we hand-write notes, we abbreviate and summarise far more than when we type, and this helps us to remember more of our notes.I find it interesting to see how many adults still carry a pen and paper and still take paper notes, despite the ubiquity and efficiency of digital devices.

Of course, there may well be potential upsides to digital exams too. We recently did some small-scale analysis of writing assessments in Australia, where the national NAPLAN assessments have recently gone digital. We found that for a small subset of students who struggled with handwriting, the change appears to have helped them. Similarly, some of the research AQA reported showed that more students are comfortable using computers for over an hour than using pen and paper.

But whether there are benefits or drawbacks, the key point to note is that digital exams are different to paper ones. They can never be exactly the same.

The medium is the message

There is a broader philosophical debate about mode effects, made famous by the Marshall McLuhan line that ‘the medium is the message’. Certain media make it easier to express certain thoughts, and harder to express other thoughts. The educationalist Neil Postman applied McLuhan’s idea to education in his 1985 book Amusing Ourselves to Death. Postman was concerned that by moving from a word-centred to an image-centred culture, we would lose the ability to express certain thoughts and ideas.

Postman died in 2003. Since then, we’ve seen how smartphones and social media make it easier to consume and create small gobbets of text and video, and harder to consume and create long-form content.

If we accept this argument, the question is therefore: what types of thinking and content will digital assessments make easier? And what type will they make harder?

So what should we do?

On this issue, I can see that there are good arguments on either side. On-screen assessments do offer genuine improvements in efficiency, and they may also allow for greater accessibility and innovation.

But pen and paper do seem to promote deeper thought and fewer distractions, and in a world where so many spaces have been colonised by screens, I like the idea that schools can be one place that is free from those pressures.

At No More Marking, we provide schools with coded sheets of paper that allow students to take assessments on paper. Teachers can scan in the coded paper to our website and still get instant digital feedback and in-depth analysis. They can also keep a digital record of students’ handwritten responses. That’s our attempt to manage the pros and cons of each format.

This may be an area where there are no solutions, only trade-offs - but at least if we recognise the trade-offs exist we are less likely to make bad decisions.