Education technology is never neutral

It is both a problem and a solution

• “Any technology can be used well or badly.”

• “Technology is just a tool - what matters is what you do with it.”

• “Kids can use a tablet to study or to play games - the issue isn’t the tablet, it’s what they are doing on the tablet.”

I hear this argument all the time: that when technology gives you a bad outcome, the problem is not the technology but the way teachers or kids are using it.

For example, last week, Matt Yglesias wrote an article called “Ed tech is not the answer or the problem”. Referring to a specific app that has come in for a lot of criticism, he said that it was probably being used well in some effective schools, but poorly in some ineffective ones. The issue was not the app, but how it was being used.

But asking whether ed tech is “good” or “bad” is like asking whether schools should have desks or whether teachers should use erasers. In both cases, they almost certainly should!

But the presence or absence of erasers is not what’s making the difference between effective and ineffective schools. If you had a building full of good teachers who were using a good curriculum and had adequate support from administrators and other stakeholders but for some reason they weren’t allowed to use erasers, they would find that annoying, but I’m sure they’d figure it out.

This is a really popular and persuasive argument, and there is a bit of truth to it, because high-functioning and well-managed organisations can make the best of a bad situation. But ultimately, I think it’s misleading. Truly high-functioning organisations do not deliberately choose tools that create bad situations. They choose the tools that are right for the job. And they do so because they understand that tools are vitally important. They are not neutral and interchangeable widgets, and they are capable of having a profound impact on the way we think and behave.

Tools change our behaviour

Tools make some behaviours more likely and others less so.

Take Yglesias’s own example of an eraser. It’s a very simple tool, but it still changes behaviour. It makes some behaviours less likely, and other behaviours more likely. In a classroom where every pupil has an eraser, the attitudes to error will be different from those in a classroom where no one has an eraser. A teacher could try to create the same culture and norms in each classroom, but the presence or absence of a specific tool will make it easier or harder.

Recently, Adam Kucharski wrote about coding using Mathematica, which only allows one “undo”. When he codes using that app, he is far more careful and cautious than if he had unlimited “undos”. The “undo” tool - basically a digital eraser - shaped the way he thought.

The argument I am making here is an extension of Marshall McLuhan’s “the medium is the message” argument. I think Neil Postman has given the best concrete example of this: if your major medium of communication is smoke signals, then your messages are unlikely to include philosophical tracts. The form of smoke signals precludes certain content and types of thought.

Screens make certain behaviours more likely

Laptops, tablets and phones are far more powerful than an eraser, and have a much more powerful effect. They often replace a textbook or an exercise book, but compared to those paper technologies they make task-switching much more likely.

You could be the best teacher in the world, and be completely committed to getting students to concentrate deeply and read difficult texts. But if you are in a classroom where every pupil accesses the content via a screen, I think you will be less likely to achieve your aims than a weaker teacher in a classroom with no screens at all.

Not only that, but there are big differences between different screen types. They are all optimised for different functions, and make those different functions more likely.

Desktops and laptops have physical keyboards, and are optimised for long-form writing, and not for messaging on the move. Mobile phones are optimised for scrolling, swiping, and short messages. You don’t see people walking down the street texting on their laptop. And people tend not to write novels on their phones. Tablets are different again. I think they are optimised for passive consumption of media, as opposed to creation of it.

The mode effects research: yes, ed tech is a problem

I don’t know enough about the specific app Yglesias refers to in his article. But I do think that regardless of the quality of the app or the content on it, there is a difference between learning on screen and learning on paper.

There is a large research literature on “mode effects” - essentially what happens when you change the medium of an assessment but keep the content the same.

One of the best and most rigorous recent studies analysed the results of more than 3,000 students in Germany, Ireland and Sweden, who had taken the 2015 Programme for International Student Assessment tests in reading, maths and science. The students were randomised into two groups. One group took the test on paper; the other took it on a computer. The paper-based group achieved a full 20 scaled-score points better than the computer-based group. That is the equivalent of about six months of additional schooling - a huge difference.

I spoke to the author of the paper, John Jerrim, about this research for an article I wrote about it for the TES, and he told me that he was really surprised by the magnitude of the effect. If an educational intervention caused that kind of improvement we would be rushing to scale it up!

AI-enhanced Comparative Judgement: can we make ed tech part of the solution?

At No More Marking, this is something we think about constantly. What do our tools and technologies make more likely? What do they make less likely?

We’ve been running Comparative Judgement assessments for nearly a decade, and have put significant effort into creating paper-based assessments that can be assessed digitally. Our system allows you to assess writing in an incredibly technologically sophisticated way - without a pupil ever seeing a screen. We’ve assessed about 3 million pieces of writing using this process.

We have now added AI judges to our assessments, which changes the dynamics yet again. What will AI assessments make more and less likely? Well, AI assessment is faster and easier than human assessment. If you make something quicker and easier, it tends to happen more often. So schools may run more writing assessments.

That could be good. It could mean better validation of interventions, reduced teacher workload, more opportunities for pupils to receive feedback — even, potentially, daily practice in the run-up to big national exams.

But it could also be bad. In younger years, for example, an increase in extended writing assessments may not be desirable. Shorter, different kinds of assessment may be more appropriate. We have some of these already, but maybe we’ll need to beef them up and make them more prominent.

We also provide a wider range of feedback, some of it directly created by AI. What will the effect of this be? We recently wrote about some focus groups we’ve been doing asking students about what kinds of feedback they prefer. We also have a project running right now where we measure how much improvement students make when they redraft their writing in response to AI feedback.

Our aim is to create tools that make good outcomes more likely and bad outcomes less likely. This will not happen by accident!

Am I denying human agency?

The attraction of the “technology is neutral” argument is that it makes us feel like we are in control. As I say, there is a grain of truth to this: there will be a range of ways you can deploy a technology.¹ My argument is that the range is limited. The technology sets the floor and ceiling you operate within. The more powerful the tool, the narrower the range you can operate within. An eraser is a relatively weak tool which still gives the teacher and students a wide range to operate in. A tablet is a much more powerful tool, and its power constrains the behaviour of teachers and students.

Your true agency is not in how you use the tool. By that point, the constraints are already in place. Your true agency involves how you select the tool, and in the input you have on its design. As Winston Churchill (almost) put it: we shape our tools, and thereafter our tools shape us.

Help us shape AI feedback & assessment!

And that is why we are constantly talking to schools about what they want in an assessment system, and reviewing the data to see what impact it’s having. If you would like to be a part of these design efforts, you can!

• My colleague Chris has created a user group for secondary schools in England who want to use our AI system to mark GCSE mocks. If you’d like to learn more about this, contact us.

• I am leading our efforts to optimise the AI feedback in different subjects. If you’d like to try out our system with 30 free credits, you can book a call with me here.

• If you would just like to learn more, sign up for our next intro webinar on Mon 27 April.

1. Matt Yglesias presents some data to show that different schools using the same app can still get different outcomes. Yes, of course that will happen. I would not expect every school with omnipresent tablets to get exactly the same outcomes. I would not expect every school with phone bans to get exactly the same outcomes. Still, it is suggestive that in England, the highest performing schools tend to have very sparse use of screens in the classroom. (“Highest performing” as measured by the very sophisticated Progress 8 measure which measures how much value every secondary school adds across 5 years of education.)