Dynamic writing assessment with AI

How did our pioneer school do?

A couple of weeks ago we wrote about our new AI-enhanced writing assessment project for Year 6, CJ Dynamo. This project is available for free for all primaries, and starts after Easter.

We’ve had one pioneer school, the wonderful Shakespeare Primary School in Leeds, who have trialled this project for us, and in this post we will share details of their results.

Our dynamic assessment of writing involves three stages: a pre-test of writing, tailored instruction leading to a redraft of the writing, and then a post-test of the redrafted writing.

In our pioneer school, 79 year 6 pupils completed their pre-test and post-test writing. The pre-test was judged 100% by humans. The post-test was judged 100% by AI. (This was partly to do with pressures of time - our recommended approach is 90% AI, 10% human). Here are some key insights.

• The correlation between the scores on the two assessments was 0.6. This is reassuring, given the two tests were assessing the writing of the same students.

• Overall, scores rose by 12 scaled score points from 542 to 554 – we expected average scores to improve given the redrafting process, and an increase of 12 points on this assessment is a 0.29 effect size - quite a bit!

• Out of the 79 pupils, 61% saw their scores increase. This was interesting – we might have expected more than 61% to improve. The average improvement was 12 points, but there was a very wide distribution, with some pupils improving and regressing by dozens of points. Maybe redrafting is harder to get right than we think!

• Weaker students improved slightly more: WTS students improved by 15 points on average, EXS by 12, GDS by 9.

Here is a graph of the pre-test vs post-test scores.

What did our pioneer school make of the process?

Here’s some feedback from Rebekah Wilson, deputy head at Shakespeare School,

Although it did feel a bit of a squeeze to complete across two lessons, we really learnt a lot through our redrafting process. We will be using some of our redrafting learnings in future writing curriculum development and teachers could really see how the teacher feedback packs added value beyond just this redrafting session. In fact, I'd go so far to say their minds were blown! Thank you for giving us the opportunity to work with you, and Chloe too!

We’ll follow up shortly with a more detailed and qualitative look at the students’ writing and the extent to which they responded to the feedback. All the students in this school redrafted their work after using the tailored workbooks we provided which featured a mix of teacher, AI and No More Marking feedback. We can see some really interesting examples of students responding to this feedback and getting higher scores, which will in turn give us ideas about how we can improve the feedback further.

Take part

If you would like to take part in this assessment after Easter, you can. Subscribe for free to CJ Dynamo here.

Our recommended approach for this project is to do 10% of the judgements with your human teachers and 90% with AI. That way you can see the extent of agreement between the AI and humans.