Assessing Oracy - the next phase

What we have learned from our trial, and what we are planning next

At the end of the last academic year, we ran a trial project to assess oracy using Comparative Judgement. 301 pupils from 92 schools took part. Since then, we have evaluated the structure of the trial and, drawing on the expertise at University of Oxford’s Centre for Educational Assessment, are finding out more about what teachers think good oracy is.

In this post, we'll update you on the first phase of the research, and explain what's happening next.

What did we learn?

Here are three key insights.

• The teacher judgements were consistent. We weren’t sure before we ran this project that teachers would agree on good oracy. In purely statistical terms, they did: the judging had a reliability of 0.79.

• You don’t have to assess every student. Our national writing projects assess every student in a year group. It’s not practical to do that for oracy - the administration and judging would take too long. However, assessing a sample of students still provided schools with useful information, and schools particularly liked listening to responses from students outside their school.

• Oracy is hard to separate from literacy. We surveyed the teacher-judges afterwards and asked what they thought were the strongest and weakest features of the students’ responses. Many of them mentioned features like “speaking in full sentences”, “incomplete sentences”, “sentence structures are poor”, and “trailing off mid-sentence”. Sentence structure is typically taught in lessons on writing, so it’s interesting to see it pop up here as an important feature of oracy. We learn to speak before we learn to write, but it’s also possible that learning to write changes the way we speak. Clare Sealy has written some fascinating blogs about this relationship lately (eg see here), and it’s something we hope to follow up on in future research.

What’s next?

• Assessing Oracy: our new oracy assessment project is now open to any school who would like to participate. You can read more details and sign up to take part here. The project will assess a sample of 5 students at the start and end of the year.

• Progress measures: we will continue to assess a sample of students, but will supplement the attainment information with progress information from the start to the end of the year. We’re using a matrix structure to make sure the progress measures are as robust as possible.

• More research on the features of good oracy: our initial survey of teachers gave us an insight into the oracy features they value. We want to identify the presence of those features in the responses and see if we can identify which are the most important. Also, many of our participating students will have completed writing assessments with us too, which opens up new possibilities for investigating the links between writing and oracy.

We have an information webinar at 4.15pm on Thursday 19th September if you would like to learn more.