In her last piece, Daisy considered whether a focus on oracy may benefit the forgotten third,
if it turns out that the data are right and that most students who struggle academically do not turn out to be star oracy and vocational students, what then?
In the summer of 2024 and the academic year 2024-2025 we ran oracy assessments for schools that allow us to bring some data to the debate. In particular we can see whether our data support the established literature that oracy and written composition have a moderate correlation, consider how pupils progressed over the course of a year and see if disadvantaged pupils performed at the same level as less disadvantaged pupils. While the national examination system does have some data on oracy, it is poorly granulated with only 3 grades and 40% of the cohort receiving a Pass. Our advantage is that Comparative Judgement creates a fine grained measurement scale for oracy.
Our limitation, however, is that oracy is hard to assess, so unlike most of our other posts the samples in our study are small, which restricts the forms of analysis we can perform and limits our findings. We ran two trials, one with 301 pupils from 57 schools, and another with two assessments in the school year with 107 matched pupils from 23 schools. Both involved pupils from ages 8 to 14.
Study 1
The first study required schools to choose a small sample of pupils from each year group they wished to assess, and record them answering a prompt to this question: “Are computer games good or bad?” Teachers could prompt when the pupils got stuck, but otherwise they were left to speak freely rather than as part of a debate. All participating schools provided judges for the study, and judged submissions from all schools, not just their own. For study 1 we could match pupils to their writing assessments.
Study 2
Designed on a similar basis to study 1, the pupils answered one of two prompts, longer school holidays or the best pet. If they answered on longer school holidays at the start of they year they answered on the best pet at the end of the year, and vice-versa. For study 2 we couldn’t match pupils to their writing assessments, but we were able to measure progress over the year.
What relationship did we find between oracy and written composition?
From study 1 we matched 151 pupils in years 5 and 6 (aged 9 to 11) to their writing assessments. The correlation was 0.43, higher than the 0.34 reported in the meta-analysis1.
Expressed in Daisy’s language we can say that those in the bottom third for their written composition are most likely to be the bottom third for their oracy assessment.
How much did pupils improve over a year?
With two assessment points in study 2, we were able to measure oracy improvements over the year.
It is encouraging to see oracy improving over a year. What is really striking is how improvement slows with age. Yes, these are small samples, but we see exactly the same in our writing composition data. Progress is strongest in the earliest years and then slows as pupils grow older.
Pupil Premium
In both studies pupils with pupil premium status2 scored lower on average than the rest of the cohort. The gap is about 0.4 of a standard deviation3, broadly comparable with national English assessments at KS2. Among the 118 Year 5 pupils with matched writing results, the PP gap was larger at approximately 0.6 standard deviations in oracy; again, the gap is similar to the writing assessment, where for these pupils it was 0.7 standard deviations4.
What are teachers rewarding when they judge oracy?
We gave teachers considerable freedom to decide what they valued in the spoken responses. Without a rubric specifying separate criteria, they could draw on their own sense of what made one response better than another.
After the assessments we followed up and asked them how they made the judgements and what they valued the most in the spoken presentations.
Most of them mentioned features that are highly rated in written communication – including sentence structure. Here are some of the teachers’ comments on what they noticed about the stronger and weaker responses.
You could argue that teachers are wrong to place so much emphasis on sentence structure when assessing oracy. Perhaps they should be rewarding other features more heavily; independently, however, they chose to reward sentence structure.
The examples below exemplify how the pupils in the middle score range use the conjunctions “Because”, “But” and “So” with varying degrees of success. These three words are the pillars of the Writing Revolution’s transformative interventions.
Transcript with a score of 52
I think computer games are quite good because you can learn education and, like, you can learn maths and stuff. And there’s also, like, screen time so you can’t, like, go on it too long. And your parent could just, like, say it’s the end of your screen time, especially on, like, weekends and school days. And then you can get on with the things you need to do.
And I think, like, computer games are, like, some of them are bad, but, like, they have, like, age restrictions. So it helps so that the younger kids don’t get them. And, like, parents could say, “You’re not old enough for that game.” And then they wouldn’t have to, like, they could stay out of trouble.
And I think computer games are quite good because there’s a lot of things to keep you safe on the internet from, like, seeing the bad side of it.
Transcript with a score of 57
I think computer games are good and bad because they can distract you from doing your schoolwork. And some games can be violent, so sometimes they need an age limit. But they are really fun to play and they pass time if you’re bored. So I think they’re good and bad.
[Teacher asks the pupil to say more.]
Yeah, and like, it is true that some people don’t like games. Just like, that’s okay because you don’t have to play games. Because you can, like, socialise with other people, like outside, and play actual physical games instead of just video games.
It would seem that some of the same fundamental rules that apply to writing are valued by teachers in oracy.
Thoughts
Returning to our original proposition, it seems fairly safe to say that many of those who struggle with writing struggle with oracy. Our findings suggest that oracy is not necessarily an area of hidden strength for pupils who struggle with writing: pupils encounter many of the same challenges in spoken and written expression, while teachers value similar features in each. Finally, if we believe that oracy is a priority, then our data suggests it is the younger pupils we should focus on. Oracy, just like writing, gets harder to improve as pupils get older.
The meta-analysis included participants across a broad educational range, including foreign language learners, from kindergarten children to postgraduate students and noted that the relationship got stronger as pupils got older.
Pupil Premium (PP) is additional funding to support disadvantaged pupils in England, including those eligible for benefits-related free school meals currently or within the previous six years, and children in care or previously in care. Department for Education
The pooled difference across all collected samples has confidence intervals that don’t cross 0, but it should be noted that the samples are taken from different year groups at different times of year and assessed against different prompts.
In this case neither result is statistically significant due to the small sample size, but the repeated pattern across samples makes this an interesting observation.
Thanks for this. Love the detail and examples. Super interesting.
Interesting that the teachers aren’t measuring the things specific to oracy - eg confidence, audibility, pace, posture. Nor does response to questioning seem to have been included in the task.