Bringing our feedback philosophy to life with AI
AI assessment reports informed by our vision of good feedback
Over the last few weeks we've written a lot about the best kinds of assessment feedback and how AI can and can’t help.
The ultimate aim of feedback is to help students improve and move closer to their eventual goal, which is why we say feedback should be a thermostat, not a thermometer. But there are some proximate steps along the way. For younger students and novice students, the process of responding to feedback may need to be heavily structured, making the information for teachers just as important as the information that goes directly to students. And whilst the information that does go directly to students needs to be useful, it also has other important functions - it needs to motivate students and make them feel that their work has value.
As well as writing about feedback, we have been putting our ideas into practice by building AI-supported student & teacher reports. In this post we will go through what the new reports look like and how their different features support our philosophy of feedback. There are some screenshots in this post but if you want to see more images, see our help site here.
We have developed four new reports: audio comments, whole-class audio comments, student reports and teacher reports.
Audio comments
Our new audio comment system allows teachers to leave audio comments on student writing while they are judging. We then use AI to transcribe all the audio comments and combine them together to produce a polished written comment for every student.
The audio comment report provides you with all of the written comments, two to a page. Here’s an example.
We've explained before that we don't necessarily think these are the most useful part of the system in terms of creating next steps for students that will improve their work. However, it is important for students to know that their work has been read and appreciated by their teachers. The early feedback is that students really like knowing that several teachers in their school have read their work and commented on it. We’ve also found that the best kind of audio comments are specific ones that pick out aspects of the students' writing.
If a school have chosen not to leave any audio comments then this report will be empty.
Whole-class audio feedback
The whole-class feedback is also based on the teacher audio comments. Our AI system combines together the comments to create three summary reports: one for each class; one for the entire year group; one for the national cohort.
If a school have chosen not to leave any audio comments, or if they haven't left enough, then they will only get the national report, not the class or year group ones.
We think this report has the potential to be more useful than the individual audio comments in helping the students improve, if teachers use the information in it to adjust their teaching and address common weaknesses.
Student Report
Each student gets an individual booklet with four parts.
An image of their writing - so they can check their work.
A paragraph of written feedback from their teachers - this is the same as the paragraph in the audio feedback PDF, just copied over to this report.
Direct AI feedback - this is an automated feedback table that is directly generated by AI with no input from teachers. For our creative tasks at primary, the AI provides feedback on the following three features: describing places, describing people and using vocabulary. We’ve found that the AI does provide some fairly useful and open-ended suggestions on these features. It is less good at providing accurate feedback on technical aspects of writing like sentence structure and punctuation. To support those aspects, we’ve created the next feature.
Multiple choice questions - we’ve provided every student with 5 multiple-choice questions, prefaced by an instructional resource explaining the content of the questions. Our initial plan was for the AI to create these questions based on its analysis of the writing, but it just wasn't accurate enough, and if something is going to direct to students accuracy really matters. Instead, we created 3 sets of 5 questions ourselves. Depending on their score, students are allocated one of the 3.
We recommend printing off these reports and returning them to students. We've deliberately left out score information as we know not all schools will want to return those to students, and we want the focus to be on engaging with the feedback and completing the activities.
We also have a further new feature in development which will allow students to redraft their writing based on all of these feedback and resubmit it for assessment.
Teacher report
This download provides teachers with a report on each student. Each report has 3 parts.
Data summary - the student's score and grade on this assessment and previous ones, together with a graph showing their progress over time. These scores are the results of our Comparative Judgement assessments. Up until now our Comparative Judgement assessments have solely involved human judgements, but we are currently investigating whether AI can provide judgements.
Direct AI feedback - this is an automated feedback table that is directly generated by AI with no input from teachers. It has more features (9 in total) and more detail than the chart that goes to students. The primary version of this report has features that are aligned to the primary Teacher Assessment Framework (TAF), which will help teachers with the admin of writing moderation.
An image of the writing - this is there so you can cross reference the automated feedback against the writing and pick out examples yourself. This is important as the direct AI feedback can be ambiguous and is not totally reliable.
Our hope is that report will help teachers to understand their students better, to plan and replan lessons and schemes of work that address their weaknesses, and to monitor their improvement over time.
Summary
These reports use a combination of artificial and human intelligence. It’s important for us that they are not completely AI-driven - partly because there are still concerns about AI accuracy, and partly because it’s important for teachers to engage with student writing.
We have integrated human intelligence into the process in a way that plays to the relative strengths of humans and AI, and that we think gives maximum value for minimal input. These reports are incredibly detailed and have a high degree of personalisation. They also include nationally standardised and moderated scaled scores. It would take an impossibly long time for a teacher to produce this level of detail without technology. However, at no stage do students have to see a screen - the entire process for the student is paper-based. (We do offer a typed option for our Australian schools where NAPLAN is now digital.)
The reports are now available for most of our national projects, and we will keep developing them based on the feedback we get from teachers - so let us know if you have any suggestions! We also have an intro webinar on Wednesday 2 April at 4pm where we will demonstrate these new features.
Our next two Substack posts will develop the research here. First, we will explore if we can use the above process to create a dynamic assessment of students’ redrafting skills. Second, we will publish data on our AI judging trial, to see if we can speed up judging even further by adding AI judges.
