Designing the perfect assessment system, part 1

What's the point of assessment?

Here in the UK, our government are running a review of England’s curriculum & assessment.

Over the next few weeks, we will publish a few articles about the challenges of designing an assessment system. We hope it will be of interest to readers in England, and the 50% or so of subscribers to this Substack who are not from England. We’ll have quite a few comparisons to international systems along the way.

In this first post, we will go back to first principles and think about the main purposes of assessment.

What is the purpose of assessment?

One of the major challenges of designing a good assessment system is that there are a lot of important and competing purposes of assessment. An assessment that would be perfectly designed for one purpose will not work well for another. Trying to make one assessment fit lots of purposes generally means it ends up fulfilling none of the purposes.

This is the main argument of my 2017 book Making Good Progress. Here’s a brief explanation of what I mean.

Here are four important purposes of assessment.

1. To give schools, universities and employers information about an individual student’s level of attainment and capacity for future study / employment (the summative function)

2. To give teachers, students & parents information about how the student can improve (the formative function)

3. To give the government and school improvement agencies information about which schools are doing well and are beacons of best practice, and which schools are struggling (the accountability function)

4. To give everyone information about whether educational standards are improving (the national standards function)

In England, the 16-year-old school leaving exam, the GCSE, provides a good case study of what happens when you use an assessment that’s been designed for one purpose for many other purposes. GCSEs were essentially designed to fulfil purpose 1 above. Over time, teachers ended up using them to give feedback and shape the curriculum (purpose 2), the government started using them to judge schools (purpose 3), and everyone started to see rising GCSE grades as a sign that national standards were improving (purpose 4). They didn’t do any of these three purposes very well, but the attempt to make them do so compromised the purpose they were designed for. So you ended up with schools creating intervention groups of students just on the D/C borderline, coaching them intensively in exam technique to get them just over the threshold. This led to lots more students getting the C grade, but the C grade stopped meaning what it had meant previously.

The last round of curriculum and assessment reforms fixed some of these problems. The accountability function is now fulfilled by Progress 8, not GCSE attainment percentage. The national standards function is fulfilled by the National Reference Test for English and Maths GCSEs.  The formative function is harder to fix with a central government lever, but there does seem to be a better understanding from Ofsted that using GCSE exams and specs to guide the entire curriculum is a bad idea.

Can we decouple assessment and accountability?

The National Reference Test is one of the most interesting innovations of the last round of English assessment reforms. It’s designed to decouple the measurement of national standards (purpose 4) from the measurement of individual attainment (purpose 1) and school accountability (purpose 3). I think it’s working pretty well, and it came in particularly handy during the pandemic, as it has allowed us to have an idea of what’s happened to standards even as the qualifications systems was hugely disrupted.

There are other examples of useful tests like this – for example, America have NAEP, the National Assessment of Educational Progress, and internationally, the PISA tests measure attainment in a range of countries at age 15.

A feature of all these tests is that they are completely decoupled from school accountability and individual attainment. Schools and individuals often don’t even find out how they have performed, and often the tests involve samples of students, not the entire cohort. So there is no incentive for schools to game them or teach to the test, and that in turn ends up giving us more faith in their findings.

However, you still have to be careful. It’s fine to decouple the measurement of national standards from the accountability system. But what about if you decide to remove accountability too? For an example of this, you can look at KS2 science. The national curriculum tests for science were abolished in 2009. As soon as 2011, surveys showed that schools had stopped teaching science as much and had reallocated the time to English and maths, which were still being tested. A few years later, a sample test was brought in which showed very low proportions of students were reaching the expected standard. International sample tests show a similar story. So the samples have been good at identifying a problem, but can’t provide much information that would help fix it.

So what should change?

I think the current secondary assessment system is not too bad, and as a result of the most recent changes does a decent job of balancing the different purposes.

At primary I think there is more scope for reform. The accountability system still has some of the problems that the secondary system had a decade ago, there are unresolved problems around writing assessment, and an over-reliance on assessment frameworks that are full of vague progression statements.

In the future posts we’ll outline some possible improvements.