Using a multiple-choice assessment to examine misconceptions in maths

In a series of posts, we will detail the start of year results for the Axiom Maths/No More Marking Mathematics Research Project being carried out this year. As part of the project, partnership schools working with Axiom Maths and schools working with No More Marking (NMM) were invited to participate in a year-long project. The project entailed assessing Year 7 students in mathematics in these schools at the start and at the end of the school year. At each point, students were/will be assessed in two different ways: (1) a multiple-choice assessment using NMM’s Automark system; and (2) a comparative judgement assessment again using NMM’s platform. The aim of the project is to look at the progress made by the students over Year 7 using what we might think of as a more traditional maths assessment, and comparing this to a comparative judgement assessment of mathematical reasoning skills.

In this post, we examine the initial results from the multiple-choice assessment, in particular looking at the analysis of distractor items for particular questions, and what insight this analysis can provide about student misconceptions.

---

As part of the project, a multiple-choice assessment was designed jointly by Axiom Maths and NMM and contained 15 items, taken with permission from Craig Barton and his bank of diagnostic mathematics questions. The assessment was designed to cover mathematics topic areas included in the Key Stage 2 Mathematics Programme of Study and in the Key Stage 3 Programme of Study.

In total 1789 students from 17 schools completed the multiple-choice assessment. Following the assessments, the coordinating teacher scanned the answer sheets and uploaded them to NMM’s Automark system, which automatically marked the sheets and calculated the scores for the students.

What was of particular interest was when we analysed the proportion of students selecting each choice in each of the multiple-choice questions to find prominent distractor items indicating misconceptions by the students. Let have a look at some examples.

The easiest item in the multiple-choice assessment was Q2 where 90% of the students answered this item correctly.

Figure 1: Q2 — the easiest item

The distractor analysis for this item gave the following:

Figure 2: Distractor analysis for Q2

The distractor analysis shows the proportion of students (y-axis) at each total score of the assessment (x-axis) choosing each mutliple-choice option. This item is shown as an example because we can see that there were no likely distractors in this case — everyone was likely to choose option A, therefore very few students showed any misconception for this item.

The first item in the assessment that showed interesting distractors was Q4.

Figure 3: Q4

Figure 4: Distractor analysis for Q4

The lowest scoring students were most likely to choose response A, 5300, perhaps interpreting fifty-three hundredths as fifty-three hundred. Students in the middle range of scores were most likely to choose response D, 0.053. They understood the hundredths part of the question but placed the 53 in the incorrect place-value position.

The next item in the assessment that showed interesting distractors was Q11.

Figure 5: Q11

Figure 6: Distractor analysis for Q11

The lower to middle scoring students were most likely to choose response, ¾. We can see the misconception that they simply subtracted the numerators and the denominators without converting the fractions to a common denominator.

The final item we examine is Q14.

Figure 7: Q14

Figure 8: Distractor analysis for Q14

All students except those at the highest-scoring end were most likely to choose response A, 2, and response B, 3. Response A is perhaps easiest to understand because students could have just multiplied the two whole numbers together. Response B is more difficult to understand but students could have understood that it is more than 2 because of the fraction parts needing to be multiplied as well.

In our next post, we will examine the initial results from the comparative judgement assessment.