Comparative Judgement is especially good at measuring the relative difficulty of different assessments. In the past we’ve used it to measure the relative difficulty of A-level maths over a 50-year period. In this article we present data on the relative difficulty of the Guardian crossword over 100 years! Enjoy! (And if you’re more interested in using Comparative Judgement for school assessment, we’ve just released a new tranche of dates to talk to us about GCSE mock marking).
Greek gods
The Guardian Crossword, now known as the Cryptic Crossword to distinguish it from the more recent Quick Crossword, has always been part of my adult life. Living in Greece in the early 1990s, the Guardian, intermittent and two days old by the time it arrived, was my only window on the wider world. Over languid afternoons sipping frappés my crossword mentor Phil initiated me in its syntax, and armed with a battered Webster dictionary I would study it with the intensity of Michael Ventris deciphering Mycenaean scripts. My Greek gods were Araucaria, Janus and Custos.
Thirty five years later, struggling with an old Araucaria puzzle I had come across in A Clue to our Lives, 80 years of the Guardian Crossword, I couldn’t help wonder if my younger self would have breezed through it, and whether that iconic cultural symbol, the cryptic crossword1, had in fact subtly, over the years, become easier, and our brains lazier. Hence was born a study, the comparative difficulty of the Guardian Crossword from the 1930s to the present day. The study was based on an earlier paper I had co-authored on the difficulty of A-level mathematics over time.
I sampled 5 clues from each of the puzzles in the book, spanning the year 1930 to 2007, omitting those with special instructions. To those I added 5 clues from every year from 2007 to the present day using the Guardian’s online archive. I was careful to sample different days of the week (Monday being traditionally the easiest day) as well as different lengths of clues, and avoided the prize crossword where the solutions are not readily available. I presented the clues with the solution to expert judges2, with the simple instruction to choose: “The harder clue?”
Pear & saffron tarts
An immediate challenge to the study is that the style of the crossword has changed from the 1930s to the present. The very first clue to appear in the first puzzle published by the Guardian was a simple definition: “One of our elder statesman? (7)” (BALFOUR). There is nothing cryptic about this clue. A cryptic clue will usually include some hint as to the wordplay involved in a solution in order to guide the solver towards the answer. For example, a modern cryptic clue may look like “Cyril’s got a plan for some homelands in the country.” If you can’t remember the Cyril in question (Ramaphosa) you should be able to form his name from the letters from A + MAP + HO inside RSA. It would have been possible to exclude non cryptic clues from the study, but that wouldn’t really answer my question on whether the difficulty of the crossword as a whole had changed over the years.
My decision, however, led to some lively discussion amongst judges as to what constitutes difficulty. Here is Ashley Knowles aka Boatman on the issue:
the older clues include examples of plain definitional clues and quotations, which wouldn’t have a place in a modern cryptic puzzle. I decided to rate all such clues as easier than any cryptic clue, and my responses will therefore automatically anchor the bottom end of the regression and guarantee an apparent increase in difficulty over time, even though there’s an argument that cryptic clues are actually easier (for an experienced solver) than plain definitional clues. I don’t see a way around this: it’s like asking whether an Elizabethan pear & saffron tart is more or less tasty than a chicken tikka masala.
Other judges disagreed, marking the simple definitional clues as harder. Remarkably, however, the reliability of the judging was high at 0.82 after 8,310 decisions on 465 clues revealing that generally there was good consensus on the perceived difficulty of the clues. The single definition clues were shown statistically to be the most divisive. There was no consensus on “Many sewers have these (6,3,3) NEEDLE AND THREAD (1954)” for example, which ended up in the middle of the scale.
Lord of the clues
Following statistical modelling of the decisions, the scaled score of each clue reveals how hard the clue is perceived to be by the judges, with the hardest perceived clue receiving a score of 100 and the easiest 0. The first thing I do to validate any scale is check the top and bottom, and I was reassured. Judges perceived the hardest clue of all to be written by a setter who is accused of playing fast and loose with the rules, Paul. Here is the clue, published in 2003:
God, Lord of the Flies’s William Ing? (4) THOR
Remarkably one of the judges3 remembered solving this at the time and was able to give this explanation:
God = Thor
Author minus gold(au) = Thor
William (gold) Ing = Author minus gold .I remember solving this a long time ago .
A fiendish subtraction worthy of top spot. The easiest? A tie. From 1945, “Is older, possibly, in uniform? (7) SOLDIER” and from 2026, “Disorganised leaders of army training session elicit apathy (2,3) AT SEA”. With a subtraction at the top and and an anagram and an acrostic at the bottom I was reassured that the scale was meaningful.
Finished!
Grouping the clues by decade a pattern emerges. The hardest decade was perceived by judges to be the 1930s and the 1980s the easiest. Over time we see a real (statistically significant) but modest fall in difficulty. Bucking the trend, Araucaria, Bunthorne and Paul were all tightening the screws in the 1990s to the 2000s, providing some support for my hypothesis that the puzzle has eased ever so slightly in recent years.
The nature of difficulty
I’m stuck on the top left-hand corner. I just don’t know any Bolivian poets.
Apart from the changing rules of the cryptic crossword, one of the difficulties in judging the difficulty of clues is the extent to which cultural figures or words may be perceived as more obscure now then they were in their time. A look at the hardest clues per decade, however, reveals that the cryptic crossword never really taps into the zeitgeist. Telephonographs had long been decommissioned when they featured in 1933, Clough’s masterpiece Amours de Voyage was written in 1849, 107 years before it was featured in a clue. Maurice Tate was a cricketer from the 1920s, 50 years before he appeared in the 1970 solution ACETATE. PUT OUT MORE FLAGS? Thomas More died in 1535, and Put Out More Flags was published in 1942, 458 and 51 years respectively before this 1993 clue. Rev Obadiah who appeared in a 1984 clue? Trollope. (d. 1882)4. In that respect, nothing has changed. Scooby Doo appeared in a clue last year while its heyday was the 1970s.
| Decade | Percentile | Date | Number | Clue | Solution |
|---|---|---|---|---|---|
| 1930s | 99 | 09 May 1930 | 110 | is what you do to hair. | RAISE |
| 1930s | 99 | 09 May 1930 | 110 | '_ you noblest English' (Harry). | ON |
| 1930s | 99 | 02 January 1933 | 1 | This will always let you know if you've been 'rung up' | TELEPHONOGRAPHS |
| 1940s | 97 | 02 December 1943 | 287 | The staff of Goliath's was like a weaver’s beam | SPEAR |
| 1940s | 96 | 02 December 1943 | 287 | To him we owe the lyre | JUBAL |
| 1940s | 95 | 09 February 1942 | 209 | We must do this to progress of production | ACCELERATE |
| 1950s | 98 | 05 November 1952 | 265 | Unique page, often of much value | FIRST FOLIO |
| 1950s | 96 | 27 January 1956 | 178 | 'The horrible pleasure of pleasing _____ people' (Clough) | INFERIOR |
| 1950s | 94 | 12 April 1954 | 87 | Alias the tiger cat | OCELOT |
| 1960s | 100 | 30 July 1966 | 179 | Play the French horn at the opera, omitting the States | CORIOLANUS |
| 1960s | 97 | 05 June 1967 | 131 | Round the circle in a fairy castle | FOTHERINGAY |
| 1960s | 92 | 17 February 1962 | 42 | Spanish displacement of country in Australia? | ASTURIA |
| 1970s | 98 | 28 December 1970 | 12819 | A salt-expert cricketer too? | ACETATE |
| 1970s | 91 | 28 December 1970 | 12819 | Laity needs core of academic body | LEARNED SOCIETY |
| 1970s | 89 | 16 June 1976 | 14486 | Pinter's conscientious temp? | CARETAKER |
| 1980s | 99 | 15 October 1987 | 17985 | . . . safer with a herd of swine | SOUNDER |
| 1980s | 94 | 29 May 1984 | 16939 | Downward path of the unctuous Rev Obadiah? | SLIPPERY SLOPE |
| 1980s | 91 | 02 April 1982 | 16271 | Groovy journey mister: I must be in business | TRAM RIDE |
| 1990s | 100 | 01 May 1997 | 20952 | Uneven contest? | FELL RACES |
| 1990s | 97 | 16 September 1992 | 19514 | Psychologist gets three points in taking transport from painter | EYSENCK |
| 1990s | 94 | 19 February 1993 | 19645 | Inconvenienced Lord Chancellor grows languid in war novel, I'm told | PUT OUT MORE FLAGS |
| 2000s | 100 | 27 December 2003 | 23023 | God, Lord of the Flies's William Ing? | THOR |
| 2000s | 96 | 20 September 2001 | 22319 | Show, in two senses, to have sickness benefit | AVAIL |
| 2000s | 89 | 05 August 2009 | 24770 | Forward or circulating, drink coffee I make | PERCOLATOR |
| 2010s | 96 | 04 August 2015 | 26641 | Catering for travellers to Burundi, going back: a poor place to live | TOURISTY |
| 2010s | 93 | 01 August 2013 | 26015 | Carol spilled the beans, all alone | SINGLETON |
| 2010s | 86 | 02 August 2018 | 27577 | Longing for home, cardinals ignored listener's complaint | OTALGIA |
| 2020s | 95 | 07 August 2024 | 29455 | Christening baby after rear wiped, bun on it combed back first | BAPTISMAL |
| 2020s | 93 | 05 August 2025 | 29765 | Clue: solvers’ dog | SCOOBY |
| 2020s | 86 | 05 August 2025 | 29765 | Dog fed old cracker | SEXPOT |
What drove the difficulty in the 1930s to the 1950s is not the obscurity of the clues, but their nature. A quote from Henry V, a biblical figure and a quote from a Victorian poem all feature in the hardest clues from that period, none of which would qualify today as “cryptic” as they do not feature any form of wordplay. D.S.Macnutt set out the established principles of clue construction for the modern cryptic in Ximenes on the Art of the Crossword in 1966. Here’s Ashley Knowles aka Boatman again,
A few of the clues were incomplete by modern standards – for example, "A roguish manner?" for ARCHWAY lacks definition, or "Laity needs core of academic body" for LEARNED SOCIETY lacks an anagram indicator. I decided to rate these clues as harder, but I’m not necessarily being consistent here: by the argument that I’ve applied to plain definitional clues, perhaps these types should be seen as easier on the grounds that a solver from an earlier time would not have been put off by the lack of post-Ximenean syntax.
He gets hung once a year, poor fellow
Quotations, general knowledge, single definitions and partial cryptic solutions would be frowned upon today in a cryptic crossword. Looking more closely at this earlier era (roughly 1930 to 1956) our judges perceived completing quotations to be the toughest early type of clue, but these were rare. Far more common, and nearly as stiff are the cryptic definitions, which are still seen, although not nearly so often, today. A cryptic definition is usually a playful misdirection, often signalled by a question mark. A classic example of this would be “He gets hung once a year, poor fellow (11)” ACADEMICIAN which appeared in 1932. The clue refers to the Royal Academy’s summer exhibition in which an artist’s paintings may be hung, while suggesting that hanging is not normally a pleasant fate.
Cryptic definitions
Re-running the analysis with only clues that would be acceptable in a modern cryptic crossword, we see that a puzzle from the 1930s would be challenging by any definition to today’s solvers. The clue “Neater part of the gun (7) TRIGGER” that appeared in 1935 is a perfectly acceptable double definition, but it helps if you’ve recently read Mark Twain’s “everything was trim and trig and bright …” The 1930s were the heyday of the aptly named setter Torquemada, who famously set a quotation from Kim which nobody could solve5. Through the 1940s the clues became simpler and more concise, until in 1949 Afrit (who set for the Listener) begun the modern age with his injunction: “You need not mean what you say, but you must say what you mean.”6 The wordplay became more varied while single definitions and general knowledge were out.
Beyond knowledge
My question when I began this study was whether the Guardian Crossword has grown easier over time, and the answer, statistically speaking, is yes, it has. The answer, however, turns out to be less interesting than the transition in the cluing that is evident in the sources of difficulty. The modern cryptic still requires the cultural memory and knowledge that was required of the earlier puzzles. It would certainly be poorer without Waugh although it is doubtful that Eysenck would get past today’s editor. Knowledge, however, is not in itself enough to solve the modern puzzle: for that you need the lateral thinking to allow you to piece together the hints in the clue. The power of making that connection is so powerful that one solver remembered a clue and its parsing from 23 years ago.
The Guardian Cryptic of the 2020s may be a little easier to solve than that of the 1930s, but it is so much richer, and, in the spirit of the paper itself, perhaps more democratic. Lacking a classical and biblical education, I too would be left feeling like Reginald Perrin if faced with 'To him we owe the lyre (5) JUBAL.'
Acknowledgements
I am hugely indebted to the wonderful community at fifteensquared for giving up their time to judge without recompense, to Ashley Knowles aka Boatman for his thoughtful comments, and to his students for dedicating their time to judging.
The full dataset is available here: https://github.com/cbwheadon/crosswords
For me there is only one cryptic crossword: the Guardian's. I believe there may be others.
Judges were recruited from fifteensquared and via Ashley Knowles aka Boatman.
https://fifteensquared.net/2026/08/22/looking-for-volunteers-for-guardian-cryptic-survey/
Perhaps not as dated as it seems as Alan Rickman played him in a 1982 BBC series.
Sandy Balfour, A Clue to Our Lives: 80 Years of the Guardian Cryptic Crossword (London: Guardian Books, 2008), p. 31. Although he set for the Observer, Balfour notes that his anarchical approach to crosswords pervaded all puzzles of the time.
Sandy Balfour, A Clue to Our Lives: 80 Years of the Guardian Cryptic Crossword (London: Guardian Books, 2008), p. 43.