The AI Death Zone: a cautionary tale
Vibe-coding off a cliff edge
This week’s Substack is a bit of an “inside baseball” report on how we are - and are not - using AI tools to develop our website. It’s by our CEO, Chris, with reference to our CTO, Brian. The same AI promises and pitfalls we’ve found when marking writing are also present when developing websites. For more on our AI marking platform sign up to our next webinar on Mon 27 April.
Brian and I have been coding together for over 20 years now. I met him at the CEM Centre at Durham University in the early 2000s, and he taught me how to code. In 2013, we founded No More Marking together and have scaled it to process millions of writing scripts every year. We’ve been through every technological fad there is.
We started with trying to deliver Python code on the web, Django, which taught us that the web and Python were not easy bed follows. I lost weekends configuring linux boxes with endless scrolling text that would end in baffling errors. Thank goodness for stackoverflow. We then moved to Meteor, which offered real-time updating of information for users that seemed magical but didn’t scale. Finally, we moved on to a proper serverless stack when serverless was just becoming a thing. At every stage, we’ve been ready to take on the latest innovation to learn how to deliver using the best of new technologies.
We’re certainly far from the stereotype of the programmers who learned to code at university using PHP and never really wanted to move beyond it. We’ve been through Ruby on Rails, Scala and fulfilled our Bayesian dreams with OpenBugs. Recently, however, we’ve been faced with a new challenge: the AI coding agent.
For the first time in over 20 years, Brian and I no longer see eye to eye. Brian has fallen prey to AI addiction in a way I fear is irredeemable. It started fairly innocently; I noticed Brian was making function calls that simply didn’t exist in the libraries he was using. When I asked him if he’d actually read the documentation, he would say we no longer need to—the AI would do it for us. He was at a loss to explain why the AI was inventing methods and properties that the library simply didn’t support.
That’s where it started. It has since got a lot worse.
The Mirage of “Vibe Coding”
Brian has now developed what I would call a severe case of AI dependency: he requires a regular hit, a new model, a new release from Anthropic or OpenAI to keep him going. A small does used to be enough, but he’s now taking stronger and stronger hits, always craving the latest designer drug.
In the early days of No More Marking Brian’s nickname was imaginary Brian. As I was out on the road talking to users, I would always refer to “Brian” who would sort things out back at the ranch. As no one had ever seen Brian, there was a rumour he didn’t actually exist. Now I fear that rumour has become reality, and I wish we could get the real Brian back.
Most recently, he convinced me that we could “vibe code” an entire application. I read up on it; I read all the blogs from the Deep Mind crew and suspended disbelief while I read about spec driven AI development. As a huge fan of Test Driven Development, where you write a test first and then write the code, this seemed like something I could get on board with. We fired up the planner from GitHub and spent hours crafting a specification for a fairly simple component of the app. We did everything we thought was expected for vibe coding success. After 10 minutes of watching GitHub whirring away, producing the most insane set of documents, specifications, features, plans and architectures I’ve ever seen, we stopped it.
But that wasn’t the end. We had to switch tools—there’s always a better tool, a different approach, a better model. We switched platform. We spent 70% of our budgeted time planning, making sure the architecture we were dictating was sensible. It was at a level where a team of human coders could have created the app feature for us. I wouldn’t let Brian press the “start coding” button until I was sure we had everything in place. As the planning stage got longer, his finger got more and more twitchy. Eventually, I let him press the button for the AI to start coding.
The Syntax vs. The System
He assured me we could go and have lunch, and when we came back, the code would be ready. It sounded fabulous. The promise was that we had done the hard work—the human thinking—and we’d leave the grunt work—the actual writing of the syntax—to the AI. Surely nobody wants to be writing syntax when they could just be thinking.
Two hours later, a rather crestfallen Brian calls me. “I don’t think we got the specification quite right,” he says. I asked to see what it had done. He says, “Well, we’ve got issues with things like data typing and interface issues, but let me just see what happens if I click this button here.”
I asked, “Have you read the code? Do you know what that button is going to do? If you attach a PDF and click that button, do you know where that PDF is going to go?”
Of course he hadn’t read the code, but thankfully when he clicked the button, nothing at all happened. At this point I knew we were about to enter the Death Zone. The Death Zone for mountaineers is where, starved of oxygen, progress slows to a slow-motion crawl. Programmers are not starved of oxygen, they are starved of understanding. They are looking at code they haven’t written, assumptions they never would have made. We only seem to hear these days from those who have summited, but I suspect the AI death zone is piled high with bodies, all with the words “just one more prompt” frozen on their lips.
The Art of the Language
Brian and I are experienced coders. We’ve delivered web apps at scale with millions of concurrent hits and performed sophisticated statistical analysis. I, for one, have been published in the Journal of Statistical Software—a career highlight. No doubt some will think we just used the wrong model or the wrong tool. Maybe. But while Brian gets a hit of adrenaline with every new model, I have that familiar sinking feeling.
From a personal point of view, I would be very happy if vibe coding turned out to be a mirage. We and others have written about how writing is thinking1. Surely coding is thinking? For the last 10 years, I’ve worked with the R language and seen how it has developed. That evolutionary process has been vital to it becoming the world’s most popular statistical language.
The ecosystem in R called the tidyverse2 is a work of great beauty. Those of us who learned R before the tidyverse remember just how difficult some things were to achieve in an elegant fashion. The tidyverse evolved on top of the base R language through the hard work and dedication of creative, and unpaid individuals and opened up new possibilities. The tidyverse became a vibrant ecosystem which makes data science accessible and fun. No one got rich from creating the tidyverse, but the world got to be a safer, more creative and more beautiful place.
I simply cannot understand how a Large Language Model that is trained to reproduce patterns could ever produce genuine evolutions in the coding languages we use. The tidyverse was carved out by people who understood the base language so deeply they knew exactly how to improve it. They didn't just "vibe" with the syntax; they mastered it and improved it. The manifesto quotes Hal Abelson, “Programs must be written for people to read, and only incidentally for machines to execute.” Are we entering a world now where only machines read programs?
It’s probably time to get back to Brian now and see if I can rescue him from the death zone. He’s soon off for a trip to Shanghai where he’s looking forward to being driven around in a driverless car. When he ends up in the Yangtze he’ll still be wondering if he got the prompt wrong.
https://tidyverse.tidyverse.org/articles/manifesto.html
Do you think there’s any place for LLMs in coding? I agree that it’s not useful for me personally for building whole features or applications - that involves giving up too much control that I want to keep to have the code look and work exactly how I want it to.
But there are definitely places where LLMs save me a massive amount of time - line autocomplete, creating sample data for testing, adding missing unit test cases, etc.
I'm on the fence with this; it can be really good, but it can also be trouble. It's great for prototyping new ideas from scratch, but it gets difficult when you get more code; it tends to make a mess of everything.
So you have to:
a) know what you want - recognise the right solution when you see it (which requires still understanding the domain)
b) have good processes in place, such as integration testing (which can also help to write to some degree)
For example, with programming, if you're using Python, it can really pay off to use a type-checked subset of the language and have a significant amount of testing in place. I think a lot of people are waking up to this and realising that they were lax on this to begin with - we've been spending quite a bit of time on this.
It's also really addictive and becomes a crutch; if it fails to converge on a solution, then it's tempting just to keep hoping that it will fix everything for you... kind of like if you're gambling and you just keep going thinking that eventually your luck will change. It somewhat discourages you from understanding the details, which can impair you. Mind you coders do this all the time even without AI coding, there's some problem you can either a) knuckle down and understand deeply what's wrong or b) randomly prod things and hope you get lucky!
On the flip side, you can get a lot of productivity out of it - but you really have to tidy up the mess, and I don't believe vibe-coding is effective in the long run (yet!?). I've heard a lot of people talk about it as a productivity multiplier, for people who are very capable it can reduce the burden of writing a lot of boilerplate so they can have time to focus on details which matter - but for those with less discipline it can lead to a giant mess.
.. and I fully agree with your sentiment on "The Art of Language" - I haven't seen any of these systems produce code which I would describe as elegant, you can get there but it currently needs a person to drive it, to setup all the abstractions and figure out the details - and even then you'll see that these LLMs always like to take the easy way out, do things the nasty way - cheat the tests etc. etc.
I also dislike this "chasing the new thing", so I'm not up there with the cool kids using the tools from 2 weeks ago.
It's hard to know if it actually makes someone more effective or not. I use it quite a lot, but I'm quite pedantic, so that I won't accept junk code - but even then it slips past you now and then! My workmate doesn't use it at all, and he's quite effective. And we have had juniors use it for things in not-so-good ways, where we've become quite exasperated and thrown it away and started again.