Do you think there’s any place for LLMs in coding? I agree that it’s not useful for me personally for building whole features or applications - that involves giving up too much control that I want to keep to have the code look and work exactly how I want it to.
But there are definitely places where LLMs save me a massive amount of time - line autocomplete, creating sample data for testing, adding missing unit test cases, etc.
I feel there's definitely a place for LLMs in coding although I've certainly found that asking it to do small chunks at a time is more productive. Unit testing needs more work as it only seems to build tests for some of the more common, basic tests rather than edge cases but hopefully that will improve.
I have noticed this - it's not good at writing good tests, so it needs to be coaxed into writing tests which actually test anything (but with some help, they can do wonders).
Frequently, doing things like copying the code into the test, then "surprise!" they match.
Then, when doing the actual testing, they prefer to relax the testing criteria rather than fix problems, and also just get caught up running tests repeatedly (leading to a lot of wasted time).
I'm on the fence with this; it can be really good, but it can also be trouble. It's great for prototyping new ideas from scratch, but it gets difficult when you get more code; it tends to make a mess of everything.
So you have to:
a) know what you want - recognise the right solution when you see it (which requires still understanding the domain)
b) have good processes in place, such as integration testing (which can also help to write to some degree)
For example, with programming, if you're using Python, it can really pay off to use a type-checked subset of the language and have a significant amount of testing in place. I think a lot of people are waking up to this and realising that they were lax on this to begin with - we've been spending quite a bit of time on this.
It's also really addictive and becomes a crutch; if it fails to converge on a solution, then it's tempting just to keep hoping that it will fix everything for you... kind of like if you're gambling and you just keep going thinking that eventually your luck will change. It somewhat discourages you from understanding the details, which can impair you. Mind you coders do this all the time even without AI coding, there's some problem you can either a) knuckle down and understand deeply what's wrong or b) randomly prod things and hope you get lucky!
On the flip side, you can get a lot of productivity out of it - but you really have to tidy up the mess, and I don't believe vibe-coding is effective in the long run (yet!?). I've heard a lot of people talk about it as a productivity multiplier, for people who are very capable it can reduce the burden of writing a lot of boilerplate so they can have time to focus on details which matter - but for those with less discipline it can lead to a giant mess.
.. and I fully agree with your sentiment on "The Art of Language" - I haven't seen any of these systems produce code which I would describe as elegant, you can get there but it currently needs a person to drive it, to setup all the abstractions and figure out the details - and even then you'll see that these LLMs always like to take the easy way out, do things the nasty way - cheat the tests etc. etc.
I also dislike this "chasing the new thing", so I'm not up there with the cool kids using the tools from 2 weeks ago.
It's hard to know if it actually makes someone more effective or not. I use it quite a lot, but I'm quite pedantic, so that I won't accept junk code - but even then it slips past you now and then! My workmate doesn't use it at all, and he's quite effective. And we have had juniors use it for things in not-so-good ways, where we've become quite exasperated and thrown it away and started again.
I feel we're still finding out how to use these new tools effectively. Breaking down problems certainly helps but there's still a need for a human in the loop.
two comments: I reckon experienced developer tell whatever AI they use 'Too Much' and the AI tries and fails to do it. I think great results can be got by not telling the AI too much and letting it decide how the coding works. the techstack to use and so on. That's how, using Replit, I created Timbles.com and I am not a coder, although I also think you have to have some coding experience and ability to understand databases for example. I have no idea whether the code is elegant if not, but it does what I want.
The other point is that having deep experience of what it is that you are trying to code is a prerequisite. For example with Timbles, it builds on having a paper version for over 30 years. As then you know when things are not quite right, or need amending. However, I think there IS a lot of hype from the 'techbros'. Here’s my definition of the verb to ‘vibe’! https://youtu.be/ZmBdpjwwTdo?si=Uf81gfgPUmZv5OO6
AlphaGo used self-play, which is a pretty nice method for self-improvement, which works because there's a universal way to evaluate a score (by the rules of Go).
Unfortunately, for general knowledge tasks, there isn't such an obvious way to do this (though maybe there is, for programming?). Current LLMs aren't trained like this - it is definitely being worked on, though.
Amusingly, AlphaGo was also fragile, and people found ways to exploit it by getting it "out of domain" - by playing silly moves which confused it - I don't think that takes away from the achievement though.
Do you think there’s any place for LLMs in coding? I agree that it’s not useful for me personally for building whole features or applications - that involves giving up too much control that I want to keep to have the code look and work exactly how I want it to.
But there are definitely places where LLMs save me a massive amount of time - line autocomplete, creating sample data for testing, adding missing unit test cases, etc.
I feel there's definitely a place for LLMs in coding although I've certainly found that asking it to do small chunks at a time is more productive. Unit testing needs more work as it only seems to build tests for some of the more common, basic tests rather than edge cases but hopefully that will improve.
I have noticed this - it's not good at writing good tests, so it needs to be coaxed into writing tests which actually test anything (but with some help, they can do wonders).
Frequently, doing things like copying the code into the test, then "surprise!" they match.
Then, when doing the actual testing, they prefer to relax the testing criteria rather than fix problems, and also just get caught up running tests repeatedly (leading to a lot of wasted time).
I'm on the fence with this; it can be really good, but it can also be trouble. It's great for prototyping new ideas from scratch, but it gets difficult when you get more code; it tends to make a mess of everything.
So you have to:
a) know what you want - recognise the right solution when you see it (which requires still understanding the domain)
b) have good processes in place, such as integration testing (which can also help to write to some degree)
For example, with programming, if you're using Python, it can really pay off to use a type-checked subset of the language and have a significant amount of testing in place. I think a lot of people are waking up to this and realising that they were lax on this to begin with - we've been spending quite a bit of time on this.
It's also really addictive and becomes a crutch; if it fails to converge on a solution, then it's tempting just to keep hoping that it will fix everything for you... kind of like if you're gambling and you just keep going thinking that eventually your luck will change. It somewhat discourages you from understanding the details, which can impair you. Mind you coders do this all the time even without AI coding, there's some problem you can either a) knuckle down and understand deeply what's wrong or b) randomly prod things and hope you get lucky!
On the flip side, you can get a lot of productivity out of it - but you really have to tidy up the mess, and I don't believe vibe-coding is effective in the long run (yet!?). I've heard a lot of people talk about it as a productivity multiplier, for people who are very capable it can reduce the burden of writing a lot of boilerplate so they can have time to focus on details which matter - but for those with less discipline it can lead to a giant mess.
.. and I fully agree with your sentiment on "The Art of Language" - I haven't seen any of these systems produce code which I would describe as elegant, you can get there but it currently needs a person to drive it, to setup all the abstractions and figure out the details - and even then you'll see that these LLMs always like to take the easy way out, do things the nasty way - cheat the tests etc. etc.
I also dislike this "chasing the new thing", so I'm not up there with the cool kids using the tools from 2 weeks ago.
It's hard to know if it actually makes someone more effective or not. I use it quite a lot, but I'm quite pedantic, so that I won't accept junk code - but even then it slips past you now and then! My workmate doesn't use it at all, and he's quite effective. And we have had juniors use it for things in not-so-good ways, where we've become quite exasperated and thrown it away and started again.
I feel we're still finding out how to use these new tools effectively. Breaking down problems certainly helps but there's still a need for a human in the loop.
That’s what breaking it down means. Do part, human adjustments, do next part.
two comments: I reckon experienced developer tell whatever AI they use 'Too Much' and the AI tries and fails to do it. I think great results can be got by not telling the AI too much and letting it decide how the coding works. the techstack to use and so on. That's how, using Replit, I created Timbles.com and I am not a coder, although I also think you have to have some coding experience and ability to understand databases for example. I have no idea whether the code is elegant if not, but it does what I want.
The other point is that having deep experience of what it is that you are trying to code is a prerequisite. For example with Timbles, it builds on having a paper version for over 30 years. As then you know when things are not quite right, or need amending. However, I think there IS a lot of hype from the 'techbros'. Here’s my definition of the verb to ‘vibe’! https://youtu.be/ZmBdpjwwTdo?si=Uf81gfgPUmZv5OO6
Those that suggest ai can’t invent or innovate need to read up AI go.
AI developed plays that changed the ways humans play just like a human innovator does.
AlphaGo used self-play, which is a pretty nice method for self-improvement, which works because there's a universal way to evaluate a score (by the rules of Go).
Unfortunately, for general knowledge tasks, there isn't such an obvious way to do this (though maybe there is, for programming?). Current LLMs aren't trained like this - it is definitely being worked on, though.
Amusingly, AlphaGo was also fragile, and people found ways to exploit it by getting it "out of domain" - by playing silly moves which confused it - I don't think that takes away from the achievement though.
Yes, just like in education rapid feedback and a clear end goal make progress much easier!