Item — HN Reader

> There's plenty of things wrong with these tools: they are often wrong, are slow and the GPT4 ones are really expensive.

So...why should they be included?

I really worry about this "often wrong" part - you only know they are wrong if you already know what you're doing. Otherwise you end up trying to use hallucinated APIs & libraries, or produce code not better than copy & pasting StackOverflow answers (which is what the AI was trained on anyway).

3 comments

If an LLM hallucinates a method name you'll find out the moment you try and run the code.

Code has a built-in form of easy fact checking, which makes it one of the most appropriate applications for LLMs. It's much harder to spot a hallucinated fact in a paragraph of prose than it is to spot a hallucinated API method.

The skills you most need to develop in order to take advantage of LLM assisted programming are code reading, code review, manual and automated testing and being really good at thinking of edge-cases that might not be covered.

It turns out these are important skills for being a great developer already - LLMs just force the issue on them a little more.

Due to the nature of my current work I haven't really used GPT for coding yet, but I keep wondering isn't it easier to write code than to read and truly understand it? So how much development time was really saved, if I still care about off-by-one errors or correct identity checks in hash maps or all those edge cases I probably should care about? Those are all things much harder to spot reading than writing the code.

So I keep wondering if we just save time by introducing more unknown bugs using GPT?

I guess this also has a lot to do with what code is written. I would be much more concerned with a system level C++ library than some JavaScript CRUD.

Your comment is a great example of how intelligent humans hallucinate!

Gpt-4 often handles errors well. The generated code is easy to review if you understand what you asked for (if it generates tests and examples too-which it can). Etc

People have different experiences with any given thing. You might find your TV remote intuitive and easy to us. Your grandparents might find it overwhelmingly difficult. Which is true?

Yeah, sorry for the snark. I was mostly referring to how the comment I replied to said they hadn’t actually used GPT for coding, but had lots of ideas about its limitations etc.

If you read closely I am not talking about the limitations of GPT, but about the limitations of the developer having to fix code of others (GPT essentially being another developer). I guess it depends on the complexity of the problem. A lot of stuff is very easy to review and for me other stuff needs more time to understand than it did to write.

I have absolutely seen Copilot generate valid C# code with a nasty logic bug. You really need to carefully review every line, and at that point I would usually rather just write it myself.

I'm sure there are good use cases, but I just haven't seen anything life-changing in my experiments with various LLM products. The only thing I use day to day is IntelliCode, which saves a bit of typing here and there.

If the LLM hallucinates a Python package, someone else might notice that and sneak in a malicious version of that hallucinated package.

Or if it hallucinates a method name, it might be in a code path that goes untested. How often are people using these tools to also write comprehensive test suites?

Like I said, you have to put the work in if you're going to use these tools effectively. I write a LOT of tests both with and without the help of LLMs. I try hard never to leave any code path from an LLM (or from code I write myself) untested - always manually and usually through automated tests too.

If you don't do that, the chances LLM-generated code will introduce weird bugs are high. But the chances you yourself (or your coworkers) will introduce weird bugs are high as well - QA, code reviews and testing are important no matter what helped write the code.

> If an LLM hallucinates a method name you'll find out the moment you try and run the code.

What about when it confabulates a module name that's been squatted by a malware distributor?

One of the things you need to learn to do when working with LLMs is to avoid blindly installing packages it suggests.

Just like if someone opens a PR against your project on GitHub you should review the dependencies they are trying to bring in as well.

While it is true that code has built-in fact checking, i would expect AI to actually do the fact checking implicitly. They are all logic in the end.

I often see GPT/Gemini propose code solutions that refer to non-existing libraries or methods inside of these libraries. Perhaps the solution is to use specialized AI for coding is more advised.

And dont use the generic ones. However these hallucinations are still facts. The issue is that humans are also prone to hallucinations; this is perhaps the most challenging aspect to solve in AI. Everything the AI says must be vetted/fact-checked.

Back to coding with AI: i noticed AI, as described by the original author, is perhaps best integrated as an assistant. It works well when given enough context and a simple task that could otherwise take hours to complete but is now in seconds. So think of:

- given this function: …

- write test cases that covers the most common inputs.

LLM’s are still long way to go to replace programmers. But at the pace it’s going, it feels scary sometimes.

Over reliance is a major part of the issues. LLMs in other contexts help bottom performers a lot, but top performers get worse because they are lulled into trusting the AI's judgment.

I've worked a good number of hours with Claude Opus and it has never produced non-compiling code (ChatGPT 4 does that for me), but it can create quite subtle bugs, which is missed by the "just make sure it compiles"-type comments in this thread.

This is like not using keyboards because of typos.

If my keyboard had some noticeable % chance of typing the G key when I hit the F key, I’d trash it in a heartbeat.

Yeah, guck that!

It's more like not using semi-autonomuous driving features with the car entirely relying on your expertise to realise and correct when it's making a mistake. The main difference is risk. Your own life at stake versus bugs in some production system.

But why are you pushing errors to production? You know you're allowed to fix the LLM's code output, right?

If a robot could paint your house, but made three small errors, would you refuse to use it? Or would you just fix the three small errors by painting over them?

There's some kind of John Henry complex going on in this AI discussion.

You're working under the assumption that you will be able to find the errors. I personally found reviewing code always way harder than writing it and we already push tons of bugs to production in written+reviewed code.

… If your keyboard is _introducing_ typos, _buy a new keyboard_.

yeah but I can type 300 wpm on this one

So… maybe I just do a different sort of software engineering to most people here, but is “how fast can I physically write code” _really_ the limiting factor in many cases?

it's not, but we're torturing an analogy here.