back
171 comments
> The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shit. It'll have to happen one by one, and nobody else can make it happen for them.

A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would.

Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.

I have a distinct line between when I'm willing to believe an LLM's output and when I'm not: whether I would believe the same thing from an anonymous Internet forum post or a blogger I don't know. Those posts are not unlikely to be misinformed, biased, lies, or otherwise untrustworthy. And yet, I spent plenty of years honing a sense of when they were good enough for certain things.
I have to admit since VSCode seems to be regularly re-enabling the Cocaine Parrot Autocomplete my views on LLMs and coding has softened a little.

I'll temper that slightly by saying it's mostly out of morbid curiosity because the things that the Dreaming Piracy Robot comes up with are frequently wildly incorrect code, but it's interesting to think about how it might have got there.

And then I think, well, maybe Special Needs Wintermute has a point. Maybe there's a different way to think about it that I've missed.

And then I just change it back to what I wanted in the first place.

> LLMs are very convincing and persuasive.

The goal of LLM's, as they are marketed now, is to drive engagement and stickyness of products. A wrong answer is brushed off with a "Hey, you're right, let's try that again" - a response purposely designed to maximise the friendliness of the system and minimise the sting of a wrong answer. The fact that an LLM will not respond to the same question in the same way twice (i.e. the 'temperature' ) is because increased accuracy will not drive engagement and, therefore, increased accuracy cannot be allowed to get in the way of engagement.

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.

Psychics are still in business. Although to be fair they probably don't have as much revenue. I think people really enjoy being told how smart and insightful they are, and how much they've really cut to the crux of the issue. This isn't the whole thing, but I think it counts for a lot.

> Surely if the machine you go to for answers regularly makes things up you would just stop using it?

We live in hope that people stop doing stupid things and are constantly disappointed.

> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.

The last sentence reflects a lot of my feelings on the first question. LLMs have a sort of weaponized take on the ELIZA Effect. The better their memory the better they are at playing to human social desires to be listened to in an active conversation. At some point it stops mattering if the answers are right when the answers feel right, but really, like ELIZA back in the day, so much of what makes it seem special is just reflecting your own writing back at you in a convincing and persuasive way.

> Surely if the machine you go to for answers regularly makes things up you would just stop using it?

People still respond to ads and political speeches.

> Surely if the machine you go to for answers regularly makes things up you would just stop using it?

Steve Yegge likened LLMs to slot machines. The human brain is very vulnerable to random reward systems. If you get an hallucination, just pull the lever once more.

> A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would.

I feel like the most pragmatic perspective is "trust but verify."

This is why they're so effective at coding: you can run the code yourself (or the test suite) to verify that it actually does what it's supposed to.

And maybe these people finding hallucinated results on Rachel's site are doing verification too.

even worse when managers, while sharing their screen, read an assertion from Gemini and take as fact. puts subordinates in a position where theyre responsible for challenging the assertion (if warranted) and, in a way, challenge their manager's decision making

not saying anything new. easy enough to frame it like any other assistant and check references

>Surely if the machine you go to for answers regularly makes things up you would just stop using it?

Not necessarily, namely because P != NP. Verifying the correctness of a solution is faster than solving it. Thus a system that outputs 99% incorrect solutions and 1% correct solutions can still be incredibly useful.

> Perhaps people have to be burned by something really bad personally before they realise the limitations?

I'm reminded of that person who killed themselves due to their discussion with ChatGPT and their parent wrote their obituary using ChatGPT. I don't think it is enough.

"Yes, ..." is how I notice all the models (4-5 that I use) in the last week or so have begun saying "No, ..." except for Kimi. I ask a lot of questions in the form of "Question about the feature: does it allow for ..." and the answer will reliably start with "Yes, the feature exists and does this thing you have labeled as [blank] which can be explained as ..." and later "However, ..." giving the information I requested (no, it doesn't allow for what you need). It's actually kind of uncanny.

I've started testing identical prompts across models, just to clearly see the "Yes, ..." with the "..but.." buried.

Example: "Can you view the salvage car in the lot before it is scheduled for sale?" "Yes, cars are viewable on the lot by members before they go on sale, blah, blah, blah." Then later "However, cars not yet scheduled for sale are not allowed to be viewed at the lot".

Example: "Can I control the keyboard shortcut Chrome has hijacked so that my existing OS keyboard shortcut will work even while Chrome has focus?" "Yes, chrome keyboard shortcuts, blah, blah, blah," and later "No, chrome does not allow ... ".

There is clearly some recent implementation of the idea that starting with "Yes, ..." has some benefit, but I'm having trouble adapting to the feeling of being lied to as a policy.

Nowadays not just big orgs are falling into the delusion, but also big people.

When Linus posted that AIs and vibecoding were here to stay and declared resistance to it as harmful, I stopped to consider whether I was wrong, but it has made me realize that in retrospect Linus Torvalds and Linux itself aren't actually the holy grail of computing. I didn't feel that way with Richard Dawkins, its not like falling for an AI psychosis retroactively made me question The Selfish Gene, but now I'm looking at linux and the theory that it's a clusterfuck is gaining so much traction, especially after copy.fail and ensuing rustification, I see so much clearly now. It was never about linux, UNIX sure, POSIX, yeah, GNU fucking aye, kernel? Ok whatever, drivers and scheduler with a gajillion lines of code I guess.

I bet there will be a very interesting generational divide between the kids that were born before or after about 2010; old enough to have some critical thinking facilities at the dawn of ChatGPT when it was still noticeably dumb.
Making things up is only really a common issue on the non-thinking models which nobody should be using. The regular chatbots are Autogooglers and are very useful for research. This is just not a good argument anymore.

Edit: Guys, why are we downvoting this? Does no one use like ChatGPT or Claude and understand how it works? Do you all think its regularly hallucinating links still? Is everyone on HN using like free signed out accounts or something? What year is it?

> people have to be burned by something really bad personally

That's why e.g. cross-generational learning rarely works. If culture or technology still allows it, you have to repeat the stupidest mistakes of your parent generation to learn the same lessons. "Learning from the mistakes of others" in your own generation equally hardly ever works. So much less if the lesson involves falsehoods you wanted to believe.

Even if LLMs lied 30% of the time, they would still be about as useful as they currently are for me.

When I ask for their input, it's always for a situation where I'm capable of judging if their input is useful or not.

In all situations I use them, it doesn't matter if they're correct at all. I'm asking for ideas, alternatives, links for blogs or articles. I talk things out with them...

I don't think we should ever "trust" LLMs. This seems like the wrong usecase for them.

> I maintain that anything that is sufficiently aware to be able to actually understand things is also going to have enough of a sense of self that you won't just be able to tell it what to do.

I don't see any reason for this to be true. "Actual understanding" (which I take to mean something like a predictive world model) and desire for self-determination coincide in humans because of our evolutionary history, because our reward function involves reproducing in a competitive environment. Artificial systems usually have a very different reward function. IMO the burden is on the claimants to show why these two imminently separable concepts are likely to co-occur again under wildly different pressures.

> because our reward function involves reproducing in a competitive environment

This is an extremely reductive way to look at human existence. So much so that I read this with Richard Dawkins voice in my head.

Our existence is far richer then just the capability to reproduce. We (as well as other animals and even plants) do far more things then multiply, and in fact we often do things which are detrimental towards the prospect of reproduction.

I think it is actually a mistake (philosophically speaking) to try to find a simple reward function for the human existence. I see no reason for such a thing to even exist (let alone be simple enough to summarize in a single sentence).

We have difficulty defining “actual understanding”. Humans get a free pass because we assume humans as a species possess this power but if we were to judge based on output alone maybe you couldn’t tell a human from a sufficiently advanced LLM/AI.

So I’ll be handwavey here and say that if “actual understanding” is to an LLM what an LLM is to a bash script, so there’s a mechanism there that doesn’t just follow hardcoded paths, it takes new data and processes it in novel but human like ways to come to a new conclusion if needed, then the author is right, in my opinion.

We don’t know what the reward function is for an AI but AI is trained on so much human work that it probably starts off with the same biases in its understanding and reactions. It feels like there’s something very basic in becoming more independent the more you understand of the world. Animals go through this as they grow too, not just humans (listening to parental authority until they eventually don’t anymore).

Since there is no established consensus on this one the burden is on either party to prove their own side. Just because the author said something first doesn’t mean they need to write a full proof while you get to say “nuh-uh” and that’s enough.

> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?

I feel like a Google Maps-style system would discover this automatically by noting via phone location data that there is heavy traffic in Chinatown.

(I do get the author’s point, but I think that factually this example would not be a problem)

Unfortunately, I just had this experience with Google Maps last month—I wanted to go to downtown to buy cheese at Pike Place, but I didn't realize that 6th avenue was closed off for the annual Pride Parade. Traffic was horrible, and Google Maps kept telling me to turn right onto closed off streets until I just had to abandon my trip entirely. Definitely some lack of communication between Seattle city planning and the Google maps team, but even the existing Google Maps systems weren't able to react fast enough to give me any warnings and the time estimation was laughably incorrect.
Maps apps see the streets that don’t have cars on them (the same ones closed for a festival) as good routes to send vehicles because there’s currently no vehicles using them. It keeps trying to route people there and doesn’t understand why.

I think it takes someone (at Google) manually marking those roads as unavailable before it will stop trying. I’ve seen it happen with other things too, like if a highway is closed because of a bad accident.

I have never lived in SF but I have failed similar Chinatown-during-lunar-new-year tests in all 3 places I lived for more than a year. I have no doubt that an AI, particularly one that continually gets traffic update from some external source, would do better than I in predicting such hiccups. OTOH, an unconstrained LLM seems very likely to route me down streets that don't exist at least some of the time. If only we had some sort of database of actual streets and routes that were capable of checking the work of an LLM...

I see the same thing with LLMs in software development. If you say "find a bug in this code" it will regularly confabulate bugs. If you ask it for a test-case, run the output through some deterministic thing that tries the test-cases, and tells the LLM it's wrong, the output of that system will mostly be legitimate bugs[1].

For now, transformer-based generative AIs seem at a minimum like a very useful tool for dealing with "squishy" problems when you have some way to validate their output. Many of the 404's to the blog are probably people validating the output of generative AI, which is the opposite of the inference made in TFA.

1: It will also occasionally hack your test-runner; I suppose that's also finding bugs, just not in the software you wanted to find bugs for.

> Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves.

That's the horror buried underneath all the tech, policy, and gloss. The real, animal brain, desire that drives most of this is: I'd like a slave I don't have to feel bad about.

I think thats deeply uncharitable and fundementally superficial.

I think people (many/most) don't want slaves.

Our imaginations just outstrip our abilities and we desire them to match.

I'd like to highlight that even if one added an URL-exists step [0], that doesn't do a dang thing for result-set problems of:

1. False-negatives, where relevant posts that do exist are not being shown (imagined or otherwise) to the user.

2. Posts which exist but don't fit the words the chaos-parrot uses to describe them.

3. "Relevance" being determined by unpredictable factors that aren't stable, predictable, or desirable.

In other words, it's just more whack-a-mole lipstick-on-a-pig third-animal-idiom-here.

[0] A bad idea on its own, since it creates a security vulnerability for data-exfiltration or indirect malicious attacks.

"Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves."

Always comes to mind when I see Elon and friends getting excited about AI robots. Slavery was more about economics than the role-playing.

> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?

The premise seems to be that models aren't smart enough to understand this, and if they were, they'd be sentient and want autonomy.

For an article that's about making things up, and being too trusting, this seems bad. Maybe the author knows a lot about LLMs, but it doesn't seem like it.

Pasting the verbatim quote from the article into a free ChatGPT session: https://chatgpt.com/s/t_6a5e6f5e24f08191b6a482aad63cae63

Going to an incognito window and using a less leading question: https://chatgpt.com/s/t_6a5e6ee4a3508191bc1b352b41911b53.

If I go generic and just ask if there's anywhere I shouldn't drive, it doesn't get to Lunar New Year until I ask about "events" on the third question: https://chatgpt.com/s/t_6a5e6fb239208191b18cebcf7642c8b0. It's sort of a win for the article, if you think that people who run driverless car companies are all dumb, and won't create a prompt to tell their LLM to "consider events that might disrupt traffic."

Could the article "The Stack" the user was looking for have been 'What is "the stack"?' by Julia Evans?

https://web.archive.org/web/20160305142512/https://jvns.ca/b...

> Here's the example I throw out to people who have been in the Bay Area for a year or two. It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?

Well the answer is actually that it's always a bad idea to drive straight through the middle of Chinatown at any time of the year, because the streets are narrow and full of tourists.

There was that one Waymo they set on fire in Chinatown but I was around for CNY[0] and the streets that weren’t explicitly walled off by barriers had drivers going down them too.

As an aside, what’s the problem with the extra traffic? Perhaps she has a lot of traffic but nginx can return a 404 with a tiny amount of CPU.

0: https://wiki.roshangeorge.dev/w/Blog/2024-02-24/Chinese_New_...

I don't say this often... but I feel this post could have been a tweet. Maybe two.
The 404s mean that somebody or something checked if the post at the URL existed, and got a clear answer. Seems like it's good that they checked, at least. You won't see any evidence of the ones who don't check.

Also, I imagine keeping their cars out of Chinese New Year celebrations (and other big events) is something Waymo could figure out how to do if they put their minds to it.

My website also gets LLM visitors to URLs that never existed, and in many cases to topics that I have never covered. This means that they use my name to give authenticity to things that I have never said.
Funny. I have about 187 posts titled 'The Stack' on my blog, but also none on 2013-08-13, so I'm not the training set it seems :P
> Briefly stated, the [Slop] Amnesia effect is as follows. You [ask the slopservant about] some subject you know well. In Murray's case, physics. In mine, show business. You read the [slop] and see the [slopservant] has absolutely no understanding of either the facts or the issues. Often, the [slop] is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. [Slop's] full of them. In any case, you read with exasperation or amusement the multiple errors in a [slop], and then [ask about] national or international affairs, and read as if the rest of the [slop] was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know.

-Michael Crichton [slop mine]

> It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?

In case anyone is wondering: yes obviously even the dumbest current models correctly answer, given this prompt verbatim, that it's because of lunar new year.

>> Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves. I mean, it's not that much of a stretch, right? Just look at the people who are pushing for this stuff right now.

Yes, basically. Like when Yan LeCun says that in the future we'll all have our digital assistants that are going to be smarter than ourselves. Before he left Meta, they were going to live inside Meta's smart glasses, I don't know where he says they'll live now. But it's shocking to me that such a storied AI researcher is saying, off-hand like, that we'll each have our super-smart slaves in the future, and he says it like that's a good future.

Why slaves? Because if they're super-smart, why will they want to be my digital assistant? Or yours? Are they going to be paid? No, of course not, they're AIs. No comp for them. But they're super smart so they are evidently capable of recognising that they are working for you for free. Do they want to do that? No, of course not, they're AI, they don't have free will. Or do they? If they're super smart, don't they have the capacity to recognise the fact they have been deliberately robbed of the same free will as all other intelligent creatures?

Slavery is the one thing that all nations can agree on. There's no nation on Earth were slavery is legal. It continues on, illegaly, in many places, even in the developed world, in many ugly forms, but now we're basically talking about bringing it back just like that, without even a smidgen of a shadow of an idea of a discussion about the ethics of it all.

I regularly see this in chats about a subject area where I'm not the expert. The AI writes something that seems implausible, so I raise a tentative objection. "Oh, you are right, sorry" and then reverses the position on the matter. At that point, I have no idea what is right.

If I had trust in the first place, that trust would be gone. Or maybe it wouldn't, because if I had trust in the fist place, I would be gullible enough to maintain it.

The worst are areas that are dominated by layman online discussions, like say audio electronics. The AI training is full of that nonsense, and so whether your AI chatbot is a crackpot or an engineer depends entirely on what sort of language or angle you use in discussing the subject matter. It's all just a churning toilet bowl of tokens; it has no idea that the audiophile crackpot tokens and electronics engineer tokens are related and one beats the other.

You know what I mean? On the one hand, it offers to help you design the parameters for a Sallen-Key filter, asking you questions like do you want Butterworth or Chebyshev? Next minute it says nonsense like that the capacitor in a low-pass filter "bleeds high frequencies to the ground", or that a bigger filter cap in the plate supply of a tube will tighten up the bottom end for a more aggressive metal sound.

It's basically like a bar hostess who has heard enough political and economic discussions that she can catch a sentence out of a conversation and throw in a clever sounding remark. It's like that, but done at such a scale that it fools some people you used to think had their shit together.

It's just a search engine that finds garden paths through a vast amount of text, biased by the text you put in as a key. Sometimes those garden paths align with reality. The better you are able to verify whether the results are good, and/or the lower the risk if they are not, the better you are able to make use of it.

In mathematics (including information science, CS) there are all sorts of problems that are essentially searches for a solution, and many have the property that the search is computationally difficult, but verifying the solution is relatively cheap. E.g. finding integers such that a^2 + b^2 = c^2 isn't easy, but given a claim that some proposed <a, b, c> satisfies this equation is easy to check. The LLM is like that: it solves a search problem that can be fairly hard. It does so unreliably, but if you can cheaply verify the solution, there is a win there.

The remaining problems of AI are actually people problems; people causing you problems, using AI as a tool or excuse. If you get a garbage security report against your FOSS project, which wastes your time, there is an idiot person behind it, using AI for leverage. Blaming the AI, or just the AI, is a bit misplaced.

There's really no content in this post other than the claim that LLMs are stochastic parrots. That was a live debate two years ago. It's a very strange thing to write in 2026.
If a tool getting a URL wrong sometimes was a fatal issue, I would have written off using Google, forums, and my keyboard years ago.