▲ 136 points
back
187 comments
The LLM AI technology generation is optimized to be fluently conversational and not to be factually correct all the time.
1) Hallucinations often appear because LLMs are designed to create fluent, coherent text.
2) LLMs have no understanding of the underlying reality that language describes.
3) LLMs use statistics to generate language that is grammatically and semantically correct within the context of the prompt.
It sacrifices accuracy for being good at conversations as it is designed to do. All these criticisms of hallucinations are missing the point.
Generative AI is generative and basically is a specialist at making things up. It’s going to take things like multiprompting, network AIs that fact check output, and a host of other technologies or even entirely new models of AI to solve these problems, but don’t make the mistake of thinking that the system is supposed to be working without hallucinations right now — that’s not what it’s optimized for.
Language models (the computation) "care" about making "correct" predictions (correct of course being determined by the data).
This is why the simplest solution to reducing hallucinations is just making them more competent/better predictors (more neurons, more data). If the prediction can benefit from all the knowledge/reasoning it has accumulated from training, it'll tap that first. One of the biggest tells of hallucinations is if the model is saying something completely different each generation vs being consistent.
What they just don't care about is communicating being out of distribution or making whack predictions. This becomes a problem for humans because they're perfectly fine making things up when the above fail.
But by all accounts, they do learn to distinguish these things. The computation is very much aware when it is going way off base.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets - https://arxiv.org/abs/2310.06824
> LLMs have no understanding of the underlying reality that language describes.
LLMs absolutely have concepts that extend beyond just words. As early as 2008 (back when there were no large language models, only "regular" language models), we've been able to demonstrate things that seem to me like the model is learning abstract concepts. For a classic example, see Linguistic Regularities in Continuous Space Word Representations [0], a 2013 paper that talks about a model with a latent space where one can take the vector representations of the words "king", "queen", "man", and "woman" and literally perform the arithmetic `king - (man - woman) ≈ queen`. To me, this clearly demonstrates that the model "understands" the concept of gender, represented by a vector in the model's latent space. The set of numbers that you get from the subtraction `man - woman` represent the concept of the difference between the male and female genders, without needing a specific word tied to that representation of the concept.
It's debatable whether or not that counts as "understanding", but that's more of a semantic debate about what it means to understand, not a debate about the model's internal knowledge of the world and ability to do things with that knowledge.
> "The LLM AI technology generation is optimized to be fluently conversational and not to be factually correct all the time."
I always find this point a bit odd because humans aren't "optimized" to be correct either.
> "LLMs have no understanding of the underlying reality"
I struggle with this one because I see both sides of it. I was making a prompt the other day and gave a CSV file as an input and told the LLM if could only answer with values from one column and it did exactly as I asked. It's hard for me to see things like that and not believe it has an understanding at some level.
When a human reads a bunch of stuff on the internet and regurgitates it as fact, we don't say they hallucinate, we say that they're wrong.
Why we baby AI on this front, I have no idea.
> All these criticisms of hallucinations are missing the point.
Not if people need to be reminded that, as you say, LLMs are not designed to give reliable answers. Many, many people appear to believe that they are, since they rely on the answers in all kinds of contexts. For example, in past HN threads on LLMs, I have seen people say that they rely on LLM-generated code.
> LLMs have no understanding of the underlying reality
Not true. They are slowly gaining an understanding of reality by reverse engineering the relationships built into human languages. The only reason LLMs are getting better is because they are better modeling the world. At some point the only way to improve token prediction is to gain an understanding of the world.
In other words, LLM's are, in the technical sense defined by Harry Frankfurt, bullshitters.
The essence of bullshit (as he defines it) is that it is neither truth nor lie but rather unconcerned with veracity altogether. A bullshitter wants to appear smart, or convince others to agree with them, or some other conversational objective. Bullshit statements may or may not ultimately turn out to be true, but the thing that makes them bullshit is that the bullshitter didn't actually know when they said it.
Hence I propose a far more succinct term to describe the phenomena at hand. A far more accurate terminology than the presently vogue "hallucinating": AB. Artificial Bullshit.
Did you read the article?
AI hallucinations are the opposite of creativity. They give the most obvious wrong answer possible. For example, ask an AI about an unknown basketball player, it will probably describe it at fit and tall, because that's how we expect basketball players to be. That may be a good observation, but certainly not creativity.
If instead it told the story of someone short and obese, with details on how he got to play basketball, that would be creative. Of course, you can ask an AI to tell the story of a short and obese basketball player, and it will tell you a likely story for such a player, but again that's not creativity, it is just the AI filling the blanks based on whatever similar stories it had in its training set.
Is creativity something innate in humans, or can it be learned? I would argue that if a human can learn creativity then an AI can too. By the way, here's GPT-4 "Write a description of an unlikely basketball player. Be creative."
> Meet Ethel, a 4'9" grandmother of six, with a penchant for knitting and a mastery of Sudoku. With bifocal glasses perched on her nose, she's far from your typical basketball star. But what Ethel lacks in height and athleticism, she makes up for with an uncanny sixth sense for predicting opponents' moves and an underhand free-throw that rivals the best in the league. With her floral-print headband and orthopedic shoes squeaking down the court, she's both an anomaly and a secret weapon on her community center's basketball team.
Creating doesn't mean having to maxmize entropy either. Lots of good stories keep very close to the "obivous" or the well-known.
I don't know, asking ChatGPT4 to be creative seems to work okay:
https://chat.openai.com/share/62180301-b7ae-46fe-bd15-bd6973...
> Jack "Shadow" Carter was a prodigy dismissed for his short stature, standing only 5'7". Ignored by scouts and overshadowed by taller players in high school, he developed a unique playing style that exploited his low center of gravity and agility. He became a master of steals and assists, zipping around the court like a shadow, hence his nickname.
(Though I get your point that "hallucinations" will tend towards lowest-common-denominator answers, not creative answers.)
This is what the temperature parameter controls. Increase it for less probable (arguably more creative) output.
Right, this is also why we don't see breakthroughs in science or literature from AI.
This is an odd article. To me, it seems like the "creative" arts are an ideal arena for AI. After all, there's no such thing as "wrong" art.
The article says "well, sometimes what it makes is bad".
Well big deal. A lot of human-created art is awful too.
> This is an odd article. To me, it seems like the "creative" arts are an ideal arena for AI. After all, there's no such thing as "wrong" art.
Yeah but who wants to consume art purely generated by AI (that is, not human-created with AI support)? Most art sites have had blanket bans, or at least required tagging, on ai-generated art because people hate it so much.
Or to put it another way: why are you in the comment section of Hacker News, and not just asking ChatGPT to generate social media comments on the article?
I really didn't think anyone was applauding the "hallucinations" but maybe I just hang around people who aren't visionaries.
It's better to treat llms as more of a language encoders and decoders and have some for of intermediate representation where facts are verifiable. Or some form of a validation mechanism that isn't exactly checked against the generated texts. Pretty sure there are people working on integrating stuff like knowledge based but state of the art llms don't seem to have any kind of fact verification mechanism built in
Contra the headling, we should "celebrate being wrong" – when it's wrong in fast, interesting, & voluminous ways that can then be filtered or corrected.
That's how lots of science, innovation, & learning work: generate many superficially-plausible candidates via a fast-and-loose process, then refine with a more rigorous evaluation.
That AIs, in the form of LLMs, are now doing this so well was unexpected, and progress in checking 'hallucinations' is proceeding very fast.
(Fortunately, the article is less dismissive than the headline, recognizing these model's potential & mainly urging an understanding of the limitations.)
Key sentence for me:
> It might be better to say that everything GPT does is a hallucination, since a state of non-hallucination, of checking the validity of something against some external perception, is absent from these models.
I try to explain this to people who are obsessed with using ChatGPT to tell them things. So far I've been telling them something like: "it does not attempt to provide you valid information, it's optimizing for what would read like a reasonable continuation of the conversation, which is really not the same thing."
"Hallucinating" is really a euphemism for fluent bullshitting. It also doesn't fit the normal understanding of a hallucination. It's just wrong information expressed confidently. Partly it's how the conversational UI works, and partly how the model works. Without the nice generated text that creates in our minds an impression of competence, it would just be an error.
I never liked that the term "hallucinating" ended up sticking.
The AI doesn't know something so it just invents something. We usually call that "bullshitting", or in more polite crowds, "lying".
So the argument is “Our brains can freewheel generative ideation without mistaking our imagination for fact”.
When a human thinks it’s able to determine the difference between imagination and fact, but when a human reads words on a screen it isn’t?
Drivel.
Yet another article that imagines we know of some secret sauce in the human brain that DNNs can't possibly have. One for the dustbin.
> Unfortunately, this promotes a misunderstanding of how large language models (LLMs) work... It might be better to say that everything GPT does is a hallucination, since a state of non-hallucination, of checking the validity of something against some external perception, is absent from these models.
Let me turn it around:
> Unfortunately, this promotes a misunderstanding of how brains work... It might be better to say that every question a human answers without research is a hallucination, since a state of non-hallucination, of checking the validity of something against some external perception, is absent when simply answering a question.
Obviously nonsense. If you're going to write an article about misconceptions you'd better make sure you are right!
(Though it is silly to celebrate hallucinations; they're definitely not desirable.)
Spot-on article, for the most part. I firmly believe that the use of an LLM as a fact-finding tool represents a misunderstanding of the technology's abilities and intent. If you're using an LLM like it's a search engine, then you should just use a search engine instead. LLMs are conversational devices. Search engines are an index of human knowledge. Right tool, right job.
I'm fine with the term "hallucination," though. Hallucinations are, by definition, not real. The term emphasizes a detachment from reality that LLMs possess, and the general public is just beginning to grasp.
All thoughts are hallucinations.
Think of requiring a minimum amount of hallucination in AI output as a safety mechanism, like mixing the distinctive odor into a propane tank. The odor is a signal that there's a gas leak that must be dealt with. A high minimum amount of hallucinations is a signal that the source is untrustworthy and must be checked. Hallucinations may turn out to be a feature that protects against over reliance. And defers the need for a Butlerian jihad.
"Hallucinate" is the wrong word when applied to LLMs. Why? Because in common usage that means you are under the influence of some kind of external substance that has entered your blood-stream, or maybe you are under hypnosis, or shamanistic chanting. That is not the case when somebody says "LLM is hallucinating, again!",
It is also sensationalist, implying we don't really know what's going on, why is this LLM hallucinating?
It even sounds like an excuse: Hey my LLM did not come out with reasonable answer, but that is only because it was hallucinating, just like humans sometimes do, so it is even more human-like than we thought. See. Or may it was drunk! That explains it.
No, it's just that LLMs are sometimes on the topic, sometimes not. When somebody says they are "hallucinating" it does not mean they are working in some kind of extra-ordinary mode of operation. They are working just as usual.
What explains what some people (want to) call LLM "hallucinating" is that LLMs sometimes make sense, sometimes they don't.
Hallucination... that's an anthropomorphism. LLMs are functions. They don't make things up, they just sometimes return non-factual data. Which makes complete sense; the model does not have the notion of a fact, otherwise we would be talking about something more like expert systems, or knowledge graphs, not LLMs.
Isn't confabulation a better term than hallucination?
Honestly AI's hallucinating isn't a whole lot different than real life. Only a small minority of humans fact check anything anymore before believing it.
So "irl", we see people like Alex Jones that get up on their big platforms and start spewing nonsense, but if they sound confident enough and it confirms what you want out of the world, then people latch onto it as fact and don't bother to verify. You see this across the internet. Just today I saw a story on instagram that had been re-posted and the person that was talking about it was many degrees removed from the original, but believed it to be real. Looking through the comments, I had to scroll past 50+ comments to find someone who finally called it out as fake. Everyone else was just posting "no way", "wow, I never knew". You never knew because its completely made up. But when we hear someone speak with confidence and we don't care enough to fact-check, then people just believe it.
This is no different than AI. AI models sound confident and reliable. We assume they are making proclamations based on fact but they aren't always (or "usually" in my experience). Many people blindly believe the AI models because they sound reliable and confident in the way they speak. They never say "i don't know".
What AI is doing is problematic for sure. On one hand I want to take the pitchforks and revolt. But on the other hand I look around and realize, that even if we fixed it or vanquished this enemy, we still have a bunch of talking heads doing the same thing.
Maybe AI hallucinations are actually the most human element of AI.
So call it fiction. We like fiction.
Truth is hard. Maybe too hard for a mere machine. But dramatic narrative and quirky dialogue might be quite doable.
3000 chapter litrpg fantasy generated overnight.
The term "hallucinating" in this context is just another way we anthropomorphize machines. It is mostly harmless but I see the author's point that it can be misleading when trying to address specific issues.
On the (rare) occasion I find it useful to avoid this very normal tendency I ask myself if it would make sense to apply the same framing to the output of an AI image generator.
Curiously enough, I don't get the same result as the author on plain ChatGPT-4.
https://chat.openai.com/share/e213e0bd-2838-45e2-9942-e52954...
https://chat.openai.com/share/64bc62d3-042c-40c4-8d50-8e28ce...
Hilariously, plugging the example in the article into Bing enhanced ChatGPT-4, ChatGPT-4 w/ Bing hallucinates because of that very article!
https://chat.openai.com/share/4fe50933-8436-44ad-a778-6297ca...
If you tell ChatGPT-4 w/ Bing to ignore thereader.mitpress.mit.edu where the article is hosted, it doesn't hallucinate the string "Evolution by Any Other Name?".
https://chat.openai.com/share/b8a94a8e-723f-40b6-b7c5-698dcd...
source: https://chat.openai.com/share/4fe50933-8436-44ad-a778-6297ca...
What a weird point and conclusions. Yes, we shouldn't treat LLMs as search engines. But who's to say what's creativity and what's not? Glitch art is "wrong" if we apply the author's judgement, but it's still creative.
FYI this is a fluff article that has zero value. Just skip it if you're curious about LLMs.
Boy, is the author out of loop!
Hallucination is always and everywhere used as a negative term for LLMs. And it is seen as a problem/challenge that we should get rid of.
Did not read the article after seeing such a wrong title.
it's just a consequence of the annoying anthropomorphizing of tech. "Human's can't do X, the AI model can't do X, look it's just like me fr, fr". Of course nobody applies that logic to a forklift or a debugger. If gdb started to hallucinate variables into existence we don't call it a creative act, we call it a bug.
Given that these AI systems just like any other machine operate at scale, automated, and fast, they must be precise and transparent, that is where the work should be.
If an AI doesn’t hallucinate you run into another problem: you start to suspect that the output was copied whole cloth from some other source somewhere and is being passed off as novel.
What is the important distinction between a human dreaming about nonsense and an LLM sounding creative but being wrong?
who is "celebrating" hallucinations?
I've only heard of people trying to "solve" them.
I remember when MIT used to do things. Same with RPI and Stanford. Now they just do papers.
Have you met artists? They're always bullshitting and hallucinating on drugs.
When I hear the word "hallucination" in my mind appears an image of a crazy guy almost with foam on his mouth, probably on drugs or having severe mental problems. That is not a thing that I associate with being creative and certainly not trust worthy.
LLMs aren't knowledge retrieval systems.
Nor are they capable of creativity.
*Confabulation