Richard Stallman on ChatGPT

Considering Stallman worked in the MIT AI lab in the era of symbolic AI, and has written GCC (an optimizing compiler is a kind of symbolic reasoner imo), I think he has a deeper understanding of the question than most famous people in tech.

He's not wrong. It's not intelligence. It's a simulacrum of intelligence. It can be useful but ought to not be trusted completely.

And it's certainly not a boon for freedom and openness.

It’s a little like saying calculators cannot do math because they don’t really understand numbers or arithmetic and they just do bit operations.

I understand the sentiment, but reality is that it does with words pretty much what you’d expect a person to do. It lacks some fundamentals like creativity and that’s why it’s not doing real problem solving tasks, but it’s well capable of doing mundane tasks that the average person gets paid to do.

And when it comes to trust and accuracy, if I ask it a question about German tax system, it will look up sources and may give an answer with an inaccuracy or two but it will for sure be more accurate than whatever I will be able to do after two hours of research.

I prefer using LLM. But many people will ask what is an LLM and then I use AI and they get it. Unfortunate.

At the same time, LLMs are not a bullshit generator. They do not know the meaning of what they generate but the output is important to us. It is like saying a cooker knows the egg is being boiled. I care about the egg, cooker can do its job without knowing what an egg is. Still very valuable.

Totally agree with the platform approach. More models should be available to be run own own hardware. At least 3rd party cloud provider hardware. But Chinese models have dominated this now.

ChatGPT may not last long unless they figure out something, given the "code red" situation is already in their company.

Absolutely hilarious that he has a "What's bad about" section as a main navigation, very self-aware.

If we define intelligence as the ability to understand an unfamiliar phenomenon, create a mental model of it, these models are not intelligent (at least at inference time), as they cannot update their own weights.

I'm not sure if these models are trained using unsupervised learning and are capable of training themselves to some degree, but even if so, the learning process of gradient descent is very inefficient, so by the commonly understood definition of intelligence (the ability to figure out and unfamiliar situation), the intelligence of an inference only model is zero. Models that do test time training might be intelligent to some degree, but I wager their current intelligence is marginal at best.

All a LLM does is hallucinate, some hallucinations are useful. -someone on the internet

> ChatGPT cannot know or understand anything, so it is not intelligence. It does not know what its output means. It has no idea that words can mean anything.

This argument does a great job anthropomorphizing ChatGPT while trying to discredit it.

The part of this rant I agree with is "Doing your own computing via software running on someone else's server inherently trashes your computing freedom."

It's sad that these AI advancements are being largely made on software you can not easily run or develop on your own.

I think we can call LLMs artificial intelligence. They don't represent real intelligence. LLMs lack real-life experience, and so they cannot verify any information or claim by experiencing it with their own senses. However, "artificial intelligence" is a good name. Just as artificial grass is not real grass, it still makes sense to include "grass" in its name.

The assertive jump from it not understanding to it being not worth using is pretty big. Things can also be useful without having trust in them.

What's so difficult to understand about LLMs? The meaning of intelligence is irrelevant. I like to call LLMs Associators. No they don't think and they don't understand. But it turns out by finding patterns and associate them on the basis of symbolic language, there comes up some useful stuff in some areas.

Perhaps we are putting human intelligence on a pedestal. I have often caught myself having a supposedly original idea only to find I inherited it from something I read months before.

We are mostly autocomplete with a mild capacity to synthesise new ideas. It’s the network effect of communicating that creates the feedback loops which amplify our collective intelligence.

Also if you think intelligent life has to have a regard for truth, not just a regard for self, tune in to any news channel.

The quality and usefulness of the service across different domains, the way it's being rolled out by management, the strategy of building many data centres when this makes questionable sense, the broader social and psychological effects, the stock market precarity around it, the support among LLMs for open source code and weights, and the applicability of the word "intelligence" are all different questions.

This reads like more a petulant rant than a cogent and insightful analysis of those issues.

That's what I'm thinking every time I hear or have to use the term "AI". It is not intelligent, but everyone is so used to call it so. LLM is much better.

Unfortunate that he starts with the thinking argument because it will be nitpicked to death, while bullshit and computing freedom arguments are much stronger and to me personally irrefutably true.

For those who will take “bullshit” as an argument of taste I strongly suggest taking a look at the referenced work and ultimately Frankfurt’s, to see that this is actually a pretty technical one. It is not merely the systems’ own disregard to truth but also its making the user care about the truthiness less, in the name of rhetoric and information ergonomics. It is akin to the sophists, except in this case chatbots couldn’t be non-sophists even they “wanted” to because they can only mimic relevance, and the political goal they seem to “care” about is merely making other use them more - for the time being.

Computing freedom argument likewise feels deceptively about taste but I believe harsh material consequences are yet to be experienced widely. For example I was experiencing a regression I can swear to be deliberate on gemini-3 coding capabilities after an initial launch boost, but I realized if someone went “citation needed” there is absolutely no way for me to prove this. It is not even a matter of having versioning information or output non-determinism, it could even degrade its own performance deterministically based on input - benchmark tests vs a tech reporter’s account vs its own slop from a week past from a nobody-like-me’s account - there is absolutely no way for me to know it nor make it known. It is a right I waived away the moment I clicked “AI can be wrong” TOS. Regardless of how much money I invest I can’t even buy a guarantee on the degree of average aggregate wrongness it will keep performing at, or even knowledge thereof, while being fully accountable for the consequences. Regression to depending on closed-everything mainframes is not a computing model I want to be in yet cannot seem to escape due to competitive or organizational pressures.

The question about what sort of "cognition" is going on in the simulated neural network of the current LLMs is an open one.

Saying "it's not intelligence" it's the wrong framing.

also, there are fully open LLM, including the training data.

He was right on a number of things, very important ones, but he's loosing it with old age, as we all will.

I never really considered this too deeply, because I've never studied "Agentic AI" before (except for natural language processing). Stallman is making a really good point. ChatGPT doesn't solve the intelligence problem. If ChatGPT was actually able to do that it would be able to make ChatGPT 2.0 on request.

OK take for Nov 2022.

Mundane for Dec 2025.

What the world calls an LLM is just a call-and-response architecture.

In the labs they’ve surely just turned them on full time to see what would happen. It must have looked like intelligence when it was allowed to run unbounded.

Separate the product from the technology and the tech starts to get a lot closer to looking intelligent.

> being capable of knowing or understanding.

The definitions of "knowing" and "understanding" are being challenged.

Also, it's no longer possible to not have a dependency on other opaque softwares.

This seems to be a complaint against general use of "Artificial Intelligence" term: none of it is "real intelligence" as we don't really have a definition for that.

He is right, once again.

There are some semantic debates going on in this thread about the term "bullshit". But there is a clear definition. The paper Stallman links to uses bullshit in the Frankfurtian sense, that is, talk without care for the truth:

> The liar, Frankfurt holds, knows and cares about the truth, but deliberately sets out to mislead instead of telling the truth. The "bullshitter", on the other hand, does not care about the truth and is only seeking "to manipulate the opinions and the attitudes of those to whom they speak"[0]

[0] https://en.wikipedia.org/wiki/Bullshit#Harry_Frankfurt's_con...

It simplifies a lot of analysis and critical thinking job for me so say what you want I call it "intelligence".

LLM is a model. So, it fits under "all models are wrong, some are useful". Of course, it can produce wrong results. But it can also help with mechanistic tasks.

And you can run some models locally. What does he think of open-weight models - there is no source code to be published. Closest thing - the training data - needs so many resources to turn into weights that it's next to useless.

Why do we continue to give increased weight to a guy whose computer science knowledge ends in 1980? Dude became an icon by preaching a rationalization to an all-too-receptive generation of cash poor college students: that Microsoft should work pro bono. Odd that he didn't extrapolate that principle to other "secret sauce" enterprises like Coca Cola or KFC.

Artificial grass is not grass, but we can use it for similar things.

I have most sympathy for the ideals of free software, but I don't think prominently displaying "What's bad about:", include ChatGPT, and not make a modicum of effort to sketch out a basic argument, is making any service to anyone. It's barely worth a tweet, which would excuse it as a random blurb of barely coherent thought spurred by the moment. There are a number of serious problems with LLMs; the very poor analogies with neurobiology and anthropomorphization do poison the public discourse to a point where most arguments don't even mean anything. The article itself present LLMs as bullshitters, which is clearly another anthropomorphization, so I don't see how this really addresses these problems.

Whats bad about: RMS Not making a decent argument make your position look unserious

The objection that is generally made to RMS is that he is 'radically' pro-freedom rather than be willing to compromise to get 'better results'. This is something that makes sense, and that he is a beacon for. It seems such argument weaken even this perspective.

I use ChatGPT for CLI app commands and it's perfect for that!

To me LLMs are glorified google search. I use them like that.

Extremely lazy take.

> ChatGPT is not "intelligence", so please don't call it "AI".

Totally ignoring the history of the field.

> ChatGPT cannot know or understand anything

Ignoring large and varied debates as to what these words mean.

From the link about bullshit generators

> There are systems which use machine learning to recognize specific important patterns in data. Their output can reflect real knowledge (even if not with perfect accuracy)—for instance, whether an image of tissue from an organism shows a certain medical condition, whether an insect is a bee-eating Asian hornet, whether a toddler may be at risk of becoming autistic, or how well a certain art work matches some artist's style and habits. Scientists validate the system by comparing its judgment against experimental tests. That justifies referring to these systems as “artificial intelligence.”

Feels absurd to say LLMs don't learn patterns in data and that the output of them hasn't been compared experimentally.

We've seen this take a thousand times and it doesn't get more interesting to hear it again.

> ChatGPT is not "intelligence", so please don't call it "AI".

I've been saying the same thing for years, but it's utterly hopeless by now. Even the critics use that ludicrous "AI" moniker.

> I call it a "bullshit generator" because it generates output "with indifference to the truth".

Sure we humans don't do this... right?

> I call it a "bullshit generator" because it generates output "with indifference to the truth".

Seems unnecessary harsh. ChatGPT is a useful tool even if limited.

GNU grep also generates output ”with indifference to the truth”. Should I call grep a “bullshit generator” too?

Old man yells at cloud.

> ChatGPT is not "intelligence", so please don't call it "AI". I define "intelligence" as being capable of knowing or understanding, at least within some domain.

Great -- another "submarines can't swim" person. [EDIT2: Apparently this is not his position, although it's only clear in a different page he links to. See below.]

By this definition nothing is AI. Quite an ignorant stance for someone who used to work at an AI laboratory.

ETA:

> Please join me in spreading the word that people should not trust systems that mindlessly play with words to be correct in what those words mean.

Please join me in spreading the counterargument to this: The best way to predict a physical system is to have an accurate model of a physical system; the best way to predict what a human would write next is to have a model of the human mind.

"They work by predicting the next word" does not prove that they are not thinking.

EDIT2, con't: So, he clarifies his stance elsewhere [1]. His position appears to be:

1. Systems -- including both "classical AI" systems like chess and machine learning / deep learning systems -- can be said to have semantic understanding, even if they are not 100% correct, if there has been some effort to "validate" the output: to correlate it to reality.

2. ChatGPT and other LLMs have had no effort to validate their output

3. Therefore, ChatGPT and other LLMs have no semantic understanding.

#2 is not stated so explicitly. However, he actually goes into quite a bit of detail to emphasize the validation part in #1, going so far as to describe completely inaccurate systems still count as "attempted artificial intelligence" because they "purport to understand". So the only way #3 makes any sense is for #2 to be presented as stated.

And, #2 is simply and clearly false. All the AI labs go to great lengths to increase the correlation between the output of their AI and the truth ("reduce hallucination"); and have been making steady progress.

So to state it forwards:

1. According to [1], a system's output can reflect "real knowledge" and a "semantic understanding" -- and thus qualify as "AI" -- if someone "validate[s] the system by comparing its judgment against [ground truth]".

2. ChatGPT, Claude, and others have had significant effort put into them to validate them against ground truth.

3. So, ChatGPT has semantic understanding, and is thus AI.

[1] https://www.gnu.org/philosophy/words-to-avoid.html#Artificia...

This is an idiotic take

"I call it a "bullshit generator" because it generates output "with indifference to the truth"."

yeah, no. the point of post-training with RL is precisely to get the truth on many tasks. Many of the answers in post training are judged on whether they are true or not, not just "RLHF / human preference".

Also it's not like human are perfectly commited to truth always itself and we don't question their overall innate "intelligence" sense.

Stallman just doesn't know what post-training is