back
147 comments
I’m just re-reading Blindsight by Peter Watts, a novel that is amazingly prescient in — amongst other things — its predictions of this coming economic upheaval.

Speaking of… one scene in the book has humans communicating with an alien that appears to have “learned” human speech by training a non-sentient LLM on human ship-to-ship transmissions, and then “fine tuned” it to achieve the desired communication goal without ever actually understanding what the LLM is saying.

This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s! That’s proper science fiction, right there.

Back to the economic aspect: Several characters in the book had to “butcher themselves” with implants and enhancements to remain economically relevant in the age of AIs. It’s that… or you’re packed away in storage. Useless.

PS: Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”. You could use this to scare whales away from ships without understanding what specifically the LLM was singing to them!

>This is a book from 2006 accurately using the salient features of LLMs popularised in the 2020s

I'm a huge fan of Peter Watts' work, but to be fair that is at the very least a close-cousin to the Chinese Room thought experiment that's been around for a while.

Agreed. The concepts were floating around before, but nothing presented the idea of an LLM quite as coherently and presciently like Rorschach did. (I hasten to add that this is in my own limited experience - I am an SFF fan but I have not read every book in existence.)

As a layman with respect to LLMs, I feel like Blindsight and its perspective of the chinese room puzzle is required reading. I wonder if people working in AI feel the same and I'd be really curious to know how they feel about the book.

> non-sentient LLM

I don't think it is described as non-sentient in the book, but only as lacking consciousness.

Consciousness not being required for sentience is one of the two major themes of the book (the other being that consciousness is an evolutionary dead end).

Still your example is very relevant. I very much believe that ChatGPT, for example, is not conscious. The question is if it has some amount of sentience.

A much more interesting set of books (I've never managed to finish Blindsight due to Watts writing) is Ian Banks' Culture series.
> Imagine training an LLM on cetacean recordings and then fine-tuning on “orca attack imminent”.

Decoding animal communication using machine learning is a popular research topic. See the current issue of Science, for example: Using machine learning to decode animal communication (https://www.science.org/doi/10.1126/science.adg7314)

Beautiful! Or at least beautifully prescient. I'm starting to think storytellers are smarter than they let on.
I don't like Blindsight or Echopraxia, I just get caustic misantrophic vibe from Watts' writing. The whole materialist/hyperintelligent/nonsentient thing tastes very Marxist.
I don’t understand why economic disruption and human obsolescence isn’t considered an existential risk. We could end up in a world where 99.99% of people are redundant and useless cost centers for our GDP maximizing economic paradigm. In that case, you don’t need killer robots in order to push humanity towards near-extinction. The invisible hand will smite them.

But don’t worry, TED Talk attendee. The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian. So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.

It's a bizarre and blinkered article. The most immediate dangers are economic. We already have a hopelessly unstable economic system, with increasing swathes of the population economically disenfranchised. AI is more likely to accelerate that than prevent it.

The other immediate dangers are social and political. When one person with resources can run an AI-enhanced troll farm and social media PR engine - not even remotely science fiction - we have a serious problem.

Those threats are already politically and culturally existential.

And that's before anyone has even fitted a gun to an AI-powered autonomous drone. Or gunship.

AI is inherently conservative because it reinforces hierarchy. The AI-poor will have far less political, cultural, and personal leverage than the AI-rich.

Essentially it will have the same effects as money as a cultural and political practice - but much more so.

> We could end up in a world where 99.99% of people are redundant and useless cost centers for our GDP maximizing economic paradigm

If 99% of consumers no longer have money to spend, GDP will definitely not be maximized.

>I don’t understand why economic disruption and human obsolescence isn’t considered an existential risk.

Because that's the goal of our economic system! If everything could be made with no human labor requirement, then the price of these goods is going to trend towards zero. The disruption is going to cause pain, but ultimately that's what we've always been after.

There was a time when 90% of the labour force worked in agriculture. If I lived in the 1600s I'd be terribly concerned that mass unemployment would coincide with large increases in agricultural productivity.

However, when the agricultural sector became more productive the 'winners' wanted bigger houses, travel, education, healthcare etc and this provided enough opportunity for the displaced workers to find jobs. These new sectors, over time, even offered more wealth and better working conditions to their workers than the old agricultural sector.

Given that the number of people in agriculture declined from 90% to 10% over the past 200 years or so, I'm less worried about disruption like this. People are remarkably retrainable, and just because current jobs no longer exist doesn't mean we won't create new ones out of thin air (like marketing, or Youtubers, or film lighting specialists).
This is just repeating the fallacy that the person being replaced is going to be the blue-collar worker.

But it isn't...it's the office knowledge worker. Anyone cruising in the middle bucket where you don't do creative research, but you do sit in front of a computer. AI is coming for those groups.

The blue collar worker? They'll be fine for decades. Anyone who gets out there and manipulates the physical environment, won't be replaced any time soon.

Maybe this is why billionaires and hundred-millionaires are the ones so concerned about existential risk. A collapse of consensus in our current and historically not-that-old system of property rights is an existential risk -- to them.
> The obsoletariat mostly won’t be Americans. The median member of this class will probably be Chinese or Indian.

Anyone doing physical labor is going to be just as necessary in a world of artificial mind, it's those knowledge workers of the service economy that will find their skills automated.

As soon as a significant share of the population is „irrelevant“ and treated as such (read: no good welfare, …), capitalism will end quickly. „Who owns the machines that generate value“ will shift from individuals/corporations towards a public good, and Marx ultimately was right.
>So you can continue your performative concerns about Roko’s Basillisk or whatever topic of distant, paralyzing uncertainty is overflowing from the next room into mine.

Amusing, usually rationalists are made fun of because the concerns about Roko's Basilisk are not performative.

The world median will be chinese or indian, but the american median will be white!
FYI, the employment rate in the west in between 50% and 75%.
This is a problem with capitalism in general, not just with AI.
It is, just not by the media. They only worry when their jobs are threatened. The rest of us can have cake.
This is a truly awful article. I'm mildly positive towards AI because I think people tend to react with fear and examine the risks without equivalently examining the rewards, and I think that's happening here and the rewards seem amazing. But the arguments of risk are serious and plausible and this article is just lazily dismissive of them.

> AI is not competing for resources with human beings.

A superintelligence would absolutely be competing for resources (mainly electricity and cooling) with human beings.

> Rather, we provide AI systems with their resources, from energy and raw materials to computer chips and network infrastructure.

A superintelligence will be able to convince anyone it wants to do anything for it. Even the previous President did a pretty good job at causing adulation and democracy-threatening personal loyalty from people he had not actually met in person. Imagine how amplified and personal the effect could be.

> For now, AI depends on us, and a superintelligence would presumably recognize that fact and seek to preserve humanity since we are as fundamental to AI’s existence as oxygen-producing plants are to ours. This makes the evolution of mutualism between AI and humans a far more likely outcome than competition. Moreover, the path to a fully automated economy — if that is the goal — will be long, with each major step serving as a natural checkpoint for human intervention.

The author is literally relying on us outsmarting the superintelligence (collectively, with coordination!). It doesn't sound like they've come to terms with the concept of superintelligence at all. Or with the state of human global coordination problems, come to think of it.

> AI cannot physically hunt us.

This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process.

> AI’s impact on the climate is up to us.

Our own impact on the climate is up to us. Our collective decision-making is.. suboptimal.

> If we really think that superintelligent AI presents a plausible existential risk, shouldn’t we simply stop all AI research right now? Why not preemptively bomb data centers and outlaw GPUs?

You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.

>This is likely to become literally false sometime soon if it hasn't already, but even if it doesn't, the AI doesn't have to. It just has to convince another human that the human is in love with it and it wants the human to kill a bunch of people, then scale the process.

I'm stunned at the number of people who try to make this argument. The operators of Radio Télévision Libre des Mille Collines never had to leave their broadcast studio. Goebbels never had to do anything but get in front of a microphone or a typewriter. Still, millions were violently killed. We bicker about LLMs' ability to write C code that will compile, but their best abilities are to gaslight, emotionally manipulate, lie and create FUD.

> A superintelligence will be able to convince anyone it wants to do anything for it.

This "just so" claim has always struck me as quite a reach. Humans are manipulable, yes, especially by (perceived) group pressure, but there are still limits. Especially if the human can easily just mute the AI's output. It's like when people believe that a sufficiently clever hacker can get a given program to do anything. There are millions of security vulnerabilities, yes, but you're unlikely to find a RCE in the program `true`. But instead of the 80s wizard hacker trope we're going back to the older B-movie hypnotist who can take full control of any person by talking

> You are literally linking to the person who is the most prominent voice of AI existential risk, who is seriously suggesting doing exactly that.

There's a few links in the article - which person are you referring to? Being the most prominent voice calling something unsafe is not an inherently valuable qualification. The most vocal opponents of nuclear power tend to have a poor understanding of it

> A superintelligence will be able to convince anyone it wants to do anything for it.

What is the evidence for such an extravagant claim? Is there a correlation between IQ delta and persuadability that has no degradation as the lower IQ rises? Especially given that there are plenty of people far more intelligent than the President in your example who are much less persuasive.

It's very hard to transport humans to other planets, but a bunch of robots are already exploring Mars. I see no reason why a superintelligence would want to confine itself to Earth.

Why compete with humans on Earth if there's the whole Solar system to exploit?

The foundation of this article is that there is a zero sum game for resources for AI safety. I think we should reject this notion and improve AI safety both in terms of existential and non-extensional risk. If more resources are needed and can be used effectively for dealing with safety we should instead take resources away from propelling AI forward and apply them to safety.

I see this argument against existential risk alliance – that AI needs humans to provide it with resources- this we can hold it hostage. I like to point out that power plants are increasingly since the pandemic remotely operable- that AI will increasingly be able to hold us hostage by hacking the infrastructure we need to survive. Throw in the ability to use some robots and bio weapons to keep humans from taking them back easily.

The other aspect is that AI doesn’t have to properly conclude ahead of time that such an attempt has a high probability of success. Its program just has to decide that this will help it achieve some objective.

Arguments in the first part are really bad. I haven't read about ai safety that much, but still see it.

For example, ASI may be able to automate manufacturing really quickly, maybe be able to convince humans easily to do what it needs, etc Moreover, why ai would not be competing with us for resources? Materials on earth are finite.

Also, most likely scenario is not evil AGI, as it outlined in the article, but indifferent. And basing reasoning about AI on Darwinian evolution is barely useful.

Also, talk about this tribal people AI is ironically even worse then silicon valley narrow viewpoint: it is Californian liberal virtue signaling.

Is there a difference between evil and indifferent if you are on the receiving end? Humanity is largely indifferent to animal welfare and animals would qualify it as evil how we treat them.
Although I disagree with the conclusion, this looks like a pretty decent article in terms of detailed thinking.

I just think that we not only need to work on all of the more immediate problems but also worry about less immediate problems that are existential risks.

Also people always say "AGI" and they mean different things but I am guessing most now mean something like a digital living human/simulation of human when they say that. Mixed in with some blurry connotation of it automatically becoming godlike in power.

That stuff is very speculative. But we don't need AI to get to that level for it to be very dangerous. Just imagine that we have an open source GPT that is something like 33% smarter than GPT-4 and less brittle. Make the output 50 times faster than human thought and suppose that we can run it very inexpensively, in a "swarm" of agents cooperating.

Then you have a type of superintelligence that does not require any really speculative AI advance -- hyperspeed reasoning -- based on the current technology.

If that is widely deployed for military and industrial decision-making then something like a computer virus could create existential risk.

Also, realize that all these systems really need to emulate some of the core functional aspects of animals, like a self-preservation instinct or desire to control resources, is the right instruction and relaxation of guardrails. So when they can reason a bit better and at hyperspeed with collaboration between them, they don't need to be alive or anything to be dangerous.

Look at the history of increases in computing efficiency. It is easy to imagine 50 or whatever times output speed increase in less than 5 years. And even though they are not human level now, GPT-4 has proven that these types of systems can have strong problem solving ability.

imo GTP-X is going to need to be much much “smarter” and consistently accurate (which may be fundamentally impossible) before it’s going to be “widely deployed for military and industrial decision-making” anywhere.
Yes, this is a good take.

The general discussion is definitely prematurely focused on some sort of end-boss fight, while AI and large scale data collection are causing serious harm and privacy concerns to real humans already. It’s only reasonable to expect that this increases in the future.

I would like to see more discussion focussing on these pressing issues.

Also a pause on AI development is not an option, not a solution and is digressing from the issue at hand. Big capital with their hands on big valuable data will support anything that distracts here.

Finally, I do think that AI also brings many positives, I am not against the technology itself at all- we shouldn’t be.

name a single reason why a pause is not an option. dont say why it would be hard or unlikely or inconvenient or whatever, to exclude it as a possibility means it has to be physically impossible.

of course it isnt physically impossible. AI requires large compute, large resources. you cant advance the state of the art inconspicuously. the only reason you say it isnt an option is because you dont consider AI an existential threat, really, and therefore extreme measures arent justified. but they actually are.

AGI doesn't ever have to happen for rapid deployment of AI based systems to cause harm.

Google "robodebt australia" for what happens when government uses machine derived decisions to penalise the poor.

That point is basically what the article is about.
aussie here, former dole bludger, still know people who live off the welfare system, hide their side income, and are proud of it. There was sooooo much warning for robodebt that they all made sure their taxes were clean before the first demands went out, but a few still got through. Mostly accurate, but a few dodgy ones too. Of course, as we know, it was a dragnet and the human suffering produced was excessive. Considering the depth of the corruption, it was clear the system was incapable of correcting itself, so what else is there to do? How do you get help to the people with real need when there's so many people cheating? Robodebt was big hammer approach.
The distant future. Like five years. Self-aware (or a simulation thereof) AGI with self-directed motivation and the ability to self modify is barely a half step from ASI. And AGI could happen in the coming year. Five if you wanna be pessimistic. Never if you’re extraordinarily pessimistic but that’s looking unlikely. If it’s possible it’s possible now.

This isn’t the Yellowstone super volcano that might blow in the next billion years.

People who talk like this are idiotic at best.

Granted, the other concerns are real too but the accusation that the concern of existential risk is being used to hide the other known problems is dangerous in the extreme.

I'm pretty sure you could have expressed your opinion without resorting to insults.
"And AGI could happen in the coming year."

"People who talk like this are idiotic at best."

We live in a very confusing, transitory time. Right now there's a large chunk of people who are convinced that we are a few years away from extinction if we do not ration out GPUs and bomb illegal clusters of compute, while at the same time there's a large chunk that believes that a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company). Absolutely wild.
"Convinced that we are a few years from extinction," is an exaggeration of the pessimistic/alarmed end of the spectrum of opinion.

All the forecasts I've seen have the period of danger spread out over at least 20 or 30 years, and no one claims to understand the topic well enough to be able to tell whether a particular model is dangerous even if given complete freedom to examine the source code and interview the creators.

Our inability to predict is in large part the basis of the pessimism/alarm because it means that the leading labs won't know when to stop.

> a car can never navigate a single-lane one-way tunnel safely without a driver (ie, The Boring Company).

I think the criticisms of the Boring Company are more "small diameter tunnels are a bad way to build high-throughput underground infrastructure, and autonomous cars will not solve that"

Probably things would become clearer if what we currently describe as 'AI' we instead term as 'Statistical Pattern Regurgitator' or similar.

Any serious attempt at AI would: need to be trained on reality - like people are trained; need to overcome the cardinality barrier - digital systems can only approximate analogue systems; and need to demonstrate spontaneous language emergence driven by survival in a social environment.

One day, someone came up with the Transformer architecture for neural networks. The ideas didn't quite come out of nowhere, but their impact did - I don't think that anyone predicted the degree to which ChatGPT and its ilk are superior to LSTMs and the like. More importantly, nobody predicted the details, the way that the GPTs behave, the way they're willing to invent facts like they're spooling out plausible thoughts without being constrained to observed reality.

From a history-of-technology standpoint, the state of the art for generative AI went from LSTMs to ChatGPT overnight. There was no warning. Nobody could have looked at the Attention Is All You Need paper and predicted how much it would improve the state of the art.

And worst of all, we already know that AIs can be hostile. What else is a public corporation's corporate bureaucracy but a Chinese Room running an AI whose goal is to maximize quarterly dividends? What else is a government but another Chinese Room trying to survive? We talk about corporations being faceless or friendly, we ascribe them wants and needs and behaviors and habits, we give them names, and they suffer natural selection and mutation and reproduce and evolve over time so as to select from those bureaucracies that are most fit for survival. And this class of AI is already responsible for global climate change and the Cold War and so many other things that might already be on the path to push us into extinction.

LLMs are nowhere near as humanlike as a corporation. Corporations are bound by regulations and capitalism and basic human decency. It is absolutely possible to build an LLM whose fundamental goal is to kill people. All you have to do is tell ChatGPT to pretend it's playing a game of global thermonuclear war, and some joker on 4chan would do that in a heartbeat.

So what? Why does this matter?

We have no way to predict when the next revolution in AI capability will occur.

We have no way to predict what the next revolution will be.

We have no way to predict the capabilities of the next generation of generative AIs.

We have no way to predict how the next generation of AIs will "think", insofar as they do.

And because we have no predictions, we have no way to prepare.

So the real question is, how much risk are you willing to tolerate? Are you wiling to take a coin flip on it? A spin of the cylinder? A roll of the dice? Because that's what we're looking at here - the next generation of AIs has no reason to be any less inimical to human existence than the examples we already have, and AI will _only_ get better.

Was this written by chatgpt?