▲ 161 points
back
243 comments
> If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster.
Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it.
Cheetahs are very fast, but humans have way more endurance.
Fun fact for me personally: because I am lucky enough to know a zookeeper who was in charge of the ambassador animal program at a major zoo, I have direct personal experience that cheetahs purr when you pet them.
You're right. They're smaller than you probably imagine (about the weight of an average Labrador). That's still definitely big enough to be a problem if they felt threatened, I'm sure, but the animals my friend was in charge of were raised to be around people for outreach purposes. That particular cheetah, for example, had once been on the Today show.
Cheetahs (genus “Acinonyx”) are the largest ‘small cat’, apparently [1] - separate from the genus “Panthera“ containing the ones that will attack humans (lions, tigers, leopards).
[1] There’s a well known viral video of a wildlife park keeper who sleeps with three cheetahs who behave pretty much like large house cats.
Ice yachts get well over 100 mph. I'm not sure how much they were used in pre-industrial times.
Why is this a "fun fact", what is this reddit-tier comment contributing?
Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”
I like this essay, and have very often referred to it when someone talks about AGI.
There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds.
It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity.
Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile.
So, less God, more WAU from SOMA or the Ocean from Solaris.
For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking.
The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.).
An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue...
Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes.
Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible.
That's hand waving.
I've only heard the God narrative from skeptics as in look at those idiots thinking they are building God. I think most people who believe in AGI arriving see it more as something like a chess computer but as well as beating us at chess it'll do other thinking too. A souped up chess computer isn't God.
Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's not driven by natural selection and standard rules of life, but it is important to frame our relationship with machines in new ways if we're ever going to make some sort of artificial intelligence.
Gotta keep in mind that "you control what you can destroy" was expressed as the crux of a single-planet guerrilla group's maximalist negotiation strategy with an interstellar government, where the destruction was of the ability to travel between the stars. It's not a coherent theory of control outside of the razor thin application of "I'm willing to accept pain as long it hurts you much more." It is not a control that builds stability and infrastructure, it is the control of base extortion
> Sure, you can turn it off but then your food would rot and you might starve.
If I live in a world where I can afford a freezer with food in it, it's practically guaranteed I can destroy my fridge without starving to death after. Heck, even if I was completely broke I could destroy my fridge and would have a pretty good (+99.999%) chance of not dying in the next year from starvation.
I get I'm nitpicking your point a bit, but I actually think most of our machines we could destroy and still be fine. We'd need to make sacrifices to our quality of life of course..
I can destroy my refrigerator pretty easily. If I care about food inside, I can take it out into a new one. So, seems I control it by that definition. Your idea "but food will spoil"/"I will lose a small amount of wealth" seems irrelevant to the strict definition.
Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.
Could you elaborate on that last part?
I can't say I disagree:
We need better scifi! And like so many things, we already have the technology.
This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly.
It's already been translated well into English, it just needs to be better distributed.
What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing.
They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.
Lem’s Cyberiad was my favorite as a kid and still is!
Related. Others?
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)
Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)
Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)
Maciej Ceglowski – Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13120213 - Dec 2016 (4 comments)
Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch.
Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles.
[0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805
[1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised...
[2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/
It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of profit. The need for way too much training data indicates something is still wrong with the current approach.
None of that was predicted.
I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players:
https://news.ycombinator.com/item?id=12168228
I even wrote up a whole article that specifically called RL loop based development as the future:
https://medium.com/@andrewkemendo/the-ai-revolution-will-be-...
> Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
The "Einstein can't get a cat into a carrier even though he is smarter" is just a hilariously bad argument. All cat owners can get their cats into a carrier! And most cats don't want to get in, because they hate the vet! And it's almost entirely because the humans are smarter!
You can even get a literal tiger into a carrier, even though it can kill you easily. You just drug its food and wait till it passes out. This is because you are smarter than it, and know that tranquilizers exist and how to obtain them, which is a strategy that cats of any size are not even able to conceive of, and probably can't understand what happened after it's been done to them.
AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
> The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process.
In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
>Sam Altman, the man who runs YCombinator, is my favorite example of this archetype. He seems entranced by the idea of reinventing the world from scratch, maximizing impact and personal productivity. He has assigned teams to work on reinventing cities, and is doing secret behind-the-scenes political work to swing the election.
>Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
How right the author was.
This was one of the rare critiques of AI doom that actually understands the case for it and presents them well, so I kept reading to see what its arguments for our safety against AI doom were. They were roughly:
1. It's hard to put a cat in a box despite us being smarter than a cat, so we're safe. (Counter: we're pretty good at putting cats in boxes when it matters.)
2. It was hard for Australia to kill Emus, so we're safe. (Counter: Australia could probably kill all Emus if it mattered enough, and we definitely accidentally kill off species when one of their inputs for life matters enough to us.)
3. Some smart humans get paralyzed by hedonism or existential angst instead of optimizing for arbitrary goals implied by their arbitrary value sets, so we're safe. (Counter: others overthrow the Czar, land rockets, etc.)
4. Modern AI is data-trained, so recursive improvement requires more data, so we're safe. (Counter: AI-crafted, synthetic data is a thing.)
5. We don't (yet) know how to improve our brains with brain surgery, so we're safe. (Counter: same as #4 above, which unlike us/evolution AI is being deliberately trained to understand and perform.)
6. Children take a long time to grow up, so we're safe. (Counter: the author's own "Premise 5: Computer-Like Time Scales", where they correctly note that computers can be arbitrarily faster than us.)
7. Individual smart humans on a desert island would be cooked, so we're safe. (Counter: nothing says the capability of a single AI must stop at that of an individual human, or that of a small group of smart humans; humans brains got dropped into a savannah and eventually they launch rockets.)
8. If AI doom is not a real threat, believing in it makes you believe some other not-real things that seem crazy or distasteful. (Counter: do we have a clear argument why it is not a real threat yet, in the list above?)
2016: this monstrous intellectual creature, through devious modeling of what our emotions and intellect are like, will be able to persuade us to do things like give it access to factories, synthesize custom DNA, or simply let it connect to the Internet, where it can hack its way into anything it likes and completely obliterate everyone in arguments on message boards.
2026: hold my molt beer
(I love how "connect an ai to the internet" is always the precursor to doom in pre-2022 scifi scenarios, and then as soon as we get something we call ai we hit that big red button)
> Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept.
This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans."
I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity.
In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.
Some previous discussions:
2016 https://news.ycombinator.com/item?id=13240811
Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human?
It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc.
(A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?)
The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place.
Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
"So I'd like to engage AI risk from both these perspectives. I think the arguments for superintelligence are somewhat silly, and full of unwarranted assumptions.
But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously.
First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity"
--
The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence".
I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous.
I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI".
You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen.
I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction.
This is just wishful thinking
We don't have a good definition of intelligence
Also, the premise that this thing would take over, it's hard to reason why it would do so
Anthropocentrism is also problematic in this article.
I note for premise 2 that the people who doubt the premise are the very stupid, but more importantly the very smart.
The former is no challenge to the premise, but the latter? That is a different story.
EDIT : For S&G I asked Claude about it. It replied :
The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this easy to accept. But that's a headcount, not an argument. The religious objection can be set aside because it rests on a premise (the soul) the materialist simply doesn't share. Penrose's can't, because it's pitched entirely inside the materialist frame — Gödelian limits on algorithmic understanding, non-computable physics in the substrate. You don't get to wave that away; you have to show it's wrong. The talk does the former and pretends it's done the latter.
The entire superintelligence thesis is a wager on the authority of intelligence — that smarter minds see further, judge better, and that this is precisely why we should fear or defer to them. If you take that seriously, then dissent from the very smartest humans on the exact question of whether minds are substrate-independent is the most expensive dissent available. You can't venerate intelligence as the thing that settles everything and then file your most intelligent objector under "outliers, moving on." The move is self-undermining on the argument's own terms.
You have to have a certain degree of intelligence in order to be convinced by stupid ideologies.
It might be childish, but I did laugh at the Ben Carson aside.
> Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society.
This article is from 2016; now it doesn't feel like backlash is strictly a function of manipulation.
> Hopefully you see the resemblance between this vision of AI and a genie from folklore. The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting.
The monkey's paw. You know, you don't need superintelligence for that.
Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
This was such a terrible piece of work back when it came out, it hasn't aged any better. By the way, it was responded to by MIRI: https://intelligence.org/2017/01/13/response-to-ceglowski-on...
"Recursive self-improvement" is in the same league as "perpetual motion".
What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
>Is superintelligence just a memetic hazard? [Overblown fear by smart people who are too easily convinced.]
Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?)
If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm.
Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive.
Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
Any of this kinda talk these days is just part of the hype train for the LLM companies tulipomania pyramid scheme.
Afaik no-one that is actually working on AGI is anywhere close atm.
In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective wisdom of those with skin-in-the-game.
If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed.
All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
Superintelligence Alignment conversations are a misdirect.
Let's talk about Billionaire Alignment, Economic alignment, Human alignment.
Classware should be M.A.D. -- in that it shouldnt even happen.
> But enough people take this scenario seriously that we have to take them seriously.
no we don't...
not sure where this notion comes from that if enough public figures are worried about something, then we must also
"If AdSense became sentient, it would upload itself into a self-driving car and go drive off a cliff."
This is a pretty bad talk.
It starts of interesting and then goes into lots of nonsense and non-sequitars when it start its takedown. (note: I'm not an AI alarmist, just reading the talk)
Arugment from Wooly Definition: An irrelevant argument - we don't need more intelligence. All we need is human intellegence + duplication and communication. An AI can clone itself immediately with its existing knowledge. A human can't. And AI can transmit thoughts perfectly "I know kung-fu style" a human can't
Argument From Stephen Hawking's Cat: This is also irrelevant. The arugment is supposed to be against Superintelligence but this argument is against controlling it, not against it happening.
Argument From Einstein's Cat: more of the same
Argument From Emus: more of the same - we can't control it
Argument From Slavic Pessimism: also not an argument against superintelligence.
Argument From Complex Motivations: Not an argument against superintelligence. Only an argument that some intelligences have mental issues
Argument From Actual AI: this didn't age well
Argument From My Roommate: not an argument against superintelligence. Only that some intelligences aren't motivated.
Argument From Brain Surgery: Not even sure that this is saying? It seems to be saying you need to learn stuff? Yea, people learn, AI can learn.
Argument From Childhood: Not an argument against AI. (1) unlike humans, AI can duplicate with full knowledge. (2) AI can learn faster than humans. Already proven.
Argument From Gilligan's Island: It takes a village is not an argument - AI can also specialize if it needs to.
Grandiosity, Megalomania, Comic Book Ethics: These argument that the people who believe in it often feel they should be charge. I agree that's true and bad. This is not an argument against superintelligence.
Transhuman Voodoo: This is an appeal to "these ideas sound too incredible therefore you should not believe them". Not sure how that's an argument
Religion: Agree, people who beleive and seem and maybe are religious. That's not an argument against superintelligence.
Simulation: Non-sequitar. This is "some of these people believe other crazy stuff QED no superintelligence". That's not an argument against superintelligence.
Data Hunger: This is actually an argument supporting the superintelligence believers. They believe sucking up all the data is bad. Not sure what argument is being made here relativel to superintelligence.
... and I stopped ... What a waste of time
None of this guys arguments are persuasive, but I do recommend everyone read Nick Bostrom's Superintelligence.
Your assumption that minds come in all shapes and sizes is wrong. Read up on embodied cognition. If anything at all, AI are true aliens, unlike known minds.