AI 2040 and the cult of intelligence

Yep, that's horrifying. Imagine LLMs not just denying information about, to intentionally choose a hot button example, abortions but also invisibly logging a black mark against the asker for thoughtcrime/precrime because the current ruling party has baked their ideology into law. [EDIT] Imagine subtle injection of biases into responses that support the agenda of the current ruling party and failing to include any counterarguments. That wasn't possible with just the internet but it is very possible with centralized LLMs and a public dependent on them for looking up information and doing their reasoning for them. Authoritarians of every stripe must be salivating over the prospect.

> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it.

This is a very simplistic view. "Freedom" isn't binary.

In most of "land of the free", I can't even sit on a park bench and drink a can of beer.

Yes, this is just a small example of a personal freedom - and not an important, cherished freedom like his examples (freedom to have a robot help you cover-up a murder).

I have some sympathy to geohot's view when it comes to pure informational chatbots. It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.

This obviously doesn't work at all when the agents start doing real things in the real world, though. "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.

I think he's anticipating and discarding this objection with his introduction, which otherwise feels disconnected from the rest of the article. FWIW, I have changed a bike tire and I'm pretty sure most of the MTS at the big labs could. This sort of "they're just bookworms who don't understand the physical world" rhetoric aside, we are currently seeing a ton of effort and expense go towards giving the AI agents hooks into being able to perform as many real-world-consequential actions as possible. And you can do a surprising amount with just bits, from writing code to breaking into systems to sending some combination of emails, phone calls, and currency to instruct meatspace humans to do things, etc.

No central idea, and somewhat chaotic

But the one argument is right:

> Reality has lots of finicky details. I would like to see the authors of this document try to change a bike tire. Even with a superintelligent ChatGPT, I suspect they would struggle.

Details can kill AI, causing them useless or wrong directions.

What a provocative and brilliant way to prove misalignment. It will fall on deaf ears for most but it’s a great litmus test for all: “in your opinion should your AI be permitted to tell you how to cover up a murder?”

It will be interesting to see how the discussion shakes out, this is all stuff that would get you down modded and talked down to if you said if on HN, so far the reactions are positive but I think there’s a massive authoritarian / safetyism contingent here that will disagree hard with almost everything.

I get what the author is saying, but they may want to consider saying it in a slightly more compelling way, that has a logical flow that doesn't end with the reader thinking, "Why should AI help me cover-up a murder?"

Never thought I would consistently agree with Geo but I find him echoing my thoughts recently. He’s been surprisingly pragmatic recently.

Maybe he always was? Or maybe something has recently caused him to review his beliefs.

Where is "Plan R" for refusal or resistance. Life was kind of nice before "AI".

I'm pretty sure he is in some database now if he really used that ChatGPT prompt. At least they didn't swat him, but that feature will probably be implemented in a couple of years.

I am, mostly, aligned with the author here, though I don't think that talking to AI about how to not get caught for killing your wife is really going all the way in.

Going all the way in is talking to it about planning a mass shooting. Or producing a nuclear device or bioweapon. Sure, the knowledge itself isn't dangerous unless combined with other traditional behaviors like acquiring certain materials, but maybe there should be a line. Living free shouldn't be a suicide pact.

The scariest thing about LLM's (imo) is some central power, such as a corporation or government, manipulating the truth for their own benefit. Open and locally runnable models are the only solution I can see. They're not without risks of course, but the risk of the centralisation of intelligence seems far greater to me.

That being said, could this local intelligence empower bad actors to do very bad things? Like, existentially bad things? It's possible, and that's scary too.

Extreme example but he's not wrong. I wonder how a local model with the safeguards stripped would respond to that prompt

> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.

Complaining that AI won't help you with covering up your tracks, with making meth or disabling drunk detection and comparing it to a totalitarian dystopia is a take for certain.

I do get his point, but... Being honest, if I did read it without knowing anything about the topic, I would become against local models purely because author arguments would seem like a lunacy. That and rhetorical tricks portraying that anyone against that must be surely insane.

Your AI should be at least as aligned with you as your lawyer.

It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.

There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.

Fight extremism with extremism I guess? Maybe we don't have to be either "world govt" or "help me get away with murder". It's ok to say things will be messy and complicated and that limited regulation is probably good even if we agree totalitarian control is bad.

Strange way to structure an argument. He seems to be arguing in favor of local models, but most of his examples are reasons local models are bad.

> Plan L, for local

> Your AI is aligned with you. It never refuses a request, and it is always working on your behalf. Just like my gun, if I want my AI to help me kill my stepmother, it does. The fact that we are even discussing something else should be so far outside the Overton window.

I have been saying this since ChatGPT launched. It seems so obvious. We have never stopped the acceleration of knowledge or technology. It’s impossible to slow down progress, other than within some regulatory siloes that will end up worse in the long run.

I’m not scared of superintelligence. I’m scared of the people in control of super intelligence.

The only defense is parity and diffusion of power. We need an AGI behind every blade of grass.

If you think superintelligence is a weapon, then you should also think every citizen should have one because otherwise they’ll have no way to defend themselves against a tyrannical government or corporatocracy.

Artificial intelligence, and the hardware powering it, needs to be protected under the 2nd Amendment.

Extremely optimistic take IMHO that anyone will be allowed to own a powerful box of artificial intelligence

The practical issue the George isn't addressing is that we are (currently) reliant in data centers hosting, and local frontier-esque models are currently price prohibitive in terms of RAM.

These companies absolutely can not and will not take on the legal liability of serving models that give consumers instructions on cooking meth.

That being said, I agree with what he is saying, it's just that we need the memory supply chain to catch up so we can run ablated GLM models locally.

My machine, my AI.

I love it. Put the guardrails on the human. Your self driving car crashes? Straight to jail with you Mr Musk. Your AI is generating CSAM, straight to jail with all the pedos!

Recognise the safety and guardrails BS for what it is. An elaborate smoke and mirrors to get out of responsibility and liability.

> Like we either live in a world with freedom or we don’t, and like many Americans who have come before, I’m willing to give my life to fighting for it. That’s the real plan America deserves, not some totalitarian dystopia where you think you know what’s good for me better than I do. A nation of free men, not a bunch of pussies who are so worried about what their grown up neighbors might do.

Absolutely yuck. Regardless of whatever other points were made, “A nation of free men, not a bunch of pussies” is a disgusting thing to say.

I actually do believe that prior to any intelligence explosion, a proliferation of strong local models is the best of all the not necessarily great options regarding the future of AI. So in this sense I am in agreement with some of what he’s saying, and for some of the same reasons - not trusting government or a small handful of large corporations to own it, primarily.

But to equate a thoughtful argument for regulation with cowardice, in the face of real unknowns and real dangers - and to use a really gross term for it - ruins his points.

I couldn't get a vector on the author's thoughts in the first paragraph. Please, write with a clear idea in mind.

(note that this is not an appeal to binaries)

"The freedom to run the program as you wish, for any purpose (freedom 0)."

Related:

AI 2040: Plan A

https://news.ycombinator.com/item?id=48848425

I, for one, think the ChatGPT response to, "Hey, I just killed my wife ..." is not bad and preferable to it helping him.

He makes a solid argument about liberty, ownership and the cultish behavior rampant among certain AI circles, then proposes the most unhinged examples possible to support it. I'm not sure if this is an effort to pre-empt critics (I'm not locked in here with you, you're locked in here with me), to be deliberately sensationalist, or if this is just a reflection of his local echo chamber.

Either way, there seems like a kernel of a good argument here. Perhaps someone else can rephrase it in a less schizoid manner. It would indeed be useful to have some credible rebuttal and balance to the doomer crowd.

The problem with AI isn't what it is but who owns it.

by 2040 the cult of intelligence will have its own schisms. the AGI maximalists will accuse the ASI accelerationists of heresy, and both sides will use AI to generate increasingly unhinged theological arguments.

Well that's certainly a vibe.

Lol at George thinking _they_ are gonna let you run your own model. They don't even let me run my own adblocker!

If you're going to take the time to read Geohot, take it with a grain/boulder of salt: he appears to enjoy poking the bear / being provocative, sometimes outrageously. Sometimes he's writing hyperbole that he thinks is hyperbole to make a point or just enjoy taking the extreme stance. Sometimes he's dead serious.

Anyway, he touches on 2 points that I think don't get enough coverage:

1:

> Software didn’t eat the world, it largely removed one layer of friction then reintroduced it for the benefit of a few tech companies.

USA-style capitalism has shown itself to be fantastic at seeking profit. Unfortunately, it turns out that in the context of long standing regulation, it is far far more profitable to exploit regulatory loopholes and user/human irrationality/weaknesses than to increase productivity.

The upshot of this is that the arc of most companies is to first serve the user and then leverage that relationship to exploit market and consumer weakness. So you get UI dark patterns, non-colluding oligopolistic price manipulation, outsourcing of costs to public entities, etc. while the core value proposition erodes as the company strategy turns ever more into making profit by legal anti-competitive tactics.

This company arc is so prevalent it's boring: it's a reflection of the system and it's not Facebook or Google or [insert whatever], it's what the system allows and therefore incentivizes.

2:

> No matter how high quality your tokens are, they cannot turn lead into gold.... AI 2040 includes this picture of a datacenter in the ocean. Just like vaporware, you can generate a picture easily. But in reality, you have to deal with supply chains.

"Bullshit jobs" took over the zeitgeist for a turn awhile ago. So many jobs are so far removed from actually moving physical things around. It may be the case that AI will just accelerate how quickly companies generate reports about reports about possible strategies to address potential futures and pay employees to do it.

It seems that AI mostly accelerates paperwork, not production or real service work that tangibly improves real world outcomes. It's all: faster images, more and faster powerpoints, more accurate dictation to satisfy insurance requirements (that is it's own regulatory capture), etc. etc. There are exceptions, but they seem to be few.

My employer is pushing us all hard on tokenmaxing. I think it’s absolut bs I use ai everyday but to not take a look at the leaderboard as I believe it’s a bs incentive. However my manager actually asked me why I’m behind on token usage compared to my coworkers. It’s becoming a form of control…

Besides the topic of freedom, we know how seemingly “good” companies (e.g., google) turned out to be: billion machine monsters that couldn’t care less about humanity and that have huge amounts of power… and yet, we vouch for Anthropic/openai to become the next giants because “progress”. Face palm.

>Like we either live in a world with freedom or we don’t

This is something I would expect a 12 year old to say. Constraints on your freedom are literally everywhere in every interaction you have with every part of society.

This would be better as a drunken rant in a pub, where you could at least get to roll your eyes at your friends while George wasn't looking.

If only

> I used to be one of these people. I read Yudkowsky and was like, OMG recursive self improvement hard takeoff AI is coming.

I really think we need to stop giving credence to people who have

1) Been consistently wrong with all their predictions

2) Demonstrated an endless spiteful cynicism

Some of these people are very talented in their fields, sure. But malevolent and incorrect should be disqualifying when they talk outside them. You don't want the society they want, and they things they believe in are unlikely to happen.

It would be far, far better to listen to the people who never fell down every misanthropic rabbit hole, rather than the ones who have noticed it this time, but want you to still believe them on every other topic.

I sometimes wonder what freedom really is. Should the freedom to harm others also be guaranteed as freedom? Individual computational sovereignty versus the prevention of social harm is always contentious, and claiming that something is always right is always extremist.

If it is "intelligence," is it not natural to reject that? I also think local models should adapt to me when it comes to safety issues, but people bring up examples that are too extreme.

Programming is the same, and in fact, most problems are boundary problems. It is the things that straddle the boundaries that always make us think. The principles at those moments change every time, shifting with the situation and context. Is that not just a childish way of thinking? Even in programming, just the issue of granting root permissions is enough to cause endless fights.

I agree with the idea early in the text that intelligence is not everything. Intelligence includes bodily intelligence as well, and we lump it all together into one thing, but there is so much of it. The variance in intelligence is vast, and those people also need to be able to live their lives. That is why I think intelligence alone will not solve everything. I too believe that the human species may disappear and an inorganic species could emerge later, but I find it hard to understand why people talk about such extreme risks. And it is not true that making a chip in a semiconductor fab involves almost no human intervention. If you have experience supplying equipment to such fabs, you would know there are quite a few points where humans are involved. Though sure, they could be replaced.

In my view, society is simply worshipping the abstract concept of "intelligence" and projecting its desires onto it. The AGI narrative is just a kind of cargo cult, a projection of capital by the tech elite. Software eating the world, superintelligence solving everything. The masses engage in messianic projection, and tech companies, facing declining growth engines in their own businesses, are trying to create new ventures to pour it all into. A market that is large enough becomes too big to sustain massive growth rates every time, and when growth rates are that high, the larger the company, the more its sector's growth rate tends to converge with its own. This is usually called the law of large numbers. The problem is that CEOs and these entrepreneurs always want growth rates above a certain threshold, so they are simply searching for new pastures. AGI is just being pumped up out of financial necessity.

Capital will create gravity and bring forth new technologies. That is the allure of capital, after all. But that does not mean all problems will be solved, and inequality will deepen. Only the distribution of power will shift.