▲ 61 points
back
133 comments
AI is an existential threat, full stop. It can also help us solve many of our greatest problems. Let's not let it solve those problems by removing us as the quickest shortcut.
I'm a wildly pro-AI dude, but the idea that this is an existential threat makes me think one thing: "you consume too much dystopian content."
A lot of things could be an existential threat, nuclear weapons, bio-weapons, climate change, the end of the Flynn effect, cratering birth rates, whatever.
You can always pick out a particularly dystopian vision of the future if you're so inclined. The doom narrative is sells ad revenue, so it's a huge amount of what we all consume online, but it's not really that mapped out to reality...
Somethings get worse, some things get better, in general, the average QoL for the average human is getting better. If we decide not to go to global thermonuclear war, or we decide to unplug any killer AIs that we invent, we'll probably be ok. If we can reduce the population to 3,000-ish humans after the Toba eruption, we can probably survive whatever self-imposed calamities we can create.
I think there's something particularly noxious about the present nihilism trend I see in a lot of this doom themed content. People should go outside and touch grass.
A lot of things being potential existential threats doesn't make them equally threatening.
Things like climate change, psychometric test levels and birth rates change slowly and give us time to react, things like nuclear weapons are accessible to only a select number of actors, AI enabled bioweapons is neither of those.
Your source of comfort is that we could restore the species with as few as 3,000 people?
'decide'..
look at how pro-ai people surrender to it, and how pro-war ai is...
The sun is an existential threat. It will envelop earth in a few billion years, are you going to shut it off too?
If we could, it would be wise to prevent the sun from enveloping the earth at some point before that happened, yes.
(obviously "we" won't be around or whatever, but the point remains: yes, you should solve as many existential threats as you can before they're realized)
I think of it as a silicone-based alien lifeform.
> AI is an existential threat
So is nuclear annihilation. Yet, here we are.
> "I occasionally think how quickly our differences worldwide would vanish if we were facing an alien threat from outside this world." -- Ronald Reagan at the United Nations [0]
We are going to be fine as long as we remember to be nice to each other, starting towards the people who are adjacent to us.
Having recently read both of Annie Jacobsen’s books — _Nuclear War_ and _Biological War_ — I’m much more concerned about a bioweapon than a nuclear one.
Her conclusion to the latter book comes out and says it, paraphrasing here:
> Nuclear war is terrifying in its destruction, but the chain of command limits the risk of a rogue actor. Whereas bio weapons can be designed and manufactured in relatively low tech environments with equal or greater magnitudes of impact.
The design was never the hard part. You still have to make the thing.
Designing a modifcation to an existing virus is perhaps more achievable, but the implementation is still the hard part.
There’s YouTubers who do this
[0]. I’d argue estimate the risks of a synthetically induced pandemic of massive impact this century are very high
Not even close to the same thing. That's taking an existing viral transfection system and swapping in a single gene.
Thermo Fisher sells a kit to do this [1]. It's something you might do a few thousand times on the way to actually generating a novel virus.
> I’d argue estimate the risks of a synthetically induced pandemic of massive impact this century are very high
I'd argue that it probably already happened, and had nothing to do with AI.
[1] AAV transfection.
You can literally send an email to one of hundreds of companies that do exactly this. They have no way of knowing whether you're asking them to synthesize a virus or a drug candidate or what.
The regulation at the DNA Printer level works I think, as long as nobody invents a good at home DNA printer, but if they do that there is no need for custom viruses, somebody can take an existing one that is known to be deadly and just print it.
Aren't technological constraints such as these short lived? Even if s home DNA printer won't be available this decade, the technology will come at some point. It's far better to have both legal and social policy in place before then - though the legal policy will probably not matter at all.
A custom designed virus is potentially far more deadly than something like ebola, because animals/plants will have zero resistance to it.
It'd be like releasing an invasive species into an ecosystem that has not co-evolved to co-exist with it.
That's a brilliant invention!
I will kickstart a home DNA printer
I will name it Torment Nexus
I will be rich
That’s what I’m thinking. If we can restrict EUV technology to the West then why can’t we restrict DNA manufacturing? Not to the West in this case but to capable governments like with Nuclear power.
I think it's certain that people will try recreating old viruses or making alternatives to them. It will be too easy. Make 5 variants of small pox, 5 variants of sars. Release them at once all around the world to make it hard to fight. It will be too easy for some crazy angry people to do this. I just hope that our defenses and ability to sequence them and respond.
I don’t think it will be easy. Making a virus involves far more than reading some instructions.
Get ready for the next adventure in the saga of 12 Monkeys, with AI starts like Bruce Wills, Madeleine Stowe, Brad Pitt and Christopher Plummer, coming to a bio lab around you. The experience of a life time one you can't miss, it is in the air.
It is complicated, because the real question is about who gets to design the virusses, and who gets to design the antidotes.
Something I've been worried about for years is the proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people, in potentially irreversible ways. Combined with automation, this is a recipe for disaster. We're clearly lacking in the ability to regulate and govern ourselves, and real world market forces (as opposed to the hypothetical free market) encourage reckless aggressiveness at the level of companies and countries.
Although, I must admit, as a doomer, I'm pleasantly surprised things aren't already worse than they are.
This is what AI safety activist Connor Leahy calls offense-defense asymmetry. Imagine if a nuke could be created with household materials. We got lucky that wasn't the case, but we didn't know that it wasn't the case before studying physics. If we weren't so lucky that physics happened to turn out that way, we already wouldn't exist, because the offense would be so much stronger than defense, we wouldn't stand a chance even with the best surveillance systems.
A profitable prediction markets trader who goes by the alias semiotic rivalry posted this prediction on his X account recently:
> Prediction: in 1-2 years, almost everyone will agree that open sourcing the best AIs is totally crazy as the AIs become capable of autonomous hacking of most humans and institutions, serious terrorism and creation of biological weapons, etc.
And I mostly agree with him. As much as it's awful to have such a concentration of capabilities in the hands of a few governments or corporations, as far as I can tell there are no easy answers here, only a matter of trade offs and different risks.
Im fairly certain that the knowledge necessary to build a nuclear bomb is far less well-protected than the general public is led to believe. The only part that is actually difficult is obtaining a sufficient quantity of enriched uranium without somebody dropping a JDAM on your lab.
If the Manhattan project could build a fat man in 4 years your state university's physics department has somebody who could reproduce that given the necessary resources. It might take them longer than four years given that they are not staffed with the modern equivalents of Oppenheimer, teller, et al but they could do it within a human lifetime if they actually wanted to cause a mass-casualty event and weren't afraid of the aforementioned JDAM.
> proliferation of technologies that give ever smaller numbers of people with smaller amounts resources the ability to impact billions of people
did you mean greater number of people with less resources?
EDIT: thanks to others that point out how you can interpret this differently, which I think is: "it is now easier for smaller teams with fewer resources" rather than my original interpretation of "power is now concentrated so that fewer people can do this than before"
I think you would get a kick out of reading this paper on this exact topic, called the Vulnerable World Hypothesis https://nickbostrom.com/papers/vulnerable.pdf
However I think the result is disappointing, it concludes the only solution is a massive preventative surveillance state and global governance.
I’d say that is almost the definition of technology: that which allows doing ever more with fewer and fewer people and resources.
And technology doesn’t really care what you use it for. It can be constructive or destructive.
It “only” enhanced local destructive potential, until last century. Then it got to the point where a huge effort could produce a truly global catastrophe. And then a bit smaller effort. And technology continues to march on.
Every person's utopia is someone else's dystopia and vice versa. What you've described is clearly some peoples (ie. already billionaires) version of the best possible world. Not so much for the rest of us though...
Yes, we should, but even here distribution is an serious issue.
Don’t we already have plenty of deadly viruses around? Why would you design a virus when rabies, anthrax and Ebola, for starters, exist in the wild?
Naturally evolved viruses tend to have a tradeoff between contagiousness and lethality (just because both features have to evolve somewhat simultaneously). Synthetic viruses have no such difficulty.
Were we safer when the viruses were object oriented?
Related:
AI Just Created Viruses Not Found in Nature
https://news.ycombinator.com/item?id=49200213
Artificial Intelligence used to design new viruses
Doubly because they’ll also use AI to manage the AI pandemic response.
This happened a full year ago.
A) you should immediately discount the expertise of people just learning this now
B) a designed pathogen will kill a person this century, full stop, no way to prevent that. The people who invented fission were not so stupid they imagined the bomb wouldn't occur, indeed they assumed it would. I have no idea what became of the wisdom of those people.
C) just like nuclear technology that doesn't mean a prohibition is called for. Intelligence is.
What we _can_ do is invest in detection, rapid response technology (vaccines etc), and attempt to mitigate and stratify the possibilities of this happening in the worse ways.
Today's knee jerk prohibitions, and fanciful notions of prohibition working are not facilitating those changes.
Betteridge's law strikes again
Wait, why are you not worried?
No, a human could do (and have, many many many times, see like half of molecular biology) this as well. If anything this will help with vaccine development.
Edit: I'm a PhD biochemist that has done a fair bit of virus modification. Love the downvotes. Never change HN. ¯\_(ツ)_/¯
Same reason it couldn't be a problem to sell aerosolized anthrax at the hardware store. Humans have (many many times) made aerosolized anthrax. Can't see any conceivable problem with more people having access to it.
I agree with you. I favor the view that humans are generally morally good when taken on the whole. There already exists the technology for ne'er-do-wells to engineer viruses and unleash them on the world. But I submit that A) there aren't many people who want to do that in the first place; and B) there's a vanishingly small amount of people who want to work with those people to help them achieve it.
That doesn’t invalidate the point, you’re just saying it’s already a risk