If there’s even a 0.01% chance of self-improving AI taking over humanity, all other concerns are secondary
The only question that matters is whether it can get away from us.
OpenAI was founded after Elon thought Google wouldn’t take safety seriously. Anthropic was founded because Dario et al. didn’t trust OpenAI to take it seriously.
Now they are both racing to create superintelligence with little regard to safety, with the justification that China will create it first and take America’s global hegemony.
If OpenAI and Anthropic build something they think is dangerous because “China will do it anyway,” they need to be extremely sure China will do that—create a recursively self-improving system—not just keep training models.
This is even more baffling because the labs (looking at you, Anthropic) keep saying that China is distilling or copying their models. In other words, China is keeping up by copying, not overtaking.
If that’s true, a slowdown at Anthropic/OpenAI would slow China down too. So:
- Why on earth are we using “China will do it anyway” as the justification?
- Even if we don’t trust Beijing, why does that distrust outrank the existence of humanity? China’s posture to AGI to date has arguably been more conservative than the US’.
If our technological progress continues the way it’s going, in the next 30 years, we will soon have industry, energy, and settlement move off-world, and Washington versus Beijing in relative positions will be a relic of the past. Great power competition still matters, but it does not outrank keeping the first self-improving systems from destroying the option value for the rest of humanity.
To those saying “show me the mechanism of how AI gets out of hand,” the control/intelligence explosion problem has been written about for almost 60 years, starting with Norbert Wiener in 1960 and I.J. Good in 1965.
To the frontier labs (Anthropic/OpenAI) fighting for economic supremacy, realise that, in its current state, AI is already going to be economically transformative. You will capture enormous rents without crossing self-improving superintelligence. Take a breath, pause, and realise your decisions are determining the future of humanity.
Every person alive today and everyone who will ever be should have the same first incentive: that humanity continues with prosperity and agency intact.
Blaise Pascal argues that you should live as if God exists. If you’re wrong, you lose a few finite pleasures. If you’re right, you gain eternity and avoid going to hell.
When the downside of an action is catastrophic and irreversible, and the cost of caution is small by comparison, you do not wait for certainty.
We do not have to go far back in history to find a time—COVID-19—where the world neglected warning signs, and the people who waited for proof ended up at the mercy of whatever happened around them.
The risk of superintelligence that we cannot control is that same risk magnified to infinity. We are not wagering with only millions of lives. We’re wagering with everyone’s, forever.
If we slow down and the danger was overstated, we only lose time and revenue. If we race ahead and it wasn’t overstated, we will not get a second attempt.
The world will little note, nor long remember what we say here, but it can never forget what @hilbertspaess did here.