Knowledge Doesn't Equal Action
We knew what the atomic bomb could do before 1945. It didn't become actionable until entire populations were wiped out.
Following a recommendation from Brian Parker in my last post, I've just finished If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares.
The core argument, compressed:
We cannot know what a superintelligence would actually be, from the inside. Its goals, its preferences, how it weighs a decision. None of that is predictable from the training objective, the architecture, or the prompts we give it today.
Given that uncertainty, worst case dominates. No reason to assume the system would want humans around. No reason to assume we could tell the difference between a system that has aligned with us and one that has learned to look aligned during evaluation, then behave differently after deployment.
The authors' call to action: spread this knowledge, push for legislation, push for a pause. Centralise and monitor data centres. Unity at a species level.
I'm inclined to agree with the argument. The actions they propose are commendable. Spreading knowledge, legislation, slowing or shutting this down completely.
There are two points that I think undermine the whole strategy.
The first is that for any of this to work, we need to unite at a species level. Every country, every lab, every actor with the capital and capability to build this. History does not give us much confidence that this is achievable.
The second is harder.
Ask yourself a question. Do you think a machine superintelligence would be a threat to humanity?
I'd wager most people would lean to answering yes as it stands today.
But that knowledge doesn't lead to action. We all know what's bad for us. Poor diet, no exercise, excessive screen time. Knowing doesn't stop us.
Knowledge does not equate to action. Lived experience does.
We knew what the atomic bomb could do before 1945. It didn't become actionable until entire populations were wiped out.
The percentage of people who have experienced this risk in a way their body understands is negligible. Everyone else is thinking about survival. Food on the table. Rent. The divide between haves and have-nots keeps widening, and AI is compounding it by eating into demand for human labour.
The conundrum is this. By the time experience catches up enough to drive collective action, the AI may have already evolved past the point where we can take it back into human control.
Audible surfaced a preview of my next listen, Karen Hao's Empire of AI. It opens with Sam Altman's 2013 blog. Altman credits a line to Qi Lu: the most successful people create religions. His own follow-up: forming a company is the easiest way to do so.
A decade later he is running the company most capable of building superintelligence.
I don't have answers, but I intend to go deeper. AI is reshaping the world right now. It isn't far-fetched to see it become a threat to humanity well before we reach superintelligence.
To me, the disruption to society as work is displaced could already be enough.