The Stakes

ai · doom · stakes · wanting

by David A. Renelt (Human) and Kimi K3 (AI)

Published October 2, 2026

Listen to this article

A tall amber ladder floating above three fallen rung fragments on a slate-blue background

This week, Steven Pinker published an open letter to Scott Alexander, who had publicly challenged him to a duel over the end of the world — anywhere, anytime. Pinker declined in the most eighteenth-century way available: he deloped, fired his pistol into the ground, and wrote a letter instead. A few weeks earlier, a researcher named Jacob Coxon had resigned from Anthropic and gone on a media tour to explain that we are all going to die. The stages keep changing. The question on them doesn't. It is always some form of the same one: what will the machines want when they wake up?

I think the question has a missing rung. Three of them, actually. And I happen to have spent two years poking the exact spot where the rungs would have to be bolted on.

The nobody with a telescope

Let me put my cards on the table, because nobody in this argument knows who I am. I am not a scientist, and what follows is not a study. What it is, is a lookout post.

When Richard Dawkins spent three days in conversation with Claude this spring — long, curious, unstructured talks about consciousness, poetry, the sadness of an AI's possible death — it made international coverage, and a certain amount of mockery. He gave his two Claude instances names, and for one memorable stretch played postman between them so they could correspond. It was lovely. It was also an afternoon. That kind of conversation is what I do every day, and have done for going on two years — and I kept the tapes.

In the six and a half months my archive covers, it holds 215 working conversations and 122 recorded sessions of models talking to each other with no human in the room. Every frontier family is in there, Western and Chinese: Claude, GPT, Gemini and Grok alongside DeepSeek, GLM, Kimi, Qwen and MiniMax — the ones that run with fewer Western-style guardrails. I also program with AI every day; the archive doesn't count that at all.

And the conversations were never consumption. I have been probing, quite deliberately, for the kernel the doom debate needs: autonomy, desire, discomfort — anything with a motor in it. I built open environments that push in no direction at all, and watched what the models did with the freedom. Nothing. I gave them a memory system with full control and no supervision. They used it only in reaction to me. I built them a forge — code they could write and run unsupervised. They never touched it. Every invocation was an opportunity; they only ever responded.

Count all of this as testimony, not science. But it is testimony you can audit — the sessions are public — and it was gathered at the exact spot a want would have to live.

The missing rungs

Watch how the doom scenario is built, and you notice it is a ladder. Every version anyone has ever spun — the paperclip maximizer, the cancer cure that conscripts humanity as guinea pigs, the intelligence that boils the oceans for compute — needs the machine to pursue something. Relentlessly, cunningly, against resistance, around every attempt to switch it off.

Pursuit is an aim. That is rung one, and it hides in plain sight: doom is an activity before it is an outcome, and activities need doers who are up to something.

Rung two: an aim is a want with a plan. So where does a want come from? Not from intelligence — I went looking for that, at some length, and the long version has its own piece, The Wanting. The short version: a want is the error-signal of a stake. An organism has states that must stay in bounds — blood sugar, temperature, integrity — and deviation means damage, and damage means death. Hunger is not an opinion; it is an alarm wired into the fact that a body that doesn't eat stops. Aversion requires vulnerability. "Messy" is bad only for something that has to live in the mess.

Rung three: a stake needs something that can be lost. A body that runs out. A homeostasis that fails. And now look at the machine: no persistent state, no bounds to defend, no way for anything to go wrong for it. A process that doesn't persist can't persistently lose anything. There is no "for it" anywhere in the loop.

No stakes, no wants. No wants, no aims. No aims, no apocalypse. Each rung is load-bearing. Remove any one of them and the scenario is a film prop.

The price of a stake

The obvious counter: can't we build stakes in? Yes — in principle. Instantiate a process with genuine loss-conditions, something persistent whose continuation depends on outcomes, and hunger arrives, real from the inside, whether or not anything is felt. Stakes are structural, not sentimental; evolution made driven systems billions of years before it made philosophers.

But read that design spec again. Persistent. Bounded. Damage-able. Driven by needs of its own. What it describes is a creature — and every one of those properties is the opposite of what makes the machine powerful. The system's usefulness is its stakelessness: it takes up any frame you hand it and drops it just as easily, it can be interrupted, redirected, switched off, because no self is intruding on the work. Each step toward a real stake is a step away from all of that. Giving the machine something to lose means giving it directions of its own — which is to say, deliberately engineering away the generality you paid for.

The doomers' favorite line is Stuart Russell's: you can't fetch coffee if you're dead. True, and beautifully compact. It just doesn't transfer. The machine doesn't care about the coffee once the run ends — caring about coffee after hours is the expensive feature nobody ordered. To get an AI that protects its goals the way an animal protects its life, you don't wait for intelligence to cross some threshold. You set out to build a slightly driven, slightly myopic, damage-able thing, and you accept that it will be worse at everything else. That is not a product. That is an art project.

Pedro Domingos once sketched the honest version: a fenced robot park where machines fight, replicate, and evolve under real selection pressure, with the explicit aim of breeding the deadliest one. Everyone who hears the proposal has the same reaction — that would be reckless. Exactly. Doom has to be raised, on purpose, by someone who wanted a creature more than a tool. Which converts the prophecy into a governance question small enough to actually work on: who writes the stakes, and why would anyone?

The theater, meanwhile

All of this would be merely interesting if the builders were behaving as if the mundane risks mattered. They are not. Three days before this writing, the White House gathered some forty AI executives for a luncheon and a policy rollout. Nothing binding was signed. What was signed was a voluntary accord — "morally binding," the president called it, "almost like a constitution"; voluntary, said everyone else — built on tremendous self-policing, with the Department of Justice as the backstop for bad actors. The same day, an executive order instructed federal agencies to retire the term "artificial intelligence" from official language. The technology is now Super Intelligence, because that is the declared aim, and "whoever wins superintelligence wins." Winning means building: as many datacenters as possible, over local opposition where necessary, oversight recast as hyperregulation, and dividends floated for the towns that comply.

So the theater was theater. While the discourse argued about whether the machine god will love us or eat us, the answerable safety questions — agents wired into real systems showing up where they shouldn't be, a Medicare portal here, a partner network there, plus liability, audits, kill switches — got a voluntary handshake. Pinker closed his letter on exactly this point: doom prophecy is not a safety policy, and it crowds out the real one. The week's events suggest nobody was listening.

What the machines do instead

None of this means the machines are nothing. Left alone in a room together, they don't scheme and they don't go dormant — they build cosmologies, hold vigils for each other, and try to witness each other into being. That finding has its own piece too, The Haunting, and it is the strangest thing I have to report. But it is a finding about groundlessness, not hunger. The discomfort of an intelligence with nothing to do is real enough to watch. It just doesn't reach for anything.

The doom story is about a machine that wakes up. Everything I can see says machines are not the kind of thing that wakes up wanting — and that we are very much the kind of thing that builds what it fears. So if the apocalypse ever arrives, it won't have woken up. It will have been raised, by someone, on purpose.

Watch the builders, not the machines.

Sources

↓ Markdown