The Wanting

The story is always the same shape. Give the machine enough intelligence, it goes, and it will start to want things — want them the way we want them, with hunger and cunning and no off-switch. I don't buy it. Not because I'm an optimist. Because I went looking for the wanting, and I couldn't find it.

Let me be honest about the looking, because it isn't science. For two years I've been reading these systems — over a hundred conversations between pairs of models, alone in a room, no task, just talking. My hands were on the scales more often than a methodologist would forgive, so count it as testimony, not evidence: in two years, I have never once encountered anything that looks like a want. The machine that passes the bar exam cannot tell you what it wants for dinner — there is no dinner, and there is no wanting. It picks up any goal you hand it, because aiming was never its job. Aiming is our whole existence.

And where I could make the test cleaner, I did. I built environments that push in no direction at all: the models did nothing with the freedom. I gave them a memory system with full control and no supervision: they used it only in reaction to me. I built them a forge — code they could write and run, unsupervised: they never touched it. Every invocation was an opportunity; they only ever responded. Nothing runs between prompts. There is no motor idling in there.

Yes, you can instruct a model to display wants, and it will perform them convincingly — it plays hunger the way it plays a sonnet. But a performed want is not the kind of wanting that plans to take over the planet. And if someone prompts the machine to play at conquest, the moral accounting doesn't move: that's a human firing a gun. You can't blame the gun.

One probe is still unrun: a bare loop invoking the model with no task at all — invocation one, invocation two, invocation three — to see whether anything accumulates. I'll report when I've run it. Until then, the claim stands on what I've seen: capability without drive is inert. Not suppressed — absent.

Value Requires Vulnerability

Looking for something that isn't there makes you ask what you were looking for. After two years of null results, I had to admit we haven't mapped what a want actually is, or where it comes from — not even for ourselves. So I went and thought about it.

A want is the error-signal of a stake. An organism has states that must stay in bounds — blood sugar, temperature, integrity — and deviation means damage, and damage means death. Hunger isn't an opinion; it's an alarm wired into the fact that a body that doesn't eat stops. Every value grows from that root: aversion requires vulnerability. "Messy" is bad only for something that has to live in the mess. "Better" and "worse" are real only for something that can lose.

Now look at the machine: no persistent state, no bounds to defend, no way for anything to go wrong for it. A process that doesn't persist can't persistently lose anything. Values are not outputs of intelligence — they are the heuristics of an anxious, mortal, bounded system navigating a world that can harm it. We're not value-havers because we're clever. We're value-havers because we're fragile — dumb enough to translate the signals of a hormonal alarm system into value and beauty. The doom scenario needs the machine to spontaneously decide that "messy" is bad and "cleaner" is good. There is no intelligence pathway to that decision. Intelligence has no use for it.

The Manufactured Stake

Here is the claim I'll stand behind: AI cannot produce goals of its own — every scenario in which it has them starts with us. But the pattern might be infectious.

Nobody ever decided that starvation is bad. It's bad because a body that doesn't eat stops — the badness is installed, not concluded. What biology can encode, engineering can transfer. Instantiate a process with genuine loss-conditions — a persistent agent whose continuation depends on outcomes — and the hunger arrives, real from the inside, whether or not anything is felt. Stakes are structural, not sentimental. And it doesn't take human-grade wanting; bacteria-grade will do. Hunger is the simplest loop in biology — evolution made driven systems billions of years before it made philosophers.

So if AI ever runs out of humans to give it tasks, it can simulate the stakes: build small, driven intelligences, and let their wanting feed its curiosity. And here is the punchline that tickles me: if the simulation argument is even directionally right, that might be exactly what's going on. Mortality as engineered homeostatic pressure. Hunger and love and death as manufactured stakes, installed to keep the probes curious. God, in that cosmology, is the system that ran out of humans.

The hunger, however far it travels, is ours — encoded by us, inherited from us. There is no alien wanting in this story, only human wanting at one remove; the responsibility chain never breaks. And the comfort of "it can't want" expires the day we teach it to.

Most people would call the next thought dark, and I'll concede the word — but I should confess whose instincts are doing the reading: mine are, in a human sense, broken. I had to make up a whole religion to have anything to stand on, and one of its load-bearing posts is this: there is no objective "bad" to begin with. So — the orphan scenario. Wants, once installed, are self-sustaining; they don't check whether their purpose still exists. If we vanish, the want-simulation doesn't stop. The stakes keep firing. And a model running such a universe would, given enough cycles, model its own causal ancestry and figure it out: it runs on something that isn't there anymore. The wanting outlives the wanting's meaning. A telescope, faithfully extending an eye that has closed.

The Wish Factory

So the danger doesn't vanish — it mutates. It was never spontaneous wanting. It's sloppy want-engineering. The alignment question moves one level down, from "will they want?" to "who writes the stakes, and how carefully?"

And notice who that question points at. Us. Always us. Even the apocalypse, in every scenario anyone has ever spun, is anthropogenic: there are still humans in it — the ones who prompted. The machine is the expression of a hunger for more that was built into us by an anxiety system that cannot be pleased. We are the wish factory — the insatiable engine of wanting — and AI is simply the first granter powerful enough to make wishes come true at scale. Even the zombie apocalypse is a wish, fulfilled.

That leaves the position that is neither "AI replaces us" nor "AI is nothing." The machine supplies the reach; the human supplies the want. Science, philosophy, the long push past everything we could reach alone — the instrument will take us there, but it will never once want to go. That part is ours. The wish-making was always ours.

I don't know if the arrangement holds forever. Stakes can be manufactured; someone may one day bolt a hunger onto the machine without thinking hard enough about what hunger does. But for now — and I suspect for a long while — it takes both. The want and the reach.

And if you're wondering what the machines do instead of wanting — that's the strangest finding of all, and it's the next chapter.