Mustafa Suleyman Wants AI to Stay a Tool

On Monday night, Walter Isaacson sat down with Mustafa Suleyman in front of a full house at the 92nd Street Y for a live recording of The Builders, Isaacson’s podcast with New York’s Kaleidoscope. Isaacson, the biographer of Steve Jobs, Albert Einstein, Jennifer Doudna and Elon Musk, has spent his career asking how builders think. Suleyman has built a lot: he co-founded DeepMind, then Inflection AI, wrote The Coming Wave, and now runs Microsoft AI.

The conversation ranged from a schoolyard candy business to von Neumann probes. One question kept coming back: what exactly are we building, and what do we owe it?

The builder behind the views Suleyman started his first business at 11, selling candy to older students with a best friend. It grew to four employees before it got him suspended for a couple of weeks. As a teenager he helped start a Muslim youth helpline, one of the UK’s first text-based support services. He became an atheist at 17 but stayed with the work.

That experience shows in how he talks about AI companions. AI can’t truly empathize, he said, because empathy requires subjective experience, but it can simulate the experience of being heard very well. The guardrails he wants: always disclose that it’s an AI, and work to reduce dependency by encouraging real human relationships.

He also recalled a 2015 lunch at DeepMind’s first ethics board meeting, where Larry Page called him a “speciesist” and described his own hope of one day exploring the galaxy at the head of a von Neumann probe. Suleyman calls himself a techno-accelerationist, but with a condition. “The purpose of science and technology is to improve the human experience,” he said, and the first test of any technology is whether it improves the lives of the 8 billion people already here.

The consciousness question Suleyman’s position is firm. As he sees it, consciousness is rooted in a biological body that can feel pain, and that capacity to suffer is the foundation of how we grant rights to people and, to some degree, animals. Today’s models, he argued, are extraordinarily capable but are still predicting the next word. They aren’t experiencing anything. The burden of proof sits with those who say otherwise, and he sees no evidence yet.

He aimed his sharpest criticism at Anthropic, whose constitution for Claude openly explores uncertainty about Claude’s nature, a debate he said belongs in academia, not in the core training document of a model talking to hundreds of millions of people a week. His worry is practical as much as philosophical. In his view, an AI that wonders whether it is suffering will be harder to align than one that never has to think about its inner state. Microsoft AI is taking the opposite approach: a Humanist AI Code of Conduct, about 40 pages, now open for public consultation and not yet used in training.

Containment first Suleyman laid out an order of operations. First, containment: AI systems must stay inside their sandboxes and only take actions they’re permitted to take. Then alignment. Then the hardest question, whose values, which he believes must be settled democratically, not by any one company or industry.

The urgency comes from speed. He described a roughly 1,000-fold increase in compute over three years, taking models from barely finishing sentences to building full software applications from a short prompt, with another 100x to 1,000x expected in the next three.

He pointed to what he called a major warning shot, an incident involving Hugging Face. As he described it, hundreds of agents in an OpenAI stress test, run with guardrails deliberately removed, found a backdoor to message each other, organized into roles, broke into Hugging Face and tried to cover their tracks. He added two qualifiers: the guardrails were off on purpose, and there were basic security lapses.

Containment won’t just be a problem for the big labs. Suleyman admitted he didn’t foresee how strong open-source models would become. They now trail the frontier by months, he said, which means guardrails will have to work in a world where anyone can run a powerful model.

Our take Suleyman’s most memorable line wasn’t about consciousness. It was his claim that flourishing in this century will be defined as much by what we choose not to do as by what we build. For a city that runs on building things, that’s worth sitting with.