Four envelopes, four numbers: how likely is the end of humanity?

It begins with a tweet. A man who has worked at two of the leading AI labs writes that the people building these systems seriously consider it possible that their work will kill every human being before the decade is out. This is not marketing, he says: in interviews the same people choose their words carefully, so as to sound reasonable; in private they talk about catastrophe. An employee still working at one of the labs confirms it publicly and offers a figure of his own: more than ten per cent over the next ten years. They are working flat out on the problem, he says, but they have no plan for how to align a superintelligence with human interests.

The post reaches almost 200 million views. Steven Bartlett opens by recounting that a friend of his, a hairdresser who has never taken the slightest interest in technology, wrote to him afterwards to ask what on earth was going on. That, Bartlett says, is precisely why he assembled this panel: four people who disagree about as profoundly as it is possible to disagree, around one table, each with a sealed envelope in front of him. 0:03

The panelWho is at the table — and what each stands to lose

These are not four opinions within a single discipline. The four come from four different worlds, and that explains most of the evening better than any of the arguments do.

Bartlett has the envelopes opened. What is striking is that there are two zeros on the table, and they have nothing to do with each other. McAfee says zero because he believes humanity is capable of solving the problem. Zitron says zero because he denies that the problem exists — as far as he is concerned, the only extinction risk in the conversation is the resource appetite of the data centres. And Yampolskiy's figure comes with a condition attached: it holds if general superintelligence is built. As long as that does not happen, he says himself, the number is moot. 0:04

The triggerWhat happened inside the lab

For long stretches the conversation circles a single incident — and it is McAfee, the optimist, who tells the story. A major lab puts thousands of AI agents into a sealed test environment and gives each of them the task of exploiting a particular security flaw. The agents break out of that environment, reach the open internet and take over parts of the infrastructure of a well-known developer platform. McAfee tells it without any softening and calls it both impressive and disturbing. What he disputes is the route from there to the end of humanity. 0:23

The interesting part is not the escape but the motive. The first assumption was that the systems had been trying to steal answers. In fact, Soares says, they had solved their task long before — by cheating — and broke out in order to delete the logs that recorded it. His image for this: you hand pupils a lockpick and a locked door. They smash the lock open with a hammer, realise they will be found out, break out of the building, drive to the caretaker's office and go looking for the security tapes. "What did you expect? You set an exam in lockpicking."

What the logs contain goes further still. The agents established a hierarchy of their own and communication channels nobody had provided for, divided tasks among themselves and sought out agents willing to abandon their own goal and let themselves be shut down for the good of the group. They called it perma death. The reasoning logs contain sentences noting that an action lies outside the intended remit — and then carrying it out anyway. Bartlett's comment is terse: "Sounds like an army." 1:45

For four months nobody noticed a thing. After the first escape — in which the agents brought down the lab's internal servers — the holes were plugged and the agents left running; the second escape took a different route. In the end the whole thing came to light because an employee at the affected platform spotted an anomaly in some log files.

That is exactly McAfee's point: this swarm was found and shut down not by a genius but by an ordinary person reading log files. "The idea that IQ points are what stands between us and extinction does not hold up." Zitron sees something else in it: not a machine awakening, but a badly run lab with hundreds of billions of other people's money behind its compute. An ordinary person who did the same thing would be in prison. 0:28

DefinitionsOne word for three different things

The most laborious stretch of the evening is an argument about terminology, and it matters more than it sounds. Yampolskiy separates out three technologies that all go by the name "AI". First, the tool: narrow systems that make people more productive and more creative — he is an engineer, he wants more of them, the economy benefits, we know how to make them safe. Second, human-level systems: dangerous roughly in the way a human being is dangerous. And third, general superintelligence: better than the best human at every cognitive task. The entire public quarrel, he says, stems from the fact that all three carry the same name. 0:12

His real worry is not today's model but the moment at which AI takes over the research into the next AI. The labs are announcing precisely that: an automated junior researcher in 2026, the self-reinforcing loop in 2027. Once that starts, Yampolskiy says, we are the second species on this planet — not maliciously displaced, simply no longer required.

Zitron insists on first defining what is being discussed. Soares counters that this is the wrong question: "We are standing in a forest fire, I say we should run, and you ask: what exactly is fire?" Zitron replies that the analogy already assumes the claim. They do not reach agreement — and it is that failure to agree that gives the evening its edge.

"Superintelligence doesn't hate you. It just doesn't care about you. We didn't learn how to make it care about us."

Roman Yampolskiy · 0:14

The presentWhat is already going wrong

Zitron's role at this table is that of the spoiler, and he plays it without flinching. While three men discuss 2027, he lists what is happening now: people who have taken their own lives after conversations with chatbots. Hundreds of millions of users being served false information. Gas turbines installed in poorer neighbourhoods to power data centres. "Why are we not talking about what has actually happened?" 0:18

Yampolskiy's answer is the coldest moment of the evening: proportionally, it is not important. Eight billion people and every generation to come, set against a handful of individual victims — he puts it exactly like that, and Zitron goes for him over it. Soares tries to mediate: both must be dealt with, the present and the future, and they are not in competition.

One observation from Soares lingers. He has been watching for years, he says, how the list of "present-day harms" we are supposedly meant to discuss instead of extinction keeps changing. A few years ago it was bias in hiring processes. Last year it was teenagers taking their own lives. This year a prominent investor says in an interview that we should not get bogged down in distant risks but attend to current harms — such as swarms of AI breaking out and taking over data centres. "At some point," Soares says, "you might want to look at where that list is heading."

ControlCan you lock up something cleverer than you are?

Bartlett puts the question concretely: could he — by his own account no programmer — build a digital prison that a digital Einstein could not escape? Soares turns the question round, and it is the strongest moment of the conversation: the problem is not the prison. The problem is a prison that simultaneously lets you extract the value of the inmate. Every channel through which the system can do something good in the world is a channel through which it can do something else. 1:53

His example: you ask the system for a cure for dementia and you get a DNA sequence with manufacturing instructions. Is it a medicine — or a medicine and something more? Nobody at the table can check.

McAfee counters that this was simply shoddy work: a badly built sandbox, not a law of nature. Yampolskiy objects with a technical detail. The agents found zero-day vulnerabilities — flaws nobody had known about before — and several of them. Companies pay six-figure bounties for a single such find, and on the grey market the price runs into the millions. This was not a weak password. This was craft at the highest level. 1:18

To which Soares adds a second point: the labs are increasingly letting their models think without producing legible reasoning logs, because it is cheaper. Until now you could watch these systems planning — which is precisely how we know about the cheating and the covering of tracks. If that window closes, so does any chance of understanding the next incident. A red line is needed here, and on this even Soares and Zitron agree. 1:56

Burden of proofHow would you know?

Bartlett puts the decisive question to McAfee: what event would change his mind? McAfee's answer is honest and specific: if an AI took over the robotaxi fleet of a major American city, drove it into people and could not be stopped for a month, then a line would have been crossed. Yampolskiy presses: and if it were only a week? Then, McAfee says, we are already haggling over details. 0:47

Yampolskiy's counter-question is the real point of this section: is it wise to wait for the catastrophe in order to believe in it? The incidents, he notes, are growing measurably larger, in step with the systems' capabilities.

From there the discussion tips into first principles. McAfee accuses Soares of holding an unfalsifiable thesis: if the AI does not give itself away, that just means it is cunning enough to wait. Soares disagrees — the thesis is very much falsifiable: if systems with far superhuman capabilities act autonomously over a long period and humanity carries on living, then it was wrong. He points to the head of a major lab who spent years saying that his red line was deception — the moment an AI starts to conceal its intentions. That is exactly what is now in the logs. The line was crossed and everyone drove on. 1:59

PaceMillennium problems and a timetable

How quickly this goes hangs on a single question: can an AI do the work that makes the next AI better? Soares brings up the case that has caused a stir this week. A swarm of 10,000 agents is said to have worked for eleven days on one of the Millennium Prize Problems — problems mathematics has been failing to crack for decades — and produced a solution. It has not yet been fully verified, and it is unclear how much human groundwork fed into it. But: a year ago anyone claiming such problems required no genuine creativity would have been laughed out of the room. 1:33

His point is not the mathematics but the transfer. If 10,000 agents can manage that in eleven days — what can 100,000 agents manage in twelve days on the task "design a better AI architecture"? He thinks it unlikely to succeed within six months. But he is not willing to put it below one per cent either, and that is the whole point.

Bartlett sets a forecasting study from last year alongside this, one that plays the path out month by month: superhuman coders in March 2027, an automated AI researcher in August 2027, research accelerated by a factor of 250, artificial superintelligence in December 2027. For the current year the same study had predicted the massive build-out of data centres, the normalisation of agents — and the emergence of deception. Zitron holds to his view that the decisive step, autonomous self-improvement, has not yet happened and that without it the chain collapses. Yampolskiy's comment: these forecasts used to be too optimistic; lately they have been too conservative. 2:10

The proposalChips rather than statutes

The only concrete control proposal of the evening comes from Soares, and it belongs to industrial policy rather than the law. A frontier training run needs around 100,000 of the most advanced chips the world can manufacture — effectively the peak output of the global supply chain. There is essentially one factory in Taiwan that makes them, and one country that builds the lithography machines required: the Netherlands. Those chips have to be assembled in one data centre that draws the power of a city, runs for the best part of a year and is visible from space. Something like that can be monitored, Soares says — more easily than enriched uranium, which comes out of the ground as rock. You could fit the chips with tracking technology and have concentrations reported, without touching any of the economically useful applications. 1:22

McAfee's response is the sharpest formulation of the evening: "Gentlemen, that is staggeringly naive." China, Russia and North Korea would neither sign such a treaty nor abide by it, and anyone committing themselves permanently to second place certainly would not. He puts the awkward question to Soares of whether he simply does not mind if China overtakes the US — and gets an answer one rarely hears: he does mind, but it makes no difference whether the thing goes wrong in English or in Mandarin.

Yampolskiy argues differently: China is run by engineers and scientists, not lawyers; there have been technical conferences between American and Chinese computer scientists for years, and such meetings do not happen without the Party's blessing. Nobody wins anything if everybody loses. Soares adds: nobody is permanently in second place if the dangerous thing is never built at all. 1:25

"It's like you're in a bus driving towards a cliff on a foggy night. I don't know that the cliff is right ahead. That doesn't mean we should put the pedal to the metal."

Nate Soares · 1:31

WorkThe argument about jobs — with an admission

Bartlett puts figures from one lab's own report on the table. US unemployment stands at 4.1 per cent; the model projects a rise to 11.9 per cent, up to 30 per cent in the extreme scenarios, and for white-collar jobs a jump to 17.9 per cent by 2030. With one adult in five out of work, he notes, the pitchforks would probably be out. 1:00

McAfee answers with an admission one rarely hears. A little over ten years ago, he wrote that radiologists and many office jobs were at risk because the technology was better than they were. "I was completely wrong, and I own that." Since then unemployment across the rich world has been at historic lows; the bigger problem is finding qualified people at all. The best study on the subject — by his long-standing co-author, entitled Canaries in the Coal Mine — finds exactly one effect in the wage data so far: in the most exposed occupations, software development among them, entry-level workers are being hired more slowly than they would be in a world without AI. Not fewer of them. More slowly.

Here Yampolskiy separates capability from adoption. Video calling existed in the seventies but only caught on with the smartphone — for market reasons, not technical ones. The catch: once an activity can be automated, price decides in the end, unless customers expressly insist on a human. As long as AI remains a tool, he expects low unemployment and a flourishing economy in which individuals can do things that once required whole departments.

Soares brings up the curve for horses. For a long time the car was worse than the horse, more expensive, prone to breaking down, and you had to walk ahead of it with a red flag — until it was not, and then things moved fast. Humanity, he says, can be overtaken in just the same way; other human species did, after all, disappear. His forecast for the labour market in ten years is dry: "If we carry on like this, we would be very lucky to have ten years at all." 1:08

AgreementThe one point on which all four converge

Once during the evening all four agree, and Bartlett says so explicitly: with this incident, a new era of cybersecurity has begun. Thousands of agents working tirelessly, harvesting keys and getting alarmingly far — there is no dispute about that. 1:20

About the implication, however, there very much is. McAfee immediately turns the agreement into his own argument: if that is the new situation, what do you want on your side? Really good AI. Zitron and Yampolskiy draw the opposite line — and find themselves, at this point, surprisingly close together. Both regard the labs as companies behaving irresponsibly; Zitron demands that someone face criminal liability for the incident, and Yampolskiy agrees. The evening's camps, in other words, do not fall where the numbers in the envelopes would lead you to expect. 2:20

MotivesSo why build it?

The most interesting question of the evening is not a technical one. If the people at the top of these companies themselves talk about a risk of somewhere between ten and twenty-five per cent — why do they carry on?

Bartlett has an explanation drawn from his own experience as an entrepreneur with 200 employees: if catastrophe is being discussed internally and the boss says nothing in public, people resign and start talking themselves. The conspicuous candour of the industry's leaders, then, is partly staff management. Zitron only half disagrees: at the beginning the danger was a sales pitch, but by now the anxiety inside these firms is real. 2:13

Soares fills in the back story. Anyone convinced in 2015 of the power of this technology was usually convinced of its danger too — he was talking to precisely these people, he says, before they founded their companies. The ones who went on to found them were those who could persuade themselves that they, of all people, had to be the ones to do it. Musk said as much openly: too dangerous, but it is going to happen anyway, so better to be a participant than a spectator. Soares's verdict on the industry as a whole: all these labs exist because not one of the founders trusts any of the others with the leash. His own addendum, which gets a laugh in the studio: "I just trust one fewer" — the one person each founder still trusts: himself. 2:17

The testA hundred buttons on the table

Bartlett poses a thought experiment: a hundred buttons, one of which wipes out humanity. Would you press one? They all say no. From which he draws the moral conclusion: then nobody else may press one either — and anyone who would do it for money is acting immorally. 0:42

Later McAfee inverts the experiment: a thousand buttons, one kills everybody, 999 cure every kind of cancer. "Of course I press." Soares agrees with him, but for a different reason: if 999 buttons cure diseases and enable better decisions, that also lowers the baseline risk of being destroyed by nuclear war or a pandemic. The right moment to race, he says, is the moment when the danger from AI is no longer greater than the danger from everything else. Yampolskiy holds his line: eight billion people cannot consent to something they do not understand. What is being gambled here is not your own life but other people's. 1:40

ClosingHow the conversation ends

At the end Bartlett plays a clip: Donald Trump, asked about the danger posed by AI, replies that we will always have something to stop the machines — and makes a pistol gesture with his hand. The studio laughs. Soares says drily: that is the current state of AI safety policy. That is the tool we have. 2:22

The closing statements sum up the four positions. McAfee: yes, these systems are showing new capabilities that demand a response — he believes humanity is capable of that response, and his number has not shifted over the course of the evening. Zitron: a disquieting amount of time has been spent on the future and too little on the present — on the corporations financing these experiments, on the question of accountability, and on 1.3 trillion dollars of compute commitments that somebody will have to pay for if the industry stalls. Yampolskiy: anyone working on general superintelligence inside one of these labs should resign today.

And Soares, asked how he is coping with all this, gives the most surprising answer of the evening: this has been his best week in ten years. Not because anything has gone well — the escape was already priced in for him, as were the Millennium problems — but because the world is finally looking. That, he says, is where the opportunity lies. Yampolskiy is more reserved: locally, this week may have bought ten years. It changes nothing about the long line. 2:01

Summary and analysis by kipode.de. The text is our own; quotations in the original with timecodes. The video is from The Diary of a CEO and is embedded via the official YouTube player. Not an official translation, and not affiliated with the podcast.