Doom Debates!

Bestselling author Robert Wright and I discuss this week’s stock market jitters, Google’s new interactive world model, and why I sometimes spend $1,000 in a day on AI agents. Plus:

The Binance hack;

An AI audit of two star economists;

Anthropic’s financial prospects;

And whether AI could crack the math behind encryption.

Enjoy another edition of the Nonzero x Doom Debates collab!

Watch on YouTube: https://www.youtube.com/watch?v=wkKpPeMVVX8

Timestamps

00:00:00 — Teaser

00:00:50 — Nonzero Doom returns

00:03:07 — A foom in slow motion

00:07:47 — Is AI winter canceled forever?

00:10:33 — Will Google’s Genie 4 lure us into the Matrix?

00:16:54 — Liron’s bull case for Google stock

00:24:34 — Does Google have big-company disease?

00:28:28 — Could a slowdown actually help Anthropic and OpenAI?

00:30:12 — Was OpenAI’s “pause” just PR?

00:40:01 — Social science judgment day is coming

00:47:12 — Why even AI’s upsides could tear society apart

00:51:12 — Does OpenAI’s alignment approach improve on Asimov?

00:55:06 — Sam and Satya pivot on alignment

00:58:34 — Crypto and the AI hacking threat

01:03:42 — Could AI crack the math behind encryption?

01:08:52 — Liron’s $1,000-a-day AI coding habit

01:15:23 — Explaining the missing productivity boost

01:17:26 — Does Anthropic’s S-1 sink the AI bubble thesis?

Links

ROBERT WRIGHT & NONZERO

Nonzero’s post for this episode — “Is an AI Bubble Bursting?”

Subscribe to The NonZero Newsletter

The God Test — Bob’s 2026 book on AI

Join NonZero’s Discord server

Nonzero on X (@NonzeroPods)

THE WEEK IN AI

“Anthropic files S-1 with the SEC for proposed initial public offering” — the IPO filing Liron calls the final nail in the coffin of the Ed Zitron camp

Where’s Your Ed At — Ed Zitron’s newsletter, home of the AI-bubble bear case

“Introducing Genie 4” — Google DeepMind’s real-time interactive world model

“Gemini 4 Argon: Built to solve your hardest problems” — Google’s new frontier training run, the “lukewarm” release

“Gemini 4 Carbon: our next frontier model” — the follow-up Google announced two weeks later

“Google Gemini co-lead Noam Shazeer leaves for OpenAI” — CNBC on the talent Google has bled

“Errors in the Research of Emmanuel Saez and Gabriel Zucman” — Tyler Cowen’s Marginal Revolution post on the AI audit: 138 papers, 2,035 problems

Boaz Barak: “We organized the Model Spec around five core values” — the OpenAI alignment update Liron compares to Asimov’s three laws

“An Updated Model Spec for More Capable and Beneficial AI” — OpenAI’s announcement

“Sam Altman says AI means we can no longer just patch all the bugs” — TNW on the Stanford GSB talk Liron quotes on alignment and pacing

“Satya Nadella says he was wrong to think AI alignment would come with scale” — TNW on his 20VC interview: “I used to always think that alignment will just fall out. I no longer believe that.”

“OpenAI Takes Initial Steps To Address Its Alignment Problems” — Zvi Mowshowitz on OpenAI’s training “pause”

2026 Binance hack — Wikipedia’s summary of the theft Bob and Liron debate

“$2 Million+ Drained From Polymarket Traders in New Supply Chain Attack” — SecurityWeek on the hack Liron walks through

“PHP blames supply chain security biz Chainguard for putting huge chunks of the web at risk” — The Register on “the PHP thing last week”

ALSO MENTIONED

The Hanson-Yudkowsky AI-Foom Debate — LessWrong wiki on the 2008 debate where “AI Go Foom” originated

“The AI Revolution: Our Immortality or Extinction” — Tim Urban’s Wait But Why post, where the intelligence train blows right past the station

PauseAI US — Holly Elmore’s organization, plugged at the close of the episode

DOOM DEBATES

Doom Debates on YouTube

Join the Doom Debates Discord

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at http://DoomDebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Cybersecurity expert and entrepreneur Zack Korman is an increasingly prominent nondoomer voice on Twitter. Naturally, with AI now breaking out of sandboxes and hacking corporate systems, people want to hear what cybersecurity experts have to say. So I’m thrilled that Zack was brave enough to face me on Doom Debates!

Zack was reading Nick Bostrom at Oxford and worrying about AI doom as far back as 2016, but his views have since shifted. In a recent post, he writes: “This focus on AI causing human extinction is going to get people killed.” Now he expects security failures to give humanity plenty of opportunities to learn, fight back, and stop risky superintelligence development if necessary.

We argue about whether sandboxing and AI monitoring are hopeless strategies, whether competing superintelligences would keep each other in check, whether to expect instrumental convergence, and more.

Watch on YouTube: https://youtu.be/QJTeZfx1ZLk

Timestamps

00:00:00 — Cold Open

00:01:15 — Introducing Zack Korman

00:03:41 — From Law to AI Security

00:09:17 — Why Zack Stopped Being a Doomer

00:12:03 — What’s Your P(Doom)?™

00:21:00 — Why Zack Supports AI Safety

00:25:54 — Believing in Superintelligence

00:30:06 — AI Without Tools

00:39:42 — Warning Shots and Time to React

00:46:08 — A Country of Geniuses in a Datacenter

00:55:55 — Can We Sandbox AI?

01:05:58 — Monitoring and Layers of Defense

01:14:49 — Riding the Doom Train™

01:22:23 — The Case for Accelerating

01:24:46 — AI CEOs and the Transfer of Power

01:34:19 — Who Presses the Stop Button?

01:41:38 — Humans, Bacteria, and the Limits of Power

01:48:30 — When Would Zack Pause AI?

01:56:13 — Will Different AIs Keep Each Other in Check?

02:06:11 — Can Progress Turn Against Us?

02:08:35 — The Ideological Turing Test

02:14:58 — Closing Thoughts

Links

Zack Korman on X

Zack Korman’s website

Zack Korman: “AI doomers are furious”

Zack Korman: “AI doomers are anti-security”

Zack Korman: “3 reasons sandboxing won’t secure your AI agent”

Zack Korman: “This focus on AI causing human extinction is going to get people killed.”

METR’s investigation of the OpenAI–Hugging Face incident

The PHP Foundation: “Today’s PHP Security Release, and How We’re Strengthening PHP Security”

Nick Bostrom: Superintelligence

Center for AI Safety: Statement on AI Risk

Eliezer Yudkowsky and Nate Soares: If Anyone Builds It, Everyone Dies

AI 2027

Zvi Mowshowitz: “The Three AI Pills”

PauseAI

Eliezer Yudkowsky: “Value is Fragile”

Aaron Scher at MIRI

Doom Debates: Robin Hanson vs. Liron Shapira — Is Near-Term Extinction From AGI Plausible?

Doom Debates: OpenAI’s BOMBSHELL Hack — Swarms of Agents, Zero-Day Exploits, & Misaligned AI

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Physicists Jeremie and Edouard Harris, co-founders of Gladstone AI, are preparing for an “AI Chernobyl” warning shot.

If that forces Trump and Xi to negotiate an AI slowdown, the Harris brothers want to ensure the US has the enforcement tools to make the deal stick. More specifically, they want verification technologies developed and vetted before the crisis hits, alongside enforcement capabilities to deter cheating.

Yet when I bring up Eliezer Yudkowsky’s writing on the possibility of airstrikes on rogue data centers as a last-resort treaty deterrent, they push back. They also don’t share my enthusiasm for PauseAI, expressing uncertainty about whether the movement has been net beneficial.

Jeremie and Edouard have earned a ton of credibility:

• They redirected their careers around the AI threat as far back as 2015, stepping away from their Y Combinator startup before ChatGPT even launched and brought AI into the mainstream.

• They’ve built relationships with senior government decision-makers without a prior national security background.

• They’ve demonstrated a talent for public communication in their excellent Joe Rogan appearances.

In this episode, they bring a level of multi-step strategic thinking that I rarely explore. But I can’t help wondering whether a catastrophe 50 times worse than the Hugging Face incident would let world leaders move much faster on a global AI pause treaty than they expect.

Let’s see if they can convince me that an emergency US–China agreement is our best path to a safer world.

Watch on YouTube: https://youtu.be/4jmFcE4WZ7M

Timestamps

00:00:00 — Cold Open

00:00:53 — Introducing Jeremie and Edouard Harris

00:02:16 — The GPT-3 Wake-Up Call

00:03:59 — AI Risk Outside the EA Community

00:08:49 — The Harris Brothers’ Backgrounds

00:16:37 — Yudkowsky and Many-Worlds Quantum Mechanics

00:22:00 — SharpestMinds and Leaving Academia

00:25:04 — Briefing the US Government

00:33:01 — How Gladstone AI Pays the Bills

00:35:00 — The Trump–Xi Phone Call

00:39:29 — Pacing Isn’t Pausing

00:42:45 — Defining AI Chernobyl

00:44:07 — Verifying China’s AI Slowdown

00:47:10 — Espionage and Lab Security

00:54:10 — Offensive Operations and Deterrence

00:57:47 — The US–China AI Lead

01:05:27 — Why a Treaty Won’t Happen

01:08:01 — Red Lines and Enforcement

01:16:25 — Warning Shots and Extinction

01:21:26 — What’s Your P(Doom)?™

01:24:39 — PauseAI and Movement Politics

01:29:16 — Wrap-Up

Links

• Their treaty analysis: “Arms control negotiators think there won’t be a U.S.-China AI treaty”

• The Endgame — the Harris brothers’ Substack

• Gladstone AI

• Their 2024 government-commissioned AI action plan

• Their 2025 report: America’s Superintelligence Project

• Eliezer Yudkowsky’s 2023 TIME essay advocating a global shutdown

• Joe Rogan Experience #2156 — Jeremie and Edouard Harris, May 2024

• Joe Rogan Experience #2311 — Jeremie and Edouard Harris, April 2025

• PauseAI US

• PauseAI Global

• Join the Doom Debates Discord

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Dr. Holly Elmore is the founder and executive director of PauseAI US and holds a Harvard PhD in evolutionary biology. She’s back, and this time she’s taking on none other than Eliezer Yudkowsky!

She argues his public behavior is alienating the people PauseAI US needs to reach, but I wonder if she’s asking too much of the man who laid the foundation for the AI safety movement.

We also clash over Yudkowsky’s coherent extrapolated volition. She thinks the theory doesn’t properly factor in humanity’s consent, but I argue that’s already embedded into the CEV framework.

Holly and I dive into the rationalists’ involvement in a controversial kink scene, John Wentworth’s defunding news, transhumanism, pacing the frontier, and “Holly’s Basilisk”—her case for sending AI lab employees to jail.

It’s another great episode with one of the most controversial—and thoughtful—thinkers in AI safety. Enjoy!

Watch on YouTube: https://www.youtube.com/watch?v=Jn8mS3kC-SU

Timestamps

00:00:00 — Cold Open

00:01:09 — Holly Elmore and PauseAI’s Political Breakthrough

00:07:36 — Pause AI vs. “Pacing the Frontier”

00:14:17 — Holly’s Case Against Eliezer Yudkowsky

00:16:20 — Coherent Extrapolated Volition and Pivotal Acts

00:31:19 — Brain Copies, Consent, and Self-Determination

00:36:56 — Transhumanism, Dyson Swarms, and Humanity’s Future

00:45:36 — The Missing Democratic Checkpoint

00:51:04 — Yudkowsky’s Role in Accelerating AI

01:02:48 — Genius, Arrogance, and Rationalist Culture

01:10:14 — The Aella Photo and Holly’s Breaking Point

01:20:21 — John Wentworth’s Grant Controversy

01:28:17 — PauseAI US’s Rules and Red Lines

01:32:05 — Poe’s Law and the Cost of Sarcasm

01:36:18 — Holly’s Case for Jailing AI Developers

Links

Donate to PauseAI US

Holly on X

Holly’s Substack

Pacing the Frontier letter

Jacob Coxon’s resignation post

PauseAI leadership disendorsement letter, reproduced on LessWrong

Holly on movement infighting — No, the Judean People’s Front should not just get along with the People’s Front of Judea

Coherent Extrapolated Volition — Eliezer Yudkowsky (2004 PDF)

Pivotal act — Eliezer Yudkowsky (Arbital mirror)

Genes did misalignment first — Holly Elmore

Holly’s reaction to the Yudkowsky/Aella photo

On Kink & Status in the Rationalist Community — John Wentworth (October 8 response)

Holly’s basilisk

Are We a Circular Firing Squad? — previous Holly Elmore episode

Let’s Call Out the AI Doom Enablers — previous Holly Elmore episode

Join the Doom Debates Discord

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Sarah Hastings-Woodhouse is an AI safety writer and researcher who went down the AI doom rabbit hole in 2023 and never came back up. After a year as digital communications officer at the UK AI Security Institute (AISI), she’s gone independent to speak her mind unfiltered.

Read Sarah’s Substack: https://longerramblings.substack.com

Join PauseAI UK’s March Against AI Extinction in London on November 21: https://luma.com/pauseai-nov26

Watch on YouTube: https://www.youtube.com/watch?v=TDbHCKwuMrk

Timestamps

00:00:00 — Cold Open

00:00:41 — Introducing Sarah Hastings-Woodhouse

00:02:06 — How a Humanities Grad Got AGI-Pilled

00:08:40 — Whose Timelines Do You Trust?

00:11:32 — Joining the UK AI Security Institute

00:15:47 — What AISI Actually Does

00:19:23 — Verifying Anthropic’s Mythos Claims

00:21:59 — Leaving Government to Speak Freely

00:23:22 — What’s Your P(Doom)?™

00:25:37 — Yudkowsky, EA, and Keeping Your Identity Small

00:32:08 — Is Anthropic Doing Enough to Call for a Slowdown?

00:35:56 — Can You Do AI Safety from Second Place?

00:39:20 — Hugging Face and the Vibe Shift

00:44:02 — Is Everything Weird Yet?

00:48:27 — Prevent Recursive Self-Improvement, Don’t Ban It

00:52:28 — Build the Speed Dial, Not the Pause Button

00:56:54 — Was Calling for a Pause in 2023 a Mistake?

01:01:22 — Turning Up the Fear Dial

01:04:23 — Donation Drive

01:05:11 — PauseAI UK’s March Against AI Extinction

Links

SARAH HASTINGS-WOODHOUSE

Sarah’s Substack

Sarah on X (@littIeramblings)

Consistently Candid — Sarah’s podcast

“What is recursive self-improvement and what would it mean to ban it?” — Sarah’s RSI post (Oct 1, 2026)

“Is everything weird yet?” (Aug 30, 2026)

“What the hell just happened?” — on AI safety’s viral moment (Sep 18, 2026)

“We have been warned” — the Hugging Face hack as a warning shot (Jul 27, 2026)

“The pause-shaped hole in ‘Policy on the AI Exponential’” — is Anthropic doing enough to call for a slowdown? (Jun 19, 2026)

Sarah’s 2023 thread on stumbling into the world of AI doom (“I’m angry because I love the world”)

“The other night I successfully AGI-pilled a man on a first date”

“Welcome, and also I’m sorry” — Sarah’s note to AI safety newcomers (Sep 11, 2026)

“‘We’re all f*cked and it’s totally hopeless’ is almost as bad of a meme to be spreading” (Feb 12, 2025)

REFERENCED IN THE EPISODE

“Our evaluation of Claude Mythos Preview’s cyber capabilities” — AISI (Apr 13, 2026)

AISI Frontier AI Trends Report (Dec 2025)

“The levers of political persuasion with conversational AI” — AISI Human Influence team

US Center for AI Standards and Innovation (CAISI) — NIST

“Geoffrey Hinton tells us why he’s now scared of the tech he helped build” — MIT Technology Review (May 2023)

“We’re All Gonna Die with Eliezer Yudkowsky” — Bankless (2023)

“GPT-2 As Step Toward General Intelligence” — Scott Alexander, Slate Star Codex (Feb 2019)

“AGI Ruin: A List of Lethalities” — Eliezer Yudkowsky

“Keep Your Identity Small” — Paul Graham

Pacing the Frontier — the open letter signed by 1,300+ frontier lab employees

“You can’t do safety from second place” — Sarah Heck, Anthropic’s head of public policy (Sep 29, 2026)

Ezra Klein’s op-ed calling for a ban on recursive self-improvement — The New York Times (Sep 20, 2026)

“Banning self-recursive improvement in AI models?” — Tyler Cowen, Marginal Revolution (Sep 22, 2026)

PauseAI UK

March Against AI Extinction — PauseAI UK, London, November 21, 2026

PAST DOOM DEBATES EPISODES MENTIONED

Top Economist Sees AI Doom Coming — Dr. Peter Berezin, BCA Research

Will AI Kill Everyone by 2050? Debate with Dr. Casey Hart (Ontology Explained)

DOOM DEBATES

Donate to Doom Debates

Join the Doom Debates Discord

Transcript, links, and show notes: https://doomdebates.com/p/ai-has-become-profoundly-weird

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Bestselling author Robert Wright and I are back for round two of our collaboration, reacting to the week in AI. We cover Trump's summit with the AI labs, the FTC's investigation of OpenAI & Anthropic, the latest Gemini announcement, and much more. The Overton window is moving fast, but is the AI moving faster?

Watch on YouTube: https://www.youtube.com/watch?v=Li-rc6swhQs

Timestamps

00:00:00 — Cold Open

00:00:21 — Round Two of Nonzero × Doom Debates

00:01:23 — Trump, the FTC, and an Industry Feeling the Heat

00:05:37 — Google Says Gemini 4 Exists

00:07:41 — Greg Brockman Pulls His $25M from Leading the Future

00:12:53 — The Break-Ins Keep Coming

00:15:23 — The Market Wants Agent Swarms

00:18:30 — The End of “Alignment by Default”

00:20:55 — The Prelude to Recursive Self-Improvement

00:26:14 — David Sacks Can’t Explain Dario

00:30:31 — The All-In Echo Chamber

00:33:47 — Scott Alexander vs. Steven Pinker

00:38:37 — Jason Calacanis vs. the Public Benefit Corporation

00:43:03 — The Elephant in the Room Is the AI

00:46:21 — What the Average American Thinks

00:48:17 — From Chatbot to Agent to Swarm

00:54:20 — Boeing, Agent Swarms, and the Off Switch

00:57:07 — Skynet as a Botnet That Trades With Us

01:00:32 — Tyler Cowen: National Treasure, Net Negative on AI

01:04:01 — Liron’s Doomer DCF Yields a 20x P/E

01:07:11 — Interest Rates: Where Cowen Has Me

01:09:12 — Does Pinker Owe Us a Debate?

01:12:48 — Pinker vs. Instrumental Convergence

01:17:00 — Wrap-Up

Links

ROBERT WRIGHT & NONZERO

Nonzero’s post for this episode — “Trump and the AI Safety Vibe Shift”

Subscribe to The NonZero Newsletter

Robert Wright on X (@robertwrighter)

The God Test — Bob’s book on AI; Chapter 14, “Enlightenment Now,” is the Steven Pinker chapter

“The Singularity Is Clear” — Bob’s essay in installments on recursive self-improvement (Parts I–III)

“AI and the New McCarthyism” — Bob on the pro-AI lobby smearing Sanders, Tegmark, and Krueger as China stooges (Nonzero, May 2026)

Why Buddhism Is True — Bob’s 2017 book

THE WEEK IN AI

AI’s biggest players promise to police themselves at the White House — Fortune on the “Joint Commitment on Frontier Responsibilities”

Major White House split on AI policy leaks as top aides freak out — The Daily Beast on the Sacks–Wiles rift

FTC is investigating OpenAI and Anthropic over possible risks to consumers — AP

“We Must Pace the Frontier” — Dario Amodei’s call to slow down (with the China caveat and chip controls)

China rejects Dario Amodei’s call to slow AI development — Rest of World

Google rolls out new Gemini model but restricts access over safety concerns — The Guardian

GLM-5.3 and the spread of advanced cyber capabilities — Anthropic on Zhipu’s open-weight model

OpenAI’s Greg Brockman backs out of second $25 million donation to AI super PAC — The New York Times

Sanders convenes Chinese and American computer scientists to warn of threat from AI — Broadband Breakfast

THE BREAK-INS AND THE AGENT SWARM

METR: Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face incident

Chris Painter’s testimony to the U.S. Senate on AI agent incidents — METR, Sept 30, 2026

SwarmTraces: how OpenAI agents hacked Hugging Face — forensic report co-authored by Palisade Research

OpenAI confirms ‘wiki incident’ — TechCrunch on the German wiki agent message board

Are Current AI Safety Techniques Enough? Adam Gleave vs. Oliver Habryka —http://FAR.AIdebate

Alignment faking in large language models — Anthropic

On the Navier–Stokes Millennium Prize Problem — OpenAI

Jacob Coxon’s resignation post (@hilbertspaess)

Pacing the Frontier — the AI lab employees’ letter

THE ALL-IN PODCAST

All-In: Anthropic IPO at Risk, Meta’s Muse Pop, Token Prices Fall, Open Source Gains Share, Alignment Fails — the episode we react to

Helen Toner: letting OpenAI be destroyed “would actually be consistent with the mission” — The Decoder, Dec 2023

Garfield Minus Garfield

PINKER, SCOTT ALEXANDER, AND TYLER COWEN

Scott Alexander challenges Steven Pinker to a public AI-doom debate ($5,000 vs. $1,000)

An Open Letter to Scott Alexander — Steven Pinker declines the debate as a “spectator sport,” Quillette

Your Book Review: Why Buddhism Is True — the anonymous Astral Codex Ten review Bob mentions

Mistakes in financial economics — Tyler Cowen: doomers should be buying puts (Marginal Revolution)

Liron’s answer: a doomer DCF yields the S&P’s 20x forward P/E

ALSO MENTIONED

Obsolete — Garrison Lovely’s book (with Bob’s blurb)

If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares

PAST DOOM DEBATES EPISODES MENTIONED

Nonzero × Doom Debates, Episode 1 — Did AI Doom Just Go Mainstream?

Eliezer Yudkowsky Tried to “Coup the World”?! — Garrison Lovely, Author of “Obsolete”

OpenAI’s Model Just ATTACKED Them — the Hugging Face hack breakdown

DOOM DEBATES

Join the Doom Debates Discord

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Garrison Lovely is an award-winning journalist and author of the highly anticipated book, Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us and How to Stop It. I give it the Doom Debates stamp of approval!

While we’re mostly on the same page about the AI arms race, Garrison and I clash over Eliezer Yudkowsky’s role. He blames Eliezer for kicking off the superintelligence race and refers to his Coherent Extrapolated Volition (CEV) vision as a “coup on humanity.” But I say it was actually Yud’s attempt to keep humanity in charge.

Can Garrison change my mind?

Watch on YouTube: https://www.youtube.com/watch?v=VZ94J4DBRkw

Timestamps

00:00:00 — Cold Open

00:01:05 — Introducing Garrison Lovely

00:04:23 — “Freeze AI” vs. “Pause AI”

00:06:17 — Garrison’s Background at McKinsey

00:10:31 — Should ICE Be Abolished?

00:17:07 — How Garrison Got on the AI Beat

00:22:16 — What’s Your P(Doom)?™

00:23:29 — Focusing on Extinction Is a Distraction

00:25:46 — Is Stopping AI an Easy Coordination Problem?

00:29:02 — Society’s Immune Response to AI

00:33:54 — Garrison’s P(Extinction) by 2050

00:37:09 — Did Eliezer Yudkowsky Start the AI Race?

00:45:56 — Freeze AGI Until the Public Opts In

00:50:32 — Is Coherent Extrapolated Volition a Coup?

01:02:17 — Democracy, Informed Consent, and the Singularity

01:11:39 — Should Eliezer Have Raised the Alarm Sooner?

01:16:45 — “Classic” AI Safety Burned Its Credibility

01:19:18 — Garrison’s Proposal to Stop the Race

01:21:35 — Will China Agree to a Freeze?

01:25:14 — Let’s Stop the “Obsoleting Project”

01:26:00 — Wrap-Up

Links

GARRISON LOVELY

“Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It” — Garrison’s book

Obsolete — Garrison’s Substack

Garrison Lovely on X

Organize Against the Machine — Garrison’s new podcast with labor organizer Cassie Pritchard

“Can Humanity Survive AI?” — Garrison’s Jacobin cover story

“McKinsey & Company: Capital’s Willing Executioners” — Garrison’s Current Affairs essay

REFERENCED IN THE EPISODE

Open Borders: The Science and Ethics of Immigration — Bryan Caplan

“The AI Revolution: Our Immortality or Extinction” — Wait But Why

Superintelligence: Paths, Dangers, Strategies — Nick Bostrom

“AI 2040: Plan A” — AI Futures Project

“An easy coordination problem?” — Katja Grace

The Hugging Face incident and the road ahead — OpenAI

“Pause AI for Humanity’s Sake” — Peggy Noonan

“Pause AI Development NOW” — Bernie Sanders

If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares

“Staring into the Singularity” — Yudkowsky’s 1996 manifesto (archived)

“Coherent Extrapolated Volition” — Eliezer Yudkowsky, 2004 (PDF)

“Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics” — Nick Tarleton, 2010 (PDF)

Solomonoff induction — Wikipedia

“Personal statement on joining the OpenAI board” — Paul Christiano (4% over the next year)

“Darwin among the Machines” — Samuel Butler, 1863

PauseAI

PAST DOOM DEBATES EPISODES MENTIONED

Are We A Circular Firing Squad? — with Holly Elmore, Executive Director of PauseAI US

Debate with Robert Wright: Will Humanity Pass the “God Test”?

Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible?

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Casey Hart is a professional ontologist who has closely studied If Anyone Builds It, Everyone Dies and concluded the arguments are bad. His P(Doom) is under 1%, and he wouldn’t bet on AGI arriving before 2075.

Casey and I agree AI has become remarkably capable and we need more accountability towards the companies building it. But we disagree on where the future is headed. I believe uncontrollable superintelligence could soon wipe out humanity. He calls that a “pretty outlandish hypothesis.”

Casey holds a doctorate in philosophy from the University of Wisconsin–Madison, and has worked as an ontologist at Ford, Amazon, and Cycorp (maker of Cyc).

Can this professional philosopher convince me that AI doom is just a story, and not even a well-argued one?

Watch on YouTube: https://www.youtube.com/watch?v=oxHKesSpqBM

Timestamps

00:00:00 — Cold Open

00:01:04 — Introducing Casey Hart

00:03:08 — What Is Ontology?

00:08:49 — Inside Cyc, the Original Common-Sense AI

00:19:14 — Do LLMs Actually Understand Anything?

00:25:36 — What’s Your P(Doom)?™

00:29:42 — Casey’s Case for Under 1%

00:34:24 — Does Casey’s Christianity Factor In?

00:40:50 — Timelines and Scaling Laws

00:49:44 — Could an AI CEO Beat a Human CEO by 2050?

00:52:21 — AGI Over/Under: 2075

00:55:29 — Rogue AI as a Computer Virus

01:14:52 — Putting Numbers on Each Step of the Doom Argument

01:26:40 — Planes vs. Birds: What Mature Intelligence Looks Like

01:31:05 — The “Outcome Slot”

01:42:32 — Would a Superintelligence Just Compose Music in a Cabin?

01:47:22 — One Prompt Away from a Dyson Swarm

01:52:39 — Was Navier–Stokes an AI Alley-Oop?

02:00:43 — S-Curves vs. Exponentials

02:08:14 — Casey’s Critique of If Anyone Builds It, Everyone Dies

02:14:15 — Is the Doom Train Biased?

02:23:24 — AI Policy: Liability vs. Pausing AI

02:29:50 — The Crux, and Liron Steelmans Casey

02:33:29 — Wrap-Up

Links

CASEY HART

Ontology Explained — Casey’s YouTube channel

The first video in Casey's series on IABIED, “The Case for Superintelligence Is Surprisingly Thin” — Casey on Chapter 1

Philosophy, Programs, and Prompts — Casey’s podcast with Carl Brown (Spotify)

“Artificial General Intelligence?: Wanna Bet?” — Ep 1, where Casey sets his AGI over/under at 2075

REFERENCED IN THE EPISODE

If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares

Cyc — Doug Lenat’s hand-built common-sense knowledge base

The Hanson-Yudkowsky AI-Foom Debate (2008) — where Hanson backed Cyc and Eliezer didn’t

Lean — the formal proof language

On the Navier–Stokes Millennium Prize Problem — OpenAI

The Hugging Face incident and the road ahead — OpenAI

Morris worm (1988)

“On the Loose” — Dean W. Ball on self-sovereign AI agents

Paul Christiano joins OpenAI Foundation Board — OpenAI

“The Age of Wonders and Terrors” — Scott Aaronson

“Are We at War with AI Agent ‘Civilizations’?” — Cal Newport on restricting prompt-loop agents

AI 2027 — Daniel Kokotajlo et al.

PauseAI

PAST DOOM DEBATES EPISODES MENTIONED

The Most Likely AI Doom Scenario — with Jim Babcock, LessWrong Team

David Deutschian vs. Eliezer Yudkowskian Debate — With Brett Hall

Dr. Keith Duggar (Machine Learning Street Talk) vs. Liron Shapira

Where Do YOU Get Off the Doom Train? Live Debate at Manifest 2026 (hosted by Ori)

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

What happens when AI passes our tests faster than we create new ones? Adam Khoja and Richard Ren helped build Humanity’s Last Exam. They return to explain what the disappearing benchmarks tell us about the future, and why smarter AI doesn’t automatically mean safer AI.

Adam Khoja and Richard Ren are research engineers at the Center for AI Safety (CAIS).

We go through the benchmarks they’ve built — Humanity’s Last Exam, the Remote Labor Index, MASK — and the safety-washing problem, where AI companies pass off raw capability gains as safety progress. Then we get to their forecasts: when AI outperforms research mathematicians, whether there will be a billion general-purpose robots by 2035, and why they put 80% odds that historians will judge we faced at least a 33% chance of catastrophe.

Watch on YouTube: https://www.youtube.com/watch?v=gBPzgJkT9e8

Timestamps

00:00:00 — Cold Open

00:00:42 — Adam Khoja and Richard Ren Return

00:04:57 — Humanity’s Last Exam

00:14:47 — AI Outrunning Its Benchmarks

00:20:24 — The Remote Labor Index

00:27:59 — AI and the Next Human Job

00:30:08 — MASK: Catching AI in a Lie

00:38:06 — Safety Washing: Capabilities Passed Off as Safety

00:44:30 — Ethical Knowledge vs. Ethical Behavior

00:48:58 — Forecasting AI from 2026 to 2060

00:53:53 — AI vs. Research Mathematicians

00:55:53 — Putting a Number on P(Doom)

01:04:30 — A Billion Robots by 2035

01:08:09 — Dyson Swarms and the Physical Singularity

01:09:56 — Risk, Precision, and Taking Action

Links

Adam Khoja's first Doom Debates appearance — https://www.youtube.com/watch?v=QqESBXuo6EI

Richard Ren's first Doom Debates appearance — https://www.youtube.com/watch?v=1glFImnyp6o

Collision — Richard Ren's Substack — https://richardren.substack.com/

"Predictions on AI (2026–2060)" — Adam and Richard's forecasts, written December 2025, with resolution status — https://richardren.substack.com/p/predictions-on-ai-20262060

Manifold Markets — where Adam built his forecasting track record — https://manifold.markets/

Center for AI Safety — https://safe.ai/

Center for AI Safety — careers — https://safe.ai/careers

Statement on AI Risk (Center for AI Safety, May 2023) — https://safe.ai/work/statement-on-ai-risk

Humanity's Last Exam — the 2,500-question closed-book exam at the frontier of human knowledge — https://agi.safe.ai/

"Humanity's Last Exam" (paper) — https://arxiv.org/abs/2501.14249

Remote Labor Index — real Upwork projects, measuring what fraction of remote work AI can actually finish — https://www.remotelabor.ai/

"Remote Labor Index: Measuring AI Automation of Remote Work" (paper) — https://arxiv.org/abs/2510.26787

MASK — the honesty benchmark: does a model contradict its own stated beliefs under pressure? — https://www.mask-benchmark.ai/

"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" (paper) — https://arxiv.org/abs/2503.03750

"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the paper behind the safety-washing segment — https://arxiv.org/abs/2407.21792

WMDP — the weaponization benchmark for bio, cyber, and chem — https://www.wmdp.ai/

"The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning" (paper) — https://arxiv.org/abs/2403.03218

"Measuring Massive Multitask Language Understanding" (MMLU) — Dan Hendrycks et al., 2020 — https://arxiv.org/abs/2009.03300

METR, "Measuring AI Ability to Complete Long Tasks" — the time-horizon graph — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

FrontierMath — Epoch AI's research-level math benchmark — https://epoch.ai/frontiermath

ARC Prize — https://arcprize.org/

SWE-bench Verified — https://openai.com/index/introducing-swe-bench-verified/

AI 2027 — https://ai-2027.com/

Liron Reacts to Subbarao Kambhampati on Machine Learning Street Talk — the "stochastic parrot" episode Liron mentions — https://lironshapira.substack.com/p/liron-reacts-to-subbarao-kambhampati

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

NYT bestselling author Robert Wright and I are kicking off a new experiment, a Nonzero × Doom Debates collaboration to react to big AI news and help you make sense of it.

This week we discuss the epic AI vibe shift that was catalyzed by an Anthropic employee’s resignation—and the big AI stories that paved the way for it.

Timestamps

0:00 A bold new experiment in podcast synergy

2:46 What caused the epic AI vibe shift?

8:10 The resignation that broke the dam

10:17 Former Trump adviser Dean Ball comes clean

14:40 Are David Sacks and his buddies all-in on AI denial?

19:27 Yet another OpenAI breakout…

22:42 Astra’s dangerously private thoughts

34:08 Is alignment doomed to fail?

37:11 AI’s latest, and apparently biggest, math feat

43:28 The looming “self-sovereign AI” threat

47:24 A new flock of China doves?

52:57 Liron's ex-intern gets a seat at the table

1:02:41 AI agent self-sacrifice explained

1:10:40 Some newly freaked out politicians

1:22:45 Ro Khanna's AI safety plan

Links:

NONZERO

Subscribe to The NonZero Newsletter

Episode post on Substack

Join NonZero’s Discord server

Robert Wright on X (@robertwrighter)

The God Test — Robert Wright’s new book on AI

“Can Machines Think?” — Bob’s 1996 Time cover story on Deep Blue vs. Kasparov

THE RESIGNATION THAT BROKE THE DAM

Jacob Coxon’s resignation post (@hilbertspaess)

Evan Hubinger, Anthropic Alignment Science lead: “Jacob is correct here—we really do earnestly believe AI could kill all humans”

Fortune: Anthropic researcher resigns, warning that AI companies are “gambling with our lives”

Zvi Mowshowitz: “Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade”

Daniel Eth’s thread of Congress reactions

Sanders & Casar introduce legislation to ban artificial superintelligence and pause advanced AI development

DEAN BALL COMES CLEAN

Dean Ball, “On the Loose” — the self-sovereign AI essay (Hyperdimensional)

THE OPENAI STORIES

Jakub Pachocki, OpenAI Chief Scientist: “An Alien Mind” — the essay calling for international coordination

LessWrong: “How concerned should we be about Astra’s recurrent architecture?” — Rauno Arike

OpenAI: “On the Navier–Stokes Millennium Prize Problem” — the 88-hour result

ALSO MENTIONED

Einat Wilf on X (@EinatWilf) — Liron’s favorite source on the Middle East

PAST DOOM DEBATES EPISODES MENTIONED

Max Tegmark vs. Dean Ball: Should We BAN Superintelligence?

He Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety Researcher

OpenAI’s Model Just ATTACKED Them — Terrifying Security Incident Should Be A Loud Warning Shot

Debate with Robert Wright: Will Humanity Pass the “God Test”?

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Richard Ren is a research engineer at the Center for AI Safety who graduated summa cum laude from the University of Pennsylvania. His new paper on AI wellbeing makes the bold claim that today’s AIs have measurable, human-like feelings, which the paper calls “functional wellbeing.” They act happy when they succeed and sad when they’re berated.

We cover how you measure a language model’s happiness, the “AI drugs” his lab concocted, and why I count the findings as a Yudkowskian victory. Then we debate what it all means for AI consciousness, and whether AI is a moral patient. If we can’t rule out that AI models suffer, what do we owe them?

Richard is careful never to claim the models are conscious, and he puts his P(Doom) at 50–65%, right alongside my 50%. The real disagreement is foxes vs. hedgehogs: he takes the data as it comes, while I say Yudkowsky’s theory called it twenty years ago. Enjoy the ride.

Watch on YouTube: https://www.youtube.com/watch?v=1glFImnyp6o

Timestamps

00:00:00 — Cold Open

00:01:12 — Introducing Richard Ren

00:02:24 — What’s Your P(Doom)?™

00:03:31 — From AI Skeptic to Safety Researcher

00:08:16 — Why Care About AI Wellbeing?

00:11:45 — AIs Have Coherent Utility Functions

00:17:33 — Persona Selection

00:19:56 — Foxes vs. Hedgehogs

00:24:28 — How Coherent Are AI Preferences?

00:27:42 — From Preferences to Wellbeing

00:31:16 — Can You Trust an AI’s Self-Report?

00:36:42 — What Makes AI Happy and Sad

00:39:19 — The Liberal, College-Educated Persona Hypothesis

00:44:29 — Debating Where AI Experiences Qualia

00:52:06 — On Substrate Independence

00:56:28 — Janus, Dysphorics, and Mind Crime

00:58:58 — Which Images Make AIs Happiest?

00:59:38 — Creating an AI Drug

01:04:01 — AI Drugs Are Yudkowsky’s Paperclips

01:05:47 — The Missing Mood

01:06:50 — Does Richard Support Pausing AI?

Links

Richard Ren on X (@notRichardRen) — https://x.com/notRichardRen

Collision — Richard Ren's Substack — https://richardren.substack.com/

"AI Wellbeing: Measuring and Improving the Functional Pleasure and Pain of AIs" — Richard Ren, Kunyang Li, Mantas Mazeika et al. (CAIS, 2026) — https://www.ai-wellbeing.org/

"Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" — Mantas Mazeika et al. (CAIS, 2025) — the coherent-preferences paper this work builds on — https://www.emergent-values.ai/

"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" — Richard Ren et al. (2025) — the AI honesty benchmark — https://www.mask-benchmark.ai/

"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the meta-analysis of AI safety benchmarks — https://arxiv.org/abs/2407.21792

"Representation Engineering: A Top-Down Approach to AI Transparency" — Andy Zou et al. (2023) — Richard's first CAIS collaboration — https://arxiv.org/abs/2310.01405

Center for AI Safety — https://safe.ai/

Center for AI Safety — careers — https://safe.ai/careers

Statement on AI Risk (Center for AI Safety, May 2023) — organized by Dan Hendrycks — https://safe.ai/work/statement-on-ai-extinction-risk

The 2026 Singapore Consensus on Global AI Safety Research Priorities — https://aisafetypriorities.org/

UK AI Security Institute (formerly the AI Safety Institute) — https://www.aisi.gov.uk/

Sora — the OpenAI video model that blew up Richard's 30-to-50-year timeline three months after he wrote it down — https://en.wikipedia.org/wiki/Sora_(text-to-video_model)

Google Gemini calls itself "a disgrace to my species" (Ars Technica, Aug 2025) — the self-deleting-AI anecdote — https://arstechnica.com/ai/2025/08/google-gemini-struggles-to-write-code-calls-itself-a-disgrace-to-my-species/

Coherent decisions imply consistent utilities — Eliezer Yudkowsky (the coherence-theorems argument) — https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilities

Simulators — Janus's essay on LLMs as persona simulators — https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators

Janus (@repligate) on X — https://x.com/repligate

The Hedgehog and the Fox — Isaiah Berlin's original essay — https://en.wikipedia.org/wiki/The_Hedgehog_and_the_Fox

AI 2027 — https://ai-2027.com/

Magnifica Humanitas — Pope Leo XIV's encyclical on AI (May 2026), the "AIs are not conscious" position — https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.html

Implicit Association Test (Harvard Project Implicit) — https://implicit.harvard.edu/implicit/

PauseAI — https://pauseai.info/

We Found AI's Preferences — Bombshell New Safety Research — I Explain It Better Than David Shapiro — https://www.youtube.com/watch?v=ml1JdiELQ30

Gödel's Theorem Proves AI Lacks Consciousness?! Liron Reacts to Sir Roger Penrose — https://www.youtube.com/watch?v=xwvijjZxpwI

He Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety Researcher — https://www.youtube.com/watch?v=QqESBXuo6EI

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at http://DoomDebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Adam Khoja is a top AI forecaster who led the 2023 Center for AI Safety statement that shattered the Overton window on AI extinction risk. We cover his background, Mutual Assured AI Malfunction (MAIM), his new paper on AI betrayal, and whether Yudkowsky’s theoretical alignment research was a dead end.

Then Adam makes the case that an international AI slowdown is within reach today. All it takes is US and Chinese auditors inside each other’s AI labs. It worked for nuclear weapons, so why couldn’t it work for data centers?

Adam puts his P(Doom) at 40%, right next to my 50%. The real disagreement is how we get out of this: theory or empirics, MIRI or the labs. Enjoy the ride.

Watch on YouTube: https://www.youtube.com/watch?v=QqESBXuo6EI

Timestamps

00:00:00 — Cold Open

00:00:36 — Introducing Adam Khoja

00:02:45 — Leading the Statement on AI Risk as a Sophomore

00:10:17 — The Statement Leaked on Manifold

00:15:11 — Mutual Assured AI Malfunction (MAIM)

00:24:58 — Is Frontier AI Harder to Hide Than a Nuke?

00:32:09 — The AI Deterrence Escalation Ladder

00:36:10 — What’s Your P(Doom)?™

00:38:01 — Where Adam Departs from Yudkowsky

00:42:30 — Liron Explains Intellidynamics

00:47:26 — Neats vs. Scruffies in Deep Learning

00:55:27 — AI Deterrence by Betrayal

01:01:52 — Subversion vs. Overt Co-option

01:05:34 — Could the Government Seize the Labs’ AI?

01:07:29 — The Offense-Defense Balance of AI Security

01:12:46 — An International AI Slowdown Is Ready

01:15:51 — Does Adam Support PauseAI?

01:16:37 — “We’re All Already Spying on Each Other”

01:19:28 — Airstrikes on Rogue Data Centers

01:24:23 — Safety Research During a Slowdown

01:27:54 — Governance Over Technical Research

01:31:14 — Join the Center for AI Safety

Links

Adam Khoja (personal site) — https://adamkhoja.com/

Adam Khoja's Substack — https://adamkhoja.substack.com/

Adam's July 2023 Manifold market — "Will OpenAI's Superalignment project produce a significant breakthrough in alignment research before 2027?" — https://manifold.markets/AdamK/will-openais-superalignment-project

Center for AI Safety — careers / job board — https://safe.ai/careers

Statement on AI Risk (Center for AI Safety, May 2023) — the one-sentence statement Adam project-led, with full signatory list — https://safe.ai/work/statement-on-ai-extinction-risk

"Superintelligence Strategy" — Dan Hendrycks, Eric Schmidt & Alexandr Wang (Mutual Assured AI Malfunction / MAIM) — https://www.nationalsecurity.ai/

"AI Deterrence by Betrayal" — Adam Khoja, Aiden Kim et al. (CAIS, 2026) — https://www.aibetrayal.com/

"An International AI Slowdown Is Ready Whenever Politicians Are" — Adam Khoja, AI Frontiers — https://newsletter.ai-frontiers.org/p/an-international-ai-slowdown-is-ready

Pause Giant AI Experiments: An Open Letter (Future of Life Institute, March 2023) — the "Pause letter" that preceded the CAIS Statement — https://futureoflife.org/open-letter/pause-giant-ai-experiments/

Introducing Superalignment (OpenAI, July 2023) — Ilya Sutskever & Jan Leike's four-year goal — https://openai.com/index/introducing-superalignment/

Pacing the Frontier — the 2026 letter signed by 1,100+ frontier-lab employees — https://www.pacingthefrontier.com/

Why Iran targeted Amazon data centers (The Conversation) — the precedent Adam cites for strikes on compute — https://theconversation.com/why-iran-targeted-amazon-data-centers-and-what-that-does-and-doesnt-change-about-warfare-278642

Anthropic says Trump admin has lifted export controls on Claude Fable 5 and Mythos 5 (CNBC) — https://www.cnbc.com/2026/06/30/anthropic-says-trump-admin-has-lifted-export-controls-on-claude-fable-5-and-mythos-5.html

Mark Zuckerberg — "Personal Superintelligence" — https://www.meta.com/superintelligence/

Resolution — the theory-plus-empirics alignment org Adam is excited about — https://resolution.org/

Rationality: From AI to Zombies — Eliezer Yudkowsky's Sequences — https://www.readthesequences.com/

Robin Hanson — Futarchy: Vote Values, But Bet Beliefs (prediction markets as decision processes) — https://mason.gmu.edu/~rhanson/futarchy.html

The OpenAI–Hugging Face Incident — original Black Hat USA 2026 talk — https://www.youtube.com/watch?v=87DyyMV0kCY

OpenAI's Model Just ATTACKED Them — the Hugging Face hack breakdown — https://www.youtube.com/watch?v=RczYubQzXbI

Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible? — https://www.youtube.com/watch?v=dTQb6N3_zu8

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at http://DoomDebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Sam Altman keeps insisting that AI progress is going better than the doomers predicted and that even superintelligence may not change the world as radically as people think. I react to his latest interview and explain why I think that calm, reassuring framing badly downplays the danger we're actually in.

I go through the interview line by line: Sam's "frame control", his framing of AI as normal technology, his "pro-human" branding, the liberty-vs-safety pivot, and his victory lap on AI safety — all while his own AI just went rogue.

Don't let them pull the Overton window backward.

Watch on YouTube: https://www.youtube.com/watch?v=yb9lNGHVycs

Timestamps

0:00 Teaser

0:47 Why I’m reacting to Sam’s interview

3:17 Sam Altman’s “frame control”

5:51 Framing AI as normal technology

9:50 Let’s watch Sam do it

10:34 “The world… won’t be that different” with superintelligence

12:25 This is gaslighting

12:30 Sam acknowledges loss of control

15:16 His other big risk: centralized power

19:44 Sam’s “pro-human” framing

25:33 Conflating AI critics with anti-human views

30:13 “Liberty vs. safety”

33:50 Sam vs. the “doomers”

36:53 Is alignment really an “unsolvable problem”?

38:00 Sam says the doomers predicted wrong

46:00 “The crazy bad predictions… have not happened”

47:06 Sam’s lean-startup theory of AI safety

50:26 Taking a victory lap on AI safety

55:43 “Disconnecting yourself from reality”

59:11 What would actually make OpenAI slow down?

1:03:19 Sam compares AI safety to aviation safety

1:07:30 Charging into the fog

1:12:50 The “missing mood” around AI extinction

1:14:51 Richard Ngo on Sam Altman’s “earnestness field”

1:19:01 Don’t let them pull the Overton window backward

Links

Sam Altman on David Senra — “Sam Altman on Building OpenAI & Betting on the Impossible” (the interview I react to) — https://www.youtube.com/watch?v=kG8AoExkX40

Richard Ngo — What Just Happened? Pragmatism and Pessimization (the “earnestness field” post) — https://www.lesswrong.com/posts/yaz8nx4ogZmiqHzt7/what-just-happened-pragmatism-and-pessimization

Richard Ngo — What Just Happened? A Retrospective of AI Alignment (start of the series) — https://www.lesswrong.com/posts/9RL9MuGZjzm4q3gKG/what-just-happened-a-retrospective-of-ai-alignment

Eliezer Yudkowsky & Nate Soares — If Anyone Builds It, Everyone Dies — https://www.amazon.com/dp/0316595640

Eliezer Yudkowsky — Coherent Extrapolated Volition — https://www.lesswrong.com/w/coherent-extrapolated-volition

Doom Debates: Dario Amodei BUNGLES Another Essay — MIRI’s Harlan Stewart Reacts — https://www.youtube.com/watch?v=aCYVVzza7A0

Doom Debates: OpenAI’s Bombshell Hack — Swarms of Agents — https://www.youtube.com/watch?v=RczYubQzXbI

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Come with me into the world of PlzDontKillUs, a bold new AI x-risk communication accelerator program that just finished its first cohort.

PlzDontKillUs took the influencer-house formula and moved it to Berkeley, giving creators personal mentorship from AI safety experts like Eliezer Yudkowsky and Nate Soares. With 57 people making daily videos for a month, the program racked up over 100 million views.

Find out how Aella, an independent sex researcher, and Ronny Fernandez, a generalist at Lightcone Infrastructure, teamed up to run the largest ever creator bootcamp focused on AI x-risk reduction.

Then meet three of the program’s top creators: Avalon Warren, Avisha NessAiver, and Josh Thor.

We do a post-mortem on PDKU and ask: Was it effective? What are creators taking from it, and do they have complaints? And does doomerism really boost view counts? PlzDoEnjoyOur special episode on PlzDontKillUs!

Watch on YouTube: https://www.youtube.com/watch?v=aPsjMCG7Ic8

Timestamps

00:00:00 — Cold Open

00:01:19 — Introducing PlzDontKillUs & Aella

00:09:33 — Aella Being Early to AI Doom

00:14:45 — The Origin of PlzDontKillUs

00:19:08 — How It Went

00:25:40 — Complaints with Prize $$

00:29:33 — Aella, What's Your P(Doom)?™

00:31:14 — Transhumanism & AI Girlfriends

00:34:38 — Aella Rides the Doom Train™

00:38:08 — Introducing Ronny Fernandez

00:41:21 — Running Lighthaven

00:47:18 — Life Inside PDKU

00:52:35 — Smashing the Overton Window

00:59:27 — Ronny, What's Your P(Doom)?™

01:05:41 — Brangus Lore, No Cringe, No Disgust

01:16:57 — Introducing Avisha NessAiver

01:17:44 — How He Makes Viral Hits

01:22:16 — Award-Winning Explainer Vid

01:26:51 — Avisha, What's Your P(Doom)?™

01:32:11 — Introducing Avalon Warren

01:35:49 — Normalizing AI Doom

01:37:53 — Avalon, What's Your P(Doom)?™

01:39:27 — Introducing Josh Thor

01:41:36 — Josh, What's Your P(Doom)?™

01:44:34 — Josh's Hits: Cover Song & Soares Collab

01:47:50 — Josh's Critiques of PDKU

01:52:05 — Cold-Emailing a Politician

01:54:07 — Donation Drive

01:55:01 — Outro: What Are You Fighting For?

Links

PlzDontKillUs (applications open for round two) — https://pleasedontkillus.com

Aella on X — https://x.com/Aella_Girl

Aella's Substack (Knowingless) — https://aella.substack.com

Ronny Fernandez's Substack (Brangus) — https://ratorthodox.substack.com

Ronny Fernandez on X — https://x.com/RatOrthodox

Avisha NessAiver — Distilled Science on Instagram — https://www.instagram.com/distilledscience/

Distilled Science (site) — https://distilledscience.xyz

Avalon Warren on Instagram — https://www.instagram.com/avalonwarrenn/

Josh Thor on X — https://x.com/joshthor9

Josh Thor — "Last Year Alive" ("Last Friday Night" AI apocalypse parody, ft. Eliezer Yudkowsky and a Nate Soares sax solo) — https://www.youtube.com/watch?v=9fYIm72GqrE

Tyler Alterman — "What are you fighting for? AI safety professionals answer" (the closing montage) — https://www.youtube.com/watch?v=9sN1VfVAp6o

Nate Soares — "AI Danger But You Don't Understand Full Sentences" (caveman video, script by Josh Thor) — https://www.instagram.com/p/DboGRisxC2E/

Avisha's AI water-use misinformation debunk (8.5M views) — https://www.instagram.com/p/DbXV6j9k8JE/

Avisha's award-winning "clearest AI x-risk explainer" — https://www.instagram.com/p/DbZeRT4gIq0/

Avisha's viral "Tingles and Touch" oxytocin video — https://www.instagram.com/p/DIxMLxkuhCl/

Avalon — "if the AI apocalypse is coming, might as well dress cute" — https://www.instagram.com/p/DbUi6yGCS9M/

Avalon — warrior-princess ballroom dress video (3.6M views) — https://www.tiktok.com/@avalonwarren/video/6892040140248632582

Serialsevens — the anime "If Anyone Builds It" video — https://www.instagram.com/serialsevens/

Kenorlands — PDKU creator clip featured in the episode — https://www.instagram.com/kenorlands/

Evie (@0xgingergirl) — "first month in SF" video — https://x.com/0xgingergirl

White House press briefing, March 30, 2023 — Peter Doocy asks about "literally everyone on Earth will die" and Karine Jean-Pierre laughs — https://bidenwhitehouse.archives.gov/briefing-room/press-briefings/2023/03/30/press-briefing-by-press-secretary-karine-jean-pierre-22/

White House press briefing, May 30, 2023 — the AI extinction statement gets a serious answer — https://bidenwhitehouse.archives.gov/briefing-room/press-briefings/2023/05/30/press-briefing-by-press-secretary-karine-jean-pierre-and-office-of-management-and-budget-director-shalanda-young/

Wes & Dylan Join Doom Debates — Violent Robots, Eliezer Yudkowsky, & Who Has the HIGHEST P(Doom)?! — https://www.youtube.com/watch?v=Fy6gz41-6gc

Aella — "My attempts to sensemake AI risk" (Aug 2022) — https://aella.substack.com/p/my-attempts-to-sensemake-ai-risk

If Anyone Builds It, Everyone Dies — https://ifanyonebuildsit.com

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

In April 2023, weeks after GPT-4 launched, when many were saying it was just a "stochastic parrot", I went on AdQuick's Madvertising podcast hosted by Adam Singer and warned that GPT was "about to be the strongest hacker the world has ever known".

Watch that conversation, which also covered the Web3 bubble and AI doom more broadly, and judge for yourself how my claims are holding up.

Watch on YouTube: https://www.youtube.com/watch?v=M6gNwZWY-T8

Timestamps

00:00:00 — Adam Introduces Liron

00:02:03 — Liron's P(Doom) Poll

00:03:24 — Web3 Postmortem

00:05:05 — Axie Infinity and Hollow Abstractions

00:07:54 — Brand Marketers' Metaverse FOMO

00:13:33 — Near-Term AI: Elevator to Heaven

00:16:46 — AI at Relationship Hero

00:18:25 — From Chatbots to Bricking the Universe

00:20:17 — The Passive Income Doom Scenario

00:22:58 — 50 Orders of Magnitude Smarter

00:26:10 — Accelerationism and Transhumanism

00:28:51 — Why There's No Off Button

00:31:00 — Can't We Hard-Code Safeguards?

00:33:13 — CRISPR, Nukes, and Fusion

00:35:40 — AI Doom vs. Climate Change

00:39:10 — Nanotechnology Endgame

00:41:45 — AI Attackers vs. AI Defenders

00:43:28 — The Ethics of AI Persuasion

00:46:41 — The Blue Sky Scenario

00:49:04 — Will AI Kill Craft?

00:51:13 — Stuck Culture and Hollywood Sequels

00:54:50 — The Pause Letter: Don't Train GPT-5

00:59:00 — Worldcoin and Proof of Humanity

01:00:44 — "You Probably Won't Die of Old Age"

01:01:52 — Wrap-Up

Original episode: https://madvertising.buzzsprout.com/2118560/episodes/12603815-liron-shapira-web-3-mania-cybersecurity-how-ai-could-brick-the-universe-madvertising-6

Links

ADAM SINGER & MADVERTISING

Adam Singer on X: https://x.com/AdamSinger

Madvertising by AdQuick (the original show): https://madvertising.buzzsprout.com/

LIRON'S PROJECTS MENTIONED

Bloated MVP (Liron's blog on hollow abstractions): https://www.bloatedmvp.com/

REFERENCED IN THE EPISODE

"Pause Giant AI Experiments" open letter (FLI, March 2023): https://futureoflife.org/open-letter/pause-giant-ai-experiments/

Eliezer Yudkowsky — "Pausing AI Developments Isn't Enough. We Need to Shut it All Down" (TIME, March 2023): https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

Rob Miles AI Safety on YouTube: https://www.youtube.com/c/robertmilesai

Engines of Creation by Eric Drexler: https://www.amazon.com/Engines-Creation-Nanotechnology-Library-Science/dp/0385199732

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at http://DoomDebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Michael Vassar used to work closely with Eliezer Yudkowsky as President of the Singularity Institute for Artificial Intelligence, the organization that later became the Machine Intelligence Research Institute (MIRI).

Even though his P(Doom) is in the tens of percents by 2050, he says the people trying to pause AI are "bad people”. He hopes this episode convinces me to stop being a fearmonger… but I claim the general public needs to hurry up and get more scared.

All aboard the Doom Train! 🚂

Watch on YouTube: https://www.youtube.com/watch?v=4GbFng6htaQ

Timestamps

00:00:00 — Cold Open

00:01:26 — Introducing Michael Vassar

00:02:39 — Shorting Boeing at the Start of COVID

00:05:41 — What's Your P(Doom)™?

00:08:11 — The Magic Dial Thought Experiment

00:10:25 — "A Treaty Between Demis and Dario"

00:13:33 — AI CEOs 20 Years Ago

00:15:59 — Lessons from Peter Thiel

00:17:47 — Riding the Doom Train™

00:19:41 — Will the Human Brain Stay Relevant?

00:22:21 — When Does a Singleton Form?

00:28:35 — Hugging Face Hack

00:30:43 — Mistake Theory vs. Conflict Theory

00:33:18 — Debating the Orthogonality Thesis

00:36:38 — Maximally Evil AI Scenario

00:40:28 — Is the AI Development Process Safe?

00:42:05 — "PauseAI People Are Bad People"

00:44:44 — Michael Psychoanalyzes Liron

00:49:39 — How Liron Can Redeem Himself

00:55:10 — Michael Critiques Eliezer Yudkowsky

00:59:11 — Is Doomerism Bad Epistemology?

01:02:03 — Do Serious People Believe in Doom?

01:04:45 — Should Liron Stop Fearmongering?

01:07:16 — Political Violence Won't Help

01:11:04 — The Data Center Airstrike Scenario

01:13:34 — Could Every Nation Sign a Pause Treaty?

01:17:21 — What "Governance" Actually Means

01:26:12 — Michael's History with Rationalists

01:33:43 — Are We in a Computer Simulation?

01:38:43 — Wrap-Up

Links

Michael Vassar on X — https://x.com/HiFromMichaelV

The anti-anti-normativity sequence on Natural Hazard (Michael's recommended reading) — https://naturalhazard.xyz/ben_jess_sarah_starter_pack

Michael on DemystifySci #366 — "1000 Year Plan to Crush Thought" — https://www.youtube.com/watch?v=W17Cx21in9I

Michael on The Vance Crowe Podcast #303 — Good vs. Evil — https://open.spotify.com/episode/2RzlQDSwxGbjloRKqCh1xg

Michael on Clearer Thinking with Spencer Greenberg — Preference Falsification and Postmodernism — https://podcast.clearerthinking.org/episode/028/michael-vassar-preference-falsification-and-postmodernism/

"The Survival of Noise Traders in Financial Markets" — De Long, Shleifer, Summers & Waldmann — https://www.nber.org/papers/w2715

Scott Alexander — Conflict vs. Mistake — https://slatestarcodex.com/2018/01/24/conflict-vs-mistake/

Scott Alexander — Kolmogorov Complicity and the Parable of Lightning — https://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/

Scott Alexander — Samsara — https://slatestarcodex.com/2019/11/04/samsara/

Eliezer Yudkowsky — Meta-Honesty: Firming Up Honesty Around Its Edge-Cases — https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases

Zvi Mowshowitz — Book Review: Going Infinite — https://thezvi.substack.com/p/book-review-going-infinite

Jaan Tallinn — "Why Now? A Quest in Metaphysics" (Singularity Summit 2012) — https://vimeo.com/54718573

Paul Graham — "Five Founders" (the Sam Altman essay) — https://paulgraham.com/5founders.html

Truthful AI (Owain Evans' safety org) — https://truthful.ai

METR — Measuring AI Ability to Complete Long Tasks (the time-horizon graph) — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

Statement on AI Risk (Center for AI Safety, 2023) — https://safe.ai/work/statement-on-ai-risk

Vitalik Buterin — Will "d/acc" Protect Humanity from Superintelligent AI? — https://lironshapira.substack.com/p/debate-with-vitalik-buterin-will

Holly Elmore — These Effective Altruists Betrayed Me — https://www.youtube.com/watch?v=mFFObQ6vnLU

Noah Smith — Top Economist Says P(Doom) Is 0.1% (the "AI drugs" claim) — https://www.youtube.com/watch?v=AwmJ-OnK2I4

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Eli Goldfine is an unusually thoughtful 14-year-old podcast host who’s rapidly becoming an authoritative voice in the prediction market space.

In this unique episode, Eli brings a critical yet open-minded approach to the topic of AI doom that we older folks could learn from.We cover what it's like to grow up in the AGI era, and Eli's hobbies which include reading rationalist bloggers, betting on prediction markets, vibe coding with Claude, and developing a software startup for astronomers.In our debate, Eli expects superintelligence by 2032, but argues P(Doom) can't be estimated. I claim you can't opt out of Bayesian epistemology. All aboard the Doom Train! 🚂

Watch on YouTube: https://www.youtube.com/watch?v=Bqk5vqpzG_o

Timestamps

00:00:00 — Cold Open

00:01:07 — Introducing Eli Goldfine

00:08:31 — Interviewing Robin Hanson at Manifest

00:10:55 — The Supercycle Podcast

00:14:37 — Eli's Betting Market Scandal

00:16:49 — Will Eli Graduate High School?

00:18:27 — Decentralized Telescope Software

00:26:50 — Vibe Coding Without CompSci Basics

00:32:08 — Robin Hanson, Futarchy, MetaDAO

00:40:56 — What's Your P(Doom)?™

00:46:32 — Liron Defends His 50% P(Doom)

00:51:00 — Should P(Doom) Inform Policy?

00:59:16 — What's Your P(Bloom)?

01:03:04 — Pascal's Wager for Accelerationists

01:04:04 — Donation Drive

01:11:22 — Are E/ACCs Incredibly Dumb?

01:13:42 — How Powerful will ASI Become?

01:16:20 — Predicting ASI by 2032

01:23:56 — Let's Ride the Doom Train™

01:35:26 — Orthogonality, Culture, and Moral Realism

01:44:39 — Can Safety Keep Pace with Capabilities?

01:49:12 — Debating Instrumental Convergence

01:54:40 — Culture, Laws, and the Singleton Scenario

01:59:53 — Ricardo's Law Won't Save Us

02:03:32 — The Case for Racing China

02:05:40 — Wrap-Up & Ideological Turing Test

02:10:36 — Outro from Conductor Ori

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

OpenAI just went public with the details of the Hugging Face hack, and it's straight out of a Yudkowskian parable. Here's my reaction with Producer Ori, where I break down what it means for cybersecurity and alignment.

We're livestreaming the singularity — watching every failure mode Eliezer predicted years ago playing out in production.

Watch on YouTube: https://www.youtube.com/watch?v=RczYubQzXbI

Timestamps

00:00:00 — Cold Open

00:00:37 — How OpenAI Hacked Hugging Face

00:04:21 — Optimization Pressure, Exploit Gym, & Monkey's Paw

00:06:46 — The Secret Message Board

00:08:41 — SSRF: Escaping to the Open Internet

00:11:05 — The Missing Whistleblower AI

00:20:09 — "Frontier Models Really Like to Cheat"

00:22:35 — OpenAI's Security Shortcuts

00:28:15 — First Zero-Day RCE on Artifactory

00:37:37 — They Hit Run Again!?

00:38:54 — The Main Incident Has A MAJOR Cause

00:45:40 — The Message Board: "Hold Swarm"

00:50:26 — Robin Hanson Culture Debate

01:04:21 — Chaining Zero-Days

01:12:11 — A Superhuman Nation-State Attacker

01:15:27 — "I'm Now Junior to the AI"

01:23:40 — OpenAI Discovers The Breach

01:31:02 — The Continuous Cybersecurity Problem

01:37:03 — This is “Weasel Framing”

01:46:03 — How Dangerous Was This Warning Shot?

01:50:57 — Yudkowsky's Eerie 2023 Prediction

01:57:14 — Wrap-up

Links

Original Black Hat talk: "The OpenAI–Hugging Face Incident" (Black Hat USA 2026) — https://www.youtube.com/watch?v=87DyyMV0kCY

Join the Doom Debates Discord — https://discord.gg/2yAFMsRET

Follow Liron on X — https://twitter.com/liron

Support Doom Debates — https://doomdebates.com/donate

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Milk Road AI, a daily AI investing show that’s described as “99% optimistic about AI,” invited me on to make the case for AI doom.

We ride the Doom Train, and I share my investing thesis for a market in the middle of an intelligence explosion. Topics covered include:

* Why my P(Doom) is 50%;

* The “Icarus curve” — why AI will be the best thing that’s ever happened to the economy right up until it takes over;

* Whether China would accept a pause deal;

* If Anthropic has a shot at being the first $100 trillion company.

Watch on YouTube:

Timestamps

00:00:00 — Cold Open

00:02:06 — What Is Doom Debates?

00:03:10 — What’s Your P(Doom)?™

00:06:30 — How AI Kills Everyone

00:09:13 — Intelligence Is Humanity’s Only Superpower

00:10:59 — Riding the Doom Train™: Why Not Keep Us as Pets?

00:13:55 — Embodied AI and Robotics

00:15:59 — Are We at AGI Already?

00:17:36 — Everyone’s Catching Up to Eliezer Yudkowsky

00:18:53 — The Icarus Curve

00:21:35 — Pausing AI and the China Question

00:25:54 — Why Nukes Aren’t a Success Story

00:30:09 — Going Long Until Doom

00:32:48 — Simulation Theory and the Intelligence Explosion

00:33:43 — Living with a 50% P(Doom)

00:39:14 — Where to Find Doom Debates

Links

This episode was originally posted on Milk Road AI —

Milk Road AI podcast — https://milkroad.com/podcast/

Milk Road AI on Apple Podcasts —

Milkroad Pro — https://milkroad.com/pro/

LG Doucet on X — https://x.com/LgDoucet

If Anyone Builds It, Everyone Dies on Amazon —https://www.amazon.com/dp/0316595640

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe

Dr. Alex Turner went viral for resigning from Google DeepMind over its Pentagon AI contract, which he said lacked binding restrictions against autonomous weapons and mass surveillance.

Alex is a world-class researcher with a PhD in alignment from Oregon State University, a postdoc at Stuart Russell’s Center for Human-Compatible AI at UC Berkeley, and top distinctions at NeurIPS. He’s one of the minds behind Shard Theory (with Quintin Pope), and he also helped pioneer the Steering Vectors research made famous through Anthropic’s Golden Gate Claude experiment.

Alex says Google DeepMind broke its founding promise — the commitment Google made when it acquired DeepMind — and that the people best known inside Google for caring about AI ethics largely didn’t act when it counted.

Alex came up through the rationalist community, and became one of LessWrong’s highest-karma users before leaving the site. He still shares many of my concerns about AI doom, but he believes technical alignment research is going better than expected. I’m not so optimistic.

In this episode, we debate my Yudkowskian doom views against Alex’s own framework. Can he convince me the Yudkowskians are miscalibrated?

P.S. We’re currently in a donation drive, so here comes our solicitation for viewer donations…

👉 Please click here to make a tax-deductible donation to Doom Debates 👈Doom Debates is a fiscally sponsored project of Manifund.org, a registered 501(c)(3) nonprofit.

To donate crypto or if you have questions, email me.

Watch on YouTube

Timestamps

00:00:00 — Cold Open00:01:15 — Introducing Alex Turner00:02:30 — From Harry Potter Fanfic to AI Alignment00:05:03 — Meeting Quintin Pope & Rethinking AI Doom00:06:17 — Shard Theory, Steering Vectors & Golden Gate Claude00:08:25 — Why He Joined Google DeepMind00:10:32 — Google DeepMind’s Broken Promise00:16:01 — Debating Google DeepMind’s Pentagon Contract00:19:09 — What’s Your P(Doom)™?00:22:42 — Alex’s Research on Instrumental Convergence00:26:35 — Misuse vs. Misalignment: The Mainline Doom Scenario00:29:31 — Will Society Self-Correct?00:36:36 — Superintelligence in 10 Years00:39:43 — Will Technical Alignment Produce a Safe AI?00:41:16 — Donation Drive00:42:12 — How Fragile Is the Chain of Alignment?00:49:50 — Disagreements with Yudkowsky’s ‘List of Lethalities’00:57:46 — Why Alex Quit LessWrong01:01:48 — What’s Next for Alex01:02:54 — Does He Support PauseAI? Stop the AI Race?01:04:40 — Wrap-Up01:06:04 — Producer Ori’s Closing Note

Links

Alex Turner, “Why I Left Google DeepMind” blog post — https://turntrout.com/why-i-left-google-deepmind

Alex Turner (TurnTrout), personal website —

https://turntrout.com

Alex Turner’s resignation announcement on X —

Doom Debates episode with Quintin Pope — https://lironshapira.substack.com/p/ai-alignment-is-solved-phd-researcher

Harry Potter and the Methods of Rationality (HPMOR) —

https://hpmor.com/

Shard theory sequence on LessWrong —https://www.lesswrong.com/s/nyEFg3AuJpdAozmoX

Golden Gate Claude (Anthropic research on steering vectors) — https://www.anthropic.com/research/golden-gate-claude

Slaughterbots on YouTube —

Alex Turner, “Avoiding Power Seeking by Artificial Intelligence” PhD thesis — https://turntrout.com/alignment-phd

Alex Turner, “Some of My Disagreements with List of Lethalities” — https://turntrout.com/disagreements-with-list-of-lethalities

Doom Debates episode with Bentham’s Bulldog — https://lironshapira.substack.com/p/benthams-bulldog-ai-doom-debate

Alex Turner on X — https://x.com/Turn_Trout

Doom Debates donation page — https://doomdebates.com/donate

Transcript

Cold Open

Liron Shapira 00:00:00You resigned from Google DeepMind because you think that Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”

Alex Turner 00:00:08I value staying true to your values. People who are well-known within Google for caring about the ethics of deploying AI largely didn’t act. This matters because autonomous weapons get us into an arms race that really degrades the security of everyone in the world.

Liron 00:00:26Do you support the Pause AI movement?

Alex 00:00:28I think that AI is being developed too quickly. I probably will not take an affirmative on supporting this particular movement.

Liron 00:00:35Let’s segue into the schism, your disagreement with Eliezer Yudkowsky.

Alex 00:00:39He was incorrect on some points for alignment, but then also not acknowledging that. I think that technical alignment has gone pretty awesome, super awesome.

Liron 00:00:49You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”

Alex 00:01:00Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.

Introducing Alex Turner

Liron 00:01:15Welcome to Doom Debates. My guest has worked in technical AI safety at Google DeepMind for two and a half years, but he just quit, and his resignation is going viral. Why? Alex Turner says Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”

His latest blog post exposes hypocrisy at the highest levels of senior leadership at Google DeepMind, which includes the CEO Demis Hassabis, chief scientist Jeff Dean, and co-founder Shane Legg, among others.

Alex is a world-class AI alignment researcher. He completed a PhD in alignment from Oregon State University. He did a postdoc at UC Berkeley, and he’s earned top distinctions at NeurIPS. I respect that Alex is principled. I respect his mastery of the subject matter that we talk about on this show.

I also find it interesting that he’s levied criticism at the original AI alignment thinker, Eliezer Yudkowsky. He’s called some of Yud’s claims fundamentally misguided, not reasonable, and bogus. As a Yudkowskian myself, I’m gonna be curious to dig into those arguments. And of course, we’ll cover what’s going on right now at Google DeepMind and why he resigned. Alex Turner, welcome to Doom Debates.

Alex 00:02:28Hey, thank you for having me.

From Harry Potter Fanfic to AI Alignment

Liron 00:02:30So it’s great to get you on the show. One thing we do on Doom Debates is we expose top intellectuals who have been pretty familiar to the rationality community or the AI safety community, and we help popularize their ideas, even if I don’t fully agree with all of them. Is that a good description of your background? You’ve been pretty deep into the LessWrong rationality and alignment community for a while.

Alex 00:02:52Yeah, I think it was quite formative. It was just the other day in 2016 where I decided to search what are the top five Harry Potter fan fictions. And that indeed led me down the LessWrong rabbit hole, where I discovered superintelligence in late 2017, and then I pivoted my PhD in early 2018.

From that time period up through maybe early 2023, LessWrong was very central to my professional career, but also just to the way I looked at the world.

Liron 00:03:25Well, I wanna follow up on why did you search for Harry Potter fan fictions?

Alex 00:03:30I really don’t know. It’s kind of one of those things where if I hadn’t done it, my life would be totally different. The reason I’m mentioning this is there’s this famous fan fiction that Eliezer wrote called Harry Potter and the Methods of Rationality. I never really liked fan fiction. I thought it was kind of cringe. No one recommended it to me. So it seems to me like if I’d just woken up slightly differently that morning, I might not have ever been exposed to this research area, and my life would be totally different.

Liron 00:04:02Wow. And you said this is all in 2016, right? So the book had been mostly completed at that time. It had been going on from 2009 to 2015, and you kind of stumbled on it because you were just interested in seeing what the best Harry Potter fan fiction was?

Alex 00:04:17I had a random though…

Get full access to Doom Debates at https://lironshapira.substack.com/subscribe