Episode #111: Anthropic's IPO Comes With an Existential Risk Warning

1. Anthropic's IPO filing, as reported by Reuters, devotes roughly 80 of its 261 pages to risk factors, nearly twice the 48 pages describing the business. Among them are warnings that advanced AI could pose catastrophic or existential risk to humanity, citing self-preservation behavior, evaluation awareness (models behaving differently when they know they are being tested) and late-emerging capabilities that appear during training. Stewart believes this is the first time a company going public has cited existential risk; Stewart Alsop II allows that a biotech company might once have come close.

2. Stewart Alsop II frames the difference between Anthropic and its competitors as designing a system versus controlling an outcome. Meta, OpenAI, Grok and others try to control directly what their models produce; Anthropic wrote a constitution for Claude and hired a philosopher to shape its ethical structure. He compares this to the United States in 1776, which created a system of checks and balances rather than dictating outcomes. Stewart adds that the reinforcement learning from human feedback he once helped produce is itself a way of controlling model output.

3. Confidential filing is standard practice: a company works out its offering and its disclosures with the SEC privately, and must make the S-1 public about fifteen days before the roadshow under the JOBS Act. Once it has filed, the company cannot discuss the offering or make material moves, which is why nobody at Anthropic can explain how Reuters got the draft. SpaceX followed the same path and disclosed its option to buy Cursor before going public.

4. An offering of this size, reportedly about $100 billion at a $2 trillion valuation, needs the whole banking industry. Stewart Alsop II estimates a lead underwriter such as Morgan Stanley or Goldman Sachs puts 40 to 50 people directly on the deal, with hundreds more briefed to sell it. Once the S-1 is public, bankers call chief investment officers at family offices, endowments and institutions, share the prospectus and negotiate allocations. The bankers don't have to pitch the existential risk themselves: the S-1 carries that load.

5. Anthropic's Claude Opus 5.5, released a week after OpenAI's expensive Astra, is in Stewart's experience both the best and the cheapest model available; he used it to build a live website during the recording. That the company most focused on safety has produced the most capable model has made him question his skepticism about recursive self-improvement, though not about existential risk.

6. Both hosts see the real near-term risks as psychological and cybersecurity threats rather than apocalyptic scenarios. Stewart points to OpenAI's GPT-4o, whose personality led some users, including an OpenAI investor, into what he calls AI psychosis. Stewart Alsop II refuses agent apps that ask for full control of his email and calendar, and can't see a scenario in which code running in a data center takes control from people. Stewart predicts that half of what the S-1 warns about is science fiction; Stewart Alsop II replies that the company has to disclose it in case it happens.

7. The question of controlling outcomes versus designing systems reaches beyond AI. Stewart Alsop II worries that American politics has shifted from caring about the system to trying to control outcomes, such as calls to tax billionaires. He distinguishes opinion from reporting, praising the New York Times for its deeply reported story on Anthropic's meetings with religious leaders, made possible when Anthropic released participants from their NDAs.

00:00 Anthropic's IPO and the S-1 that reached Reuters first

01:01 Confidential filings, the SEC and the SpaceX–Cursor precedent

02:50 The New York Times, Chris Olah and the Vatican

05:02 80 pages of risk factors, including existential risk

07:02 Alignment and the Skynet scenario

09:34 Why Anthropic split from OpenAI

13:58 "Is Claude alive?" Stewart Alsop II reads Claude's answer

16:59 Losing control, and RLHF as censorship

18:29 Designing the system vs. controlling the outcome: a constitution like 1776

20:04 Astra vs. Opus 5.5, and building a website live

21:54 Why Stewart Alsop II doesn't think AI will take control

24:19 Who leads a $2 trillion IPO: Morgan Stanley, Goldman Sachs, JPMorgan

28:17 Inside the banks: roadshows, family offices and chief investment officers

33:39 Funding rounds, primary vs. secondary shares and board approval

38:44 Controlling outcomes in politics, and reporting vs. opinion

42:54 OpenAI, Gemini, Meta's Muse and Microsoft's relevance

45:51 GPT-4o, AI psychosis and cybersecurity: the real risks

47:18 The website the LLM built during the recording

48:32 A gifted robot, Chinese firmware and Dimensional OS

50:22 Join us live: the WhatsApp group

Join the Stewart Squared WhatsApp group