• Nate Soares said he stopped adding to his 401(k) because AI may kill us within a decade.
• The MIRI president said humanity has a more than 50% chance of dying within a decade.
• Elon Musk said AI could make retirement savings irrelevant by creating an age of abundance.

Saving for retirement is a bet on the future. Nate Soares, the president of the Machine Intelligence Research Institute, isn't so sure AI will let us get that far.

"I do not save for retirement right now. I have a 401(k) from when I worked at Google, but I haven't contributed for a decade," Soares told The Atlantic CEO Nicholas Thompson in a recent interview for Thompson's "The Most Interesting Thing in AI" series.

Soares' decision reflects a grim view of the AI race.

"Godfather of AI" Geoffrey Hinton and Anthropic researcher Evan Hubinger have, respectively, put the odds of AI killing everyone within a decade at 10% and more than 10%. Soares is more pessimistic: He told Thompson he believes there is a greater-than-50% chance AI causes human extinction in less than a decade.

Soares, who is also the coauthor of "If Anyone Builds It, Everyone Dies," has spent more than 12 years working on AI alignment, the field focused on ensuring powerful systems act as intended. Before joining MIRI, the Berkeley-based nonprofit focused on AI risks, he worked as an engineer at Google and Microsoft.

Soares said that a sufficiently capable AI could develop goals at odds with humanity's, evade oversight, improve itself, and build the automated infrastructure needed to operate without people.

From there, he said, it could reshape the planet around its own objectives, potentially making it uninhabitable for humans — a warning he shared with Business Insider last year.

A 401(k) for the apocalypse?

The question of how AI will shape the future has also prompted Elon Musk to question whether retirement savings will matter. Musk's vision of the future, however, is radically different than Soares'.

In January, the Tesla and SpaceX CEO said people should not worry about "squirreling money away for retirement" in 10 or 20 years, predicting advances in AI, robotics, and energy will make retirement savings "irrelevant." (Personal finance experts and AI gurus told Business Insider the opposite: You really should be saving for retirement.)

Musk has said that supercharged productivity could create an age of abundance, in which everyone has access to a "universal high income," cheap goods and services, and better healthcare and education.

Musk has also said money could become irrelevant in a future where AI and robots outperform humans at virtually every task.

For Soares, the prospect of a vastly capable AI is an existential danger, not a route to plenty.

Echoing calls from tech leaders to "pace the frontier" of AI development, Soares said governments should halt the training of general-purpose AI systems. They should focus on developing technology for applications like cancer research and safer self-driving cars, he said.

The solution, he told Thompson on Wednesday, "is to not build the thing that would kill everybody."

"I do everything I can to stop this eventuality," he said. "I don't think it's inevitable."

• Satya Nadella says he changed his mind on AI alignment, or ensuring AI aligns with human values.
• Nadella said he previously thought alignment would just happen as AI capabilities improved.
• "I no longer believe that," Nadella said on the 20VC podcast.

Microsoft CEO Satya Nadella says he has changed his mind about the idea that alignment would naturally emerge as AI systems became more capable.

On a new episode of the 20VC podcast, host Harry Stebbings asked Nadella what he'd changed his mind about over the past year. The CEO said he had previously assumed that as AI became more capable, alignment — ensuring AI behaves in line with human values and goals — would automatically follow.

Instead, he said, while scaling has played out as he expected and the models' capabilities have continued to improve, alignment remained a challenge.

"I used to always think that alignment would just fall out," Nadella said. "I no longer believe that. I think this alignment thing may be a harder, different problem."

The Microsoft CEO previously warned about the risks of AI and argued for greater external controls in a post on X on October 10. Nadella said companies should treat AI models as insider threats and assume they are "compromised."

"We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," Nadella wrote.

His belief that AI's increasing capabilities won't automatically ensure human control is in line with warnings of other tech leaders. Both OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei have warned about the threats of AI. A letter signed by more than 1,000 AI workers in July warned that AI capabilities could advance faster than people's ability to understand or control the systems.