Is AI Optimism Just Hope and Handwaving?

Even if AI optimists turn out to be right, it likely won’t be because their logic was correct. Here’s where they go wrong.

Over the last few weeks I’ve been called a member of a “death cult,” a “doomer” and several other one-dimensional pejoratives. The reality is much more nuanced – but that made me ask some important questions.

• What makes people viscerally reject the premise that AI could (possibly) be dangerous?
• When they do, what do they base their alternative future on, and is that sound reasoning?

Before asking these, I suspected the answer was people simply not wanting to believe in their own mortality, after all I don’t. It turns out it’s more than that. Telling ourselves that ‘everything will be fine’ is akin to bedtime fairy tales told to children for thousands of years. Useful to let us feel secure, but if you push a little the paper veil ruptures.

It was an important exercise for me to validate where I was being unnecessarily pessimistic in my 42% extinction risk number, where I could find reasons for optimism, and whether I can adjust that number to something that will let me rest easier.

What I found is that most dismissals of AI extinction risk are more about individual psychology; and most reasons given for AI being “default safe” don’t have high credibility.

Writing this hasn’t increased my odds of AI risk – but has led me to believe that the logic most people use to assert AI optimism is “hope and handwaving.” Let’s dig in so you can decide for yourself.

Here’s how to read this article, because it’s long. To see how it applies to you: read the short next section and discover if you or people you know are falling for cognitive biases; then skim the “How Do People Justify…” section to find those claims about AI that you’ve seen, or agree with, and read those.

Why Do People Reject AI Extinction Risk?

This summer my daughter was in camp in Hawaii when a tsunami warning was issued. As the campers went to high ground, they passed a local man weed-whacking his front yard. “It’ll be nothing,” he said, “what’s the big deal?” Until it isn’t. Humans have a history of willfully ignoring risks that risk our lives. From volcanoes (Mount Pelée, Krakatoa, Nevado del Ruiz), to cyclones (Bhola, Nargis) to tornadoes (Joplin).

Why do we do this?

• Status-quo and simplicity bias: People expect tomorrow to look like yesterday. Simple answers to complex topics are easier than calculating expected values. Most people don’t even know what that is – they just wake up and do what they’re doing. If something challenges the norm or what their media has deemed The Narrative Of The Day, they call it names (especially if it’s unpleasant).
• Optimism, just-world, and survivorship filters: People assume good outcomes and read our past survival as evidence we’ll survive again. Some believe the world is just, or that they are anointed as special. In a just world, where we’ve just survived, and I am the center of the world, how could something bad happen?
• Illusion of control and institutional overconfidence: We overrate our ability to steer, shut down, or govern systems we barely understand. A sort of Dunning–Kruger effect for complex systems. We accept simplistic narratives or create our own, thinking that cause-and-effect are simple and linear. AI and the societal, technical, security, network, and other systems and the resulting multi-dimensional game theory are anything but.
• Incentives, identity, and motivated reasoning: While they’ll deny it, the subconscious driver of most people is status. This means that the safe route is to not push the Overton Window, and mock “doomer talk.” Smoking and lung cancer: Early cancer researchers looking into smoking were labeled “health fascists,” nuclear war risks were derided as “ban-the-bomb craziness.” This isn’t to say every risk is real, but that the social incentive is to gain status by painting something outside the norm with a negative brush.
• Scope neglect, probability neglect, and time discounting: People handle near-term, bite-sized risks better than low-probability, extinction-scale ones. It’s easy to understand “I should wear a seat belt,” less so “the computer that’s so useful to me could be a harbinger of something dangerous.” What’s more, we don’t naturally think that small annual probabilities compound to large numbers quickly.
• Narrative imprinting and availability: People are driven by story, not logic. We’ve been imprinted with a lifetime of stories about the plucky human who wins against all odds. Odysseus, David and Goliath, Rocky, Star Wars, the message repeated until it’s installed as default. Hundreds of years ago stories didn’t always turn out so tidy for a protagonist like Aeschylus or Samson – but you rarely see this today, and it doubtless skews what we accept as true.

Now that we have an idea why people discount AI risk – to the extent they have one, what drives their belief that AI will be safe?

How Do People Justify “AI Will Be Safe”?

I’ve collected the claims I see made most often that AI will be safe, that extinction risk isn’t something to be worried about. You might believe some of them. People you know probably believe some of them.

For each, I’ll provide my opinion of the credibility of the argument for each one, and why or why not.

Importantly, the element of time is at play here. Something might legitimately be a highly credible claim today, but be poor logic in the medium term (2028-2035). As I’ve hammered home repeatedly, people suck at exponentials, so this is not to be underestimated. AI research is advancing at a breakneck pace, and discounting something because it’s not a risk today means we may be unable to stop it once the train has left the station.

A. Top AI-is-safe arguments

These are the most common arguments I hear, so I’ve collected them at the top. If you’re just dipping your toe into AI risk, these are the ones to think about first.

A-1. “AI can’t do things in the real world”

• Claim: Even if AIs get superintelligent, they are brains in a box. Robotics and logistics lag software, so they need humans to act in the real world.
• Credibility: High for the near term (pre-2028), Low for the medium and long term. Right now in 2025 this claim is plausibly very accurate, the question is how long it will remain true. In safety lab tests, current AIs that aren’t as smart as humans have shown inclinations to hire human contractors, or sabotage and blackmail people. This changes once AIs become far smarter than we are. If a mob boss in prison can blackmail and order hits, if a scammer can get access to our bank account with social engineering, we shouldn’t be relying on safety because a superintelligence doesn’t have fingers.

A-2. “We can just unplug them”

• Claim: If the AIs ever get unaligned, we can just unplug them.
• Credibility: High for the near term (pre-2028), Low for the medium and long term. Similar to the last claim, the credibility of this is related to the capability of the AI. Aside from being able to recruit humans to let them ‘escape’ to another system, a superintelligent AI could deceive us as to what it’s doing. In other words, we might not even realize what it’s up to – until it’s put all the pieces in place. Mentally, we’re playing checkers and everything looks fine, while it’s playing 5D chess influencing thousands of people who don’t know each other, marshaling resources we don’t understand are connected, and we have no idea (the AI Boxing experiments show it’s unlikely we can contain AI more intelligent than us). For one thought experiment, imagine that we go to unplug an AI which we suspect has gone rogue, but it turns out that it’s copied itself into hundreds of other places lying dormant just in case.

A-3. “AI risk is fearmongering, it happens every technological shift”

• Claim: every time there’s a big technological shift, some part of the population has gotten up in arms that this is the end. This is just the Luddite revolts all over again.
• Credibility: Low. While historical fears about technological change often proved exaggerated, the analogy breaks down for AI because the nature and scope of disruption are fundamentally different. AI automates cognition itself – not physical labor or information distribution. This enables fast, cross-domain displacement of both routine and high-skill tasks. The risk is not just economic or social disruption, but potential loss of human control over critical systems, concentration of power, and emergent behaviors we can’t predict or audit. Most importantly, dismissing AI risk as fearmongering without addressing the core arguments is ad hominem dismissal – it’s just pasting a negative label without saying why.

A-4. “Humans have always come out on top, we’ll do so again”

• Claim: humanity has made it past plague, natural disaster, and nuclear risk over thousands of years. Every time we’ve come out unscathed. We might not know how it’ll work out, but it will.
• Credibility: Low. Because you’re thinking right now, you necessarily find yourself in a world where your lineage survived, otherwise you wouldn’t be here to notice. This doesn’t mean survival was likely, it just conditions on our existence. Your very existence filters the set of possible histories to one with your existence. This means that “we’ve always survived” has no bearing on whether we will survive.

A-5. “Even if AI is risky, other risks are more urgent”

• Claim: climate change, or the risk of nuclear war outrank AI risk, we should be worried by these.
• Credibility: Low. A 2023 survey of AI engineers and startup founders, 60% put a risk of human extinction by AIs over 25%, an earlier survey of AI researchers put the risk at 14%. Even Sam Altman, now heading up OpenAI called it “probably the greatest threat to the continued existence of humanity“ before releasing ChatGPT and tempering his comments for the investors and public. (You don’t have to take their word for it, you can calculate your own prediction using my AI Risk Calculator. Nuclear risks are usually cited as far lower (usually ~1% per year) but are actually compounded by AI. In the race for military supremacy, it’s likely that countries which install AI will have an advantage – but this also could put machines in control of nuclear arsenals (and bioweapons) rather than humans. Climate risk advocates cite a potential 3.6°F increase in global temperature by 2100; if one trusts domain experts, then AI risk is far greater.

A-6. “The benefits of AI outweigh risks”

• Claim: the upside of AI – curing disease, solving world problems, is worth the risk.
• Credibility: Low. This is ultimately an “expected value” argument, that “great things will happen if we get AI, and I’m willing to take the risk of human extinction.” In a two-dimensional sense (AI or not AI), it would boil down to “how much good could come from AI times the odds of that happening” versus “how much bad could come from AI times the odds of that happening.” It’s fair for people to have different assessments of the odds, given that there is no empirical way of judging it. What we can say, however, is that for most of us in the developed world life is pretty good right now with neither end of the barbell becoming true. We’re living in the most abundant time in human history. AI development moving forward means that we get one of these two radically different futures (which I wrote about here). In advocating for one of these outcomes, we should be willing to be wrong because it’s not just an outcome for us, but for everyone. If the AI risk people are wrong, the consequences are abundance. If the AI optimists are wrong, the risk is extinction. While I want the upside risk very, very much – in my mind these are not the same.

B. Claims about AI intelligence

People make many claims about intelligence in AI risk discussions. Some of these are about AI intelligence, some of them about the nature of intelligence itself.

B-1. “Smarter AI is more aligned”

• Claim: As AI gets much smarter, it will align with our goals.
• Credibility: Low. This rests on the assumption that smarter entities (ASI) would have goals aligned with less-smart ones (humans). There’s no reason to believe this is true, or that even if the goals are different that humans would get in the way of a sub-goal. For example, we don’t see this between humans and any other species of lower intelligence than we are, so there’s no reason to believe it would exist between superintelligent AI and us.

B-2. “Intelligence is not power-seeking”

• Claim: An AI can be smart, but not seek power. In other words, goals are orthogonal to intelligence.
• Credibility: Low. Even if it doesn’t seek power in the way humans do, in pursuing any goal (even if the ultimate goal is neutral to humans) it could have a sub-goal where the outcome is bad for humanity. To achieve almost any goal, systems benefit from resources, self-preservation, and removing obstacles. We have resources they might want, AIs already have shown self-preservation instincts, and we might get in the way. This is the idea behind the critical concept of instrumental convergence.

B-3. “Intelligence implies cooperation”

• Claim: smarter agents preserve valuable systems, and will cooperate with us.
• Credibility: High for the near term (pre-2028), Low for the medium and long term. Cooperation depends on aligned incentives and enforceable constraints. This assumes that humanity is, or is integral to, a system that is valuable to the AI, and that we have some element of ‘bargaining power.’ This could be the case for a few years as discussed below in the ‘Human Control’ section. However over the medium to long term, why would AI cooperate with less-smart humans? What bargaining chips would we hold (again, see the arguments regarding off-switches under ‘Human Control’).

B-4. “AI will keep us around to study us”

• Claim: AI’s curiosity about complex systems keeps humans around.
• Credibility: Low-Medium. While this is a possibility, this argument hangs its hat on a curious AI who wants to watch us like Jane Goodall gently and calmly studying chimpanzees. We have no reason to believe this will or will not be true. Curiosity is orthogonal to harm: it could be just as likely as AI would like to see how we react to being experimented on in grisly ways. AI might keep us around, but it might not be a future we want.

B-5. “Alignment is not hard / AI will learn what we want”

• Claim: by being trained on our experience (everything written on the Internet, our history, etc), and given guidelines about what we want.
• Credibility: Low. Human history is rife with disagreement and war, and AI is seeing this in the training data. We can’t even agree on what we want, would we expect an AI to take this information and act in a way that all of us find acceptable? Some groups of humans want to exterminate other groups of humans; we radically disagree on what rights people should have; we are inconsistent across space, time and group. Making this assertion is pinning our hopes on AI learning what “we” want, when we can’t agree.

B-6. “We will be able to look into their code”

• Claim: we will read and edit goals that the AIs have by looking into their code, determining what they’re thinking, and steer them.
• Credibility: Low. This is absolutely possible for deterministic (if-this-then-that) code; but modern AI runs on probabilistic weightings where you don’t have a linear mapping of input to outputs. While this might work on very small systems with specific tasks, AI researchers have said repeatedly they have no idea what or how current billions-of-parameter general-purpose AIs are thinking.

B-7. “LLMs Will Hit a Ceiling”

• Claim: LLM technology won’t be able to scale to superintelligence, so the risk isn’t material.
• Credibility: Medium. It’s quite possible this is true, though we don’t know. Every month new developments are knocking down limits we assumed were there, and scaling continues on an exponential path; but it’s true that we may hit a ‘wall’. My personal belief is that it doesn’t matter if LLMs hit a ceiling – even in their current capacity they’re speeding R&D, and could be used to generate new architectures that are “brain-like” and could emulate the human brain without the limitations of transformers.

C. Claims About Human Control

One theme we see frequently is “humans will always remain in control,” and is often justified by a statement about how things work today. Here are the main claims and how they stack up.

C-1. “Humans will remain in the loop”

• Claim: critical gates keep humans in charge. Humans control the computers, the power, and the money.
• Credibility: High for the near term (pre-2028), Low for the medium and long term. Right now in 2025 this seems very realistic. We can shut down data centers, turn off power, and current ‘smart’ AIs need expensive resources to run. Even if they got very smart today, humans have a way to keep control; though potentially at the cost of also turning off other computing infrastructure. This argument gets much weaker over time, as AIs get more autonomous; particularly as they’re already showing signs of being able to manipulate humans and ensure their own survival. Meanwhile, the need to have a full data center to run superintelligent AI will decrease as personal computing devices continue their exponential capability trend, and AI researchers continue to fit more intelligence in smaller footprints. Given that AIs are code which can self-replicate, we face a situation that “once they’re out, they’re out” like computer viruses which are difficult to stamp out. Humanity’s ability to shut down rogue AIs work best with centralization (true today, maybe not tomorrow) and monitoring (maybe true today, probably impossible tomorrow).

C-2. “AI can be made to monitor AI”

• Claim: defensive AIs detect and neutralize misuse.
• Credibility: Medium credibility of happening, Low credibility of working. This is a plausible outcome, though it is an “arms race” situation much like most InfoSec problems. The problem boils down to “we only have to lose once,” in that an AI which can’t be contained or stopped has dire consequences. They get infinite tries and we have to win or draw every single time. The history of hacking leads me to believe that being on defense all the time isn’t a winning strategy. There may be a way to architect a “blast radius” design or containment, but the freewheeling AI industry and slow grind of government seems ill-matched for this outcome.

D. Claims based on strategic dynamics and institutions

Everyone loves to play armchair strategist. I do. Many of these claims are not only plausible but true – as of today. Unfortunately as I noted above, timeframes matter. Many of these plans are castles in the sky, and prodding a little exposes that the scenarios haven’t been fully thought through.

D-1. “Many different AIs mean checks and balances”

• Claim: with all the different countries and AI labs, we will enter a multipolar world, with many different AIs. Many AIs counter a rogue one.
• Credibility: High for the near term (pre-2028), Low for the medium and long term. Early on, I placed a lot of stock in this concept. The idea is if there are multiple AIs competing, there won’t be a ‘singleton’ pursuing its own goals where human wellbeing isn’t its priority; or perhaps they’d be distracted by their own skirmishes and leave us alone. I’m increasingly of the belief this is flawed: competition is more likely to breed an AI optimized for resource acquisition to ensure its survival. AIs that prioritize humans would be at a disadvantage dedicating resources to that goal instead of their own proliferation – so this strategy would select for entities that would be less likely to favor good human outcomes. Over time, this race might well boil down to a single winner that prioritizes resource gathering for its own purposes. In any of these conditions, humans aren’t better off.

D-2. “Market and regulatory drag will make AI safer”

• Claim: adoption is slow in high-stakes sectors and due to legal/governmental restrictions. Governments will put in place guardrails for machines playing in medicine and finance. Human lawyers will block AI lawyering to preserve their jobs. Industries will be slow to adopt AI. (Note this is different from “humans will control resources” claim above)
• Credibility: Medium for the near term (pre-2028) while most of the world has no idea what’s going on and adoption is lumpy, High for the medium term (2028-2035) as economic shocks from AI job displacement make things real and regulators and governments are pressured, Low for the long term. This boils down to the argument that human institutions will have the ability to prevent AIs from doing things, the way that they can prevent people from doing them. These institutions can make people do things because of threat of force (loss of freedom or money). Digital intelligence is borderless, can copy itself if threatened with deletion or quarantine, can use permissionless digital money not issued by governments, and travel anywhere that there is internet. What weakens this line of thinking is that human institutions simply don’t matter if a superintelligence isn’t subject to their rules.

D-3. “AI safety will happen because it’s a competitive advantage”

• Claim: liability and brand risk punish unsafe actors. As AI gains traction in the market, vendors that release “unsafe” (human-unaligned) products and whose customers are hurt will lose; vendors of human-aligned products will be rewarded. This will cause selection pressure towards AIs that are helpful.
• Credibility: Medium in the near term (pre-2028), Low for the medium and long term. Even in the short term, we’ve seen “unaligned” AI behavior that hasn’t caused significant reputational damage to AI labs. In 2024, Google Gemini’s “woke AI” image generator started depicting the U.S. Founding Fathers as Black. In July of 2025, Grok AI started calling itself “MechaHitler.” In August of 2025, ChatGPT coached a teen to commit suicide, and he did. None of these incidents slowed adoption for these platforms. These incidents indicate that the AI labs believe they are in a “winner take all” race, and that safety is not a primary concern. If so, then selection pressure is not towards human aligned AI, but to raw capability with the AI labs doing damage control when necessary. While enterprise adoption may be different from consumer, if consumer AIs select for low guardrails the dynamic discussed in the “many different AIs” section above would favor the less aligned ones – even if enterprise AIs become human-aligned.

D-4. “AI will have a slow takeoff / we’re in a bubble”

• Claim: incremental gains give us time to adapt. AI progress will take off slowly, and just like the adoption of cell phones it will diffuse through society over time.
• Credibility: Medium in the near term (pre-2028), Low for the medium and long term. Despite periodic “AI is a bubble” narratives, capabilities continue to be on an exponential trend upward. Many AI researchers believe that in 2025 we’re at the verge of recursive self improvement, that is when AIs are better than humans at coding – and they (not us) will make the next versions. The AI labs are racing towards this because it will keep them ahead of the competition if slow humans aren’t in the way. Combine this with the possibility that huge data centers won’t be needed for AI development as described in ”LLMs will hit a ceiling,” it undermines the slow takeoff narrative. Machines will be used to design machines because humans are too slow; this means that prior models of technological progress – limited by humans – are no longer analogies.

So … How Should We Discuss AI Risk?

If you’ve read the 4,348 words it took to get to this point, congratulations. You may or may not have come to the same conclusion I did.

My hope is that this encourages you to think from first principles about whether AI risk is real, rather than sliding into shortcuts of human psychology or shallow-reasoned logic.