An underlying assumption of both visions of the future might be completely wrong, and it changes everything.
Once again, my worldview on our future with AI was too simplistic, and I have to re-assess what the next five years will bring.
For the last nine months I’ve written that we’re living in an existential barbell with AI. I’ve written a lot about disruption in education, jobs and the economy. Or… We could live forever.
Last week, my friend Ken challenged it. He argued I was overweighting doom. The same AI that destroys jobs could collapse the cost of living to near-zero. Energy, food, education would be available to all. He argued dystopia isn’t a physics problem, it’s a governance problem (how do we get there).
I’m glad he pushed me, because for the week after writing him back something nagged me.
Ken’s incredibly smart, and we took the same set of intuitions and weight each end differently. On one end, the human species blooms and we’re living forever in utopia. In the other, we’re headed for extinction.
What I hadn’t challenged is whether the barbell is the right shape.
This is the hidden structure of most AI debates. Bloomers and Doomers sound like opposites, but they share an ontology – what AI actually is. Both assume intelligence is optimization power – and this ability scales until it surpasses humans across all domains. They just disagree on whether we can align that power with human interests.
But what if that shared assumption is wrong? What if I’m guilty of not breaking my own frame?
Join me, and down the next rabbit hole we go –
In both the Doom and Bloom scenarios, we short-cut complexity in search of a convenient narrative. But the reality is far more nuanced (for better and worse, as we’ll see):
- In AI Will Take Your Job I discussed Polanyi’s Paradox, the idea that we know more than we can tell. Economist Friedrich Hayek similarly argued that a lot of society’s knowledge is tacit and contextual, generated through interaction. In other words, knowledge isn’t just sitting in a database or email inbox waiting to be extracted.
- I’ve been reading about autopoiesis – a term that comes from biology. It refers to systems that maintain themselves through continuous coupling with their environment. Think of how an organism isn’t just sitting static in an environment, but is constantly exchanging with it to stay alive.
- My current tinkering is in embodied cognition (how does the “body” substrate, whether biological or silicon, affect goal achievement) – and while I’m not sure I agree, others in the space argue that understanding requires a physical body as part of having stakes in the game.
These are different ways of saying the same thing: it’s possible some knowledge can’t be extracted easily because it doesn’t exist until the interaction creates it.
The implication: my existential barbell might not be the right shape. If we examine this more, our likely outcomes could be the fat, messy middle – because the easy narratives are missing something important.
Let’s look at how the risk we face isn’t superintelligence that’s capable of making us gods or skeletons, it might be AI that’s capable enough to be both helpful and dangerous while being systematically wrong about its own limits.
What is “Smart,” Anyway?
I’m very not smart at a lot of things. I have a terrible memory, I especially can’t remember people. But I remember patterns and think in dependency chains, and figure it out by tinkering. My wife, on the other hand, can remember personal details about several thousand patients, absolute fucking sorcery, but will sooner sit in the dark than test which circuit breaker to flip to get the lights back on.
So when we ask “how smart will AI get,” we’re assuming we know what smart is.
But there are two fundamentally different answers, and which one we hold determines everything downstream: what AI can do, when it will do it, and whether the abundance-doom barbell is the right shape.
- View A: Cognition is computation. The brain is a computer made of meat. Intelligence is pattern recognition, optimization, prediction. Given enough compute and the right architecture, any cognitive capacity can be replicated. This view built the AI systems we have. It feels science-y, and left-brainers like me are attracted to it.
- View B: Cognition is embodied. Understanding isn’t just pattern-matching over data. It requires grounding through interaction with the world. Some knowledge doesn’t exist prior to the process that generates it. You can’t extract it from a database because it was never in a database. Pieces of it may be, but it’s not all there. It feels a little squishy.
View B sounds philosophical. It’s not. You already know it’s true from experience: the nurse who knows something’s wrong before the vitals show it. The founder who can’t write down what makes a good hire but knows it in the first five minutes.
The way your kid learned to catch a ball wasn’t by reading physics equations, it’s by catching the damn ball.
I’m not being mystical here, claiming there’s some special animal spirit that only humans have. Rather, that some knowledge is constituted through interaction. Polanyi called it tacit knowledge. Hayek called it local knowledge. The embodied cognition folks call it “enaction.”
Different names, but it’s the same idea: some knowing requires being there, in a body, in a context, with something at stake.
If this is true, then there are things AI will not learn just by scaling compute and data. This doesn’t mean AI will never get there – we don’t have some magic force field protecting us from obsolescence. But it may mean there’s no ‘foom,’ no overnight leap to godhood. Or if there is, it’ll be jagged. Until humanoid robots roam about interacting with us, gaining tacit knowledge from each interaction, there’s runway.
So in this world view the question becomes: what happens in the messy middle?
The AI Content Human Centipede
It looked stupid, so in 1996 I didn’t go to see Multiplicity in the theater. Now it’s teaching me a lesson. Michael Keaton clones himself to balance work and family, and… The first clone is fine, it’s basically a more-motivated him. The second clone is a little off. By the fourth clone, you get a guy who eats pizza with a spoon and can barely form sentences.
Even you young ‘uns who never made mix tapes or photocopied a photocopy intuitively understand: copies of copies degrade. Something essential gets lost.
When GPT-3 came on the scene, my prediction was an extension of Dead Internet Theory, mine was The AI Content Centipede – it’s what happens when crap AI content is used to generate more crap AI content (for those not getting the Human Centipede reference, do NOT click on this link unless you are prepared to be very, very disturbed).
First-generation AI content – the stuff trained on human writing, human art, human code – is now damn good. But it’s parasitic on human grounding. There’s something to the texture of human output that makes the first imitation work. Despite – or perhaps because – human creative output is highly variable and has large amounts of uniquely human crap, it works well. The AI didn’t grieve; it learned from text written by people who did and wrote a poem that felt mostly right.
Now we’re training AI on AI outputs. Second-generation content. This is parasitic on the parasite.
And it’s getting weird as the centipede grows. Not just lower quality, it’s getting kind of uncanny. It’s coherent, and when I see it, it’s hard to put into words why it’s wrong (or bad), it just is. It’s just off.
What’s being lost isn’t “data quality,” it’s grounding. Human content is tethered to embodied experience, and right now our would-be AI overlords (or saviors) don’t have that. They simulate it really, really well (and better by the day) in the first generation. But each step the tether stretches thinner. Eventually it snaps, and we’re left with AI Michael Keaton eating pizza with a spoon.
But here’s why this matters beyond crappy SEO articles and weird AI art: the same principle that we can all see for AI content generation applies to how AI will do things in the real world.
This is the autopoiesis problem from earlier. Living things stay coupled to reality with a feedback loop that includes ongoing interaction. Skin in the game (literally). AI doesn’t yet. It processes representations of representations, untethered from the ground truth those representations once pointed to.
It’s not just a content/data problem. It’s a world-model problem.
An AI planning real-world actions is building on the same ungrounded foundation. Its model of reality is parasitic on human perceptions of reality, interpreted through language. We have experience, filtered through our unconscious brain processes, transmitted through a low-fidelity narrow pipe of language. The AI consumes this, and outputs same, and on we go.
This opens a gap in the doomer argument of “Even if AIs don’t have hands, they can just use humans to do real-world work” (claim # A-1 in my article Is AI Optimism Just Hope and Handwaving) – the grounding layer is still one level off.
If the AI acts on a world model without its own grounding, it faces drift – it becomes confident but miscalibrated.
This is where risk lives: in systems that think they know, while the tether to reality stretches thinner as datacenters fill with steaming piles of AI slop.
What This Predicts About AI
So if this “grounding thesis” is right – if some knowledge requires embodied interaction and can’t be extracted from data, the important question becomes “what does this tell us about where AI is going?”
It predicts the jagged frontier of AI capability isn’t random.
In The One AI Metric That Matters in 2026 I focused on the METR statistics and that the important number to watch is “time to failure,” in other words, how long can AIs run unsupervised.
I still believe this is the most important metric, but the side note I left – that the real world is messy – gets renewed focus. Because if the embodied cognition view is right, the jaggedness has structure and where we see differences in Time to Failure is the signal:
- The Deep Incursions Into Human Territory: Domains where performance comes from pattern-matching over large datasets. Language, code, image generation, game-playing, test-taking, classification, prediction from historical data. These are “View A” tasks (cognition as computation). AI excels because the knowledge is in the data, waiting to be extracted.
- Where We Have More Time: Domains requiring tacit knowledge, contextual judgment, or understanding constituted through interaction. Physical reasoning in novel situations. Reading a room. Knowing when the standard playbook doesn’t apply. Generating solutions that emerge from the negotiation itself. These are “View B” tasks (cognition is embodied), and they stay “valleys” of AI advancement not because we need more compute and scale, but because the knowledge was never in a database to begin with.
If I’m right, this is a long(ish) term feature of the landscape, determined by the structure of knowledge itself. That sounds fancy, but it just means that “not all knowing is the same, and the kind of knowing determines what AI can do about it (for better and worse).”
Now here’s where it gets dangerous:
AI systems don’t know which side of the frontier they’re on.
They’re trained to be confident, and produce outputs with the same fluency whether they’re pattern-matching from solid ground or eating pizza with a spoon. This is the Dunning-Kruger effect for machines: systems confident in their models may not even know what they don’t know.
In fact, LLM overconfidence is empirically documented. Recent research shows LLMs are systematically overconfident about their capabilities and cannot retroactively adjust their confidence after learning they were wrong (unlike humans).
This means AI will be like my 18 year old daughter, who thinks that with six months of Muay Thai she can defeat an attacker who has 50 pounds on her: most confident, precisely where least competent.
In non-existential situations, it means AIs making bad decisions. The coding assistant that’s genuinely superhuman at syntax might be dangerously wrong about whether the code should be written at all. The strategic advisor that synthesizes information brilliantly might miss the thing that anyone in the room would have felt.
In existential situations, it means AI gets an idea in its head about something it wants to do which is unaligned with our interests, and executes not knowing it’s going to fail. In other words it doesn’t have to be competent to be dangerous, only confident enough to try.
For the middle of the barbell scenario, this becomes the risk: AI that’s smart enough to be credible, confident enough to act, and systematically miscalibrated about where its competence ends.
They don’t know what they don’t know. And worse, neither do we – until something breaks.
This is the shape of the messy middle. Not AI that can do everything, or that can do nothing. AI that’s transformatively powerful in the peaks, confidently wrong in the valleys, and navigating a world where it can’t yet tell the difference.
Reframing Our AI Future
So let’s revisit the Bloomer/Doomer debate I opened with. Are we all gonna die, or are we going to live out a “Star Trek future” (which I will advocate against until I die, because we must at all cost avoid a skintight leotard future)
Today I’ve been constructing the case that both camps might be wrong. The stakes don’t change, but the distributions of outcomes (all-or-nothing), how it will happen, and the risk profile do.
Here’s why:
Bloomers and Doomers disagree about outcomes, but they agree on the ontology.
They share View A: intelligence is optimization power, and it scales. Give AI enough compute, enough data, enough architectural cleverness, and very soon it will eventually surpass human capacity across all domains.
From there, the debate becomes about alignment:
- Bloomers: We’ll figure out how to point that optimization power at human flourishing. Therefore: abundance, solved problems, radical improvement in quality of life.
- Doomers: We won’t figure it out in time, or it’s impossible. The orthogonality thesis – that intelligence and goals are independent – means a superintelligent AI could optimize for something utterly alien to human values. Therefore: existential risk.
This is why the debate feels so stuck. We’re arguing about a parameter (can we solve alignment?) within a shared frame (intelligence is optimization that scales without limit). If we accept the frame, we have to pick a side.
We create the Silicon Golem, and either we align him, or he destroys us. Barbell.
But what if the frame is wrong?
The embodied cognition view – View B – doesn’t answer the alignment question. Which I don’t like, and neither will the ‘abundance’ folks, because it doesn’t give a neat and clean answer. It’s messy.
What it does instead is change the nature of the discussion to something that (at least to dorky me) is a very interesting question: if some knowledge requires grounding through interaction, then pure optimization can’t do everything, no matter how much raw compute you throw at it.
The scaling stops, or at least slows dramatically, until the embodied cognition catches up to physical world tasks. We start to look at where the jagged frontier is to calibrate risk and opportunity, and we start to look at what AI thinks it understands about the world.
The barbell turns into something messier. Not “abundance or extinction” but a complicated transition with AI that’s transformatively powerful in some domains, persistently limited in others, and dangerously overconfident about which is which.
And thus, it changes the role of humanity and what the world will be for our kids. Human relevance in this scenario isn’t wishful thinking or species-level cope. We are necessarily involved, even if we can’t stop the trip down the greased slide towards our weirdly cyborg future. There are kinds of knowledge-constitutive work that require humans in the loop, because the knowledge doesn’t exist until the interaction happens. Someone has to be in the room, have skin in the game.
This reframe matters because it changes what we should actually worry about and do:
- The Bloomer frame (accelerate toward abundance) underestimates the danger in the transition – trusting that the future is default-good and missing the messy middle where miscalibrated AI can do real damage if we trust them before they know their own limitations.
- The Doomer frame (slow down or stop) may be fighting the wrong enemy – shouting about superintelligent takeover that the majority of people will tune out, while the actual risk is a thousand confident-but-wrong systems making high-stakes decisions in domains they don’t understand.
At the risk of taking the analogy too far, the fat middle of the barbell is harder to grasp. It’s not as dramatic as either utopia or extinction. But it might be closer to true.
Here’s Why It Matters to Us
If the messy middle is real, what we do today matters.
The simplistic abundance narrative has us just waiting, because we’ve chosen to believe it’ll all just be OK, and we can go about our lives as today. The doomer narrative tells us we should be afraid.
The barbell made it easy to defer – one side didn’t require you to think, the other required you to think there wasn’t much to be done. The lesson I took from that way of thinking was to savor the time we have left, the only sane choice for a polarized outcome.
But if you’re undecided, or if – like me – you oversimplified the world, it might be worth asking: what in my world is truly on the human side of contextual judgment? For how long? How soon before the jagged edge cuts me?
This updates our vision for what the world looks like in the liminal era.
I’ll be honest, this reframe is much harder to digest. I want something simple. To know if we’re going heaven or hell, for it to not be so messy. Humans want to belong to a tribe of belief and we will create narratives to fit our predisposition. But that’s a recipe for regret.
So until Skynet gets us, or we’re spacefaring in a leotard, maybe there’s a new puzzle to solve.
