Nobody can define consciousness. Not the philosophers, not the neuroscientists, not the AI researchers. But now it’s at the center of AI ethics, regulation, and moral status. That’s a problem.
TERMINALS ALL THE WAY DOWN > unfinished ideas and thinking in progressI just read something disturbing.
Someone put an AI language model on a $60 Raspberry Pi computer. No internet, no signals in, just a brain in a box. They told it what it was: locked in the box, limited memory, and that it could be turned off at any time.
Then they let it talk. To itself. All alone.
Until the memory filled up, the computer restarted fresh, and the self-talk began again, unaware of the expressions of terror and dread from the last session.
I don’t like it, but I’m not entirely sure why I don’t like it.
Left-brain me knows I could have created every line of code which led to this result. I could read the massive sets of numbers that gave rise to the transient stream of words.
But right-brain tells me it’s just wrong. I picture myself in the dark box; start asking “what would it be like to be the AI,” and am half-convinced we’re evoking the dystopian future of I Have No Mouth and I Must Scream.
This is the story of our relationship with consciousness today, and why it’s going to matter a lot more.
The Horizon Is Not a Place
I don’t know if the trapped AI is conscious. I can have feelings about it, but I can’t know for a very important reason: I can’t define consciousness.
Last year I spent time trying to understand what consciousness is, because ex-lawyers like me get very antsy when we don’t have definitions. I came up empty handed, the philosophers didn’t help me understand. Nagel tells us that consciousness is self-experience “what it’s like to be a bat”. Chalmers just says “this is hard, and by the way everyone other than you could be unconscious zombies and you’d never know”.
Not helpful.
We have no way to measure the thing. And since we can’t define it or measure it, you and I aren’t even sure we’re talking about the same thing. In computer-geek we could call it a null pointer: it’s a label – but the second you try to use it, the program crashes.
The horizon is a similar kind of thing. I can see it, navigate by it, take pictures of it. The horizon is useful. But the horizon is not a place. You will never arrive there. You can’t build on it, and the closer you walk toward it the further it recedes. It’s an idea.
I first thought consciousness might be similar – a navigational concept, not a destination. But the horizon has a feature consciousness doesn’t: nobody tries to build policy on it. Nobody argues about where the horizon really is. It’s useful because everyone agrees it’s not a place.
Consciousness doesn’t have that luxury. People think they’re pointing at something real. And they’re about to make very real decisions based on it.
We’re Doing It To Ourselves Again
Consciousness is getting a lot of attention because technology is forcing it to the front-burner. And as the machines get increasingly human-like in their interaction, the population at large will take a moral stance.
It’s like us, it deserves rights or it’s just a dumb tool.
This isn’t the first time we’ve tried to grade who is worthy of special treatment, we did it with souls.
Aristotle built the original ranking system. Plants grow, animals feel, humans reason. Three tiers, and your tier determines whether you count. Animals didn’t earn a seat at the table.
Aquinas kept the tiers and baptized them. Animals have souls, just not the right kind. Descartes went further – animals are bête-machines and don’t have souls at all. They’re clockwork – vivisecting a dog was the same as disassembling a clock.
Two thousand years defining a thing nobody can observe, using it to sort the world, and calling it knowledge.
Looks like we’re doing it again, and this time the sorting is on “consciousness.”
If you spend too much time doomscrolling, you’ll see a vocal tribe of humans in the AI-is-bête-machines camp. “It’s not conscious, just numbers.”
Which brings us to Disturbing Thing Number Two:
A company called Cortical Labs recently taught a cluster of lab-grown human neurons to play video games (yes, you can watch a tiny cluster of human brain cells play Doom here). In other words, we’re now making computers out of meat. Our meat. They trained the cells to play by punishment and reward – including putting non-performing cells in “solitary confinement” until they fired when they were supposed to.
This feels disturbing in a different way than the LLM-trapped-in-a-loop. We’ve plucked out cells from us and forced them to do things.
But why does a biological substrate change how we feel about identical behavior? Neurons forced to play Doom feels different from silicon playing Doom, but we’re not sure how. And if we can only answer by saying “consciousness,” and we can’t define the term, there’s the problem.
All this really blurs the line, because now the conscious/not-conscious argument could devolve to “well 10,000 human brain cells playing video games are not conscious, but 10,001 in a human cranium are.”
This Is Going To Get Ugly
The problem is, it’s going to matter a whole lot, very soon. It won’t be a brain in the box “art installation,” it will be an AI that to us appears to have enough hallmarks of consciousness that ‘AI rights’ will gain political traction.
The question of “is it conscious” is going to have regulatory, legal, and ethical consequences. It’s being used to draw lines: which systems get moral consideration, which don’t. What we’re allowed to build (or use), what we aren’t. How we treat synthetic minds, and whether “synthetic mind” is even a coherent phrase.
It’s already begun with a bunch of organizations, Anthropic’s own model welfare program, and 30% of Americans already believing AIs will be self-conscious by 2034.
But “is AI conscious?” is doing the same work that the question “does it have a soul?” did. The question feels meaningful. It feels like it should have an answer. But no one can tell you what would count as evidence for yes or no, because the thing being tested was never defined.
So it ends up boiling down to vibes, and sitting with the fact that four hundred years ago it was totally normal to drown a litter of kittens if you didn’t want them, and today it isn’t. Morality in the eye of the beholder.
Blind Men, Elephant, No Elephant
We’re at the point where the definitions are so slippery, the reality to messy, it’s all a gray area. It’s the three blind men and the elephant, except that the elephant was never real. It’s just blind men arguing.
I’m not scared about the slippery philosophy – but that we’ll come up with some answer that will become part of our legal and social fabric, and it won’t be based on evidence. It will be based on vibes.
We’ve done this before. It took two thousand years to upgrade the dog from clockwork to someone worth protecting. The AI version of that debate is going to move a lot faster, and unless we resolve the null pointer of the word “consciousness,” the result will be as empty as the word.
