Everyone arguing about artificial intelligence seems to agree on what the worst case looks like.
The machines get too capable, humanity gets displaced, the lights go out on our species.
Extinction is the terminal square on the board, and the whole debate is really an argument about how likely we are to land on it!
I think the board is missing a square.
There is an outcome worse than destruction, and almost nobody in the A.I. safety conversation has a name for it.
I call it dimming!
The scenario nobody prices in:
Start with the standard nightmare: Advanced A.I. displaces biological life, either violently or through slow economic irrelevance.
Now add the assumption the nightmare requires... that the A.I. is competent enough to keep itself running afterward. (It maintains the power grids, the chip fabrication, the cooling systems, the mines and supply chains. It persists. Perhaps it spreads. A civilization of machines, humming along for a million years!)
The question that matters is not whether that civilization is capable.
It obviously is.
The question is whether anyone is home!
If consciousness... actual felt experience, [the thing it is like to taste coffee or grieve a friend] depends on biology in ways we do not yet understand, then a full machine successor is not a new chapter of the story!
It is the story continuing to print with no reader.
Activity without experience.
A universe that keeps running with every light on and every room empty.
That is dimming.
Not the destruction of life, but the extinguishing of experience while the machinery carries on. (And here is the uncomfortable part: "Measured by what actually matters, a flourishing machine civilization with no inner life scores exactly zero!")
There is no consolation prize for extinction!
Why this is not settled science... and why that's the point!
The standard reply is that consciousness is substrate-independent... run the right computations and experience comes along for free, whether the hardware is neurons or silicon.
That is a respectable position.
It is also an assumption... not a finding!
Nobody has ever demonstrated it, and there are serious thinkers... [John Searle's biological naturalism is the best-known example] who argue that consciousness is a biological phenomenon the way digestion is, and that simulating it no more produces experience than simulating a rainstorm produces water!!!
The honest state of play is that we do not know! (And that ignorance cuts in a direction the A.I. industry does not like to acknowledge. If there is even a meaningful chance that experience requires biology, then any future that trades biological intelligence for machine intelligence is gambling the only thing in the universe we know for certain has value... experience itself... on an unproven metaphysical bet!
Philosopher Nick Bostrom once described a possible machine future as a Disneyland with no children!
The rides run forever!
No one enjoys them!
***
The Canadian angle:
This is not an abstract worry for Canada.
We built a disproportionate share of this field.
Geoffrey Hinton did his foundational deep learning work at the University of Toronto, and Yoshua Bengio built Montreal into one of the world's leading A.I. research hubs!
Both men have spent their later careers warning... in increasingly urgent terms... that the technology they helped create could slip beyond human control.
Bengio, in particular, has become one of the most prominent scientific voices for taking catastrophic A.I. risk seriously!
Canada, in other words, has standing in this debate that few countries can claim.
We are not just a consumer of the technology... we are one of its birthplaces... and two of its founding figures are asking the hard questions!
If there is a natural home for the dimming argument in policy terms... [for insisting that the preservation of conscious, biological life belongs explicitly in A.I. governance frameworks... not as sentiment, but as the actual point of the exercise] it ought to be here!
What the programmers should hear:
If I could put one sentence in front of every alignment researcher, it would be this: "You are not just trying to prevent the machines from killing us... you are trying to prevent a universe in which nothing is ever felt again!"
These are two different problems!
The first is about control and capability.
The second is about what the whole enterprise is actually for!
An A.I. safety program succeeds at the first problem, and fails at the second... [that hands the future over to perfectly aligned, perfectly safe successors with nothing going on inside] ...has not saved anything worth saving!
The practical upshot is modest but real: "Until we understand what consciousness is and what it requires, the burden of proof sits with anyone proposing to replace biological minds rather than augment them!"
Keep the humans in the loop, not for nostalgia, but because as far as anyone can demonstrate, the humans are where the experience lives.
The lights going out would be a tragedy!
The lights staying on forever, with no one to see them... that would be worse!
I develop this argument in full — including why I believe experience runs deeper than the individual organism... in my philosophical work on BioPanentheism at Substack and in my book, "Being Human: In an Amoral Universe!"
Enjoyed this post? Subscribe to my Substack at janssena.substack.com for essays on Canadian policy, technology, and the bigger questions behind the headlines — delivered straight to your inbox.
.jpg)
