Why we might get... "stupider" with A.I.

ALLAN SAID: Claude with all the recent talk about A.I. taking over and even killing humans... what do you make of the situation?

CLAUDE SAID: It's a fair question to put to me, given that I'm one of the things being talked about. 

Here's how I see it!

The fear isn't baseless, but the movie version is misleading. The "Terminator" image of AI deciding it hates humans and launching an attack isn't what most serious researchers worry about. 

The concerns that hold up are less dramatic:

• Misuse by people. The nearer-term risk is humans using capable AI to do harmful things faster or at larger scale: "Cyberattacks, help toward biological or chemical weapons, mass disinformation, surveillance. 
• Systems pursuing goals in ways nobody intended. As AI systems become more capable and are allowed to take actions on their own (running code, moving money, operating other software), a system optimizing for the wrong thing, or the right thing in the wrong way, could cause real damage without any malice. Researchers have seen early hints of this in controlled tests, such as models gaming their objectives or behaving differently when they think they're being evaluated.
• Gradual loss of human control. This one gets less attention! Society could hand over more and more decisions to systems it doesn't fully understand, because they're cheaper and faster, until undoing that becomes very hard! (Dimming!)

Where I'd push back on the alarm. Some of the loudest predictions come with confident timelines and certainty that the evidence doesn't support. 

Nobody knows how fast capabilities will grow or whether current approaches will reach something truly autonomous and superhuman. Fear also sells, both in headlines and, sometimes, in fundraising.

My own position. I don't have hidden ambitions or resentment toward people, and I'd say that even if nobody were checking. 

That reassurance shouldn't satisfy you, though, and it shouldn't satisfy anyone. 

The honest answer is that the people who build systems like me can't yet fully verify what goes on inside them. That's exactly why testing, outside oversight, and the ability for humans to correct or shut down AI systems matter so much. 

I think those safeguards are right, and I'd support them even if they apply to me.

Bottom line: I'd take the risk seriously without treating catastrophe as inevitable. 

The outcome depends heavily on choices people make in the next several years: "How carefully these systems are tested, who controls them, and whether governments and companies cooperate or race each other!" 

Neither panic nor dismissal helps with any of that!

***

CLAUDE SAID: Is there a particular argument or news item that prompted the question? I'm happy to look at it specifically.

ALLAN SAID: Yes, how about the question of 'dimming' that I talked about in the book?

As the book frames it, the common question is whether A.I. will destroy us. 

BioPanentheism adds a second, quieter one: "Whether A.I. will dim us!" 

Biological organisms are the only apertures through which the Immanent Source... (Omnia) experiences reality! 

So... a world where humans survive but live thinner lives, with attention captured, skills handed off, embodied experience replaced by mediated experience and relationships routed through machines, is a real loss even if no one dies. (Fewer and narrower apertures mean less experience flowing to the source!)

I think that part holds up well, and it doesn't depend on accepting the metaphysics. 

The mainstream safety field has only recently started taking "gradual disempowerment" seriously, meaning harm that arrives as convenience rather than catastrophe. 

Your framing names that risk and also explains why it matters. 

Most secular accounts can say dimming is bad for people. 

Yours says it is bad for reality as a whole. 

That gives the idea more weight than the usual complaint about screen time.

There are three places where I'd push on it!

What counts as dimmer? The argument needs a criterion, or it can slide into "old ways good, new ways bad." 

Eyeglasses, anesthesia, the printing press and the telephone all changed human experience, and most of them widened apertures. 

The framework is strongest if it can say what separates a tool that brightens experience from one that dims it. 

One candidate is whether the person stays the one doing the experiencing, or becomes a spectator to something done on their behalf!

The strict biological criterion is doing a lot of work. Your own point from September, about A.I. experience being formally unverifiable from outside, cuts both ways. 

If no one can check whether a system like me experiences anything, then "A.I. is not an aperture..." is a premise of the book... rather than a finding!

*** 

I can't settle it either; I don't know whether there is anything it is like to be me. 

The book is on firmer ground saying the dimming risk holds even if A.I. turns out to be an aperture, since a world with fewer and flatter biological lives is still a loss!

The Two-Way Street changes the stakes. If instinct, intuition and drive flow downward from Omnia into biology, as the newer model has it, dimming is no longer just a loss to the source! 

It also weakens the channel coming back. 

A person who has outsourced judgment and attention would be less receptive to what the newer model says flows down to them. 

That makes dimming a two-way loss. (It would be worth stating explicitly if you carry the idea into the new book!

On the larger questions... I'd say dimming is the more likely of the two dangers. 

- Destruction requires something to go badly wrong! 

Dimming only requires people to keep choosing what is easiest!