After my last post, AI Safety Is a Category Error, I found myself sitting with a question I couldn’t shake.

If safety is a system property, not a model property, then why does the entire industry keep trying to install safety into models directly? Why does everyone keep making this mistake?

I sat with it for a while. And then the answer hit me.

Chatbots.

There’s an old saying: a cupful of fine wine in a barrel of sewage doesn’t improve the sewage but a cupful of sewage in a barrel of fine wine ruins the wine. Chatbots are the cupful of sewage (maybe a barrelfull). They have infected the entire AI safety discourse, and until we deal with that honestly, the rest of the conversation goes nowhere.

This week I attended the STAMP Safety Design Workshop at MIT. I went in hoping to get some answers about AI safety. I didn’t get answers. I got better questions. That turns out to be the more valuable outcome.

Here’s why. The first move in STAMP-based safety design is to define the system, its goals, and its losses, where losses means anything a stakeholder would be pissed off about. That’s the actual definition. Not “failure modes.” Not “risk events.” Anything a stakeholder would be pissed off about. I love that framing because it is bracingly honest and it forces you to think about the whole picture, not just the parts you already instrument.

When I started applying that lens to the AI safety debate, something clicked. The entire conversation (the Senate hearings, the red-teaming frameworks, the responsible AI checklists) is built on a category error. And category errors are special. They don’t just produce wrong answers. They make it impossible to ask the right questions.

A category error is when you ascribe a property to something that cannot, by its nature, possess that property. “The number seven is heavier than the number four.” “That melody smells like pine.” These statements aren’t false in the ordinary sense; they’re not even in the right zip code of falseness. They belong to the wrong frame entirely.

AI is not a system. AI is a component of a system. Systems can be safe or unsafe. Components cannot.

A few years ago I was in a meeting with developers and someone asked a simple question: “What’s the right framework for a new Windows desktop app?”

Dead silence. One person suggested WPF. Another said WinUI 3. A third asked if they should just use Electron. The meeting went sideways and we never did answer the question.

That silence is the story. And the story goes back thirty-plus years.

When a platform can’t answer “how should I build a UI?” in under ten seconds, it has failed its developers. Full stop.

I’ve always had a stone in my shoe about being a college dropout.  As such, I never imagined I’d be walking into Harvard Law School as anything other than someone who needed directions to MIT. I’m an engineer. I build systems. I’ve spent my career building things that work — reliably, predictably, and without prayer-based assumptions about what’s happening underneath.

Five weeks into retirement, I realized I’m constitutionally incapable of watching the most important conversation in the world go sideways without doing something about it.

I’ll never forget the day the world stopped. I picked up the phone and heard: “Hello. This is LCA. I’m calling to inform you that you are the subject of a formal investigation into anti-trust violations.”

At that point, everything got blurry. I think I forgot to breathe for a few minutes. I was in shock.

I had a few encounters with Microsoft lawyers that were, in a word, scary. But in a good way. Every time I worked with them, they were hardcore on following the law, and I greatly appreciated and respected that. 

A Microsoft engineer (the awesome .Net Perf architect, inventor of the phrase “Pit of Success”, and now Distinguished Engineer Rico Mariani) recently told me he used Copilot to script a way out of a laggy system. He was delighted. The story triggered a memory from the early days of PowerShell when my diplomacy was clearly … pre-operational.