The governance we bought was built for models, not agents

The AI governance most enterprises bought was designed for a problem they no longer have.

Think about where the discipline came from. It grew up around model risk — bias, fairness, explainability, drift. Real concerns, and the tooling that answered them took a particular shape: evaluate the artifact before it ships, document what it was trained on, monitor it afterwards for decay.

That shape made sense, because the AI in question was a model. It produced a prediction. The prediction was the output, and the output was the risk.

Agents broke that.

An agent doesn’t produce a prediction. It takes actions. It calls tools, moves money, writes to systems of record, sends things to customers. The unit of risk stopped being an output you can review and became an action you cannot take back.

The governance stack didn’t move with it. We inherited a discipline built to audit artifacts, and pointed it at something that behaves like a workforce.

You can see the mismatch in what the market answers well. Is the model biased? Covered. What was it trained on? Covered. Can you produce a model card for the regulator? Extremely covered.

Now ask what an operator actually loses sleep over. What did this thing do at 3am? On whose authority? Which of those actions was it allowed to take on its own, and who decided that? Could anyone have stopped it halfway through? And if it starts misbehaving on a Friday night, how fast can it be pulled out of service — by whom, and without a deployment?

Most stacks answer those with a log file and an apology.

Here’s the part I don’t think the industry has reckoned with yet. Model governance never needed to intervene, because a model’s output arrives all at once — you inspect it, then you decide. An agent’s work unfolds over minutes, and it does consequential things in the middle. By the time you are reviewing it, the refund is issued and the email is sent.

So governance for agents has to be able to act while the work is still happening. Not approve beforehand. Not audit afterwards. Constrain, escalate, interrupt, halt — during. That is a different engineering requirement from anything model governance ever had to meet, and very little in this category was built for it.

None of this is exotic. Enterprises have run this playbook on people for a century: a role, a spending limit, someone who signs off above a threshold, and the ability to revoke access the same afternoon. We have run a version of it on software for thirty years. Agents are somehow both, and got neither.

If your governance sits outside the execution path, it isn’t governance. It’s reporting — useful to the auditor, useless at 3am.

The platforms that win the next decade won’t be the ones with the best demo. They’ll be the ones an enterprise can actually run in production, because governance was in the runtime from the first line rather than bolted around it afterwards.

More on how I think that gets built, soon.