What happens when your AI gets something wrong

Every vendor demo shows the system succeeding. The question that decides whether you can actually deploy one is what it does the rest of the time, and almost nobody asks it before signing.

It will get something wrong. Not often, and less often than the person it is helping, but the failure will be different in character and that difference is what you have to design for.

A person who is unsure hesitates. They ask someone. They write a hedged reply. An AI system that is unsure produces a confident answer at the same speed as a correct one, and nothing in the output tells you which you are looking at. That is the whole problem, and it is a design problem rather than a model problem.

Three failures, in order of how much they cost you

1. It answers something it should have escalated

The expensive one. A question that needed a human got a plausible reply instead, and because the reply was plausible nobody noticed. In a clinic that might be a clinical question. In a property business it might be a commitment about a lease.

The fix is not a better model. It is a hard boundary: the system answers from what you loaded into it, and anything outside that becomes "someone from the team will follow up". Not because it could not generate an answer. Because it should not.

2. It acts when it should have asked

Reading is safe. Writing is not. An agent that can send a message, book an appointment, or change a record can do something you cannot undo.

So consequential actions sit behind an approval gate. A human sees the drafted action and confirms it. This sounds like it removes the point of automation, and it does not: the work was never in clicking approve, it was in composing the reply, finding the availability, and remembering to do it at all.

3. It drifts

The quiet one. Nothing breaks. The system keeps answering, but a price changed three weeks ago, or a service was discontinued, and it is confidently describing a business that no longer exists.

Drift is invisible unless you instrument for it, which is why quality telemetry belongs in the build rather than added later when someone complains.

What good looks like when it goes wrong

The measure of a deployment is not its success rate. It is whether a failure is visible, contained and explainable.

• Visible. Somebody knows it happened, and they know before the customer tells them.
• Contained. The blast radius was one drafted message, not fifty sent ones.
• Explainable. You can reconstruct what the system saw, what it decided, and who approved it.

The 2am question

Here is a useful test for any vendor. Ask what happens when the system does something strange at two in the morning. If the answer is about model accuracy, they have not operated one. If the answer is about who gets notified, what the system refuses to do unsupervised, and how you reconstruct the decision afterwards, they have.

Every one of the boundaries described above exists because something went wrong first. That is not a confession, it is how operating anything works. The difference between a prototype and a production system is roughly a list of things that went wrong once.

Related: Who is accountable for the decision · How you know it is actually working