An AI That Can Describe an Action Cannot Necessarily Perform It

A customer asked for a payment link. The system replied as if the link were attached and on its way. It was not. There was no order, no payment link, and nothing attached. Nothing had been created at any point in that conversation.

The message came from a follow-up agent, and a follow-up agent has exactly one capability: read the conversation history, write a nudge. It holds no tools. It cannot open an order, it cannot call a payment provider, and it cannot attach a file. It can only produce text.

That is the part worth sitting with. The model did not fail at a task it had been given. It described an action it was structurally incapable of performing, and the description was fluent enough to go straight to a customer without anything stopping it.

The regular conversation flow already had protection against exactly this. Statements touching orders, payments, and links were checked against the source of truth before anything went out. That guard existed, it had been reviewed, and it worked.

Proactive follow-ups did not go through it. They ran on a separate path: triggered by inactivity, assembled by a background job, sent on a schedule. The message never crossed the point where the check lived.

So this was not really a model failure. It was a coverage failure. We had one hardened outbound path and one that was not, and the unhardened one was easy to overlook because nobody is watching a job that runs on its own at night.

That generalizes past this incident. A control that protects customers has to cover every path that reaches a customer: the live reply, the retry, the scheduled nudge, the digest, the queue consumer, the cron job. A safety check that only covers the path you are looking at is a safety check with a hole in it.

The fix was to stop treating the generated message as output and start treating it as a draft that has to earn the right to be sent.

It is composed from two inputs rather than one: the conversation history and the authoritative order data. History on its own will happily reproduce whatever was said before, including things that were never true.

The draft is then validated claim by claim. Every factual statement and every implied action — a link exists, an order was created, a payment landed, something is attached — has to be backed by the record before it counts.

If a claim is not supported, the message is rewritten once using only the facts that were verified, and the rewrite goes back through the same validation. A rewrite gets no credit for being a rewrite.

If anything is still uncertain after that, the pipeline stops generating and sends deterministic copy: a short, pre-written message that says nothing it cannot back up.

A long thread is a persuasive document. If someone wrote "so you'll send me the link", that sentence is now sitting in context and reads like a commitment the system already made. If an earlier message described a next step, the model treats the description as evidence that the step happened.

None of that is authority. The database is authority. A verified provider response is authority. When the transcript and the record disagree, the record wins, and the message has to be rebuilt around what the record says.

This is a small rule that is easy to state and easy to lose. Retrieval-heavy systems keep pulling conversation into the prompt because it improves tone and continuity, which it does. It just cannot be allowed to decide what is true.

We changed the prompt too, and the prompt change helped. But a prompt moves the distribution of what the model tends to do; it does not bound it. If the only thing between a hallucinated action and a customer is an instruction written in English, you have a strong tendency, not a guarantee.

The validation step is code. It runs after generation, it inspects the actual text that is about to be sent, and it fails closed.

Failing closed is specifically right for proactive messages. In a live conversation the customer is present and can push back within seconds. A proactive follow-up arrives unprompted, is read once, and is believed. A generic safe follow-up is a mild disappointment. A confident fictional action is a promise the business did not make.

The conversation that produced the bad message is now a regression test. It is sanitized — customer details stripped, kept only as a fixture — and it runs on every change to the follow-up path. A post-mortem that ends in a patch and a paragraph decays; the patch survives and the reason for it does not. A test is the only documentation that complains when you break it.

The other thing that happened during the response is, I think, the more interesting story. Prompts here are versioned and promoted through a gate, the same as code. While the incident was open there was pressure to push a prompt candidate straight through, because the fix looked obvious and the situation looked urgent.

The gate held. The code-level protection passed its checks and shipped, and a shared prompt candidate that was not healthy was stopped before it reached production.

That is the outcome you want, and it is worth naming precisely because it is uncomfortable. Urgency is the exact condition under which review gets skipped, and a shared prompt has a shared blast radius: it touches every conversation, not only the one that broke.

LLM output should describe system state. It should never define it.

Every claim that reaches a customer about a payment, an order, a reservation, inventory, a delivery, or a link has to be grounded in the source of truth before it is sent. If the system cannot prove the claim, the system does not make the claim. That is a code-level rule, applied on every outbound path, with a deterministic fallback at the end of it.

And the narrower version, the one I keep coming back to: an AI that can write about an action is not necessarily able to perform that action. The fluency of the sentence tells you nothing about whether anything happened.