IA en Producción

Most AI demos look brilliant on stage. Most AI in production quietly breaks in ways nobody is watching.

That gap is the whole talk. A demo runs on a handful of inputs someone chose in advance, in a room where the person driving already knows which prompt works. Production runs on everything else: the malformed input, the ambiguous request, the user who phrases things in a way nobody on the team would have thought to try.

Nothing crashes. There is no stack trace and no red dashboard. The system answers confidently and answers wrong, and unless you built something to notice, it keeps doing that indefinitely.

The difference between an AI that works in the demo and one you can put in front of real users is not model choice, and it is rarely prompt engineering. It comes down to whether you can see what the system is actually doing once it leaves your hands.

Traditional observability assumes failure is loud — an exception, a timeout, a 500. Language models fail quietly and fluently. The output is well-formed, plausible, and incorrect. That means the usual instrumentation tells you almost nothing about the failures you actually care about, and you have to go build the layer that does.

Evals are the regression suite for systems that do not produce the same answer twice. They will not tell you the model is correct, but they will tell you when a prompt change, a model upgrade, or a retrieval tweak made things worse — which is the question you are usually asking.

LLM-as-judge takes the qualities you can only describe in words — is this helpful, is it grounded in the source, does it follow the format — and turns them into something you can score, chart, and alert on. It is imperfect, and it is dramatically better than reading samples by hand and calling it a review.

The last piece is the one teams resist most: treating real-world failures as the input to the next iteration. Every bad output a user hits is a test case you did not have to invent. Capture it, label it, put it in the eval set. The alternative is shipping on the assumption those cases will not happen, and they will.

The tooling has made building an AI feature almost trivial. A working prototype is an afternoon. That is genuinely new, and it has moved the hard part somewhere else.

Knowing whether the thing is actually doing its job — that is the hard part now, and it is the part that decides whether anyone keeps using what you shipped.

If you are shipping AI and not measuring it yet, that is the gap worth closing first.