Our Production Model Finished Fifth
For months our WhatsApp sales agent answered customers on a model nobody had ever benchmarked. When I finally tested that slot against ten alternatives it came fifth — and my own harness lied to me four times on the way to finding out.
Age is not evidence
A value that has been in the config long enough starts to look deliberate. Nobody questions it, because questioning it would imply somebody once decided it, and nobody wants to be the person who did not know why.
That is the whole failure mode. grok-4.6 was answering real customers about real prices and real delivery dates, and the only argument for it was that it had been there a while. It had never won a comparison. It had never been in one.
Meanwhile we had been tuning the part of the system we could see. The agent does two jobs, and we had spent months iterating on the second one while the first sat frozen.
RENDERING DIAGRAM…
One variable, eleven models
The design was deliberately boring. Same system prompt. Same tools. Same eval suite. Same structuring model behind it. The only thing that moved was the model answering the customer.
Boring is the point. If two things change and the score moves, you have learned nothing you can act on.
grok-4.6 plus ten alternatives
prompt, tools, eval suite and extractor held fixed
out of eleven
cheaper than the model it replaced
Eleven models do not fit in one batch
They ran in two waves, on different days, against different provider weather. That alone can manufacture a ranking. So one model ran in both waves as an anchor — if the anchor drifts between waves, the waves are not comparable and the cross-wave ranking is fiction.
grok-4.6 scored the same in both. That is the only reason I trust a single table with eleven rows in it.
RENDERING DIAGRAM…
The frontier model came fifth
Here is the top of the table. I am publishing ranks and labels rather than raw scores, because the numbers are only meaningful against our prompt, our tools and our domain — and a leaderboard lifted out of its harness is exactly the kind of artifact this whole post is arguing against.
The point is not that GLM is always best. The point is that we had never tested the slot at all, and the moment we did, four models beat the one we were paying for.
The gap was not cosmetic either.
0.632 → 0.822
0.655 → 0.931
0.931 → 1.000
count, not a score
The job was obedience, not creativity
A general AI benchmark rewards a model that explains, brainstorms and reasons freely. Our sales bot is graded on the opposite instinct: use the tool, quote the exact price, and do not improvise.
A model that fills in the blank sounds helpful right up until the blank was a delivery date. For a bot quoting prices, products and dates, plausible-but-wrong is strictly worse than "I am not sure" — one costs a refund, the other costs a follow-up message.
The winner was not smarter. It was better at not making things up.
That is what the groundedness number is measuring, and it is why a model with worse general benchmarks can be the right hire for this seat. The expensive frontier model did not lose because it was bad. It lost because this job rewards one behaviour above almost everything else.
One model was talking to itself in public
Some customers had been seeing the bot think out loud:
"I'll look up your box contents and delivery zone first…"
We assumed our framework was mishandling reasoning traces, so we built a guard to strip those lines on the way out. Reasonable fix. Wrong diagnosis.
The sweep answered it by accident: across all eleven models, exactly one produced that leak. It was not our framework. It was that model's output, and we had spent engineering time building a filter for a behaviour we could have simply stopped buying.
Sometimes the app bug is just the model.
I was wrong four times before I was right once
The useful version of this story includes the mistakes, because every one of them would have shipped a confident, wrong conclusion if I had stopped one step earlier.
| Blamed | What I concluded | What was actually true |
|---|---|---|
| Provider | "This model times out." | One bad provider hour. It looked like model truth because it arrived as model output. |
| Sample size | "That metric regressed." | 29 cases moved the metric by 0.12. At that n, that is noise wearing a decimal point. |
| Harness | "This model completely failed." | My parser was reading the model's reasoning trace instead of its answer. The model was fine; my measurement was not. |
| My laptop | "46 timeouts is a real signal." | Local concurrency manufactured them. Production, on the same model, had zero. |
That last one is the one I think about. A benchmark measures the model and the harness, and it reports a single number for both. It never tells you which half failed.
RENDERING DIAGRAM…
on my machine, under load
same model, same window
moved the metric 0.12
not the framework
Five rules I am keeping
None of these are really about model selection. They are about not fooling yourself while you measure.
- 01Find the config values nobody has measured.Start with the ones that have been there longest. Age reads as intention, and it is the cheapest lie in the repo.
- 02Benchmark the seat, not general intelligence.Score the exact job: your prompt, your tools, your domain. A public leaderboard is measuring a different job with a different harness.
- 03Repeat one model across every batch.The anchor is what converts several runs into one comparison. Without it you are ranking days, not models.
- 04Trust big gaps; treat small ones as clues.A 0.28 shift on safety is a decision. A 0.12 shift on 29 cases is a thing to go investigate, not a thing to ship.
- 05Check production before blaming the model.Your harness can fail in ways that look exactly like model failure, and it reports both with the same confidence.
What it actually cost
We swapped the model, the leak went away with it, and the guard we had built for the leak turned out to be scaffolding around someone else's bug.
But the swap is not really the lesson. Any of this could have been found at any point in those months by someone asking one question about one line of config.
We did not need a better model. We needed to measure the model doing the actual job.
Somewhere in your stack there is a value that has been correct-by-default since the day it was typed. Age is not evidence. Go measure the hot path.