Why your AI pilot never reached production

Something like two thirds of the AI work we get called into is a rescue. A team built a proof of concept, demonstrated it internally, everyone was impressed — and then it sat there. Six months later it is still sitting there, and nobody can quite explain why.

The reason is nearly always the same, and it is not the model.

The demo problem

A proof of concept is built by running a handful of inputs through a prompt and looking at the outputs. If they look good, the demo works. This is a perfectly reasonable way to establish that an approach is plausible.

It is a terrible way to establish that an approach is reliable, because you have measured nothing. You have a sample size of maybe twenty, selected by the person who wanted it to succeed, judged by eye.

So when someone asks "what happens on the weird cases?" — and in a regulated environment someone always asks — there is no answer. Not a bad answer. No answer at all. And no responsible person signs off on a system whose failure rate is unknown.

What an evaluation harness actually is

It is less sophisticated than it sounds. Three parts:

1. A golden dataset. A few hundred real inputs with known-correct outputs, assembled by the people who do the work today. This is the expensive part and there is no way around it.
2. An automated scorer. Something that compares model output against expected output and produces a number. Exact match where the task allows, a rubric-based model grader where it does not.
3. A gate in CI. The score runs on every change and fails the build when it drops.

That is it. No new infrastructure, no platform purchase.

Why this changes everything

Before the harness, every conversation about the system is an argument about anecdotes. Someone found a bad output; someone else insists it usually works. Neither can prove anything, so the discussion resolves by seniority rather than evidence.

After the harness, the conversation is: accuracy is 94.2%, here are the seventeen failures, twelve are the same category, let us fix that category. That is a normal engineering conversation. It has a next step.

More importantly, it makes improvement safe. Teams stop being frightened to change the prompt, because a regression is caught in minutes rather than discovered by a customer.

We build the evaluation harness before we tune anything. It feels like a detour for about a week, and then it is the only reason the project moves at all.

The order matters

The instinct is to get the thing working first and add measurement later. In practice "later" does not arrive, because by then the system is entangled with production and building a golden dataset means reconstructing history.

Build the harness first. It is genuinely faster, even though it does not feel like it on day three.

What good looks like

When we hand over an AI capability, the client gets a dashboard showing accuracy over time, cost per request, p95 latency, and the current failure categories ranked by frequency. Their own engineers can change a prompt and see within ten minutes whether it helped.

That is the difference between a pilot and a product. It has very little to do with which model you picked.