Your AI demo works. Production is a different job.

Getting an AI pilot to production is not a final polish on the demo. It is a separate body of work that the demo was allowed to skip: handling the messy inputs, deciding what happens when the model is wrong, watching quality over time, and keeping costs sane at real volume. The demo proved the idea is possible. Production proves you can be trusted with it on a Tuesday afternoon when a real customer is depending on it.

We have built enough of these to stop being surprised by the gap. A pilot answers a curated question for a friendly audience. Production answers whatever a stranger types, at any hour, and lives with the consequences. Almost everything that makes the second one hard is invisible in the first.

When you build a pilot, you feed it the inputs you understand. Clean documents, well-formed questions, the happy path. That is fair, you are testing whether the core idea has legs. But it means the demo never met the half-finished form, the scanned PDF that is really a photo of a fax, or the question that assumes context the system does not have.

Production is mostly those inputs. The first real work is making the system behave when the data is bad, because in the field the data is usually a little bad. That is not a model problem, it is plumbing, and it is where a lot of the time goes.

A demo gets to be right. A production system has to have a plan for being wrong, because it will be, and the question is only how it fails. Does it say something confident and incorrect, or does it notice it is unsure and hand off? The difference is the whole product.

So we design the failure cases on purpose. The threshold where the system declines to answer, the path to a human, the message a user sees when something is uncertain. None of this shows up in a demo, and all of it shows up in whether people keep trusting the thing after the second week.

After a handful of good demos it is tempting to call the quality settled. It is not, you have just seen it behave a few times. Before we put an AI feature in front of users we build an evaluation set, real examples with known good answers, so quality becomes something we can measure on every change instead of something we feel.

That harness is what lets us improve the system without breaking it. Swap a prompt, change the retrieval, try a cheaper model, and the number tells you whether you helped or hurt. Without it, every change is a guess and every regression is a surprise a customer finds first.

In a demo nobody is counting tokens or milliseconds. In production both decide whether the feature is viable. A flow that calls the largest model three times might be lovely and also too slow and too expensive to ship to everyone.

Part of the production job is finding the cheapest, fastest version that still clears your quality bar. Often that means a smaller model for the easy cases and the big one held in reserve, with the evaluation set deciding where the line sits. These are not optimisations you do later, they shape what the feature can be.

The last piece is the one teams forget to budget for. Once real people are using an AI feature, you have to watch how it behaves, because they will use it in ways you did not imagine. Logging, monitoring, and a way to review the bad answers are not extras, they are how the system gets better after launch.

A pilot ends at the launch. A production system is where the real learning starts, and the work is keeping it honest as the inputs drift and the usage grows.

If your AI pilot is stuck in a notebook, the work left is rarely more cleverness. It is the engineering that turns something promising into something dependable. That work is less exciting than the demo and far more valuable, because it is the part your customers meet.