How we decide an LLM feature is good enough to ship.

We decide an LLM feature is good enough to ship when it clears a quality bar we wrote down before we built it, measured against real examples rather than a few good demos. The bar is specific to the feature, the cost of a wrong answer sets how high it sits, and we keep measuring it after launch because the inputs drift. Good enough is a number we agree on, not a feeling we have in a demo.

Most teams ship LLM features on vibes. They try a few prompts, the answers look great, and that confidence becomes a launch. Then a real user finds the case the team never tried, and the feature loses trust faster than it earned it. The fix is to make quality measurable before it is public.

The first question is not which model, it is what does a good answer look like, and what does a bad one cost. A summary that misses a nuance is annoying. A wrong dosage, a wrong number on an invoice, a confidently invented policy, those are different. The price of being wrong sets how high the bar has to be, and it is different for every feature.

We decide that with the people who own the outcome, not just the engineers. If the team cannot describe what good looks like in plain words, the feature is not ready to build, because there is nothing to aim at and nothing to test against.

Before we tune anything, we collect real examples with known good answers. Some are the easy, common cases. Many are deliberately the hard ones: the ambiguous question, the input with a trap, the case where the right answer is to refuse. This set is the closest thing an LLM feature has to a test suite.

It does not need to be huge to be useful. A few dozen well-chosen cases that cover the ways the feature matters will catch more real problems than a thousand random ones. The skill is in picking examples that represent what users will really do, including the things they should not.

An overall score hides the failures that matter. So we track quality where it counts: how often the answer is factually grounded in the source instead of invented, how it does on the hard cases specifically, and how it behaves when it should decline rather than guess. A feature can look great on average and still be unshippable because it fails badly on the cases that carry the most risk.

We watch the floor, not the average. The question is not how good is it usually, it is how bad is it at its worst, and can we live with that.

Numbers tell you something is wrong. They do not tell you why. So someone sits and reads the failures, the actual bad outputs, because the pattern is usually obvious once you look. The retrieval missed a document. The prompt was ambiguous. The model was asked to do two jobs at once.

This is the least automated and most valuable step. Most quality gains come from one person reading twenty bad answers and noticing the same mistake three times. We do it before launch and we keep doing it after.

Good enough to ship does not mean good enough to trust blindly. We launch LLM features with a way to fail safely: a confidence threshold below which the system hands off, a human in the loop where the stakes are high, a clear path back when the answer is uncertain. The bar decides what we ship. The fallback decides what happens when we are wrong anyway.

That combination is what lets us ship with a straight face: a system that stays safe even when the model is wrong.

Real usage is the only honest test. Once people are using the feature, we keep the evaluation set growing with the new cases they surprise us with, and we watch quality over time, because the inputs change as the product and the users do. A feature that cleared the bar in May can slip by August without anyone touching the code.

Good enough to ship is a decision you make with evidence, and then keep making. The work does not end at launch, it just gets quieter.