How do you know when your AI feature is wrong?

The feature works. You have used it, the team has used it, the demo went well, and the output looks right almost every time you look at it. That last clause is the problem. "Almost every time you look at it" is not a measurement, and the cases you look at are the cases you designed for. The ones that will hurt you are the ones nobody thought to type. This is written for the engineer who already shipped, not for someone deciding whether to. Most of it is unglamorous. ## What is actually wrong with spot-checking? It samples from the wrong distribution. When you check your own feature you generate inputs the way its author generates inputs — you know what it is for, so you ask it things it is for. Real users arrive with truncated pastes, two questions in one sentence, a document from a vendor whose export is subtly malformed, and the assumption that it can do something adjacent that it cannot. Spot-checking also has no memory. You fix a prompt so case A behaves, and case B — which you fixed three weeks ago and have not thought about since — quietly regresses. Without a suite you find out about that from a customer, and the customer will not phrase it as a regression. They will phrase it as "your product is unreliable", which is a much more expensive sentence. ## Separate the retrieval failure from the generation failure If your feature does anything with your own data, this is the highest-value thing on the page. Most reports of "the model got it wrong" are not generation failures at all. The right document never made it into the context window. The model was then asked a question it had no way to answer and did the thing models do, which is produce something fluent. You cannot fix that by editing the prompt, and teams spend weeks trying. So measure the two separately, and measure retrieval first because it has a correct answer. For a set of real questions, record which chunk *should* have been retrieved, then check whether it appeared in the context at all. That is a deterministic assertion with a yes or no answer — no model call, no judgement. When retrieval is landing, and only then, is a generation score meaningful. ## What actually goes in a harness Less than the word implies. Four parts: **A fixture set.** Inputs paired with what must be true of the output. Not ideal answers — *properties*. "Contains a valid ICD-10 code", "cites at least one retrieved source", "does not state a dosage", "refuses". Properties survive model upgrades and prompt rewrites; a golden string does not. **Graders, deterministic first.** If the output should be valid JSON, parse it. If it should contain a real product code, look the code up. Reach for a model-as-judge only for the subjective residue, and when you do, calibrate it: label a sample by hand and check the judge agrees with you. An uncalibrated judge is a second unmeasured system sitting on top of the first one. **A runner that scores by category**, so a drop tells you *where*. A single aggregate number tells you something changed and nothing else. **A CI gate.** The suite runs on the branch, and a drop below the line blocks the merge. Everything above is a document until this part exists. ## The change that helps most is letting it refuse A system with no abstention path will fabricate, because fabricating is the only move available to it. Give it somewhere else to go. If retrieval confidence is under the threshold, or the required field is not present in any retrieved chunk, return "I do not have that" rather than generating. Then make abstention a first-class outcome in the harness — you want to know both how often it refuses and how often it refuses something it should have answered, because a system that refuses everything scores beautifully on correctness and is useless. ## Do you need an evaluation framework? Probably not, and this is where most teams lose a month. What you need first is a file of real failures. Every time somebody says "that answer is wrong", the input goes in the file with a note on what was wrong with it. That file, after a few weeks of production, is worth more than any framework, because it is drawn from the distribution that is actually hitting you rather than one you imagined. Then write a script that runs those inputs and asserts the properties. Plain code, in your repo, in your language. Adopt a framework later if the script gets unwieldy — many teams find it never does. The dataset is the asset. The harness is plumbing. ## Where I learned this I designed and built a multi-tenant clinical platform — 95 TypeScript source files, 29 test files, 36 migrations. It is built and owned by me and has no live client, so I am not claiming a deployment; what I am claiming is the constraint I built under, which was that a wrong answer reaching a patient was not a bug you ship and fix on Thursday. That constraint changes the engineering. Output has to be measured and gated rather than hoped over, abstention has to be designed in from the start instead of retrofitted, and the interesting question stops being "is the model good" and becomes "what does the system do when the model is wrong". That second question has an answer you can build. The first one does not. ## If you want another pair of eyes The awkward part of evaluation work is that the person best placed to design the harness is the person who wrote the feature, and they are also the person least able to imagine how it breaks. **[Run the free website check](/website-audit)** — that is a public, unauthenticated endpoint that takes an arbitrary URL from a stranger and drives headless Chromium at it, which is an SSRF liability unless the boundary is real. 36 tests cover that boundary — the inputs it must refuse, and the public ones it must keep serving. Poke at it; it is a fair sample of how I build the parts nobody sees. If the eval question is the live one, I do a fixed $2,500 review — your prompts, your retrieval, your failure modes — and you get a harness design and a prioritised fix list in about a week, with no retainer after it. [Tell me what you shipped](/contact). If the answer is that your spot-checking is adequate for the stakes you are operating at, I will tell you that instead.