Evaluation That Can Fail: Why “95% Accurate” Decides Nothing

AI readiness for small-business managers — Part 2 of 5.

In Part 1 you wrote the boundary of your service and labelled your claims Verified, Probable or
Unverified. This part takes the claim that sells most prototypes — “it’s 95% accurate” — and shows
why that number cannot support a launch decision, then walks you through building an evaluation
that can.

The word to hold onto is fail. An evaluation that cannot fail is a demo with a spreadsheet
attached. If no result could have stopped the launch, the testing was decoration.

In August 2026 I paid Mistral to do 470 items of clerk work for my shops, and the record says it
passed: about 12 corrections in every 100, against a limit of 30. I didn’t make any of those
corrections. It was only the AIs working between them, approving their corrections and identifying
their mistakes. I don’t remember any 30% limit; I had to be reminded it existed. We stopped paying
Mistral because we realised we wanted to try working only with Claude and ChatGPT for a while and
see how this works. If they decided Mistral is not good, then it’s not good. The evaluation could
have failed. It passed, and it decided nothing.

Why the accuracy number decides nothing

“95% accurate” hides the three questions a manager actually needs answered:

• Accurate on what? If the test set is the same tidy documents the demo used, the number
  measures the demo again. Production sends the scanned invoice, the odd layout, the second
  language, the form someone filled in by hand.
• Which 5% failed? Ninety-five with trivial misses is a different service from ninety-five
  where the misses are payment amounts. Average performance and operational risk are different
  measurements, and only the second one belongs in a launch decision.
• Failed how? A document system has failure modes a percentage cannot see: the confident
  statement with no support in the source, the field it silently skipped, the answer that is
  right but incomplete. Wrong-but-fluent costs more than visibly broken.

Step 1 — take your prototype’s headline number and write, in one line each: what it was measured
on, what the worst miss inside it was, and whether anyone would have noticed. Unknown is an
acceptable answer; it goes on the Part 1 list with an owner and a date.

Step 2 — build a test set that looks like production

From the checklist and the diagnostic, the test set must represent:

• the real document types the service will receive, including the ugly ones;
• the real languages and layouts;
• the real user groups — new and occasional users, not only the expert who built the demo;
• the difficult cases: edge layouts, missing fields, inputs the system should refuse.

Size matters less than honesty. Fifty documents pulled from last month’s actual queue beat five
hundred synthetic ones, because the queue contains the failures nobody thought to invent.

Step 3 — measure the failures that matter

Four measurements, from the training. Each is a count, not an impression:

1. Critical-field errors — mistakes in the fields a decision depends on: amounts, names,
   dates, classifications.
2. Unsupported statements — output the source document does not back.
3. Omissions — what the system left out while sounding complete.
4. Breakdowns by severity, user group and document type — because an error rate that is fine
   on average can be concentrated exactly where it hurts.

Step 4 — rewrite accuracy into acceptance criteria

The training’s exercise, and the heart of this part: turn “95% accurate” into three acceptance
criteria a launch decision can use. A decision-useful criterion names a measurement, a threshold
and what happens when the threshold is missed. For example, following the shape used in the
worked case:

1. “Critical-field error rate below 2% on the representative set — above it, no launch.”
2. “Unsupported or missing output below 5% in every important subgroup, not just on average.”
3. “Every high-severity failure found in testing has a documented cause and a fix or a control.”

Write your own three. The test of each one: could this criterion block the launch? If nothing
could fail it, rewrite it until something could.

Step 5 — fix a regression set

Models change, prompts change, suppliers update things without asking you. Freeze a fixed set of
test cases and rerun it after every material change; a fall of more than a few points, or the
return of a failure you had closed, is a stop signal. This is one checklist line and an hour of
discipline, and it is the difference between knowing your service changed and finding out from a
customer.

What you have when you finish

An evaluation that can fail: a production-shaped test set, four failure counts instead of one
average, three criteria that could block launch, and a frozen regression set for the day something
changes. In the worked document-agent case this whole domain was honestly labelled Unverified —
which is exactly what made “pilot only” the defensible decision there.

Numbers alone launch nothing, though. Someone has to read the output, doubt it and be able to say
no — Part 3 is about making that person real: meaningful human oversight, and why a named reviewer
without authority is a checkbox, not a control.

---

Authorship: HAC — human-directed, AI-assisted. The passage on the Mistral clerk contract, in the opening, is the author’s own spoken words, transcribed and arranged, not generated. This part is converted from the 90-minute manager training “From AI prototype to production service” (segment 25–40 min), the Day 10 readiness checklist (domain 5), and the readiness diagnostic (evaluation section). No new frameworks were created for it.

The AI readiness course: Part 1 — The prototype trap · Part 2 — Evaluation that can fail · Part 3 — Meaningful human oversight · Part 4 — Running it for real · Part 5 — The launch decision.