Every AI component fails sometimes
A classifier misclassifies an input. A retrieval system returns the wrong document. A language model hallucinates a fact. These failures come from the statistical behavior of the systems and can occur even when everything is operating as designed. Engineering can reduce their frequency but cannot drive it to zero.
A failure budget defines how often those outcomes are acceptable. Without one, a bad output triggers an investigation and a fix, but the team has no objective way to tell whether overall performance improved or regressed. The budget provides the test: is the system within tolerance?
Accuracy needs context
"The model is 95% accurate" sounds good in a slide deck. If accuracy was measured per request and production traffic resembles the evaluation data, roughly 1 in 20 requests will be wrong. At 10,000 requests a day, that is about 500 failures. Whether that rate is acceptable depends entirely on the failures and who they affect.
A 2% error rate on low-stakes document classification might be fine. A 2% error rate on medical triage routing is a serious problem. Its significance depends on the failure class, its severity, and the operating context.
Categorize failures by impact
Wrong outputs carry different costs. Separate them into classes based on their downstream impact:
- Silent wrong answers. The system gives a confident, incorrect response and the user has no reason to doubt it. These are especially dangerous because they erode trust without anyone noticing until damage accumulates.
- Obvious wrong answers. The output is clearly broken: garbled text, irrelevant results, or nonsensical formatting. Users can recognize these failures and work around them.
- Refusals and timeouts. The system declines to answer or fails to respond. These failures disrupt work and prevent incorrect output from being acted on.
- Access control violations. The system exposes information the user shouldn't have. This is a security failure with a budget of zero.
A single accuracy number blends these failure classes together even though each warrants a different tolerance. Track them separately so the thresholds reflect the actual risk.
Setting the budget
Start with the downstream consequence. Human review permits a higher tolerance because every output is checked before it matters. Direct handoff to an automated workflow requires a lower tolerance. Safety or legal compliance may require a zero budget for specific failure classes.
Give each component its own budget. A RAG pipeline needs separate measures for retrieval (did the right documents surface?), generation (did the model use them correctly?), and citation accuracy (can each claim be traced?). A combined "system accuracy" number hides the component that is breaking.
Concrete example. An internal knowledge-base assistant might set these targets: retrieval recall above 90% for known-answer test cases, zero citation fabrications, abstention on every query where source evidence is missing, and fewer than 5% of answers flagged as unhelpful in user feedback. Each target has its own measurement and its own investigation trigger.
Measuring in production
Eval suites test controlled cases. Production traffic exposes real operating conditions. Each catches failures the other can miss, so you need both. An eval set may omit question patterns that users introduce in production. Production monitoring may not surface regressions on rare inputs deliberately covered by the eval set.
Production measurement should include:
- Automated checks on output structure, citation validity, and refusal triggers.
- Human review of a random sample of responses, scored against a rubric.
- User feedback signals such as thumbs down, re-asked questions, and escalations to a human operator.
- Comparison against the eval set on a regular schedule to catch drift.
Log enough to reconstruct any individual failure: the input, the retrieved context (if applicable), the model version, prompt version, and full output. Without reconstruction, investigations are guesswork.
When the budget is exceeded
Define the response to a budget breach before the system ships. Crossing a threshold must trigger a specific action, and an incident is the wrong time to invent the protocol.
Common responses, roughly ordered by severity:
- Increase monitoring frequency and review the failing cases to confirm it's a real trend.
- Roll back to the previous model, prompt, or retrieval configuration that was within budget.
- Add a human review step for the failing output class until the root cause is fixed.
- Disable the affected component entirely and fall back to a non-AI path.
Rollback depends on preserving the previous configuration. Keep restorable versions of model weights, prompts, and retrieval indexes. Otherwise, a regression forces the team to fix forward under pressure.
Talking about risk with non-technical stakeholders
Engineers often describe model performance with metrics such as precision, recall, F1, and perplexity. Most stakeholders need those results translated into business impact before they can evaluate risk. That includes the people who approve budgets, sign contracts, and accept liability.
Translate the failure budget into the business questions they need answered:
- How many users per day will see a wrong answer?
- What's the worst thing that happens when it's wrong?
- How fast can we detect and correct a problem?
- What's the fallback if the AI component goes down entirely?
- How does this compare to the error rate of the current process (manual or otherwise)?
If the current manual process and the AI system are measured on the same task under comparable conditions, a drop from a 12% data-entry error rate to 3% with full logging and rollback is a clear improvement despite the remaining mistakes. Using the status quo as the baseline keeps the risk comparison honest and useful.
What the failure budget record looks like
Maintain the failure budget as part of the system's documentation. It should contain:
- Each AI component and its role in the system.
- Failure classes defined for that component, with examples.
- Tolerance thresholds per failure class.
- How each threshold is measured (eval, production sampling, automated checks).
- What happens when a threshold is exceeded: the specific escalation path.
- Who owns the investigation and who owns the decision to roll back.
- Review cadence, including how often the budget is re-evaluated against production data.
This document supports procurement review, release approval, incident response, and quarterly system reviews. Each review then has recorded thresholds and evidence for answering the basic question: is the AI operating within its approved tolerance?