Start with the record
A demo can be useful because it gives the team a concrete example. Release review asks for a wider record: what changed, what passed, what failed, and what evidence supports the decision. That distinction matters once a model becomes part of a product, workflow, or reviewable system.
The harness records the task, model configuration, inputs, outputs, scores, tool calls, retrieved context, costs, latency, and software version for each run. A reviewer can open a result and trace it back to the exact prompt, response, scorer, and runtime settings from the record itself.
What to record
Task surface. Version the inputs, expected behavior, and reason each case exists. Keep held-out cases separate from tuning work.
Runtime configuration. Record model identity, sampling settings, prompt scaffolding, tools, retrieval policy, and context-window handling for every run.
Scoring functions. Separate structural checks, behavioral checks, safety checks, and semantic scoring. Disagreement between scorers creates an investigation item.
Run record. Store inputs, outputs, scores, costs, timing, logs, and artifacts in a form that can be queried later.
These fields form the comparison boundary for the system. If a prompt changed, a model changed, retrieval changed, or the scorer changed, the report says so directly, because otherwise two runs can look comparable while measuring different systems.
Handle nondeterminism directly
Language-model output varies. The report needs to show how variance was handled. Use fixed seeds where a provider supports them, repeated runs where sampling remains variable, and tolerance bands where exact equality fails the review need. The useful view shows spread, tail behavior, and failure examples, because a single average can hide important release cases.
Model-graded checks need their own tests, including a labeled calibration set rerun whenever the scoring model or scoring prompt changes. A metric that moved because the scorer drifted is a different event from a metric that moved because the system improved.
Design the test surface
Good evaluation suites are organized by risk: capability checks ask whether the system can perform the target behavior, regression checks protect behavior that already worked, safety checks cover refusal and escalation, and integration checks exercise retrieval, tool use, and downstream consumers.
Mixing those categories into one score makes failures harder to act on. A capability gain travels with any refusal regression it introduced. A passing unit-style prompt confirms the prompt case and leaves retrieval to its own integration evidence. The report leaves each failure in the category where it belongs, so the next engineering action is clear before review begins.
Keep examples close to the metric
Aggregate charts are useful for direction. Defect review needs representative failures near the chart: the prompt, the relevant context, the model response, the scorer output, and the reason the case was counted as pass, fail, or blocked. When examples are separated from the metric, teams waste review time deciding whether the number is credible before they can discuss the actual fix.
Examples also protect the suite from becoming ceremonial. If a safety score improves while the visible failures become more serious, the suite is telling the team something about the scoring design, and the harness has to make that discrepancy easy to notice.
Plan for real operating constraints
Hosted models bring rate limits, token budgets, cost controls, and data-handling rules. Local models bring hardware placement, dependency drift, and cleanup work. The harness has to schedule runs, resume after interruption, label partial failures, and leave enough logs for another operator to see which cases finished and which cases never ran.
Cost and throughput belong next to accuracy in the report. A change that improves a benchmark while increasing runtime or cost may still be the right change, and the tradeoff belongs in the same review packet as the quality result.
Use the harness during development
The same system can run at different depths. Fast checks catch common regressions during development. Scheduled runs cover the wider suite, including repeated samples and integration paths that take too long for every commit. Release runs freeze the configuration and produce the report that can be kept with the shipped version.
That separation keeps the harness useful while the product is still moving. Engineers get quick feedback, reviewers get a stable record, and evaluation becomes part of normal engineering work earlier than the final release review.
Inspectable reports
Generate reports from immutable run records, with charts linking back to the cases, prompts, responses, scoring rationale, and configuration behind them. Diffs between runs show the code, model, prompt, retrieval, and scoring changes that produced the new result.
The report is finished when a skeptical reader can follow the evidence from the artifact itself. A serious evaluation harness supports that level of review.