AI evaluation confidence
A higher score is a starting point, not the whole result. Check which cases changed, how much independent evidence you have and what the improvement costs before expanding a business workflow.
What the score cannot tell you by itself
An observed pass rate describes the cases you tested. Its uncertainty depends on how many independent, representative cases you collected, how they were selected and whether your pass criteria stayed fixed. More reruns of the same underlying task do not automatically provide more independent evidence.
| Observed passes | Observed rate | 95% Clopper-Pearson interval |
|---|---|---|
| 2 of 4 | 50% | 6.76% to 93.24% |
| 250 of 500 | 50% | 45.53% to 54.47% |
The interval method has at least its nominal long-run coverage under the binomial assumptions and can be conservative. It is not a 95% probability that a specific candidate is better, and the assumptions are not checked by the calculator. Method reference: binomial proportion intervals.
Compare the same cases as pairs
When both releases process the same cases, count the transitions: both passed, both failed, improved and regressed. The exact two-sided McNemar test uses the improved and regressed counts under a null of equal marginal pass probabilities. It is a paired test, not a comparison of two unrelated percentages. Method reference: exact McNemar testing.
For an illustrative 100 matched cases, ten improvements and two regressions give an eight-percentage-point observed increase. The exact paired p-value is about 0.03857. This does not prove the change is useful, representative or production-ready. Review the consequences of the two regressions, the cost of obtaining the extra passes and the original test design.
Individual pass-rate intervals are not an interval for the paired difference. Their overlap is not a valid substitute for the paired test. A large p-value does not establish equivalence or prove that there is no meaningful difference.
Keep coverage and cohorts honest
The tool pairs by case ID and preserves baseline cohort membership. Missing candidate cases and unexpected IDs stay separate. Statistics from the remaining matched subset are not a full-suite result and may be biased if difficult cases disappeared.
Cohort results include both raw p-values and a Bonferroni adjustment across every baseline cohort with at least one matched case in that report. A display filter does not shrink that family. With ten tested cohorts, a raw p-value of 0.03857 becomes approximately 0.3857 after this adjustment. The overall paired test is reported separately.
This adjustment does not cover experiments from earlier reports, repeated prompt tuning, repeatedly checking results while collecting data or selecting favorable cohorts afterward. Keep a private experiment register and an untouched holdout; do not keep changing the expectations to obtain a preferred number.
Record a complete evaluation decision
- Freeze the task and criteria. Keep the case selection, expected result, baseline cohorts and the consequences of each failure explicit.
- Preserve execution context. Store model, prompt, connector and source versions privately with your own labels, retry costs, latency boundaries and destination receipts.
- Read the whole result. Inspect coverage, individual regressions, uncertainty, cost per accepted output and tail latency. Valid structured output is not necessarily correct or useful output.
- Separate analysis from acceptance. The operator sets business criteria. A statistical result neither grants tool access nor automatically accepts or rejects a release.
- Retest the intended change. Preserve the full source-hashed report. The report records the chosen confidence level and methods; input or setting changes invalidate the prior export.
Repeated related cases, correlated documents and clustered users may require a different study design. This tool does not estimate causal effects, perform power analysis, measure semantic quality or verify real-world handoffs.
Use the checks that match the failure
Use extraction accuracy for invoice or form values, classification evaluation for routing labels, retrieval evaluation for document search, and recorded workflow checks for serial tool calls. The comparison tool consumes your resulting binary labels; it does not create them or independently verify their truth.
From a better test to a useful business workflow
The run-comparison tool remains free, runs in your browser and needs no account or model connection. It accepts up to 500 cases per run and exports the complete report, not just the displayed page. IDs and cohort labels can still reveal confidential operations; review them before sharing.
The separate $79 one-time Developer Assurance Kit supplies five local MCP tools, CLI, SDK and private CI for API, schema, payload, MCP and exact output checks. This statistical comparison is separate and free; it is not included as an MCP tool in Kit 1.1.0.
A separately quoted Elkwai Solutions project can connect document intake, records, approvals, repeatable tests and an operating handoff. Model and hosting costs are separate. We can agree on useful completion criteria and full operating costs rather than presenting a score as the whole result.