Matched outcomes, not mismatched totals
Headline comparisons use only IDs found in both inputs. Missing and unexpected IDs remain separate. Matched-subset statistics do not describe a full suite if cases are missing.
Your models. Your systems. Repeatable evidence.
AI run comparison
Compare two recorded evaluations by the cases that changed. Review paired outcomes, sample uncertainty, baseline cohorts and recorded operating costs without calling a model or uploading your data.
Compare recorded runs
Session-only inputs. No upload, autosave, provider connection or model calls.
500 runs and 512 KiB per input. Pass labels are supplied by you. No prompt, response, API key or personal information is needed.
Run comparison
Pass-rate intervals
Cohort uncertainty uses a Bonferroni adjustment across all matched baseline cohorts in this report. Prior experiments and post-hoc selection are not corrected. Methods and assumptions.
Missing and unexpected cases
Export contents
Includes case IDs, cohort labels, supplied metrics and source hashes, but not the raw source arrays. Review before sharing.
Input format and measurement boundaries
Supply arrays of unique id and boolean passed. Optional fields: group, latency_ms and integer cost_microusd. One USD is 1,000,000 microUSD. For example, $0.002 is 2,000 microUSD. Unknown fields and duplicate IDs are rejected rather than silently discarded.
[{"id":"invoice-01","group":"Extraction","passed":true,"latency_ms":900,"cost_microusd":2000}]Use the same cases and acceptance criteria in both runs. Include retries, tool fees and other execution costs once within your chosen per-case measurement boundary. The tool does not obtain provider pricing or validate your measurements.
How to read a comparison
Sample uncertainty
Clopper-Pearson intervals describe each matched pass rate. An exact two-sided McNemar test compares improved and regressed pairs. Individual interval overlap is not a paired test or an interval for the difference.
Cost and latency coverage
Cost per pass includes failures and is unavailable when costs are missing or nothing passed. P50 and P95 use linear interpolation over measured rows only. Missing costs or latency are not zero.
Stable cohorts and repeated testing
Groups keep baseline membership. Raw and Bonferroni-adjusted cohort p-values remain available across the complete report family. Prior experiments, repeated tuning and post-hoc selection are not corrected.
Independent, representative pairs and unchanged pass criteria are assumed, not verified. Repeated variants of a case are not necessarily independent. A p-value is not the chance a release is better; a large p-value does not prove equivalence. Nothing here automatically accepts a release.
Third-party notices. Local calculations use jStat 1.9.6 plus the existing strict JSON parser and source hashing utilities.
Need an evaluation connected to real work?
A separately quoted Elkwai Solutions pilot can connect your documents, records, tools and review steps, with representative tests and an operating handoff. Model and hosting costs are separate.
Is your document search finding the right evidence?
The free RAG retrieval evaluator measures precision, recall, MRR and binary nDCG from judged document IDs and ranked results. Inspect individual misses, missing runs, cohorts and no-answer tests without uploading document content.
Keep the checks in your own development stack
The Developer Assurance Kit is a separate $79 one-time software purchase: offline browser, CLI, SDK, private CI and five local MCP tools for API/schema changes, payload validation, MCP catalog changes and deterministic AI output regression. The free run-comparison tool on this site remains separate and free; it is not included as an MCP tool in Kit 1.1.0. The free retrieval evaluator is also separate from the Kit.
No hosted model, per-run fee or custom AI integration is required. Your own provider, runner and infrastructure costs still apply.