Your models. Your systems. Repeatable evidence.

AI output regression testing

Treat a prompt, model or connector change as a software change. Write explicit expectations, replay your own workflow, and check the recorded output without asking another AI to grade itself.

Open free regression checks Expected suite JSON Observed runs JSON

Define the expected result first

An expected suite contains cases with stable IDs. Each case needs at least one check: an output schema, exact JSON-pointer assertions, required source quotes, a latency budget or a cost budget. Keep the suite separate from the observed run.

{"cases":[{"id":"invoice-01","assertions":[{"path":"/currency","equals":"USD"}],"max_latency_ms":1500,"max_cost_microusd":5000}]}

[{"case_id":"invoice-01","output":{"currency":"USD"},"latency_ms":900,"cost_microusd":2000}]

Assertions use RFC 6901 JSON-pointer paths. Exact equality is deliberately narrow: a value either matches or it does not. Missing observations, required measurements and unexpected case IDs remain findings rather than silently disappearing.

Four useful types of regression check

Structured extraction

Check required fields, types, enum values and exact reference values for records your workflow creates. Do not mistake valid JSON for a correct record.

Tool and agent handoffs

Assert stable identifiers, routing labels and expected argument values. Test consequential execution and authorization separately.

Supplied evidence quotes

Require a source ID and verify that the quoted string occurs in the supplied excerpt. This tests quote membership, not the truth or support of the surrounding answer.

Recorded budgets

Compare supplied latency and cost with per-case limits. The engine does not call providers, read invoices or measure runtime independently.

Use fixtures without overfitting to them

Capture representative cases. Include important normal cases and failure cases. Keep sensitive source material in your own environment.
Preserve a holdout. Do not repeatedly tune on every case you use to report final quality. Keep acceptance criteria stable across releases.
Record the execution context. Retain prompt, model and settings versions, source revisions and measurement boundaries beside your outputs.
Inspect individual failures. Use run comparison for matched outcomes and baseline cohorts; an unchanged aggregate rate can hide improved and regressed cases.

node cli.mjs regression expected.json observed.json --fail-on high

Free browser checks support five cases. The $79 Kit supports 100 cases plus local CLI, SDK, private CI and MCP. Exact expectations do not prove semantic correctness or eliminate hallucinations.

Technical references: MCP tool specification and JSON Schema 2020-12. The software implements documented checks, not full protocol or schema conformance.

Is your document search finding the right evidence?

The free RAG retrieval evaluator measures precision, recall, MRR and binary nDCG from judged document IDs and ranked results. Inspect individual misses, missing runs, cohorts and no-answer tests without uploading document content.

Evaluate retrieval Read the RAG evaluation method

Keep the checks in your own development stack

The Developer Assurance Kit is a separate $79 one-time software purchase: offline browser, CLI, SDK, private CI and five local MCP tools for API/schema changes, payload validation, MCP catalog changes and deterministic AI output regression. The free run-comparison tool on this site remains separate and free; it is not included as an MCP tool in Kit 1.1.0. The free retrieval evaluator is also separate from the Kit.

No hosted model, per-run fee or custom AI integration is required. Your own provider, runner and infrastructure costs still apply.

Review the $79 Kit Compare free tools