MCP tool catalog diff
Review an MCP release before the agent consuming it finds the mismatch. Compare complete tools/list snapshots by stable tool name, then inspect changes to inputs, outputs, descriptions and declared behavior.
What the comparison checks
| Change | Why it matters | Next check |
|---|---|---|
| Removed tool | A saved consumer call may reference a name that no longer exists. | Replay consumers and migrate callers. |
| Newly required input | Existing calls may omit an argument now required by the schema. | Test the previous valid call against the proposed contract. |
| Output schema changes | A downstream parser may rely on fields or types being returned. | Validate representative structured results. |
| Description or behavioral hint changes | An agent may select or interpret a tool differently. | Review tool selection and actual server-side authorization. |
Inputs may be direct tools/list results or JSON-RPC result wrappers. Tool order is not identity. Duplicate names, unsupported schemas and a remaining pagination cursor are rejected; gather every page before comparing.
A repeatable MCP release review
- Capture both catalogs using authorized access. Store the full results before and after the change. Do not paste bearer tokens, headers or credentials.
- Run the static diff. Load both JSON snapshots in the free workspace and inspect high-impact and review findings.
- Replay consumer cases separately. A contract diff does not call the server or prove that its implementation follows the declared schema.
- Retain the report. Export rule versions, source hashes and JSON-pointer locations alongside the release under review.
node cli.mjs mcp before.json after.json --fail-on high
The CLI and local MCP server are part of the licensed Kit, not required for the free browser review.
Annotations are not authorization
Read-only, destructive, idempotent and open-world annotations are behavioral hints. They do not enforce scope, permissions or a human approval step. Review those controls in the server and client independently.
No catalog comparison can prove prompt-injection resistance, runtime security or end-to-end compatibility. Reordering tools alone should not produce a removal finding; changing the contract still deserves consumer tests.
Technical references: MCP tool specification and JSON Schema 2020-12. The software implements documented checks, not full protocol or schema conformance.
Connect the rest of the evidence
Use AI output regression checks for structured expectations and run comparison to inspect per-case outcomes, costs and latency after a change. API validation tools cover the HTTP interfaces around your MCP server.
Is your document search finding the right evidence?
The free RAG retrieval evaluator measures precision, recall, MRR and binary nDCG from judged document IDs and ranked results. Inspect individual misses, missing runs, cohorts and no-answer tests without uploading document content.
Keep the checks in your own development stack
The Developer Assurance Kit is a separate $79 one-time software purchase: offline browser, CLI, SDK, private CI and five local MCP tools for API/schema changes, payload validation, MCP catalog changes and deterministic AI output regression. The free run-comparison tool on this site remains separate and free; it is not included as an MCP tool in Kit 1.1.0. The free retrieval evaluator is also separate from the Kit.
No hosted model, per-run fee or custom AI integration is required. Your own provider, runner and infrastructure costs still apply.