AI agent workflow trace checker

Compare intended steps with recorded tool calls

Session-only inputs. No upload, autosave, model call or tool execution.

Up to 500 cases and 50 calls per case; 512 KiB, 20,000 JSON nodes and 32 levels per input. These independent limits can reduce usable case counts. Serial traces only. Exclude arguments, outputs, credentials and customer content. Reports retain supplied IDs, tool names, rules and measurements.

Workflow evidence

All attempts contribute to supplied cost and tool-time totals. Unknown measurements stay unknown. Tool time is a duration sum, not end-to-end latency. A recorded success is not proof that a business effect occurred.

Make the test expectations explicit

Each expected case has a stable id, a complete successful-call sequence and allowed_tools. Successful calls must match the entire sequence exactly, including repeated steps. Failed attempts do not advance it. All calls, including failed and pending attempts, must use listed tools.

{"cases":[{"id":"request-01","group":"Intake","sequence":["lookup","draft"],"allowed_tools":["lookup","draft"],"max_failed_calls":1,"max_cost_usd":"0.01","max_total_tool_ms":2000}]}

[{"id":"request-01","calls":[{"id":"call-01","tool":"lookup","status":"succeeded","cost_usd":"0.002","duration_ms":400},{"id":"call-02","tool":"draft","status":"succeeded","cost_usd":"0.003","duration_ms":600}]}]

Statuses are succeeded, failed and pending. Failed-attempt allowance defaults to zero; set it deliberately when a recovered retry is acceptable. Pending calls never pass. Missing traces never pass, even for an empty expected sequence. An explicitly recorded empty trace can pass an empty sequence.

Budgets are optional. USD values are non-negative decimal strings, up to 9 integer and 6 fractional digits; no locale symbols or conversion. Durations are non-negative integer milliseconds, up to 3,600,000 per call; a total tool-time budget is at most 180,000,000. Failed attempts count toward both. Missing or pending measurements cannot establish a budget; a known subtotal already over it is still over. No budget means no budget check, not evidence of zero cost.

IDs and tool names use 1-128 letters, digits, dots, colons, underscores or hyphens. Tool lists and sequences have at most 50 entries. Call IDs are unique within a trace. Cohorts are reference-only text up to 80 characters. Duplicate JSON keys, duplicate case IDs, unknown keys and unsupported shapes are rejected. There is no automatic OpenTelemetry import, parallel-call ordering or fuzzy sequence alignment.

Use the same checker in a local test job

The self-contained free CLI requires Node 22 or newer. It reads two bounded UTF-8 files, makes no network requests and writes the complete report to standard output. No installation, API key or account is required.

node agent-trace-audit.mjs --expected workflows.json --observed traces.json > report.json
node agent-trace-audit.mjs --expected workflows.json --observed traces.json --fail-on-errors > report.json

Exit 0 means valid report generation, not a workflow pass. With --fail-on-errors, exit 2 means an expected workflow failed a supplied check or an unknown case ID was recorded. Input/I/O errors exit 1. This check does not replay calls or enforce server permissions.

Know what this evidence does and does not prove

The four-case illustrative fixture passes two workflows: a clean run and a recovered retry with an explicit one-failure allowance. Another run reverses two successful steps, lacks a cost measurement and includes an unlisted pending call. The fourth trace is missing. One unknown case is retained separately, excluded from metrics. None of these are client results.

A model can report success while the business action failed. This checker accepts your recorded statuses; it does not verify trace authenticity, actual permissions, arguments, side effects or answer correctness. Validate those independently and preserve an untouched holdout before changing test expectations.

For payment, message or record-write effects, reconcile against the destination system. Webhook delivery evidence separates acknowledgement from recorded effect counts. Output regression checks supplied structured outcomes.

From an isolated check to a working business system

Nothing here sends messages, changes records, calls an agent or starts a paid engagement. Supplied tool lists are test expectations, not live access permissions.