Accuracy can hide rare failures
Accuracy uses every expected case. Macro F1 and balanced recall give each expected-supported class equal weight. The per-class table shows which category is being missed or over-predicted.
Recorded decisions. Visible errors. Your own AI.
AI classification evaluator
Is your workflow putting each item in the right place? Compare reference labels with recorded decisions from any model or rule-based system. Inspect missed classes, abstentions and missing runs without uploading the underlying content.
Evaluate recorded classifications
Session-only inputs. No upload, autosave, provider connection or model calls.
500 cases per input; at most 30 distinct labels in the expected set and matched predictions; 512 KiB and 20,000 parsed JSON nodes per input. Use opaque IDs, not message text, personal information or API keys. Labels and IDs can still be sensitive.
Classification evidence
Macro F1 and balanced recall average only classes with expected support in the selected cohort. Conditional accuracy excludes abstentions and missing runs; always read it with coverage. The all-cohorts table remains unfiltered.
Unexpected case IDs
Export contents
Exports the complete evaluation, not just the selected view: case IDs, labels, cohorts, confusion counts, metric definitions and source hashes. No raw content is included. Review identifiers before sharing.
A reference label and a recorded decision for each case
The reference array contains unique id, expected label and optional group. The prediction array contains the same id and a single recorded label. Use label: null only when the workflow explicitly abstained. A missing row is a missing run, not an abstention.
[{"id":"ticket-01","label":"billing","group":"Support"}][{"id":"ticket-01","label":null}]IDs and labels use 1-128 letters, digits, dots, colons, underscores or hyphens. Groups are non-empty text up to 80 characters. Unknown fields, duplicate IDs and malformed JSON are rejected. This is a single-label evaluator, not a multilabel or confidence-calibration tool.
Read errors and coverage together
Abstention is a separate operating choice
Classified-only accuracy can improve simply by routing difficult work to review. Prediction coverage exposes that trade-off. Missing runs are reported separately and do not count as correct.
Confusion shows where work goes
Rows are expected labels and columns are recorded labels, explicit abstention and missing runs. Labels absent from the reference set are retained as errors, not silently discarded.
Evidence is not a certificate
Labels are judgments you supply. A hash identifies the exact input bytes, not their truth. Inspect representative held-out cases and consequences before deciding where automation is appropriate.
For the six-case illustrative example: two correct, two wrong, one abstention, one missing run and one unexpected ID excluded. All-case accuracy is 33.333%, prediction coverage 66.667% and classified-only accuracy 50%. Macro F1 is approximately 0.292.
Formula reference: scikit-learn precision, recall, F-score and support. Undefined denominators are reported as null. Missing-run and expected-supported macro rules are explicitly defined in this implementation. Third-party notices.
Build a routing workflow around measurable results
Elkwai Solutions can implement document intake, ticket routing or a structured review workflow using your existing systems and chosen local or cloud AI. A scoped pilot includes reference cases, measured failure types, review handling and handoff.
This evaluator is free and separate from the $79 Developer Assurance Kit; it is not included as an MCP tool in Kit 1.1.0. Custom implementation is quoted separately. No model calls or subscription start when you use the evaluator.