Support routing
Inspect technical tickets sent to billing and billing tickets sent to technical support separately. Record re-routing effort, waiting time and review cost in your runner; the evaluator does not invent those costs.
Recorded decisions. Visible errors. Your own AI.
How to evaluate AI classification
A useful routing system needs more than a reassuring accuracy percentage. Define the categories, preserve each decision and inspect the failure types that matter to the business.
1. Define the work and the reference judgments
Choose a specific task: routing support tickets, separating invoices from contracts, or sending incoming work to a department. Define one expected label per case, with examples and boundary cases. Keep raw content, reference reasons and reviewer disagreements privately with the test-set version. The public evaluator only needs IDs and labels.
Do not ask the producing model to declare its own output correct. Use reviewed reference labels, preserve judgment changes and retain cases where people disagree. A reference set is not a complete list of every category that could exist.
2. Keep uncommon classes and hard cases visible
Evaluate ordinary cases alongside rare but consequential ones. Use expected cohorts such as document family, intake channel or operating team. Include ambiguous inputs, empty or damaged records and cases that should go to review. The tool reports each cohort from the reference set; a prediction cannot move itself into an easier cohort.
For a 99-to-1 test set, predicting the common class every time gives 99% accuracy and zero recall for the rare class. Balanced recall is 50%. A balanced score is diagnostic, not proof that the reference set matches your incoming workload.
3. Record classification, abstention and failure separately
Export the actual chosen label for each stable case ID. A null label is an explicit abstention; an absent row is a missing run. Keep run timestamps, model or rule versions, prompt settings, preprocessing and review outcomes in your private experiment record. Do not omit difficult cases from the expected suite.
The evaluator counts wrong non-null labels as incorrect, including labels absent from the reference set. Unknown case IDs are preserved separately and excluded from all expected-suite metrics. It does not obtain decisions from your model, call a provider or measure latency itself.
4. Interpret the complete measurement boundary
| Measure | Definition | Boundary |
|---|---|---|
| Precision | TP / (TP + FP) | Unavailable if a class was never predicted. |
| Recall | TP / (TP + FN) | Missing runs and abstentions contribute false negatives for the expected class. |
| F1 | 2 TP / (2 TP + FP + FN) | Unavailable only when its denominator is zero. |
| Macro F1 / balanced recall | Equal-weight mean of class F1 / recall | Only classes with expected support in the selected population; predicted-only classes remain visible separately. |
| All-case accuracy | Correct / every expected case | Missing and abstained cases do not count as correct. |
| Conditional accuracy / coverage | Correct / classified; classified / expected | Read together. Abstaining on hard cases can raise conditional accuracy without solving more work. |
Weighted F1 uses each class's expected support. The confusion matrix retains raw counts with separate abstention and missing columns. Cohort macro scores use that cohort's supported classes; do not assume identical class mix between cohorts.
5. Diagnose business consequences, not only model scores
Document intake
Inspect invoices mistaken for contracts, unknown categories and missing runs. Check extraction, downstream validation and authorization separately; a correct category does not prove the document fields are correct.
Assign consequences to observed error types using measured handling costs and your own risk assumptions. Compare automation plus review effort with the actual baseline. Use AI run comparison for recorded costs and latency and the automation value planner for explicit operating assumptions.
6. Change one thing, preserve a holdout and retain evidence
Freeze label definitions and case IDs before comparing a prompt, model, ruleset or preprocessing change. Keep an untouched holdout and avoid selecting a release only because it won on repeatedly inspected cases. Preserve per-class and cohort reports, source hashes, run configuration, reference reasons and unresolved failures.
Do not infer statistical significance, confidence calibration or production readiness from these descriptive metrics. There is no universal passing score: acceptance depends on the workflow's consequences, coverage and review burden.
Primary technical reference: scikit-learn classification metrics. Elkwai Solutions uses explicit missing-run, abstention and macro-denominator rules documented above.
Build a routing workflow around measurable results
Elkwai Solutions can implement document intake, ticket routing or a structured review workflow using your existing systems and chosen local or cloud AI. A scoped pilot includes reference cases, measured failure types, review handling and handoff.
This evaluator is free and separate from the $79 Developer Assurance Kit; it is not included as an MCP tool in Kit 1.1.0. Custom implementation is quoted separately. No model calls or subscription start when you use the evaluator.