RAG retrieval evaluator
Does your document search find the evidence the answer needs? Compare judged document IDs with ranked retrieval results, inspect every miss and export reproducible evidence without sending your inputs to a server.
Evaluate ranked retrieval
Session-only inputs. No upload, autosave, provider connection or model calls.
Up to 500 queries, 100 unique document IDs per query, 512 KiB and 20,000 parsed JSON nodes per input. Use identifiers, not document text, questions, API keys or personal information. IDs and cohort labels can still be sensitive.
Retrieval evidence
Unexpected query IDs
Export contents
Contains query IDs, document IDs, cohort labels, metrics, definitions and source hashes. It excludes document content, but review identifiers before sharing.
Use a judged test set and a ranked result list
The expected array supplies each query's stable id, optional group and binary relevance judgments in relevant_ids. The result array supplies the same query id and document_ids in rank order. All IDs use 1-128 letters, digits, dots, colons, underscores or hyphens. Duplicate query IDs, duplicate document IDs and unknown fields are rejected.
[{"id":"query-01","group":"Policies","relevant_ids":["doc-refund","doc-exceptions"]}][{"id":"query-01","document_ids":["doc-shipping","doc-refund","doc-exceptions"]}]Use relevant_ids: [] for an explicitly judged no-answer test. An observed empty document_ids is an abstention. An absent query is a missing run, not a successful abstention. Relevance is supplied by you; documents not judged relevant count as non-relevant.
What each metric means
| Metric | Calculation | Interpretation |
|---|---|---|
| Precision @ k | Relevant hits in the first k / k | How much of the allowed retrieval budget is useful. Unfilled slots count against precision. |
| Recall @ k | Relevant hits in the first k / all judged relevant IDs | Whether the evidence needed for the task was found. |
| MRR @ k | Mean of 1 / first relevant rank; zero when no relevant hit occurs within k | How soon at least one useful result appears. It does not reward finding the rest. |
| Binary nDCG @ k | Sum of relevance / log2(rank + 1), divided by the ideal ranked sum at k | How well multiple relevant results are ordered, with binary rather than graded relevance. |
All expected answerable queries enter the averages equally, including missing runs scored zero. No-answer tests are reported separately. Unknown queries never inflate the result. A small, incomplete or repeatedly tuned test set does not establish general performance.
For the illustrative example at k=3: recall is 50%, precision is 22.222%, MRR is 0.5 and binary nDCG is approximately 0.415. One of four expected queries is missing, and one unexpected query is excluded.
Method reference: Stanford Introduction to Information Retrieval: ranked retrieval evaluation. This implementation uses binary relevance and explicitly defined missing-run rules. Third-party notices.
Turn the findings into a working document workflow
The free evaluator measures recorded retrieval. It does not build an index, generate an answer or call a model. If you need the workflow implemented, Elkwai Solutions can scope a document-search or review pilot around your information, test cases and operating constraints.
A written project quote names the workflow, connectors, milestones, evaluation and handoff. Model, hosting, integration and ongoing support costs are identified separately; using this tool does not start a subscription.