How to evaluate RAG retrieval
A fluent answer can hide a failed search. Evaluate the retrieval step separately, preserve the evidence behind each judgment and compare changes on the same cases before deciding what to deploy.
1. Freeze the question, corpus and measurement boundary
Choose a real business task: finding an exception in a policy, locating a service instruction or collecting the documents needed to review an invoice. Retain the query text privately. Assign an opaque query ID and record the source corpus version, document or chunk IDs, permissions, retrieval settings and run timestamp in your own experiment record.
Decide whether you evaluate whole documents or chunks. Do not judge a document ID then compare it with unrelated chunk IDs. If a source is updated or split, preserve the mapping and create a new corpus version. Keep reranking, filters, hybrid search and permission checks in the recorded configuration.
2. Judge evidence before tuning the retriever
Build a representative set across task types, departments, rare terms, paraphrases and difficult cases. Label all known relevant IDs for each case; the evaluator treats unjudged IDs as non-relevant. Record who judged the evidence and the reason privately. Recheck disagreements instead of allowing the generating model to mark its own answer as correct.
Include cases where the approved corpus contains no useful evidence. An empty expected list is an explicit no-answer judgment, not a missing label. Keep a held-out set you do not repeatedly tune against. If you revise judgments, retain the previous version and explain why.
3. Record every query, even when the run fails
Export the ranked document IDs from your own retriever. An observed empty list means it ran and returned nothing. A missing row means the result is unavailable; do not erase it from the test denominator. Unexpected query IDs are recorded separately and excluded from the expected-suite metrics.
The evaluator computes macro averages across answerable queries. Precision uses a fixed k denominator, including empty slots. Recall asks whether the judged evidence was found. MRR rewards the first relevant result; binary nDCG also rewards placing multiple relevant results early. No-answer abstentions are tracked separately, with missing runs counted as unsuccessful.
4. Diagnose the failure before changing the model
| Finding | Inspect | Test one change |
|---|---|---|
| No relevant hit | Source ingestion, access filters, identifiers, term coverage and corpus completeness | Verify the relevant source exists and is eligible before changing retrieval settings. |
| Relevant evidence below k | Ranking, reranking, query formulation and conflicting near-duplicates | Compare one ranking change on the unchanged held-out cases. |
| Some evidence missing | Chunk boundaries, multi-document questions and overly narrow filters | Compare recall and context cost together, not simply a larger k. |
| False retrieval on no-answer tests | Fallback behavior and score thresholds specific to the retriever | Measure abstention and missed valid answers together; scores are not universally calibrated. |
| Good retrieval, poor answer | Context assembly, citations, instructions and answer evaluation | Evaluate the answer separately. Retrieval metrics cannot prove answer correctness. |
5. Compare the operating trade-off, not just one score
Record retrieval latency, reranking cost, context-token cost and review effort in your own runner. The retrieval evaluator does not measure these. Use the AI run comparison with separately defined pass criteria and recorded costs to compare releases by stable case ID and cohort.
Preserve the source-hashed retrieval report, configuration, relevance judgments and corpus manifest with the release. Hashes identify exact input bytes; they do not certify their truth. A good report remains reproducible after someone changes an embedding model, ranking rule or source document.
There is no universal passing score. Define acceptance around the workflow's consequences, required evidence, unresolved failure cases and review cost. Separate retrieval evidence from answer accuracy, authorization tests and end-to-end operating checks.
A complete evaluation packet
- Private query text, opaque IDs, cohort labels, relevance judgments and reasons.
- Corpus manifest and document/chunk identity mapping, including source revisions and permissions.
- Retrieval configuration, run timestamp, ranked IDs and explicit failures.
- Source-hashed report, per-query misses, cohort results and no-answer outcomes.
- Separate answer checks, latency/cost measurements, release decision and known limitations.
Technical references: Stanford ranked retrieval evaluation and scikit-learn nDCG documentation. The free evaluator is a retrieval diagnostic, not an automated factuality or production-readiness certificate.
Turn the findings into a working document workflow
The free evaluator measures recorded retrieval. It does not build an index, generate an answer or call a model. If you need the workflow implemented, Elkwai Solutions can scope a document-search or review pilot around your information, test cases and operating constraints.
A written project quote names the workflow, connectors, milestones, evaluation and handoff. Model, hosting, integration and ongoing support costs are identified separately; using this tool does not start a subscription.