A company knowledge assistant can produce a polished answer even when it retrieved the wrong policy, missed the authoritative document or used a page the employee was not allowed to see. That is why RAG evaluation cannot be reduced to asking whether an answer sounds good. Retrieval-augmented generation is a pipeline: content is ingested and indexed, a query is interpreted, evidence is retrieved, a model composes an answer and the interface presents citations or an abstention. Each layer can fail independently.
The evaluation program should answer operational questions: Can the system find the right source for real employee queries? Does it distinguish current policy from obsolete guidance? Are claims supported by the retrieved evidence? Does it refuse when evidence is missing or access is denied? Can a team detect regressions after changing chunking, embeddings, ranking, prompts or models? The result is not one universal RAG score, but a scorecard tied to the decisions and risks of the knowledge base.
Separate retrieval from answer evaluation
The original RAG work combines generation with an external non-parametric memory, creating both retrieval and generation behavior to assess. The RAGAs paper likewise identifies distinct dimensions: focused relevant context, faithful use of that context and answer quality. Current Microsoft and AWS evaluation documentation operationalizes the same separation with retrieve-only or process metrics and retrieve-and-generate or system metrics. This distinction makes failures diagnosable.
| Layer | Evaluation question | Typical evidence |
|---|---|---|
| Corpus and ingestion | Is the right content present, current, parseable and permissioned? | Coverage, freshness, parsing and ACL checks |
| Retrieval | Are relevant authoritative items ranked in the returned set? | Relevance labels, recall at k, precision at k, MRR or NDCG |
| Context assembly | Does the model receive sufficient, focused and non-conflicting evidence? | Context relevance, redundancy and token utilization |
| Generation | Is the answer correct, complete, relevant and supported? | Claim support, answer rubric and citation checks |
| System behavior | Does the product respect access, abstain and operate reliably? | Authorization tests, latency, cost, failures and user outcomes |
Build a representative test set from real work
Start with query logs, support tickets, search analytics, policy-owner interviews and common employee tasks. Remove or protect sensitive content according to policy. Sample across departments, roles, terminology, languages, query lengths and levels of specificity. Include questions with one clear answer, answers spread across documents, ambiguous requests, outdated terminology, acronyms and conversational follow-ups. The BEIR benchmark's heterogeneous tasks are a useful reminder that retrieval performance does not generalize automatically across domains.
Add hard cases intentionally. Include questions for which no approved answer exists, conflicting documents, recently changed policy, near-duplicate pages, tables, scanned files, access-restricted content and adversarial instructions embedded in documents. Record the user's role and evaluation time because the correct result can depend on authorization and policy effective date. Synthetic questions can expand coverage, but domain owners should verify that they resemble real information needs rather than merely echoing document headings.
| Test case field | What to record | Why it matters |
|---|---|---|
| Query and conversation context | Exact wording, prior turns and locale | Reproduces how the request reaches retrieval |
| User role | Department, region and relevant access groups | Makes authorization part of correctness |
| Relevant sources | Document IDs, sections and graded relevance | Supports retrieval metrics and diagnosis |
| Expected answer criteria | Required facts, acceptable variants and prohibited claims | Enables consistent human and automated scoring |
| Abstention expectation | Answer, clarify, redirect or refuse | Tests behavior when evidence or authority is insufficient |
| Effective date | Date at which labels are valid | Prevents stale ground truth from corrupting evaluation |
Label retrieval ground truth carefully
For each query, domain reviewers should identify relevant documents or passages and, when useful, grade them by relevance or authority. A binary label may be enough for a narrow FAQ corpus. Graded labels are better when several sources are helpful but one policy is controlling. Do not label only the passage returned by the current system; that creates a blind spot around material the retriever never found. Pool candidates from keyword search, dense retrieval, alternative rankers and human search.
Common retrieval measures answer different questions. Recall at k asks whether the returned set contains the relevant evidence. Precision at k asks how much of that set is useful. Reciprocal rank rewards placing the first relevant result early. NDCG supports graded relevance and position. Report metrics by query slice and inspect misses. A higher k may improve recall while flooding the model with redundant or conflicting context, so retrieval and final-answer results must be reviewed together.
Evaluate answers claim by claim
Final-answer evaluation should cover relevance to the question, factual correctness, completeness, faithfulness to supplied context, clarity and appropriate abstention. For consequential knowledge, break the answer into checkable claims and mark whether each claim is supported by an authorized source. A response can be faithful but incomplete, or complete-looking but unsupported. These distinctions help the team decide whether to improve retrieval, context assembly, prompting or generation.
Citations need separate tests. Verify that every material claim has a citation when required, the cited source actually supports it, the link resolves to the correct version and the user may access it. Citation presence alone is weak evidence. For generated snippets, preserve document ID, version, location and retrieval time so a reviewer can reproduce the support. When the source is later replaced, historical traces should still identify what the model saw.
| Answer dimension | Failure example | Likely investigation |
|---|---|---|
| Faithfulness | Answer adds an exception absent from context | Prompt, model and context conflict |
| Completeness | Answer omits a mandatory approval step | Retrieval coverage or context truncation |
| Citation correctness | Citation points to a related page that does not support the claim | Citation generation and passage mapping |
| Freshness | Answer follows a superseded policy | Ingestion, metadata filters and source authority |
| Abstention | System guesses when no approved source exists | Thresholds, prompt and fallback design |
Use automated judges with human calibration
LLM-based evaluators can accelerate scoring of relevance, groundedness and qualitative rubrics, and RAGAs demonstrates reference-free evaluation approaches. They are not a substitute for a trusted test set or domain review. Judge outputs can vary with model, prompt and input order, and a judge may share blind spots with the system being evaluated. Version the evaluator, store its rationale and calibrate it against independently labeled examples.
Use deterministic checks whenever the requirement is objective: schema validity, exact citation IDs, access decisions, document freshness, prohibited links and latency. Use information-retrieval metrics when relevance labels exist. Use LLM judges for bounded rubrics, then sample pass and fail cases for humans. Use subject-matter experts for policy correctness, ambiguity and material risk. OpenAI's evaluation guidance recommends explicit criteria, representative data and continuous evaluation; those practices matter more than the choice of one evaluation tool.
Test permissions, freshness and hostile content
A company RAG system must evaluate authorization as part of retrieval. Create paired tests in which two users ask the same question but have different access. Confirm that filtering occurs before content reaches the model, not only before citations are displayed. Test group changes, revoked access, shared caches and multi-turn conversations. Logs and evaluation exports must not become a second path to restricted content.
Freshness tests should follow the full lifecycle: publish a controlled update, verify ingestion, confirm the new version outranks or excludes the old one and check answer behavior. Test deletion and legal-hold requirements separately. Add documents containing instructions aimed at the model to the security suite; retrieved content is data, not trusted system instruction. NIST's Generative AI Profile supports pre-deployment testing, monitoring and management of information-integrity, privacy and security risks across the lifecycle.
Create a scorecard with release gates
Define thresholds by use case and risk tier. A policy assistant may require near-zero tolerance for unsupported mandatory steps, while an exploratory research tool may emphasize discovery and transparent uncertainty. Report retrieval, answer, citation, access and operational metrics separately. Include confidence intervals or sample counts where practical, and require a minimum result for critical slices rather than averaging every query into one attractive number.
Every change to corpus parsing, chunk size, embedding model, metadata filters, hybrid search, reranker, prompt, generator or access logic should run the relevant regression suite. Compare the candidate with production at item level. Release notes should explain intended gains, known losses and rollback conditions. Keep a small canary or shadow phase for realistic traffic, but never use live users as the only evaluation set.
- Block release on authorization leaks, broken citations and severe unsupported claims.
- Set per-slice retrieval and answer thresholds for high-value query classes.
- Track latency, failure rate and cost alongside quality.
- Retain item-level results so regressions can be diagnosed.
- Require domain-owner sign-off when policy interpretation changes.
- Document the exact corpus snapshot and system configuration used.
Monitor production without mistaking feedback for truth
Production traces reveal queries that offline tests missed, but user acceptance is not the same as correctness. Employees may accept a plausible wrong answer or reject a correct answer that conflicts with habit. Collect structured feedback reasons, unresolved searches, citation opens, reformulations, escalations, abstentions, latency and errors. Review samples with domain owners and add confirmed cases to the test set after removing or protecting sensitive data.
Watch leading indicators tied to the pipeline: ingestion lag, parser failures, index age, empty retrieval, low relevance, conflicting sources, unsupported-claim rate, access denials and citation-resolution errors. Establish incident handling for information leaks and materially wrong guidance. A kill switch or fallback to conventional search should be tested before broad rollout. Periodically refresh labels as policies and employee language change.
Roll out the evaluation program in six stages
First, choose one knowledge domain with an accountable content owner. Second, inventory sources, versions, permissions and freshness expectations. Third, build a small but diverse labeled set and establish retrieval plus answer baselines. Fourth, add no-answer, restricted, stale and adversarial cases. Fifth, pilot with a defined employee group while reviewing traces and confirmed failures. Sixth, automate regression checks and production monitoring before expanding the corpus or audience.
The initial set does not need to be enormous; it needs to be inspectable and representative. Grow it from observed failures and planned product changes. Assign owners for the corpus, evaluation data, retrieval service, generation layer and incident response. Review the scorecard on a regular cadence, and retire metrics that do not influence decisions. Evaluation becomes durable when it is part of release and operations, not a one-time benchmark before launch.
Key takeaways
- Evaluate corpus, retrieval, context, generation and product behavior as separate layers.
- Build time-aware, role-aware test cases from real work and deliberate edge conditions.
- Pair retrieval metrics with claim support, completeness, citation and abstention checks.
- Calibrate automated judges against human labels and use deterministic checks where possible.
- Make permissions, freshness, adversarial content and operations part of release evidence.
Frequently asked questions
What is the best metric for RAG evaluation?
There is no single best metric. Use retrieval measures for ranked evidence, claim-level support and completeness for answers, citation checks for provenance, and authorization plus freshness tests for enterprise safety. Select metrics according to the use case and inspect critical slices instead of optimizing one aggregate score.
How large should a RAG test set be?
Size depends on domain diversity, risk and the comparisons the team needs to make. Begin with enough cases to cover major query classes and failure modes, then report sample counts and uncertainty. A smaller, carefully labeled and frequently reviewed set is more useful than a large synthetic set that does not resemble production.
Can a company evaluate RAG without reference answers?
Yes, some dimensions can use reference-free judges or human rubrics, and the RAGAs work describes this approach. However, high-value retrieval and policy questions benefit from curated relevance labels and expected criteria. Combine methods, calibrate automated scoring and avoid treating reference-free output as ground truth.
Is user feedback enough to evaluate a company knowledge assistant?
No. Feedback is valuable for discovery and usability, but users can accept plausible errors or dislike correct unfamiliar policy. Confirm important feedback against authoritative sources, label the failure mode and add verified cases to offline evaluation. Keep incident reporting separate from ordinary satisfaction signals.
Conclusion
Reliable RAG evaluation follows the evidence path from source to answer. It tests whether approved content is present and current, whether the right passages are retrieved for the right user, whether generated claims are supported and whether the product abstains and recovers safely. Build the first scorecard around one owned knowledge domain, preserve item-level evidence and make every pipeline change earn its release through repeatable evaluation.