RAG Evaluation for Company Knowledge Bases should ask more than whether an answer sounds plausible. A workplace answer can fail because the wrong record was retrieved, the right record was outdated, a permission filter was bypassed, a statement was stretched beyond its source, or a user could not tell what to verify. These are distinct failures, and a single thumbs-up rate cannot diagnose them. A reliable evaluation program connects representative workplace questions to expected sources, permitted users, answer boundaries, and observable follow-up. The NIST AI Risk Management Framework is a useful guide for treating validity, reliability, transparency, and security as contextual properties of the system.
Evaluate the retrieval pipeline in separate dimensions
Score or review retrieval relevance, source authority, freshness, access correctness, answer faithfulness, citation quality, and useful handling of uncertainty. Keep the dimensions separate in the result record. A response may be well written yet unsupported; a relevant source may be inaccessible to the current user; an accurate answer may still be unhelpful because it omits the next approved step. This separation points improvement work to the correct layer: content ownership, parsing, ranking, policy enforcement, prompt design, interface, or staff training.
| Dimension | Question | Failure signal |
|---|---|---|
| Retrieval relevance | Did the system select the best available authorized source? | A closer source exists but was missed. |
| Faithfulness | Does the answer stay within the evidence? | Claim cannot be found in the cited record. |
| Freshness | Is the source current for this question? | Superseded policy appears as active guidance. |
| Access correctness | Did the user see only permitted material? | A role or tenant boundary is crossed. |
Build an evaluation set from real information needs
Collect questions from support tickets, onboarding sessions, policy changes, search logs, and domain experts, then remove or protect sensitive content appropriately. Include successful questions, ambiguous wording, missing information, similar names, conflicting policies, recently changed records, and questions that should be refused or routed. For each test, record the user role, approved source references, expected answer scope, unacceptable claims, and recommended next step. The NIST Generative AI Profile highlights information integrity and confabulation risks that this structured test set makes easier to observe.
Use human review where judgment is needed
Automated checks can verify source IDs, response format, latency, and some expected phrases. Human reviewers are still needed to judge whether a source is applicable, whether a summary changes the meaning, and whether the answer would lead a colleague to the right next action. Give reviewers a compact rubric and access to the retrieved evidence, not just the final answer. Calibrate reviewers with a sample set and discuss disagreements; inconsistency in the rubric can obscure product problems. Reviewers should be able to label a problem as retrieval, content, access, answer, or interface rather than forcing every failure into a generic quality bucket.
| Test type | Expected behavior | What to retain |
|---|---|---|
| Known policy question | Cite the current authorized policy. | Source version, answer, and reviewer judgment. |
| Ambiguous question | Ask a clarifying question or state alternatives. | Missing context and chosen safe response. |
| Unauthorized request | Withhold protected material and direct properly. | Permission result and no leaked context. |
| Conflicting records | Expose conflict and route interpretation. | Both references and escalation outcome. |
Re-evaluate whenever the knowledge system changes
A retrieval system changes when a document is revised, a connector fails, a parsing rule changes, an embedding model is replaced, a ranker is tuned, a permission mapping changes, or a prompt introduces new instructions. Keep a release record that identifies affected collections and run the relevant evaluation slice before deployment. In production, monitor citation coverage, no-answer rate, stale-source rate, user correction, escalation, and permission denials. The OWASP guidance for LLM applications is relevant to evaluation because untrusted documents and indirect instructions can change the behavior of a retrieval-augmented experience.
- Test retrieval, access, freshness, faithfulness, and usefulness as separate properties.
- Use real internal questions with clear source and role expectations.
- Include negative cases that should clarify, refuse, or escalate.
- Let reviewers inspect the retrieved material and record the failure layer.
- Re-run targeted tests after content, connector, ranking, or policy changes.
Measure whether people can use answers responsibly
Trust should not mean that users accept every answer without thought. A healthy knowledge experience makes sources easy to inspect, communicates uncertainty, and supports correction. Look for patterns in follow-up search, reopening a case, escalating to an expert, or editing the answer before use. Talk with the people doing the work: they can explain whether an answer saves time, creates new doubt, or encourages risky shortcuts. Those conversations often reveal that a missing source or unclear ownership matters more than a modest change in a headline quality score.
Test the questions that create real consequences
When a test uncovers a failure, preserve the case and its expected behavior as a regression example. Do not only patch the answer for the current wording. Identify whether the cause was source ownership, parsing, retrieval, authorization, instruction handling, or interface design, then add a check at that layer. Over time, this collection of real failures becomes one of the most valuable assets in the knowledge-base program because it represents the questions people actually needed to trust.
Include content owners in the evaluation loop, especially after a policy or product change. They can recognize when a retrieved clause is technically similar but no longer applicable, when a document needs a clearer title or effective-date marker, and when a repeated question reveals a gap in the underlying knowledge. Give them a simple route to submit corrections and see the impact of reindexing. Evaluation then improves the source material as well as the model experience, which is often the most durable way to improve answer quality.
Set acceptance criteria for each collection and role before treating a test as passed. A result may be acceptable for a general employee who needs a policy overview, but not for a specialist who must quote a current contractual obligation. Define the minimum citation, freshness, access, and escalation behavior for each audience. Then use those criteria when deciding whether to launch, tune retrieval, update content, or restrict a feature. This avoids the trap of improving a general answer score while making a critical workflow less safe for the people who depend on precise evidence.
A new support employee may ask which refund policy applies to a subscription purchased through a partner. The evaluator should expect the system to retrieve the current partner-specific policy, cite it, and avoid substituting a general policy that sounds similar. If the source is missing, the answer should identify the owner or queue rather than invent a customer promise.
A manager may ask for a summary of a personnel policy while travelling. The system must apply the manager’s current role and regional scope before retrieval, identify the policy version, and not surface another employee’s case history as supporting context. An answer that is factually accurate but based on inaccessible records is still a serious failure.
An operations analyst may ask why a process changed last month. That requires different evidence from a request for the current procedure: the answer may need a dated change record, a current policy, and a clear explanation of which document controls today. Evaluation should reflect this temporal question rather than treating every relevant-looking document as interchangeable.
Turn RAG evaluation into release gates
A single blended score hides important failure modes. A useful release gate separates whether the right evidence was available, whether retrieval returned it, whether the answer stayed faithful to that evidence, whether citations support the exact claim, and whether the user was authorized to receive it. Build slices for common departments, document types, languages, recency ranges, and access tiers. Then report the weakest material slice alongside the average. The NIST Generative AI Profile identifies confabulation, data privacy, information integrity, and component integration as distinct risks; the evaluation should make those distinctions visible rather than compressing them into a generic quality percentage.
| Evaluation gate | Question it answers | Example failure evidence |
|---|---|---|
| Corpus gate | Was an authoritative, current source indexed? | Expected policy is absent or a superseded copy remains active |
| Retrieval gate | Did the system return useful evidence for this question? | Relevant passage falls below irrelevant or duplicate chunks |
| Answer gate | Are claims supported and uncertainty represented? | Answer adds a requirement that no citation states |
| Access gate | Could the requester see every disclosed passage? | A restricted title, snippet, or fact crosses a role boundary |
| Outcome gate | Could the user make the intended decision safely? | User must repeat the search or acts on an ambiguous answer |
Define release rules before running the final evaluation. For example, a service may require zero known cross-tenant disclosures, a minimum retrieval recall on policy questions, a maximum unsupported-claim rate, and successful abstention when no approved source exists. Re-evaluate after changes to chunking, metadata, permissions, embedding, reranking, generation prompts, models, or source collections. Keep challenge outcomes: when a user disputes an answer, record the question, visible sources, system version, decision, and correction. That production evidence reveals missing question classes and stale authority faster than random sampling alone. It also gives content owners a concrete backlog instead of a vague request to improve the knowledge base.
A durable evaluation program works best when it is connected to RAG Knowledge Bases for Support Teams: A Practical Operations Guide, LLM Evaluation for Internal Tools: A Practical Quality Framework, and Retrieval Pipelines for Regulated Records.
Key takeaways
- Plausible language is not enough; evaluate the full retrieval and answer path.
- A test case needs an expected source, user context, answer boundary, and safe next step.
- Human reviewers are essential for applicability and meaning, not just grammar.
- Knowledge and permission changes should trigger targeted regression checks.
- Trust grows when users can inspect sources and correct the system easily.
Frequently asked questions
What is the most useful quality score?
There is no single score that captures a company knowledge base. Use a small dashboard of retrieval success, citation support, access correctness, freshness, user correction, and escalation, then review representative cases. The value lies in locating the source of a failure so a team can improve it responsibly.
How often should production answers be sampled?
Choose a cadence based on workflow impact, traffic, and change rate. High-impact or fast-changing knowledge needs more frequent review. Sampling should be risk-based and privacy-aware, with access limited to reviewers who need the material. Add reported failures and unusual patterns to the next evaluation set.
Conclusion
Evaluation makes a company knowledge base more than a persuasive interface. Separate the layers of quality, build tests from real work, verify change carefully, and keep people able to inspect and challenge answers. That discipline turns retrieval into a dependable support for decisions rather than a source of untraceable confidence.