Retrieval-augmented generation pairs a generative model with an information-retrieval system. At query time, the application searches an external knowledge collection and supplies selected evidence to the model. The original RAG paper combined parametric model memory with non-parametric memory to address knowledge-intensive tasks and provenance challenges. Enterprise RAG broadens that pattern into ingestion, authorization, search, reranking, prompt assembly, generation, citation and evaluation services. It is an application architecture, not a guarantee that an answer is true.
Use RAG when the task depends on a changing or private body of knowledge and users benefit from inspectable sources. It is less suitable when the task needs deterministic calculation, authoritative database transactions or knowledge absent from the corpus. A support assistant can retrieve approved troubleshooting procedures; an account balance should come from the transactional system through a controlled tool. Begin with user questions, source authority and acceptable abstention before choosing an embedding model or vector database.
Key takeaways
- Govern source quality, permissions, version and deletion before indexing documents.
- Evaluate retrieval independently from generation so missing evidence is not mistaken for model failure.
- Use hybrid retrieval, metadata filters and reranking when the corpus and questions justify them.
- Carry document-level authorization into chunks, indexes, caches, citations and logs.
- Treat retrieved text as untrusted content that can contain malicious instructions.
- Require citations to support claims and let the system abstain when evidence is insufficient.
Define the task and authoritative corpus
Write an answer contract. Specify users, question classes, sources, recency, languages, response form, citation behavior, refusal conditions and escalation. Decide whether the system summarizes, locates, compares or recommends. The broader the task, the harder it is to establish complete evidence. Select one corpus with an accountable owner, such as approved product manuals and resolved procedures. Exclude drafts, duplicates and personal workspaces unless their authority and access rules are understood.
Create a source registry containing owner, system, document type, approval status, version, effective dates, access policy, retention and refresh method. Preserve provenance from source to parsed document and chunk. Ingestion should be idempotent and observable: detect changed, deleted, failed and quarantined records. A stale index is a data quality incident when policy or safety instructions changed. Deletion must remove primary chunks, derived embeddings, caches and replicas according to the governing rule.
| Corpus decision | Question | Required evidence |
|---|---|---|
| Authority | Which source wins when records conflict? | Owner and precedence rule |
| Freshness | How soon must a change be searchable? | Refresh objective and index timestamp |
| Access | Who may retrieve each document? | Source ACL mapping and negative test |
| Granularity | What unit preserves useful context? | Chunk experiment on representative questions |
| Deletion | Which derived stores contain the record? | Propagation log and reconciliation |
| Quality | Which documents should be excluded? | Quarantine reasons and remediation owner |
Build an ingestion pipeline that preserves meaning
Parsing should retain headings, tables, lists, page references and stable document identifiers. Remove repeated navigation and boilerplate without flattening distinctions. Chunk by semantic structure first, then tune size and overlap from evaluation. Small chunks may match precisely but lose definitions; large chunks can dilute ranking and consume context. Attach metadata for source, section, version, effective date, language and access. Do not infer high-stakes metadata from content when the source system can supply it.
Embedding converts text into vectors for similarity search, but exact terms, identifiers, dates and error codes often benefit from lexical search. Hybrid retrieval combines signals; reranking applies a stronger relevance model to a candidate set. These techniques add latency and cost, so justify them with measured gains. Keep embedding model and index version explicit. A model change can alter neighborhood behavior and require reindexing and evaluation rather than an invisible in-place upgrade.
Design query and retrieval as a testable subsystem
The query path may classify intent, normalize terms, apply user and tenant filters, generate alternate queries, retrieve candidates, fuse rankings and rerank. Keep transformations traceable because a rewritten query can remove a critical qualifier. Apply authorization before content leaves the retrieval boundary, not after the model has seen it. Filters must be derived from trusted identity and source policy. A zero-result response should remain zero results rather than silently broadening to unauthorized or low-quality sources.
Set the number of retrieved passages and context budget from evidence. More context can introduce distraction, conflict and cost. Deduplicate overlapping chunks and preserve enough neighboring text to interpret a match. Where sources disagree, expose the conflict and effective dates instead of asking the model to blend them. Retrieval should return source identifiers and ranking signals along with text. This allows debugging, citation and separate measurement of whether the needed evidence was available.
| Evaluation layer | Metric or question | Failure it reveals |
|---|---|---|
| Corpus | Is the authoritative answer represented and current? | Coverage or freshness gap |
| Retrieval | Recall at k for relevant evidence | Needed evidence never reaches generation |
| Ranking | How early is the best evidence returned? | Weak scoring or chunking |
| Answer | Are claims supported and complete? | Unsupported synthesis or omission |
| Citation | Does each citation entail the associated claim? | Decorative or mismatched reference |
| System | Latency, cost, denial, abstention and user correction | Operational or workflow failure |
Generate answers that remain tied to evidence
Prompt assembly should identify the user task, trusted application instructions, retrieved evidence boundaries and response requirements. Ask the model to use provided sources, distinguish uncertainty and abstain when evidence is insufficient. Citation identifiers should be generated from retrieved source metadata and validated after generation. A citation link proves where text came from, not that the model's claim follows from it. Check claim-to-source entailment for important use cases and display the relevant source passage or section for human verification.
Do not let fluent output conceal missing authority. For policy, legal, medical, financial or safety-sensitive workflows, route decisions to qualified people and authoritative systems. RAG can help locate material and draft a summary; it should not silently turn ambiguous documents into policy. Separate answer text from actions. If the application can call tools, subject those calls to independent identity, authorization, schema validation and approval. Retrieval context must never grant tool authority.
Protect permissions, content and telemetry
Mirror source permissions carefully. Document-level controls may need to be copied to chunks and enforced during every retrieval. Reconcile membership and revocation changes quickly enough for the use case. Segregate tenants in indexes or through strongly tested filters and resource boundaries. Protect service identities, encryption keys and ingestion connectors. Prevent the model from revealing retrieved text beyond the user's purpose, including through summaries, citations, conversation history or cached responses.
Minimize logs and traces while retaining enough provenance to investigate. Prompts, chunks and answers may contain confidential or personal information. Store identifiers and structured evaluation signals where full content is unnecessary, restrict trace access and define retention. Defend ingestion against unsupported types, active content and parser vulnerabilities. Mark external text as untrusted; scan or quarantine where appropriate. Red-team direct and indirect injection, cross-tenant questions, membership changes, extraction attempts and malicious citation targets.
Evaluate offline, in shadow mode and in production
Build a versioned dataset of real question patterns with expected relevant sources, answer criteria and refusal behavior. Include ambiguous, unanswerable, adversarial and permission-sensitive cases. Domain experts should adjudicate a sample and document disagreement. Run retrieval and generation evaluation for every material change to parser, chunking, embedding, ranking, prompt or model. Automated model judges can assist at scale, but calibrate them against human ratings and do not let one model be the sole authority for its own class of output.
Use shadow mode before exposing answers, then pilot with a bounded user group and visible feedback. Monitor retrieval miss, citation use, unsupported claim, abstention, correction, escalation, latency and cost per acceptable answer. Investigate by question class rather than one average. Maintain runbooks for source outage, failed ingestion, index corruption, model degradation and leaked content. Rollback requires versioned indexes and configuration, not only a previous prompt. Reconcile deleted or corrected documents after recovery.
Plan delivery and cost around evidence
A sensible sequence is task framing, corpus audit, retrieval baseline, answer prototype, evaluation set, security review, shadow operation, pilot and expansion. Start with lexical and vector baselines before adding query rewriting or agents. Each component should earn its complexity through measurable improvement. Define ownership across knowledge, application, security and operations. A production RAG service needs a content correction path and service support, not just model engineers.
Cost includes source integration, parsing, embedding and reindexing, storage, search, reranking, model tokens, evaluation, human review, telemetry and support. Model cost per accepted user outcome and include failed retrieval, retries and correction. Caching can reduce spend but must respect identity, source version and deletion. Keep assumptions about document volume, refresh rate, query mix and context length. A small curated corpus can outperform a large unmanaged one while costing less to govern.
Frequently asked questions
When should RAG be used instead of fine-tuning?
RAG fits changing, private knowledge that benefits from source inspection. Fine-tuning is often better for behavior, format or task adaptation, though it does not provide a simple way to update or cite facts. Systems can combine them. Choose from the task and evaluation, not a presumed hierarchy.
Is a vector database required?
No. Small or term-heavy corpora may work with conventional search. Vector retrieval helps semantic matching; hybrid search often handles mixed language. Select infrastructure after a baseline shows which retrieval errors need improvement.
Does RAG eliminate hallucination?
No. It can provide useful evidence and provenance, but retrieval may be wrong and generation may misread or ignore sources. Evaluate retrieval, support, citation and abstention separately and retain human review for consequential use.
Conclusion
A reliable RAG system is primarily a governed knowledge and evaluation system. Define the question boundary, curate authoritative sources, preserve provenance and enforce permissions before retrieval. Measure whether the right evidence arrives, whether claims follow from it and whether the system abstains appropriately. Protect tools and data from untrusted content, version every material component and operate ingestion as a production pipeline. With those foundations, retrieval can make generative interfaces more current and inspectable without pretending that fluent synthesis replaces authority.