A retrieval pipeline is a chain of custody for knowledge. It moves records from an authoritative source through extraction, normalization, chunking, indexing, filtering, retrieval, and answer generation. Every handoff can change meaning, freshness, visibility, or provenance. A fluent answer does not prove that the right document was indexed, that the document is current, or that the requester was allowed to see it.
The original retrieval-augmented generation research describes the value of combining parametric model memory with an explicit non-parametric memory, while also identifying provenance and knowledge updates as open problems. For adjacent implementation context, see the RAG for company knowledge and support guide and the RAG evaluation guide. In production, treat retrieval as a separately testable data product. Give every stage an owner, a version, a reconciliation check, and a failure mode that an operator can act on.
Define the Source Contract
Start with the question the pipeline must answer and the source systems that are authoritative for it. A support assistant may use published troubleshooting articles, approved release notes, and account-specific records, but those sources do not have the same ownership or access rules. Record which source wins when two documents disagree, how quickly a change must become searchable, and what the assistant should say when no approved source supports an answer.
Inventory Records Before Indexing
Create a source inventory with a durable document identifier, canonical location, content type, owner, publication state, effective time, expiry or deletion state, access label, and last successful synchronization. Keep the identifier stable when a document is revised so the system can distinguish an update from a second copy. Capture an immutable version or content digest for each indexed representation. This makes a stale answer traceable to a source change, parser defect, missed event, or index lag.
| Lifecycle stage | Typical failure | Control and evidence |
|---|---|---|
| Source change | A revised or withdrawn record is not emitted to downstream consumers. | Use change events or scheduled inventory comparison; retain source version, event time, and reconciliation result. |
| Extraction | A parser drops a table, heading, attachment, or language-specific section. | Store extraction status, parser version, page or section counts, and a reviewable failure sample. |
| Normalization | Dates, identifiers, headings, or access labels become ambiguous after conversion. | Apply typed transformations with validation errors, field-level provenance, and a quarantine path. |
| Index write | Only some chunks arrive, or an old version remains alongside a replacement. | Use idempotent document-version keys, write manifests, and source-to-index count checks. |
| Query filtering | The result contains relevant text but ignores the requester's entitlement. | Enforce access filters from trusted identity claims and test both allowed and denied fixtures. |
| Answer assembly | The model blends unsupported passages or cites the wrong version. | Return source identifiers and passages with the context; require abstention when support is insufficient. |
Make Extraction and Normalization Observable
Extraction is where useful structure is often lost. Preserve headings, lists, table relationships, footnotes, code samples, page boundaries, and document language when they affect meaning. Do not silently turn an unreadable file into an empty record that appears successful. Route encrypted, malformed, or unsupported files to quarantine and expose that status to the source owner.
Normalize only what the query experience needs. Clean repeated headers and navigation text, but keep the original location and enough surrounding structure to support an answer citation. Store normalized text separately from the source representation. When a transformation changes a number, date, identifier, or access label, fail validation rather than guessing. The NIST Generative AI Profile emphasizes provenance, versioning, and evaluation of data quality across the AI lifecycle.
Choose Chunks for the Question
Chunking is a retrieval decision, not just a token limit. A chunk should contain enough context to answer the intended question and enough specificity to avoid matching unrelated topics. Keep a parent-document identifier, heading path, section position, effective date, and access metadata on every chunk. For procedures, a complete step group may be a better unit than a fixed character count. For a reference table, preserve the header relationship so a value cannot be retrieved without its column meaning.
Test chunk boundaries with real questions. If a question requires two adjacent sections, allow a controlled parent expansion or retrieve linked chunks rather than copying the entire document into the prompt. Keep lexical matching for exact product names, error codes, and identifiers, and use semantic retrieval for paraphrased requests. A hybrid design still needs measured ranking behavior; adding a second retriever does not fix missing or mislabelled source content.
Carry Access Controls Into the Index
Relevance and authorization are separate filters. A highly relevant chunk is still an invalid result if the requester cannot read its source. Derive entitlement metadata from the source system or an approved identity service, validate it during ingestion, and apply it before the model sees retrieved text. Avoid asking the model to decide whether a passage is private; that decision belongs to deterministic application controls.
Test inheritance and revocation. A user who loses access should not continue seeing an old chunk because the index update is delayed. A new group membership should not expose a record before the source authorization is effective. The OWASP Top 10 for LLM Applications calls out sensitive information disclosure, supply-chain vulnerabilities, and excessive agency; retrieval filters are part of the application security boundary for each of them.
Trace Every Record Through the Pipeline
Keep a trace that connects the answer to query normalization, retrieved chunk identifiers, ranking scores or rank positions, access decisions, prompt context, model version, and final citations. The trace does not need to expose sensitive text to every operator; it does need a protected route for authorized investigation. Record enough metadata to answer which source version was used and why a different candidate was excluded.
- Use a stable correlation identifier across ingestion jobs, index writes, queries, and answer events.
- Retain source version, parser version, chunking configuration, embedding or ranking configuration, and prompt version.
- Log filtered candidates and denial reasons in a protected form so access bugs can be distinguished from relevance misses.
- Expose citation links to the canonical record, subject to the same authorization check as the answer.
- Make deletion and replacement operations idempotent so retries cannot create duplicate or resurrected content.
- Alert on reconciliation gaps, permission propagation lag, extraction failures, and unusual empty-result rates.
Evaluate the Whole Retrieval Path
Do not score only the final answer. Build a labelled evaluation set with the question, expected source or source set, allowed answer boundary, entitlement context, and an abstention condition. Measure whether the right evidence appears in the retrieved set, whether the answer stays supported by that evidence, whether citations point to the effective version, and whether denied content remains absent. Evaluate newly ingested documents as well as long-standing ones.
| Test case | Expected retrieval behavior | Acceptance evidence |
|---|---|---|
| Exact error code | Rank the current troubleshooting entry even when the surrounding natural-language question is short. | Expected document in the top result set, current version, and citation to the relevant procedure. |
| Paraphrased policy question | Find the approved policy section and retain the conditions that qualify its rule. | Required clause coverage, heading path, and answer grounded in the retrieved section. |
| Outdated document query | Prefer the effective replacement or explain that the source is expired when no replacement exists. | Version comparison, effective-date decision, and no citation to withdrawn content. |
| Restricted account record | Exclude the record for an unauthorized requester even when it is the most semantically relevant match. | Denied candidate record, safe response, and no restricted text in model context or logs visible to the user. |
| No supporting source | Abstain or ask for clarification instead of filling the gap with general model memory. | Explicit no-support state, user-safe explanation, and absence of an invented citation. |
| Source deletion | Remove all searchable chunks for the deleted version and prevent stale cache reuse. | Deletion event, index reconciliation, cache invalidation, and a denied or empty post-delete query. |
Work Through a Policy Update
Suppose a service team publishes a new refund policy on Monday and marks the previous version withdrawn. The source system emits a revision event. The ingestion worker extracts the new text, preserves its heading structure, applies the policy access label, and produces a manifest containing the document identifier and version. The index writer replaces the old chunk set atomically or marks the transition so a query cannot combine clauses from both versions.
During the propagation window, the assistant should report that the policy is being updated or route the question to a human, depending on the service contract. It should not cite the withdrawn document simply because that vector is still nearby. After the write, an evaluation query checks the qualifying conditions, a deletion query confirms that the old version is absent, and a permission test confirms that only the intended audience can retrieve the policy.
Operate Freshness, Deletion, and Incidents
Monitor each stage separately. A successful scheduled job is not proof that the index matches the source inventory. Compare source counts with indexed document versions, track extraction quarantine, measure time from source change to searchable state, and sample content for structural loss. Set alerts around business impact, such as an urgent policy source that remains stale, rather than relying only on infrastructure health.
When an answer is wrong, trace the record before changing the prompt. Ask when the source changed, which parser and chunking configuration handled it, which metadata reached the index, what candidates were filtered, and what context was passed to generation. That sequence distinguishes a source defect from an indexing gap, ranking issue, entitlement bug, or unsupported answer. The NCSC secure AI system development guidance places logging, monitoring, update management, and incident processes inside secure operation; retrieval needs the same operational ownership.
Define recovery actions before launch: replay a source event, quarantine a parser version, rebuild a collection, invalidate a cache, revoke an access label, or disable answer generation for an affected source class. Preserve the incident timeline and the repaired evaluation case. A quick rebuild without a root-cause record can restore search temporarily while leaving the same failure ready to return.
Key Takeaways
- Treat retrieval as a versioned data product with source ownership, reconciliation, and a clear abstention behavior.
- Give every document and chunk a durable identity, effective state, provenance, and access metadata.
- Preserve structure during extraction and choose chunk boundaries that match the questions the system must answer.
- Apply authorization before generation and test access revocation, inheritance, and delayed propagation.
- Evaluate source selection, support, citation version, and deletion behavior in addition to answer wording.
- Trace incidents through the source-to-index path before changing the model or prompt, then turn the repair into a regression test.
Frequently Asked Questions
How should a team choose chunk size?
Choose a unit that preserves the context needed for the target question. Test headings, procedures, tables, and linked sections with representative queries. A fixed size can be a useful implementation starting point, but it should not split a rule from its conditions or a table value from its header.
Are citations enough to make an answer reliable?
No. A citation is useful only when it points to an authorized, effective source and the answer is actually supported by the cited passage. Test citation version, source coverage, access filtering, and unsupported claims as separate controls.
What should be monitored in production?
Monitor source-to-index freshness, extraction and quarantine rates, reconciliation gaps, empty and low-support results, authorization denials, retrieval quality on a fixed evaluation set, citation validity, and deletion completion. Link those signals to an owner and a documented recovery action.
Conclusion: Make Retrieval Explainable
Reliable retrieval pipelines make the journey from source record to answer inspectable. Build around durable identity, versioned transformations, deterministic access controls, question-shaped evaluation, and operational recovery. When an answer fails, the team should be able to locate the broken handoff and repair that stage instead of hiding the problem with more generated context.