Vector search retrieves items whose representations are close in an embedding space. That can improve discovery when users express concepts rather than exact terms, but similarity is not relevance, permission, truth, or business priority. A production implementation needs a corpus contract, evaluation set, access controls, freshness path, and an explicit decision about where approximate retrieval is acceptable.
This vector search implementation checklist covers the retrieval service itself. Edilec's embeddings mistakes guide, RAG architecture guide, and semantic search production guide address adjacent model and product choices.
Key takeaways
- Define relevance with judged queries and task outcomes before choosing an index.
- Preserve source identity, version, permission metadata, chunk position, and deletion state beside every vector.
- Apply authorisation before content is returned; filtering strategy must be tested with restrictive real cases.
- Compare approximate results with exact search to measure recall and tune latency deliberately.
- Version embeddings and indexes, support dual-read migration, and prove deletion across derived stores.
Write the retrieval contract
Name the user task, corpus, unit returned, top-k limit, latency objective, freshness expectation, and fallback. Search for support resolution may return passages with document links; product discovery may return products with availability and category rules. Decide whether lexical matches, semantic matches, recency, popularity, and business constraints are combined. A vector database should not silently decide policy through distance alone.
Create a query set from actual logs and interviews, protecting personal or confidential information. Include navigational queries, paraphrases, acronyms, rare entities, exact codes, ambiguous terms, negation, and no-answer cases. Label relevant items and importance with domain reviewers. Use separate development and holdout sets so tuning chunk size and index parameters does not overfit the final report.
| Query class | Example intent | Expected behavior |
|---|---|---|
| Exact identifier | Find policy FIN-042 | Lexical or metadata path returns exact item first |
| Conceptual | How are supplier bank changes approved? | Relevant procedure sections rank highly |
| Ambiguous | Mercury limits | Clarify domain or show distinct clusters |
| Restricted | Executive compensation process | Return only items permitted to caller |
| No answer | Policy for an unsupported country | State no reliable result; do not invent one |
Build traceable chunks and metadata
Chunk along semantic and structural boundaries where possible: headings, paragraphs, table rows with headers, code units, or product descriptions. Preserve enough surrounding context to interpret the passage, but avoid embedding an entire long document when the task needs precise evidence. Store source ID, source version, title, section path, offsets, language, timestamps, owner, content type, and access attributes. A result must map back to a viewable current source.
Normalize conservatively. Removing identifiers, punctuation, or table structure may destroy the exact clues a user needs. Deduplicate boilerplate without collapsing legitimate repeated clauses. Keep the raw canonical content outside the vector index and treat embeddings as derived data. The ingestion event should be idempotent, and deletion or permission changes should propagate with measurable latency.
Choose embeddings and index from evidence
Evaluate candidate embedding models on your judged set, languages, dimensions, throughput, privacy constraints, and cost. Record model identifier and preprocessing version on every vector. Begin with exact nearest-neighbour search when the corpus and latency allow it; it provides a reference for quality. The pgvector documentation notes that approximate indexes such as HNSW and IVFFlat trade recall for speed, with different build, memory, and query characteristics.
Tune against service-level evidence. Compare approximate top-k with exact top-k for representative filters and corpus sizes. Increase candidate breadth or search parameters only when relevance gain justifies latency and compute. Monitor index size, build time, update rate, memory, and tail latency. Do not assume benchmark results from a uniform public dataset transfer to a multilingual, highly filtered enterprise corpus.
| Measure | What it reveals | Release use |
|---|---|---|
| Recall@k | Whether judged relevant items are retrieved | Primary relevance gate |
| MRR or nDCG | Whether better items rank earlier | Ranking comparison |
| ANN recall | Approximate overlap with exact results | Index tuning gate |
| Filter completion | Whether restrictive queries return enough items | Access and product correctness |
| P95 latency | Tail response under target load | Capacity gate |
| Freshness lag | Delay from source change to searchable state | Operational objective |
Enforce permissions and filters correctly
Authorisation belongs to the retrieval boundary, based on current identity and policy. Do not retrieve forbidden content and rely on a language model or interface to hide it. Metadata must carry tenant, role, classification, geography, lifecycle, and other required constraints. The service should fail closed when permission data is missing or stale, and audit the policy decision without logging unnecessary query content.
Filter timing changes results. OpenSearch's filtering documentation distinguishes efficient filtering, post-filtering, and exact approaches; post-filtering can yield fewer than k results under restrictive filters. Test the algorithm and engine you actually deploy. A query limited to one small department may need exact filtered search or a larger candidate scan, not the default ANN setting.
Use hybrid retrieval and reranking deliberately
Lexical search remains strong for names, identifiers, quoted phrases, and rare terms. Combine lexical and vector candidates when the task benefits, normalise scores carefully, and use a transparent fusion method or learned reranker validated on the same query set. Apply hard policy filters before ranking and business boosts only when documented. Keep raw component scores for diagnosis; one opaque final score makes regressions difficult to explain.
For a support knowledge base, retrieve 30 lexical and 30 semantic candidates, deduplicate by source passage, apply permission and product-version filters, rerank the remaining set, and return five cited passages. If all candidates are weak, the system should ask for clarification or return no result. NIST's AI RMF supports this measured, lifecycle approach to mapping context and managing observed risk.
Release, monitor, and re-embed safely
Build a versioned index beside production, validate document counts, permission distributions, missing vectors, and judged quality, then shadow queries. During migration, write updates to both versions or keep a replayable change log. Compare results by query cohort before shifting traffic gradually. Retain the prior index until rollback and deletion reconciliation are proven.
Monitor zero-result rate, weak-score rate, click or downstream success where meaningful, reformulations, permission denials, stale-source clicks, latency, and cost. Sample results for expert judgment because clicks can reward attractive but wrong items. Protect against untrusted content influencing downstream generation; the OWASP LLM application risks include prompt injection and sensitive disclosure that retrieval systems must account for.
- Canary by tenant or query cohort and compare against a stable baseline.
- Alert on ingestion backlog, permission-sync lag, missing-source links, and deletion failures.
- Run exact-versus-approximate recall checks on a scheduled sample.
- Keep a kill switch to lexical-only or prior-index retrieval.
- Reconcile every source deletion across raw content, chunks, vectors, caches, and logs.
Diagnose retrieval failures systematically
When a result is wrong, classify the failure before tuning. The needed content may be absent or stale; segmentation may have separated a heading from its meaning; preprocessing may have removed an identifier; the embedding may not represent the language; a permission filter may have removed the only relevant passage; ANN search may have missed an exact neighbour; fusion may have over-weighted popularity; or reranking may have promoted a plausible but weaker item. Preserve intermediate candidates and scores so investigators can locate the failing stage.
| Failure class | Diagnostic comparison | Likely response |
|---|---|---|
| Corpus | Search source system directly | Repair ingestion or freshness |
| Chunking | Inspect neighbouring source spans | Change segmentation by content type |
| Embedding | Compare model candidates on judged set | Re-embed only after holdout gain |
| ANN | Run exact search with same filters | Tune index or use exact cohort path |
| Filtering | Compare authorised candidate counts | Fix metadata or filter strategy |
| Ranking | Inspect lexical, vector, and reranker lists | Retune fusion with judged queries |
Maintain a failure library with the original query, caller policy context, expected sources, observed candidates, root cause, and fix version. Add each material case to regression evaluation without allowing the set to become only a collection of past successes. Periodically refresh with new queries and documents. Search quality changes as vocabulary, products, permissions, and the corpus evolve, so yesterday's benchmark is not a permanent certificate.
Operational incidents also need a customer-safe response. If permission metadata is stale, fail closed or fall back to a source that enforces current access. If embedding ingestion is delayed, expose freshness and use lexical search over current content where appropriate. If a new index harms ranking, route back to the prior version without losing recent deletions. Recovery is complete only after caches, candidate stores, and user-visible links reconcile with the canonical source.
Frequently asked questions
Do we need a dedicated vector database?
Not necessarily. Evaluate corpus size, filtering, update rate, latency, operations, and existing data systems. A current search engine or relational extension may be sufficient and simpler. Choose on measured requirements, not the label on the product.
What is the best chunk size?
There is no universal token count. Use the smallest unit that preserves enough meaning for the task, then evaluate alternatives by document type and query cohort. Tables, code, policies, and transcripts often require different segmentation.
How do we know vector search is production-ready?
Require holdout relevance results, permission tests, restrictive-filter tests, load and failure tests, freshness and deletion objectives, a versioned rollback, and named owners. A good demonstration query is not release evidence.
Conclusion
Reliable vector search is an evaluated retrieval service, not an index feature. Ground it in real queries, traceable sources, current authorisation, measured recall, and versioned operation. When the team can explain missing results, restricted results, migrations, and deletion, semantic relevance becomes useful without outrunning control.