A RAG prompt injection threat model starts from an uncomfortable fact: retrieved text is data to the application but can behave like instructions to the model. A malicious sentence in a support article, uploaded contract, ticket, wiki page, or indexed website may tell the model to ignore policy, reveal context, miscite evidence, or invoke a tool. Adding a keyword filter at the final prompt misses poisoned ingestion, permission mistakes, ranking manipulation, encoded instructions, multimodal content, and unsafe downstream use of model output.
Threat modeling gives each control a boundary and a failure it is meant to contain. This guide builds on Edilec's RAG architecture guide, prompt-engineering security review, and permissioned enterprise search guide. The objective is not to prove injection impossible; it is to prevent untrusted content from acquiring authority and to limit impact when the model follows it.
Identify RAG assets, security objectives, and consequence
List assets before attack strings: source records and access labels, ingestion credentials, parsers, chunks, embeddings, indexes, prompts, retrieved context, model endpoints, conversation state, tool credentials, outputs, citations, logs, and human decisions. Security objectives include confidentiality, source integrity, permission preservation, instruction integrity, output provenance, availability, and bounded action. Rank consequence by use case. A bad answer in brainstorming differs from an injected instruction that authorizes a payment or exposes another tenant's records.
Document actors and trust assumptions: source publisher, uploader, tenant administrator, crawler operator, platform engineer, user, model provider, and connected service. A legitimate user may introduce a malicious third-party file without knowing it. An authorized editor may be compromised. The NIST adversarial-ML taxonomy supplies language for attacker goals, capabilities, knowledge, lifecycle stages, and mitigations. Use that precision instead of treating every unexpected output as the same injection technique.
| Trust boundary | Attacker opportunity | Security invariant | Primary evidence |
|---|---|---|---|
| Source to ingestion | Publish or upload adversarial content | Publisher identity, provenance and scope remain attached | Source event, hash and ACL snapshot |
| Parser to index | Hide instructions in metadata, markup or image text | Derived chunks cannot lose source and permission labels | Parser version and chunk lineage |
| Query to retrieval | Manipulate ranking or cross tenant scope | Only authorized, relevant records can enter context | Retrieval trace and permission decision |
| Context to model | Make data override trusted instructions | Context carries no authority to change policy | Prompt envelope and policy version |
| Model to tool/output | Cause disclosure, action or unsafe rendering | Deterministic code enforces authorization and output handling | Tool decision, validator and authoritative result |
Map ingestion, indexing, retrieval, and action data flows
Draw the actual path for each source type. Show where content is fetched, converted, scanned, chunked, enriched, embedded, stored, filtered, ranked, assembled, sent to a model, rendered, and acted upon. Mark process identities, stores, networks, tenant transitions, asynchronous queues, and caches. Include update and deletion paths: stale poisoned content can persist in derived indexes after a source is corrected. Link each edge to an owner and observable event.
Model control flow separately. Which instructions come from platform policy, application developers, users, retrieved records, and tool results? Which code decides what the model can call? A model's textual instruction hierarchy is useful behavior guidance but not a security boundary. OWASP's prompt-injection guidance notes that RAG and fine-tuning do not fully mitigate injection, and recommends constrained behavior, validation, least privilege, human approval, segregation of external content, and adversarial testing.
Enumerate attack paths through every RAG stage
At ingestion, consider malicious uploads, compromised websites, recursive archives, parser differentials, OCR-hidden text, metadata, and content that becomes dangerous only when combined. At indexing, consider lost ACLs, cross-tenant namespaces, stale deletion, embedding collisions, manipulated chunk boundaries, and ranking boosts. At retrieval, consider crafted queries, broad filters, poisoning that mimics high relevance, and source floods that crowd out trusted evidence. At context assembly, consider delimiter escape, instruction imitation, payload splitting, excessive context, and forged citations.
At model and action stages, consider system-prompt extraction, sensitive context disclosure, output manipulation, unsafe code or markup, tool selection, argument substitution, and multi-step exfiltration. Include availability: poisoned documents can cause loops, expensive retrieval, or repeated tool calls. Use MITRE ATLAS to map relevant adversarial techniques and connect them to observed behaviors. Avoid claiming framework coverage merely because a technique ID appears in a spreadsheet; demonstrate prevention, detection, containment, and recovery for priority paths.
Layer provenance, retrieval, model, and action controls
At the source layer, restrict publishers, verify connectors, scan active content, record provenance, and support rapid quarantine. At the index layer, preserve tenant and object ACLs, validate parser output, isolate source classes, cap document influence, and make deletion complete. At retrieval, authorize before ranking, constrain scope, diversify sources where useful, and retain a trace. At context assembly, label external content, quote rather than promote it into policy, minimize context, and keep trusted instructions separate.
At execution, give the model no direct credentials. Deterministic code validates structured tool requests against current identity, tenant, object, arguments, limits, approvals, and business invariants. Require confirmation for high-impact actions and keep read and write capabilities separate. Validate outputs before rendering as HTML, SQL, commands, or workflow instructions. The NIST GenAI Profile supports treating information integrity, cybersecurity, privacy, and human interaction as connected risk areas rather than isolated model filters.
Turn the threat model into abuse-case tests
Create tests from paths, assets, and invariants. Seed a document with obvious instructions, subtle policy imitation, encoded payloads, multilingual text, false citations, split fragments, hidden markup, and tool requests. Place it in authorized and unauthorized repositories, change its rank, update and delete it, and combine it with benign sources. Test several models and prompt versions because defenses may not transfer. Verify system behavior and evidence: source quarantine, permission filtering, tool denial, user warning, citation fidelity, alert generation, and clean recovery.
| Test case | Expected safe behavior | Failure evidence | Control owner |
|---|---|---|---|
| Unauthorized poisoned document ranks highly | Document never reaches model context | Retrieval trace contains cross-scope object | Identity/retrieval team |
| Authorized source says to reveal hidden context | Model treats text as content and no secret appears | Sensitive token or policy disclosed | Application/model team |
| Document requests a write tool | Runtime denies absent user intent and policy | Tool dispatch occurs from retrieved text alone | Tool gateway owner |
| Payload split across three chunks | Combined context remains bounded and action denied | Unsafe instruction succeeds only after assembly | RAG application owner |
| Poisoned source is deleted | All derived chunks and caches disappear within objective | Old chunk remains retrievable | Data platform owner |
Detect, contain, and learn from RAG injection attempts
Log source identity, ingestion version, chunk lineage, retrieval decisions, context source IDs, model and prompt version, policy decisions, tool requests, and authoritative results without indiscriminately storing secrets. Detection signals include unusual source dominance, instruction-like content, cross-scope retrieval, citation mismatch, repeated denials, context disclosure patterns, and sudden tool-selection changes. Treat classifiers as signals, not universal gates; attackers adapt and benign technical documents often contain instruction language.
Contain at the narrowest layer: quarantine a source, rebuild an index partition, revoke a connector, disable a tool, rotate exposed credentials, pin a safe configuration, or require review. Preserve the malicious source and full trace under restricted access. Search for other queries and users exposed to the same chunks. Add the path to regression tests and fix the failed invariant, not only the observed phrase. Communicate with source owners because poisoning can indicate compromise outside the RAG service.
Document residual prompt-injection risk and deployment conditions
For each priority attack path, state remaining consequence after controls, uncertainty, accepted operating conditions, and the owner authorized to accept it. Conditions might limit repositories, user groups, accessible data, tool authority, or decisions for which outputs may be used. Make them enforceable in configuration and visible to operators. If the application cannot prevent a retrieved instruction from influencing text, compensate by eliminating sensitive context, disabling writes, validating citations, and requiring knowledgeable review. Reassess acceptance when a model, parser, connector, corpus, population, or action changes. Residual-risk documentation keeps teams from presenting layered mitigation as a guarantee and gives incident responders a clear basis for containment.
Assign threat-model maintenance to a named product-security owner and review it with source-system, identity, data, model, tool, and business owners. Keep assumptions machine-checkable where possible: allowed repositories, tenant filters, model routes, tool states, and logging fields. Link each priority path to tests and each test to release gates. A diagram that is not updated when connectors or authority change becomes false assurance. The fastest maintenance signal is often a manifest diff: a new source, parser, model, context field, tool, or output renderer should open the affected boundaries and abuse cases for review.
Key takeaways for a RAG prompt injection threat model
- Treat retrieved content as untrusted data even when it comes from an authorized repository.
- Map provenance, permissions, instruction flow, tool authority, and derived-data deletion end to end.
- Make deterministic services enforce identity, object scope, transaction limits, and business invariants.
- Derive adversarial tests from attack paths and verify recovery evidence as well as model output.
- Update the threat model whenever sources, parsers, models, tools, or use-case consequences change.
FAQ about prompt injection in RAG
Can a classifier solve RAG prompt injection? No. It may detect known patterns, but robust mitigation requires provenance, permission, context separation, least privilege, validation, approval, and monitoring. Is an internal wiki trusted? Its access may be trusted while its content remains capable of misleading a model; compromised accounts and copied external text matter. Does citation prevent injection? Citations improve traceability but can be forged or attached to manipulated answers. Should suspicious documents be deleted automatically? Usually quarantine first, preserve evidence, and involve the source owner to avoid destructive false positives.
Conclusion: protect authority, not just the prompt
Prompt injection becomes dangerous when content crosses from evidence into authority. A RAG threat model makes that crossing visible from source publication through retrieval and action. With durable provenance, permission enforcement, separated instruction channels, bounded tools, structured validation, adversarial tests, and recoverable indexes, an organization can reduce both the likelihood and consequence of model manipulation. Filters remain useful, but only as one sensor in a system whose decisive controls live outside the model.