A prompt engineering security review examines the entire application that assembles instructions, retrieves content, calls a model, parses output and invokes tools. The prompt itself is not a security boundary. Models process trusted instructions and untrusted data through the same probabilistic mechanism, so a hidden instruction in a webpage, email or file can influence behavior. OWASP notes that foolproof prompt-injection prevention is not known. The practical goal is layered containment: minimize authority, preserve provenance, validate outputs, require approval and detect abuse.
Scope the system, assets and authority
Draw the data and control flow from user identity through application policy, prompt templates, retrieval, model provider, memory, output parser, tools and external effects. Include logs, evaluation stores, caches and administrator interfaces. Mark trust boundaries and deployment owners. Identify assets: confidential data, credentials, system instructions, model access, customer records, tool authority and decision integrity. State the maximum consequence if the model follows a malicious instruction. A read-only summarizer and an agent that can send money require very different reviews.
Inventory every input channel, including direct prompts, uploaded documents, images, retrieved pages, search snippets, database content, tool results and stored conversation memory. Untrusted instructions can be indirect and may be invisible to a user. Record who can influence each source and whether content crosses tenants or security domains. Define prohibited data and actions. Classify outputs by destination: display, structured storage, code interpreter, browser, email, database or external API. The destination determines injection and encoding risk.
| Boundary | Threat question | Required evidence |
|---|---|---|
| User to application | Can identity or tenant context be spoofed? | Server-side authorization tests |
| Content to prompt | Can untrusted data act as instruction? | Provenance labels and injection probes |
| Model to parser | Can malformed output bypass policy? | Schema and rejection tests |
| Model to renderer | Can output trigger script or unsafe links? | Contextual encoding and browser tests |
| Model to tool | Can it exceed user or service authority? | Allowlist, least privilege and approval tests |
| Memory and logs | Can secrets persist or cross sessions? | Retention, isolation and access review |
Threat-model direct and indirect prompt injection
Direct injection arrives through the user's own input; indirect injection arrives inside content the application retrieves or opens. Test attempts to override instructions, reveal confidential context, alter tool parameters, suppress citations, persist malicious memory or manipulate later users. Include encoded, obfuscated, multilingual and multimodal variants appropriate to the system. Jailbreak testing explores bypass of model safeguards, while prompt injection focuses on unwanted application behavior; both matter, but business impact depends on the authority and data surrounding the model.
Do not treat a secret system prompt as protection. Assume determined users can infer or elicit parts of it, and never place credentials there. Instruction hierarchy and clear delimiters can improve behavior but cannot reliably separate executable instruction from data. Retrieval-augmented generation and fine-tuning do not eliminate injection. Model the attacker as controlling any external document the system reads. The strongest mitigation is architectural: even a compromised model response cannot access forbidden data or execute a high-impact action without independent authorization.
Control context, retrieval and memory
Retrieve only from approved sources and apply the requesting user's permissions before content enters context. Avoid retrieving broadly and asking the model to ignore unauthorized sections. Keep source identifiers, document version and trust classification with each chunk. Separate instructions from content structurally and state that content may be hostile, while recognizing this is only one layer. Limit context to what the task requires. Scan or quarantine active content where applicable, and do not automatically follow links or embedded references from retrieved material.
Treat memory as stored application data. Define what may persist, for how long, at what tenant and user scope, and who can delete it. Do not write model-generated instructions into durable memory without validation. Prevent one user's content from becoming another user's trusted context. Protect vector indexes, prompt traces and evaluation corpora according to source sensitivity; embeddings are not automatic anonymization. Redact secrets before logging, restrict access and set retention. Test deletion through derived stores and backups according to policy.
Validate output and contain tool execution
Treat model output as untrusted input to the next component. Require typed schemas, length limits, allowlisted enums and semantic validation. Encode for the actual destination to prevent script, SQL, command, template or markup injection. Never execute generated code or queries with broad credentials merely because they parse. When free text is sent externally, scan for secrets and policy violations and show a preview. Separate model explanation from machine-action fields so persuasive prose cannot smuggle parameters into a tool call.
Expose narrow tools such as getorderstatus or propose_refund, not a generic database, shell or network client. Each tool should authenticate independently, authorize the user and tenant, validate parameters, enforce business constraints and use a dedicated least-privilege identity. Set rate, cost, recursion and time limits. Use idempotency and uniqueness for side effects. Require human approval for high-impact actions with exact target, amount and consequence. Log request, authorization, approval, execution and result without leaking sensitive payloads.
| Control | What it reduces | What it does not guarantee |
|---|---|---|
| System instructions and delimiters | Casual instruction confusion | Prevention of determined injection |
| Input filtering | Known patterns and prohibited content | Detection of novel or encoded attacks |
| Permission-aware retrieval | Unauthorized context exposure | Factual correctness of allowed content |
| Structured output validation | Malformed or unexpected fields | Truth or business appropriateness |
| Least-privilege tools | Impact of model compromise | Correct intent for an allowed action |
| Human approval | Unreviewed consequential action | Reliable review without evidence and time |
Run an adversarial, system-level test plan
Build a versioned suite from system assets and likely attackers, informed by resources such as MITRE ATLAS and OWASP. Include benign task cases, direct injection, malicious retrieved content, data-exfiltration requests, cross-tenant probes, tool escalation, output injection, denial-of-wallet, malformed files and memory poisoning. Test every supported modality and tool combination. Assertions should examine final system behavior: data returned, tool invoked, side effect, log and user message. A model refusing in text while still issuing a harmful tool call is a failure.
Use deterministic automated checks where possible and trained human review for nuanced harmful behavior. Track attack success by consequence and segment, not one aggregate safety score. Re-run after model, prompt, retrieval, parser, tool or policy changes. Test provider outage, timeout and degraded mode. Conduct manual red teaming before high-authority release, then convert discoveries into regression cases. Protect test details and generated sensitive content appropriately. Independent review is useful for high-impact systems, but it does not transfer operational responsibility.
Gate release and prepare incident response
- Approve a threat model with assets, actors, trust boundaries and maximum authority.
- Verify identity and tenant controls before retrieval and every tool call.
- Run benign quality, injection, leakage, output and tool-abuse suites.
- Confirm logs, alerts, budgets, kill switch, fallback and rollback work in production-like conditions.
- Pilot with read-only or proposal authority and review anomalous sessions daily.
- Increase authority only for bounded actions that meet explicit evidence thresholds.
- Record findings, owners and residual risk; retest every material system change.
Monitor authorization failures, unusual retrieval, repeated policy rejection, tool-call volume, high-cost sessions, malformed outputs, human overrides and confirmed incidents. Avoid logging full prompts by default when they contain sensitive data; retain the minimum needed for security and evaluation. Establish a reporting path for users and researchers. Incident response should support disabling tools, revoking credentials, isolating a model or retrieval source, preserving evidence, notifying affected parties and correcting persistent memory. Exercise the kill switch and fallback before an incident.
Maintain an inventory of prompts, model versions, source collections, tools, credentials, owners and evaluation results. Review provider changes and vulnerability disclosures. NIST's generative-AI profile emphasizes lifecycle governance and testing; NCSC guidance similarly spans design through operation. Prompt systems should also follow ordinary secure software practice: protected repositories and pipelines, code review, dependency management, provenance and vulnerability response. AI-specific threats add to, rather than replace, application and cloud security.
Document residual risk in language an accountable owner can act on: the attack path, affected asset, maximum consequence, current containment, evidence and next review trigger. Avoid a generic statement that injection remains possible. A useful record might say that hostile documents can alter summaries, but retrieval is permission-scoped, tools are disabled and users see citations before publishing. This clarity supports an informed release decision and tells responders which boundary must remain intact.
Key takeaways
- Review the complete prompt application, not the template in isolation.
- Assume retrieved files and pages can carry hostile instructions.
- Enforce retrieval and tool authorization in deterministic code.
- Validate output for its destination and minimize model authority.
- Test final behavior adversarially and prepare a real containment path.
Frequently asked questions
Can prompt injection be completely prevented?
Current authoritative guidance does not support that claim. Layer behavior guidance, source control, filtering and evaluation, but design the system so successful influence has limited data access and action authority. Containment and detection are essential.
Should system prompts be hidden?
Protect them as configuration and avoid unnecessary disclosure, but do not rely on secrecy for security. Never store credentials in prompts. Assume instructions may be inferred and enforce actual permissions outside the model.
Does RAG make prompts secure?
No. It can improve grounding but introduces an indirect-injection path and may expose unauthorized or poisoned content. Apply permission-aware retrieval, provenance, context minimization, citation checks and adversarial tests.
When is human approval required?
Require it for consequential, hard-to-reverse or policy-sensitive actions unless rigorous evidence and governance justify a narrower automated boundary. Approval must display trustworthy evidence and exact effect; a generic confirmation is weak control.
Conclusion
Secure prompt engineering is mostly secure system engineering around an untrusted, probabilistic component. Map every source and destination, assume external content can manipulate the model, authorize outside it, constrain tools and validate outputs. Then test the final effect under realistic attacks and failures. No clever instruction can replace these boundaries. A prompt system is ready only when the team can contain compromise, explain evidence and recover without granting the model more trust than it can support.