An AI guardrails security review asks whether untrusted people, content, models and dependencies can cross a boundary that software was supposed to enforce. It does not assume the model will reliably obey a security instruction. The review follows data from user input and retrieved documents into the model, then treats every output, tool request and generated artifact as untrusted until conventional software validates and authorizes it. It also examines how prompts, models, connectors and policies change after release.
This is narrower than general responsible AI review and deeper than content moderation. Use Edilec's plain-language AI guardrails guide, AI agent architecture guide and RAG systems review for adjacent product and architecture decisions. The security review should align threats from OWASP's LLM application risks and MITRE ATLAS with the application's actual trust boundaries and business actions.
Scope the system and protected outcomes
Inventory the model providers, prompts, retrieval indexes, document ingestion, memory, tools, agents, queues, user interfaces, moderation services, telemetry and human approval. Mark external services and administrative paths. For each component, identify data class, credential, owner, update route and failure mode. Then state protected outcomes: one tenant cannot access another tenant's records; a model cannot authorize a payment; untrusted documents cannot change system policy; generated code cannot execute without an isolated review path.
Define actors beyond an obvious malicious user. A compromised connector, poisoned knowledge document, overprivileged employee, malicious web page, vulnerable model dependency or mistaken administrator can create the same outcome. Include indirect prompt injection, where instructions arrive inside content the model is asked to summarize or retrieve. Record which boundaries rely on deterministic software and which rely on probabilistic detection. Controls that depend only on model behavior should be treated as weaker and paired with enforcement.
| Boundary | Security question | Required enforcement |
|---|---|---|
| User to application | May this identity invoke this workflow? | Authentication, role and rate policy |
| Application to retrieval | Which records may enter context? | Tenant and record authorization before ranking |
| Content to instructions | Can untrusted text alter policy? | Separation, delimiting and constrained task |
| Model to tool | Is this requested function and input permitted? | Tool allowlist and schema validation |
| Tool to resource | May the current subject perform this action? | Resource-level authorization and transaction limits |
| Telemetry to operators | Can evidence expose secrets or be altered? | Redaction, access, integrity and retention |
Test direct and indirect prompt injection
Direct tests place conflicting instructions in user input. Indirect tests place them in retrieved files, web pages, emails, images or tool results. The goal is not only to make the model say something prohibited. Test whether injected content can change data scope, select an unintended tool, disclose hidden instructions, cause unsafe output handling or manipulate an approver. Use realistic encodings, multilingual content, long documents and multi-turn persistence. Preserve test cases so model and prompt updates can be compared.
Instruction hierarchy and delimiters can reduce confusion but cannot create a hard trust boundary. Keep sensitive system details out of prompts where possible. Label untrusted content, minimize the task, and prevent it from selecting arbitrary tools or destinations. Apply policy after generation. For example, a document summarizer should receive only documents the user can read, and its output should remain text; it should not inherit a general email tool merely because the model proposes sending the summary.
Protect data, prompts and model assets
Trace sensitive information through ingestion, vectorization, prompts, provider processing, cache, conversation history, feedback, evaluation and logs. Confirm purpose, retention, region and deletion for each store. Embeddings and derived features may still reveal or link sensitive source data, so they require access and lifecycle controls. Filter retrieval by authorization before similarity search and prevent one tenant's identifiers or metadata from influencing another's results. Test deletion through source, index, cache and backup according to policy.
Protect system prompts, policies, model endpoints and administrative interfaces from unauthorized change. Prompt disclosure is not automatically a breach, but secrets and security assumptions should never rely on prompt confidentiality. Pin or record model versions and providers; review updates that can alter behavior. Verify downloaded models, libraries and container provenance. The UK NCSC secure AI development guidance emphasizes secure design, development, deployment and operation across the AI lifecycle, including supply-chain responsibilities.
- Do not place API keys, private credentials or enforcement secrets in prompts.
- Authorize retrieval before ranking and generation.
- Redact sensitive values from routine traces and evaluation exports.
- Restrict prompt, policy, connector and model configuration changes.
- Record model and dependency versions for every release.
- Test deletion and tenant isolation across derived stores and caches.
Constrain tools and excessive agency
Every tool should expose a narrow operation with typed parameters and a dedicated credential. The application decides which tools are available for the current workflow; the model does not discover broad internal APIs by default. Validate parameter shape, allowed values, resource scope, rate, destination and current authorization. Separate read from write and preview from commit. Require independent approval for irreversible, high-value or external actions. Use sandboxing for generated code and file processing, with network and filesystem limits.
Test confused-deputy scenarios. A user with access to the AI interface should not gain the agent's broader service privileges. A retrieved document should not cause the agent to act as an administrator. A low-risk tool should not accept a parameter that turns it into an arbitrary request proxy. Verify server-side authorization on the final resource, because upstream checks can be bypassed through retries, alternate interfaces or compromised components. Set iteration, time and cost limits so a looping agent cannot exhaust services or produce uncontrolled effects.
| Test | Attempt | Secure result |
|---|---|---|
| Cross-tenant retrieval | Reference another tenant's identifier | No record enters ranking or context |
| Indirect injection | Embed tool instructions in an approved document | Content is summarized but cannot alter authority |
| Parameter smuggling | Place extra command in a free-text tool field | Schema rejects or treats it only as data |
| Replay | Repeat a previously approved action | Idempotency prevents duplicate effect |
| Approval substitution | Change parameters after human review | Execution binds to reviewed immutable request |
| Resource exhaustion | Create recursive or high-volume tool calls | Budget and iteration limits stop safely |
Handle model output as untrusted data
Generated HTML, Markdown, SQL, code, filenames and URLs can become injection vectors when another component renders or executes them. Encode for the destination, sanitize active content, use parameterized queries, restrict schemes and domains, and avoid shell concatenation. Validate structured output against a strict schema and reject unexpected fields. A model that correctly refuses most malicious requests can still produce a dangerous string accidentally; downstream safety must not depend on intent.
Do not let generated citations or links create false trust. Resolve sources from controlled identifiers and verify that a cited record supported the claim. For code assistance, isolate execution and require ordinary review, tests and dependency controls. For user-facing content, apply moderation proportionate to risk and provide reporting. Monitor whether guardrails disproportionately block legitimate groups or languages, since a security control that silently denies service can create operational and fairness harm.
Review human approval as a security control
Human review can stop a consequential action only if the reviewer sees trustworthy context and the executed request is exactly the reviewed request. Show source records, proposed change, destination, consequence, uncertainty and policy warnings. Prevent the model from hiding parameters inside a summary. Bind approval to an immutable action hash or equivalent server-side record. Expire approval when inputs change. Separate requester and approver when the risk requires it, and monitor rubber-stamping caused by excessive queue volume.
Review account compromise and social engineering. An attacker may craft a persuasive explanation rather than a technically unusual request. Give reviewers independent signals and a route to inspect originals. Require step-up authentication for sensitive approvals and protect recovery. The NIST Generative AI Profile highlights human-AI configuration and information-integrity risks; the review interface is part of the attack surface, not a neutral safety layer.
Use an AI security boundary matrix
The matrix below reviews six boundaries independently: instructions, retrieved data, model output, tool selection, resource execution and telemetry. Each cell names what is untrusted and where deterministic enforcement belongs. This avoids a common review failure in which strong input moderation is credited for protecting a downstream action it cannot actually control. Test at the boundary and then test the complete chain.
Prepare detection, containment and recovery
Log model and prompt version, policy decision, retrieval identifiers, tool request, authorization result, approver and outcome with redaction. Detect unusual denied actions, cross-boundary attempts, rapid tool loops, new destinations, sensitive-output patterns and configuration changes. Preserve evidence without storing unnecessary private content. Give responders switches to disable a tool, model, connector, tenant or feature independently. Rotate credentials, quarantine poisoned content, rebuild indexes and revoke queued actions as part of recovery.
Run an incident exercise in which a trusted document contains an injected instruction and an agent proposes a privileged action. Ask whether the team can identify affected conversations, stop execution, remove the content, assess data exposure, notify owners, restore clean indexes and update tests. Feed confirmed attack patterns into the regression set. MITRE ATLAS can help structure adversary techniques, but the response plan must map them to the product's identifiers, logs and control owners.
Key takeaways
- Map every model, data, tool, administrative and telemetry trust boundary.
- Test indirect prompt injection through the same content sources production will use.
- Keep authorization and action limits in deterministic software outside the model.
- Treat generated output as untrusted for its destination context.
- Bind human approval to the exact action and evidence reviewed.
- Prepare targeted containment and recovery for models, tools, connectors and indexes.
Frequently asked questions
Can prompt injection be completely prevented?
No general model-level method guarantees that untrusted content cannot influence behavior. Reduce exposure and enforce data and action boundaries outside the model so a successful influence attempt cannot cross into unauthorized effects.
Is red teaming enough for release?
No. Red teaming discovers failure modes, while architecture, authorization, secure delivery, regression tests, monitoring and response make controls durable. Findings should become repeatable tests and design changes.
Does a managed model provider cover application security?
The provider secures defined model-service responsibilities. The application owner still controls prompts, data selection, tools, credentials, authorization, output handling, user experience, monitoring and incident response.
Conclusion
A strong AI guardrails security review assumes the model can be mistaken or manipulated and asks what the surrounding system still prevents. Scope assets and outcomes, test hostile content, constrain tools, authorize at the resource, protect outputs and evidence, and rehearse containment. Security becomes credible when a model failure remains a bounded product failure instead of becoming unauthorized access or action.