Prompt engineering is the work of specifying a model task with instructions, context, examples, output constraints, and evaluation. It is not a security boundary. A prompt can guide behaviour, but untrusted text, retrieved documents, and user messages can still compete with or subvert those instructions. Engineering teams should therefore review a prompt-enabled feature as an application: what data enters, what authority the model has, which outputs are accepted, and who owns a failure. The NIST Generative AI Profile supports this system-level view by treating governance, measurement, and management as connected activities.
Define The Work Boundary
Describe the workflow in a sentence before drafting instructions. A useful boundary might be: a support assistant may summarize approved case notes for an authenticated agent, but it may not disclose account data to a customer or alter a record. Specify the input sources, the allowed result, the prohibited result, and the human handoff. That makes security review concrete. It also stops a team from mistaking a general-purpose chat interface for a capability with a known purpose. For the practical mechanics of task specification, the plain-language prompt engineering guide provides useful adjacent context.
| Review question | Weak pattern | Stronger design |
|---|---|---|
| Where does untrusted content enter? | Retrieved files and user text are inserted beside privileged instructions with no label or separation. | Mark content by trust level, delimit it clearly, and keep policy decisions in application code. |
| What can the output do? | Free-form text is passed directly into a tool or workflow. | Parse a constrained schema, validate every field, and authorize the resulting action independently. |
| Who can see context? | A broad system prompt or shared thread carries data across users and tenants. | Scope retrieval, session state, and logs to the requester and purpose. |
| How is prompt change controlled? | Instructions are edited in production without test cases or approval. | Version prompts, review material changes, and compare releases against a fixed evaluation set. |
Design The Instruction Boundary
Separate three things that are often mixed together: business policy, model instructions, and user-provided content. Business policy belongs in deterministic code and access rules. Model instructions explain the bounded task, format, and escalation behaviour. User content is evidence or a request, never an authority to redefine the application. This separation makes prompt injection easier to reason about. A malicious message may still be shown to the model, but it cannot grant itself a tool, change a role, or bypass validation. OWASP's LLM guidance is particularly useful when mapping those indirect prompt-injection paths.
- Use structured outputs for decisions that feed software, then reject fields that do not match the expected schema.
- Give the model the least context and the least tool authority needed for the immediate task.
- Keep secrets, credentials, and privileged policy logic outside prompts and model-visible conversation history.
- Add explicit instructions for uncertainty, unsupported requests, and escalation, but enforce sensitive rules in code.
- Log prompt version, input provenance, validation result, and tool outcome with appropriate data minimization.
- Review translated, pasted, and document-derived instructions because indirect injection can arrive through any content channel.
Test Adversarial Cases
A security review needs more than examples that demonstrate the intended answer. Build cases containing conflicting instructions, a document that asks the model to reveal its system message, an unauthorized request embedded in a legitimate ticket, malformed structured output, and a request that should be escalated. Test the model response and the application response. The important result is often that the model noticed an instruction but the application declined the consequential action. The UK secure AI development guidance is a useful reference for treating testing, deployment, and monitoring as part of the same assurance work.
Review Tool Use Separately
Tool calling changes the risk profile because generated text can request an action. Give each tool a narrow purpose, typed parameters, a timeout, idempotency behaviour, and an audit event. The server receiving the call must authenticate the caller and check authorization for that exact operation; it should not trust that the assistant already did so. High-impact actions such as refunds, permission changes, and external messages deserve a confirmation or human review gate. The OpenAI Agents guide is a useful implementation reference, but its patterns still need to sit inside the organisation's own identity and approval model.
| Failure signal | Likely cause | Containment step |
|---|---|---|
| The model follows text from a retrieved document | Untrusted content was given the same status as application instructions. | Quarantine the source, retain the trace, and add a test that proves the application ignores the attempted override. |
| A tool receives unexpected parameters | The output schema or server-side validation is too permissive. | Reject the call, alert the owner, and require explicit allowlisted fields and ranges. |
| Users obtain context from another account | Retrieval or session keys are not scoped to tenant and actor. | Disable the affected path, investigate access logs, and correct authorization before re-enabling. |
| A prompt edit changes sensitive behaviour | Prompt releases are not evaluated or approved as configuration changes. | Restore the prior version and add regression cases covering the changed instruction. |
Operate The Review Loop
Treat prompts as versioned production configuration. Record their purpose, owner, expected input classes, evaluation cases, release date, and rollback condition. Monitor refusal and escalation rates, structured-output validation failures, tool-call denials, user corrections, latency, and cost. None of these signals alone establishes safety, but together they reveal drift and unexpected usage. Sample traces after important changes, especially when a new retrieval source or tool is introduced. A short review that connects a real failure to a test or rule is more valuable than a large prompt library with no operating owner.
Governance And Change Control
Governance for prompt engineering should describe the instruction set and tool boundary in the same practical terms used to run the service: who can change it, what evidence is required, how a change is reviewed, and how the previous state can be restored. Treat a new source type, tool, or user role is introduced as a production event rather than routine maintenance. The owner should assess whether the change alters the permitted decision, data scope, safety controls, user explanation, or support obligation. Record the outcome in a change log that links to test results and the responsible approver. That modest discipline avoids a common failure mode in which a technically small update changes behaviour but nobody can later explain why.
Build A Review Pack
Keep a compact review pack for prompt engineering, made from adversarial prompts, structured-output failures, and denied actions. For every example, retain the expected result, the evidence a reviewer should inspect, the unacceptable result, and the recovery action. Refresh the pack when operations expose a new failure class, but preserve a stable core so releases can be compared over time. Invite the people who own source records, resolve exceptions, and answer customer questions to review samples with engineering. Their observations often reveal that a failure is caused by an outdated record, confusing state, or incomplete policy rather than a model defect. A useful review ends with an owner and a measurable follow-up, not a vague request to improve quality.
Turn Feedback Into Improvement
Do not expand prompt engineering simply because early users like the experience. First classify feedback by severity, frequency, affected user group, and reversibility. Fix issues that expose protected data, create unsupported decisions, or trap people in an unclear state before pursuing broader coverage or lower costs. Then decide whether the repair belongs in source stewardship, interface design, policy logic, model configuration, evaluation data, or team training. This framing keeps the improvement loop honest: the model is one component of a service with people and systems around it. Publish the decision and its expected signal, then verify after release that the change reduced the observed problem without moving it into an invisible manual process.
- Name the accountable owner for the instruction set and tool boundary and the person who approves material changes.
- Define which change events must trigger testing, access review, communications, or rollback preparation.
- Keep a representative case pack with expected evidence and a documented reason for each outcome.
- Review correction and override data with operations, security, and the people who own the underlying records.
- Prioritize failures by user impact and reversibility before optimizing speed, appearance, or model cost.
- Close each recurring issue with a specific test, control, owner, and a date to check the effect.
Key Takeaways
- Prompt engineering improves task clarity; it does not replace authorization or input validation.
- Keep policy, privileged instructions, and untrusted content visibly separate.
- Constrain machine-consumed results with schemas and server-side checks.
- Test indirect prompt injection and unsafe tool requests as normal release cases.
- Version prompts and connect observed failures to explicit changes in tests or controls.
Frequently Asked Questions
What is the first thing to build for prompt engineering? Start with a narrow, high-frequency task with a known owner, accessible evidence, and a safe fallback. How much human review is required? Match review to impact, uncertainty, reversibility, and legal obligations; low-risk drafts differ from irreversible record changes. What should be logged? Keep the minimum information needed to reconstruct inputs, policy and validation decisions, versioned configuration, and outcome, with retention and access controls. When should a team expand prompt engineering? Only after the evaluation set, live exception data, and user feedback show the existing boundary is stable enough to support more work.
Conclusion: Review The Whole Application
A secure prompt-enabled product makes the model helpful without making it authoritative. Define the work, isolate untrusted context, validate outputs, and place consequential actions behind identity and policy checks. That gives teams a feature they can improve deliberately rather than an instruction set they hope will behave.