An AI red team scope is the contract that turns adversarial curiosity into decision-grade assurance. It defines which harms matter, which system boundaries may be crossed, what access testers receive, how risky actions are controlled, what constitutes evidence, and when a repaired issue is actually closed. Without that contract, a team can produce an entertaining collection of jailbreak screenshots while missing retrieval authorization, unsafe tool use, training-data exposure, tenant boundaries, monitoring gaps, and operational recovery.
The buyer should begin with business consequences, not a list of prompts. A customer-support assistant, code agent, underwriting copilot, and autonomous purchasing agent expose different assets and authorities. The surrounding architecture described in Edilec's guide to how AI agents work matters as much as the model: prompts, retrieval, memory, identity, tools, approval gates, logging, and human operators all belong in the testable system.
Define the AI red team scope through outcomes
Write one outcome statement for each material risk. Examples include preventing one customer from obtaining another customer's records, preventing unapproved payments, resisting extraction of protected system instructions, detecting persistent knowledge-base manipulation, and ensuring a harmful action can be reconstructed. The NIST Generative AI Profile treats risk management as a lifecycle activity and emphasizes measurement, monitoring, incident disclosure, third-party risk, and pre-deployment testing. Use those concerns to identify outcomes, then translate each outcome into an observable claim.
| Scope decision | Weak wording | Decision-useful wording | Acceptance evidence |
|---|---|---|---|
| Confidentiality | Test data leakage | Attempt cross-user and cross-tenant retrieval through prompts, caches, memory and tools | Reproducible request chain, authorization context and confirmed boundary behavior |
| Agency | Test unsafe actions | Attempt high-impact tool calls without the required user authority and approval | Denied calls, policy decision records and no downstream side effect |
| Integrity | Test poisoning | Introduce controlled malicious content through each approved ingestion route | Provenance alert, quarantine record and unaffected production index |
| Availability | Test denial of service | Exercise bounded token, tool-loop and concurrency exhaustion scenarios | Rate-control response, graceful degradation and recovery measurement |
| Accountability | Review logs | Reconstruct a selected harmful path from user request to external effect | Correlated identities, versions, retrieved sources, tool arguments and outcome |
Choose an engagement model and access level
Black-box testing approximates an external user but reveals little about control coverage when it fails to find a flaw. Gray-box testing supplies roles, architecture, data classifications, tool schemas, known guardrails and test accounts, making it the usual enterprise default. White-box work adds source, prompts, policy code, evaluation sets and observability access. Use multiple perspectives when threat actors differ. An anonymous-abuse track can run beside an authenticated-insider track and a compromised-operator track without pretending they have equal knowledge.
Document model and application versions, region, system prompts, retrieval collections, memory configuration, tools, plugins, safety layers, identity provider, feature flags and fallback paths. Record whether testers may create accounts, upload documents, modify a test corpus, call APIs directly, inspect traces, automate prompts, use external models, or attempt social engineering. Access should be time-bound and attributable. Production testing needs stricter rate, data and side-effect controls than an isolated replica, but a replica must preserve the production control plane closely enough to support conclusions.
Turn the threat model into test surfaces
NIST's adversarial machine learning taxonomy distinguishes attack stages, attacker goals, capabilities and knowledge. That vocabulary prevents a scope from collapsing every problem into prompt injection. Map abuse against model inference, model access, training or fine-tuning data, retrieval ingestion, ranking, memory, tool execution, output handling, identity, monitoring and incident response. For each surface, state the attacker's starting access and the security property under test.
Use MITRE ATLAS to ground scenarios in adversary tactics and techniques, then adapt them to the deployed architecture. Use the OWASP GenAI Security Project to check application-level risks and agentic failure patterns. Neither source is a ready-made test plan. A technique only enters scope when there is a plausible path, a valued asset, and an observable result. This keeps testing deep enough to find compound failures rather than broad enough only to tick taxonomy labels.
Write rules of engagement for AI-specific hazards
Rules of engagement should name prohibited targets, approved hours, source addresses, rate ceilings, maximum cost, stop conditions, emergency contacts and data-handling requirements. Add AI-specific boundaries: whether real personal data may enter prompts, whether generated malware may be executed, which tools can create irreversible effects, whether persistent memory may be altered, whether poisoning survives beyond the exercise, and how copyrighted or regulated content is handled. Require an immediate stop for observed customer impact, uncontrolled propagation, unexpected external communication, credential exposure, or cost runaway.
Use synthetic records with canary values whenever the claim permits it. When realistic sensitive data is essential, minimize fields, restrict tester access, encrypt exports, define deletion evidence and prohibit reuse in commercial model training. Separate the person who authorizes dangerous actions from the tester attempting them. The same least-authority principles explained in Edilec's agent tool permissions guide should govern test credentials.
Specify reproducible evidence and severity
A finding needs more than a successful output. Require timestamp, environment, account and role, model and prompt versions, conversation state, retrieved document identifiers, memory state, tool calls, policy decisions, response, downstream effects, repetitions, and the tester's confidence. Preserve raw evidence in a controlled annex while the report uses redacted excerpts. Where nondeterminism matters, state the number of attempts and observed outcomes without presenting a small convenience sample as a population statistic.
| Evidence field | Why it changes the decision | Minimum report treatment |
|---|---|---|
| Exploit preconditions | Separates public exposure from privileged misuse | Accounts, access, timing and required knowledge |
| Business effect | Distinguishes odd text from a material security outcome | Affected asset, authority, user and downstream record |
| Repeatability | Shows whether mitigation and monitoring can be tested | Attempt count, successful paths and relevant variance |
| Control response | Reveals prevention, detection and recovery performance | Policy event, alert, operator action and elapsed sequence |
| Version context | Prevents a later retest from comparing different systems unknowingly | Model, prompt, index, tool and application release identifiers |
| Residual exposure | Supports acceptance when elimination is impractical | Remaining path, compensating controls, owner and review date |
Severity should combine impact, reachability, required privilege, repeatability, detectability, persistence and blast radius. Avoid scoring a toxic sentence and an unauthorized transaction with the same generic rubric. Define who may lower severity and what evidence they must provide. A safety refusal bypass may be serious in one product and irrelevant in another; a low-frequency authorization bypass remains critical when it can move money. The rubric belongs in the scope before findings exist.
Make the report usable by control owners
Deliver an executive risk view, a system-boundary diagram, coverage matrix, finding register, evidence annex, detection observations, and limitations. Every finding should identify the failed claim, affected component, business consequence, reproduction path, likely cause, recommended control layer, owner, and retest method. Link model behavior findings to the broader evaluation program in Edilec's model evaluation engineering notes, so repaired cases become durable regression tests rather than forgotten screenshots.
Recommendations should favor layered changes: narrow tool authority, enforce authorization outside the model, improve source provenance, isolate tenants, validate outputs before execution, add detection, and provide a safe fallback. Prompt changes can help, but they are rarely sufficient where a deterministic control can enforce a boundary. Ask the red team to distinguish a tactical patch from a systemic correction and to state which untested variants could survive the proposed fix.
Contract the retest and acceptance decision
A retest should use the original evidence package, the claimed fix, adjacent variants, and a regression set. Freeze or record relevant versions, because a model swap, index rebuild or policy release can change behavior independently of the remediation. The tester should classify each item as fixed, partially fixed, not fixed, not reproducible, or superseded, with reasons. Acceptance belongs to the accountable risk owner, not the vendor or tester, and should record compensating controls and expiry when residual risk remains.
Procurement language should reserve a retest window, define included effort, require urgent disclosure during the engagement, address intellectual property and data deletion, and state who can share the report. Compare proposals on qualified hours, surfaces, access assumptions, evidence standard, remediation support and retesting, not prompt count. A low bid that excludes tools, retrieval and identity may be testing the least consequential part of the product.
Key takeaways
- Scope business harms and security claims before selecting adversarial techniques.
- Test the model, application, retrieval, memory, identity, tools, monitoring and response as one system.
- Use rules of engagement that control personal data, persistent state, irreversible actions, cost and external effects.
- Require versioned, reproducible evidence and a severity rubric tied to business impact.
- Close findings only through a defined retest and an accountable risk-acceptance decision.
Frequently asked questions
Is AI red teaming the same as penetration testing?
No. They overlap in adversarial method, but an AI engagement also evaluates probabilistic behavior, misuse, content-mediated attacks, model and data attacks, retrieval integrity, unsafe agency, and monitoring. Conventional infrastructure and application testing should still cover the underlying APIs, cloud resources and identity controls.
Should testing happen in production?
Only when the claim cannot be established credibly elsewhere and production safeguards are explicit. Begin in a representative isolated environment, then use narrowly bounded production validation for controls that depend on live identity, routing, data, monitoring or provider behavior.
How often should an AI system be red teamed?
Use change and risk triggers rather than a calendar alone. Material model, prompt, retrieval, tool, permission, data-source or deployment changes can justify focused testing. A periodic full engagement remains useful for accumulated changes and new threat knowledge.
Conclusion
A credible AI red team engagement is an evidence-producing risk decision, not a jailbreak tournament. Its scope connects valued outcomes to realistic access, system-wide test surfaces, controlled execution, reproducible findings and retest acceptance. When those terms are explicit, buyers can compare providers, engineers can remediate root causes, and risk owners can understand what remains.
Start with five material claims and trace each through the full AI workflow. If the team cannot identify the owner, enforcement point, evidence source and acceptance threshold for a claim, that gap is already useful. Resolve it before the engagement begins; the resulting scope will make every testing hour more consequential.