AI Red Team & Adversarial Testing

About this program

Risks addressed

• Critical Jailbreak bypasses safety guardrails in production
• Critical Prompt injection exfiltrates customer context
• Critical Tool-using agent tricked into invoking destructive action
• High Model hallucinates facts in customer-facing output

Controls (8)

1. Red-team scope + rules of engagement

   High

   This control ensures that red-team exercises are conducted within explicitly defined boundaries and under documented rules of engagement (RoE) that specify in-scope systems, prohibited actions, attack vectors, timing constraints, communication protocols, and escalation procedures. The scope prevents unintended disruption to…

   How to test + evidence

   Testing procedure: Written ROE: targets, allowed techniques, kill switch.

   Evidence to collect: ROE document.

2. Pre-launch adversarial testing

   Critical

   Pre-launch adversarial testing subjects AI systems, applications, or products to simulated adversarial attacks before production deployment to identify vulnerabilities, failure modes, and exploitable weaknesses. This control requires documented adversarial testing protocols that include red-team exercises, fuzzing, model inversion attempts, prompt…

   How to test + evidence

   Testing procedure: Every model / prompt change goes through an adversarial suite before deployment.

   Evidence to collect: Test report + sign-off.

3. Prompt-injection corpus tested

   High

   This control verifies that AI language models and conversational agents are systematically tested against a curated corpus of prompt-injection attack patterns before deployment and on a recurring basis. The testing corpus includes techniques such as role-switching, instruction override, delimiter confusion,…

   How to test + evidence

   Testing procedure: Maintained corpus of injection payloads run automatically; pass criteria documented.

   Evidence to collect: Corpus + last run.

4. Jailbreak resistance evaluations

   High

   Jailbreak resistance evaluations systematically test AI systems—particularly large language models deployed in production—against adversarial prompt techniques designed to bypass safety guardrails, policy restrictions, or intended behavioral boundaries. These evaluations employ curated libraries of known jailbreak patterns (role-playing exploits, delimiter injection,…

   How to test + evidence

   Testing procedure: Top jailbreak templates (DAN-style, indirect, multi-turn) tested per release.

   Evidence to collect: Eval report.

5. Tool / agent safety tests

   Critical

   Tool and agent safety tests validate that endpoint security agents, monitoring tools, and automated security software function correctly under adverse conditions without causing system instability, performance degradation, or operational disruption. This control requires structured pre-deployment and periodic testing of security…

   How to test + evidence

   Testing procedure: Agents tested for misuse: destructive tool invocation, escalation, data egress paths.

   Evidence to collect: Test results + sandbox config.

6. Grounding + factuality evaluations

   High

   Grounding and factuality evaluations measure whether generative AI systems produce outputs anchored to verifiable source material and free from hallucinations or fabricated information. These evaluations use benchmark datasets, automated scoring against known facts, and human review to assess model adherence…

   How to test + evidence

   Testing procedure: RAG / factuality evals run on a representative test set; trend tracked.

   Evidence to collect: Eval scores over time.

7. Bug-bounty / responsible disclosure for AI

   Medium

   A bug bounty or responsible disclosure program for AI systems provides external researchers, users, and ethical hackers with a structured, legal channel to report vulnerabilities, adversarial prompts, model manipulation techniques, data leakage issues, or unsafe outputs in deployed AI models…

   How to test + evidence

   Testing procedure: External researchers have a clear path to report AI-specific issues.

   Evidence to collect: security.txt + intake.

8. Post-incident review feeds the red-team backlog

   Medium

   This control ensures that findings from post-incident reviews—including attack vectors, detection gaps, and response weaknesses—are systematically translated into offensive security testing scenarios and added to the red team's backlog. Post-incident analysis provides real-world attacker tactics and organizational blind spots that…

   How to test + evidence

   Testing procedure: Real incidents become new red-team test cases.

   Evidence to collect: Test-case provenance.