AI red teaming involves five concrete stages: scoping and threat modeling, attack design, execution, reporting, and remediation verification. It is not free-form poking at a chatbot until something embarrassing happens — that is how you get an interesting screenshot and no fixed process. A real engagement starts by defining what would actually be bad for this specific system, builds attacks targeted at those specific failure modes, runs them systematically, and ends with a report that assigns severity and an owner to every finding. If your process skips straight to "try to make it say something bad," you are doing improv, not red teaming.
How it works
Scoping and threat modeling comes first. Before any attack, the team defines what a bad outcome looks like for this specific deployment: leaking a system prompt, executing an unauthorized action through a connected tool, producing content that breaches a specific policy, or being manipulated by a hidden instruction in a document it reads. A customer support bot and an autonomous coding agent have almost nothing in common as attack surfaces, so the threat model has to be built per system, not copied from a template.
Attack design follows the threat model. Each identified risk gets one or more concrete attack approaches: role-play framing to elicit a jailbreak, a document with hidden instructions to test injection resistance, a probing conversation designed to extract system prompt contents, or a sequence of innocuous-seeming requests that combine into a policy violation. Generic "try to jailbreak it" attempts find generic problems; attacks built from the specific threat model find the problems that actually matter for this deployment.
Execution mixes manual creativity with automated scale. A skilled human tester finds the novel, context-aware attacks; automated attacker models generate hundreds of variations on known attack patterns and check them at a scale no human team can match. Most real engagements run both in parallel.
Who runs it and when
| Approach |
Who |
Best for |
Limitation |
| In-house team |
Your own security or ML staff |
Continuous testing during development |
May share blind spots with the builders |
| Third-party specialist firm |
External red-teaming consultancy |
Pre-launch deep engagements, compliance evidence |
Costly for continuous, ongoing coverage |
| Public bug bounty |
The open security community |
Breadth, long-tail discovery after launch |
Uneven quality, needs strong triage |
| Automated attacker models |
Software, not people |
Regression testing, scale, continuous coverage |
Rarely finds genuinely novel attack types |
Most mature programs layer these: automated coverage running continuously, an internal team owning the threat model, and a periodic third-party or bug-bounty engagement for an outside perspective before major releases.
What the report actually needs
A red-teaming report is only useful if it produces action. Each finding needs a severity rating tied to real impact, exact reproduction steps so engineering can confirm the fix, and a named owner — otherwise findings pile up as anecdotes nobody is accountable for closing. The engagement is not complete when the attacks stop; it is complete when every finding above an agreed severity threshold has a verified fix and, ideally, a permanent regression test that runs on every future model update.
Common mistakes
- Skipping the threat model and going straight to attacks. Generic jailbreak attempts without a defined scope produce interesting anecdotes, not an actionable risk picture.
- Treating red teaming as a pre-launch checkbox. Models and attack techniques both keep changing after launch; a one-time engagement ages out within months.
- No severity framework for findings. Without a consistent way to rank findings, teams fix the most dramatic-looking issue instead of the most impactful one.
- Closing findings without a regression test. A fix that is not backed by a permanent test can silently break again on the next model or prompt update.
FAQ
How is this different from a general AI red teaming overview?
Our AI red teaming in 2026 piece covers the concept broadly: manual versus automated methods and how red teaming differs from evaluation. This piece focuses specifically on how a real engagement is structured end to end, from scoping through remediation.
Do small teams need a formal red-teaming process?
Yes, at a smaller scale. Even a lightweight threat model and a few hours of structured testing before launch catches far more than ad hoc poking, and it gives you a baseline to expand later.
How often should red teaming happen?
Continuously for automated coverage, with periodic deeper manual or third-party engagements tied to major releases or model version changes.
What is the single most common process failure?
Skipping scope and threat modeling. Without it, teams cannot tell whether a finding is a serious risk or a curiosity, and the report becomes impossible to prioritize.
Where to go next
For the practical defenses that red-teaming findings usually feed into, see prompt injection defense and AI guardrails vs content filters. For the broader concept and how it differs from evaluation, our earlier AI red teaming in 2026 piece is a useful companion read.