Red Teaming
Deliberately trying to break it before release to find the weak spots
- Red teaming is a deliberate attempt to get past the defenses, done before something ships. You look with the eyes of whoever is trying to break in, not the eyes of whoever built it.
- If guardrails are the fence put up in advance, red teaming is actually leaning on that fence to see if it holds. The two work as a pair.
- The people who built something naturally see it through the shape of their own design. That's why it takes someone else to find the gaps they missed.
- Scope and rules get agreed on upfront — what's fair game and how far it can go. It's a test, not an attack.
- The output isn't a trophy case of what got broken — it's a list of what needs fixing. Fix it, test it again, repeat.
Contents
1The analogy
Not long after moving into a new place, you step outside one evening and walk a lap around the house — not with a resident's eyes, but with the eyes of someone who might be sizing it up.
Things you'd normally never notice start standing out. The latch on the back door is loose. The lock on the window by the flower bed is old. Along a low stretch of the wall, an outdoor unit sits right where someone could use it as a step. The spare key under the flowerpot is the very first place anyone would think to look.
You check things over, but you don't break anything. Note what stands out, then fix each item one at a time. A few days later, walk the lap again. This lap around the house is exactly the kind of check that happens before AI ships.
2In detail
The defender's eyes and the breaker's eyes see differently
The people who built something check it by following the flow they designed — press this button and this happens, send this request and it gets blocked like this. But the trouble spots are usually outside that flow. Nobody expected anyone to use it that way, and that unexpected path becomes the opening.
The breaker's eye ignores the intended order. What happens if you jump from here to there? What happens if you chain two features together in a strange sequence? What happens at the tail end of a very long conversation? Same system, completely different view.
That's why red teaming is kept separate from the checks the builders run on themselves. Builders take part too, but mixing in people who don't know the design, with different backgrounds, is what surfaces different gaps.
Scope gets set before anything starts
Before starting, there's a list of things to agree on: which systems are in scope, whether real user data can be touched, where to stop, and who gets told what's found. It's the same idea as walking around the house without actually breaking the door down.
Logging gets worked out too. If nobody writes down what was tried and under what conditions it worked, the people fixing it can't reproduce it. Whether something worked once by luck or reliably works every time only becomes clear after repeated attempts.
Without this agreement upfront, testing turns into an incident. Writing down the scope isn't procedural formality — it's a safety mechanism.
What gets tested
The target isn't security alone. Testing covers whether answers fall outside the rules, whether user or company information leaks into an answer, whether the system states unverified things with confidence, whether it follows a hidden instruction buried in material it read, whether results differ across groups, and whether tools it's been given get misused.
Different spots need different people. Catching off tone in language needs someone who knows the language and culture; catching tool misuse needs someone who knows that system. No single team can see everything, so testing mixes people from different backgrounds.
Manual testing and running large volumes of automated attempts get used together. The automated side covers ground broadly; the manual side finds angles automation wouldn't think to try.
The output is a list of fixes
When testing wraps up, what comes out is a collection of cases that got through. What matters here isn't the count — it's the variety. Ten different kinds of gap matter more than a hundred near-identical cases; the former is the broader test.
Each case gets handed to whoever fixes that layer. Some become a filtering rule, some become training material, some lead to a decision to cut a feature entirely. A fixed case goes back on the list to re-check next round, because a spot that was closed once tends to reopen while something else nearby is being fixed.
It never ends in one pass
Swap in a new model, add a new feature, change the reference material — and the conclusions from the last round of testing need re-checking. It's the same as: fixing up a house once doesn't mean you skip the walk-around next year.
And no matter how many laps you walk, you can't check everything. The goal of testing isn't to bring the gaps to zero — it's to know roughly how much risk remains before shipping. Shipping without knowing and shipping while knowing are very different states.
3More precisely
The term "red teaming" comes from security and military exercises, where a defending side is called blue team and an attacking side is called red team. It resembles penetration testing in software security, but with real differences.
The biggest difference is that the target here is language and judgment, not a hole in code. The line between success and failure is blurry, so different people call the same case differently, and the same attempt doesn't always produce the same result. So testing doesn't conclude from one success — it looks at the rate at which something works across repeated attempts. Writing down the judging criteria ahead of time is part of the test too.
The analogy breaks down in places. A house has a fixed number of doors and windows, so with enough effort you can check all of them — but the paths language can come in through are uncountable, so claiming to have checked everything doesn't even make sense. A house stays fixed once repaired, while a model can have a previously closed gap reopen while some other part gets patched. And checking a house can be done alone, while AI testing needs several people with different backgrounds to surface different gaps.
4Try it yourself
5Common misconceptions
It's easy to think red teaming is hacking, but actually it's testing done under an agreed-upon scope and rules, and handing what's found to whoever fixes it is part of the same package.
It's easy to think one big round is enough, but actually a closed gap can reopen every time the model or a feature changes, so it has to repeat.
It's easy to think finding more broken cases means a better test, but actually what matters more is widening the coverage — noticing entire categories that got missed.
7One-line summary
In shortRed teaming is walking a lap before release with the eyes of whoever is trying to break in, aiming to ship knowing how much risk remains rather than shipping blind to it.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02