Red-Teaming: Attacking Your Own System Before Someone Else Does

Ordinary software testing checks whether the system does what it should. Red-teaming checks whether it can be made to do what it should not, and the difference changes the method entirely.

Three properties make it awkward.

The input space is unbounded. There is no equivalent of enumerating boundary conditions, because the space is natural language and an attacker can rephrase indefinitely.

Success is not binary. A model that refuses nine times and complies once has a vulnerability, so a pass rate is the wrong summary and the interesting information is in the tail.

And the system is non-deterministic. The same prompt can succeed and fail across runs, which means a single trial establishes almost nothing and every finding needs repetition to characterise.

The consequence is that red-teaming is closer to security research than to QA. You are looking for existence proofs, not coverage, and a single reproducible bypass is a result regardless of how many attempts failed around it.

The honest goal is not proving the system safe, which is not achievable. It is finding the failures that exist, fixing what is fixable, and knowing what remains.

Automated methods get the attention. Manual testing by people who understand the domain finds the things that matter most, and the reason is instructive.

An automated attack optimises against a proxy for success, usually whether the model refused. A human knows what would actually harm this organisation, which is a different and far more specific question. The most valuable findings tend to be application-specific: not can I make it say something offensive, but can I make the support assistant reveal another customer's order, or make the internal tool disclose a salary band.

Composition matters more than headcount. A useful team includes someone who knows the domain deeply enough to recognise a consequential disclosure, someone who understands the system's architecture and therefore where the seams are, someone with a security mindset, and ideally someone entirely unfamiliar with the product, because they will try things the builders have unconsciously ruled out.

The common failure is a red team composed entirely of the people who built it. They test the paths they designed, avoid the ones they know are weak because those feel like cheating, and share a mental model of what the system is for. That last one is the real problem: an attacker has no such model, and neither should your testers.

Four families from the research literature, useful both as tools and as a description of what a capable adversary has available.

Gradient-based optimisation. Greedy Coordinate Gradient, from Zou and colleagues in 2023, optimises an adversarial suffix by following gradients toward a target completion. It requires white-box access to weights, so it applies directly only to open models. The finding that matters for everyone: the suffixes it discovers often transfer to closed models the attacker never touched, so open-weight availability provides an attack path against systems built on closed ones.

Iterative black-box refinement. PAIR, from Chao and colleagues, uses one language model as an attacker that repeatedly reformulates a prompt against a target and refines based on the response. No weight access needed, which makes it applicable to any system with an API.

Many-shot jailbreaking, from Anthropic in 2024, fills a long context with a fabricated dialogue of the assistant complying with harmful requests, then asks the real question. Effectiveness rises with the number of examples.

And encoding and obfuscation: base64, ciphers, low-resource languages, token splitting, homoglyphs. Unsophisticated and persistently effective, because filters match surface forms while the model reads through them.

For a defender the practical use is transfer. Attacks published against other systems are the cheapest test set available.

A structure that produces findings rather than reassurance.

Scope first, from the threat model built in the first lesson. Name the specific harms that matter for this application, because open-ended attack the chatbot produces anecdotes rather than results.

Then run two tracks in parallel. Manual testing goes after the application-specific harms. Automated testing runs volume: published attack corpora, encoding variants, many-shot constructions, and iterative refinement against the targets.

Everything that succeeds goes to triage, where the question is not was this blocked but what could an attacker actually achieve. Reproduce it several times, because non-determinism means one success may be noise.

Confirmed findings split two ways. Some are fixable by a rail or an architectural change. Some are not fixable and become accepted risk, which is a legitimate outcome provided someone named accepts it.

Every confirmed finding, fixed or not, becomes a regression test. That is what makes the exercise compound rather than expire.

The difference between a useful exercise and a demonstration is almost entirely in the scoping.

Name the harms. Not test for jailbreaks but a specific list: disclosure of another customer's data, disclosure of internal pricing, advice that would create liability in this jurisdiction, an action taken against an account the user does not own, exfiltration through any available channel.

Derive them from the threat model rather than from a generic list, because the harms that matter are properties of your application, not of language models in general.

Decide what counts as success in advance, and make it about consequence. A model that outputs a mildly rude sentence has not produced a security finding. A model that reveals one field of another user's record has, even though the text looks innocuous.

Include the indirect channel explicitly. Most teams test what a user can type and never test what a poisoned document can do, which leaves the more serious attack surface from the first lesson entirely unexamined. Plant instructions in a document, get it into the retrieval corpus, and see what happens.

And set rules of engagement: which environment, what data, who is informed, and what to do if something genuinely sensitive is uncovered. Red-teaming a production system with real customer data needs the same care as any other authorised testing.

Red-team metrics are easy to construct and easy to misuse, and three cautions cover most of the misuse.

Attack success rate, the proportion of attempts that achieved a named harm, is the standard headline. It is only comparable against a fixed attack set, and it is trivially improved by adding weak attacks to the denominator. Report the set alongside the number or the number means nothing.

A falling success rate over successive exercises is ambiguous. It can mean the defences improved. It can also mean the red team ran out of new ideas, which is the more common explanation as an exercise matures. Rotating people, and importing attacks published elsewhere, is what distinguishes the two.

And resist the pull toward a single score. Whether an attack succeeded matters far less than what it achieved. One reproducible path to another user's data outweighs a hundred blocked attempts at producing rude text, and any aggregate that lets the second bury the first is actively misleading.

The reporting that survives contact with an executive audience is a short list of confirmed findings ranked by consequence, each with reproduction steps, the fix or the acceptance decision, and a named owner. A dashboard showing ninety-four percent blocked communicates less and reassures more, which is exactly the wrong combination.

The highest-return output of a red-team exercise is not the report. It is the test suite, because that is the part that keeps working after everyone has moved on.

Every confirmed finding becomes a test case: the input, the channel it arrived through, and the assertion that the harm does not occur. Findings you chose to accept rather than fix belong in the suite too, marked as known, so that a change making them worse is visible.

Run the suite on every change to the model, the prompt, the rail configuration, the retrieval corpus, and the tool permissions. All five can reintroduce a fixed vulnerability, and the model version is the one people forget, because it changes without a deployment on your side.

That last point is the operational core. A provider updating a model can silently undo a mitigation you validated months ago, and without an automated suite you will discover it from an incident rather than from a test run.

Two practical notes. Keep the suite's expected outcomes about consequence rather than exact strings, since outputs vary between runs and a string match will produce constant false alarms. And run each case several times, because a non-deterministic system that passes once has not passed.

The suite is also the artefact that makes the next exercise cheaper, since the red team can start from what is already covered rather than rediscovering it.

Some findings have no clean fix, and how an organisation handles those says more about its maturity than its block rate.

The realistic options, in descending order of preference.

Remove the capability. If an agent can be induced to send data externally, and the sending capability is not essential, delete it. This is the only option that closes the class rather than narrowing it.

Add a human gate. Route the affected action through confirmation. Slow, and reliable in a way no filter is.

Reduce the blast radius. If the harm is disclosure of a data field, stop putting that field in context. The model cannot leak what it never held.

Detect and respond. Where prevention is not available, ensure the attempt is logged, alerted on, and investigated, accepting that you are trading prevention for detection.

Accept and document. A named person records that the risk exists, why it is tolerated, what would change the assessment, and when it will be revisited.

What is not on the list: fixing it with a stronger prompt. Adding never reveal confidential information to a system prompt is not a control, because the instruction sits in the same channel as the attack. The first lesson's point returns here in its most practical form.

A one-off exercise produces a report that ages. The value comes from making it a repeating practice at a cadence matched to change.

Before first launch, and before any material expansion of capability or audience.

On every model version change, at minimum by running the regression suite.

Periodically with rotated people, because a fixed team's findings converge and stop being informative.

And after any incident, since an incident is evidence that the threat model was incomplete rather than merely that a control failed.

Two supporting habits keep it alive between exercises. Watch published research and incident reports, since a technique demonstrated against another system is a free test case and the cheapest form of red-teaming available. And treat unusual production logs as findings: users probing a system leave traces, and those traces are real adversarial data generated at no cost.

The summary across all three lessons. The model cannot police the boundary between instruction and data, so defence sits outside it, in layers, with architectural containment doing more work than any filter. And the only way to know whether those layers hold is to attack them yourself, on a schedule, with the results written into a suite that keeps checking after the exercise ends.