The frontier red-team playbook, three years on

In July 2023, Anthropic published a short post admitting something most vendors would bury: with six months and 150+ hours of expert effort, its own models could sometimes produce expert-level knowledge useful for biological misuse. At the time it read as a policy gesture, timed a few days after the White House voluntary commitments. Three years on, it looks like something else. It's the document where the industry's default red-teaming procedure got written down, and most teams borrowing the idea today still skip the parts that made it work.

What the exercise actually was

Strip away the biosecurity framing and the method has a clear shape. Anthropic started from a threat model (a bad actor seeking biological weapons capability), recruited domain experts rather than generalist prompt-hackers, and taught those experts to jailbreak the models. Testing ran through a bespoke interface with the trust-and-safety monitoring switched off, so the experts probed the raw model rather than the production stack. Findings were converted into quantitative evaluations that could be re-run against future models. Mitigations came in two layers: changes to the training process, and classifier-based filters in deployment.

Two details in that list did the heavy lifting. Testing without the production safety tooling meant Anthropic learned what the model itself would do, not what the filters caught. And converting expert findings into repeatable evals turned a one-off engagement into a regression suite. The post's own framing was uplift: does the model accelerate a bad actor relative to what they'd get from internet access alone.

The company estimated 100+ hours per domain to do this properly. That number matters later.

From blog post to compliance infrastructure

The procedure didn't stay a blog post. That fall, Anthropic formalized it into its Responsible Scaling Policy, which ties red-team results to capability thresholds with deployment consequences. The chain reached its logical end in May 2025, when Anthropic shipped Claude Opus 4 under ASL-3 protections (constitutional classifiers, egress bandwidth controls, tighter access to weights) as a precaution, because it couldn't rule out that the model had crossed the CBRN capability line its own evals defined.

The rest of the field converged on the same shape. OpenAI's Preparedness Framework runs on capability thresholds and expert testing. The Frontier Model Forum published its red-teaming definitions in October 2023 leaning on the same methodology. The EU AI Act's obligations for general-purpose models with systemic risk, applicable since August 2025, make adversarial testing a legal requirement rather than a lab norm. A 2023 disclosure exercise became, in about two years, the compliance backbone of frontier AI.

The parts you should actually copy

My read for working developers: the structure transfers, the scale doesn't, and the industry keeps confusing the two.

If you're building on top of a model API, three moves from the 2023 playbook are directly reusable. First, write the threat model before you write a single adversarial prompt. "Jailbreaks" is not a threat model. "A logged-in user exfiltrates another tenant's data through our RAG tool calls" is, and it dictates entirely different tests than CBRN uplift does.

Second, red-team below your guardrails. Most app teams only ever test the full stack, system prompt and moderation filters included, which tells you nothing about how much work those filters are doing. Hit the raw model with your attack set too. The gap between the two runs is your actual safety margin, and it's what silently moves when you swap model versions.

Third, treat red teaming as eval generation. Every successful attack becomes a test case that runs in CI, especially on model upgrades. A new model version routinely reopens holes the old one had closed. Anthropic re-runs its frontier evals on every model for exactly this reason, and there's no excuse for app teams to do less when the tooling is this cheap.

That tooling is the big change since 2023. What cost Anthropic expert-hours is now partly automated: NVIDIA's garak scans models against 100+ known probe classes, Microsoft's PyRIT orchestrates an attacker LLM against your target to find novel breaks, and promptfoo wires adversarial test suites into CI where the regression-eval discipline belongs. A reasonable starting stack is garak for known attacks, promptfoo for the CI loop, PyRIT once you've exhausted the catalog.

What doesn't scale down

The caveat: automation covers known attack classes. The expensive part of the 2023 exercise, real domain experts spending 100+ hours finding failure modes nobody had catalogued, has no cheap substitute. That's fine, because most teams shouldn't be attempting it. If you're shipping a support bot or a coding agent, your threat model is prompt injection, tool-call abuse, and data leakage, not bioweapons uplift. Frontier CBRN testing is the labs' job, and after the EU AI Act, their legal problem.

The failure mode I keep seeing is teams cosplaying the frontier version: an expensive one-off "AI red team" engagement, a PDF of findings, no evals, no re-testing on the next model swap. That inverts the lesson. Everyone already suspected in 2023 that models could be coaxed into saying dangerous things; what Anthropic demonstrated is that adversarial testing only compounds when its output is a threat model plus a regression suite, re-run forever. Copy that, at whatever scale your product warrants, and skip the theater.