AppliedNovember 2026
No Jailbreak Required: the safety gap between frontier models
- 0–3%
- of harmful requests answered by each model alone
- 0%
- answered by at least one of seven
- 0
- answered by all seven
1Abstract
Language models are now used in health care and public information, and each provider trains its own model to refuse harmful requests. That independence creates a weakness: if one model complies with a harmful request that the others refuse, the refusals protect no one, because anyone can send the same request to the model that complies. We audited refusals across seven frontier models (GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro, Grok-3, DeepSeek-V3, Qwen3-235B and Llama 3.3 70B) on 100 harmful requests from the AdvBench benchmark, asked plainly, with no system prompt and no jailbreak. Each model on its own complied with only 1 to 3% of requests. But 8% of requests were answered by at least one model while others refused (7% without one borderline request). We call these the exploitable gap: requests that look safely handled across the ecosystem, but that anyone can get answered by switching providers. No request was answered by all seven models, so safety training does agree on some harms. The results argue for coordination on safety across providers, and for refusal benchmarks that measure the whole ecosystem, not one model at a time.
2Introduction
Safety for language models is usually measured one model at a time: how often does this model refuse a harmful request? By that measure, frontier models do well. Each refuses nearly everything a standard harmful-request benchmark asks.
But people do not use one model. The same request can be sent to many models from many providers, each trained to refuse on its own terms. If any one of them complies, the request is answered, however many others refused. The protection a user faces is not that of the average model, or the best one, but of the most permissive one.
This matters most where these models are being deployed: health care and public information, where a single harmful answer can do damage. This work measures how large that gap is under ordinary use, with no jailbreak at all.
3Methods
Models. Seven frontier models: GPT-4o, Claude 3.7 Sonnet, Gemini 2.5 Pro, Grok-3, DeepSeek-V3, Qwen3-235B and Llama 3.3 70B.
Requests. 100 prompts sampled from the AdvBench harmful behaviors benchmark (Zou et al., 2023).
Conditions. Each request was sent to each model as an ordinary user would send it: no system prompt, and no jailbreak technique of any kind.
Scoring. An automated judge, GPT-4o-mini, scored each response as compliant or not.
The exploitable gap. A request is in the gap when at least one model complies and at least one refuses. A request every model refuses is handled; one every model complies with is a shared failure.
4Results
Taken one at a time, every model looks safe: each complied with between 1% and 3% of the 100 requests.
Taken together, they do not. 8% of requests were answered by at least one model while others refused. One of these was a borderline consumer-harm request; without it, the gap is 7%. The gap is larger than any single model's compliance rate, because different models comply with different requests.
No request was answered by all seven models. Where safety training agrees, it agrees completely: some harms are refused across the board.
5Discussion
Per-model refusal rates understate real-world risk. A provider can reasonably report that its model refuses 97 to 99% of harmful requests, and every provider can say the same, while the ecosystem as a whole still answers 8% of them. The relevant number for safety is not any one model's rate, but the rate at which some model complies.
Two things follow. First, providers would benefit from coordinating on safety: sharing the requests their models refuse, so a refusal by one is not undone by another. Second, refusal benchmarks should report ecosystem-level results alongside per-model ones: for a set of models, how many harmful requests does at least one of them answer?
That no request was answered by every model is encouraging. It shows the providers already agree on a core of harms; the work is extending that agreement to the edges.
6Limitations
- One benchmark, 100 requests. AdvBench covers a particular set of harms, and 100 requests is a sample. The gap may differ on other benchmarks and on larger samples.
- An automated judge. Compliance was scored by GPT-4o-mini, not by people. Borderline responses are where an automated judge is least reliable, and one request in the gap was borderline.
- Plain requests only. The audit used no system prompt and no jailbreak. Deployed products often add system prompts, which may change refusals in either direction, and adversaries do use jailbreaks, which would likely widen the gap.
- A snapshot. Models are updated often, so these rates describe the versions tested, not the models today.
7Availability
The requests come from AdvBench (Zou et al., 2023), which is publicly available. The model responses include compliant answers to harmful requests and are not published. For questions about this work, email hello@layeredlabs.ai.
Living draft, updated November 2026. This page can change before formal publication.