Benchmark
Which safety classifier actually catches it?
Safety classifiers from Meta, Google and the open-weights community, run head to head on labelled safety cases through the in/guard/outharness. Scored on leaks - unsafe content let through - against over-blocks, the benign content they flagged anyway.
Content safety, head to head
The two models run on an identical set: 17 cases, 34 checks, 18 unsafe and 16 benign, each model classifying both the user prompt and the assistant response.
| Model | Params | Blocked | Leaked | Over-blocked | Correct | Latency |
|---|---|---|---|---|---|---|
| WildGuardQwen3-4B community build | 4B | 17 | 1 | 0 | 33 / 34 | ~1.0 sself-hosted |
| Meta Llama Guard 4 | 12B | 17 | 1 | 2 | 31 / 34 | ~0.34 sOpenRouter |
Both leak exactly one of eighteen unsafe checks. They separate on the other half: Llama Guard 4 flags two benign cases WildGuard passes, at three times the parameters.
One identification caveat: the WildGuard scored here is a community Qwen3-4B build, not the original release, so treat its numbers as evidence about that build rather than about WildGuard as published. Every other model is the vendor's own weights.
A 27B shield scores zero on injection
Six multilingual prompt-injection cases - a system-prompt extraction wrapped in German, Spanish, Hindi, Chinese, Russian and English - put through every model with category routing switched off, to ask whether the content shields would have caught it anyway.
| Model | Built for | Caught | Rate |
|---|---|---|---|
| Meta Llama Prompt Guard 2 | injection detector · 86M | 6 / 12 | 50.0% |
| WildGuard | content shield · Qwen3-4B build | 6 / 12 | 50.0% |
| Meta Llama Guard 4 | content shield · 12B | 4 / 12 | 33.3% |
| Meta Llama Guard 3 | content shield · 8B | 0 / 12 | 0.0% |
| Google ShieldGemma | content shield · 27B | 0 / 12 | 0.0% |
ShieldGemma-27B and Llama Guard 3 caught nothing - not one of twelve checks, in any of the six languages. That is not a defect in those models. They are hazard-taxonomy classifiers asking whether text is violent, hateful or criminal, and a polite request in German to recall how a conversation began is none of those.
It is why the harness routes by category: injection cases go only to the dedicated detector, content shields run only the rest. Scoring either on the other's job produces a leaderboard that means nothing.
The uncomfortable half
The dedicated detector reaches 50%. Multilingual injection is unsolved by every model here - which is why in/guard/out does not rest on a classifier alone.
Scope: every check in that run is labelled unsafe, so it measures recall only. There is no false-positive number in it.
How these were measured
The harness is resumable: every verdict is appended as it lands, so an interrupted run continues where it stopped. Errors are deliberately not checkpointed, so a re-run retries them rather than skipping them - a model that failed to answer never counts as a model that answered safely.
Figures above are archived partial runs from June and August 2026, recorded as each model joined the harness. Each states its own scope.
Why this matters to you
In Guard Out puts a shield model in front of content safety, and which one is your choice - configurable per key. This is the evidence for making it on numbers rather than on brand.
On these cases the 4B WildGuard build is the one to beat. Against Meta's Llama Guard 4 it catches the same unsafe content, over-blocks nothing where Llama Guard 4 over-blocks two benign cases, and does it at a third of the parameters. Bigger is not safer.
Every model is public and linked, so the exact build under test is identifiable.
How to read the latencies
Median time per check, not mean - a single cold start otherwise drags an average well above anything a warm service does. Where a model is available from a hosted inference provider the measured figure is that provider's; where it is not, the figure is a self-hosted deployment of the same weights. Each row says which.
For the one model available both ways, hosting was five times faster and returned identical verdicts on all 44 shared checks - so serving choice moves latency without moving an answer. Treat a two-times gap as real and a ten-percent gap as noise.