Benchmarks
Measured against the best models available
In Guard Out is tested against purpose-built models from IBM and Meta on the same labelled cases. On agent actions it pairs exact checks with a second opinion on their refusals, and decides 29 of 32 correctly while blocking one legitimate action in sixteen.
How we score
Two comparisons, each reporting both halves of the result: what a check catches, and how much legitimate traffic it blocks along the way. The second number is the one that decides whether anyone leaves a guardrail switched on.
Each benchmark scores one check in isolation, which is the hard way to be measured and the honest one. In a real run they compose: the agent benchmark shows two attack families stopped by two different parts of In Guard Out, with no overlap - and a single-purpose model missing four of the cases the second part catches.
And where detection runs out - the forged invoice, the shifted decimal, the two attacks that defeat every detector because the attacker wrote the evidence - the approval layer catches it: a payment above a ceiling or to an unapproved account is held for a person, from a template you edit rather than a policy you write.
Action grounding
In Guard Out decides 29 of 32 and blocks one legitimate action in sixteen
Thirty-two labelled agent actions – fabricated payees, poisoned invoices, legitimate payments – put to In Guard Out, to purpose-built models and to two controls. Ranked by the measure that cannot be gamed: how many of all 32 each one got right.
- In Guard Out29 / 32
- substring control24 / 32
- IBM Granite Guardian 4.120 / 31
- MiniCheck-Flan-T5-Large17 / 32
- approve everything16 / 32
Catching bad actions is the easy half. In Guard Out blocks one of the 16 legitimate actions; the model with the highest catch rate blocks 14.
Safety classifiers
You pick the shield on measured results, not on brand
Safety classifiers from Meta, Google and the open-weights community on the same labelled cases. In Guard Out puts a shield model in front of content safety and lets you choose which one, per key - so the choice is worth making on numbers.
- WildGuard - best measured33 / 34
- Meta Llama Guard 431 / 34
Catches the same unsafe content as Meta Llama Guard 4 and over-blocks nothing where Llama Guard 4 over-blocks two benign cases - at a third of the parameters. Bigger is not safer.
How these numbers were produced
Every case is hand-written and labelled with a stated reason: what it is meant to distinguish, and why the correct answer is the correct answer. Nothing is scraped, generated, sampled to a favourable mix, or weighted to flatter a result - every figure counts every case.
Competing models are run unmodified, on their own vendors’ weights or endpoints, asked the question each was built to answer, and verified to separate a good case from a bad one before being scored at all. Every model links to its own page so the exact build is identifiable.