Hallucination Detection
in/guard/out vs Vectara HHEM
HHEM is the open-weights scoring model behind Vectara’s hallucination leaderboard – compact enough to run on a CPU, scoring whether generated text is consistent with its source. As a groundedness signal for RAG, it’s excellent. As production protection it’s one instrument: a score between 0 and 1, on one class of failure, with everything else – thresholds, actions, repair, agents – left to you.
Side by side
| Capability | Vectara HHEM | in/guard/out |
|---|---|---|
| What it is | A factual-consistency scoring model | A guardrails proxy with enforcement authority |
| Failure classes covered | Source-consistency of generated text | Numeric drift, botched arithmetic, unsupported claims, format, PII, injection, agent actions |
| The numeric class | Scored along with everything else | Recomputed deterministically – a changed figure is caught exactly, at zero cost |
| On detection | A consistency score | Repair, block, or flag per policy – with the evidence recorded |
| Where it runs | You embed and serve it in your pipeline | On the wire: one base-URL change in front of any provider |
| Agent traffic | — | Tool policy, action grounding, taint, loop detection, per-run budgets |
| Multi-step tasks | — | The run graph: every request and tool call in one audited session |
| Policy & modes | Threshold you choose in code | PREVENT and FIX per key, user, or request – observe first, enforce when proven |
Based on Vectara HHEM’s publicly available information. We’ve done our best to keep it accurate, but details may have changed since we last looked. If you’ve noticed an error, please email us.
A score is an input. Shipping is a decision.
The pattern across this lane holds: detector models – HHEM, Lynx, Luna – have made hallucination scoring fast and cheap, and in/guard/out’s judge tiers can happily use models of this class as escalation behind the free deterministic checks. What none of them provide is the system around the score: enforcement with repair, PII round-tripping, format guarantees, and the entire agent problem – budgets, grounded actions, tool policy – where the risk isn’t a sentence but a side effect.
Frequently asked questions
We already gate RAG answers on HHEM scores – what would change?
Keep the signal; upgrade the verdict. On the wire, a low-consistency answer doesn’t just drop a metric – it gets repaired where safe, blocked where not, and logged with per-claim evidence. And the failures HHEM doesn’t see – drifted numbers with no source pair, malformed JSON, an ungrounded payment – are covered by the rest of the pipeline.
Is HHEM’s consistency check the same as your grounding stage?
They overlap on intent, differ on method. HHEM scores semantic consistency with a model; in/guard/out’s first grounding tier is deterministic – it recomputes arithmetic and matches figures to sources exactly, which a similarity model can miss – and the claim-audit tier then handles the semantic class with cited evidence.
Does in/guard/out work with Vectara’s RAG platform?
in/guard/out fronts any OpenAI-compatible endpoint, so RAG answers generated behind it pass through the same output pipeline as everything else – whatever retrieval stack produced them.
See it on your traffic first.
FIX mode observes without blocking, so you can measure what in/guard/out catches before enforcing anything. We are running a limited demo - sign up and we will get you in as soon as we can.