Hallucination detection
in/guard/out vs Vectara HHEM
HHEM is the open-weights scoring model behind Vectara’s hallucination leaderboard – compact enough to run on a CPU, scoring whether generated text is consistent with its source. As a groundedness signal for RAG, it’s excellent. As production protection it’s one instrument: a score between 0 and 1, on one class of failure, with everything else – thresholds, actions, repair, agents – left to you.
Side by side
| Capability | Vectara HHEM | in/guard/out |
|---|---|---|
| What it is | A factual-consistency scoring model | A guardrails proxy with enforcement authority |
| Failure classes covered | Source-consistency of generated text | Numeric drift, botched arithmetic, unsupported claims, format, PII, injection, agent actions |
| The numeric class | Scored along with everything else | Recomputed deterministically – a changed figure is caught exactly, at zero cost |
| On detection | A consistency score | Repair, block, or flag per policy – with the evidence recorded |
| Where it runs | You embed and serve it in your pipeline | On the wire: one base-URL change in front of any provider |
| Agent traffic | — | Tool policy, action grounding, taint, loop detection, per-run budgets |
| Multi-step tasks | — | The run graph: every request and tool call in one audited session |
| Policy & modes | Threshold you choose in code | PREVENT and FIX per key, user, or request – observe first, enforce when proven |
A score is an input. Shipping is a decision.
The pattern across this lane holds: detector models – HHEM, Lynx, Luna – have made hallucination scoring fast and cheap, and in/guard/out’s judge tiers can happily use models of this class as escalation behind the free deterministic checks. What none of them provide is the system around the score: enforcement with repair, PII round-tripping, format guarantees, and the entire agent problem – budgets, grounded actions, tool policy – where the risk isn’t a sentence but a side effect.
Frequently asked questions
We already gate RAG answers on HHEM scores – what would change?
Keep the signal; upgrade the verdict. On the wire, a low-consistency answer doesn’t just drop a metric – it gets repaired where safe, blocked where not, and logged with per-claim evidence. And the failures HHEM doesn’t see – drifted numbers with no source pair, malformed JSON, an ungrounded payment – are covered by the rest of the pipeline.
Is HHEM’s consistency check the same as your grounding stage?
They overlap on intent, differ on method. HHEM scores semantic consistency with a model; in/guard/out’s first grounding tier is deterministic – it recomputes arithmetic and matches figures to sources exactly, which a similarity model can miss – and the claim-audit tier then handles the semantic class with cited evidence.
Does in/guard/out work with Vectara’s RAG platform?
in/guard/out fronts any OpenAI-compatible endpoint, so RAG answers generated behind it pass through the same output pipeline as everything else – whatever retrieval stack produced them.
See it on your traffic first.
FIX mode observes without blocking, so you can measure what in/guard/out catches before enforcing anything. We are running a limited demo - sign up and we will get you in as soon as we can.