Hallucination prevention
Catch hallucinations before they reach a user
Even frontier models are not 100% safe on hallucinations – and the cheaper the model, the higher the rate. A company shipping LLM features is one confidently-wrong answer away from a wrong number in a report, a fabricated fact to a customer, or a decision made on invented data. The clouds now sell grounding checks too – here is how the approaches differ.
Side by side
| Capability | Cloud grounding checks (Bedrock, Azure) | in/guard/out |
|---|---|---|
| Where it runs | Inside that cloud’s service – Bedrock Guardrails, Azure AI Content Safety | On the wire, in front of any provider – OpenAI, Anthropic, OpenRouter, local |
| First-pass method | An ML model scores groundedness – every check is a model call | Deterministic first: recompute arithmetic, match figures to source – zero extra tokens |
| Directional claims (“up 12%” when it went down) | Depends on the scoring model | Caught deterministically – arithmetic doesn’t hallucinate about hallucinations |
| Formal verification | Bedrock Automated Reasoning: policy rules checked by formal logic – genuinely novel for policy Q&A | Determinism applied to the numbers themselves; judge escalation only when a claim needs judgment |
| Correction | Azure offers a correction feature (preview) | FIX mode repairs arithmetic in place; PREVENT blocks with a 422 |
| Source material | You pass a reference source and query explicitly per check | Uses the sources already in the request; RAG, web search, and consilium escalate optionally |
| Evidence | A groundedness score | A per-request grounding report: each claim marked up with what failed and why |
| Cost shape | Per-check API pricing, on every request | Cheapest-first: the deterministic pass is free; guard-model calls metered only on escalation |
The strategy: escalate only when a claim needs it
in/guard/out attacks hallucination with a layered, cheapest-first strategy. The first pass is deterministic – it recomputes the model’s arithmetic, detects figures that drifted from the source, and catches directional contradictions like “up 12%” when the numbers went down – with no extra model call. Only when a claim needs judgment does it escalate to a fact-check judge, and only when it needs outside evidence does it reach for retrieval.
In FIX mode, safe repairs happen automatically – wrong arithmetic is corrected in place. In PREVENT mode the response is blocked with a 422 before your app ever shows it. Every finding is recorded in a per-request grounding report: the answer marked up with exactly what was caught and why.
The layers, cheapest first
Deterministic numeric grounding
Recomputes every arithmetic claim, matches every figure in the answer to the source, and flags directional / percentage contradictions. Zero extra tokens; it doesn’t hallucinate about hallucinations.
Fact & consistency judge
A guard model flags internal contradictions, claims false by common knowledge, and overconfident extrapolation. An honest “I don’t know” or a hedged estimate isn’t punished – only unhedged certainty without support.
RAG against your documents
Retrieves your ingested document chunks (pgvector) and verifies the answer is supported by them.
Web search & consilium
Checks claims against live web-search results, or puts the answer in front of one or more independent critic models – disagreement is a grounding signal.
Private data via MCP Coming soon
Verify a claim against a private source – e.g. a market-share figure only your own data can confirm.
Detecting LLM hallucinations: most well-known approaches
Types of LLM hallucinations: faithfulness vs. factuality
“Hallucination” is really two problems. A faithfulness failure is an answer that contradicts, or isn’t supported by, the source material it was given – the RAG system that cites a document, then states something the document never says. A factuality failure is an answer that is simply false about the world – the confidently invented statistic, the fabricated legal citation, the API that doesn’t exist. The distinction matters because they’re caught differently: faithfulness is checkable against the context in the request, while factuality needs outside evidence or a model that knows the domain.
Within faithfulness, it’s worth separating contradictions (the answer directly disagrees with the source) from unsupported claims (the answer adds detail the source neither confirms nor denies). A contradiction is almost always wrong; an unsupported claim may be true but ungrounded – and for a report or a customer reply, “true but we can’t show why” is often still a problem. Fabricated citations are the most notorious extrinsic case: fluent, plausible references to papers, cases, or URLs that don’t exist.
Detecting LLM hallucinations: most well-known approaches
Lexical overlap (BLEU, ROUGE)
Measures how much of the answer’s wording appears in the source. Almost free and very high-precision – if the words aren’t there at all, something is off – but low-recall, because a faithful paraphrase shares few exact tokens. A cheap first filter, not a verdict.
Semantic similarity (embeddings)
Embeds the answer and the source and compares them by cosine distance, catching paraphrases that lexical overlap misses. Better at meaning, but it blurs exactly the small, decisive edits – a flipped number, a changed name – that matter most, so on their own these scores tend to be precise but miss a lot.
Self-consistency / stochastic sampling
The idea behind SelfCheckGPT and BERT-based consistency checks: generate the same answer several times and measure how much the samples agree. A fact the model actually knows stays stable across runs; a hallucination wobbles. It needs no reference source, which makes it valuable for factuality, but it costs several generations per check.
LLM-as-a-judge
Hands a second model the source and the answer and asks it to rule. The strongest versions are rubric-guided: the judge must quote the exact passage from the source and the exact claim from the answer, label each as contradiction, unsupported, or agreement, and reason before it scores. The highest-recall method and the one that reads intent, but every check is a model call with a real token bill.
RAG hallucination detection: run cheap checks first
No single technique wins on both precision and recall at an acceptable cost, which is why practitioners – AWS’s RAG guidance among them – converge on the same shape: run a cheap, high-precision filter first to clear the obvious cases, then spend an expensive judge only on what survives. in/guard/out is built on exactly that gradient: the free deterministic checks above run first, and only the claims they can’t settle move up a tier. What the deterministic tier can’t judge – nuance, paraphrase, unsupported-but-plausible claims – escalates to a rubric-style claim audit that must cite verbatim evidence, and, when a claim needs an outside opinion, to retrieval over your own documents, live web search, or a consilium of independent critic models whose disagreement is itself a grounding signal. Each escalation is opt-in and metered, so you pay for the expensive methods only on the answers that actually need them.
Automatic hallucination repair on the wire
Most detectors – and every technique above – stop at a number: a groundedness score, a faithfulness verdict, a consistency percentage. In production that leaves the hardest part undone, because a flagged answer has still been generated and is still sitting in front of your user. in/guard/out closes that gap by living on the wire: a caught arithmetic error is repaired in place in FIX mode, an unsupported answer is blocked with a 422 in PREVENT mode before your app receives it, and every finding is written to a per-request grounding report. The techniques in this guide produce the signal; the value is in acting on it in line, before the response ships.
in/guard/out’s own checks
Beyond these general-purpose scorers, in/guard/out incorporates custom-developed hallucination guardrails that detect possible hallucinations and FIX or BLOCK them.
Frequently asked questions
How do you detect hallucinations without another LLM?
The first layer is deterministic: it recomputes arithmetic, compares every number in the answer against the source material, and checks stated directions against the underlying values. That class of hallucination – altered figures, botched math, contradicted trends – is caught with no model call at all.
Can hallucinations be fixed automatically?
Where a safe fix exists, yes. In FIX mode wrong arithmetic is repaired in place; findings with no single safe rewrite (contradictions, unsupported claims) are flagged and recorded. In PREVENT mode the response is blocked before your app receives it.
What does it cost per request?
The deterministic pass adds no model tokens. Judge and retrieval layers are opt-in, and every guard-model sub-call is priced into the request, so the dashboard shows the true cost of checking – measured, not estimated.
Does this let us run a cheaper model?
That is the core economics: in/guard/out watches the output, so you can route traffic to a cost-effective model and keep the quality bar. The guarantees come from the checks, not from the model tier.
Related guardrails
Catch hallucinations on your data
Upload a document and watch in/guard/out catch altered figures, unsupported claims, and contradictions in real time. We are running a limited demo - sign up and we will get you in as soon as we can.