Choosing an approach
Library, gateway, firewall, or proxy?
“LLM guardrails” names at least four different product shapes, and they are not interchangeable – each enforces at a different point, with different authority. Here is the honest map, including where a proxy is not the right tool.
Validation library
Checks run inside your application process, wired in with code.
Maximum flexibility; no extra infrastructure; great for a single, deeply customized app.
Every app integrates and updates separately; uninstrumented apps and low-code tools get nothing; no cross-request view, so agent runs, loops, and budgets are invisible; enforcement lives inside the process it is meant to police.
AI gateway
A proxy focused on routing, caching, key management, and rate limits – with moderation filters bolted on.
One integration point; solid for cost routing, failover, and provider abstraction.
Safety is a feature, not the product: typically keyword or moderation-API filters on input and output. No grounding of answers in sources, no tool-call policy, no run-level budgets or action-level enforcement for agents.
Prompt firewall
A detection layer classifying inputs (and sometimes outputs) for injection and abuse.
Strong classifiers for the injection problem specifically; useful signal for a SOC.
Detection without enforcement of consequences: it can say “this looks like an attack” but not “this payment is ungrounded, block it”. Hallucination, format, PII restore, and agent budgets are out of frame.
Eval & observability platform
Measures quality on test sets and traces production calls for later analysis.
Essential for development: regression suites, prompt comparison, failure analysis.
After the fact by design. A trace of the incident is not a block of the incident – nothing stands in line to stop the response or the tool call that fails the check.
Where the guardrails proxy sits
On the wire, in both directions, with enforcement authority: it screens and restores PII, grounds answers in their sources, validates and repairs output, and – for agents – checks every tool call against policy, tiers, grounding, taint, and budgets before it runs. It embeds the best of the library ecosystem (Presidio, NeMo Guardrails, Outlines, Prompt Guard) and operates it as infrastructure: one integration, one policy surface, one audit trail across the LLM boundary, workflow checkpoints, and the MCP gateway.
Keep PII/PHI out of model providers
A proxy is the natural fit: screening must happen before the request leaves your boundary, and restore must happen on the way back – on every app, including the ones nobody instrumented.
Stop hallucinated figures from shipping
Needs in-line output enforcement with access to the source material in the request – deterministic grounding plus a judge, with block/repair authority. Libraries can do parts per-app; evals only measure it.
Control what an AI agent may do
Needs cross-request state: runs, budgets, loop detection, tool history for sequence policies. Only something on the wire sees all of an agent’s requests; in-process libraries see one call at a time.
Secure MCP tool traffic
Needs an MCP-speaking gateway between agent and servers, where a deny is a protocol error the client cannot ignore – plus manifest pinning against rug pulls.
Route, cache, and manage provider keys
An AI gateway’s home turf. The proxy focuses on enforcement; it composes with a gateway (either side of it) if routing is a separate concern in your stack.
Regression-test prompts before release
An eval platform’s job, and the proxy does not replace it. Evals catch what you can reproduce; the proxy catches what happens anyway in production.
Is this a NeMo Guardrails / Guardrails AI alternative?
It is the deployment alternative: enforcement on the wire instead of a library in each app. Not a rip-and-replace of the ecosystem, either – the proxy embeds respected open-source components (Presidio for PII, NeMo Guardrails for topic rails, Outlines for structured output, Prompt Guard for injection) and operates them behind one base URL, adding what a library cannot: cross-request runs, budgets, tool-call enforcement, and one audit trail.
Do I need a guardrails proxy if I already use an AI gateway?
They solve different jobs and compose: the gateway handles routing, caching, and key management; the proxy enforces safety – grounding, PII screen/restore, tool policy, budgets. Chain them if you need both.
Proxy vs library – what is the honest trade-off?
The proxy adds a network hop and a component to run, and per-app custom logic is easier in code. The library costs you N integrations, no coverage for uninstrumented traffic, and no cross-request view of agent runs. Teams with one app and strong engineers often start with a library; teams with many apps, agents, or compliance duties end up on the wire.
Can the proxy replace offline evals?
No, and it is not trying to. Evals measure quality before you ship; the proxy enforces policy after you ship. The overlap is that proxy findings – what was actually caught in production – make an excellent source of new eval cases.
Decide on evidence: run it next to what you have.
FIX mode observes without blocking, so you can measure what the wire would have caught before changing anything. We’re running a limited demo.