AI gateway
in/guard/out vs LiteLLM
LiteLLM solves routing – a hundred-plus providers behind one API, load balancing, failover, key and team management. Keep it. in/guard/out answers the question a router doesn’t ask: can you trust what the traffic is doing? It governs the run – the whole loop of model calls, tool calls, and side effects an agent makes – and it sits behind LiteLLM with the few lines of verified config below.
Side by side
| Capability | LiteLLM | in/guard/out |
|---|---|---|
| Unit of governance | The request and the API key | The run – every request, tool call, and workflow checkpoint in one governed session |
| Agent budget enforcement | Dollar + iteration caps per session, in a cache counter that checks committed spend and expires when idle | Hard caps: admission atomically reserves worst-case cost against a durable ledger |
| Runaway-loop protection | Counts calls per session | Fingerprints identical tool calls – catches the agent stuck in a loop, not just a long run |
| Tool-call control | Name/argument regex rules; per-tool trust flags | JSON-schema validation, permission tiers (read → payment), per-key policy, dangerous-arg scanning |
| Action grounding | — | A payment’s IBAN, amount, or ID must appear in something the run legitimately read |
| PII handling | Presidio masking, with optional un-masking of the response | Screen & restore plus egress control: taint tracking follows screened values into tool arguments, per-tool rules decide which entities may leave |
| Hallucination checks | LLM-as-judge hook; vendor integrations | Deterministic first – arithmetic recomputed, number drift caught, zero tokens – then claim audit and judges |
| When a check fails | Block or mask | Repair where safe: fix the JSON, correct the drifted number, restore the PII |
| Workflow conformance | — | Tool-ordering policies enforced as a state machine – auto-compiled from an n8n export |
| Steps that never touch an LLM | — | A checkpoint API with drop-in n8n node, LangChain callback, and Python SDK |
| Run visibility | Spend logs and tracing callbacks | A reconstructed run graph – what the agent actually did, with cost burn-down and coverage |
Gateways govern requests. Runs are where the risk lives.
Every failure that makes teams afraid to turn an agent on is run-scoped: the loop that burns a budget across forty requests, the poisoned document that plants a payee three steps before the payment, the screened email address that exits through a tool argument five calls later. A request-scoped hook – however good – sees each call in isolation. in/guard/out correlates the run by session id and enforces across it: budgets that stop the next call, payees grounded in sources, taint tracking on tool arguments, and sequence policies over tool order.
The chain is deliberately boring: your apps keep calling LiteLLM; LiteLLM forwards to in/guard/out as an OpenAI-compatible backend; in/guard/out runs the pipeline and reaches your providers. No app code changes, routing stays where it is, and every agent run comes back with a graph, a true cost, and an audit trail it never had.
litellm config.yaml – point a deployment at in/guard/out (verified chain)
model_list:
- model_name: gpt-4o-mini # what your apps request – unchanged
litellm_params:
# "openai/" tells LiteLLM to speak the OpenAI protocol to api_base;
# what follows it is the in/guard/out model string, forwarded as-is.
model: openai/openai/gpt-4o-mini
api_base: https://guard.example.com/v1
api_key: os.environ/INGUARDOUT_API_KEY # any in/guard/out API key
general_settings:
forward_client_headers_to_llm_api: true # forwards X-Guardrails-Session-Id per run
# Nothing to configure on the in/guard/out side – every key from the
# dashboard's Keys tab accepts OpenAI-compatible traffic out of the box.Frequently asked questions
Is in/guard/out a LiteLLM alternative?
Only where the jobs overlap – and they mostly don’t. LiteLLM is a routing gateway; in/guard/out is a runtime control plane for what your apps and agents actually do. The recommended architecture is both, chained: routing in front, governance behind.
LiteLLM has session budgets – isn’t that the same thing?
Same intent, different foundation. A cache-counter cap checks committed spend and resets when the session goes idle – an agent that pauses resumes with a clean slate, and simultaneous requests can slip past the same reading. in/guard/out admits each request by atomically reserving its worst-case cost against a durable ledger, so the cap holds under concurrency, across restarts, and in the audit record.
Both products integrate Presidio – what’s different?
What happens after the mask. Both can mask on the way in and swap real values back into the response. in/guard/out also follows the screened values through the rest of the run: taint tracking stops a screened email or account number from exiting through a tool argument, and per-tool egress rules say which entity types each tool may see at all. Masking protects the provider; taint protects everything the agent touches.
Does tool calling survive the chain?
Yes. LiteLLM forwards OpenAI-compatible tool calls unchanged, so the agent guardrails – tool policy, permission tiers, action grounding, loop detection, budgets – engage exactly as if the agent called in/guard/out directly.
What does the chain cost in latency?
One extra hop plus the guardrail work itself, which the dashboard reports per stage, per request – deterministic checks are sub-millisecond, and model-based checks run concurrently and only when enabled. You see the guardrail tax as a measured number, not an estimate.
Keep your gateway. Govern your runs.
Chain in/guard/out behind LiteLLM with a few lines of config - FIX mode observes first, so you can measure what it catches before enforcing anything. We are running a limited demo - sign up and we will get you in as soon as we can.