AI Gateway
in/guard/out vs LiteLLM
LiteLLM solves routing – a hundred-plus providers behind one API, load balancing, failover, key and team management. Keep it. in/guard/out answers the question a router doesn’t ask: can you trust what the traffic is doing? It governs the run – the whole loop of model calls, tool calls, and side effects an agent makes – and it sits behind LiteLLM with the few lines of verified config below.
Side by side
| Capability | LiteLLM | in/guard/out |
|---|---|---|
| Unit of governance | The request and the API key | The run – every request, tool call, and workflow checkpoint in one governed session |
| Agent budget enforcement | Dollar + iteration caps per session, in a cache counter that checks committed spend and expires when idle | Hard caps: admission atomically reserves worst-case cost against a durable ledger |
| Runaway-loop protection | Counts calls per session | Fingerprints identical tool calls – catches the agent stuck in a loop, not just a long run |
| Tool-call control | Name/argument regex rules; per-tool trust flags | JSON-schema validation, permission tiers (read → payment), per-key policy, dangerous-arg scanning |
| Action grounding | — | A payment’s IBAN, amount, or ID must appear in something the run legitimately read |
| PII handling | Presidio masking, with optional un-masking of the response | Screen & restore plus egress control: taint tracking follows screened values into tool arguments, per-tool rules decide which entities may leave |
| Hallucination checks | LLM-as-judge hook; vendor integrations | Deterministic first – arithmetic recomputed, number drift caught, zero tokens – then claim audit and judges |
| When a check fails | Block or mask | Repair where safe: fix the JSON, correct the drifted number, restore the PII |
| Workflow conformance | — | Tool-ordering policies enforced as a state machine – auto-compiled from an n8n export |
| Steps that never touch an LLM | — | A checkpoint API with drop-in n8n node, LangChain callback, and Python SDK |
| Run visibility | Spend logs and tracing callbacks | A reconstructed run graph – what the agent actually did, with cost burn-down and coverage |
Based on LiteLLM’s publicly available information. We’ve done our best to keep it accurate, but details may have changed since we last looked. If you’ve noticed an error, please email us.
Gateways govern requests. Runs are where the risk lives.
Every failure that makes teams afraid to turn an agent on is run-scoped: the loop that burns a budget across forty requests, the poisoned document that plants a payee three steps before the payment, the screened email address that exits through a tool argument five calls later. A request-scoped hook – however good – sees each call in isolation. in/guard/out correlates the run by session id and enforces across it: budgets that stop the next call, payees grounded in sources, taint tracking on tool arguments, and sequence policies over tool order.
The chain is deliberately boring: your apps keep calling LiteLLM; LiteLLM forwards to in/guard/out as an OpenAI-compatible backend; in/guard/out runs the pipeline and reaches your providers. No app code changes, routing stays where it is, and every agent run comes back with a graph, a true cost, and an audit trail it never had.
litellm config.yaml – point a deployment at in/guard/out (verified chain)
model_list:
- model_name: gpt-4o-mini # what your apps request – unchanged
litellm_params:
# "openai/" tells LiteLLM to speak the OpenAI protocol to api_base;
# what follows it is the in/guard/out model string, forwarded as-is.
model: openai/openai/gpt-4o-mini
api_base: https://guard.example.com/v1
api_key: os.environ/INGUARDOUT_API_KEY # any in/guard/out API key
general_settings:
forward_client_headers_to_llm_api: true # forwards X-Guardrails-Session-Id per run
# Nothing to configure on the in/guard/out side – every key from the
# dashboard's Keys tab accepts OpenAI-compatible traffic out of the box.Frequently asked questions
Is in/guard/out a LiteLLM alternative?
Only where the jobs overlap – and they mostly don’t. LiteLLM is a routing gateway; in/guard/out is a runtime control plane for what your apps and agents actually do. The recommended architecture is both, chained: routing in front, governance behind.
LiteLLM has session budgets – isn’t that the same thing?
Same intent, different foundation. A cache-counter cap checks committed spend and resets when the session goes idle – an agent that pauses resumes with a clean slate, and simultaneous requests can slip past the same reading. in/guard/out admits each request by atomically reserving its worst-case cost against a durable ledger, so the cap holds under concurrency, across restarts, and in the audit record.
Both products integrate Presidio – what’s different?
What happens after the mask. Both can mask on the way in and swap real values back into the response. in/guard/out also follows the screened values through the rest of the run: taint tracking stops a screened email or account number from exiting through a tool argument, and per-tool egress rules say which entity types each tool may see at all. Masking protects the provider; taint protects everything the agent touches.
Does tool calling survive the chain?
Yes. LiteLLM forwards OpenAI-compatible tool calls unchanged, so the agent guardrails – tool policy, permission tiers, action grounding, loop detection, budgets – engage exactly as if the agent called in/guard/out directly.
What does the chain cost in latency?
One extra hop plus the guardrail work itself, which the dashboard reports per stage, per request – deterministic checks are sub-millisecond, and model-based checks run concurrently and only when enabled. You see the guardrail tax as a measured number, not an estimate.
Keep your gateway. Govern your runs.
Chain in/guard/out behind LiteLLM with a few lines of config - FIX mode observes first, so you can measure what it catches before enforcing anything. We are running a limited demo - sign up and we will get you in as soon as we can.