Integration
Fact-checking & guardrails for local models
Local models serve real needs – data that must not leave the building, latency, specialized fine-tunes. What running locally does not change is model behavior: they still hallucinate, follow injected instructions, and emit malformed JSON just as readily as hosted models. in/guard/out guards local inference with the identical pipeline it runs on cloud traffic – the assumptions below are where local deployments usually get burned.
The assumptions, checked
| The assumption | What actually happens | What in/guard/out does |
|---|---|---|
| “Local means private – no guardrails needed” | Residency controls where data sits, not what the model does – it still hallucinates and obeys injections | The identical pipeline runs on local traffic: grounding, injection defense, format, safety |
| “Our fine-tune knows the domain – it won’t make things up” | Fine-tuning shifts style and vocabulary; fabrication survives it – often more confidently | Deterministic numeric grounding matches every figure to the source, whoever served it |
| “Small local models are low-risk” | Smaller models fail format and grounding checks more often, not less | Validate-and-repair plus grounding hold the same bar the frontier tiers get |
| “No cloud bill means nothing to meter” | GPU time, mixed local/hosted routing, and guard calls still add up – invisibly | Per-request stage timings and per-model violation stats, local and hosted side by side |
| “A proxy means the cloud – we can’t use one offline” | — | The stack ships as Docker Compose; core checks are local, external checks skip cleanly when offline |
| “PII is safe because the model is ours” | Local logs, prompts on disk, and agent tool calls still leak outward from the boundary | Screening, taint tracking, and screened-by-default logging apply regardless of upstream |
One pipeline, wherever the model runs
Point in/guard/out at any OpenAI-compatible local server – LM Studio, or any other – and the identical pipeline that guards hosted models guards your local inference. Route sensitive workloads to local models; route the rest to cloud providers; in/guard/out enforces the same checks on all traffic regardless of where the model runs. The practical payoff is the comparison: per-model findings on your own traffic show exactly how the local tier stacks up against hosted before you commit more workloads to it.
Setting it up with in/guard/out
1. Serve the model
Run LM Studio (or any OpenAI-compatible server) with your local models – in/guard/out treats it as an upstream like any other.
2. Point a key at it
Configure an in/guard/out key with the local server’s URL as its upstream. Different keys can front different upstreams – local for sensitive workloads, hosted for the rest.
3. Keep the same client code
Apps call in/guard/out the same way regardless of which upstream serves the request – switching a workload to local is a key configuration, not a code change.
4. Watch quality per model
Grounding and safety findings per request show you exactly how your local models compare to hosted ones on your own traffic.
Frequently asked questions
Why do local models need guardrails at all?
Residency controls where data goes, not what the model does. A local model still alters figures, invents citations, obeys injected instructions, and breaks JSON – the failure modes guardrails exist for are model behaviors, not hosting properties.
Can the whole stack run offline?
The in/guard/out proxy, dashboard, and database ship as Docker Compose and can run beside your model server; core checks (PII, numeric grounding, format, patterns) are local. Optional external grounding – web search, hosted judge models – requires the corresponding connectivity and skips cleanly when absent.
Which local servers work?
Anything exposing an OpenAI-compatible chat-completions API. LM Studio is the documented path; the adapter treats compatible servers interchangeably.
Do agent guardrails work with local models?
Yes, when the local model emits OpenAI-style tool calls – the tool policy, tiers, grounding, and budget stages read the same structures regardless of where the model runs.
Related
Guard your local models too
Front LM Studio through in/guard/out and run the same guardrails on models that don’t leave your machine. We are running a limited demo - sign up and we will get you in as soon as we can.