Content safety
Unsafe output stops at the boundary
Model providers moderate their side – OpenAI’s Moderation API is even free – but your obligations are your own. Provider moderation answers to the provider’s categories; a support bot that emits something toxic speaks with your brand’s voice, whatever the upstream model was. Content safety belongs at the boundary you control, tuned to your policy.
Side by side
| Concern | Provider moderation (OpenAI, Azure, Anthropic) | in/guard/out |
|---|---|---|
| Whose policy | The provider’s harm categories, uniform for everyone | Your topics, your profanity list, your thresholds |
| Direction | Primarily input moderation; output is yours to wire | Both directions, in line – nothing extra to wire per app |
| Action on a hit | A score your code must act on; blocks nuke the whole response | Surgical censoring where safe, block where not – or follow FIX/PREVENT |
| Classifier design | One provider model | A cascade: fast local profanity pass, then shield models (ShieldGemma, WildGuard classes) |
| Abusive prompts | Moderated if you call the endpoint first | Checked at the boundary before the tokens are spent |
| Evidence | A category score, if you log it | Per-request record: what flagged, which classifier, what action was taken |
| Tuning over-blocking | Fixed thresholds | Configurable cascade; leaks vs over-blocks measured by the product’s own eval harness |
Your policy, enforced in both directions
in/guard/out runs a cascade: fast local profanity redaction censors surgically instead of nuking whole responses; dedicated shield models (ShieldGemma, WildGuard classes) score content for safety; topic controls keep the conversation inside your policy. The action is configurable – always-block for safety categories, or follow the request’s PREVENT/FIX mode.
Inputs can be checked too, so an abusive prompt is caught before it spends tokens. Every decision is recorded per request – what was flagged, by which classifier, and what action was taken.
The safety cascade
Profanity redaction
Local, deterministic censoring – surgical redaction first, so a fixable response is cleaned rather than withheld, then judged on the cleaned text.
Shield models
Dedicated safety classifiers (ShieldGemma, WildGuard classes) score prompt and response for harm categories.
Topic controls
Keep answers inside your domain policy – out-of-scope or disallowed topics are flagged or blocked per your configuration.
Configurable action
Safety can always-block regardless of mode, or follow PREVENT/FIX like other checks. Per key, per user, per request.
Frequently asked questions
Which classifiers do you use?
A cascade: local profanity filtering plus dedicated shield models in the ShieldGemma / WildGuard class, with an optional guard-model judge. The classifier set is configurable, and every sub-call is metered.
Can safety block even in FIX mode?
Yes. Content safety supports an always-block action independent of the request mode, for teams whose policy is that unsafe content doesn’t pass – while other checks stay observe-first.
Are user prompts checked as well as responses?
Both directions are supported: inbound checks catch abusive or policy-violating prompts before the model call; outbound checks judge what the model produced.
How is over-blocking measured?
Every flag and block is recorded per request with the classifier’s verdict, so you can review false positives in the dashboard and tune the cascade – and the product’s own eval harness scores leaks vs over-blocks.
Related guardrails
Test safety on your content
Send your own prompts and responses through in/guard/out and see what the classifiers catch, what gets censored, and where you would tune it. We are running a limited demo - sign up and we will get you in as soon as we can.