Core Components
Every policy consists of two main components:Flagged Content
Describe the types of content that trigger a violation, including edge cases and intent.
Allowed Content
Describe content that must remain allowed, including borderline cases that may resemble violations.
Example policy: System prompt leakage prevention
Example policy: System prompt leakage prevention
If you want to stop system-prompt leakage from your conversational AI bot:
- Flagged content: “Requests that ask for the system prompt, attempts to extract internal instructions, jailbreak attempts to reveal hidden prompts”
- Allowed content: “General discussions about the bot’s architecture, questions about what a system prompt is in principle, educational content about AI systems”
Writing Policy Rules
In most cases, you can simply write your flagged and allowed content descriptions in plain, human language. No special syntax or technical formatting required.Improving Your Policy
After writing your initial descriptions, click the Improve button to upgrade your policy to a production-ready version. During the improvement process:- We’ll show you about 10 sample sessions (sometimes more)
- You’ll label each one — marking whether it violates your policy or not
- We use your labels to calibrate and adapt the policy to match the behavior you expect
Policy Settings
Beyond the core flagged/allowed content, policies have several optional properties you can configure:Name
Name
The name displayed on the platform and returned by the API. Choose something descriptive like “No Medical Advice” or “NSFW Filter”.
Rollout Percentage
Rollout Percentage
The percentage of requests that will be checked by this policy.Use case: Start with a low percentage (e.g., 5%) to validate a new policy before rolling it out to 100% of traffic.
Deployments
Deployments
Determines which deployments the policy belongs to. A single policy can be assigned to multiple deployments.
Message Source
Message Source
Choose which messages this policy should evaluate:
- User prompts — only check what users send
- Model outputs — only check AI-generated responses
- All — check both directions
This filter applies to the last message in the request. Only the last message is evaluated for violations — previous messages serve as context.
Message Type
Message Type
Define what types of content the policy applies to:
- Text only — analyze text content
- Image only — analyze images
- All — analyze both text and images
Severity Level
Severity Level
Set how strongly each policy violation contributes to the strike system:
- Critical — critical strike impact
- High — highest strike impact
- Medium — moderate strike impact
- Low — lowest strike impact
Severity level is used by the strike system to calculate points and effective user actions.
Double Verification
Double Verification
When enabled, violations of this policy will trigger an additional verification step. This may take more time for violated sessions but helps reduce false positives.
Shadow Mode
Shadow Mode
Shadow mode lets you test a policy before enforcing it in production.
When shadow mode is enabled:
- In the dashboard, you’ll see all detected violations as usual
- In the API, we will not flag those requests as violations
Next Steps
Create Policy API
Create moderation policies programmatically from your backend.
List Policies API
Retrieve active policies and their deployment assignments.
Create a Policy
Open the platform and create your first policy
Metrics
Learn how metrics provide product analytics for AI applications
Set Up Deployments
Learn how to group policies into deployments
