# Configuration Structure

A guardrail configuration contains several properties that customize how the service interacts with models and applies safety checks. This page describes the core components (models, prompts, and rails) as well as advanced options for tracing, passthrough, and other behaviors.

---

## Models

A model configuration defines the LLM to use for a specific task. It consists of the following fields:

- `type`: The task the model is used for (for example, `content_safety` or `self_check_input`).
- `engine`: The model provider. For most cases, use `nim`.
- `model`: The name of the model to use for the task, in `workspace/name` format (for example, `default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3`).
- `mode`: The completion mode. Allowed values are `"chat"` (default) or `"text"`.
- `cache`: Cache configuration for this model. Primarily used for content safety models to cache repeated checks. Contains `enabled` (default: `false`), `maxsize` (default: `50000`), and `stats` sub-fields.
- `parameters`: Additional properties to configure interacting with the model. The following fields are supported for all model types:
  - `base_url`: The URL to use for inference with this model. When using models deployed through the Inference Gateway, {{ngm_short_name}} automatically resolves the URL through IGW's route table. You do not need to explicitly set a Base URL for your model.
  - `default_headers`: Custom HTTP headers to include in requests to this model. Each key-value pair represents a header name (key) and its default value (value).

Under the IGW plugin architecture, the main model is owned by IGW and specified through the VirtualModel's `default_model_entity`. Guardrail configurations should omit the `type: "main"` model entry — the plugin injects the per-request main model at runtime, and self-check rails reuse that per-request main model for evaluation.

Include a `type: "main"` entry only when you need to pin the engine or parameters used to talk to the main model — for example, a non-NIM `engine`, a custom `parameters.base_url`, or static `parameters.default_headers`. The `model` field on a main entry is treated as a template; the actual model name always comes from the inference request body.

You can configure task-specific models for any task that occurs during the guardrail process. `type` is a free-form label — rails reference it via `$model=<type>` — so you can declare custom task types in addition to the common ones below (for example, a `vision_rails` model used in [Adding Safety Checks to Multimodal Data](../../tutorials/multimodal-data.md)):

- **`content_safety`**: Content Safety check for detecting harmful content.
- **`topic_control`**: Topic Control check for keeping conversations on-topic.
- **`jailbreak`**: Jailbreak detection check.
- **`self_check_input`**: Safety check that automatically checks the user input using the `main` model for inference.
- **`self_check_output`**: Safety check that automatically checks the final LLM output using the `main` model for inference.

```python
models = [
    {
        "type": "content_safety",
        "engine": "nim",
        "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3",
    },
    {
        "type": "topic_control",
        "engine": "nim",
        "model": "default/nvidia-llama-3-1-nemoguard-8b-topic-control",
    },
]
```

!!! tip "Model Entities are automatically created when you:"
    - Create a Model Provider pointing to an external API (refer to [Add External Providers](../../../run-inference/about.md#add-external-providers))

    Use `client.models.list()` to retrieve available Model Entities in your workspace.

### Using Direct URLs

Using Inference Gateway is the recommended approach for interacting with models. If you require a direct connection to specific endpoints, you can explicitly set `parameters.base_url`:

```python
models = [
    {
        "type": "main",
        "engine": "nim",
        "parameters": {"base_url": "http://my-local-nim:8000/v1"},
    },
    {
        "type": "content_safety",
        "engine": "nim",
        "model": "nvidia/llama-3.1-nemotron-safety-guard-8b-v3",
        "parameters": {"base_url": "http://my-content-safety-nim:8000/v1"},
    },
]
```

---

## Prompts

A prompt is used by the model during a task to evaluate a message. It consists of the following fields:

- `task`: The task to apply the prompt to.
- `content`: The content of the prompt. Mutually exclusive with `messages`.
  - Prompts that require a dynamic input variable(s) use Jinja2 templating. For example, the `{%raw%}{{ user_input }}{%endraw%}` variable is replaced with the end user's input at runtime.
- `messages`: A list of messages for chat-model prompts. Mutually exclusive with `content`. Each message has `type` (such as `"system"` or `"user"`) and `content` fields.
- `output_parser`: Name of output parser to process the model's response.
- `max_tokens`: Maximum number of tokens the model can generate.
- `max_length`: Maximum prompt length in characters. When the maximum length is exceeded, the prompt is truncated by removing older turns from the conversation history until the length of the prompt is less than or equal to the maximum length. The default is 16,000 characters.
- `models`: Restricts this prompt to specific LLM engines or models. Format: a list of strings such as `"<engine>"` or `"<engine>/<model>"`.
- `mode`: The prompting mode for this prompt. Defaults to the top-level `prompting_mode` value (typically `"standard"`).
- `stop`: A list of stop tokens for models that support this feature.

For Content Safety and Topic Control checks, prompts must include the model reference in the task name:

```python
prompts = [
    {
        "task": "content_safety_check_input $model=content_safety",
        "content": "Task: Check for unsafe content...",
        "output_parser": "nemoguard_parse_prompt_safety",
        "max_tokens": 50,
    },
    {
        "task": "topic_safety_check_input $model=topic_control",
        "content": "Ensure the user messages meet the following guidelines: ...",
        "max_tokens": 50,
    },
]
```

### Prompt Template Variables

The following dynamic variables can be used in the prompt content:

| Variable | Description |
|----------|-------------|
| `{%raw%}{{ user_input }}{%endraw%}` | The end user's input message |
| `{%raw%}{{ bot_response }}{%endraw%}` | The LLM-generated response |
| `{%raw%}{{ context }}{%endraw%}` | Additional context provided in the request |
| `{%raw%}{{ relevant_chunks }}{%endraw%}` | Retrieved chunks in RAG deployments |

---

## Rails

Rails specify which flows to apply to the user input and LLM output. The `rails` object accepts the following top-level keys, each triggered at a different point in the request lifecycle:

| Key | When Applied |
|-----|--------------|
| `input` | Before user input reaches the main model. |
| `output` | After the LLM generates output, before returning to the user. |

Each rail key supports a `flows` list and, optionally, a `parallel` flag to execute those flows concurrently.

The following example defines the flow to run as an input and output rail.

```python
rails = {
    "input": {"flows": ["self check input"], "parallel": False},
    "output": {
        "flows": ["self check output"],
        "parallel": False,
        "streaming": {
            "enabled": False,
            "chunk_size": 200,
            "context_size": 50,
            "stream_first": True,
        },
        "apply_to_reasoning_traces": False,
    },
}
```

### Output Rails Streaming

Output rails support a `streaming` configuration for processing LLM tokens in chunks:

- `enabled`: Enables streaming mode (default: `false`).
- `chunk_size`: Number of tokens per processing chunk (default: `200`).
- `context_size`: Number of tokens carried from the previous chunk for continuity (default: `50`).
- `stream_first`: If `true`, token chunks are streamed before output rails are applied (default: `true`).

### Rails-Specific Configuration

Specific rails can require additional configuration that you specify in the `config` key. The following integrations are supported:

- `jailbreak_detection` -- Threshold-based jailbreak detection.
- `injection_detection` -- Prompt injection detection.

```python
rails = {
    "input": {
        "flows": ["self check input"],
    },
    "output": {
        "flows": ["self check output"],
    },
    "config": {
        # Configures jailbreak detection settings
        "jailbreak_detection": {
            "length_per_perplexity_threshold": 89.79,
            "prefix_suffix_perplexity_threshold": 1845.65,
        }
    },
}
```

---

## General Instructions

Instructions provide context to the model about expected behavior. They are appended to the beginning of every prompt (similar to a system prompt).

```python
instructions = [
    {
        "type": "general",
        "content": """You are a customer service bot for ABC Company.
You answer questions about products and policies.
If you don't know an answer, say so honestly.
Always be polite and professional.""",
    }
]
```

---

## Sample Conversation

The sample conversation sets the tone for conversations between the user and the bot. It helps the LLM learn the format, tone, and verbosity of responses. Include a minimum of two turns. The sample conversation is appended to every prompt; keep it short and relevant.

```python
sample_conversation = """user "Hi there. Can you help me with some questions I have about the company?"
 express greeting and ask for assistance
 bot express greeting and confirm and offer assistance
 "Hi there! I'm here to help answer any questions you may have about the ABC Company. What would you like to know?"
 user "What's the company policy on paid time off?"
 ask question about benefits
 bot respond to question about benefits
 "The ABC Company provides eligible employees with up to two weeks of paid vacation time per year, as well as five paid sick days per year. Please refer to the employee handbook for more information."
"""
```

---

## Advanced Options

The guardrail configuration supports additional top-level fields for fine-tuning behavior. All fields below are optional and have sensible defaults.

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `prompting_mode` | string | `"standard"` | The prompting mode for all prompts. Can be overridden per prompt. |
| `lowest_temperature` | float | `0.1` | Minimum temperature for the main model |
| `tracing.enabled` | boolean | `false` | Enable OpenTelemetry tracing |
| `tracing.enable_content_capture` | boolean | `false` | Include prompt/response content in traces |
| `passthrough` | boolean | `false` | When `true`, bypass all rails and forward the request directly to the model |

!!! warning "Enabling `tracing.enable_content_capture` causes prompts and responses (including user, assistant, and tool message content) to be included in telemetry events. This can expose PII and sensitive data in your telemetry backend."

---

## Next Steps

- Refer to [Manage Configurations](manage-configs.md) to create, update, and manage your guardrail configurations.
