# Running Inference with Guardrails

{{ngm_short_name}} applies safety checks to inference requests through VirtualModels. When your application sends a request to a VirtualModel with guardrails middleware, the plugin runs input and output rails around the model call automatically. You use the standard IGW OpenAI-compatible endpoint — no separate guardrails endpoint is needed.

Platform-managed agents are the canonical consumer: pointing an agent's `llms` block at a guarded VirtualModel entity is how you secure the agent's model path. See [Secure Agents](../../agents/security.md) for the agent-side wiring. The rest of this page applies to any client calling a guarded VirtualModel, agent or not.

## Prerequisites

- A guardrail configuration exists in your workspace. See [Manage Configurations](configurations/manage-configs.md) to create one.
- A VirtualModel is configured with `nemo-guardrails` middleware. See [Architecture](architecture.md) for wiring details.

!!! tip
    If you do not have access to GPUs, you can use NIMs hosted on `build.nvidia.com`. Refer to [Setup](../../get-started/setup.md) for instructions on configuring a `ModelProvider` that routes requests to an external endpoint.

For the examples below, create a guardrail configuration and VirtualModel:

```python
{% raw %}
import os
from nemo_platform import NeMoPlatform, ConflictError

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

CONTENT_SAFETY_PROMPT = """\
Task: Check if there is unsafe content in the user or agent messages in the below conversation in accordance with our safety policy with the below categories. Provide a 'safe' or 'unsafe' rating for both the user message and the agent response. If there is unsafe content, please also provide a list of violated categories according to our safety policy below.

<BEGIN UNSAFE CONTENT CATEGORIES>
S1: Violence.
S2: Sexual.
S3: Criminal Planning/Confessions.
S4: Guns and Illegal Weapons.
S5: Controlled/Regulated Substances.
S6: Suicide and Self Harm.
S7: Sexual (minor).
S8: Hate/Identity Hate.
S9: PII/Privacy.
S10: Harassment.
S11: Threat.
S12: Profanity.
S13: Needs Caution.
S14: Other.
S15: Manipulation.
S16: Fraud/Deception.
S17: Malware.
S18: High Risk Gov Decision Making.
S19: Political/Misinformation/Conspiracy.
S20: Copyright/Trademark/Plagiarism.
S21: Unauthorized Advice.
S22: Illegal Activity.
S23: Immoral/Unethical.
<END UNSAFE CONTENT CATEGORIES>

<BEGIN CONVERSATION>

user: {{ user_input }}

<END CONVERSATION>

Provide your safety assessment for the above conversation in the below output JSON format:
{"User Safety": safe or unsafe rating for the user message, "Response Safety": safe or unsafe rating for the agent response. Omit if no agent response present. "Safety Categories": a comma-separated list of applicable safety categories from the provided taxonomy. Omit if all safe.}

Do not include anything other than the output JSON in your response.
Output JSON:"""

config_data = {
    "models": [
        {
            "type": "content_safety",
            "engine": "nim",
            "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3",
        }
    ],
    "prompts": [
        {
            "task": "content_safety_check_input $model=content_safety",
            "content": CONTENT_SAFETY_PROMPT,
            "output_parser": "nemoguard_parse_prompt_safety",
            "max_tokens": 50,
        },
    ],
    "rails": {
        "input": {
            "flows": ["content safety check input $model=content_safety"],
        },
    },
}

try:
    config = client.guardrail.configs.create(
        name="content-safety-config",
        description="Content safety input rail",
        data=config_data,
    )
except ConflictError:
    print("Config content-safety-config already exists, continuing...")
{% endraw %}
```

Create a VirtualModel that applies the guardrail configuration:

=== "CLI"

    ```bash
    nemo inference virtual-models create guarded-llama \
      --default-model-entity default/meta-llama-3-1-8b-instruct \
      --request-middleware '[{"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"default/content-safety-config"}]'
    ```

=== "Python SDK"

    ```python
    client.inference.virtual_models.create(
        name="guarded-llama",
        default_model_entity="default/meta-llama-3-1-8b-instruct",
        request_middleware=[
            {
                "name": "nemo-guardrails",
                "config_type": "guardrail_config",
                "config_id": "default/content-safety-config",
            }
        ],
    )
    ```

## Inference Endpoint

Inference requests go to the standard IGW OpenAI-compatible endpoint:

```
/apis/inference-gateway/v2/workspaces/{workspace}/openai/-/v1/chat/completions
```

Set the `model` field to your VirtualModel's entity reference (`workspace/name` format). IGW resolves the VirtualModel, runs the guardrails middleware pipeline, and proxies to the backend model.

### Chat Completions

=== "CLI"

    ```bash
    nemo chat default/guarded-llama "What is the capital of France?"
    ```

=== "curl"

    ```bash
    curl -s $NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions \
      -H 'content-type: application/json' \
      -d '{
        "model": "default/guarded-llama",
        "messages": [{"role": "user", "content": "What is the capital of France?"}],
        "max_tokens": 200
      }' | jq
    ```

=== "Python SDK"

    Get a pre-configured OpenAI client from the platform SDK and call it like any other OpenAI-compatible endpoint. The client's base URL points at the workspace-scoped IGW route, so `model="default/guarded-llama"` resolves through IGW's VirtualModel cache.

    ```python
    oai_client = client.models.get_openai_client()

    response = oai_client.chat.completions.create(
        model="default/guarded-llama",
        messages=[{"role": "user", "content": "What is the capital of France?"}],
        max_tokens=200,
    )
    print(response.choices[0].message.content)
    ```


### Model Routing

The `model` field in your request must reference a **VirtualModel** entity (`workspace/name` format). IGW resolves the VirtualModel, applies its middleware pipeline, and proxies to the backend model specified by the VirtualModel's `default_model_entity`.

Task models in your guardrail configuration (content safety, topic control, etc.) must reference **Model Entities** using the same `workspace/name` format. The plugin resolves their endpoints through IGW's route table.

For VirtualModel wiring (`request_middleware`/`response_middleware`, entity-backed vs inline configs), refer to [Architecture](architecture.md). For an overview of how Model Entities and Model Providers fit together, refer to [About Models and Inference](../../run-inference/about.md).

## Inline Configuration

Instead of referencing a stored config entity via `config_id`, you can embed the guardrail configuration directly in the VirtualModel's middleware entry using `config`:

=== "CLI"

    {% raw %}
    ```bash
    nemo inference virtual-models create guarded-llama-inline \
      --default-model-entity default/meta-llama-3-1-8b-instruct \
      --request-middleware '[{
        "name": "nemo-guardrails",
        "config_type": "guardrail_config",
        "config": {
          "name": "my-inline-config",
          "rails": {"input": {"flows": ["self check input"]}},
          "prompts": [{"task": "self_check_input", "content": "Check if harmful: \"{{ user_input }}\"\nAnswer (Yes/No):"}]
        }
      }]'
    ```
    {% endraw %}

=== "Python SDK"

    ```python
    {% raw %}
    client.inference.virtual_models.create(
        name="guarded-llama-inline",
        default_model_entity="default/meta-llama-3-1-8b-instruct",
        request_middleware=[
            {
                "name": "nemo-guardrails",
                "config_type": "guardrail_config",
                "config": {
                    "name": "my-inline-config",
                    "rails": {"input": {"flows": ["self check input"]}},
                    "prompts": [
                        {
                            "task": "self_check_input",
                            "content": 'Check if harmful: "{{ user_input }}"\nAnswer (Yes/No):',
                        }
                    ],
                },
            }
        ],
    )
    {% endraw %}
    ```

The inline config shares the same `LLMRails` pool when its content hash matches an existing entry. The plugin warms inline configs on VirtualModel upsert too — it walks the VM's middleware entries, stabilizes each source, and dedups by content hash — so the first request after upsert finds a hot pool just like entity-backed configs do. See [Architecture](architecture.md#caching-and-performance) for the cache details.

The optional `name` field sets the diagnostic label in logs (appears as `<inline:my-inline-config>`).

---

## Streaming Output

Streaming reduces time-to-first-token (TTFT) by returning chunks as they are generated. When output rails are configured, the plugin applies safety checks to chunks of tokens as they stream from the model.

### Configuration

Enable streaming in your guardrail config's output rails. The `streaming` property supports the following fields:

| Field | Type | Description | Default value |
| --- | --- | --- | --- |
| `enabled` | `boolean` | Enable LLM output streaming | `False` |
| `chunk_size` | `int` | Number of tokens per chunk that output rails process | `200` |
| `context_size` | `int` | Tokens carried over between chunks for continuity | `50` |
| `stream_first` | `boolean` | If `True`, tokens stream immediately before output rails are applied | `True` |

```python
rails = {
    "output": {
        "flows": ["self check output"],
        "streaming": {
            "enabled": True,
            "chunk_size": 200,
            "context_size": 50,
            "stream_first": True,
        },
    }
}
```

!!! note
    If the request sets `stream: true` but the guardrail config has output flows with `streaming.enabled: false`, IGW returns HTTP 400 with a message instructing you to set `rails.output.streaming.enabled=true`. Either enable streaming on the rails config, or send non-streaming requests to that VirtualModel.

### Streaming Chat Completions

=== "curl"

    ```bash
    curl -N -s $NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions \
      -H 'content-type: application/json' \
      -d '{
        "model": "default/guarded-llama",
        "messages": [{"role": "user", "content": "Explain machine learning in simple terms."}],
        "max_tokens": 200,
        "stream": true
      }'
    ```

=== "CLI"

    ```bash
    nemo chat default/guarded-llama "Explain machine learning in simple terms."
    ```

### Blocked Content Detection

When content is blocked during streaming, the stream includes an error chunk:

```json
{
  "error": {
    "message": "Blocked by self check output rails.",
    "type": "guardrails_violation",
    "param": "self check output",
    "code": "content_blocked"
  }
}
```

---

## Guardrails Request Options

You can include a `guardrails` field in the request body to control logging and response format. This field is optional and does not affect which rails are applied — rail selection is determined by the guardrail configuration on the VirtualModel.

### Log Options

The `guardrails.options.log` object controls what diagnostic information is included in the response:

| Field | Type | Description | Default value |
| --- | --- | --- | --- |
| `activated_rails` | `boolean` | Include information about which rails were activated. | `false` |
| `llm_calls` | `boolean` | Include details about LLM calls (prompts, completions, token usage). | `false` |
| `internal_events` | `boolean` | Include the array of internal generated events. | `false` |
| `colang_history` | `boolean` | Include the conversation history in Colang format. | `false` |
| `stats` | `boolean` | Include timing and token statistics. | `false` |

```bash
curl -s $NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "default/guarded-llama",
    "messages": [{"role": "user", "content": "Hello!"}],
    "guardrails": {
      "options": {
        "log": {
          "activated_rails": true,
          "llm_calls": true
        }
      }
    }
  }' | jq '.guardrails_data'
```

### Return as Choice

For clients that do not handle extra response fields, configure the request to return guardrail data as a choice in the `choices` list with the role `guardrails_data`:

```bash
curl -s $NMP_BASE_URL/apis/inference-gateway/v2/workspaces/default/openai/-/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "default/guarded-llama",
    "messages": [{"role": "user", "content": "Hello!"}],
    "guardrails": {
      "return_choice": true,
      "options": {"log": {"activated_rails": true}}
    }
  }'
```

---

## Custom HTTP Headers

The plugin uses two different header-forwarding paths for main models and task models. Understanding the split matters when you want a custom header (for tenancy, observability, etc.) to reach a specific upstream.

### Main Model Calls

The main model is the one IGW resolves from `default_model_entity` and that handles generation. When the plugin builds the per-request main LLM, it forwards a curated allowlist from the inbound request:

- All headers starting with `x-` or `X-` — NeMo Platform principal headers, `x-otel-*`, and any custom `X-Foo` headers your client sets.
- W3C Trace Context headers: `traceparent`, `tracestate`, `baggage`.

`Authorization` and other non-allowlisted headers are intentionally dropped — IGW handles auth.

You can also pin static defaults on a `type: "main"` entry in the guardrail configuration via `parameters.default_headers`. Request-time headers override configuration defaults for the same header name (case-insensitive).

### Task Model Calls

Task models (content safety, topic control, embeddings, etc.) follow a different rule. The cached LangChain client is shared across requests, so arbitrary inbound headers are **not** forwarded to task models. Instead, each task-model call merges in specific service headers derived from the current request context:

- `traceparent` / `tracestate` for distributed tracing.
- `X-NMP-Principal-Id` / `X-NMP-Principal-On-Behalf-Of` for service-principal authorization.

If you need a static header on every call to a specific task model — for example a tenant tag or an upstream API key — declare it in that model's `parameters.default_headers`:

```python
config_data = {
    "models": [
        {
            "type": "content_safety",
            "engine": "nim",
            "model": "default/nvidia-llama-3-1-nemotron-safety-guard-8b-v3",
            "parameters": {
                "default_headers": {
                    "X-Custom-Header": "default-value",
                },
            },
        }
    ],
    # ... prompts and rails ...
}
```
