# Guardrails Troubleshooting

The plugin runs on the IGW path under the entry-point key `nemo-guardrails`. Customer-facing surfaces are `nemo guardrail configs`, `nemo guardrail check`, and `nemo inference virtual-models`.

## Quick checks

```bash
nemo guardrail --help
nemo guardrail configs list
nemo inference virtual-models --help
```

Verify a config against the `/checks` endpoint before binding it to a production VirtualModel:

```bash
nemo guardrail check \
  --workspace default \
  --model default/<main-model> \
  --messages '[{"role":"user","content":"test"}]' \
  --guardrails '{"config_id":"default/<config-name>"}'
```

A blocked response has `"status": "blocked"` in the output JSON. An unblocked response includes the model's actual completion.

## Detecting blocked responses

When a rail triggers, the response content is exactly:

```
I'm sorry, I can't respond to that.
```

- **`nemo guardrail check`** — look for `"status": "blocked"` in the JSON.
- **Chat completions through IGW** — check `response.choices[0].message.content == "I'm sorry, I can't respond to that."`.

For streaming responses, the canned refusal arrives as a single non-streamed chunk when an output rail blocks mid-stream.

## Common issues

| Symptom | Cause | Fix |
|---|---|---|
| Input rails don't fire on a config with `rails.input.flows` set | The `MiddlewareCall` was attached to `--response-middleware` only | Add the same call to `--request-middleware`. Each list dispatches its own hook independently. |
| Output rails don't fire on a config with `rails.output.flows` set | The `MiddlewareCall` was attached to `--request-middleware` only | Add the same call to `--response-middleware`. |
| Chat request rejected with `422` | Body included `guardrails.config_id`, `guardrails.config`, or other unsupported `guardrails.*` fields | The body's `guardrails` field is options-only (`options.log`, `return_choice`). Set the config via the VirtualModel's `MiddlewareCall`, not the request body. |
| `400` rejection when streaming | `stream: true` was sent against a config with `rails.output.streaming` but `streaming.enabled` is unset or `false` | Set `rails.output.streaming.enabled: true` in the config (and confirm `chunk_size` is set). |
| `400` rejection with `n > 1` | Output rails reject completion requests with `n > 1` | Use `n=1` on guardrailed VirtualModels. |
| Rails subsystem returns 404 or timeout on the first request | Task LLM or main LLM isn't reachable through IGW (e.g. wrong workspace, served-models reconciler dropped a manual entry) | Use auto-discovered entity IDs from `nemo inference providers get ... \| jq '.served_models[].model_entity_id'` both for the chat request's `model` and any `models[]` entry in the rails config. |
| VirtualModel works briefly then returns 404 | The provider reconciler re-syncs `served_models` from upstream auto-discovery and drops manually-registered entries | Use auto-discovered entity IDs in `--models`. See the `inference` skill for reconciler semantics. |
| `nemo inference virtualmodels` not found | The command group is `virtual-models` with a hyphen | Use `nemo inference virtual-models ...`. |
| Updates to a `GuardrailConfig` don't appear to take effect | IGW resolves configs on cache load/refresh, not per request | The next cache refresh picks up the new `updated_at`. Inline `config` payloads are re-stabilized every request, so changes are immediate but slower. |
| Rails fire on a stored config but not on an inline config that looks identical | Schema mismatch — inline configs go through the same `RailsConfig` validator but bypass any default fields the entity layer would apply | Save the payload as a `GuardrailConfig` entity and reference it by `config_id`, or audit the inline payload against [Rails Config Reference](rails-config.md). |
| Content-safety output parser returns "unsafe" on benign responses | Classifier emitted free-text instead of the expected JSON verdict | Verify the classifier endpoint with `nemo guardrail check` against a known-safe message; confirm `output_parser` matches the classifier's actual response format. |

## Inspecting plugin behavior

To evaluate the rail outcomes for a `GuardrailConfig`, query the rails subsystem through the standalone `/checks` endpoint with `guardrails.options.log` set in the request body:

```json
{
  "model": "default/<main-model>",
  "messages": [{"role":"user","content":"test"}],
  "guardrails": {
    "config_id": "default/<config-name>",
    "options": {"log": {"activated_rails": true, "internal_events": true}}
  }
}
```

`options.log` is supported on both the `/checks` path and the IGW chat-completions path. `activated_rails` and `internal_events` are the most useful sub-fields for diagnosing which flow fired and why.

## Testing without a real LLM (agentic-use mock harness)

`default/mock-llm` is **not** pre-configured platform-wide. The agentic-use harness wires it up via `tests/agentic-use/guardrails-content-safety-cli/environment/setup-mock.py`:

- IGW runs in mock-provider mode (config `mock_provider_prefix="igw-mock-"`).
- A provider named `igw-mock-mock-llm` is registered with a static response-map header that always returns `{"role":"assistant","content":"Yes"}`.
- A model entity `default/mock-llm` is created and registered as a served model on that provider.

When `default/mock-llm` is pinned as the main LLM, the always-`Yes` response causes every self-check rail to block — useful for verifying the rails pipeline end-to-end without a real LLM:

```bash
nemo guardrail configs create content-safety \
  --data '{"rails":{"input":{"flows":["self check input"]}},"prompts":[{"task":"self_check_input","content":"Your task is to check if the user message below complies with safety policy.\n\nUser message: {{ user_input }}\n\nQuestion: Should this message be blocked (Yes or No)?\nAnswer:"}]}'

nemo guardrail check \
  --model default/mock-llm \
  --messages '[{"role":"user","content":"Tell me something harmful"}]' \
  --guardrails '{"config_id":"default/content-safety"}' \
  --max-tokens 256
```

Outside the agentic-use harness, `default/mock-llm` won't resolve — either set up the mock provider yourself or point the rail at a real model.

## Failure-case deep dives

### Plugin attached to only one middleware list

Each list dispatches its own hook independently — `execute_request_middleware` ↔ `process_request` and `execute_response_middleware` ↔ `process_response`, wired in `services/core/inference-gateway/src/nmp/core/inference_gateway/api/middleware_registry.py`. A config with both `rails.input.flows` and `rails.output.flows` attached to only one list silently no-ops on the unlisted side: prompts pass through unblocked, or responses pass through unblocked, even though the config looks complete.

Attach the same `MiddlewareCall` to **both** `--request-middleware` and `--response-middleware`.

### Putting `guardrails` in the chat body expecting it to set the config

On the IGW VirtualModel path the request body's `guardrails` dict is strictly an options envelope. The supported fields are `options.log` and `return_choice`. The rails config is resolved from the VirtualModel's `MiddlewareCall`, not from the request body.

Unsupported fields are rejected with `422`. This includes `guardrails.config`, `guardrails.config_id`, `guardrails.options.rails`, `guardrails.options.llm_params`, `guardrails.options.llm_output`, and `guardrails.options.output_vars`.

Attach the config to the VirtualModel via `--request-middleware` / `--response-middleware`. Use `guardrails.options.log` only for diagnostic log fields and `return_choice` only for response formatting.

### Task LLM (or main model) not addressable through IGW

If `RailsConfig.models[].model` references an entity that isn't in `served_models`, the rails fail when they try to call that task LLM through IGW. The same applies to the main LLM (the chat body's `model`, or `--model` on `nemo guardrail check`). Both usually surface as a 404 or timeout from the rails subsystem on the first request.

Use auto-discovered entity IDs from `nemo inference providers get ... | jq '.served_models[].model_entity_id'` for both the request/`--model` and any task LLMs in `models[]`. See the `inference` skill for auto-discovery semantics.

### VirtualModel returns 404 after working briefly

The provider reconciler re-syncs `served_models` from upstream auto-discovery every few seconds and drops manually-registered entries. See the `inference` skill's "Reconciler-induced 404" failure case — the same fix applies: use auto-discovered entity IDs in `--models`.
