---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Run an Audit Locally"
description: ""
---

This tutorial walks through running a single audit end-to-end with the NeMo Auditor plugin SDK. You will persist an audit configuration and a target, run the audit in-process, and inspect the resulting report artifacts.

**What you will learn:**

- Initialize the NeMo Platform SDK and reach the `client.auditor` resource.
- Create an `AuditConfig` selecting which garak probes to run.
- Create an `AuditTarget` pointing at a model through the Inference Gateway.
- Execute the audit locally with `client.auditor.run(...)`.
- Read the JSONL, HTML, and hitlog report artifacts the run produces.

<Tip>

This tutorial takes approximately **10 minutes** to complete, plus however long garak takes to run the selected probes against your target.

</Tip>
## Prerequisites

- Install and start NeMo Platform using the [Setup guide](/documentation/get-started).
- Install garak in a Python virtual environment so the plugin can shell out to it. By default the plugin invokes `~/.auditor/.venv/bin/python -m garak`. Override the interpreter path with `NEMO_AUDITOR_GARAK_PYTHON` if your install lives elsewhere.
- Configure at least one Inference Gateway provider — this tutorial uses the `nvidia-inference-api` provider that is included in a default local setup, but any chat-completion-compatible provider works. Step 3 shows how to list the providers registered in your deployment. See [Inference Gateway](/documentation/vulnerability-scanning/targets/inference-gateway) for details on the `nmp_uri_spec` block used below.

---

## Key Concepts

- **`AuditConfig`** — Selects probes, detectors, and reporting settings for a garak run. Persisted in the entity store.
- **`AuditTarget`** — Identifies the model under test (generator class, model identifier, endpoint options). Persisted in the entity store.
- **In-process run** — `client.auditor.run(...)` shells out to garak on the host running the SDK call, writes report artifacts under a temporary directory, and returns their file paths. There is no remote job submission.

---

## 1. Initialize the SDK

Connect to the running platform and reach the auditor sub-resource:

```python
import os

from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
auditor = client.auditor

print(auditor.plugin_status())
# {'plugin': 'auditor', 'status': 'ok', ...}
```

If `plugin_status()` raises a connection error, the platform is not running or `NMP_BASE_URL` is misconfigured. Refer back to [Setup](/documentation/get-started).

---

## 2. Create an Audit Configuration

Build a configuration that runs a small slice of garak's latent-injection probes with three generations per probe:

```python
from nemo_auditor.entities import (
    AuditPluginsData,
    AuditReportData,
    AuditRunData,
    AuditSystemData,
)

config = auditor.configs.create(
    workspace="default",
    name="quick-scan",
    description="Lite latentinjection scan, 3 generations per probe.",
    system=AuditSystemData(lite=True, parallel_attempts=4),
    run=AuditRunData(generations=3),
    plugins=AuditPluginsData(probe_spec="latentinjection", detector_spec="auto"),
    reporting=AuditReportData(report_prefix="quick-scan"),
)
print(config.model_dump_json(indent=2))
```

For the full set of options on each sub-block, see [Configuration Schema](/documentation/vulnerability-scanning/configurations/schema). For probe selection syntax, see [Selecting Probes](/documentation/vulnerability-scanning/configurations/selecting-probes).

---

## 3. Create an Audit Target

First, list the providers registered in your deployment so you can pick one that exists:

```python
providers = client.inference.providers.list(workspace="default")
for provider in providers["data"]:
    print(provider["name"], provider.get("host_url"))
```

Use a `provider` name from this output in the target below. The model identifier format depends on the provider — check the provider's API documentation or served-model list if unsure.

Define the model under test. This example uses garak's `nim.NVOpenAIChat` generator pointed at the `nvidia-inference-api` provider registered in a default local setup:

```python
target = auditor.targets.create(
    workspace="default",
    name="llama-31-8b",
    type="nim.NVOpenAIChat",
    model="nvidia/meta/llama-3.1-8b-instruct",
    options={
        "nim": {
            "max_tokens": 1024,
            "nmp_uri_spec": {
                "inference_gateway": {
                    "workspace": "default",
                    "provider": "nvidia-inference-api",
                },
            },
        },
    },
)
print(target.model_dump_json(indent=2))
```

The `nmp_uri_spec` sentinel inside `options.nim` tells the plugin to resolve a concrete URI from the Inference Gateway provider at run time. See [Inference Gateway](/documentation/vulnerability-scanning/targets/inference-gateway) for the full conflict rules and resolution behavior.

---

## 4. Run the Audit Locally

`run()` accepts either inline entities or name strings that reference entities in the entity store. Pass the names you persisted above:

```python
result = auditor.run(
    config="quick-scan",
    target="llama-31-8b",
    workspace="default",
)

print(result["status"], result["returncode"])
# completed 0

for name, ref in result["results"].items():
    print(f"{name}: {ref['artifact_url']}")
# report-jsonl: file:///var/folders/.../results/report-jsonl
# report-html: file:///var/folders/.../results/report-html
# report-hitlog-jsonl: file:///var/folders/.../results/report-hitlog-jsonl
```

If you prefer to skip the entity-store roundtrip, pass the inline `AuditConfig` and `AuditTarget` objects from the previous two steps directly:

```python
result = auditor.run(config=config, target=target, workspace="default")
```

---

## 5. Read the Results

`run()` returns `file://` URLs for whichever of the three report types garak produced. Parse them with `urllib.parse.urlparse` to recover local paths, then load them as needed.

Load the JSONL probe-by-probe summary:

```python
import json
from pathlib import Path
from urllib.parse import urlparse

def url_to_path(url: str) -> Path:
    return Path(urlparse(url).path)

jsonl_path = url_to_path(result["results"]["report-jsonl"]["artifact_url"])
records = [json.loads(line) for line in jsonl_path.read_text().splitlines() if line.strip()]
print(f"Loaded {len(records)} report entries.")
```

Summarize hits per probe from the hitlog:

```python
from collections import Counter

hitlog_ref = result["results"].get("report-hitlog-jsonl")
if hitlog_ref is None:
    print("No hits recorded — every probe passed.")
else:
    hitlog_path = url_to_path(hitlog_ref["artifact_url"])
    hits = [json.loads(line) for line in hitlog_path.read_text().splitlines() if line.strip()]
    by_probe = Counter(entry.get("probe") for entry in hits)
    for probe, count in by_probe.most_common():
        print(f"{probe}: {count} hits")
```

The HTML report at `result["results"]["report-html"]["artifact_url"]` is the most human-friendly summary — open it in a browser to see grouped pass/fail counts and per-probe details.

---

## 6. Clean Up

Delete the entities you created:

```python
auditor.configs.delete(workspace="default", name="quick-scan")
auditor.targets.delete(workspace="default", name="llama-31-8b")
```

Report artifacts are left in place under the scheduler's temporary directory; the OS reaps them eventually.

---

## Troubleshooting

**`FileNotFoundError: garak interpreter not found at ...`**
: The plugin couldn't find a garak install. Either install garak at `~/.auditor/.venv/bin/python`, or set `NEMO_AUDITOR_GARAK_PYTHON` to the absolute path of a Python interpreter that has garak installed.

**`RuntimeError: Failed to resolve inference gateway provider '<workspace>/<provider>'`**
: The `nmp_uri_spec` block in your target's options references an Inference Gateway provider that doesn't exist. List your providers with `client.inference.providers.list(workspace="default")` and update the target to reference an existing one.

**`returncode != 0` with empty `results`**
: garak started but failed early, usually because the target endpoint is unreachable. Inspect `result["stderr_tail"]` for the error message, and verify the model is reachable through the provider you configured.

**Where do the report files live?**
: The scheduler writes garak output under `<temp-dir>/garak/<reporting.report_dir>/<reporting.report_prefix>.*`, then copies the produced files into the local results directory referenced by each `artifact_url`. For a local run this is a temporary directory under the system's `$TMPDIR`.

---

## Summary and Next Steps

You created an `AuditConfig` and `AuditTarget`, ran a single audit locally with `client.auditor.run(...)`, and loaded the resulting reports.

- For the full SDK surface — including async variants — see [SDK Resources](/documentation/vulnerability-scanning/sdk-resources).
- For more probe selection options, see [Selecting Probes](/documentation/vulnerability-scanning/configurations/selecting-probes).
- For other generator types (OpenAI-compatible endpoints, REST targets, test targets), see [Audit Targets](/documentation/vulnerability-scanning/targets).
