---
name: run-tests
description: Use when you need to validate SimReady USD assets against a profile's benchmark suite. Plans, runs, and reports test cases with the simready-benchmark CLI.
license: Apache-2.0
compatibility: Requires simready-benchmark (pip install simready-benchmark). For Kit-backed tests, also requires simready-benchmark-engine-kit (pip install simready-benchmark[kit]) and a separately-installed Kit / Isaac Sim runtime.
metadata:
  author: NVIDIA
  library: simready_benchmark
  version: "2026.05"
---

## Confirm the Installed Version

This skill ships inside the `simready-benchmark` wheel, so the copy installed with the framework matches the installed code. If you are working from a cached copy or from memory rather than the installed package, confirm the framework version with `simready-benchmark --version`, and treat `simready-benchmark --help` and `simready-benchmark --show-config` as the authoritative source for flags and configuration. A flag documented here might have changed in a different version.

## Run the Full Pipeline (Default Mode)

A single command runs all three phases (plan, run, report) in one pass.

```bash
simready-benchmark --project-config /path/to/project_config.toml
```

`project_config.toml` is the single pointer for the project inputs: asset
content roots, `sr_specs` (specifications), profiles, features, and the
benchmark packs (`[tests].paths`). Point the framework at it once, via
`--project-config`, the `SIMREADY_PROJECT_CONFIG` env var, or
`[paths].project_config` in `engines.toml` -- and all of those resolve. The
engine binary is the only thing `engines.toml` must carry. Without a project
config you can pass paths ad-hoc:

```bash
simready-benchmark --sr-specs /path/to/sr_specs \
              --assets /path/to/asset.usd /path/to/dir_of_assets
```

`--assets` accepts one or more file paths (`.usd` / `.usda` / `.usdc`) and
directories (walked using SimReady packaging-spec rules). Entries can be local
paths or `omniverse://` and `s3://` URLs; the whole pipeline routes them through
the file-access layer. Nucleus (`omniverse://`) paths require `omni.client`,
which ships with Kit and Isaac Sim and not the standalone wheel, so run from
inside Kit or Isaac Sim for Nucleus assets. S3 paths require `boto3`.

## Outputs

Land in `_testing/` next to the project config, or the current directory if no
config is set. Override with `--output-dir DIR`:

- `plan.json`: the immutable plan for this run
- `state/events.jsonl`: JSON-Lines event stream (tail this for live progress)
- `results/<asset-dir>/<asset>.usd`: stamped mirror copy
- `results/<asset-dir>/.simready/validation.json`: `runtime_tests[]` receipt
- `results/<asset-dir>/.simready/runtime/<test_name>/result.json`: per-test artifacts
- `index.html` (plus `page-N.html` for extra pages): aggregated HTML report
- `report/test_results_index.json`: aggregated JSON report
- `logs/<engine>_<n>.log`: per-engine subprocess logs

The original asset directory is never written to.

## Mode Flags (Mutually Exclusive)

```bash
simready-benchmark --project-config ... --plan-only      # writes plan.json only
simready-benchmark --resume                              # runs an existing plan.json
simready-benchmark --report-only                         # re-aggregates the report only
simready-benchmark --list-tests                          # print discovered features and tests
simready-benchmark --edit-plan ...                       # refer to the edit-plan skill
simready-benchmark --show-config                         # diagnostic: print resolved config + exit
simready-benchmark --init-engines-toml [PATH]            # set up engines.toml with the wizard
simready-benchmark --update                              # refresh foundations and Isaac; report framework
simready-benchmark --skills-path                         # print the bundled agent skills directory
```

`--show-config`, `--init-engines-toml`, and `--skills-path` are diagnostic/scaffold modes and need no other inputs.

`--resume` reads `plan.json` from the default `_testing/` (or `--output-dir DIR`).
`--report-only` re-walks `results/<asset>/.simready/runtime/<test>/result.json`
without re-running tests.

## Filter to a Subset of Features or Tests

```bash
simready-benchmark --project-config ... --features FET003           # prefix match
simready-benchmark --project-config ... --tests presence            # exact test name
simready-benchmark --project-config ... --features FET003 --tests light_response  # union
```

`--features` plays a second role: it bypasses the validation gate for matching
prefixes. This is useful when running a feature against an unstamped asset.

## Concurrency

```bash
simready-benchmark --project-config ... --max-concurrent 4
```

Default is 2. The framework caps the practical maximum based on detected GPU memory.

## NVCF Workers (Optional)

```bash
simready-benchmark --project-config ... \
              --workers https://nvcf-fn-1.example,https://nvcf-fn-2.example
```

Routes execution through NVCF cloud functions. Local Kit and NVCF workers
combine by default; pass `--no-local-engines` for NVCF-only.

## Engine Configuration

The setup wizard writes a complete `engines.toml` for you. It acquires `simready_foundations`, detects or installs Isaac Sim, and fills in the engine, `output_dir`, and `project_config` with no placeholders to edit:

```bash
simready-benchmark --init-engines-toml                  # writes ~/.simready-benchmark/engines.toml
simready-benchmark --init-engines-toml ./               # writes ./engines.toml (per-project)
simready-benchmark --init-engines-toml --no-overwrite   # keep an existing file
```

The wizard overwrites an existing file by default (`--no-overwrite` refuses, and a terminal confirms first), prompts before it clones foundations and before the multi-gigabyte Isaac install, and accepts these flags for unattended runs.

| Flag | Effect |
|---|---|
| `--foundations-path DIR` | Use an existing foundations clone instead of cloning. |
| `--foundations-url URL` | Override the clone URL for a forked or moved repo. |
| `--clone-foundations` / `--no-clone-foundations` | Clone foundations, or refuse to. |
| `--install-isaac` / `--no-install-isaac` | Install Isaac with pip when none is found, or refuse to. |
| `--executable-path PATH` | Record a specific Isaac launcher and skip detection. |
| `--project-config PATH` | Point at an existing project config directly. |

To build or repair the file by hand instead, for a second engine, NVCF, or a per-machine override, refer to the configure-engines-toml skill.

The framework reads engine paths from `engines.toml` using a **4-tier lookup** (first match wins):

1. `--engines-toml <FILE>` CLI flag
2. `$SIMREADY_ENGINES_TOML` env var (missing file is a hard error)
3. `engines.toml` in the current working directory
4. `~/.simready-benchmark/engines.toml`

A `[engines.toml] <path>` line goes to stderr one time per process so the user can identify which file is in effect. Suppress with `SIMREADY_QUIET_ENGINES_TOML=1`.

In the common case `engines.toml` carries only the engine; `sr_specs` and
test packs come from `project_config.toml`.

```toml
# engines.toml

[kit.isaac_sim]
executable_path = "C:/Program Files/NVIDIA/IsaacSim/2025.1/isaac-sim.bat"
version = "5.1.0"
tags = ["isaac", "kit"]

[paths]
output_dir = "D:/simready_runs"        # optional; otherwise <CWD>/_testing
```

`engines.toml` can also carry `[paths].sr_specs` and `[tests].paths`, but
those are overrides: set them only when there is no project config, or to
override the project config's value on this machine. Resolution for both
`sr_specs` and tests is first-non-empty-wins in the order CLI flag
(`--sr-specs` / `--tests-path`), then the matching environment variable, then
`engines.toml`, then `project_config.toml`; the project config is the
lowest-priority default. For tests, two sources always merge on top of that
winner: pip-installed entry-point packs, and the bundled FET suite derived
from the project config (its fixed location under `nv_core`), so a foundations
project config discovers the FET suite with no `[tests].paths`.

Verify the resolution with `simready-benchmark --show-config`. It prints which file matched (and which tier), with `[ok]`/`[missing]` markers on each path.

## Keeping the Setup Current

`simready-benchmark --update` refreshes an existing install in one command.

| Component | What `--update` does |
|---|---|
| Foundations | Fast-forwards the `simready_foundations` clone with `git pull --ff-only`. A copy that is not a git checkout (archive, Perforce, or S3) is reported and skipped. |
| Isaac Sim | Upgrades only when a newer version is published, and only after you confirm on a terminal. A non-interactive session reports the available upgrade and skips it rather than starting a multi-gigabyte download. A standalone (non-pip) Isaac is left untouched. |
| Framework | Prints the installed `simready-benchmark` and `simready-benchmark-engine-kit` versions and the exact `pip install --upgrade` command to run in a fresh shell. The running wheel cannot be replaced in place, so this step reports rather than upgrades. |

Skip parts with `--no-update-foundations` or `--no-update-isaac`.

## Pinning Runtime Selection

When more than one `[kit.*]` engine is declared in `engines.toml`, pin a specific runtime with `--runtime`:

```bash
# Use only the isaac_sim engine, ignore everything else in engines.toml
simready-benchmark --project-config ... --runtime isaac_sim

# Use multiple specific engines, for example a multi-runtime CI matrix
simready-benchmark --project-config ... --runtime isaac_sim newton
```

`--runtime` accepts a list (same `nargs="+"` shape as `--assets`). Names not present in `engines.toml` cause a hard error so a typo cannot silently degrade to a different engine. Omitting `--runtime` is the default: use every engine declared in `engines.toml`.

## Driving the Pipeline from Python (Advanced)

The CLI is the recommended entry point. The Python surface is low-level and
takes pre-built discovery / validation-source objects:

```python
from simready_benchmark import Planner, Runner, Reporter

# Planner is a static-method class.
plan = Planner.plan(
    sr_specs_path=str(sr_specs),
    test_dirs=test_dirs,                 # list[str] of dirs containing @test fns
    discovery=discovery,                 # AssetDiscovery instance
    validation_source=validation_source, # ValidationSource instance
)

from simready_benchmark.core.models import save_plan
save_plan(plan, "_testing/plan.json")

# Runner is a static-method class. Returns a RunHandle.
handle = Runner.start(plan, output_dir="_testing", max_concurrent=2, stamp=True)
handle.wait()

# Reporter generates HTML + JSON. generate() is a static method.
Reporter.generate(output_dir="_testing", fmt="both")
```

Building `discovery` and `validation_source` is non-trivial. The CLI does it
in `cli/main.py:_select_discovery()` and `_build_validation_chain()`. For
end-user runs, stay on the CLI.

## Exit Codes

| Code | Meaning |
|------|---------|
| `0`  | All tests passed |
| `1`  | One or more tests failed, or a runtime error |
| `2`  | CLI usage error (mutually exclusive flags, missing required input) |
| `3`  | `--resume` / `--report-only` invoked but no plan.json or results found |
| `4`  | Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). Refer to the Engines section of the report. `4` outranks `1`. |
| `130`| Interrupted by user (Ctrl-C) |

## Interactive In-Isaac Test Runner

To watch a benchmark execute and inspect the resulting scene, run it inside
a GUI Isaac Sim session. Launch the pip-installed `isaacsim` from the same
virtualenv as `simready-benchmark`, then paste this into the Script Editor:

    from simready_benchmark_engine_kit.interactive import open_runner
    open_runner()

A window opens, auto-populated with the discovered tests and the project asset
library. Pick an asset (from the dropdown or through Browse), pick a test, and
click Run Test. The test runs the same execution path as the headless runner,
the viewport animates while physics settles, and the scene is left intact
afterward so you can inspect it. The per-test `result.json` is written to the
configured output directory, in the same location and format the headless
runner uses. To reproduce a specific headless run faithfully, Browse to the
exact asset path recorded in that run's `result.json`.

## Key Rules

- Mode flags (`--plan-only`, `--edit-plan`, `--resume`, `--report-only`, `--list-tests`) are mutually exclusive. Passing more than one is exit code 2.
- The default invocation runs plan + run + report (+ stamp) in one pass; mode flags stop early or resume.
- Stamping happens by default after the run completes (per-asset, based on each asset's results). Pass `--no-stamp` to skip. Refer to the [stamp-results](../stamp-results/SKILL.md) skill.
- Asset listing accepts files (`.usd` / `.usda` / `.usdc`) and directories (walked under packaging-spec rules) interchangeably.
- The original asset directory is never written to. Stamped USDs and result.json files go under `_testing/results/<asset-mirror>/`.

For full CLI flag details refer to [references/cli-flags.md](references/cli-flags.md).
For the Python API surfaces refer to [references/python-api.md](references/python-api.md).
