# CLI Flag Reference for `simready-benchmark`

Authoritative source: `cli/main.py:build_parser()`. This is a curated summary.

## Mode Flags (Mutually Exclusive)

| Flag | Behavior |
|---|---|
| (default) | Plan + run + report in one pass |
| `--plan-only` | Generate plan.json and stop |
| `--edit-plan` | Edit an existing plan.json (no run, no report). Refer to the [edit-plan](../edit-plan/SKILL.md) skill. |
| `--resume` | Run + report against an existing plan.json |
| `--report-only` | Regenerate the report from existing per-test result.json files |
| `--list-tests` | Print the registered features and tests; no plan generated |
| `--show-config` | Print the resolved configuration (engines.toml location and tier, [kit.*] engines, [paths].sr_specs, [tests].paths with existence markers, sr_specs source, project_config, discoverable test packs) and exit. Diagnostic-only, needs no other inputs. |
| `--init-engines-toml [PATH]` | Run the setup wizard: acquire `simready_foundations`, detect or install Isaac Sim, and write a complete `engines.toml` (engine, `output_dir`, and `project_config`). Default destination `~/.simready-benchmark/engines.toml`; PATH can be a file or directory. Overwrites an existing file by default. Refer to the setup-wizard options below. |
| `--update` | Refresh an existing install: fast-forward the foundations clone, upgrade Isaac only when a newer version exists (confirmed on a terminal, skipped in a non-interactive session), and report the framework upgrade command. Skip parts with `--no-update-foundations` or `--no-update-isaac`. |
| `--skills-path` | Print the absolute path to the bundled agent skills directory and exit. |

## Setup-Wizard Options (with `--init-engines-toml`)

| Flag | Description |
|---|---|
| `--foundations-path <DIR>` | Use an existing `simready_foundations` clone instead of cloning. |
| `--foundations-url <URL>` | Override the clone URL for a forked or moved repo. |
| `--clone-foundations` / `--no-clone-foundations` | Clone foundations, or refuse to. Unset prompts on a terminal. |
| `--install-isaac` / `--no-install-isaac` | Install the latest Isaac with pip when none is found, or refuse to. Unset prompts on a terminal. |
| `--overwrite` / `--no-overwrite` | Overwrite an existing `engines.toml`. Default overwrites; unset prompts on a terminal. |
| `--executable-path <PATH>` | Record a specific Isaac launcher and skip detection and install. |
| `--project-config <PATH>` | Point the wizard at an existing project config directly, skipping foundations acquisition. |

## Subcommands

| Subcommand | Behavior |
|---|---|
| `simready-benchmark stamp --output-dir <_testing>` | Stamp benchmark metadata into mirror USDs and validation.json. Refer to the [stamp-results](../stamp-results/SKILL.md) skill. |
| `simready-benchmark stamp-one ...` | Stamp a single asset's benchmark metadata (advanced). |

## Discovery

| Flag | Description |
|---|---|
| `--project-config <PATH>` | Path to `project_config.toml` (carries content roots, `sr_specs`, profiles, features, and `[tests].paths`). |
| `--sr-specs <PATH>` | Path to the sr_specs directory (overrides project config and engines.toml). |
| `--assets <PATH> [<PATH>...]` | One or more `.usd` / `.usda` / `.usdc` files, or directories walked under packaging-spec rules. Entries can be local paths or `omniverse://` / `s3://` URLs; Nucleus paths need `omni.client` (ships with Kit / Isaac Sim, not the standalone wheel) and S3 paths need `boto3`. |
| `--tests-dir <PATH>` | Single extra test-definition directory. |
| `--tests-path <PATH> [<PATH>...]` | One or more test-pack source directories (parent goes onto `sys.path`; module walked for `@test`). Highest-priority test-path source: overrides `SIMREADY_BENCHMARK_TESTS_PATH`, `engines.toml [tests].paths`, and `project_config.toml [tests].paths`. |
| `--use-asset-search` | Use `simready_search`'s AssetLibrary for asset discovery + as a validation source. Requires `--project-config`. |

## Filters

| Flag | Description |
|---|---|
| `--features <ID> [<ID>...]` | Match feature ID prefixes, for example `FET003`. Also bypasses the validation gate for matching features. |
| `--tests <NAME> [<NAME>...]` | Exact test names to keep. |

## Run Options

| Flag | Description |
|---|---|
| `--output <FILE>` | Plan output JSON path (default `{project_root}/_testing/plan.json`). |
| `--output-dir <DIR>` | Output directory for run + report (default `{project_root}/_testing`). |
| `--max-concurrent <N>` | Up to N engine sessions in parallel (default 2). |
| `--session-mode {batch,per_asset}` | Force per-session work shape. |
| `--engine-path <PATH>` | Direct engine binary override (debug). |
| `--engines-toml <FILE>` | Override the `engines.toml` lookup chain. |
| `--workers <url1,url2,...>` | Comma-separated NVCF worker URLs; routes execution through `NvcfPlugin`. |
| `--no-local-engines` | Skip local engine discovery; use ONLY the engines explicitly constructed by other flags (currently: `--workers`). |
| `--runtime <NAME> [<NAME>...]` | Pin execution to one or more named engines from `engines.toml`, for example `--runtime isaac_sim` or `--runtime isaac_sim newton` for multi-runtime runs. Names not present in `engines.toml` cause a **hard error** so typos do not silently degrade to a different engine. Omit the flag to use every engine declared in `engines.toml`. Same `nargs="+"` shape as `--assets`. |
| `--no-stamp` | Disable post-run stamping. |
| `--keep-outputs` | Do not clear `_testing/` before planning. |
| `--format {html,json,both}` | Report format (default `both`). |
| `--publish-to <DEST>` | Publish completed `_testing/` to a destination after the run. |

## Edit-plan Flags

Used with `--edit-plan`. Refer to the [edit-plan](../edit-plan/SKILL.md) skill for the full workflow.

| Flag | Description |
|---|---|
| `--asset <PATH>` / `--all-assets` | Edit scope (mutually exclusive). |
| `--disable-profile <NAME>` / `--enable-profile <NAME>` / `--force-profile <NAME>` | Profile-level edits. |
| `--disable-feature <NAME>` / `--enable-feature <NAME>` | Feature-level edits. Exact name or `FETxxx` prefix. |
| `--disable-test <NAME>` / `--enable-test <NAME>` | Test-level edits. |
| `--enable-all` | Remove all overrides for the targeted asset(s). |
| `--only-features A,B,...` | Disable all features except matches. |
| `--set-config TEST:K=V` | Append a config override for a test. |

## Verbosity

| Flag | Description |
|---|---|
| `--version` | Print framework version and exit. |

## `engines.toml` Lookup Order (First Match Wins)

| # | Source | Notes |
|---|---|---|
| 1 | `--engines-toml <FILE>` | CLI flag |
| 2 | `$SIMREADY_ENGINES_TOML` | Env var. Pointing at a missing file is a **hard error**; explicit config should never silently fall through. |
| 3 | `./engines.toml` | In the current working directory |
| 4 | `~/.simready-benchmark/engines.toml` | Per-user default |

When `engines.toml` is found, the framework emits a single `[engines.toml] <path>` line to stderr (one time per process). Set `SIMREADY_QUIET_ENGINES_TOML=1` to suppress.

## `[paths].sr_specs` in engines.toml

`sr_specs` is normally supplied by `project_config.toml`. `[paths].sr_specs` in `engines.toml` is an override, used when there is no project config or when this machine needs a different value. The resolution order for `sr_specs` is first-non-empty-wins:

| # | Source |
|---|---|
| 1 | `--sr-specs PATH` (CLI flag) |
| 2 | `$SIMREADY_SR_SPECS` (env var) |
| 3 | `engines.toml [paths].sr_specs` |
| 4 | `project_config` (sr_specs root derived from the `[validate]` paths) |
| 5 | repo-relative + `./sr_specs` fallbacks |

For test packs the explicit ordering is `--tests-path`, then `$SIMREADY_BENCHMARK_TESTS_PATH`, then `engines.toml [tests].paths`, then `project_config.toml [tests].paths` (first non-empty wins among those, project_config lowest). On top of the winner, two sources always merge in: pip-installed entry-point packs, and the bundled FET suite derived from the project config (its fixed location under `nv_core`). So a foundations project config discovers the FET suite with no `[tests].paths`, and explicit entries add packs rather than replacing it. Relative `engines.toml` paths resolve against the toml's own directory, not the shell CWD.

## Environment Variables

| Variable | Purpose |
|---|---|
| `SIMREADY_BENCHMARK_TESTS_PATH` | Same as `--tests-path` but colon-separated (`:`) on Linux/macOS, semicolon-separated (`;`) on Windows. |
| `SIMREADY_ENGINES_TOML` | Override the `engines.toml` lookup (tier 2). Missing-file is a hard error. |
| `SIMREADY_SR_SPECS` | Override the `sr_specs` directory lookup. |
| `SIMREADY_PROJECT_CONFIG` | Override the `project_config.toml` lookup. |
| `SIMREADY_QUIET_ENGINES_TOML=1` | Suppress the `[engines.toml] <path>` stderr banner. |

## Exit Codes

| Code | Meaning |
|------|---------|
| `0` | All tests passed |
| `1` | One or more tests failed, or a runtime error |
| `2` | CLI usage error (mutually exclusive flags, missing required input) |
| `3` | `--resume` / `--report-only` invoked but no plan.json or results found |
| `4` | Setup or environment not ready (no engine could run part of the plan, or an engine failed preflight). Refer to the Engines section of the report. `4` outranks `1`. |
| `130` | Interrupted by user (Ctrl-C / SIGINT) |
