---
name: diagnose-failures
description: Use when a simready-benchmark run reports failures and you need to understand exactly what failed and re-test only the failed assets. Reads the report, lists the failed asset, feature, requirement, and test, and proposes a scoped re-run.
license: Apache-2.0
compatibility: Requires simready-benchmark (pip install simready-benchmark). Operates on the output directory of a prior run (default _testing/), reading report/test_results_index.json, the per-test result.json files, and state/events.jsonl.
metadata:
  author: NVIDIA
  library: simready_benchmark
  version: "2026.05"
---

## Confirm the Installed Version

This skill ships inside the `simready-benchmark` wheel, so the copy installed with the framework matches the installed code. If you are working from a cached copy or from memory rather than the installed package, confirm the framework version with `simready-benchmark --version`, and treat `simready-benchmark --help` and `simready-benchmark --show-config` as the authoritative source for flags and configuration. A flag documented here might have changed in a different version.

## What This Skill Produces

Given a finished run, this skill produces two things:

1. A complete, precise failure list. For every failed asset it names the failed feature, the requirement code, the test name, and the test's failure message, and it separates honest test failures from engine crashes.
2. A scoped re-test proposal that re-runs only the failed assets, using a real asset path the CLI accepts rather than the report's rel-key.

Work the diagnosis in the order below. Do not skip the rel-key mapping step. Passing a report rel-key straight to `--assets` is the most common mistake and it silently fails to find the asset.

## Locate the Run Output

Everything this skill reads lives under one output directory, by default `_testing/` next to the project config, or whatever was passed to `--output-dir`. The files of interest are:

- `report/test_results_index.json`: the aggregated JSON report. The primary input.
- `results/<asset-dir>/.simready/runtime/<test_name>/result.json`: per-test detail (metrics, Kit logs, crash flag).
- `state/events.jsonl`: the structured event stream. Used to tell test failures apart from runtime errors.
- `plan.json`: the plan. Holds the real asset paths needed to map a rel-key back to a path.
- `index.html`: the human-facing HTML report (same data as the JSON report).

For the exact shapes of each file, refer to [references/report-structure.md](references/report-structure.md).

## Step One Parse the Report Tree

Read `report/test_results_index.json` and parse the nested tree. The shape is:

```
metadata { schema_version, generated, framework_version, run_duration, total_test_time, sessions[] }
assets {
  "<asset-rel-key>": {
    asset_path: "<asset-rel-key>",
    profiles: [
      { id, version, status, features: [
        { id, version, status, tests: [
          { name, version, status, duration, engine, engine_version,
            message, metrics, media, kit_logs, description, expected_video }
        ] }
      ] }
    ],
    stamping { ... }
  }
}
```

Check `metadata.schema_version` first (the current value is `5.0`). If it does not match the shape above, stop and re-read the installed `references/report-structure.md` rather than guessing.

Tests attach directly at feature level; there is no intermediate requirement grouping in the report shape. The rollup rules are:

- Test to feature is fail-dominates. A feature fails if any one of its tests fails (a crashed test is recorded as `fail`, so it counts). A passing test never masks a failing sibling. A feature passes only when at least one test passes and none fail; skipped tests do not force a fail.
- Feature to profile is strict-AND. Any failed feature makes the profile fail. A profile shown as `fail` has at least one failing feature below it.

Still descend to the individual test `status` fields to build the exact failure list: a `fail` feature may have several tests under it, only some of which actually failed. A test is a real failure only when its own `status` is `fail`.

A feature with `"status": "validation_failed"` did not run any tests: its static validation failed. The `failing_requirements` field on that feature lists the requirement ids that failed validation.

## Step Two Build the Failure List

Walk every asset, every profile, every feature, and every test. For each test whose `status` is `fail`, record:

- the asset rel-key (the `assets` map key),
- the feature `id`,
- the test `name`, and
- the test `message`.

The report's `message` is a summary. For the full detail, open the matching per-test file at `results/<asset-dir>/.simready/runtime/<test_name>/result.json`. The `<asset-dir>` is the rel-key with its trailing `<basename>.usd` filename removed. From that file pull:

- `metrics`: the structured numbers the test wrote (for example a measured height or a settle time).
- `kit_logs`: the engine log lines tagged warning, error, or fatal.
- `crash`: present and `true` only on a synthetic crash result.

## Step Three Distinguish a Failure From a Crash

The fix for an honest test failure (the asset behaved wrong) is different from the fix for an engine crash (the run could not complete), so classify each failed test before proposing a remedy.

A synthetic crash result is written by the runner when an engine session dies before the test could finish. It carries `status="fail"`, `crash=true`, and a message that the session crashed with retries exhausted. Treat any failed test whose `result.json` has `crash: true` as a crash, not as an asset defect.

An honest test failure has no `crash` field (or `crash` is absent) and carries a real `message` and `metrics` from the test code describing why the asset did not meet the requirement.

## Step Four Confirm Failures Against the Event Stream

First check the exit code. Exit code `4` is not a test-failure code at all: it means setup was not ready, so the run could not execute part of the plan. Do not chase test failures in this case. Instead read the `engines` block of `report/test_results_index.json` and the `engine_readiness` events in `state/events.jsonl`. Each readiness event carries a `status`, a `severity`, and a `remediation_key` that names what to install or configure. A test that no ready engine could run is recorded with status `blocked` rather than `fail`, so a `blocked` test is an environment problem to fix, not an asset defect.

Exit code 1 conflates two cases: at least one test failed, and the run could not finish cleanly. To tell them apart, parse `state/events.jsonl`:

- `test_complete` events with `status="fail"` are honest test failures.
- `engine_error`, `engine_crash`, `engine_stuck`, and `run_error` events indicate runtime errors the framework could not recover from.

If the failure list is dominated by `engine_crash` or `engine_stuck`, the problem is the engine or the environment, not the assets, and a plain re-run can change the result. If the failures are `test_complete` with `status="fail"` and no crash events, the assets genuinely did not meet the requirements.

## Step Five Map the Rel-Key Back to a Real Asset Path

This is the step agents get wrong. The report keys each asset by a normalized rel-key relative to the run's `content_root`, with forward slashes and the source drive or root stripped, for example `props/cup_01/cup_01.usd`. That rel-key is a lookup key, not a filesystem path. Passing it to `--assets` does not find the asset and the re-run plans nothing.

To re-test a failed asset, recover the real path the CLI accepts. There are two correct ways.

### Option A Map Through the Plan, Then Scope With --assets

`plan.json` lists each asset's real path in `assets[].path`. The reporter derives the rel-key from that path with `asset_rel_key(path, content_root)`. Reverse the mapping: for each failed rel-key, find the plan asset entry whose `path` produces that rel-key, and use the entry's `path` with `--assets`.

```bash
simready-benchmark --project-config /path/to/project_config.toml \
              --assets /real/path/to/cup_01.usd /real/path/to/tray_02.usd
```

This re-plans and re-runs only the named assets. Add `--features` to narrow to the failed feature prefixes when only some features failed.

### Option B Disable the Passing Work, Then Resume

When you want to keep the existing plan and re-run in place, use `--edit-plan` to disable everything that passed, then `--resume`. `--edit-plan --asset PATH` takes the same real asset path that lives in `plan.json` (`assets[].path`), not the rel-key. The plan editor matches on that path. Refer to the [edit-plan](../edit-plan/SKILL.md) skill for the editing mechanics.

```bash
# Disable a passing feature on an asset, then re-run only what remains
simready-benchmark --edit-plan --asset /real/path/to/cup_01.usd --disable-feature FET003
simready-benchmark --resume
```

`--resume` reads `plan.json` from the default output directory (or `--output-dir DIR`) and runs only the tests that resolve to enabled after the override log is applied.

## Hand Off

After the diagnosis is complete:

- To edit the plan (disable passing work, scope to a feature subset, override a config value), refer to the [edit-plan](../edit-plan/SKILL.md) skill.
- To propose concrete fixes for the failed assets, refer to the [remediate-assets](../remediate-assets/SKILL.md) skill.
- For the full report and per-test result.json field reference, refer to [report-structure.md](references/report-structure.md).

## Key Rules

- Read the test `status` fields directly. Tests attach at feature level in the report; iterate `feature["tests"]` to build the failure list. Feature and profile status alone never give a complete failure list.
- Fail-dominates applies from test to feature: any failing test fails the feature. Strict-AND applies from feature to profile: any failing feature fails the profile. A `fail` at the feature or profile level guarantees a failing test below, but a `pass` does not guarantee none.
- A feature with `"status": "validation_failed"` carries a `failing_requirements` list. This is a validation issue, not a benchmark result; no tests ran for that feature.
- A failed test with `crash: true` in its `result.json` is an engine crash, not an asset defect. Classify it separately and fix the engine or environment, not the asset.
- Exit code 1 covers both test failures and runtime errors. Parse `state/events.jsonl` to separate `test_complete status="fail"` from `engine_error`, `engine_crash`, `engine_stuck`, and `run_error`.
- The report rel-key is not a path `--assets` accepts. Map it back to a real path through `plan.json` `assets[].path` before re-testing, or edit the plan by the same real path and `--resume`.

For the exact JSON shapes, rollup rules, event signals, and rel-key mapping, refer to [references/report-structure.md](references/report-structure.md).
