# Report Structure Reference

This page gives the exact shapes of the files the diagnose-failures workflow reads, the rollup rules the reporter applies, the event signals that separate test failures from runtime errors, the rel-key-to-path mapping, and the output-tree paths. Every shape here is verified against the installed source.

## Output Tree Paths

Every path resolves through `simready_benchmark.core.output_layout`. The default output directory is `_testing/` next to the project config, overridable with `--output-dir DIR`.

```
<output_dir>/
  plan.json                                                  # the plan (real asset paths)
  run_summary.json                                           # final run summary
  index.html                                                 # HTML report, page 1
  page-2.html                                                # HTML report, additional pages
  state/
    events.jsonl                                             # structured event stream
    work_pool.json                                           # live work-item database
  logs/
    orchestrator.log                                         # orchestrator text log
    <engine>_<N>.log                                         # per-session text log
  report/
    test_results_index.json                                 # aggregated JSON report
  results/
    <asset-dir>/
      <asset>.usd                                            # stamped mirror copy
      .simready/
        validation.json                                      # benchmark receipt
        runtime/<test_name>/
          result.json                                        # per-test result
          *.mp4, *.png                                       # captured media
```

The report is at `<output_dir>/report/test_results_index.json`. The HTML report is at `<output_dir>/index.html`, not under `report/`. Per-test results are at `<output_dir>/results/<asset-dir>/.simready/runtime/<test_name>/result.json`. The event stream is at `<output_dir>/state/events.jsonl`.

## test_results_index.json

Top-level shape, with `schema_version` currently `5.0`:

```json
{
  "metadata": {
    "schema_version": "5.0",
    "generated": "2026-05-28T14:00:00Z",
    "framework_version": "2026.5.1",
    "run_duration": 412.3,
    "total_test_time": 1187.4,
    "sessions": [
      {
        "id": "isaac_sim_1",
        "engine": "isaac_sim",
        "log_file": "logs/isaac_sim_1.log",
        "duration": 391.5,
        "tests_run": 14,
        "test_time": 376.2
      }
    ]
  },
  "assets": {
    "<asset-rel-key>": {
      "asset_path": "<asset-rel-key>",
      "profiles": [
        {
          "id": "Prop-Robotics-Neutral",
          "version": "1.0.0",
          "status": "pass",
          "features": [
            {
              "id": "FET003_BASE_NEUTRAL",
              "version": "0.1.0",
              "status": "pass",
              "tests": [
                {
                  "name": "rigid_body_falls_isaac",
                  "version": "1.0",
                  "status": "pass",
                  "duration": 12.4,
                  "engine": "isaac_sim",
                  "engine_version": "5.0.0",
                  "message": "",
                  "metrics": {"final_height_m": 0.001},
                  "media": [{"kind": "video", "role": "summary", "filename": "fall.mp4"}],
                  "kit_logs": [],
                  "description": "Asset settles within tolerance after free-fall.",
                  "expected_video": ""
                }
              ]
            }
          ]
        }
      ],
      "stamping": { "...": "..." }
    }
  }
}
```

### Field Notes

- `assets` is a map keyed by the asset rel-key. The same rel-key is repeated in `asset_path`.
- Each asset carries one profile entry. The profile `id` and `version` come from the asset's stamped validation record in the plan. An unstamped or override-only asset shows `id` as `<unstamped>` with an empty version.
- Each feature carries a flat `tests` list. Tests attach directly at feature level; there is no intermediate requirement grouping in the report shape.
- A test entry always carries these keys: `name`, `version`, `status`, `duration`, `engine`, `engine_version`, `message`, `metrics`, `media`, `kit_logs`, `description`, `expected_video`. The per-test `metrics` and `kit_logs` are mirrored into the report from the per-test `result.json`.
- Tests sort failed-first within a feature, features sort failed-first within the profile, and assets sort failed-first in the map.
- Status values everywhere are `pass`, `fail`, `skipped`, `incomplete`, or `blocked`. A `blocked` test is one that no ready engine could run (a setup or environment problem), distinct from a `fail`. It comes from the engine-readiness preflight, not from test code.
- A feature whose static validation failed carries `"status": "validation_failed"` and a flat `"failing_requirements"` list of requirement ids that failed validation. This is a validation issue, not a benchmark failure; no tests ran for that feature.

## Rollup Rules

### Test to Feature Is Fail-Dominates

A feature's status is computed across its tests directly with fail-dominates semantics:

- If any test is `fail`, the feature is `fail` (a crashed test is recorded as `fail`, so it counts).
- Otherwise, if any test is `incomplete`, the feature is `incomplete`.
- Otherwise, if all tests are `skipped`, the feature is `skipped`.
- Otherwise (at least one `pass`, rest pass or skipped), the feature is `pass`.

A failing test is never masked by a passing sibling: any failure fails the feature. `passed` is `true` on a feature only when its status is `pass`.

### Feature to Profile Is Strict-AND

- If any feature is `fail`, the profile is `fail`.
- Otherwise, if any feature is `incomplete`, the profile is `incomplete`.
- Otherwise, if all features are `skipped`, the profile is `skipped`.
- Otherwise (at least one `pass`, rest pass or skipped), the profile is `pass`.

Mixed pass plus skipped rolls up to `pass`; skipped features do not block passing. A profile requires all of its features to be met.

## Per-Test result.json

Written for every executed test at `<output_dir>/results/<asset-dir>/.simready/runtime/<test_name>/result.json`:

```json
{
  "asset": "props/cup_01/cup_01.usd",
  "test_name": "rigid_body_falls_isaac",
  "status": "pass",
  "duration": 12.4,
  "message": "",
  "metrics": {"final_height_m": 0.001, "settled_after_s": 1.8},
  "media": [
    {"kind": "video", "role": "summary", "filename": "fall.mp4"},
    {"kind": "image", "role": "summary", "filename": "frame_final.png"}
  ],
  "kit_logs": [],
  "engine": "isaac_sim",
  "engine_version": "5.0.0",
  "test_version": "1.0"
}
```

- `asset` is the normalized rel-key (the same key used in the report's `assets` map).
- `status` is normalized by the reporter: an `error` status becomes `fail`, and a `completed` status without an explicit pass or fail becomes `incomplete`. Tests should write `pass`, `fail`, `skipped`, or `incomplete` directly.
- `message`, `metrics`, and `kit_logs` carry the detail behind the report's summary. Pull them from this file for a full diagnosis.

### Synthetic Crash Result

If an engine session dies before a test could finish, the runner writes a synthetic `result.json` for each affected test so the test does not silently disappear from the report:

```json
{
  "status": "fail",
  "duration": 0,
  "message": "Engine session isaac_sim_1 crashed before the test could complete (retries exhausted).",
  "test_name": "rigid_body_falls_isaac",
  "asset": "props/cup_01/cup_01.usd",
  "session_id": "isaac_sim_1",
  "engine": "isaac_sim",
  "crash": true
}
```

The `crash: true` flag is the signal. A failed test with `crash: true` is an engine crash, not an asset defect. An honest test failure has no `crash` field and carries a real `message` and `metrics` from the test code.

## events.jsonl Failure Versus Crash Signals

`state/events.jsonl` is one JSON object per line. Every event carries at least `type` and `timestamp`. Use it to separate the two cases that exit code 1 conflates.

Honest test failures:

- `test_complete` with `"status": "fail"`. Example:

```json
{"type": "test_complete", "timestamp": "2026-05-28T14:02:33.412Z", "session_id": "isaac_sim_1", "asset": "props/cup_01/cup_01.usd", "test": "rigid_body_falls_isaac", "status": "fail", "duration": 12.4}
```

Runtime errors the framework could not recover from:

- `engine_error`. A session failed to start or reported a structured fatal error.
- `engine_crash`. A session process died and its claimed tests were reclaimed.
- `engine_stuck`. A session reported ready but emitted no `test_start` within the no-progress timeout and was killed.
- `run_error`. The run aborted on an unrecoverable error, for example repeated spawn failures or a global timeout.

Setup not ready (the run exits `4`):

- `engine_readiness`. Emitted during preflight, once per configured engine. Carries `engine`, `kind`, `severity` (`info`, `warning`, or `error`), `status` (for example `plugin_missing`, `missing_executable`, `unreachable`, `worker_warming`, or `driver_incompatible`), `detail`, and `remediation_key`. An `error`-severity event, or a required capability with no configured engine, makes the run `not_ready` and exits `4`. The tests that could not run are written with status `blocked`.

A run dominated by `engine_crash` or `engine_stuck` points at the engine or the environment, and a plain re-run can change the result. A run whose failures are all `test_complete` with `status="fail"` and no crash or error events means the assets genuinely did not meet the requirements.

## Rel-Key to Path Mapping

The report and the per-test `result.json` files key assets by a normalized rel-key, not by the path the user passed on the command line. The reporter computes the key with `asset_rel_key(asset.path, content_root)`:

- When the asset path is under the run's `content_root`, the rel-key is the path relative to `content_root`, with backslashes converted to forward slashes. For example, `content_root` is `D:/assets` and the asset is `D:/assets/props/cup_01/cup_01.usd`, so the rel-key is `props/cup_01/cup_01.usd`.
- When there is no `content_root`, or the path is on a different drive, the key is the asset path with its drive and leading slash stripped, again with forward slashes.

The rel-key always includes the `<basename>.usd` filename at the end. The mirror `<asset-dir>` is the rel-key with that filename removed (`props/cup_01` in the example).

To re-test, recover the real path:

- `plan.json` lists each asset's real path in `assets[].path`, and the run's `content_root` is at the top level of `plan.json`. For a failed rel-key, find the plan asset entry whose `path` produces that rel-key under `asset_rel_key(path, content_root)`, and use that `path`.
- Pass the recovered `path` to `--assets` to re-plan and re-run only those assets, or pass it to `--edit-plan --asset PATH` (the plan editor matches on the same `assets[].path` value) and then `--resume`.

Passing the rel-key itself to `--assets` does not find the asset. The re-run plans nothing because the rel-key is not a filesystem path.
