# Python API for Planner, Runner, and Reporter

These classes back the `simready-benchmark` CLI. End-user runs should use the CLI;
this surface is for advanced integrations, such as driving the framework from a
larger script or embedding it in a service. Both `Planner` and `Runner` expose
only static methods.

## `simready_benchmark.Planner`

```python
class Planner:
    @staticmethod
    def discover(sr_specs_path: str, test_dirs: list[str]) -> dict:
        """Return the full capability tree (profiles > features > tests)."""

    @staticmethod
    def plan(
        sr_specs_path:     str,
        test_dirs:         list[str],
        discovery:         AssetDiscovery,          # from simready_benchmark.core.asset_discovery
        validation_source: ValidationSource,        # from simready_benchmark.planner.validation_sources
        feature_overrides: list[str] | None = None,
    ) -> Plan: ...
```

`test_dirs` is a list of directories containing `@test`-decorated functions.
Use `simready_benchmark.core.discovery.discover_test_dirs(extra_paths=...)` to build
this list (it combines entry-point-registered packs with extra paths).

## `simready_benchmark.core.models` Plan I/O

```python
from simready_benchmark.core.models import load_plan, save_plan

plan = load_plan("path/to/plan.json")      # returns a Plan dataclass
save_plan(plan, "path/to/plan.json")
```

The `Plan` dataclass holds `assets`, `features`, `profiles`, `tests`, and
`overrides` lists keyed by integer IDs. Refer to `core/models.py` for the field set.

## `simready_benchmark.Runner`

```python
class Runner:
    @staticmethod
    def start(
        plan:            Plan,
        output_dir:      str,
        engine_plugins:  dict[str, EnginePlugin] | None = None,
        max_concurrent:  int  = 2,
        stamp:           bool = False,
        session_mode:    str | None = None,                 # "batch" | "per_asset"
        **kwargs,
    ) -> RunHandle:
        """Start execution in a background thread; returns the handle immediately."""
```

When `engine_plugins` is `None`, the runner calls
`simready_benchmark.engines.registry.discover_engine_plugins()` to load every plugin
registered under the `simready_benchmark.engines` entry-point group.

### Useful Kwargs Absorbed through `**kwargs`

| Kwarg | Purpose |
|---|---|
| `runtime=["isaac_sim", ...]` | Pin execution to one or more named engines from `engines.toml`. Same semantics as the CLI's `--runtime NAME [NAME ...]` flag: unknown names hard-fail with an `engine_error` event and a `status="failed"` summary, listing the available engines. Omit to use every discovered engine. |
| `global_timeout=86400` | Watchdog timeout in seconds. After `global_timeout + 300` (5-minute grace), the orchestration force-exits with `os._exit(1)`. |
| `engines_toml=<path>` | Override the 4-tier `engines.toml` lookup. Equivalent to the CLI's `--engines-toml FILE`. |
| `engine_path=<path>` | Direct engine-binary override (debug). |

## `simready_benchmark.RunHandle`

```python
handle.wait(timeout: float | None = None) -> dict | None  # returns the run summary
handle.events() -> Iterator[dict]   # blocking event iterator until the run completes
handle.is_done() -> bool            # True when a terminal state is reached
handle.cancel() -> None             # request graceful cancellation
handle.state            # RunState enum (RUNNING | COMPLETED | CANCELLED | ERROR)
handle.cancelled        # bool, True if cancel() was called
```

`wait()` blocks until every engine session has finished or the global timeout
(default 86400 s, override with `Runner.start(..., global_timeout=...)`) fires,
and returns the run summary dict (keys `total_work_items`, `completed`,
`failed`, `status`, `readiness`, `started_at`, `finished_at`).

## `simready_benchmark.Reporter`

```python
class Reporter:
    def __init__(self, output_dir: str | None = None) -> None: ...

    @staticmethod
    def generate(
        output_dir:  str | None = None,
        fmt:         str = "both",
        publish_to:  str | None = None,
        **kwargs,
    ) -> dict: ...
```

`generate` is a static method, so call it as `Reporter.generate(output_dir=...,
fmt=...)`. `fmt` is `"html"`, `"json"`, or `"both"` (default). Returns the
report dict (keys `metadata`, `assets`, and `engines`) and writes `index.html` and
`report/test_results_index.json` under `output_dir`. Passing `publish_to` also
publishes the completed output directory to that destination.

## `simready_benchmark.PROTOCOL_VERSION`

Bumped when the plan / result.json on-disk schemas change. Use it to refuse to
process an older or newer plan than your installed framework understands.

## Exported Names (One-Line Summary)

| Name | What it is |
|---|---|
| `Planner` | Static-method class that builds a `Plan` |
| `Runner` | Static-method class that runs a `Plan` |
| `Reporter` | Aggregates per-test `result.json` files into HTML + JSON |
| `RunHandle` | Background-execution handle returned by `Runner.start()` |
| `RunContext` | Per-test runtime context passed to each `@test` function |
| `SceneHandle` | Engine-agnostic scene API (`ctx.scene`) |
| `Plan` (from `core.models`) | Plan dataclass: assets, features, profiles, tests, overrides |
| `PlanEditor` | Mutates a `Plan` by appending override entries |
| `EnginePlugin`, `EngineSession`, `EngineConfig` | Engine ABC + helpers |
| `test` | The `@test` decorator |
| `PROTOCOL_VERSION` | On-disk schema version |
| `publish_results(...)` | Publish a completed `_testing/` to a destination (S3, Nucleus, local) |
