---
name: write-test-pack
description: Use when extending the framework with custom OEM tests, or building the next FET phase pack. Authors a pip-installable wheel of async @test-decorated functions that the framework discovers through the simready_benchmark.tests entry-point group.
license: Apache-2.0
compatibility: Requires simready-benchmark (pip install simready-benchmark). For Kit / Isaac Sim tests, also requires simready-benchmark-engine-kit and the runtime installed separately.
metadata:
  author: NVIDIA
  library: simready_benchmark
  version: "2026.05"
---

## Confirm the Installed Version

This skill ships inside the `simready-benchmark` wheel, so the copy installed with the framework matches the installed code. If you are working from a cached copy or from memory rather than the installed package, confirm the framework version with `simready-benchmark --version`. Treat the installed module docstrings (rendered in the API reference) and `simready-benchmark --list-tests` as the authoritative source for the APIs, entry points, and names described here, because a signature or name documented here might have changed in a different version.

## Anatomy of a Test

```python
# oem_acme_tests/fet028_gripper/close_and_lift.py
from simready_benchmark import test
from simready_benchmark.runner.test_context import RunContext

@test(
    features=[{"id": "FET028_GRIPPER", "version": ">=1.0.0"}],
    requirement="GR.LIFT.001",
    name="close_and_lift",
    description="Close the gripper, then lift; verify grasp holds without slip.",
    expected_video="Gripper closes around the object, lifts ~30cm, holds for 1s; object stays inside the jaws.",
    version="1.0.0",
)
async def close_and_lift(ctx: RunContext) -> None:
    ctx.step("Load asset and room")
    handle = ctx.scene.load_asset(ctx.asset_path)
    room = ctx.scene.add_room()
    room.auto_size(handle)

    ctx.step("Add physics and play")
    physics = ctx.scene.add_physics(gravity=9.81, fps=240.0)
    physics.play()

    ctx.step("Settle for camera convergence")
    await ctx.settle(frames=8)

    ctx.step("Lift the asset")
    ctx.scene.set_camera(position=(2.0, 2.0, 1.5), target=(0.0, 0.0, 0.5))
    frames = []
    for i in range(60):
        await ctx.physics_step()
        frames.append(await ctx.capture_frame(label=f"lift_{i:03d}"))

    ctx.encode_video(frames, fps=60, label="trajectory", role="summary")

    pos = ctx.get_asset_position()
    ctx.add_metric("final_z", pos[2], unit="m")
    if pos[2] < 0.25:
        await ctx.capture_frame(label="failure_drop", role="error")
        ctx.fail("Asset fell below 0.25m - grasp slipped", details={"final_z": pos[2]})
```

Key shape:
- `async def`. The framework awaits each test. Plain `def` raises `TypeError` at registration.
- Takes a single `RunContext` argument (`ctx`).
- Call `ctx.step(message)` between phases so the liveness watchdog (60s no-progress -> session dead) stays satisfied in long tests.

## Required Decorator Metadata

| Kwarg | Required | Purpose |
|---|---|---|
| `features` | yes | `list[dict]` of `{"id": str, "version": str}`, the features this test covers |
| `requirement` | yes | requirement code from the spec, for example `"GR.LIFT.001"` |
| `name` | yes | snake_case test name; unique within the feature |
| `description` | yes (non-empty) | one paragraph of what the test verifies (asset-author POV) |
| `expected_video` | yes (non-empty) | one paragraph describing what a passing run looks like in the captured video |
| `version` | yes | semver for the test definition (bump on behavior change) |
| `engine` | no | engine constraints: `{"tags": ["kit"], "version": ">=2024.2"}` |
| `config_defaults` | no | default `ctx.config` dict; overridden per-asset by the plan |
| `max_duration` | no | seconds; default `300`. Runner kills tests that exceed this. |
| `enabled` | no | set `False` to keep the source but skip registration |

The decorator validates `description` and `expected_video` even when
`enabled=False`, so re-enabling never trips on missing metadata.

## Media Roles

`capture_frame` and `encode_video` accept a `role` kwarg controlling how the
file is surfaced in the HTML report's per-test grid:

| Role | Use |
|---|---|
| `"summary"` | The headline capture / video. Usually the `encode_video` result. |
| `"error"` | A capture taken at the point of failure. |
| `"worst"` | A capture of the worst-performing frame in a metric loop. |
| `"normal"` | Default; everything else. |

Sort: videos before images, then `summary`, `error`, `worst`, and `normal`
in that order, then alphabetical by filename.

## Make Tests Config-Driven

```python
@test(
    features=[{"id": "FET003_PHYSICS", "version": ">=1.0.0"}],
    requirement="PH.DROP.001",
    name="ground_drop",
    description="Drop the asset from a configurable height; assert it settles within settle_seconds.",
    expected_video="Asset drops, bounces once, comes to rest. Wireframe overlay visible.",
    version="1.0.0",
    config_defaults={"drop_height": 1.0, "settle_seconds": 1.5},
)
async def ground_drop(ctx: RunContext) -> None:
    h = ctx.config["drop_height"]
    secs = ctx.config["settle_seconds"]
    # ... use h and secs ...
```

Users override per-asset by using the CLI's `--set-config` flag or
`PlanEditor.set_config_override(...)`.

## Package the Test Pack as a Wheel

```toml
# oem-acme-tests/pyproject.toml
[project]
name            = "oem-acme-tests"
version         = "0.1.0"
description     = "OEM-specific benchmarks for ACME robotic assets"
requires-python = ">=3.11"
dependencies    = [
    "simready-benchmark>=2026.5.1",
]

[project.entry-points."simready_benchmark.tests"]
acme = "oem_acme_tests"

[build-system]
requires      = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
package-dir = {"" = "src"}
```

The entry-point group is literally `simready_benchmark.tests`. The value is the
**bare module path** of your package (no `:callable` suffix). The framework
loads the module and uses `os.path.dirname(module.__file__)` as the directory
to walk for `@test`-decorated functions.

## Source Layout

```
oem-acme-tests/
├── pyproject.toml
└── src/
    └── oem_acme_tests/
        ├── __init__.py
        ├── fet028_gripper/
        │   ├── __init__.py
        │   └── close_and_lift.py
        └── fet003_physics/
            ├── __init__.py
            └── ground_drop.py
```

After `pip install oem-acme-tests`, the framework discovers the tests
automatically, with no engines.toml changes required. Verify with:

```bash
simready-benchmark --list-tests
```

## Iterate Without Packaging

While developing, point the framework at the source directory:

```toml
# engines.toml
[tests]
paths = ["/path/to/oem-acme-tests/src/oem_acme_tests"]
```

Or use the CLI or env var. Highest priority first: `--tests-path` > `SIMREADY_BENCHMARK_TESTS_PATH` > `engines.toml [tests].paths` > `project_config.toml [tests].paths`. The first non-empty source wins; pip-installed entry-point packs (and, for a foundations project, the bundled FET default) always merge on top.

```bash
simready-benchmark --tests-path /path/to/oem-acme-tests/src/oem_acme_tests

# Linux/macOS
export SIMREADY_BENCHMARK_TESTS_PATH=/path/to/oem-acme-tests/src/oem_acme_tests

# Windows PowerShell
$env:SIMREADY_BENCHMARK_TESTS_PATH = "C:/path/to/oem-acme-tests/src/oem_acme_tests"
```

Path-discovered tests COMBINE with entry-point-discovered tests; they do not
replace them.

## Key Rules

- The decorated function MUST be `async def`. The decorator raises `TypeError` otherwise.
- The function MUST accept a single `RunContext` argument and return `None`.
- `description` and `expected_video` are required, non-empty strings. Validated even when `enabled=False`.
- `features` is required and must be non-empty: every test declares at least one feature, as `{"id": str, "version": str}` with both non-empty. The decorator raises `TypeError` otherwise, even when `enabled=False`.
- Failures are signaled by `ctx.fail(message, details=None)`. `fail()` does NOT stop execution. Follow with `return` to exit early.
- `ctx.skip(reason)` marks the test skipped (prerequisite not met). Similarly does NOT stop execution.
- `ctx.precheck_failure(message)` raises `PrecheckAborted` (caught by the runner) AND chooses fail-vs-skip based on the asset's validation claim.
- Call `ctx.step(message)` between long phases (>30s) so the 60s liveness watchdog does not kill the session.
- Path-discovered + entry-point-discovered test packs combine; the framework loads ALL of them.

For full decorator, RunContext, and SceneHandle details, refer to [references/test-decorator.md](references/test-decorator.md).
For packaging conventions (pyproject layout, versioning, distribution), refer to [references/packaging.md](references/packaging.md).
