# The Subprocess Pattern in KitPlugin and KitSession

`KitPlugin` and `KitSession`, from the `simready-benchmark-engine-kit` package, are
the production reference for a subprocess-driven plugin. The plugin spawns one
local engine process per session. The engine inside that process claims tests
from the shared work pool and writes per-test `result.json` files. The plugin's
job is to launch the process, wait for it to become ready, demultiplex its
output into events, monitor liveness, and tear it down.

This page walks the pattern in the order the lifecycle runs.

## Discovering an Executable from engines.toml

`KitPlugin.discover(self, engines_toml_path=None)` reads `engines.toml` and
returns one `EngineConfig` per `[kit.<instance>]` subtable. The lookup order:

1. The `engines_toml_path` argument, when it points at an existing file.
2. The `SIMREADY_ENGINES_TOML` environment variable. When it is set but the file
   is missing, `discover` raises `FileNotFoundError` rather than silently
   falling through.
3. `./engines.toml` in the current working directory.
4. `~/.simready-benchmark/engines.toml`.

The plugin parses the file with `tomllib`, reads the top-level `kit` table, and
for each subtable builds an `EngineConfig`. A relative `executable_path` is
resolved against the directory holding `engines.toml`, so a config file can ship
next to a relative pointer at the runtime:

```python
toml_dir = os.path.dirname(os.path.abspath(toml_path))
exe_path = config_data.get("executable_path", "")
if exe_path and not os.path.isabs(exe_path):
    exe_path = os.path.normpath(os.path.join(toml_dir, exe_path))
```

`validate` then checks that `executable_path` exists on disk (a hard
`valid=False` when missing) and consults `driver_compatibility` to warn, not
fail, when the installed GPU driver falls outside a declared range.

## Building the Command Line

`command_builder.build_kit_command` composes the launch command. The signature
is `build_kit_command(executable_path, kit_runner_path, custom_args=None, session_index=0)`.
The command injects a runner script through Kit's `--exec` flag and gives each
session its own HTTP port to avoid collisions when several Kit instances run in
parallel:

```python
http_port = 8011 + session_index
command = [
    executable_path,
    "--exec",
    kit_runner_path,
    "--no-window",
]
command.extend([
    "--/app/useFabricSceneDelegate=true",
    "--/app/window/hideUi=1",
    "--/exts/omni.services.transport.server.http/port=%d" % http_port,
])
if custom_args:
    command.extend(custom_args)
```

The executable can be a `.bat` file, a `.sh` file, or a direct `kit.exe`. The
session is tracked through the file protocol and event stream, not a single
process PID, so all three launch forms work.

## Spawning the Process

`KitPlugin.start_session` increments a session counter, builds the command,
copies `os.environ`, and overlays the autonomous-mode keys:

```python
env = os.environ.copy()
if session_env:
    env.update(session_env)
env["SIMREADY_SESSION_ID"] = session_id
env["SIMREADY_ENGINE_TAGS"] = ",".join(engine_config.tags)
env["SIMREADY_ENGINE_VERSION"] = engine_config.version
```

It launches through `SubprocessEngine.launch(command, env, log_file)`, which
opens the process with `stdout=subprocess.PIPE`, `stderr=subprocess.STDOUT`, and
`bufsize=0`, then registers it in a process registry so an atexit handler can
kill orphans on unexpected exit. Piping stdout is what lets the log-capture
thread read the event stream line by line.

## The Log-Capture Thread

`KitLogCapture` is a background thread that drains the process stdout and
demultiplexes it. Kit prefixes every structured event line with `@@EVENT@@`.
The thread separates those from ordinary engine log noise:

```python
for raw_line in iter(self._process.stdout.readline, b""):
    if self._stop_event.is_set():
        break
    self.last_output_time = time.monotonic()       # liveness timestamp
    line_str = raw_line.decode("utf-8", errors="replace")
    if line_str.startswith(EVENT_PREFIX):           # "@@EVENT@@"
        event_json = line_str[len(EVENT_PREFIX):].strip()
        if self._event_callback:
            self._event_callback(event_json)
    else:
        log_file.write(line_str)
        log_file.flush()
```

Two outputs come off this loop. Event lines go to the callback, which decodes
the JSON and appends each event to a shared list. Noise lines go to the session
log file. The thread also records `last_output_time` on every line, event or
noise, which feeds the liveness check below. `start_session` wires the callback
to an `events_received` list, then calls `log_capture.start()`.

## Waiting for the Engine-Ready Event

After starting capture, `start_session` polls the shared event list for an
`engine_ready` event with a bounded timeout, yielding to the loop between polls:

```python
timeout = 120.0
elapsed = 0.0
poll_interval = 0.5
ready_event = None
while elapsed < timeout:
    for evt in events_received:
        if evt.get("type") == "engine_ready":
            ready_event = evt
            break
    if ready_event is not None:
        break
    if process.poll() is not None:
        raise RuntimeError(
            "Kit process exited with code %s during startup. See log: %s"
            % (process.returncode, log_file)
        )
    await asyncio.sleep(poll_interval)
    elapsed += poll_interval

if ready_event is None:
    self._subprocess.kill(process)
    raise TimeoutError("Kit did not emit engine_ready within %ss. See log: %s" % (timeout, log_file))
```

Two failure modes are handled distinctly. If the process exits before
`engine_ready`, the plugin raises `RuntimeError` with the exit code and log
path. If the timeout elapses with the process still running, it kills the
process and raises `TimeoutError`. The poll uses `await asyncio.sleep`, never a
blocking `time.sleep`, so the framework event loop keeps driving other sessions.

After the event arrives, the plugin constructs a `KitSession` holding the
process handle, the log-capture thread, the engine-info dictionary from the
ready event, and the shared event list. The runner drains buffered events later
through the session's `drain_events()` method, which copies and clears the list.

## The Inactivity Timeout and Liveness

`KitSession.is_alive` combines the process poll with the activity timestamp the
capture thread maintains:

```python
def is_alive(self):
    process_running = self.process.poll() is None
    last_output = getattr(self.log_capture, "last_output_time", None)
    if isinstance(last_output, (int, float)):
        age = time.monotonic() - last_output
        return age < OUTPUT_INACTIVITY_TIMEOUT   # 180.0 seconds
    return process_running
```

The session is considered dead when no stdout has appeared for
`OUTPUT_INACTIVITY_TIMEOUT` (180 seconds), even when the OS still reports the
process running. The 180-second window accommodates long synchronous GPU and
PhysX work that does not yield output, while still catching a genuine lockup.
This is the central reason the subprocess pattern wants activity-based liveness
rather than a bare poll.

## Cleanup That Kills the Process Tree

`KitPlugin.stop_session` is force-kill only; there is no graceful protocol,
because sessions exit naturally when their work is done. It stops log capture,
force-kills the process tree, and kills any tracked child PIDs:

```python
session.log_capture.stop()
_force_kill_process_tree(session.process)
```

The process tree matters because the engine often runs under an intermediate
launcher. On Windows a `.bat` file launches `cmd.exe`, which launches `kit.exe`
as a grandchild, and the launcher can exit while the engine keeps running.
`_force_kill_process_tree` discovers child PIDs recursively, kills each with
`taskkill /F`, then kills the parent tree with `taskkill /F /T /PID`; on Linux
it kills the process group with `SIGKILL`. `KitSession.cleanup` is lighter: it
only stops the log-capture thread, because `stop_session` already handled the
process.

## Mapping the Pattern to Your Engine

To adapt this pattern, replace four engine-specific pieces and keep the
skeleton:

- The command builder: substitute your engine's launch flags and the path to
  whatever in-process runner claims tests from the work pool.
- The event prefix and readiness signal: substitute your engine's marker for
  `@@EVENT@@` and its ready signal for `engine_ready`. If your engine cannot
  emit a structured ready event, poll a file it writes or open a socket instead.
- The inactivity timeout: tune `OUTPUT_INACTIVITY_TIMEOUT` to your engine's
  worst-case quiet stretch.
- The teardown: keep the process-tree kill if your engine spawns children;
  simplify to `process.terminate()` plus a bounded `wait` if it does not.

The minimal `acme_engine` example in the SKILL shows this skeleton without the
Kit-specific helpers.
