---
title: "Evaluate a Harbor Task Suite"
description: "Run an existing Harbor task dataset through NeMo Evaluator's Harbor runner — Harbor executes each task in a Docker sandbox and its verifier emits a reward, which the SDK scores and reports like any other agent-eval run."
---

[Harbor](https://www.harborframework.com) is a container-based harness for agentic tasks: it runs each
task in a Docker sandbox, lets an agent work in it, then runs a **verifier** that emits a **reward**
(see Harbor's [Core Concepts](https://www.harborframework.com/docs/core-concepts) for its task, trial,
and job model). If you already have Harbor task datasets, the **Harbor runner** runs them and scores
the verifier reward through agent-eval — the same
[`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the same result and bundle as the
[quickstart](/documentation/evaluate-models/agent-eval/quickstart). Only the runner changes.

<Warning>

Unlike the quickstart, this runner is **not** zero-dependency — it shells out to Harbor and Docker:

- **Python ≥ 3.12**
- **Docker** installed and running
- **Harbor**, installed separately: `uv pip install "harbor>=0.16.1"`. Harbor is intentionally **not**
  a dependency of `nemo-platform[nemo-evaluator-sdk]`, so the rest of the SDK stays lightweight.

The runner raises a clear error pointing at this install step if `harbor` is missing.

</Warning>

## The dataset

A Harbor dataset is a directory of **task folders**. For discovery, the runner needs two files per
task — the rest of the task format (environment, verifier, and solution config) is Harbor's own:

```
my-suite/
  hello-world/
    task.toml         # [task] name = "harbor/hello-world"
    instruction.md    # Create a file called hello.txt with "Hello, world!" as the content.
  ...
```

`discover_harbor_tasks` reads each folder into an `AgentEvalTask`: the `[task] name` becomes the task
`id` and its human-readable `intent`, and `instruction.md` becomes `inputs["instruction"]` — the
instruction the agent is prompted with. See
[Harbor's task documentation](https://www.harborframework.com/docs/tasks) for the full task format.

<Note>

The repo ships a one-task
[example dataset](https://github.com/NVIDIA-NeMo/nemo-platform/tree/main/packages/nemo_evaluator_sdk/examples/harbor/hello_world_dataset)
you can clone and point at (it is not shipped in the installed wheel). Or point the dataset path at
your own Harbor suite.

</Note>

## Run it

A Harbor run is a normal agent-eval run: `AgentEvaluator().run(tasks=..., target=runner)`, exactly like
the [quickstart](/documentation/evaluate-models/agent-eval/quickstart) (a callable) and the
[deployed-agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) guide (an HTTP
target). Here the target is a `HarborAgentTaskRunner`.

```python
import asyncio
from pathlib import Path

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import (
    HarborAgentTaskRunner,
    HarborRuntimeConfig,
    discover_harbor_tasks,
)


async def main() -> None:
    # INPUT: your Harbor task suite (a directory of task folders).
    tasks = discover_harbor_tasks("path/to/my-suite")

    runner = HarborAgentTaskRunner(
        config=HarborRuntimeConfig(
            # OUTPUT: where Harbor writes its <job_name>/ results tree (not the dataset).
            jobs_dir=Path("./harbor-jobs"),
            agent_name="oracle",  # a built-in Harbor agent — see "Choosing an agent"
        )
    )

    # Same AgentEvaluator as every agent-eval run; only the target changes.
    result = await AgentEvaluator().run(tasks=tasks, target=runner)

    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")


asyncio.run(main())
```

The three pieces map onto the model: `discover_harbor_tasks` turns the suite into tasks,
`HarborAgentTaskRunner` runs the Harbor job and returns one trial per Harbor trial, and `AgentEvaluator`
scores each trial's reward with `HarborRewardMetric`.

<Note>

**`dataset_path` is input; `jobs_dir` is output.** The dataset is your read-only task suite. `jobs_dir`
is a directory the runner writes into — Harbor's per-trial results land under `jobs_dir/<job_name>/`,
and that directory doubles as a re-run cache (see [below](#attempts-concurrency-and-caching)).

</Note>

The `oracle` agent is Harbor's reference agent — it produces a passing trial on a well-formed task —
so for the one-task example above the verifier reward is `1.0`:

```
harbor_reward.reward: 1.0
```

Swap `agent_name` (or `agent_import_path`) for your own agent to get a real score.

## Choosing an agent

`HarborRuntimeConfig` decides what runs inside each sandbox (see
[Harbor's agent documentation](https://www.harborframework.com/docs/agents)):

- **`agent_name`** — a built-in Harbor agent (for example `"oracle"`, Harbor's reference agent, handy
  as a smoke test that the harness and dataset are wired up).
- **`agent_import_path`** — your own Harbor agent, e.g. `"my_agent_module:MyAgent"`. Set `agent_dir`
  as well when it's a loose file rather than an installed package. Overrides `agent_name`.
- **`agent_model_name`** — the model slug handed to the agent.

## How scoring works

1. Harbor runs each task in its sandbox and writes a `<task>__<hash>/result.json` per trial, including
   the reward from the task's [verifier](https://www.harborframework.com/docs/tasks).
2. The runner reads that reward onto each trial's metadata, and `HarborRewardMetric` scores it — the
   `reward` value, or `0.0` for a trial whose verifier emitted none.
3. `result.summary` aggregates the reward across tasks (`harbor_reward.reward`), `result.trials` holds
   each trial's status and evidence, and — if you pass a `config` with an `output_dir` to `run()` — the
   run **bundle** (including `report.html`) is written like any other agent-eval run.

## Attempts, concurrency, and caching

- **`n_attempts`** — trials Harbor runs per task; **`n_concurrent_trials`** — how many run at once.
- The job directory doubles as a **cache**. Pin a stable `job_name` and a completed job whose results
  already cover every requested task is re-scored instead of re-run. The default timestamped `job_name`
  always runs fresh; set `force_rerun=True` to delete an existing job dir first.

## Shortcut: `run_harbor_eval`

When a run is exactly "one Harbor suite, scored by its reward," `run_harbor_eval` collapses the three
steps above — discover, run, score — into a single call:

```python
from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval

result = await run_harbor_eval(
    HarborRuntimeConfig(jobs_dir="./harbor-jobs", agent_name="oracle"),
    dataset_path="path/to/my-suite",
)
```

It uses the same `AgentEvaluator` and `HarborRewardMetric` under the hood. Prefer the explicit form
above when you want to mix Harbor tasks with other tasks or metrics, or share one evaluator across
targets.

## Next steps

<Cards>
  <Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
  <Card title="Agent Evaluation Quickstart" href="/documentation/evaluate-models/agent-eval/quickstart" />
</Cards>
