---
title: "Agent Evaluation Quickstart"
description: "Evaluate a simple agent end to end with the NeMo Evaluator SDK — define tasks and a metric, run locally, and read the scores and HTML report. No platform services or API keys required."
---

In this quickstart you'll evaluate an agent on two tasks, score its answers, and produce a run
bundle with a browsable HTML report — all on your machine, with **no platform services and no API
keys**. It takes about five minutes.

For the concepts behind tasks, trials, runners, and metrics, see
[Agent Evaluation](/documentation/evaluate-models/agent-eval). This page is the hands-on version.

## Prerequisites

- Python 3.11–3.14
- The SDK:

```bash
pip install nemo-platform[nemo-evaluator-sdk]
```

<Note>

Everything here runs in a single local Python process. You'll swap the stand-in agent for a real
model or deployed agent once you've seen the flow.

</Note>

## 1. Define a metric

A metric scores one trial. It implements three things: a `type` (its name), an `output_spec()` (the
values it emits), and `compute_scores()` (the scoring logic). Here's a tiny one that scores `1.0`
when the agent's answer contains an expected keyword and `0.0` otherwise. It reads the agent's answer
from `input.candidate.output_text` and the task's grader-only `reference` from `input.row.data`.

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult


class KeywordMatchMetric:
    """Score 1.0 when the agent's answer contains the expected keyword, else 0.0."""

    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## 2. Define your tasks

Each `AgentEvalTask` is one unit of work: an `intent`, the `inputs` the agent acts on (here, an
`instruction`), a grader-only `reference` (held-out truth the agent never sees), and the `metrics`
that score **this** task. The `intent` is a human-readable note about the task's goal — metadata for
whoever reads or maintains the suite. It is **not** passed to the agent, which only ever sees
`inputs`. Because metrics live on the task, different tasks can use different scorers.

```python
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask


def make_tasks() -> list[AgentEvalTask]:
    return [
        AgentEvalTask(
            id="capital-france",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of France?"},
            reference={"expected": "Paris"},
            metrics=[KeywordMatchMetric()],
        ),
        AgentEvalTask(
            id="capital-japan",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of Japan?"},
            reference={"expected": "Tokyo"},
            metrics=[KeywordMatchMetric()],
        ),
    ]
```

## 3. Provide an agent

An agent is anything that turns a task into an answer. The simplest option is an async function; the
SDK provides a runner — `CallableAgentTaskRunner` — that wraps your function so it plugs into a run.
The runner hands your function the whole task and expects an answer back (here, a canned response
keyed by task `id`). We'll use a stand-in so the quickstart runs with no external services; later
you'll point the evaluation at a real model or a
[deployed agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) instead.

```python
async def my_agent(task: AgentEvalTask) -> str:
    # Stand-in agent — replace with a real model or deployed agent later.
    answers = {
        "capital-france": "The capital of France is Paris.",
        "capital-japan": "The capital of Japan is Tokyo.",
    }
    return answers[task.id]
```

## 4. Run the evaluation

`AgentEvaluator.run()` sends the tasks to the runner, collects the trials, scores them, and returns
an `AgentEvalResult`. Setting `output_dir` also writes a run bundle to disk. (`run()` is async, so it
lives inside an `async` function — see the full script below.)

```python
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig

result = await AgentEvaluator().run(
    tasks=make_tasks(),
    target=CallableAgentTaskRunner(my_agent),
    config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
)

for aggregate in result.summary.scores.scores:
    print(f"{aggregate.name}: {aggregate.mean}")
```

## 5. Read the results

`run()` returns an `AgentEvalResult` you can work with directly in Python:

- `result.summary` — aggregated scores per metric output (the mean you just printed), plus coverage
  counts (how many trials were scored, failed, or missing an output).
- `result.scores` — one entry per (task, trial, metric): the metric outputs and their status.
- `result.trials` — each trial: the agent's answer, the evidence it produced (trajectory, final
  state, logs), and its status (`completed` / `partial` / `failed`).
- `result.run_id` — a stable identifier for this run.

Because you set `output_dir`, the same data was also written to `./agent-eval-run/` as a run bundle:

| File | Contents |
|---|---|
| `summary.json` | aggregated scores per metric output (mean / min / max / std-dev / counts) and coverage |
| `scores.jsonl` | one row per (task, trial, metric) — the metric outputs, status, and any diagnostics |
| `trials.jsonl` | one row per trial — the agent's output, its evidence, and status |
| `tasks.jsonl` | the tasks that were evaluated |
| `run.json` | the run manifest — the run id and a map of the artifact files |
| `benchmark.json` | benchmark-grouping metadata for the run |
| `report.html` | a browsable dashboard of the run — open it in a browser |

The in-memory result and the on-disk bundle hold the same information: use the result object for
programmatic follow-up, and the bundle (especially `report.html`) to inspect or share a run.

## Full script

```python
import asyncio

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.callable_runtime import CallableAgentTaskRunner
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult


class KeywordMatchMetric:
    """Score 1.0 when the agent's answer contains the expected keyword, else 0.0."""

    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])


def make_tasks() -> list[AgentEvalTask]:
    return [
        AgentEvalTask(
            id="capital-france",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of France?"},
            reference={"expected": "Paris"},
            metrics=[KeywordMatchMetric()],
        ),
        AgentEvalTask(
            id="capital-japan",
            intent="Answer the geography question.",
            inputs={"instruction": "What is the capital of Japan?"},
            reference={"expected": "Tokyo"},
            metrics=[KeywordMatchMetric()],
        ),
    ]


async def my_agent(task: AgentEvalTask) -> str:
    # Stand-in agent — replace with a real model or deployed agent later.
    answers = {
        "capital-france": "The capital of France is Paris.",
        "capital-japan": "The capital of Japan is Tokyo.",
    }
    return answers[task.id]


async def main() -> None:
    result = await AgentEvaluator().run(
        tasks=make_tasks(),
        target=CallableAgentTaskRunner(my_agent),
        config=AgentEvalRunConfig(output_dir="./agent-eval-run", parallelism=2),
    )
    for aggregate in result.summary.scores.scores:
        print(f"{aggregate.name}: {aggregate.mean}")


asyncio.run(main())
```

Run it:

```bash
python quickstart.py
```

Expected output (both answers contain the expected keyword, so the mean is `1.0`):

```
keyword_match.score: 1.0
```

Open `./agent-eval-run/report.html` to see the run in a browser.

## Next steps

<Cards>
  <Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
  <Card title="Evaluate a Deployed Agent over HTTP" href="/documentation/evaluate-models/agent-eval/evaluate-deployed-agent" />
</Cards>

- Swap the stand-in agent for a **real model or a deployed agent over HTTP**.
- Add a **trajectory** metric that scores *how* the agent worked (its tool use), not just the answer.
- Give a task **more than one metric**, or combine metric outputs into a named **view**.
