---
title: "Agent Evaluation"
description: "The task-driven evaluation model — define tasks, run an agent to produce trials, and score each trial (output and trajectory) with per-task metrics."
---

Agent evaluation is the **task-driven** shape of NeMo Evaluator (see
[Dataset-Driven vs Task-Driven Evaluation](/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation)
for how it compares to metrics). You define **tasks**, an **agent** performs each one,
and you score the resulting **trial** — not just whether the final answer is right, but *how the agent
got there*. Each task carries its own metrics, so a single suite can grade heterogeneous work.

<Note>

**Platform-plugin support is in progress.** The pages in this section run agent evaluations through
the **local SDK** (`AgentEvaluator().run()`). Running them through the **NeMo Platform plugin** as
durable platform jobs — the way [dataset-driven metrics](/documentation/evaluate-models/metrics)
already can — is under active development; for now, use the local SDK path shown here.

</Note>

## The model

One run flows through five pieces:

**Task → (runner) → Trial → (metrics) → Scores → Result**

- **Task** (`AgentEvalTask`) — the unit of work. Its fields:
  - `intent` — a human-readable description of the goal; metadata for the suite's authors, **not** shown to the agent.
  - `inputs` — the instruction and anything the agent starts from (what the agent actually sees).
  - `reference` (optional) — grader-only, held-out ground truth the agent never sees.
  - `metrics` — the scorers for **this** task.
  - `views` (optional) — combine several of the task's metric outputs into one named, reported score (see *Score by component* under [Key properties](#key-properties)).
- **Runner** (`AgentTaskRunner`) — performs each task and returns trials through one small interface,
  `run_tasks(tasks) -> [AgentEvalTrial]`. The SDK ships runners for a plain async callable, a deployed
  agent over HTTP, and container/harness backends; you can supply your own. Scoring never depends on
  which runner produced a trial.
- **Trial** (`AgentEvalTrial`) — the durable record of one attempt: the agent's final `output`, the
  `evidence` it produced (an [Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trace of steps and tool calls, final filesystem state, logs), a
  `status` (`completed` / `partial` / `failed`), and arbitrary metadata. The trial — not the runner — is what
  gets scored, and it can be re-scored offline later.
- **Metrics** (`Metric`) — each task's metrics score its trials through
  `compute_scores(input) -> MetricResult`. A metric can read the final output **and** the evidence, so
  it can grade the outcome (did it answer correctly?) or the trajectory (did it use the expected tool?).
- **Result** (`AgentEvaluator` → `AgentEvalResult`) — the evaluator orchestrates the run and returns
  the trials, per-trial scores, and an aggregated `summary`. Point it at an output directory and it
  also writes a run **bundle**: `run.json`, `trials.jsonl`, `scores.jsonl`, `summary.json`, and a
  browsable `report.html`.

## A minimal example

<Note>

Conceptual — for a runnable, end-to-end version see the quickstart (coming in this section).

</Note>

```python
from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask

task = AgentEvalTask(
    id="capital-france",
    intent="Name the capital of France.",
    inputs={"instruction": "What is the capital of France?"},
    reference={"expected": "Paris"},   # grader-only; never shown to the agent
    metrics=[OutcomeMetric()],         # this task's own scorer(s)
)

# target is a Model, a deployed Agent (HTTP), or any AgentTaskRunner.
result = await AgentEvaluator().run(tasks=[task], target=my_agent)
print(result.summary)
```

## Key properties

- **Runner-agnostic scoring.** The scorer only ever sees an `AgentEvalTrial`, so the same tasks and
  metrics score any runner's output — or previously stored trials re-scored offline.
- **Per-task metrics.** Metrics are attached to each task, not once for the whole run, so a suite can
  grade heterogeneous work (docs, Q&A, tests, net-new code) each with the checks that fit it.
- **Score by component.** A single run can score at several levels:
  - the **task outcome** — the final answer;
  - the **trajectory** — how the agent worked (its tool use and steps);
  - **views** — named roll-ups you define on a task that combine two or more of its metric outputs into one reported score (for example, averaging an accuracy metric and a tool-use metric into a single `quality` score);
  - the **run-level aggregate** — results also roll up across the whole run.
- **Runs locally.** A full run — including the `report.html` dashboard — is produced on your machine
  with no platform services required. (Running the same suite as a durable platform job through the
  evaluator plugin is in progress — see the note above.)
- **Measurement, not decisions.** The evaluator produces scores, aggregates, and provenance — it
  doesn't decide pass/fail, gate a release, or compare runs. Those decisions belong to whatever
  consumes the results.

## Targets

An evaluation target is what performs the tasks:

- a **`Model`** — a chat/completions endpoint;
- a deployed **`Agent`** reachable over HTTP — a `GenericAgent` (any JSON endpoint) or a
  `NemoAgentToolkitAgent`;
- a custom **runner** (`AgentTaskRunner`) — anything that can turn tasks into trials.

See [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)
for how to describe an HTTP agent target.

## In this section

- [Quickstart](/documentation/evaluate-models/agent-eval/quickstart) — evaluate an agent end to end in
  a local process, no services required.
- [Evaluate a Deployed Agent over HTTP](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent)
  — point a run at an agent behind an HTTP endpoint.
- [Evaluate a Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner) — run an
  existing Harbor dataset in Docker and score its verifier reward.
- [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) — score the
  trajectory (not just the answer) and roll metrics into a named view.
- [Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners) — reference for
  what a run can point at.
- [Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) — reference for the
  `Metric` protocol.
- [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) — what a run returns and
  the on-disk bundle it writes.

## Related

<Cards>
  <Card title="Dataset-Driven vs Task-Driven Evaluation" href="/documentation/evaluate-models/dataset-driven-vs-task-driven-evaluation" />
  <Card title="Evaluate a Deployed Agent over HTTP" href="/documentation/evaluate-models/agent-eval/evaluate-deployed-agent" />
  <Card title="Evaluation Metrics" href="/documentation/evaluate-models/metrics" />
</Cards>
