---
title: "Dataset-Driven vs Task-Driven Evaluation"
description: "Two complementary shapes of evaluation in NeMo Evaluator — scoring model outputs over a dataset, versus scoring an agent that performs tasks — and how to choose between them."
---

NeMo Evaluator supports two complementary shapes of evaluation. They share the same scoring
foundation but differ in **what produces the thing you score** and **what you're allowed to look at**
when you score it.

- **Dataset-driven evaluation** — you have a dataset of homogeneously structured rows, a target
  generates an output for each row (or you supply outputs), and metrics score each output against a
  reference. This is the path
  behind [metrics](/documentation/evaluate-models/metrics).
- **Task-driven evaluation (agent evaluation)** — you have a set of *tasks*, an agent performs each
  task, and you score the resulting *trial* — the agent's final output **and** how it got there
  (its trajectory, tool calls, and other evidence).

<Note>

Both shapes score with the **same `Metric` interface**. A deterministic/code scorer or an
LLM-as-a-judge scorer works in either one — the difference is the shape of the input being scored,
not the scorer.

</Note>

## Dataset-driven evaluation

You start from a **dataset** — rows of inputs, usually with a reference/expected value. A **model**
(or pipeline) produces an output per row, either generated online at evaluation time or supplied
offline, and metrics score each output against its reference. The suite is uniform — the **same
metric set applies to every row**.

**Use it when** the thing under test is a model or pipeline you can run over a fixed dataset and you
care about output quality: model quality checks, RAG pipelines, and regression testing against a
labeled set.

Entry point: [Evaluation Metrics](/documentation/evaluate-models/metrics).

## Task-driven evaluation (agent evaluation)

You start from **tasks**. Each task gives an agent an instruction and any inputs it needs; the agent
performs the task through a **runner**, producing a **trial** — the agent's final output plus
evidence such as its trajectory (an [Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trace of steps and tool calls), final filesystem state, and
logs. Container-based runners also provide an environment (and its final state) the agent works in.
Metrics then score the trial. Because the trajectory is part of the trial, you can score not
just *whether the answer is right* but *how the agent worked* — for example, whether it used the
expected tool.

Metrics are attached **per task**, not once for the whole run — each task carries its own scorers.
That lets a single evaluation grade **heterogeneous work**: a coding agent whose tasks span
documentation, Q&A, unit tests, and net-new code can score each task type with the checks that fit
it — an LLM-as-a-judge for documentation clarity, a tests-pass check for code, an exact-match check
for a factual answer — all in one suite. Dataset-driven runs, by contrast, apply the same metric set
to every row.

**Use it when** the thing under test is an agent that *acts*: it calls tools, takes multiple steps,
or edits state — and you care about the process as well as the outcome, or you're comparing skills or
models on agentic tasks.

Entry point: [Agent Evaluation](/documentation/evaluate-models/agent-eval).

## At a glance

| | Dataset-driven | Task-driven (agent evaluation) |
|---|---|---|
| **Unit of evaluation** | a dataset row | a task, and the trial it produces |
| **What produces the candidate** | a model/pipeline generates an output per row | an agent performs the task (via a runner) |
| **What you score** | the output, against a reference | the trial: final output **and** trajectory/evidence |
| **Evidence available to the scorer** | the output (and row fields) | ATIF trace, tool calls, final state, logs |
| **Metric assignment** | one metric set applied to every row | each task defines its own metrics (heterogeneous suites) |
| **Scoring granularity** | per-row metric + aggregate | task · trajectory · component (views) · session |
| **Target** | a `Model` or `Agent` (plus the dataset) | a `Model`, a deployed `Agent` (HTTP), or a custom runner |
| **Entry point** | metrics (`RunConfig`, dataset rows) | `AgentEvaluator` + `AgentEvalTask` |
| **Result** | metric results + aggregates | an `AgentEvalResult` bundle (trials incl. ATIF trajectory & file evidence, scores, summary, report) |
| **Best for** | model quality, RAG, regression on labeled data | agents, tool use, multi-step behavior, skills, model/skill A/B |

## Which should I use?

<Tip>

Start from *what produces the answer*. If you can generate outputs by running a model over a fixed
dataset, use **dataset-driven**. If an agent has to *do something* to produce the answer — and how it
does it matters — use **task-driven**.

</Tip>

- **Dataset-driven** — you have (or can produce) a labeled dataset and want to score outputs against
  references: LLM quality, RAG, or a regression suite.
- **Task-driven** — the system under test is an agent that uses tools or takes multiple steps, you
  want to score the trajectory as well as the outcome, your suite is **heterogeneous** so different
  tasks need different metrics (e.g. a coding agent producing docs, Q&A, tests, and net-new code), or
  you're running a skills / model A/B on agentic tasks.
- **Both** — the two compose. It's common to track base model quality with dataset-driven metrics
  *and* agentic behavior with task-driven evaluation.

## What both share

- **One scoring interface.** Every scorer implements the same `Metric` protocol, so the same
  deterministic and LLM-as-a-judge scorers apply to either shape.
- **Measurement, not decisions.** The evaluator *measures* — it produces scores, aggregates, and
  provenance. It does not decide pass/fail, gate a release, or compare runs against each other —
  those decisions belong to whatever consumes the results. Both evaluation shapes emit measurements;
  decisions live above them.

## Related

<Cards>
  <Card title="Evaluation Metrics" href="/documentation/evaluate-models/metrics" />
  <Card title="Agent Evaluation" href="/documentation/evaluate-models/agent-eval" />
  <Card title="Agent Configuration (targets)" href="/documentation/evaluate-models/metrics/agent-configuration" />
</Cards>
