---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Writing Metrics"
description: "Reference for the Metric protocol — the three members every metric implements, what compute_scores receives (the answer, the trajectory/evidence, the grader-only truth), the output value types, and how results are validated and reported."
---

A **metric** scores one trial. Metrics are attached to each task (not once per run), so a suite can
grade heterogeneous work — a Q&A task and a coding task can carry different scorers. Every metric,
however simple or elaborate, implements the same small protocol.

## The protocol

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutputSpec, MetricResult

class MyMetric:
    @property
    def type(self) -> str:
        """A unique name for this metric within a task (also the summary key prefix)."""

    def output_spec(self) -> list[MetricOutputSpec]:
        """The named values this metric emits, and their types."""

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        """Score one trial and return its outputs."""
```

No base class — a metric is any object with these three members (a structural `Metric` protocol).

## What `compute_scores` receives

`input.candidate` is the trial under evaluation:

| Field | Type | What it holds |
|---|---|---|
| `candidate.output_text` | `str \| None` | the agent's final answer |
| `candidate.evidence` | `CandidateEvidence \| None` | trajectory, final filesystem state, logs — see [Reading evidence](#reading-evidence) |
| `candidate.metadata` | `dict` | trial metadata (e.g. a reward a runner stamped on) |

`input.row.data` is a dict describing the task and trial:

| Key | What it holds |
|---|---|
| `input.row.data["reference"]` | grader-only ground truth (the task's `reference`), never shown to the agent |
| `input.row.data["inputs"]` | the task `inputs` (`instruction`, …) |
| `input.row.data["task"]` | `{id, intent, metadata}` |
| `input.row.data["trial"]` | `{id, task_id, status, metadata}` |

So an **outcome** metric reads `candidate.output_text` and `row.data["reference"]`; a **trajectory**
metric reads `candidate.evidence`.

## Declaring outputs

A metric declares its outputs up front; the runtime validates that `compute_scores` returns **exactly**
those names, each coercible to the declared type (a missing or undeclared output raises). Build specs
with the `MetricOutputSpec` factories:

| Factory | Value type | Use for |
|---|---|---|
| `MetricOutputSpec.continuous_score(name)` | `float` | a numeric score (0–1 or unbounded) |
| `MetricOutputSpec.discrete_score(name)` | `int` | counts or ordinal levels |
| `MetricOutputSpec.boolean(name)` | `bool` | a pass/fail check |
| `MetricOutputSpec.label(name)` | `str` | a category label |
| `MetricOutputSpec.model(name, value_schema)` | your `BaseModel` | structured/custom values |

A metric may emit **several** outputs — for example an efficiency metric returning both a boolean and a
count:

```python
def output_spec(self) -> list[MetricOutputSpec]:
    return [
        MetricOutputSpec.boolean("efficient_tool_use"),
        MetricOutputSpec.discrete_score("max_repeated_tool_calls"),
    ]
```

<Tip>

Prefer `continuous_score` when you want a numeric **mean** in the run summary. A `boolean` output
reports per-trial pass/fail but does not aggregate to a numeric mean on its own (though it still
contributes as 0/1 to a [view](/documentation/evaluate-models/agent-eval/score-by-component)).

</Tip>

## Returning a result

Return a `MetricResult` whose `outputs` match `output_spec` by name:

```python
from nemo_evaluator_sdk.metrics.protocol import MetricInput, MetricOutput, MetricOutputSpec, MetricResult

class KeywordMatchMetric:
    @property
    def type(self) -> str:
        return "keyword_match"

    def output_spec(self) -> list[MetricOutputSpec]:
        return [MetricOutputSpec.continuous_score("score")]

    async def compute_scores(self, input: MetricInput) -> MetricResult:
        expected = str(input.row.data.get("reference", {}).get("expected", "")).lower()
        answer = (input.candidate.output_text or "").lower()
        return MetricResult(outputs=[MetricOutput(name="score", value=1.0 if expected and expected in answer else 0.0)])
```

## Reading evidence

`candidate.evidence` (a `CandidateEvidence`) holds named descriptors, each exposed through a typed
handle. Always guard first — a trial may not carry a given descriptor:

```python
from nemo_evaluator_sdk.values.evidence import EVIDENCE_TRACE

evidence = input.candidate.evidence
if evidence is not None and evidence.get(EVIDENCE_TRACE) is not None:
    calls = await (await evidence.trace(EVIDENCE_TRACE)).tool_calls()
```

| Evidence | Handle | Reads |
|---|---|---|
| **trace** (`EVIDENCE_TRACE`) | `await evidence.trace(name)` | `.tool_calls()`, `.steps()`, `.token_usage()` — the [Agent Trajectory Interchange Format (ATIF)](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trajectory |
| **filesystem** (`EVIDENCE_FINAL_STATE`, `EVIDENCE_INITIAL_STATE`) | `await evidence.filesystem(name)` | `.run_verifier(command)`, `.diff(other)` — run a check or diff two snapshots |
| **logs** | `await evidence.logs(name)` | `.read_text(file)`, `.tail(file)` |

The [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) guide has a
complete, runnable trajectory metric; the SDK's `example_metrics.py` (under
`examples/run_agent_eval/`) shows filesystem and trace metrics.

## How outputs are reported

Each output aggregates in `result.summary` under the key `<metric.type>.<output>` (mean / min / max /
std-dev). To roll several outputs into one reported score, define a **view** on the task — see
[Score by Component](/documentation/evaluate-models/agent-eval/score-by-component).

## Related

<Cards>
  <Card title="Score by Component" href="/documentation/evaluate-models/agent-eval/score-by-component" />
  <Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
  <Card title="Quickstart" href="/documentation/evaluate-models/agent-eval/quickstart" />
</Cards>
