---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Manage Tasks & Tasksets"
description: ""
---

A **task** is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent
should do), the inputs it receives, and the metrics that score it. A **taskset** is a named grouping
of tasks. Both are first-class entities in the Evaluator plugin, addressed by `workspace/name` and
managed through the `nemo_platform` SDK.

Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across
runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and
metrics inline every time.

## Concepts

| Concept | What it is | Members |
|---------|------------|---------|
| **Task** | A reusable agent-eval unit: `intent`, `inputs`, and the `metrics` that score it. | References the metrics that score it. |
| **Taskset** | A flexible grouping of tasks with a description and metadata. | References member tasks by `workspace/name`, each pinned to an exact revision. Membership is a **set** — order is not significant and duplicate references are rejected. |
| **Revision** | An immutable published snapshot of a task's or taskset's content, addressed by a content digest. | Belongs to the task or taskset it snapshots. |

Both are addressed by `workspace/name`. Names are unique within a workspace, limited to 255
characters, and must match `^[\w\-\.]+$`.

Every stored task and taskset is **versioned**. Creating one publishes revision 1; replacing its
content publishes the next revision. Earlier revisions stay readable for as long as the task or
taskset exists — deleting it removes its revisions with it — which is what lets an evaluation be
re-run against exactly the content it ran against the first time.

<Note>
Publishing is **idempotent**. Replacing a task with content identical to its current revision
publishes nothing and returns the existing revision — so a pipeline can re-submit the same
definition freely without accumulating versions.
</Note>

## Initialize the SDK

```python
import os

from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

tasks = client.evaluator.tasks        # EvaluatorTasksResource
tasksets = client.evaluator.tasksets  # EvaluatorTasksetsResource
```

## Manage Tasks

A task scores its output with metrics. Store the metric first, then reference it from the task by
`workspace/name`. See [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) for the
metric classes and options.

```python
from nemo_evaluator_sdk import ExactMatchMetric

# Store a metric the task will reference.
client.evaluator.metrics.create(
    "answer-exact-match",
    metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"),
)
```

Send the task as a `TaskInput` (the authorable subset of a task) and address it by name on create.
Reference the stored metric with a `MetricRef` (`workspace/name`, or a bare `name` resolved against
the task's workspace). The service returns the stored `Task`.

```python
from nemo_evaluator.api.schemas import EvaluatorTaskDefinition, MetadataItem, MetricRef, TaskInput, TaskInputs

task = TaskInput(
    spec=EvaluatorTaskDefinition(
        kind="evaluator",
        intent="Answer the user's geography question with the capital city.",
        inputs=TaskInputs(instruction="What is the capital of France?"),
        metrics=[MetricRef("default/answer-exact-match")],
    ),
    metadata=[MetadataItem(key="suite", value="geography")],
)

stored = tasks.create("capital-of-france", task=task)
print(stored.id, stored.spec.metrics)
```

### `TaskInput` fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `spec` | `TaskDefinition` | Yes | The task's content, discriminated by `kind` — see below. |
| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |

### Task kinds

A task is an evaluation unit; its `kind` says which runner executes it. There are two:

- `evaluator` — the task's content is fields you author, scored by platform metrics.
- `harbor` — the task's content is a packaged directory of files, scored by Harbor's own reward.

Both are stored as the same record type, so a taskset can group them and you manage every evaluation
unit in one place regardless of which runner executes it.

`EvaluatorTaskDefinition` (`kind="evaluator"`):

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `intent` | `str` | Yes | Human-readable description of the desired agent behavior. |
| `inputs` | `TaskInputs` | No | The task's recognized input fields. `instruction` is the agent's prompt; it falls back to `intent` when unset. |
| `reference` | `dict[str, Any]` | No | Grader-only ground truth (held-out tests, expected outputs, rubric data). Surfaced to metrics but never seeded into the agent's workspace or shown to the agent. Held out from the *agent*, not from the API. |
| `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. |
| `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. |

`HarborTaskDefinition` (`kind="harbor"`):

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `archive_ref` | `str` | Yes | Files reference to the task's packaged directory (`workspace/fileset#path`). One fileset per task, so a task shared by several tasksets is stored once. |
| `archive_digest` | `str` | Yes | Content hash Harbor computed over the task directory. |
| `instruction` | `str` | No | The task's instruction text, when it has one. |
| `config` | `dict` | No | Harbor's own task configuration (verifier, agent, environment, steps), stored as published. |

<Note>
Storing a Harbor task is supported; **running one from storage is not yet**. A taskset may group both
kinds, but expanding a `harbor` member is rejected with `422` before the run starts, whatever target
you submit against. Harbor evaluations continue to run through the existing dataset-driven path.
</Note>

<Note>
A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored
as a content-addressed *derived* metric, and the task record is normalized to reference it. This is
why `stored.spec.metrics` always comes back as a list of `MetricRef` references.
</Note>

### Retrieve, list, and delete

```python
# Retrieve one task by name (its current content)
task = tasks.retrieve("capital-of-france")
print(task.spec.kind, task.revision, task.tags)  # e.g. evaluator 1 {'latest': 1}

# List tasks in the workspace (paginated)
page = tasks.list(page=1, page_size=100, sort="-created_at")
for item in page.data:
    print(item.name, item.spec.kind)

# Delete a task (this also removes all of its revisions)
tasks.delete("capital-of-france")
```

`sort` accepts `name`, `created_at`, or `updated_at`, each optionally prefixed with `-` for
descending order.

## Revisions

### Publish a new revision

Use `replace` to publish new content. It creates the task if it does not exist, so a publisher needs
no existence check.

```python
revised_task = TaskInput(
    spec=EvaluatorTaskDefinition(
        kind="evaluator",
        intent="Answer the user's geography question with the capital city.",
        inputs=TaskInputs(instruction="Name the capital city of France."),
        metrics=[MetricRef("default/answer-exact-match")],
    ),
    metadata=[MetadataItem(key="suite", value="geography")],
)

updated = tasks.replace("capital-of-france", task=revised_task)
print(updated.revision)  # 2
```

Submitting content identical to the current revision publishes nothing and returns the existing
revision — but any tags in the body are still applied, which is how you tag a revision after the
fact.

### List revisions

Each entry carries the `content_hash` used to pin a reference.

```python
page = tasks.list_revisions("capital-of-france")
for revision in page.data:      # newest first
    print(revision.revision, revision.content_hash, revision.tags)
```

### Read a specific revision

Pass a content digest or a tag. This returns the content **as published**, not the current content.

```python
# Revisions come back newest-first and paginated, so index by ordinal rather than by position —
# `data[-1]` is only the oldest entry on the page you happen to have fetched.
page = tasks.list_revisions("capital-of-france")
digest = next(revision.content_hash for revision in page.data if revision.revision == 1)

original = tasks.retrieve("capital-of-france", revision=digest)  # revision 1, as published
current = tasks.retrieve("capital-of-france")                    # revision 2, the current content

assert original.revision == 1 and current.revision == 2
assert original.spec.inputs.instruction != current.spec.inputs.instruction
```

### Tag a revision

A tag is a mutable pointer to a revision — useful for marking one as reviewed or approved after it
has been evaluated. Read it back with `tag=`, the counterpart to `revision=`:

```python
tasks.tag("capital-of-france", tag="blessed", revision=digest)

blessed = tasks.retrieve("capital-of-france", tag="blessed")
```

`revision=` takes a content digest and `tag=` takes a tag name. They select the same thing two ways,
so pass one or the other — passing both raises `ValueError`.

<Note>
A tag names exactly one revision. Re-tagging moves the pointer rather than adding a second one, so
`retrieve(tag=...)` always resolves to a single revision — there is no way for two revisions to
share a tag.

`latest` is managed automatically and always names the most recently published revision; it cannot be
moved by hand. A tag name may not be empty, and may not look like a content digest (64 hexadecimal
characters) — such a tag could be stored but never resolved, because a digest-shaped reference is
looked up as a digest rather than as a tag.
</Note>

## Manage Tasksets

A taskset references existing tasks by `workspace/name`. All referenced tasks must already exist when
the taskset is created; a missing or duplicate reference is rejected.

Member references are **resolved to an exact revision when the taskset is stored**. You may submit a
bare name (`capital-of-france`), a tag (`capital-of-france#latest`), or a digest — what gets stored
is always `workspace/name#<digest>`. This is why a stored taskset keeps naming the same content even
after a member task publishes something new, and it is what makes a suite reproducible.

```python
from nemo_evaluator.api.schemas import TaskRef, TasksetInput

taskset = TasksetInput(
    description="Geography questions for smoke-testing the agent.",
    tasks=[
        TaskRef("default/capital-of-france"),
        TaskRef("default/capital-of-japan"),
    ],
)

stored = tasksets.create("geography-suite", taskset=taskset)
print(stored.tasks)
# ['default/capital-of-france#a1b2...', 'default/capital-of-japan#c3d4...']
```

### `TasksetInput` fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `description` | `str` | No | Human-readable description of the grouping. |
| `tasks` | `list[TaskRef]` | No | References to member tasks (`workspace/name`, or bare `name` within the same workspace), optionally pinned with `#<tag-or-digest>`. Each is resolved to an exact digest when stored. Set semantics — duplicates rejected. |
| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
| `tags` | `list[str]` | No | Tags to point at the revision this request publishes. `latest` is always applied server-side. |

### Retrieve, list, and delete

```python
taskset = tasksets.retrieve("geography-suite")

page = tasksets.list(page=1, page_size=100, sort="name")
for item in page.data:
    print(item.name, len(item.tasks))

tasksets.delete("geography-suite")
```

Tasksets carry the same revision surface as tasks — `replace`, `list_revisions`, `tag`, and
`retrieve(revision=...)` / `retrieve(tag=...)`:

```python
# Re-resolving membership after a member task published new content cuts a new revision.
tasksets.replace("geography-suite", taskset=taskset)

for revision in tasksets.list_revisions("geography-suite").data:
    print(revision.revision, revision.content_hash)
```

<Note>
Re-submitting the *same* member names can still publish a new revision. Members are re-resolved on
every write, so if a member task published in the meantime the grouping now names different content
and genuinely differs. A taskset's identity is the exact revisions it names, not the names alone.

Member *order*, by contrast, is not part of that identity: membership is a set, so it is stored in a
canonical order and reordering the same members publishes nothing.
</Note>

Deleting a taskset does not delete its member tasks — a taskset only holds references.

## Run an evaluation over a taskset

An agent evaluation is submitted with an `AgentEvalInputSpec`, whose `tasks` field is either an
inline list of tasks or a **reference to a stored taskset**. Referencing a taskset lets you keep the
task definitions in one place and evaluate the whole set by name, instead of inlining every task on
each run.

```python
from nemo_evaluator.api.schemas import TasksetRef
from nemo_evaluator.jobs.agent_spec import AgentEvalInputSpec, ModelTarget
from nemo_evaluator_sdk.values import Model

# Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset.
input_spec = AgentEvalInputSpec(
    tasks=TasksetRef("default/geography-suite"),
    target=ModelTarget(
        model=Model(url="https://integrate.api.nvidia.com/v1", name="meta/llama-3.3-70b-instruct"),
    ),
)
```

When the job runs, the taskset reference is resolved like this:

- The **taskset revision** the ref names is loaded — the current one unless the ref pins a revision
  (see [Pin the taskset itself](#pin-the-taskset-itself)).
- Each member of that revision is loaded at the **revision pinned in the taskset**, not the task's
  current tip.
- Metric references on those members are hydrated into runnable metrics, the same as for inline tasks.
- Re-running the same taskset therefore evaluates the same content, even if a member has been
  republished since.

```text
# What is stored on the taskset (digests fixed at create/replace time)
geography-suite@1
  └─ tasks:
       - default/capital-of-france#aaa111...   ← revision 1 content
       - default/capital-of-japan#bbb222...

# Later: the member task publishes new content
capital-of-france tip → revision 2 (#ccc333...)

# Re-grade still uses the taskset pins, not the tip
JobRun A  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...
JobRun B  ──TasksetRef("geography-suite")──► load #aaa111..., #bbb222...   ← same content
```

To evaluate the *new* task content, publish a new taskset revision (for example
`tasksets.replace(...)`) so membership is re-resolved to the newer digests.

Submit this spec as the agent-evaluate job input; see
[Agent Evaluation](/documentation/evaluate-models/agent-eval) for the full run, target, and
results flow.

The inline form remains available for one-off tasks — swap `tasks=TasksetRef(...)` for
`tasks=[AgentEvalTaskInput(...), ...]`.

### Pin the taskset itself

A bare `TasksetRef` expands the taskset's current revision, so it follows the suite forward every
time the taskset is republished. Add a `#<tag-or-digest>` fragment to pin the grouping too:

```python
# The digest lives on the revision, not on the taskset record — revisions come back newest first.
current = tasksets.list_revisions("geography-suite").data[0]

# Follows the suite forward — the next `replace` is picked up on the next run.
tracking = TasksetRef("default/geography-suite")

# Frozen: this exact membership and content, regardless of later `replace` calls.
pinned = TasksetRef(f"default/geography-suite#{current.content_hash}")

# A tag works the same way, and can be moved deliberately when you bless a new suite.
blessed = TasksetRef("default/geography-suite#blessed")
```

What each form is stable against:

| | A member task republishes | The taskset is replaced |
|---|---|---|
| `TasksetRef("suite")` | unaffected | follows the new revision |
| `TasksetRef("suite#<digest>")` | unaffected | unaffected |

Every taskset revision pins its members by digest, so **neither** form is disturbed when a member
task publishes new content on its own — that is the guarantee stored membership buys you.

The two differ on `replace`. A bare ref tracks the taskset's own revisions, and members are
re-resolved on every write — so a `replace` can change *both* which tasks are named *and* the
content they resolve to, even when the submitted member names were identical. Pin the taskset when a
benchmark number has to stay comparable across that.

A fragment that no longer resolves fails the evaluation rather than falling back to the current
revision.

<Note>
A member's grader-only `reference` (held-out ground truth) is loaded from the pinned revision along
with the rest of its content, so a taskset-driven run grades against the ground truth that revision
fixed. Because `reference` is covered by the revision digest, changing it publishes a new revision —
a pin fixes the grading, not just the prompt.
</Note>

## Async usage

`AsyncNeMoPlatform` exposes the same surface; await each call.

```python
import asyncio

from nemo_platform import AsyncNeMoPlatform

async def main() -> None:
    client = AsyncNeMoPlatform(base_url="http://localhost:8080", workspace="default")
    page = await client.evaluator.tasks.list()
    for item in page.data:
        print(item.name)

asyncio.run(main())
```

## Workspaces and projects

Every method accepts an optional `workspace` argument that overrides the client's default workspace.
On create, an optional `project` argument associates the task or taskset with a project. When you
omit `workspace`, the client's configured workspace is used.

## REST API

The SDK resources are a thin client over the Evaluator plugin REST API, mounted under
`/apis/evaluator/v2/workspaces/{workspace}`:

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/tasks` | List tasks (paginated). |
| `POST` | `/tasks/{name}` | Create a task and publish revision 1. |
| `PUT` | `/tasks/{name}` | Replace a task's content and publish; creates it if absent. |
| `GET` | `/tasks/{name}` | Retrieve a task's current content. |
| `GET` | `/tasks/{name}/revisions` | List published revisions (paginated, newest first). |
| `GET` | `/tasks/{name}/revisions/{revision}` | Retrieve content as of a digest or tag. |
| `PUT` | `/tasks/{name}/tags/{tag}?revision=` | Point a tag at an existing revision. |
| `DELETE` | `/tasks/{name}` | Delete a task and all of its revisions. |
| `GET` | `/tasksets` | List tasksets (paginated). |
| `POST` | `/tasksets/{name}` | Create a taskset and publish revision 1. |
| `PUT` | `/tasksets/{name}` | Replace a taskset's membership and publish; creates it if absent. |
| `GET` | `/tasksets/{name}` | Retrieve a taskset's current membership. |
| `GET` | `/tasksets/{name}/revisions` | List published revisions (paginated, newest first). |
| `GET` | `/tasksets/{name}/revisions/{revision}` | Retrieve membership as of a digest or tag. |
| `PUT` | `/tasksets/{name}/tags/{tag}?revision=` | Point a tag at an existing revision. |
| `DELETE` | `/tasksets/{name}` | Delete a taskset and all of its revisions. |

### Edge Cases

`PUT` distinguishes its two outcomes by status: **`201`** when a new revision was published, and
**`200`** when the submitted content was already the current revision and nothing was cut. The rest:

| Case | Status | Behavior |
|------|--------|----------|
| `PUT` with new content | `201` | A revision is published and `latest` moves to it. |
| `PUT` with content identical to the current revision | `200` | Nothing is published; any tags in the body are still applied. |
| `PUT` with taskset members reordered | `200` | Membership is a set, so a reordering is not a content change. |
| `PUT` reverting to *older* content | `201` | The record genuinely changed, so it publishes a new ordinal rather than reusing the old one. |
| `PUT` re-resolving a member that has since republished | `201` | A taskset's identity is the exact revisions it names. |
| Two identical `PUT`s racing | `201` + `200` | One publishes; the other adopts its revision rather than cutting a duplicate. |
| `POST` on an existing name | `409` | Create is strict; use `PUT` to upsert. |
| `PUT` losing a race with a concurrent write | `409` | Retry against the current state. |
| Invalid metric reference (task) | `422` | Rejected at validation. |
| Missing or duplicate task reference (taskset) | `422` | Members must exist, and must resolve to distinct tasks. |
| Reserved or malformed tag name | `422` | `latest` cannot be moved by hand; a digest-shaped tag is refused. |
| Retrieving or deleting an unknown name or revision | `404` | Applies to both records and revisions. |
| `TasksetRef` pinning a revision that no longer resolves | job fails | Expansion refuses rather than falling back to the current revision. |
| `DELETE` on a task a taskset pins | `204` | Not prevented; the taskset's reference dangles and fails on read. |

## Related Topics

- [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) - Define and reuse the metrics that score a task
- [SDK Resources](/documentation/evaluate-models/sdk-resources) - Run and submit evaluations through the Evaluator plugin
- [Agent Evaluation](/documentation/evaluate-models/agent-eval) - How agent-eval tasks are executed and scored
