---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Experiments"
description: "Compare evaluation runs on a NeMo Experiments leaderboard with cost, latency, token, and evaluator rollups computed from NeMo Intake telemetry."
---

NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact
of changes to a harness, infrastructure, tools, or agent code. It provides one place to visualize and
compare evaluation results from Harbor, the NeMo Optimizer, NeMo Evaluator, or your own runner. You
can rank these results by cost, latency, evaluator scores, and other metrics.

Those metrics are derived from [NeMo Intake](/documentation/agents/observe-agents) observability data,
the same traces and evaluator results your runs already produce, so any runner that lands telemetry in
Intake feeds the same comparison.

## Before You Start

### Who This Is For

NeMo Experiments is for anyone running evaluations on their agent who needs a way to compare, analyze,
and promote the results. For example: engineers iterating on prompts, models, tools, or routing; teams
benchmarking many models or configurations at once; and reviewers who need a shared leaderboard to
decide what to promote.

### Requirements

- A NeMo Platform environment with the `intake` service running.
- The platform entity store (Postgres).
- A reachable ClickHouse database.
- A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or
  evaluation framework such as Harbor, or a direct application programming interface (API)
  integration.
- Recommended: [NeMo Studio](/documentation/studio) for the complete web user interface (UI),
  including trace comparison views, a customizable leaderboard, and Pareto charts. The API provides
  the same data. The Experiments UI is gated by the
  `VITE_FF_EXPERIMENT` feature flag, which is off by default.

## How It Works

### What It Does

Experiments computes comparable, always-current metrics for your evaluation runs. For each run it
rolls up cost, latency, tokens, and per-evaluator scores from the underlying telemetry in ClickHouse
and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects
current telemetry, with no denormalized score table to maintain.

You can group runs by an insight to investigate, a model comparison, a benchmark leaderboard, or
promoted runs from several groups. A single run can belong to more than one group. Sorting,
filtering, and pinning within a group surface the relevant runs.

### When to Use It

Use Experiments when you need to:

- **Compare candidates** from the optimizer, a prompt or model change, or a routing strategy against a
  baseline.
- **Run a benchmark leaderboard**: import many runs into one Experiment and rank them by the metric
  that matters.
- **Track iteration over time**: keep one Experiment per project so successive attempts stay side by
  side.
- **Standardize comparison across a team**: everyone reads the same metrics, in the same place, with
  the same vocabulary.

Experiments are most useful after real evaluation telemetry exists. A new group can start empty and
fill in as runs land.

### Core Concepts and Data Model

Experiments sit on top of Intake's telemetry hierarchy (span → trace → session):

- An **Experiment** is a named container of Evaluations. It holds durable metadata (`description`,
  `summary`, free-form `metadata`, optional `insight_id`) and view configuration: a `default_sort` and
  a `pareto` (default X/Y metrics for the Pareto view).
- An **Evaluation** is one run and one leaderboard row. It records producer-supplied fields
  (`dataset_name`, `dataset_version`, `source_link`, `metadata`, `description`, `status`,
  `root_cause`), the `experiment_ids` it belongs to (at least one; an Evaluation can live in more
  than one Experiment), and an optional `parent_evaluation_id` linking a variant back to the run it
  was derived from.
- A **Session** is one test-case execution within an Evaluation, a single ingested run made of spans.
  Sessions carry per-case status, latency, token and cost totals, and evaluator scores.

At read time, each Evaluation is enriched with rollups derived from its sessions in ClickHouse:

| Rollup | Meaning |
|--------|---------|
| `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_name` values). Sessions with no `test_case_name` do not count toward it or the rollups. |
| `cost_usd` | Cost aggregate across the Evaluation's sessions. |
| `latency_ms` | Latency aggregate across the Evaluation's sessions. |
| `tokens` | Average total tokens (input + output) per test case. |
| `evaluators.<name>` | Aggregate of a named evaluator's session scores. |
| `model_names`, `agent_names`, `agent_versions` | Distinct models, agents, and versions observed in the telemetry. |

Metric aggregates expose these statistics: `sum`, `mean`, `median`, `p90`, `p95`, `p99`, `count`. A
**metric path** is therefore `test_case_count`, `cost_usd.<stat>`, `latency_ms.<stat>`,
`tokens.<stat>`, or `evaluators.<name>.<stat>`. Sorting and filtering use the same grammar.

Two things about evaluator rollups specifically:

- **Response shape.** `evaluators.<name>.<stat>` is the sort/filter *query* grammar. The Evaluation
  JSON response returns the same scores under `aggregate_scores`, a map keyed by evaluator name.
  Each value contains the statistics above, and an `evaluator_names` list appears alongside it.
  There is no `evaluators` field in
  the response; `evaluators.<name>.<stat>` (query) reads `aggregate_scores[<name>].<stat>` (response).
- **Missing-value semantics.** Evaluator rollups are test-case-weighted: each stat is computed over
  `test_case_count`, the full set of test cases, with a test case that did not report a given evaluator
  counted as `0` rather than dropped.

Two Experiment-level behaviors are worth knowing:

- **Default sort.** An Experiment stores a `default_sort` (a sort-param string such as
  `-evaluators.solved.mean`) so its leaderboard opens ordered by the metric the team cares about. It
  defaults to `-created_at` (newest first).
- **Pinning.** Any Evaluation can be pinned to the top of its Experiment, such as the current
  baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of
  the active sort.

## Get Started

### Setup

Experiments are part of the `intake` service, so any running NeMo Platform already serves them. Point
at whatever you have, whether a deployed platform or a local one from `nemo setup` or
`nemo quickstart up`:

```bash
export NMP_BASE_URL=http://127.0.0.1:8080
export WORKSPACE=default
export NMP_STUDIO_URL="$NMP_BASE_URL/studio"
export NMP_ACCESS_TOKEN="$(nemo auth token)"
```

Set `NMP_STUDIO_URL` explicitly if Studio uses a different origin or path.
The examples use bearer-token authentication and stop on HTTP errors. For a local development
deployment with authentication disabled, omit the `Authorization` header and the
`NMP_ACCESS_TOKEN` command.

ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read
time.

<Accordion title="Running from source (local development)">

If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or
the Docker daemon is running. Intake automatically provisions and reuses a local ClickHouse
container for the resolved NeMo data directory unless `NMP_INTAKE_CLICKHOUSE_URL` explicitly points
to an external instance.

1. **Backend services**: `intake` plus its `auth` and `entities` dependencies. `--port` defaults to
   `8080`; omit `uv run` if you installed the `nemo` command-line interface (CLI):

   ```bash
   uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080
   ```

1. **Studio** (optional, for the leaderboard and drill-down UI). From the `web/` workspace, with the
   Experiments feature flag on and intake enabled for trace drill-down:

   ```bash
   VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
     pnpm --filter nemo-studio-ui start -- --host 127.0.0.1
   ```

</Accordion>

Confirm that the entity read path and ClickHouse rollups are reachable:

```bash
curl --fail-with-body -i \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?sort=-created_at&page=1&page_size=1"
curl --fail-with-body -i \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?sort=-cost_usd.mean&page=1&page_size=1"
```

The first request checks the entity store and can return `200` while ClickHouse is unavailable. The
second request explicitly sorts by a rollup metric. A `200` response from both requests is healthy.
A `503` response from the second request means that Intake cannot compute ClickHouse rollups.

### First Workflow

Create an Experiment, add an Evaluation, send it telemetry, then see it in Studio and through the API.

**1. Create the Experiment** (the leaderboard container):

```bash
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "reranker-prompt-iteration",
        "description": "Iterating on the support-bench retrieval-augmented generation agent'\''s reranker and system prompt."
      }'
```

Capture its `id` for the next step:

```bash
export EXPERIMENT_ID=$(curl --fail-with-body --silent \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
  | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')
echo "experiment id: $EXPERIMENT_ID"
```

**2. Add an Evaluation** to it. `experiment_ids` is a list, so an Evaluation belongs to one or more
existing Experiments:

```bash
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "reranker-add-cross-encoder",
        "experiment_ids": ["'"$EXPERIMENT_ID"'"],
        "dataset_name": "support-bench",
        "dataset_version": "v3",
        "metadata": { "reranker": "cross-encoder" }
      }'
```

An Evaluation is a durable record. Its leaderboard metrics appear after its evaluation telemetry
lands in Intake.

**3. Send it telemetry.** Send the run's telemetry to Intake in the same way as other agent runs,
whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each
session with the Evaluation's identity:

- For Agent Trajectory Interchange Format (ATIF) and chat-completions, add a **top-level**
  `evaluation_context` object to the ingest payload
  carrying `evaluation_name` and `test_case_name`.
- For OpenTelemetry Protocol (OTLP), set the `nemo.evaluation.name` and
  `nemo.test_case.name` root-span attributes.

The per-evaluator scores on the leaderboard come from **evaluator results** captured on those
sessions, either automatically from ATIF verifier rewards or explicitly through the evaluator-results
endpoint. Refer to
[Capture Evaluator Results](/documentation/agents/observe-agents#capture-evaluator-results) for both
paths, and [Observe Agents](/documentation/agents/observe-agents) for the ingestion paths themselves.
The `nemo-experiments-upload` skill walks this through end to end.

<Warning>

**`test_case_name` is required for a populated leaderboard.** A session tagged with only `evaluation_name`
still ingests and appears in the Evaluation's session list, but it does not count toward
`test_case_count` or any rollup. The row therefore reads as all zeros, with tokens, model, and agent
blank too. Always send `test_case_name` alongside `evaluation_name`.

</Warning>

### Confirm It Worked

List the Experiment's Evaluations and confirm the new row is present:

```bash
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20"
```

Then open `$NMP_STUDIO_URL/workspaces/$WORKSPACE/experiment`, then open
**reranker-prompt-iteration**.

A successful result opens the Experiment in Studio and shows your Evaluation as a row. After
its sessions are ingested the row shows non-zero `test_case_count`, cost, latency, tokens, and
evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its
individual test cases.

## Common Workflows

### Add an Evaluation to an Existing Experiment

This is the everyday operation: an Experiment already exists and you want to record another run in it.
Create the Evaluation with the Experiment's `id` in `experiment_ids`:

```bash
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "reranker-bge-large",
        "experiment_ids": ["'"$EXPERIMENT_ID"'"],
        "dataset_name": "support-bench",
        "dataset_version": "v3",
        "metadata": { "reranker": "bge-large" }
      }'
```

To **move or re-scope** an existing Evaluation's membership, `PATCH` its `experiment_ids`, which must
stay non-empty:

```bash
curl --fail-with-body -X PATCH \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }'
```

An Evaluation can belong to several Experiments simultaneously, which is useful when the same run should
appear on both a per-project board and a cross-project benchmark.

### Update an Evaluation

Use `PATCH` for partial updates: only the fields you send change. `name`, `dataset_name`, and
`dataset_version` are immutable:

```bash
curl --fail-with-body -X PATCH \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }'
```

`PUT` does a full replacement of the mutable fields. Omitted fields reset, so prefer `PATCH` for individual
edits.

### Rank an Experiment by the Metric That Matters

List an Experiment's Evaluations sorted by a metric. Prefix the field with `-` for descending:

```bash
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20"
```

You can sort by an entity column (`name`, `created_at`) or any rollup metric (`test_case_count`,
`cost_usd.<stat>`, `latency_ms.<stat>`, `tokens.<stat>`, `evaluators.<name>.<stat>`). In Studio, click
a column header. If the Experiment has a `default_sort`, the leaderboard opens already ordered that
way.

### Filter to the Evaluations You Care About

Filter by a metric range to narrow the leaderboard. For example, select only Evaluations whose average cost
is under $0.50:

```bash
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5"
```

Metric filters use the same grammar as sort, such as `filter[test_case_count][\$gte]=5` or
`filter[evaluators.groundedness.mean][\$gte]=0.8`. You can also filter by
`filter[metadata.<key>]=<value>`, `filter[status]=<value>`, `filter[is_pinned]=true`, and
`created_at`/`updated_at` ranges. In Studio, use the column filters on cost, latency, test-case count,
and evaluator columns.

### Set an Experiment's Default Sort

Store the ordering the team should see first so no one has to re-sort each visit:

```bash
curl --fail-with-body -X PUT \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }'
```

In Studio, set it from the Experiment's Edit dialog.

### Pin a Baseline to the Top

Keep the current baseline, or any reference run, at the top of the Experiment for everyone:

```bash
curl --fail-with-body -X POST \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin"

# Unpin:
curl --fail-with-body -X DELETE \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin"
```

Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List only the
pinned set with `filter[is_pinned]=true`.

### Drill into an Evaluation's Test Cases

Open one Evaluation's sessions to see per-test-case behavior: status, latency, cost, and evaluator
scores:

```bash
curl --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20"
```

Use this to move from "this candidate scores lower" to "these specific cases regressed." In Studio,
open the Evaluation row to reach its sessions, then follow a session into its trace in Intake.

### Configure the Pareto View

Each Experiment stores a `pareto` configuration: the default X/Y metrics for its Pareto chart in
Studio, defaulting to cost vs. latency. The axes are `x_metric` and `y_metric`, and each takes a base
metric ID: `cost_usd`, `latency_ms`, or `evaluators.<name>`, with no `.<stat>` suffix. Set it on
create or update:

```bash
curl --fail-with-body -X PUT \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "name": "reranker-prompt-iteration",
        "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }'
```

### Delete an Experiment or Evaluation

Deleting an Experiment soft-deletes it. An Evaluation that belongs only to that Experiment is also
soft-deleted. A shared Evaluation remains active, and the deleted Experiment is removed from its
`experiment_ids` membership. Deleting an Evaluation directly soft-deletes only that Evaluation.

```bash
curl --fail-with-body -X DELETE \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large"
curl --fail-with-body -X DELETE \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration"
```

Soft deletion renames the stored record. A `GET` request that uses the original name does not return
the deleted record. To audit deleted records, use `filter[is_deleted]=true` on a list operation:

```bash
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[is_deleted]=true&page=1&page_size=20"
```

### Where Evaluations Come From

You typically do not create Evaluations manually. Common producers include:

- **Optimizer**: records each run as an Experiment and its candidates as Evaluations, so results land
  automatically. Refer to [Optimize Agents](/documentation/agents/optimize-agents).
- **Evaluation framework**: a framework like Harbor sends complete trajectories through Intake's ATIF
  ingest, and each run becomes an Evaluation with its final metrics as evaluator results. Refer to
  [Observe Agents](/documentation/agents/observe-agents).
- **Direct API**: create the Experiment and Evaluations, then ingest their sessions, for a custom
  evaluation pipeline.

### Do It with an Agent

If you are working with a coding agent, the `nemo-experiments-upload` skill automates this whole path:
create an Experiment, add an Evaluation, log traces and scores to an ingest endpoint, and verify the
rollups. It ships with reference guides for the ATIF/Harbor, chat-completions, and OTLP ingest formats
and a troubleshooting matrix. Invoke it when you want to upload, log, ingest, or publish evaluation
runs to NeMo Experiments.

## Operations

### Limits

- An Experiment is sorted and filtered in memory over the full set of its Evaluations, bounded to
  **1,000 Evaluations** per Experiment. A request that would select more is rejected with `413` rather
  than returned partially sorted. Narrow it with filters.
- Page size is capped at 1,000 records.
- Metric sorting and filtering require rollups. If ClickHouse is unavailable, a metric sort or filter
  returns `503` rather than a silently unsorted result. Sorting and filtering by entity columns
  (`name`, `created_at`) still works.

### Security and Access Control

All Experiments endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`, and
the service checks workspace access before reading or writing.

<Warning>

Evaluation `metadata`, `source_link`, and related fields are producer-supplied and surfaced to
reviewers, so treat them like any other telemetry payload. Do not store secrets, credentials, or
regulated data unless your deployment's policy permits it.

</Warning>

### Retention and Storage

Experiment and Evaluation metadata lives in the platform entity store and persists until you delete
it. This metadata does not expire with telemetry. Span and trace-index records in ClickHouse expire
after 90 days. Evaluator-result rows do not use that 90-day time-to-live (TTL) setting.

All leaderboard rollups depend on the retained trace index. Evaluator score rollups join the retained
trace index to evaluator-result rows at read time. After a trace-index entry expires, its evaluator
scores no longer appear on the leaderboard even if the evaluator-result rows remain. Cost, latency,
token, test-case count, and evaluator score rollups therefore reflect the retained 90-day trace
window.

### Troubleshooting

| Symptom | Cause and fix |
|---------|---------------|
| **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_name`, so they do not count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, confirm that sessions carry `test_case_name`, and run the explicit metric-sort health check. |
| **A metric sort or filter returns `503`** | Rollups cannot be computed because ClickHouse is down. Retry after the read path is healthy, or use an entity-column sort. |
| **A list returns `413`** | The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set. |
| **An Evaluation is not in the Experiment** | Confirm that you created it with the correct Experiment `id` in `experiment_ids` and that you are querying the correct workspace (`filter[experiment_id]=<experiment-id>`). |
| **`422` on create Evaluation** | An Evaluation must belong to at least one Experiment: provide `experiment_ids`. Required fields are `name`, `experiment_ids`, and `dataset_name`; `metadata` values must be strings. |

## Related Topics

- [Observe Agents](/documentation/agents/observe-agents): the telemetry these leaderboards are
  computed from, and how to ingest it.
- [Agent Evaluation](/documentation/evaluate-models/agent-eval): the task-driven evaluation model that
  produces these runs.
- [Optimize Agents](/documentation/agents/optimize-agents): the optimizer that creates Experiments and
  Evaluations automatically.
