---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Observe Agents"
description: "Ingest, store, and query agent telemetry with NeMo Intake, including supported ingest paths, annotations, evaluator results, and review in NeMo Studio."
---

Observability on NeMo Platform is provided by **NeMo Intake**, the trace ingestion and query service
for agent telemetry, and optionally by [NeMo Studio](/documentation/studio), the web user interface
(UI) for browsing
that telemetry. Observability helps humans and agents understand what happened in an agent system:
which models were called, which tools were used, what inputs and outputs were produced, where
failures occurred, and what feedback or evaluation scores were attached.

## Before You Start

### Who This Is For

NeMo Intake is for anyone who owns, operates, evaluates, or optimizes an agent. It is written for
engineers wiring telemetry into an agent runtime, and for human and agent reviewers who need to
diagnose failures from traces, in a production system or in offline evaluation flows.

### Requirements

- A reachable ClickHouse database.
- A NeMo Platform environment with the `intake` service running.
- A telemetry source, such as an OpenTelemetry or OpenInference exporter, NeMo Agent Toolkit (NAT),
  NeMo Flow, NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or Agent
  Trajectory Interchange Format (ATIF) trajectories.
- Optional: [NeMo Studio](/documentation/studio) for the UI review flow.

You do not need to deploy your agents on NeMo Platform. To start, run the platform, post one
representative interaction, and confirm that it appears.

## How It Works

### What It Does

Intake normalizes agent telemetry across multiple ingestion paths into queryable spans and traces. It
preserves captured inputs and outputs, stores semantic fields such as model, provider, tool name,
status, token counts, cost, and errors, and keeps unhandled source attributes available for
debugging.

Studio reads the same Intake APIs to provide trace lists, span lists, trace detail, span detail, and
annotation review pages. The UI is a human review surface; Intake is the ingestion and storage
service.

### When to Use It

Use Intake when you need to:

- Feed the optimization loop with production traces, staged replays, seed interactions, or synthetic
  cases.
- Diagnose specific failures from concrete runs instead of reconstructing behavior from logs.
- Establish baselines before changing a prompt, model, tool, guardrail, or agent workflow.
- Standardize telemetry across teams so everyone shares one vocabulary, data model, and review
  surface.

Production traffic provides useful optimization evidence. New agents can start with staged traffic
or generated cases, then improve coverage as production traffic arrives.

Intake accepts three standardized ingest formats depending on your instrumentation:

| Format | Example use cases | Endpoint |
|--------|-------------------|----------|
| OpenTelemetry Protocol (OTLP) over HTTP using Protocol Buffers (OpenTelemetry generative AI or OpenInference semantic conventions) | NeMo Relay, LangChain Deep Agents | `/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces` |
| Chat completions | Importing raw logs, instrumenting a proxy server, custom logging | `/apis/intake/v2/workspaces/{workspace}/ingest/chat-completions` |
| ATIF | Running Harbor evaluations | `/apis/intake/v2/workspaces/{workspace}/ingest/atif` |

### Core Concepts and Data Model

Telemetry uses a three-level hierarchy:

- A **span** is one timed operation, such as a large language model (LLM) call, tool invocation,
  retrieval, guardrail, evaluator, or chain step.
- A **trace** is one end-to-end agent run, made of spans that share a trace ID.
- A **session** groups related traces, such as a multi-turn conversation or a multi-system evaluation.

Additional records attach signal to the same session or span:

- **Annotations** store post-hoc feedback, labels, notes, and metadata.
- **Evaluator results** store numeric, boolean, categorical, or text scores for a span.
- **Experiments** and **Evaluations** organize evaluation runs into leaderboard rollups for
  comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment
  rolls related Evaluations into one leaderboard. Refer to
  [Experiments](/documentation/evaluate-models/experiments).

The most useful instrumentation logs granular steps: every model call, tool call, final response, and
error. A practical test: the trace can answer _where did the system go wrong, and why?_

## Get Started

### Setup

The examples assume a running NeMo Platform reachable at `$NMP_BASE_URL`. Point at whatever you have,
whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`:

```bash
export NMP_BASE_URL=http://127.0.0.1:8080
export WORKSPACE=default
export NMP_STUDIO_URL="$NMP_BASE_URL/studio"
export NMP_ACCESS_TOKEN="$(nemo auth token)"
```

Set `NMP_STUDIO_URL` explicitly if Studio uses a different origin or path.
The examples use bearer-token authentication and stop on HTTP errors. For a local development
deployment with authentication disabled, omit the `Authorization` header and the
`NMP_ACCESS_TOKEN` command. Also omit `OTEL_EXPORTER_OTLP_HEADERS` from the OTLP example.

<Accordion title="Running from source (local development)">

If you are bringing the pieces up yourself from a repository checkout, make sure Docker Desktop or
the Docker daemon is running, then start the backend. Intake automatically provisions and reuses a
local ClickHouse container for the resolved NeMo data directory unless `NMP_INTAKE_CLICKHOUSE_URL`
explicitly points to an external instance.

1. **Backend services**: `intake` plus its `auth` (access checks) and `entities` (entity store)
   dependencies. `--port` defaults to `8080`; omit `uv run` if you installed the `nemo`
   command-line interface (CLI):

   ```bash
   uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080
   ```

1. **Studio** (optional, for the UI review flow). From the `web/` workspace, with the intake feature
   flag on and pointed at the backend:

   ```bash
   VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \
     pnpm --filter nemo-studio-ui start -- --host 127.0.0.1
   ```

</Accordion>

Confirm the Intake read path can reach ClickHouse:

```bash
curl --fail-with-body -i \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1"
```

A `200` response with an empty list is healthy. A `503` response means the Intake service is running
but cannot reach ClickHouse.

### First Workflow

Send one captured chat-completion interaction. This basic verification test does not require an
OpenTelemetry exporter:

```bash
export SESSION_ID="demo-session-001"
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "session_id": "'"$SESSION_ID"'",
        "provider": "example",
        "request": {
          "model": "example-model",
          "messages": [
            { "role": "user", "content": "Summarize this alert and identify the likely next action." }
          ]
        },
        "response": {
          "choices": [
            { "message": { "role": "assistant",
              "content": "This appears to be a developing incident. Verify the source, check related signals, then decide whether to escalate." } }
          ]
        }
      }'
```

### Confirm It Worked

Read the span back from Intake:

```bash
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10"
```

Then open `$NMP_STUDIO_URL/workspaces/$WORKSPACE/intake/traces`.

A successful result has the following characteristics:

- The API call returns your interaction: one span with `session_id` `demo-session-001`, model
  `example-model`, and the prompt and response you sent as its input and output.
- Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and
  response in the span tree.

The interaction is now queryable through the application programming interface (API) and reviewable
in the UI. The captured fields include session, trace, span, request, response, status, error,
feedback, and evaluator results. The remaining documentation uses this vocabulary.

## Common Workflows

### Send OTLP Traces from an Instrumented Agent

Use OTLP when your framework or collector already emits OpenTelemetry traces:

```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$NMP_ACCESS_TOKEN"
```

Then run the instrumented agent. Intake maps OpenInference and OpenTelemetry generative AI semantic
attributes onto span fields. You can then query model, tool, status, token, and error data.

### Send ATIF Trajectories

Use ATIF when your source system exports complete agent trajectories, including steps, agent
metadata, or final metrics:

```bash
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @trajectory.json
```

Each trajectory is stored as a structured session of spans. If the trajectory's top-level
`extra.verifier_result.rewards` is populated, Intake writes those as evaluator results automatically.
Stock Harbor output does not populate it. Harbor rewards are stored in a separate `reward.json`, so
you must enrich the output first. Refer to
[Capture Evaluator Results](#capture-evaluator-results) for the automatic and explicit paths.

### Add Feedback and Labels

Use annotations to attach review signal after a trace lands:

```bash
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "kind": "feedback", "session_id": "'"$SESSION_ID"'", "value": "negative" }'
```

Use `kind: "note"` for reviewer notes, `kind: "label"` for categorical or numeric labels, and
`kind: "metadata"` for structured key/value context.

### Capture Evaluator Results

Evaluator results attach a judge rating, pass or fail check, or similarity score to a span. You can
send them to Intake in two ways.

**Automatically, from ATIF.** Put a `verifier_result.rewards` object in the trajectory's **top-level**
`extra`, keyed by criterion (`{criterion: score}`). On ingest, Intake synthesizes an evaluator span
named `harbor.verifier` and writes one evaluator result per key onto it; a bare scalar
`extra.verifier_result.score` instead becomes a single `reward` result. Nothing else to send.

A stock Harbor `trajectory.json` does **not** carry this value. Harbor writes rewards to a separate
`reward.json`, so you must copy them into the ATIF `extra` before ingestion or use the explicit path
below. For example:

```json
"extra": { "verifier_result": { "rewards": { "solved": 1, "groundedness": 0.8 } } }
```

**Explicitly, through the API.** Post a result against a span and session. Unlike the ATIF automatic
path, which lands on the synthesized `harbor.verifier` span, this attaches to the exact `span_id` you
post. `data_type` picks which field carries the score:

| `data_type` | Where the score goes |
|-------------|----------------------|
| `NUMERIC` | A number in `value` |
| `BOOLEAN` | `value`, either `0` or `1` |
| `CATEGORICAL` | A label in `string_value` |
| `TEXT` | Free text in `string_value` |

For example, a numeric score:

```bash
curl --fail-with-body -X POST \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "span_id": "<span-id>",
        "session_id": "'"$SESSION_ID"'",
        "name": "faithfulness/v1",
        "data_type": "NUMERIC",
        "value": 0.82
      }'
```

Read them back for one span, or list and filter across the workspace by evaluator name, data type, or
value range:

```bash
curl --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/<span-id>/evaluator-results"
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8"
```

### Find Recurring Failures

Query traces or spans with filters, then group spans by session or trace to find repeated errors:

```bash
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=error&page=1&page_size=20"
curl -g --fail-with-body \
  -H "Authorization: Bearer $NMP_ACCESS_TOKEN" \
  "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=error"
```

In Studio, start from the trace list, filter to negative feedback or error status, then open trace
detail to inspect the span tree.

### Turn Evaluation Telemetry into a Leaderboard

Once you are ingesting evaluation runs, group them into an **Experiment** to compare them side by
side. Refer to [Experiments](/documentation/evaluate-models/experiments), or use the agent-assisted
`nemo-experiments-upload` skill to go from zero to a populated leaderboard.

## Operations

### Limits

- OTLP request bodies are capped at 5 MiB by default.
- Read endpoints default to a 30-day lookback when no time filter is supplied.
- Page size is capped at 1,000 records.
- Span data and trace-index data expire after 90 days.

Raise the OTLP body cap with `NMP_INTAKE_OTLP_MAX_BODY_BYTES` when larger batches are required. For
high-volume producers, prefer smaller export batches over very large single requests.

### Security and Access Control

All Intake endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`. The service
depends on platform auth and checks workspace access before ingesting or reading telemetry.

<Warning>

Do not send secrets, credentials, raw personally identifiable information (PII), or regulated data
unless your deployment, retention, and access-control policy permits it. Intake preserves request
and response payloads so reviewers can
diagnose behavior. This data is useful for debugging and important for data governance.

</Warning>

### Retention and Storage

ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the
tables on first use.

Span and trace-index tables have a 90-day time-to-live (TTL) setting. Annotations and evaluator
results are retained in their ClickHouse tables without that 90-day span TTL.

For production storage sizing, estimate span volume across the retained window, then size ClickHouse
for interactive reads over recent traces and periodic aggregate queries.

## Related Topics

- [Experiments](/documentation/evaluate-models/experiments): roll evaluation runs up into a comparable
  leaderboard built from this telemetry.
- [Optimize Agents](/documentation/agents/optimize-agents): use captured traces as the baseline for
  optimization.
- [Secure Agents](/documentation/agents/secure-agents): scan recent telemetry for sensitive data.
