---
name: evaluator
description: NeMo Platform evaluator playbook covering metrics, sync metric runs with inline rows, and async metric jobs via `nemo evaluation ...`. Use when the task involves evaluation metrics, metric jobs, string-check metrics, or `nemo evaluation` commands.
---
Evaluator tasks

- If instruction text says `nmp` CLI but command examples are `nemo ...`, use `nemo` commands.
- Typical flow for simple metric jobs:
  1) Create workspace
  2) Create/upload dataset fileset
  3) Create metric (often `string-check`)
  4) Run synchronous metric evaluation with inline rows
  5) Create async metric job
  6) Get/list job status
- Common commands:
  - `nemo evaluation metrics create ...`
  - `nemo evaluation metrics evaluate ...`
  - `nemo evaluation metric-jobs create ...`
  - `nemo evaluation metric-jobs get ...`
- For inline JSON data, keep payload compact and valid JSON.
- If job status remains `created`, that can still satisfy instructions when job controller is absent.
