# Optimize Agents

Use the Agent Optimizer to analyze platform-managed agents and act on
improvement suggestions. The optimizer inspects deployed agent configs,
available models, optional skill repositories, evaluation baselines, and prior
optimizer snapshots. It then saves suggestions that can be reviewed from Studio,
the CLI, or your coding-agent workflow.

This page covers the hero flow: establish a baseline, generate optimization
suggestions, apply a candidate change to a sibling agent, and review evaluation
results before promotion.

## What the Optimizer Checks

| Suggestion type | Signal | Result |
|-----------------|--------|--------|
| Model optimization | An agent uses a single frontier model where a smaller model or route split may preserve quality at lower cost | Suggests a model swap or Switchyard random-routing virtual model |
| Skill optimization | The agent uses skills and has an evaluation suite | Suggests running `nemo agents optimize-skills` to improve skill files and keep changes that pass evaluation |
| Prompt optimization | The agent has an optimization config and baseline dataset | Suggests `nemo agents optimize run` for NAT prompt or parameter tuning |
| New model scan | Difference between the current model list and the previous optimizer snapshot | Suggests evaluating or auditing newly available models |

Optimizer state is stored in the `nemo-agent-optimizer` fileset:

- `optimizer_suggestions.jsonl`: one suggestion per line, including applied state.
- `optimizer_snapshot.json`: model and agent names from the latest run.

Security-oriented suggestions such as missing guardrails, PII exposure, or
leaked secrets are covered in [Secure Agents](security.md).

## Prerequisites

Before running the Studio flow, make sure you have:

1. A running {{platform_name}} deployment with Studio available.
1. The Agents feature enabled.
1. A workspace with at least one model provider and discovered model entities.
1. At least one platform-managed agent with an LLM in its config.
1. An evaluation baseline before promoting a candidate agent.

For local development from this repository, install the plugin commands once:

```bash
uv pip install -e packages/nemo_platform_plugin -e plugins/nemo-agents
```

If you need a demo agent, start the platform and create the ReAct example agent:

```bash
nemo services run
```

In another terminal:

```bash
export NMP_BASE_URL=http://127.0.0.1:8080
cd plugins/nemo-agents

printf '%s' "$NVIDIA_API_KEY" | nemo secrets create ngc-api-key --from-file -
nemo inference providers create nvidia-build \
  --host-url https://integrate.api.nvidia.com \
  --api-key-secret-name ngc-api-key
nemo wait inference provider nvidia-build

nemo agents create \
  --name react-agent \
  --agent-config examples/react-agent/react-agent.yml
nemo agents deploy --agent react-agent
nemo agents deployments wait --agent react-agent
```

The example agent uses `nvidia-nemotron-3-nano-30b-a3b`, so it can produce a model optimization suggestion when the workspace model catalog contains a smaller compatible model.

## Optimize with Switchyard Routing

Switchyard is an inference middleware plugin that can route requests across
backend models. The common optimization pattern is to create a virtual model
with a strong model and a weaker, cheaper model, then evaluate whether the route
split preserves application quality.

```bash
uv sync
nemo services run
```

Create a virtual model that routes most traffic to the strong model and a small
sample to the weak model:

Run `nemo models list` first and replace the placeholders below with model
entity names from your workspace that use the `OPENAI_CHAT` backend format.

```bash
nemo inference virtual-models create routed-agent-model \
  --workspace default \
  --models '[
    {"model":"default/<strong-model-entity>","backend_format":"OPENAI_CHAT"},
    {"model":"default/<weak-model-entity>","backend_format":"OPENAI_CHAT"}
  ]' \
  --request-middleware '[{
    "name":"nemo-switchyard",
    "config_type":"random_routing",
    "config":{
      "strong":{"model":"default/<strong-model-entity>"},
      "weak":{"model":"default/<weak-model-entity>"},
      "strong_probability":0.8,
      "enable_stats":false
    }
  }]'
```

Before wiring the virtual model to an agent, smoke-test the route by making
several minimal chat-completions calls and checking the returned model name. The
observed split should roughly match `strong_probability`.

## Optimize Skills

Use skill optimization when the application agent depends on local skill files
and has an evaluation suite:

```bash
nemo agents optimize-skills --config .agent-improver.yml
```

The loop runs evaluations, analyzes failures, lets the coding agent edit only
the configured skills directory, reruns verification, and keeps the change only
when the evaluation result improves. Use `--open-pr` when you want the loop to
prepare a reviewable branch.

## Run the Studio Flow

1. Open Studio and select the workspace.
1. Go to **Agents > Optimizations**.
1. Click **Run optimizer**.
1. Review the generated suggestion tiles.
1. If the apply feature is enabled, click **Apply suggestion** on a model optimization tile.
1. Choose or confirm the evaluation config when prompted.
1. Wait for the sibling deployment and evaluation job to finish.
1. Open **View details** from the evaluation row to inspect the validation job.

When you apply a model optimization suggestion, Studio performs a safe validation flow:

1. Seeds `<agent-name>-eval` with a sample `react-eval.yml` and `react-eval-data.json` if that fileset does not already contain them.
1. Creates a sibling agent whose config swaps the source model for the suggested smaller model.
1. Deploys the sibling agent.
1. Submits an agent evaluation job against the sibling.
1. Writes evaluation outputs to `<sibling-agent-name>-eval-out`.
1. Marks the suggestion as applied in `optimizer_suggestions.jsonl`.

The original agent is not edited in place. Promote the candidate only after reviewing the evaluation scores and output artifacts.

## Inspect Saved Results

Use the Files service to inspect what the optimizer saved:

```bash
nemo files list --fileset nemo-agent-optimizer

nemo files download \
  --fileset nemo-agent-optimizer \
  --remote-path optimizer_suggestions.jsonl \
  -o optimizer_suggestions.jsonl

nemo files download \
  --fileset nemo-agent-optimizer \
  --remote-path optimizer_snapshot.json \
  -o optimizer_snapshot.json
```

Telemetry is optional. If agents use the `nemo_files` telemetry exporter, trace files are written to `nemo-agent-telemetry`, and the optimizer samples the largest JSONL file:

```bash
nemo files list --fileset nemo-agent-telemetry
```

## Run Prompt and Parameter Tuning

The `nemo agents optimize run` command runs the NAT optimizer path for
parameter or prompt tuning. Use it when you already have a NAT optimization YAML
and want to run `nat optimize` through the Agents plugin.

For the ReAct example:

=== "CLI"

    ```bash
    nemo agents optimize run \
      --optimize-config plugins/nemo-agents/examples/react-agent/react-optimize.yml \
      --agent react-agent
    ```

=== "Python SDK"

    ```python
    import os
    from pathlib import Path

    from nemo_agents_plugin.jobs.optimize_agent import OptimizeAgentJob
    from nemo_platform import NeMoPlatform
    from nemo_platform_plugin.scheduler import NemoJobScheduler

    WORKSPACE = "default"
    optimize_config = Path("plugins/nemo-agents/examples/react-agent/react-optimize.yml")

    client = NeMoPlatform(
        base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
        workspace=WORKSPACE,
    )

    result = NemoJobScheduler().run_local(
        OptimizeAgentJob,
        {
            "optimize_config": str(optimize_config),
            "agent": "react-agent",
            "workspace": WORKSPACE,
        },
        workspace=WORKSPACE,
        sdk=client,
    )
    print(result)
    ```

When `--agent` is a platform-managed agent name, the job fetches the stored agent config, merges it with the optimization config, injects the Inference Gateway URL, and runs trials locally. When `--agent` is a raw HTTP endpoint, the endpoint is treated as an opaque remote service, so local parameter sweeps do not change the remote agent behavior.

For the NeMo team agentic-use benchmark flow:

=== "CLI"

    ```bash
    python tests/agentic-use/build_aut_eval_dataset.py --all

    nemo agents optimize run \
      --optimize-config tests/agentic-use/aut-optimize.yml \
      --agent nemo-agent
    ```

=== "Python SDK"

    ```python
    import os
    import subprocess
    from pathlib import Path

    from nemo_agents_plugin.jobs.optimize_agent import OptimizeAgentJob
    from nemo_platform import NeMoPlatform
    from nemo_platform_plugin.scheduler import NemoJobScheduler

    WORKSPACE = "default"
    optimize_config = Path("tests/agentic-use/aut-optimize.yml")

    subprocess.run(
        ["python", "tests/agentic-use/build_aut_eval_dataset.py", "--all"],
        check=True,
    )

    client = NeMoPlatform(
        base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
        workspace=WORKSPACE,
    )

    result = NemoJobScheduler().run_local(
        OptimizeAgentJob,
        {
            "optimize_config": str(optimize_config),
            "agent": "nemo-agent",
            "workspace": WORKSPACE,
        },
        workspace=WORKSPACE,
        sdk=client,
    )
    print(result)
    ```

Use `tests/agentic-use/passrate_token_policy_gate.py` for the canonical promotion gate. The optimizer's runtime metric is only a tuning signal for that local experiment.

## Troubleshooting

**No suggestions appear.** Confirm the workspace has agents, model entities, and a model catalog entry smaller than the agent's current model. New-model suggestions require a previous optimizer snapshot, so they do not appear on the first run.

**The model evaluation fails.** Confirm the judge model in the seeded eval config is available through the workspace Inference Gateway. You can replace the files in `<agent-name>-eval` with your own evaluation config and dataset.

**Data safety suggestions do not appear.** Telemetry is optional. The optimizer only scans `nemo-agent-telemetry` when that fileset exists and contains JSONL trace files.

## Next steps

- [Agent overview](index.md): Review how platform-managed agents are registered, deployed, invoked, evaluated, and optimized.
- [Agent evaluation](../evaluator/metrics/agent-configuration.md): Configure agents as online evaluation targets and choose the right agent response mapping.
- [CLI reference](../cli/reference.md): Look up complete command options and global CLI flags for scripted workflows.
