---
title: "Agentic Benchmarks"
description: ""
---

Evaluate agent tool-calling capabilities using industry-standard benchmarks.

## Prerequisites

Before running agentic benchmarks, ensure you have:

1. **Workspace**: Have a workspace created. All resources (metrics, secrets, jobs) are scoped to a workspace.

2. **Model endpoint**: A deployed model to evaluate for tool-calling capabilities.

3. **API key secrets** *(for some benchmarks)*: Some BFCL benchmarks require external API keys.

```python
import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)
```

---

## Berkeley Function Calling Leaderboard (BFCL)

BFCL is a benchmark for evaluating language model tool-calling capabilities. Use this evaluation type to benchmark tool-calling tasks using the [Berkeley Function Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html).

View all available BFCL benchmarks (`system/bfclv3-*`) with a label filter:

```python
bfcl_system_benchmarks = client.evaluation.benchmarks.list(
    workspace="system", extra_query={"filter[labels.eval_harness]": "bfcl"}
)
print(bfcl_system_benchmarks)
```

BFCL benchmark evaluation requires a model for online evaluation.

Some BFCL benchmarks call external APIs that require API keys (for example, `system/bfclv3-exec-*` benchmarks). Create secrets for the API keys before referencing them in the job.

- RapidAPI (free tier; subscription required): [Yahoo Finance](https://rapidapi.com/sparior/api/yahoo-finance15), [Real‑Time Amazon Data](https://rapidapi.com/letscrape-6bRBa3QguO5/api/real-time-amazon-data), [Urban Dictionary](https://rapidapi.com/community/api/urban-dictionary), [COVID‑19](https://rapidapi.com/api-sports/api/covid-193), [Time Zone by Location](https://rapidapi.com/BertoldVdb/api/timezone-by-location)
- Direct APIs: [ExchangeRate‑API](https://www.exchangerate-api.com), [OMDb](http://www.omdbapi.com/apikey.aspx), [Geocode](https://geocode.maps.co/)

```python
client.secrets.create(
    workspace=workspace, name="rapid_api_key_secret", value="<your RapidAPI key>"
)
benchmark_params = {
    "rapid_api_key": "rapid_api_key_secret",
}
```

<Tabs>

<Tab title="Job">

```python
from nemo_platform.types.evaluation import SystemBenchmarkOnlineJobParam

job = client.evaluation.benchmark_jobs.create(
    spec=SystemBenchmarkOnlineJobParam(
        benchmark="system/bfclv3-live-simple",
        model={
            "url": "<your-nim-endpoint>/v1",
            "name": "nvidia/llama-3.3-nemotron-super-49b-v1",
        },
        benchmark_params={},
    )
)
```

</Tab>
<Tab title="Result">

Accuracy of tool call predictions

- Score name: tool-calling-accuracy
- Value range: 0.0–1.0

```json
{
  "scores": [
    {
      "name": "tool-calling-accuracy",
      "score_type": "range",
      "count": 1,
      "nan_count": 0,
      "sum": 1.0,
      "mean": 1.0,
      "min": 1.0,
      "max": 1.0,
      "std_dev": 0.0,
      "variance": 0.0
    }
  ]
}
```

</Tab>

</Tabs>
---

## Job Management

After creating a job, navigate to [Benchmark Job Management](/documentation/evaluator-benchmark-reference/benchmarks/job-management) to oversee its execution and monitor progress.

---

<Note>

- [Agent Configuration](/documentation/evaluate-models/metrics/agent-configuration) - Use agents (generic or NAT) as targets in online evaluation and benchmark jobs
- [Agentic Evaluation Metrics](/documentation/evaluate-models/metrics/agentic-metrics) - Detailed metric documentation for evaluating agentic workflows
- [Managing Secrets](/documentation/get-started/core-concepts/manage-secrets) - Store API keys for external APIs
- [Evaluation Results](/documentation/evaluator-benchmark-reference/benchmarks/benchmark-results) - Understanding and downloading results

</Note>
