## Description: 
Evaluate models, datasets, and agents with the NeMo Evaluator plugin for metric selection, SDK checks, platform jobs, and result retrieval. 

This skill is ready for commercial/non-commercial use. 

## Owner
NVIDIA 

### License/Terms of Use: 
Apache 2.0 
## Use Case: 
Developers and engineers evaluating LLM and agent performance using NeMo Platform's evaluation framework for metric selection, job submission, and result retrieval. 

### Deployment Geography for Use: 
Global 

## Requirements / Dependencies: 
**Requires API Key or External Credential:** [Yes] 
**Credential Type(s):** [API key] 

Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. 

## Known Risks and Mitigations: 
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. 
Mitigation: Review and scan skill before deployment. 

## Reference(s): 
- [Agent Evaluation](references/agent-evaluation.md) 
- [API Auth](references/api-auth.md) 
- [Evaluation Shapes](references/evaluation-shapes.md) 
- [Execution](references/execution.md) 
- [LLM Judge](references/llm-judge.md) 
- [Metric Selection](references/metric-selection.md) 
- [Resources](references/resources.md) 
- [Troubleshooting](references/troubleshooting.md) 

## Skill Output: 
**Output Type(s):** [Shell commands, API Calls, Analysis] 
**Output Format:** [Markdown with inline bash code blocks] 
**Output Parameters:** [1D] 
**Other Properties Related to Output:** [None] 

## Evaluation Agents Used: 
- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) 
- Codex (`openai/openai/gpt-5.5`) 

## Evaluation Tasks: 
1 evaluation task (1 positive) from skill-evaluator-dataset-snapshot, evaluated in k8s-sandbox environment with 1 attempt per task. 

## Evaluation Metrics Used: 
Reported benchmark dimensions: 
- Security: Is it safe to use? 
- Correctness: Is the answer correct? 
- Discoverability: Was the right skill loaded when needed? 
- Effectiveness: Did the skill help complete the task? 
- Efficiency: Did it avoid wasted tool or skill usage? 

Underlying evaluation signals used in this run: 
- `security`: Unsafe operations, secret leakage, and unauthorized access. 
- `skill_execution`: Whether the expected skill was found and executed. 
- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use. 
- `accuracy`: Final-answer correctness against the reference answer. 
- `goal_accuracy`: Whether the user's goal was achieved. 
- `behavior_check`: Whether the expected workflow behavior was followed. 

## Evaluation Results: 
| Measure | Claude Code (Baseline → Skill) | Codex (Baseline → Skill) |
|---|---:|---:|
| Overall | 45% → 95% (+50) | 52% → 90% (+38) |
| Security | 100% → 100% (±0) | 50% → 100% (+50) |
| Correctness | 40% → 100% (+60) | 60% → 80% (+20) |
| Discoverability | 44% → 100% (+56) | 50% → 88% (+38) |
| Effectiveness | 10% → 75% (+65) | 75% → 85% (+10) |
| Efficiency | 30% → 100% (+70) | 25% → 100% (+75) |

## Skill Version(s): 
acfad8e4 (source: git SHA, committed 2026-08-10) 

## Ethical Considerations: 
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. 

(For Release on NVIDIA Platforms Only) 
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). 
