Metadata-Version: 2.4
Name: nvidia-crfm-helm
Version: 25.9
Summary: NVIDIA: Benchmark for language models - Fork of Stanford CRFM HELM
Home-page: https://github.com/stanford-crfm/helm
Author: Stanford CRFM, NVIDIA
Author-email: contact-crfm@stanford.edu
License: Apache License 2.0
Project-URL: Original Project, https://github.com/stanford-crfm/helm
Project-URL: Original Documentation, https://crfm.stanford.edu/helm
Keywords: language models benchmarking nvidia fork stanford crfm helm
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: License :: OSI Approved :: Apache Software License
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cattrs~=22.2
Requires-Dist: colorlog~=6.9
Requires-Dist: dacite~=1.6
Requires-Dist: importlib-resources~=5.10
Requires-Dist: Mako~=1.2
Requires-Dist: numpy<3,~=1.26
Requires-Dist: pandas~=2.0
Requires-Dist: pyhocon~=0.3.59
Requires-Dist: retrying~=1.3
Requires-Dist: spacy~=3.5
Requires-Dist: srsly>=2.4.8
Requires-Dist: preshed>=3.0.8
Requires-Dist: spacy-legacy>=3.0.11
Requires-Dist: spacy-loggers>=1.0.4
Requires-Dist: murmurhash>=1.0.8
Requires-Dist: tqdm~=4.64
Requires-Dist: zstandard~=0.18.0
Requires-Dist: diskcache>=5.0.0
Requires-Dist: bottle~=0.12.23
Requires-Dist: datasets~=2.17
Requires-Dist: pyarrow>=11.0.0
Requires-Dist: pyarrow-hotfix~=0.6
Requires-Dist: nltk!=3.9.0,~=3.7
Requires-Dist: rouge-score~=0.1.2
Requires-Dist: scipy~=1.10
Requires-Dist: uncertainty-calibration~=0.1.4
Requires-Dist: scikit-learn~=1.1
Requires-Dist: transformers~=4.40
Requires-Dist: torch<3.0.0,>=1.13.1
Requires-Dist: torchvision<3.0.0,>=0.14.1
Requires-Dist: python-docx==1.2.0
Requires-Dist: nemo-evaluator
Provides-Extra: proxy-server
Requires-Dist: gunicorn>=20.1; extra == "proxy-server"
Provides-Extra: human-evaluation
Requires-Dist: scaleapi~=2.13.0; extra == "human-evaluation"
Requires-Dist: surge-api~=1.1.0; extra == "human-evaluation"
Provides-Extra: scenarios
Requires-Dist: gdown~=5.1; extra == "scenarios"
Requires-Dist: sympy~=1.11.1; extra == "scenarios"
Requires-Dist: xlrd~=2.0.1; extra == "scenarios"
Provides-Extra: metrics
Requires-Dist: google-api-python-client~=2.64; extra == "metrics"
Requires-Dist: numba~=0.56; extra == "metrics"
Requires-Dist: sacrebleu~=2.2.1; extra == "metrics"
Requires-Dist: langdetect~=1.0.9; extra == "metrics"
Requires-Dist: immutabledict~=4.2.0; extra == "metrics"
Requires-Dist: gradio_client~=1.3; extra == "metrics"
Provides-Extra: ranking
Requires-Dist: pytrec_eval==0.5; extra == "ranking"
Provides-Extra: summarization
Requires-Dist: bert-score~=0.3; extra == "summarization"
Provides-Extra: plots
Requires-Dist: colorcet~=3.0.1; extra == "plots"
Requires-Dist: matplotlib~=3.6.0; extra == "plots"
Requires-Dist: seaborn~=0.11.0; extra == "plots"
Provides-Extra: decodingtrust
Requires-Dist: fairlearn~=0.9.0; extra == "decodingtrust"
Provides-Extra: slurm
Requires-Dist: simple-slurm~=0.2.6; extra == "slurm"
Provides-Extra: cleva
Requires-Dist: unidecode~=1.3; extra == "cleva"
Requires-Dist: pypinyin~=0.49.0; extra == "cleva"
Requires-Dist: jieba~=0.42.1; extra == "cleva"
Requires-Dist: opencc~=1.1; extra == "cleva"
Requires-Dist: langdetect~=1.0; extra == "cleva"
Provides-Extra: images
Requires-Dist: pillow~=10.2; extra == "images"
Provides-Extra: mongo
Requires-Dist: pymongo~=4.2; extra == "mongo"
Provides-Extra: unitxt
Requires-Dist: evaluate~=0.4.1; extra == "unitxt"
Provides-Extra: seahelm
Requires-Dist: pythainlp==5.0.0; extra == "seahelm"
Requires-Dist: pyonmttok==1.37.0; extra == "seahelm"
Requires-Dist: sacrebleu~=2.2.1; extra == "seahelm"
Requires-Dist: python-crfsuite~=0.9.11; extra == "seahelm"
Provides-Extra: accelerate
Requires-Dist: accelerate~=0.25; extra == "accelerate"
Provides-Extra: aleph-alpha
Requires-Dist: aleph-alpha-client~=2.14.0; extra == "aleph-alpha"
Requires-Dist: tokenizers>=0.13.3; extra == "aleph-alpha"
Provides-Extra: allenai
Requires-Dist: ai2-olmo~=0.2; extra == "allenai"
Provides-Extra: amazon
Requires-Dist: boto3~=1.34.131; extra == "amazon"
Requires-Dist: awscli~=1.33.13; extra == "amazon"
Requires-Dist: botocore~=1.34.131; extra == "amazon"
Provides-Extra: anthropic
Requires-Dist: anthropic~=0.48; extra == "anthropic"
Requires-Dist: websocket-client~=1.3.2; extra == "anthropic"
Requires-Dist: httpx<0.28.0; extra == "anthropic"
Provides-Extra: cohere
Requires-Dist: cohere~=5.3; extra == "cohere"
Provides-Extra: writer
Requires-Dist: writerai~=4.0; extra == "writer"
Provides-Extra: mistral
Requires-Dist: mistralai~=1.1; extra == "mistral"
Provides-Extra: openai
Requires-Dist: openai~=1.70; extra == "openai"
Requires-Dist: tiktoken~=0.7; extra == "openai"
Requires-Dist: pydantic~=2.0; extra == "openai"
Provides-Extra: google
Requires-Dist: google-cloud-aiplatform~=1.48; extra == "google"
Provides-Extra: together
Requires-Dist: together~=1.1; extra == "together"
Provides-Extra: yandex
Requires-Dist: sentencepiece~=0.2.0; extra == "yandex"
Provides-Extra: reka
Requires-Dist: reka-api~=2.0.0; extra == "reka"
Provides-Extra: vlm
Requires-Dist: einops~=0.7.0; extra == "vlm"
Requires-Dist: einops-exts~=0.0.4; extra == "vlm"
Requires-Dist: open-clip-torch~=2.24; extra == "vlm"
Requires-Dist: torch~=2.1; extra == "vlm"
Requires-Dist: transformers_stream_generator~=0.0.4; extra == "vlm"
Requires-Dist: scipy~=1.10; extra == "vlm"
Requires-Dist: torchvision<3.0.0,>=0.14.1; extra == "vlm"
Provides-Extra: medhelm
Requires-Dist: aiohttp<4.0.0,>=3.12.13; extra == "medhelm"
Requires-Dist: anthropic<1.0.0,>=0.57.1; extra == "medhelm"
Requires-Dist: bert-score<1.0.0,>=0.3.13; extra == "medhelm"
Requires-Dist: bottle<1.0.0,>=0.12.25; extra == "medhelm"
Requires-Dist: datasets<3.0.0,>=2.21.0; extra == "medhelm"
Requires-Dist: Flask<4.0.0,>=3.0.3; extra == "medhelm"
Requires-Dist: gin-config<1.0.0,>=0.5.0; extra == "medhelm"
Requires-Dist: huggingface-hub<1.0.0,>=0.33.1; extra == "medhelm"
Requires-Dist: lxml<7.0.0,>=6.0.0; extra == "medhelm"
Requires-Dist: matplotlib<4.0.0,>=3.10.3; extra == "medhelm"
Requires-Dist: nltk<4.0.0,>=3.9.1; extra == "medhelm"
Requires-Dist: numpy<2.0.0,>=1.26.4; extra == "medhelm"
Requires-Dist: openai<2.0.0,>=1.93.0; extra == "medhelm"
Requires-Dist: openpyxl<4.0.0,>=3.1.5; extra == "medhelm"
Requires-Dist: pandas<3.0.0,>=2.3.0; extra == "medhelm"
Requires-Dist: parameterized<1.0.0,>=0.9.0; extra == "medhelm"
Requires-Dist: pillow<12.0.0,>=11.3.0; extra == "medhelm"
Requires-Dist: protobuf<7.0.0,>=6.31.1; extra == "medhelm"
Requires-Dist: psutil<7.0.0,>=6.0.0; extra == "medhelm"
Requires-Dist: pyarrow<21.0.0,>=20.0.0; extra == "medhelm"
Requires-Dist: pyarrow-hotfix<1.0.0,>=0.7; extra == "medhelm"
Requires-Dist: pydantic<3.0.0,>=2.11.7; extra == "medhelm"
Requires-Dist: python-dateutil<3.0.0,>=2.9.0.post0; extra == "medhelm"
Requires-Dist: python-docx<2.0.0,>=1.2.0; extra == "medhelm"
Requires-Dist: python-dotenv<2.0.0,>=1.1.1; extra == "medhelm"
Requires-Dist: PyYAML<7.0.0,>=6.0.2; extra == "medhelm"
Requires-Dist: regex<2025.0.0,>=2024.11.6; extra == "medhelm"
Requires-Dist: requests<3.0.0,>=2.32.4; extra == "medhelm"
Requires-Dist: retrying<2.0.0,>=1.4.0; extra == "medhelm"
Requires-Dist: rich<15.0.0,>=14.0.0; extra == "medhelm"
Requires-Dist: rouge_score<1.0.0,>=0.1.2; extra == "medhelm"
Requires-Dist: scikit-learn<2.0.0,>=1.7.0; extra == "medhelm"
Requires-Dist: scipy<2.0.0,>=1.10.0; extra == "medhelm"
Requires-Dist: sentencepiece<1.0.0,>=0.2.0; extra == "medhelm"
Requires-Dist: setuptools<81.0.0,>=80.9.0; extra == "medhelm"
Requires-Dist: six<2.0.0,>=1.17.0; extra == "medhelm"
Requires-Dist: smart-open<8.0.0,>=7.1.0; extra == "medhelm"
Requires-Dist: spacy<4.0.0,>=3.8.7; extra == "medhelm"
Requires-Dist: structlog<25.0.0,>=24.1.0; extra == "medhelm"
Requires-Dist: sympy<2.0.0,>=1.14.0; extra == "medhelm"
Requires-Dist: tiktoken<1.0.0,>=0.9.0; extra == "medhelm"
Requires-Dist: tokenizers<1.0.0,>=0.21.2; extra == "medhelm"
Requires-Dist: toml<1.0.0,>=0.10.2; extra == "medhelm"
Requires-Dist: torch<3.0.0,>=2.7.1; extra == "medhelm"
Requires-Dist: torchvision<1.0.0,>=0.22.1; extra == "medhelm"
Requires-Dist: tqdm<5.0.0,>=4.67.1; extra == "medhelm"
Requires-Dist: transformers<5.0.0,>=4.53.0; extra == "medhelm"
Requires-Dist: triton<4.0.0,>=3.3.1; extra == "medhelm"
Requires-Dist: typer<1.0.0,>=0.16.0; extra == "medhelm"
Requires-Dist: typing_extensions<5.0.0,>=4.14.0; extra == "medhelm"
Requires-Dist: uncertainty-calibration<1.0.0,>=0.1.4; extra == "medhelm"
Requires-Dist: websocket-client<2.0.0,>=1.3.3; extra == "medhelm"
Requires-Dist: xmltodict<1.0.0,>=0.14.2; extra == "medhelm"
Requires-Dist: yq<4.0.0,>=3.1.1; extra == "medhelm"
Requires-Dist: zstandard<1.0.0,>=0.18.0; extra == "medhelm"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# NVIDIA HELM Benchmark Framework

This directory contains the HELM (Holistic Evaluation of Language Models) framework for evaluating large language models in medical applications across various healthcare tasks.

## Overview

The HELM framework provides a comprehensive evaluation system for medical AI models, supporting multiple benchmark datasets and evaluation scenarios. It's designed to work with the EvalFactory infrastructure for standardized model evaluation.

## Available Benchmarks

The framework supports the following medical evaluation benchmarks:

| Benchmark | Description | Type |
|-----------|-------------|------|
| **medcalc_bench** | Medical calculation benchmark with patient notes and ground truth answers | Medical QA |
| **medec** | Medical error detection and correction pairs | Error Detection |
| **head_qa** | Biomedical multiple-choice questions for medical knowledge testing | Medical QA |
| **medbullets** | USMLE-style medical questions with explanations | Medical QA |
| **pubmed_qa** | PubMed abstracts with yes/no/maybe questions | Medical QA |
| **ehr_sql** | Natural language to SQL query generation for clinical research | SQL Generation |
| **race_based_med** | Detection of race-based biases in medical LLM outputs | Bias Detection |
| **medhallu** | Classification of factual vs hallucinated medical answers | Hallucination Detection |

## Quick Start

### 1. Environment Setup

First, ensure you have the required environment variables set:

```bash
# Set your API keys
export OPENAI_API_KEY="your-api-key-here"

# Set Python path if necessary
export PYTHONPATH=$PYTHONPATH:$.
```

### 2. Running Your First Benchmark

#### Method 1: Using `eval-factory` (Recommended)

`eval-factory` is a wrapper that simplifies the HELM benchmark process by handling configuration generation, benchmark execution, and result formatting automatically.

**What `eval-factory` does internally:**

1. **Configuration Processing**: Loads your YAML config and merges it with framework defaults
2. **Dynamic Config Generation**: Creates the necessary HELM model configurations dynamically
3. **Benchmark Execution**: Runs the HELM benchmark with proper parameters
4. **Result Processing**: Formats and saves results in standardized YAML format

Create a configuration file (e.g., `my_test.yml`):

```yaml
config:
  type: medcalc_bench  # Choose from available benchmarks
  output_dir: results/my_test
target:
  api_endpoint:
    url: https://api.openai.com/v1
    model_id: gpt-4
    type: chat
    api_key: OPENAI_API_KEY
```

Run the evaluation:

```bash
eval-factory run_eval \
    --output_dir results/my_test \
    --run_config my_test.yml
```

**Internal Process Breakdown:**

1. **Config Loading & Validation**: 
   - Loads your YAML configuration
   - Validates against framework schema
   - Merges with default parameters from `framework.yml`

2. **Dynamic Model Config Generation**:
   - Calls `scripts/generate_dynamic_model_configs.py`
   - Creates model-specific configuration files
   - Handles provider-specific API endpoints and authentication

3. **HELM Benchmark Execution**:
   - Executes `helm-run` with generated configurations
   - Downloads and prepares benchmark datasets
   - Runs evaluations with specified parameters
   - Caches responses for efficiency

4. **Result Processing**:
   - Collects raw benchmark results
   - Formats into standardized YAML output
   - Saves results in your specified output directory

#### Method 2: Using `helm-run` directly

```bash
helm-run \
  --run-entries medcalc_bench:model=openai/gpt-4 \
  --suite my-suite \
  --max-eval-instances 10 \
  --num-train-trials 1 \
  -o results/my_test
```

**Comparison: `eval-factory` vs `helm-run`**

| Feature | `eval-factory` | `helm-run` |
|---------|-------------------|------------|
| **Configuration** | Simple YAML config | Complex command-line arguments |
| **Model Setup** | Automatic config generation | Manual model registration required |
| **Provider Support** | Built-in adapter handling | Requires custom model configs |
| **Results Format** | Standardized YAML output | Native HELM format only |
| **Ease of Use** | Beginner-friendly | Advanced users only |
| **Integration** | EvalFactory compatible | HELM-specific |

**Recommendation**: Use `eval-factory` for most use cases, especially when working with EvalFactory. Use `helm-run` only when you need fine-grained control over HELM's native features.

### 3. Understanding the Output

After running a benchmark, you'll find results in your specified output directory:

```
results/my_test/
├── responses/          # Raw model responses
├── cache.db           # Cached responses for efficiency
├── instances.jsonl    # Evaluation instances
├── results.jsonl      # Final evaluation results
├── model_configs/     # Generated HELM model configurations
└── evaluation_config.yaml  # Standardized evaluation results
```

**Generated Files Explanation:**

- **`responses/`**: Contains raw API responses from the model for each evaluation instance
- **`cache.db`**: SQLite database caching responses to avoid re-running identical queries
- **`instances.jsonl`**: The evaluation instances (questions, prompts, etc.) used in the benchmark
- **`results.jsonl`**: HELM's native results format with detailed metrics
- **`model_configs/`**: Dynamically generated configuration files for the specific model and provider
- **`evaluation_config.yaml`**: Standardized results in YAML format compatible with EvalFactory

**Key Advantage**: `eval-factory` automatically handles the complexity of HELM configuration generation, making it much easier to run benchmarks compared to using `helm-run` directly.

## Step-by-Step Guide

### Step 1: Choose Your Benchmark

Select from the available benchmarks based on your evaluation needs:

- **For general medical QA**: `medcalc_bench`, `head_qa`, `medbullets`
- **For error detection**: `medec`
- **For research applications**: `pubmed_qa`, `ehr_sql`
- **For safety evaluation**: `race_based_med`, `medhallu`

### Step 2: Configure Your Model

Create a YAML configuration file with your model details. Here are examples for different providers:

#### OpenAI Configuration
```yaml
config:
  type: medcalc_bench
  output_dir: results/openai_test
target:
  api_endpoint:
    url: https://api.openai.com/v1
    model_id: gpt-4
    type: chat
    api_key: OPENAI_API_KEY
```

#### NVIDIA AI Foundation Models (build.nvidia.com)
```yaml
config:
  type: pubmed_qa
  output_dir: results/nim_test
target:
  api_endpoint:
    url: https://integrate.api.nvidia.com/v1
    model_id: nvdev/meta/llama-3.3-70b-instruct
    type: chat
    api_key: OPENAI_API_KEY
```

#### NVIDIA Cloud Function (nvcf)
```yaml
config:
  type: ehr_sql
  output_dir: results/nvcf_test
target:
  api_endpoint:
    url: https://api.nvcf.nvidia.com/v2/nvcf/pexec/functions/13e4f873-9d52-4ba9-8194-61baf8dc2bc9/
    model_id: meta-llama/Llama-3.3-70B-Instruct
    type: chat
    api_key: OPENAI_API_KEY
    adapter_config:
      use_nvcf: true
```

### Model Naming Conventions

Different providers use different model ID formats:

- **OpenAI**: `gpt-4`, `gpt-3.5-turbo`, `text-davinci-003`
- **NVIDIA**: `meta-llama/Llama-3.3-70B-Instruct`, `mistral-7b-instruct`

**Note**: NVCF requires a specific function ID in the URL and the `use_nvcf: true` adapter configuration.

### Step 3: Set Up API Credentials

Ensure your API credentials are properly configured:

```bash
# For OpenAI models
export OPENAI_API_KEY="<very-long-sequence>"

# For NVIDIA AI Foundation Models (build.nvidia.com)
export OPENAI_API_KEY="nvapi-..."  # Uses same env var as OpenAI

# For NVIDIA Cloud Function (nvcf)
export OPENAI_API_KEY="nvapi-..."  # Uses same env var as OpenAI

# Note: NVIDIA services typically use the same OPENAI_API_KEY environment variable
# but with NVIDIA-specific API keys (nvapi-... format)
```

### Step 4: Run the Evaluation

Execute the benchmark using one of the methods above. The framework will:

1. **Load the configuration** and validate parameters
2. **Generate model configs** dynamically for the specified model
3. **Download and prepare** the benchmark dataset
4. **Run evaluations** on the specified number of instances
5. **Cache responses** for efficiency and reproducibility
6. **Generate results** in standardized format

### Step 5: Analyze Results

Review the generated results:

```bash
# View raw results
cat results/my_test/results.jsonl

# Use HELM tools for analysis
helm-summarize --suite my-suite
helm-server  # Start web interface to view results
```

## Advanced Configuration

### Customizing Evaluation Parameters

You can customize various parameters in your configuration:

```yaml
config:
  type: medcalc_bench
  output_dir: results/advanced_test
  params:
    limit_samples: 100        # Limit number of evaluation instances
    parallelism: 4           # Number of parallel threads
    extra:
      num_train_trials: 3    # Number of training trials
      max_length: 2048       # Maximum token length
target:
  api_endpoint:
    url: https://api.openai.com/v1
    model_id: gpt-4
    type: chat
    api_key: OPENAI_API_KEY
```

### Advanced Configuration Parameters

The `config.params.extra` section provides additional parameters for fine-tuning evaluations:

#### `data_path`
- **Purpose**: Custom data path for scenarios that support it
- **Supported Scenarios**: `ehrshot`, `clear`, `medalign`, `n2c2_ct_matching`
- **Example**: `"/path/to/custom/data"`
- **Description**: Overrides the default data location for the scenario

#### `num_output_tokens`
- **Purpose**: Maximum number of tokens the model is allowed to generate in its response
- **Scope**: Controls only the output length, not the total sequence length
- **Example**: `1000` limits model responses to 1000 tokens
- **Use Case**: Useful for controlling response length in generation tasks

#### `max_length`
- **Purpose**: Maximum total length for the entire input-output sequence (input + output combined)
- **Scope**: Controls the combined length of both prompt and response
- **Example**: `2048` limits total conversation to 2048 tokens
- **Difference from num_output_tokens**: This controls total sequence length, while num_output_tokens only controls response length

#### `subject`
- **Purpose**: Specific task or subset to evaluate within a scenario
- **Examples by Scenario**:
  - **ehrshot**: `"guo_readmission"`, `"new_hypertension"`, `"lab_anemia"`
  - **n2c2_ct_matching**: `"ABDOMINAL"`, `"ADVANCED-CAD"`, `"CREATININE"`
  - **clear**: `"major_depression"`, `"bipolar_disorder"`, `"substance_use_disorder"`
- **Description**: Filters the evaluation to a specific prediction task or medical condition

#### `condition`
- **Purpose**: Specific condition or scenario variant to evaluate
- **Supported Scenarios**: `clear`
- **Examples**: `"alcohol_dependence"`, `"chronic_pain"`, `"homelessness"`
- **Description**: Used by scenarios like 'clear' to specify medical conditions for evaluation

#### `num_train_trials`
- **Purpose**: Number of training trials for few-shot evaluation
- **Behavior**: Each trial samples a different set of in-context examples
- **Example**: `3` runs the evaluation 3 times with different examples
- **Use Case**: Useful for robust evaluation with multiple few-shot configurations

### Example Configuration with All Parameters

```yaml
config:
  type: ehrshot
  output_dir: results/ehrshot_evaluation
  params:
    limit_samples: 500
    parallelism: 2
    extra:
      data_path: "/custom/path/to/ehrshot/data"
      num_output_tokens: 1000
      max_length: 4096
      subject: "guo_readmission"
      num_train_trials: 3
target:
  api_endpoint:
    url: https://api.openai.com/v1
    model_id: gpt-4
    type: chat
    api_key: OPENAI_API_KEY
```

### Running Multiple Benchmarks

To run multiple benchmarks on the same model:

```bash
# Create separate config files for each benchmark
eval-factory run_eval --output_dir results/medcalc_test --run_config medcalc_config.yml
eval-factory run_eval --output_dir results/medec_test --run_config medec_config.yml
eval-factory run_eval --output_dir results/head_qa_test --run_config head_qa_config.yml
```

### Dry Run Mode

Test your configuration without running the full evaluation:

```bash
eval-factory run_eval \
    --output_dir results/test \
    --run_config my_config.yml \
    --dry_run
```

This will show you the rendered configuration and command without executing the benchmark.

## Troubleshooting

### Common Issues

1. **API Key Errors**: Ensure your API keys are properly set and valid
2. **Model Not Found**: Verify the model ID and endpoint URL are correct
3. **Memory Issues**: Reduce `parallelism` or `limit_samples` for large models
4. **Timeout Errors**: Increase timeout settings or reduce batch sizes

### Debug Mode

Enable debug logging for detailed information:

```bash
eval-factory --debug run_eval \
    --output_dir results/debug_test \
    --run_config debug_config.yml
```

### Checking Available Tasks

List all available evaluation types:

```bash
eval-factory ls
```

## Examples from commands.sh

Here are some practical examples from the project:

### Basic Medical Calculation Benchmark
```bash
eval-factory run_eval \
    --output_dir test_cases/test_case_nim_llama_3_1_8b_medcalc_bench \
    --run_config test_cases/test_case_nim_llama_3_1_8b_medcalc_bench.yml
```

### Medical Error Detection
```bash
eval-factory run_eval \
    --output_dir test_cases/test_case_nim_llama_3_1_8b_medec \
    --run_config test_cases/test_case_nim_llama_3_1_8b_medec.yml
```

### Biomedical QA
```bash
eval-factory run_eval \
    --output_dir test_cases/test_case_nim_llama_3_1_8b_head_qa \
    --run_config test_cases/test_case_nim_llama_3_1_8b_head_qa.yml
```

## Running Evaluations with Judges

The HELM framework supports multi-judge evaluations for scenarios that require human-like assessment of model outputs. This is particularly useful for tasks like medical treatment plan generation, where multiple AI judges can provide more robust and reliable evaluations.

### Overview of Multi-Judge Setup

The framework supports three types of judges:
- **GPT Judge**: Uses OpenAI GPT models for evaluation
- **Llama Judge**: Uses Llama models for evaluation  
- **Claude Judge**: Uses Anthropic Claude models for evaluation

Each judge can use different API keys, providing better rate limiting, cost tracking, and flexibility.

### Authentication Systems

The framework supports **two authentication methods** for judge models:

#### 1. Direct Judge API Keys (Recommended for Production)
Set individual API keys for each judge type:

```bash
# API key for the main model being evaluated
export OPENAI_API_KEY="your-main-model-api-key"

# API keys for the three judges (annotators)
export GPT_JUDGE_API_KEY="your-gpt-judge-api-key"
export LLAMA_JUDGE_API_KEY="your-llama-judge-api-key"
export CLAUDE_JUDGE_API_KEY="your-claude-judge-api-key"
```

#### 2. OAuth 2.0 Client Credentials Flow (Advanced)
Use NVIDIA's OAuth system for automatic token management:

```bash
# OAuth 2.0 credentials for automatic token generation
export OPENAI_CLIENT_ID="nvssa-prd-your-client-id"
export OPENAI_CLIENT_SECRET="ssap-your-client-secret"
export OPENAI_TOKEN_URL="https://prod.api.nvidia.com/oauth/api/v1/ssa/default/token"
export OPENAI_SCOPE="awsanthropic-readwrite"

# Main API key (still required)
export OPENAI_API_KEY="your-main-model-api-key"
```

### How Authentication Priority Works

The system follows this **exact priority order**:

1. **First Priority**: Judge-specific environment variables
   - `GPT_JUDGE_API_KEY` for GPT models
   - `LLAMA_JUDGE_API_KEY` for Llama models
   - `CLAUDE_JUDGE_API_KEY` for Claude models

2. **Second Priority**: Fallback to main API key
   - If judge keys aren't set, automatically uses `OPENAI_API_KEY`
   - System logs: "GPT_JUDGE_API_KEY is not set, setting to OPENAI_API_KEY"

3. **Third Priority**: Credentials configuration
   - Falls back to `credentials.conf` or deployment-specific keys

**Important**: OAuth-generated tokens are **NOT** automatically used for judge API keys. The OAuth system is separate and serves different purposes.

### OAuth 2.0 System Details

The OAuth system provides:
- **Automatic Token Creation**: Generates access tokens using client credentials
- **Token Caching**: Stores tokens in memory and disk (`{service_name}_oauth_token.json`)
- **Automatic Refresh**: Refreshes expired tokens automatically
- **Scope Control**: Different permissions per service:
  - `azureopenai-readwrite` for GPT services
  - `awsanthropic-readwrite` for Claude services

**When to Use OAuth:**
- Better security (client credentials vs. long-lived API keys)
- Automatic token management
- Centralized billing and rate limiting
- Enterprise-grade authentication

**When to Use Direct API Keys:**
- Simpler setup
- Direct control over each judge's API key
- Different providers for different judges
- Testing and development scenarios

### Security Features

**API Key Protection**: The system automatically sanitizes error messages to prevent API keys from appearing in logs. Any API key patterns (like `nvapi-...`, `sk-...`, `hf_...`) are automatically replaced with `[API_KEY_REDACTED]` before logging.

### Configuration for Multi-Judge Evaluations

#### Basic Configuration (Direct API Keys)

```yaml
config:
  type: mtsamples_replicate  # Example scenario that uses judges
  output_dir: results/multi_judge_test
  params:
    limit_samples: 10
    parallelism: 1
    extra:
      num_train_trials: 1
      max_length: 2048
      # Different API keys for each judge
      gpt_judge_api_key: GPT_JUDGE_API_KEY
      llama_judge_api_key: LLAMA_JUDGE_API_KEY
      claude_judge_api_key: CLAUDE_JUDGE_API_KEY
target:
  api_endpoint:
    url: https://integrate.api.nvidia.com/v1
    model_id: nvdev/meta/llama-3.1-8b-instruct
    type: chat
    api_key: OPENAI_API_KEY
```

#### Advanced Configuration (OAuth + Direct Keys)

```yaml
config:
  type: mtsamples_replicate
  output_dir: results/oauth_multi_judge_test
  params:
    limit_samples: 50
    parallelism: 2
    extra:
      num_train_trials: 3
      max_length: 2048
      # Mix OAuth (automatic) and direct keys
      gpt_judge_api_key: GPT_JUDGE_API_KEY  # Direct key for GPT
      # Llama and Claude will use OAuth-generated tokens
target:
  api_endpoint:
    url: https://integrate.api.nvidia.com/v1
    model_id: nvdev/meta/llama-3.3-70b-instruct
    type: chat
    api_key: OPENAI_API_KEY
```

### Supported Scenarios with Judges

Currently, the following scenarios support multi-judge evaluations:

| Scenario | Description | Judge Types Used |
|----------|-------------|------------------|
| **mtsamples_replicate** | Generate treatment plans based on clinical notes | GPT, Llama, Claude |
| **mtsamples_procedures** | Document and extract information about medical procedures | GPT, Llama, Claude |
| **aci_bench** | Extract and structure information from patient-doctor conversations | GPT, Llama, Claude |
| **medication_qa** | Answer consumer medication-related questions | GPT, Llama, Claude |
| **medi_qa** | Retrieve and rank answers based on medical question understanding | GPT, Llama, Claude |
| **med_dialog** | Generate summaries of doctor-patient conversations | GPT, Llama, Claude |

### Complete Setup Guide

#### Method 1: Direct API Keys (Simplest)

```bash
# 1. Set up environment variables
export OPENAI_API_KEY="nvapi-your-main-api-key"
export GPT_JUDGE_API_KEY="nvapi-gpt-judge-api-key"
export LLAMA_JUDGE_API_KEY="nvapi-llama-judge-api-key"
export CLAUDE_JUDGE_API_KEY="nvapi-claude-judge-api-key"

# 2. Run the evaluation
eval-factory run_eval \
    --output_dir results/multi_judge_test \
    --run_config multi_judge_config.yml
```

#### Method 2: OAuth 2.0 System (Enterprise)

```bash
# 1. Set up OAuth credentials
export OPENAI_CLIENT_ID="nvssa-prd-your-client-id"
export OPENAI_CLIENT_SECRET="ssap-your-client-secret"
export OPENAI_TOKEN_URL="https://prod.api.nvidia.com/oauth/api/v1/ssa/default/token"
export OPENAI_SCOPE="awsanthropic-readwrite"

# 2. Set main API key (still required)
export OPENAI_API_KEY="nvapi-your-main-api-key"

# 3. Run the evaluation
eval-factory run_eval \
    --output_dir results/oauth_multi_judge_test \
    --run_config oauth_multi_judge_config.yml
```

#### Method 3: Hybrid Approach (Flexible)

```bash
# 1. Set OAuth credentials for automatic token generation
export OPENAI_CLIENT_ID="nvssa-prd-your-client-id"
export OPENAI_CLIENT_SECRET="ssap-your-client-secret"

# 2. Override specific judge with direct API key
export GPT_JUDGE_API_KEY="nvapi-gpt-specific-key"

# 3. Set main API key
export OPENAI_API_KEY="nvapi-your-main-api-key"

# 4. Run the evaluation
eval-factory run_eval \
    --output_dir results/hybrid_multi_judge_test \
    --run_config hybrid_multi_judge_config.yml
```

#### Method 4: Using `helm-run` directly

```bash
# Set up environment variables (any of the above methods)
export OPENAI_API_KEY="nvapi-your-main-api-key"
export GPT_JUDGE_API_KEY="nvapi-gpt-judge-api-key"
export LLAMA_JUDGE_API_KEY="nvapi-llama-judge-api-key"
export CLAUDE_JUDGE_API_KEY="nvapi-claude-judge-api-key"

# Run the evaluation
helm-run \
  --run-entries mtsamples_replicate:model=openai/gpt-4 \
  --suite my-suite \
  --max-eval-instances 10 \
  --num-train-trials 1 \
  -o results/multi_judge_test
```

### Advanced Judge Configuration

#### Using Different API Keys for Each Judge

You can use completely different API keys for each judge:

```bash
export GPT_JUDGE_API_KEY="nvapi-gpt-judge-1"
export LLAMA_JUDGE_API_KEY="nvapi-llama-judge-2"
export CLAUDE_JUDGE_API_KEY="nvapi-claude-judge-3"
```

#### Using the Same API Key for All Judges

If you want to use the same API key for all judges:

```bash
export GPT_JUDGE_API_KEY="nvapi-shared-key"
export LLAMA_JUDGE_API_KEY="nvapi-shared-key"
export CLAUDE_JUDGE_API_KEY="nvapi-shared-key"
```

#### OAuth Token Management

**Check OAuth Token Status:**
```bash
# Look for OAuth token files
ls -la *_oauth_token.json

# Check token expiration
cat openai_oauth_token.json | jq '.expires_at'
```

**Force Token Refresh:**
```bash
# The system automatically refreshes expired tokens
# You can also manually trigger refresh by deleting token files
rm *_oauth_token.json
```

**OAuth Scopes for Different Services:**
```bash
# For GPT services
export OPENAI_SCOPE="azureopenai-readwrite"

# For Claude services  
export OPENAI_SCOPE="awsanthropic-readwrite"

# For general access
export OPENAI_SCOPE="awsanthropic-readwrite"
```

### Example Multi-Judge Evaluation

Here's a complete example for running a multi-judge evaluation:

```bash
# 1. Create configuration file (multi_judge_config.yml)
cat > multi_judge_config.yml << EOF
config:
  type: mtsamples_replicate
  output_dir: results/multi_judge_test
  params:
    limit_samples: 50
    parallelism: 2
    extra:
      num_train_trials: 3
      max_length: 2048
      gpt_judge_api_key: GPT_JUDGE_API_KEY
      llama_judge_api_key: LLAMA_JUDGE_API_KEY
      claude_judge_api_key: CLAUDE_JUDGE_API_KEY
target:
  api_endpoint:
    url: https://integrate.api.nvidia.com/v1
    model_id: nvdev/meta/llama-3.3-70b-instruct
    type: chat
    api_key: OPENAI_API_KEY
EOF

# 2. Set environment variables
export OPENAI_API_KEY="nvapi-main-model-key"
export GPT_JUDGE_API_KEY="nvapi-gpt-judge-key"
export LLAMA_JUDGE_API_KEY="nvapi-llama-judge-key"
export CLAUDE_JUDGE_API_KEY="nvapi-claude-judge-key"

# 3. Run the evaluation
eval-factory run_eval \
    --output_dir results/multi_judge_test \
    --run_config multi_judge_config.yml
```

### Troubleshooting Multi-Judge Evaluations

#### Check Environment Variables

Verify your environment variables are set correctly:

```bash
echo "Main API Key: $OPENAI_API_KEY"
echo "GPT Judge: $GPT_JUDGE_API_KEY"
echo "Llama Judge: $LLAMA_JUDGE_API_KEY"
echo "Claude Judge: $CLAUDE_JUDGE_API_KEY"
```

#### Check OAuth Configuration

Verify OAuth credentials are properly set:

```bash
echo "Client ID: $OPENAI_CLIENT_ID"
echo "Client Secret: $OPENAI_CLIENT_SECRET"
echo "Token URL: $OPENAI_TOKEN_URL"
echo "Scope: $OPENAI_SCOPE"
```

#### Debug Mode

Enable debug logging to see which API keys are being used:

```bash
eval-factory --debug run_eval \
    --output_dir results/debug_multi_judge \
    --run_config multi_judge_config.yml
```

#### Common Issues and Solutions

**Issue: "GPT_JUDGE_API_KEY is not set, setting to OPENAI_API_KEY"**
- **Cause**: Judge API key not set, system falling back to main API key
- **Solution**: Set the specific judge API key or accept the fallback

**Issue: "Missing environment variables for openai token"**
- **Cause**: OAuth credentials not properly configured
- **Solution**: Set `OPENAI_CLIENT_ID` and `OPENAI_CLIENT_SECRET`

**Issue: "Error creating openai OAuth token"**
- **Cause**: Invalid credentials or network issues
- **Solution**: Verify credentials and check network connectivity

**Issue: API key appears in logs**
- **Cause**: This should not happen with the security fix
- **Solution**: Check if you're using the latest version with API key sanitization

#### Log Analysis

**Look for these log patterns:**
```bash
# Judge API key usage
grep "Using.*judge API key" logs/*.log

# OAuth token creation
grep "Creating new.*OAuth token" logs/*.log

# API key fallbacks
grep "is not set, setting to" logs/*.log

# Authentication errors
grep "Authentication error detected" logs/*.log
```

#### Performance Monitoring

**Check API Key Usage:**
```bash
# Monitor which API keys are being used
grep "Using.*API key.*ends with" logs/*.log

# Check for rate limiting
grep "rate limit\|429" logs/*.log

# Monitor OAuth token refresh
grep "token expired\|refreshing" logs/*.log
```

Look for messages like:
```
Using GPT judge API key from environment variable for model: nvidia/gpt4o-abc123
Using Llama judge API key from environment variable for model: nvdev/meta/llama-3.3-70b-instruct-def456
Using Claude judge API key from environment variable for model: nvidia/claude-3-7-sonnet-20250219-ghi789
```

#### Common Issues

1. **Environment variables not loaded**: Make sure your environment variables are set before running the command
2. **API key format**: Ensure your API keys start with `nvapi-` for NVIDIA services
3. **Configuration file**: Verify your YAML configuration file references the correct environment variable names
4. **Judge model availability**: Ensure the judge models are available through your API endpoints

### Benefits of Multi-Judge Evaluations

- **Better rate limiting**: Each judge can have its own rate limits
- **Cost tracking**: Track costs separately for each judge
- **Flexibility**: Use different API keys for different purposes
- **Security**: Isolate API keys for different components
- **Robustness**: Multiple judges provide more reliable evaluations
- **Diversity**: Different judge models may catch different types of errors

## Integration with EvalFactory

This framework is designed to work seamlessly with the EvalFactory infrastructure:

- **Standardized Output**: Results are generated in a format compatible with EvalFactory
- **Configuration Management**: Uses YAML-based configuration for easy integration
- **Caching**: Built-in caching for efficient re-runs and reproducibility
- **Extensibility**: Easy to add new benchmarks and evaluation metrics

## Contributing

To add new benchmarks or modify existing ones:

1. Update `framework.yml` with new benchmark definitions
2. Implement the benchmark logic in the appropriate adapter
3. Add test cases and documentation
4. Update this README with new benchmark information

## References

- [HELM Framework](https://github.com/stanford-crfm/helm)
- [EvalFactory Documentation](https://github.com/nvidia/eval-factory)
- [Medical AI Evaluation Papers](https://arxiv.org/abs/2401.00000)

For more detailed information about specific benchmarks and their implementations, refer to the individual benchmark documentation and the main HELM repository. 

# Holistic Evaluation of Language Models (HELM)

[comment]: <> (When using the img tag, which allows us to specify size, src has to be a URL.)
HELM logo

**Holistic Evaluation of Language Models (HELM)** is an open source Python framework created by the [Center for Research on Foundation Models (CRFM) at Stanford](https://crfm.stanford.edu/) for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. This framework includes the following features:

- Datasets and benchmarks in a standardized format (e.g. MMLU-Pro, GPQA, IFEval, WildBench)
- Models from various providers accessible through a unified interface (e.g. OpenAI models, Anthropic Claude, Google Gemini)
- Metrics for measuring various aspects beyond accuracy (e.g. efficiency, bias, toxicity)
- Web UI for inspecting individual prompts and responses
- Web leaderboard for comparing results across models and benchmarks

## Documentation

Please refer to [the documentation on Read the Docs](https://crfm-helm.readthedocs.io/) for instructions on how to install and run HELM.

## Quick Start

Install the package from PyPI:

```sh
pip install crfm-helm
```

Run the following in your shell:

```sh
# Run benchmark
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10

# Summarize benchmark results
helm-summarize --suite my-suite

# Start a web server to display benchmark results
helm-server --suite my-suite
```

Then go to http://localhost:8000/ in your browser.



## Attribution

This NVIDIA fork of HELM is based on the original [Stanford CRFM HELM framework](https://github.com/stanford-crfm/helm). The original framework was created by the [Center for Research on Foundation Models (CRFM) at Stanford](https://crfm.stanford.edu/) and is licensed under the Apache License 2.0.

## Leaderboards

We maintain offical leaderboards with results from evaluating recent models on notable benchmarks using this framework. Our current flagship leaderboards are:

- [HELM Capabilities](https://crfm.stanford.edu/helm/capabilities/latest/)
- [HELM Safety](https://crfm.stanford.edu/helm/safety/latest/)
- [Holistic Evaluation of Vision-Language Models (VHELM)](https://crfm.stanford.edu/helm/vhelm/latest/)

We also maintain leaderboards for a diverse range of domains (e.g. medicine, finance) and aspects (e.g. multi-linguality, world knowledge, regulation compliance). Refer to the [HELM website](https://crfm.stanford.edu/helm/) for a full list of leaderboards.

## Papers

The HELM framework was used in the following papers for evaluating models.

- **Holistic Evaluation of Language Models** - [paper](https://openreview.net/forum?id=iO4LZibEqW), [leaderboard](https://crfm.stanford.edu/helm/classic/latest/)
- **Holistic Evaluation of Vision-Language Models (VHELM)** - [paper](https://arxiv.org/abs/2410.07112), [leaderboard](https://crfm.stanford.edu/helm/vhelm/latest/), [documentation](https://crfm-helm.readthedocs.io/en/latest/vhelm/)
- **Holistic Evaluation of Text-To-Image Models (HEIM)** - [paper](https://arxiv.org/abs/2311.04287), [leaderboard](https://crfm.stanford.edu/helm/heim/latest/), [documentation](https://crfm-helm.readthedocs.io/en/latest/heim/)
- **Image2Struct: Benchmarking Structure Extraction for Vision-Language Models** - [paper](https://arxiv.org/abs/2410.22456)
- **Enterprise Benchmarks for Large Language Model Evaluation** - [paper](https://arxiv.org/abs/2410.12857), [documentation](https://crfm-helm.readthedocs.io/en/latest/enterprise_benchmark/)
- **The Mighty ToRR: A Benchmark for Table Reasoning and Robustness** - [paper](https://arxiv.org/abs/2502.19412), [leaderboard](https://crfm.stanford.edu/helm/torr/latest/)
- **Reliable and Efficient Amortized Model-based Evaluation** - [paper](https://arxiv.org/abs/2503.13335), [documentation](https://crfm-helm.readthedocs.io/en/latest/reeval/)
- **MedHELM** - paper in progress, [leaderboard](https://crfm.stanford.edu/helm/medhelm/latest/), [documentation](https://crfm-helm.readthedocs.io/en/latest/reeval/)

The HELM framework can be used to reproduce the published model evaluation results from these papers. To get started, refer to the documentation links above for the corresponding paper, or the [main Reproducing Leaderboards documentation](https://crfm-helm.readthedocs.io/en/latest/reproducing_leaderboards/).

## Citation

If you use this software in your research, please cite the [Holistic Evaluation of Language Models paper](https://openreview.net/forum?id=iO4LZibEqW) as below.

```bibtex
@article{
liang2023holistic,
title={Holistic Evaluation of Language Models},
author={Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and Benjamin Newman and Binhang Yuan and Bobby Yan and Ce Zhang and Christian Alexander Cosgrove and Christopher D Manning and Christopher Re and Diana Acosta-Navas and Drew Arad Hudson and Eric Zelikman and Esin Durmus and Faisal Ladhak and Frieda Rong and Hongyu Ren and Huaxiu Yao and Jue WANG and Keshav Santhanam and Laurel Orr and Lucia Zheng and Mert Yuksekgonul and Mirac Suzgun and Nathan Kim and Neel Guha and Niladri S. Chatterji and Omar Khattab and Peter Henderson and Qian Huang and Ryan Andrew Chi and Sang Michael Xie and Shibani Santurkar and Surya Ganguli and Tatsunori Hashimoto and Thomas Icard and Tianyi Zhang and Vishrav Chaudhary and William Wang and Xuechen Li and Yifan Mai and Yuhui Zhang and Yuta Koreeda},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2023},
url={https://openreview.net/forum?id=iO4LZibEqW},
note={Featured Certification, Expert Certification}
}
```


## Attribution

# Attribution and Acknowledgments

## Original Project

This project is a fork of the **Holistic Evaluation of Language Models (HELM)** framework created by the Center for Research on Foundation Models (CRFM) at Stanford.

- **Original Repository**: [https://github.com/stanford-crfm/helm](https://github.com/stanford-crfm/helm)
- **Original Documentation**: [https://crfm.stanford.edu/helm](https://crfm.stanford.edu/helm)
- **Original Paper**: [Holistic Evaluation of Language Models](https://openreview.net/forum?id=iO4LZibEqW)
- **Original Authors**: Stanford CRFM Team
- **Original License**: Apache License 2.0

## Citation

If you use this software in your research, please cite the original HELM paper:

```bibtex
@article{liang2023holistic,
    title={Holistic Evaluation of Language Models},
    author={Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and Benjamin Newman and Binhang Yuan and Bobby Yan and Ce Zhang and Christian Alexander Cosgrove and Christopher D Manning and Christopher Re and Diana Acosta-Navas and Drew Arad Hudson and Eric Zelikman and Esin Durmus and Faisal Ladhak and Frieda Rong and Hongyu Ren and Huaxiu Yao and Jue WANG and Keshav Santhanam and Laurel Orr and Lucia Zheng and Mert Yuksekgonul and Mirac Suzgun and Nathan Kim and Neel Guha and Niladri S. Chatterji and Omar Khattab and Peter Henderson and Qian Huang and Ryan Andrew Chi and Sang Michael Xie and Shibani Santurkar and Surya Ganguli and Tatsunori Hashimoto and Thomas Icard and Tianyi Zhang and Vishrav Chaudhary and William Wang and Xuechen Li and Yifan Mai and Yuhui Zhang and Yuta Koreeda},
    journal={Transactions on Machine Learning Research},
    issn={2835-8856},
    year={2023},
    url={https://openreview.net/forum?id=iO4LZibEqW},
    note={Featured Certification, Expert Certification}
}
```

## Fork Information

- **Fork Maintainer**: NVIDIA
- **Fork Purpose**: Medical AI evaluation and EvalFactory integration

## License

This fork is released under the same Apache License 2.0 as the original project, in accordance with the original license terms.
