---
title: "Checking Your Customization Job Metrics"
description: ""
---

After completing a customization job, you can monitor its performance through training and validation metrics. You can access these metrics in three ways:

1. Using the API
2. Through MLflow (optional)
3. Using Weights & Biases (optional)

<Note>

The time to complete this tutorial is approximately 10 minutes.

</Note>

## Prerequisites

<Markdown src="/snippets/_snippets/tutorials/prereqs.mdx" />
<Markdown src="/snippets/customizer/tutorials/_snippets/customizer-prereqs.mdx" />
### Tutorial-Specific Prerequisites

- Completed customization job with a valid job name
- (Optional) A job created with `spec.integrations.mlflow` and access to its configured MLflow tracking server

## Available Metrics

Each customization job tracks two key metrics:

- **Training Loss**: Calculated during training, logged every 10 steps (default, configurable via hyperparameters)
- **Validation Loss**: Calculated during validation, logged at each validation interval

## Viewing Your Metrics

### Using the API

Get job status and training metrics through the platform Jobs service:

```python
import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

# Get job status with metrics
job_name = "my-sft-job"
status = client.jobs.get_status(name=job_name, workspace="default")

print(f"Job: {status.name}")
print(f"Status: {status.status}")

# Check training step progress
for step in status.steps or []:
    if step.name == "training":
        for task in step.tasks or []:
            details = task.status_details or {}
            print(f"Training Phase: {details.get('phase')}")
            print(f"Step: {details.get('step')}/{details.get('max_steps')}")
            print(f"Epoch: {details.get('epoch')}/{details.get('num_epochs')}")
            print(f"Training Loss: {details.get('train_loss')}")
            print(f"Validation Loss: {details.get('val_loss')}")
            print(f"Learning Rate: {details.get('lr')}")
            print(f"Gradient Norm: {details.get('grad_norm')}")
```

The response includes training progress and metrics including loss, learning rate, and validation loss.

### Using MLflow

If your customization job was created with an `integrations.mlflow` configuration (see [MLflow Integration](/documentation/customizer-reference/manage-customization-jobs/customization-job-reference#mlflow-integration)):

1. Access the MLflow UI at the configured `tracking_uri`
2. Locate the configured `experiment_name` (defaults to the output model name)
3. Find the configured run `name` (defaults to the customization job ID)
4. View detailed metrics, including training and validation loss curves, under the "Metrics" tab

<Note>

MLflow tracking is requested per job through `spec.integrations.mlflow`; it is not automatically enabled for every job in a cluster. The tracking server can be selected with `tracking_uri` in the job spec or the platform-side `MLFLOW_TRACKING_URI` environment variable. Contact your administrator if you need access to that server.

</Note>

### Using Weights & Biases

If your customization job was created with W&B integration enabled (see [Weights & Biases Integration](/documentation/customizer-reference/manage-customization-jobs/create-a-customization-job)):

1. Go to [wandb.ai](https://wandb.ai/home) and navigate to your project
2. Find the run corresponding to your customization job
3. View training and validation loss curves, learning rate schedules, and other metrics under the run's dashboard

```python
from nemo_automodel_plugin.schema import AutomodelJobInput

# Create an Automodel job with W&B integration
spec = AutomodelJobInput(
    model="default/llama-3-2-1b",
    dataset={"training": "default/my-dataset"},
    training={"training_type": "sft", "finetuning_type": "lora"},
    schedule={"epochs": 3},
    batch={"global_batch_size": 16, "micro_batch_size": 1},
    optimizer={"learning_rate": 1e-4},
    integrations={
        "wandb": {
            "project": "my-finetuning-project",
            "entity": "my-team",
            "tags": ["fine-tuning", "llama"],
            "api_key_secret": "my-wandb-key",
        }
    },
)

job = client.customization.automodel.jobs.create(
    name="my-wandb-job",
    workspace="default",
    spec=spec,
)

print(f"Submitted job: {job.job.name}")
```

The `api_key_secret` field references a stored secret containing your `WANDB_API_KEY`.
Use the secret name (e.g., `"my-wandb-key"`) to resolve it from the request workspace.
To create the secret, see [Weights & Biases Keys](/documentation/get-started/core-concepts/manage-secrets).

Then view your results at [wandb.ai](https://wandb.ai/home) under your project.
![W&B charts example](../_images/wandb_charts_example.png)

<Note>

The W&B integration is optional and must be configured when [creating the customization job](/documentation/customizer-reference/manage-customization-jobs/create-a-customization-job). When enabled, training metrics are sent to W&B using your API key. While we encrypt your API key and don't log it internally, please review W&B's terms of service before use.

</Note>
