---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Checking Your Customization Job Metrics"
description: ""
---

After completing a customization job, you can monitor its performance through training and validation metrics. You can access these metrics in three ways:

1. Using the API
2. Through MLflow (optional)
3. Using Weights & Biases (optional)

<Note>

The time to complete this tutorial is approximately 10 minutes.

</Note>

## Prerequisites

<Markdown src="/snippets/_snippets/tutorials/prereqs.mdx" />
<Markdown src="/snippets/customizer/tutorials/_snippets/customizer-prereqs.mdx" />
### Tutorial-Specific Prerequisites

- Completed customization job with a valid job name
- (Optional) A job created with `spec.integrations.mlflow` and access to its configured MLflow tracking server

## Available Metrics

A customization job tracks every numeric metric its backend reports. The two you
will always find are:

- **Training Loss** (`train_loss`): recorded every step the training framework logs one
- **Validation Loss** (`val_loss`): recorded on every validation pass

Alongside those you will typically see `train_lr` (learning rate) and
`train_grad_norm`, plus whatever else the algorithm produces — a DPO job also
reports `train_preference_loss` and `val_accuracy`, for example.

An algorithm that scores validation on something other than a loss reports no
`val_loss`, and the series stays empty rather than filling with zeros.

### How Metrics Are Named

Each metric is named `<phase>_<metric>`, where the phase is `train` or `val` and
the metric keeps whatever name the training framework gave it. A metric reported
during both training and validation therefore stays separate: DPO's `accuracy`
becomes `train_accuracy` and `val_accuracy` rather than one interleaved series.

`train_loss` and `val_loss` are simply what this rule produces for a metric named
`loss`.

### Where Metrics Appear

Each metric shows up in two places in a training task's `status_details`:

- **The latest value**, as a top-level field under its full name (`train_loss`,
  `train_lr`, ...). Present only when the metric was actually reported, so a
  missing field means no value rather than a zero.
- **The full history**, under `metrics`, as a list of `{step, epoch, value}`
  points per metric. This is what the loss curves in the UI are drawn from.

Non-numeric values a framework emits alongside the scalars — histograms, tables,
nested dictionaries — are not charted and do not appear in either place.

### How Often Metrics Are Sent

Recording and sending are separate, and only the second is throttled.

Every value your training framework logs is recorded, at full resolution. What
is limited is how often the accumulated set is *sent* to the platform, because
each update carries the whole history and is written more than once server-side.
Reports are buffered in memory and sent at most once every ten seconds by
default; a buffered report loses nothing, since its points travel with the next
one that goes. The final values are always sent when the job finishes.

The practical effect is that a chart may lag the run by a few seconds, and the
curve you eventually read is complete.

### Controlling Metric Detail

Two optional fields under `schedule.progress_reporting` change this. Most jobs
need neither.

- **`min_report_interval_seconds`** (default `10`): the least time between
  updates reaching the platform. Raise it to spend less training time on
  reporting; lower it — or set `0`, which sends every logged step — for a
  progress bar that moves more often. Recording is unaffected either way.
- **`time_series_metrics`**: which metrics keep a *history* rather than only a
  latest value. Names are the full ones you read back, so they carry the phase
  prefix, and glob patterns are accepted: `["*_loss", "*_lr"]`. Omit the field to
  take your backend's default, which keeps the loss, learning rate and gradient
  norm. Use `["*"]` to keep a history of everything the backend reports.

A metric left out of `time_series_metrics` is still reported as a latest value
on every update — only its history is dropped. That keeps throughput counters
like `train_tps` costing one number instead of several hundred points.

Set them in the `schedule` block when you create the job:

```python
schedule={
    "epochs": 3,
    "progress_reporting": {
        "min_report_interval_seconds": 30,
        "time_series_metrics": ["*_loss", "*_lr"],
    },
}
```

<Note>

These control what the *platform* stores. They are unrelated to the training
framework's own logging cadence — Unsloth's `schedule.logging_steps`, for
example — which decides how often your framework produces a value in the first
place, and drives its stdout and any W&B or MLflow run.

</Note>

## Viewing Your Metrics

### Using the API

Get job status and training metrics through the platform Jobs service:

```python
import os
from nemo_platform import NeMoPlatform

client = NeMoPlatform(
    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
    workspace="default",
)

# Get job status with metrics
job_name = "my-sft-job"
status = client.jobs.get_status(name=job_name, workspace="default")

print(f"Job: {status.name}")
print(f"Status: {status.status}")

# Check training step progress
for step in status.steps or []:
    if step.name == "training":
        for task in step.tasks or []:
            details = task.status_details or {}
            print(f"Training Phase: {details.get('phase')}")
            print(f"Step: {details.get('step')}/{details.get('max_steps')}")
            print(f"Epoch: {details.get('epoch')}/{details.get('num_epochs')}")
            print(f"Training Loss: {details.get('train_loss')}")
            print(f"Validation Loss: {details.get('val_loss')}")
            print(f"Learning Rate: {details.get('train_lr')}")
            print(f"Gradient Norm: {details.get('train_grad_norm')}")
```

To read the curves rather than the latest values, use the `metrics` payload. It
carries every metric the job reported, so iterating it picks up backend-specific
ones without naming them in advance:

```python
for name, points in (details.get("metrics") or {}).items():
    if not points:
        continue
    print(f"{name}: {len(points)} points, latest {points[-1]['value']}")
```

### Using MLflow

If your customization job was created with an `integrations.mlflow` configuration (see [MLflow Integration](/documentation/customizer-reference/manage-customization-jobs/customization-job-reference#mlflow-integration)):

1. Access the MLflow UI at the configured `tracking_uri`
2. Locate the configured `experiment_name` (defaults to the output model name)
3. Find the configured run `name` (defaults to the customization job ID)
4. View detailed metrics, including training and validation loss curves, under the "Metrics" tab

<Note>

MLflow tracking is requested per job through `spec.integrations.mlflow`; it is not automatically enabled for every job in a cluster. The tracking server can be selected with `tracking_uri` in the job spec or the platform-side `MLFLOW_TRACKING_URI` environment variable. Contact your administrator if you need access to that server.

</Note>

### Using Weights & Biases

If your customization job was created with W&B integration enabled (see [Weights & Biases Integration](/documentation/customizer-reference/manage-customization-jobs/create-a-customization-job)):

1. Go to [wandb.ai](https://wandb.ai/home) and navigate to your project
2. Find the run corresponding to your customization job
3. View training and validation loss curves, learning rate schedules, and other metrics under the run's dashboard

```python
from nemo_automodel_plugin.schema import AutomodelJobInput

# Create an Automodel job with W&B integration
spec = AutomodelJobInput(
    model="default/llama-3-2-1b",
    dataset={"training": "default/my-dataset"},
    training={"training_type": "sft", "finetuning_type": "lora"},
    schedule={"epochs": 3},
    batch={"global_batch_size": 16, "micro_batch_size": 1},
    optimizer={"learning_rate": 1e-4},
    integrations={
        "wandb": {
            "project": "my-finetuning-project",
            "entity": "my-team",
            "tags": ["fine-tuning", "llama"],
            "api_key_secret": "my-wandb-key",
        }
    },
)

job = client.customization.automodel.jobs.create(
    name="my-wandb-job",
    workspace="default",
    spec=spec,
)

print(f"Submitted job: {job.job.name}")
```

The `api_key_secret` field references a stored secret containing your `WANDB_API_KEY`.
Use the secret name (e.g., `"my-wandb-key"`) to resolve it from the request workspace.
To create the secret, see [Weights & Biases Keys](/documentation/get-started/core-concepts/manage-secrets).

Then view your results at [wandb.ai](https://wandb.ai/home) under your project.
![W&B charts example](../_images/wandb_charts_example.png)

<Note>

The W&B integration is optional and must be configured when [creating the customization job](/documentation/customizer-reference/manage-customization-jobs/create-a-customization-job). When enabled, training metrics are sent to W&B using your API key. While we encrypt your API key and don't log it internally, please review W&B's terms of service before use.

</Note>
