---
title: "Credential Propagation"
description: ""
---
How credentials flow through the system when NeMo Platform runs jobs or serves inference.

## Managed Job Workload Identity

When a user submits a job (customization, evaluation, data generation), the job
runs in a workload container that needs to call NeMo Platform APIs to download
datasets, upload results, and read secrets. Managed job credentials are
propagated through workload identity token exchange, not by using a serialized
principal as an API credential.

The flow:

1. The user submits a job via the API. The control plane authorizes job creation
   and records the job metadata.
2. When workload identity token exchange is enabled, the managed backend creates
   a workload with `NMP_WORKLOAD_IDENTITY_TOKEN_FILE` set to a subject-token file
   path.
3. The backend owns that subject-token file. Kubernetes uses a projected service
   account token volume, while Docker uses a controller-managed volume that is
   refreshed by the jobs controller.
4. The SDK in the workload detects `NMP_WORKLOAD_IDENTITY_TOKEN_FILE`, reads the
   subject token, discovers the workload token exchange endpoint, and exchanges
   the subject token for a NeMo Platform access token using OAuth 2.0 Token
   Exchange (RFC 8693).
5. API calls from the workload use the exchanged access token. When that access
   token nears expiry, the SDK rereads the subject-token file and performs
   another exchange.

Job containers still need to run inside the platform trust boundary. Network
policies and gateway configuration should prevent workloads from reaching
internal endpoints they do not need and should prevent external callers from
forging trusted identity headers. For the full architecture, see [Security
Model](/documentation/access-control/security-model#job-credential-propagation).

## Workload Identity Token Files

For external IdP-backed workload identity, managed backends inject
`NMP_WORKLOAD_IDENTITY_TOKEN_FILE`. The file contains a workload identity
subject token, not a final NeMo API access token.

The SDK reads that file immediately before OAuth 2.0 Token Exchange (RFC 8693),
exchanges the subject token at the discovered IdP token endpoint, caches the
returned access token until near expiry, and then rereads the file for the next
exchange.

Backend ownership:

- Kubernetes uses a projected service account token volume. Kubelet rotates the
  file.
- Docker uses a dedicated controller-managed workload identity volume. The
  Docker backend writes and refreshes the file.

Users must not provide `NEMO_WORKLOAD_TOKEN`, `NEMO_WORKLOAD_TOKEN_FILE`, or
`NMP_WORKLOAD_IDENTITY_TOKEN_FILE` in managed job requests. Direct SDK users may
set `NMP_WORKLOAD_IDENTITY_TOKEN_FILE` only when they own the refreshed subject
token file.

This follows the same shape as common cloud SDKs: AWS uses
`AWS_WEB_IDENTITY_TOKEN_FILE`, Azure uses `AZURE_FEDERATED_TOKEN_FILE`, and
Google Workload Identity Federation uses `GOOGLE_APPLICATION_CREDENTIALS` to
point at a credential config that references a projected token file.

## Inference Auth Context

When a model is deployed as an inference endpoint, incoming requests are authenticated at the gateway or service level before reaching the model. The user's auth context is evaluated before the request reaches the model container.

## Trust Implications

The workload identity subject-token file is not the final NeMo Platform access
token, but it can be exchanged for one. Protect it as credential material, mount
it read-only into the workload container where possible, and keep its lifetime
short.

<Warning>

Ensure Kubernetes RBAC prevents unauthorized access to job pods. Network policies should restrict which pods can reach NeMo Platform internal endpoints.

</Warning>
## Related

- [Security Model](/documentation/access-control/security-model) — Trust boundaries and job credential propagation.
- [Auth Configuration](/documentation/access-control/deployment/configuration) — Platform auth configuration.
